On the Effectiveness of Combinatorial Interaction Testing: A Case Study
2017 (English)In: Proceedings - 2017 IEEE International Conference on Software Quality, Reliability and Security Companion, QRS-C 2017, 2017, p. 69-76Conference paper, Published paper (Refereed)
Abstract [en]
Combinatorial interaction testing (CIT) stands as one of the efficient testing techniques that have been used in different applications recently. The technique is useful when there is a need to take the interaction of input parameters into consideration for testing a system. The key insight the technique is that not every single parameter may contribute to the failure of the system and there could be interactions among these parameters. Hence, there must be combinations of these input parameters based on the interaction strength. This technique has been used in many applications to assess its effectiveness. In this paper, we are addressing the effectiveness of CIT for a real-world case study using model-based mutation testing experiments. The contribution of the paper is threefold: First we introduce an effective testing application for CIT; Second, we address the effectiveness of increasing the interaction strength beyond the pairwise (i.e., interaction of more than two parameters); Third, model-based mutation testing is used to mutate the input model of the program in contrast to the traditional code-based mutation testing process. Experimental results showed that CIT is an effective testing technique for this kind of application. In addition, the results also showed the usefulness of model-based mutation testing to assess CIT applications. For the subject of this case study, the results also indicate that 3-way test suite (i.e., interaction of three parameters) could detect new faults that can not be detected by pairwise. © 2017 IEEE.
Place, publisher, year, edition, pages
2017. p. 69-76
Keywords [en]
Combinatorial interaction testing, fault detection, model-based mutation testing, Software testing, Application programs, C (programming language), Computer software selection and evaluation, Software reliability, Testing, Effective testing, Interaction strength, Mutation testing, Single parameter, Testing technique, Three parameters, Usefulness of models
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kau:diva-86846DOI: 10.1109/QRS-C.2017.20Scopus ID: 2-s2.0-85034424921ISBN: 9781538620724 (print)OAI: oai:DiVA.org:kau-86846DiVA, id: diva2:1608369
Conference
2017 IEEE International Conference on Software Quality, Reliability and Security Companion, QRS-C 2017, 25 July 2017 through 29 July 2017
Note
Conference code: 129875; Cited By :11; Export Date: 3 November 2021; Conference Paper; References: Kacker, R.N., Richard Kuhn, D., Lei, Y., Lawrence, J.F., Combinatorial testing for software: An adaptation of design of experiments (2013) Measurement, 46 (9), pp. 3745-3752; Ahmed, B.S., Abdulsamad, T.S., Potrus, M.Y., Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the cuckoo search algorithm (2015) Information and Software Technology, 66, pp. 13-29; Cohen, D.M., Dalal, S.R., Parelius, J., Patton, G.C., The combinatorial design approach to automatic test generation (1996) IEEE Softw, 13; Nie, C., Leung, H., A survey of combinatorial testing (2011) ACM Computing Surveys, 43 (2), pp. 1-29; Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J., Ipog: A general strategy for t-way software testing 4th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, pp. 549-556. , IEEE Computer Society. 1253335 549-556; Hervieu, A., Marijan, D., Gotlieb, A., Baudry, B., Practical minimization of pairwise-covering test configurations using constraint programming (2016) Information and Software Technology, 71, pp. 129-146; Zamli, K.Z., Klaib, M.F.J., Younis, M.I., Isa, N.A.M., Abdullah, R., Design and implementation of a t-way test data generation strategy with automated execution tool support (2011) Information Sciences, 181 (9), pp. 1741-1758; Yuan, X., Cohen, M.B., Memon, A.M., Gui interaction testing: Incorporating event context (2011) IEEE Transactions on Software Engineering, 37 (4), pp. 559-574; Cheng, C.S., Orthogonal arrays with variable numbers of symbols (1980) The Annals of Statistics, 8 (2), pp. 447-453; Colbourn, C., Ḱeri, G., Soriano, P.R., Schlage-Puchta, J.-C., Covering and radius-covering arrays: Constructions and classification (2010) Discrete Applied Mathematics, 158 (11), pp. 1158-1180; Hartman, A., Software and hardware testing using combinatorial covering suites volume 34 of graph theory (2005) Combinatorics and Algorithms, , Springer US; Cawse, J.N., (2003) Experimental Design for Combinatorial and High Throughput Materials Development, , Wiley-Interscience; Shasha, D.E., Kouranov, A.Y., Lejay, L.V., Chou, M.F., Coruzzi, G.M., Using combinatorial design to study regulation by multiple input signals: A tool for parsimony in the post-genomics era (2001) Plant Physiology, 127 (4), pp. 1590-1594; Hoskins, D.S., Colbourn, C.J., Montgomery, D.C., Software performance testing using covering arrays: Efficient screening designs with categorical factors (2005) Proceedings of the 5th International Workshop on Software and Performance, pp. 131-136. , WOSP '05, (New York, NY, USA), ACM; Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K., A systematic review of the application and empirical investigation of search-based test case generation (2010) IEEE Transactions on Software Engineering, 36 (6), pp. 742-762; Colbourn, C.J., McClary, D.W., Locating and detecting arrays for interaction faults (2008) Journal of Combinatorial Optimization, 15 (1), pp. 17-48; Fraser, G., Gargantini, A., Generating minimal fault detecting test suites for boolean expressions (2010) Third International Conference on Software Testing, Verification, and Validation Workshops, pp. 37-45; Cohen, M.B., Dwyer, M.B., Shi, J., Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach (2008) IEEE Transactions on Software Engineering, 34 (5), pp. 633-650; Petke, J., Yoo, S., Cohen, M.B., Harman, M., Efficiency and early fault detection with lower and higher strength combinatorial interaction testing (2013) Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 26-36. , ESEC/FSE 2013, (New York, NY, USA), ACM; Da Mota, S.N.P.A., Carmo, M.I.D., McGregor, J.D., De Almeida, E.S., De Lemos, M.S.R., A systematic mapping study of software product lines testing (2011) Information and Software Technology, 53 (5), pp. 407-423; Sulaiman, D.R., Ahmed, B.S., Using the combinatorial optimization approach for dvs in high performance processors (2013) International Conference on Technological Advances in Electrical Electronics and Computer Engineering (TAEECE), pp. 105-109; Sahib, M.A., Ahmed, B.S., Potrus, M.Y., Application of combinatorial interaction design for dc servomotor pid controller tuning (2014) Journal of Control Science and Engineering, 2014, p. 7; Ahmed, B.S., Sahib, M.A., Gambardella, L.M., Afzal, W., Zamli, K.Z., Optimum design of PI D controller for an automatic voltage regulator system using combinatorial test design (2016) PLOS ONE, 11 (11), pp. 1-20; Ahmed, B.S., Zamli, K.Z., Lim, C.P., Application of particle swarm optimization to uniform and variable strength covering array construction (2012) Applied Soft Computing, 12 (4), pp. 1330-1347; Ahmed, B.S., Zamli, K.Z., A variable strength interaction test suites generation strategy using particle swarm optimization (2011) Journal of Systems and Software, 84 (12), pp. 2171-2185; Ahmed, B.S., Sahib, M.A., Potrus, M.Y., Generating combinatorial test cases using simplified swarm optimization (sso) algorithm for automated gui functional testing (2014) Engineering Science and Technology, An International Journal, 17 (4), pp. 218-226; Bryce, R.C., Colbourn, C.J., Test prioritization for pairwise interaction coverage (2005) Proceedings of the 1st International Workshop on Advances in Model-based Testing, pp. 1-7. , A-MOST '05, (New York, NY, USA), ACM; Qu, X., Cohen, M.B., Woolf, K.M., Combinatorial interaction regression testing: A study of test case generation and prioritization (2007) IEEE International Conference on Software Maintenance, ICSM, pp. 255-264. , 2007 IEEE Computer Society; Yilmaz, C., Cohen, M.B., Porter, A., Covering arrays for efficient fault characterization in complex configuration spaces (2004) ACM SIGSOFT Software Engineering Notes, 29 (4), pp. 45-54. , 1007519; Offutt, J., A mutation carol: Past, present and future Information and Software Technology, 53 (10), pp. 1098-1107. , 2011. Special Section on Mutation Testing; Jia, Y., Harman, M., An analysis and survey of the development of mutation testing (2011) IEEE Trans. Softw. Eng, 37, pp. 649-678. , Sept; Kuhn, D.R., Wallace, D.R., Gallo, A.M.J., Software fault interactions and implications for software testing (2004) IEEE Transactions on Software Engineering, 30 (6), pp. 418-421; Belli, F., Budnik, C.J., Hollmann, A., Tuglular, T., Wong, W.E., Model-based mutation testing-approach and case studies (2016) Science of Computer Programming, 120, pp. 25-48; Ahmed, B.S., Gambardella, L.M., Afzal, W., Zamli, K.Z., Handling constraints in combinatorial interaction testing in the presence of multi objective particle swarm and multithreading (2017) Information and Software Technology, 86, pp. 20-36; Ahmed, B.S., Zamli, K.Z., Lim, C.P., Constructing a t-way interaction test suite using the particle swarm optimization approach (2012) International Journal of Innovative Computing, Information and Control (IJICIC), 8 (1), pp. 431-452
2021-11-032021-11-032025-10-16Bibliographically approved