A Search-Based Benchmark Generator for Constrained Combinatorial Testing Models

In combinatorial testing, the generation of bench-marks that meet specific constraints and complexity requirements is essential for the rigorous assessment of testing tools. In previous work, we presented BenCIGen, a benchmark generator for combinatorial testing, which had the limitation of often discarding input parameter models (IPMs) that fail to meet the targeted requirements in terms of ratio (i.e., the number of valid tests, or tuples, over the total number of possible tests or tuples) and solvability. This paper presents an extension to BenCIGen, BenCIGenS, that integrates a search-based generation approach, aimed at reducing the number of discarded IPMs and enhancing the efficiency of benchmark generation. Instead of rejecting IPMs that do not fulfill the desired characteristics, BenCIGenS employs search-based techniques that iteratively mutate IPMs, optimizing them accordingly to a fitness function that measures their distance from target requirements. Experimental results demonstrate that BenCIGenS generates a significantly higher proportion of benchmarks adhering to specified characteristics, with sometimes a reduced computation time. This approach not only improves the generation of suitable benchmarks but also enhances the tool's overall effectiveness.