Combinatorial Testing tools in Software Testing
Combinatorial testing is a testing technique in which multiple combinations of the input parameters are used to perform testing of the software product. The aim is to ensure that the product is bug-free and can handle different combinations or cases of the input configuration. One of the most commonly used combinatorial testing methods is the pairwise testing technique which involves testing all pairs of input variable values.
Why Do We Need Combinatorial Testing Tools?
Combinatorial testing tools are easy-to-use test case generators that allow to provide the input and constraints to the input parameter model and then generate the test configurations using the model.
Since combinatorial testing follows a complex procedure and it can be a tedious task to manually perform this testing on many input parameters, we, therefore, use combinatorial testing tools. Not only are these tools easy to use with many input parameters, but they can also add constraints in the input parameters and generate test configurations accordingly. There are numerous tools available on the internet to perform combinatorial testing. In this article, we will discuss a few such tools that are available for free on the internet to generate test configurations.
Combinatorial Testing Tools
In this section, we will be discussing some easy to use, free, and popular combinatorial testing tools.
1. Pairwise Pict Online
This tool is the simplest to use because we just have to write the test factors and constraints (if any) and the test configurations are generated. This tool allows us to write the constraints using an If-Then format as shown below. It is a good tool for beginners to start with combinatorial testing.
Click to Try: Pairwise Pict Online
2. CAGen for covering array generator
This tool gives a bit more functionality and features as compared to the previous tool. Firstly, it allows us to create multiple input parameter models instead of just one like in the previous tool. Also, we can save these models as well and export them to a different type of files. It also gives the user an option to import input configuration models. Another advantage of this tool is that it can use three different algorithms (FIPOG, FIPOG-F, FIPOG-F2) to generate the combinatorial object array.
In this tool, factors and levels have to be added one by one. But before that, these steps are to be followed:
- Click here to open the CAGen tool.
- Create a new workspace and give it a relevant name.
- Now click on the workspace created and you will be able to see an Input Parameter Model on your screen.
- Enter factors, its levels, and the cardinality (number of levels) in this model.
- After all factors with their levels have been added to the input parameter model, click on the ‘generate button’ on the left panel to generate test configurations.
This tool requires a user account login. After you have logged in, you can use this tool for free. Just like the CAGen tool, Pairwiser too requires one by one entering of all factors and their values. Similarly, constraints can be entered too in the format of if-then.
Unlike other tools, Pairwiser provides a wide range of functionalities and features that one can explore in combinatorial testing.
- The tool comes with a visualizer that shows the coverage of each test case that is added to the test suite.
- Another amazing feature is the generation of test scripts. If you want to perform combinatorial testing for your application, then you can simply use these test scripts for detailed evaluation. The test scripts can be modified according to our needs using numerous options available in the tool.
- Pairwiser is quite a popular tool in the case of combinatorial testing (pairwise-testing) as they offer a faster pairwise algorithm in the market.
Click to Try: Pairwiser tool
Another intuitive tool for performing combinatorial testing is testcover.com where factors, values, and constraints are simply written in the editor, and test configurations are generated. This tool has an extremely fast and efficient algorithm and can generate about 15 test cases in 1 second. Another advantage is the tool’s easy generator requests where we just have to write the factors and values in new lines, that’s it! Testcover.com also offers the use of functions to add constraints. This tool is a good option for software engineers and testers.
The following things should be noted for using this tool:
- First, all factors are written
- Values for a factor are written in the same line separated by spaces
- Constraints are written at the end using ‘with’ keyword and parenthesis
Click to Try: Testcover.com
Advantages Of Combinatorial Testing
- One main advantage is the reduction in the number of test cases for execution
- Since the number of test cases is reduced, the time taken to execute the test suite comes out to be less due to its small size
- Test coverage is increased (up to 100 percent)
- The increased coverage also leads to an increase in the bug yield ratio
- Enhanced quality of the product as test coverage is increased
- The overall cost of testing the product is decreased
Disadvantages Of Combinatorial Testing
- In combinatorial testing, if the selection of values for input variables is not done correctly, then the resulting test configurations are futile
- Similar, the above point is applicable while selecting and applying the constraints
- Lack of understanding of the input parameters can result in an inappropriate combinatorial object array
- Manually performing combinatorial testing is a costly and tedious task
- It is a time-consuming process to generate test configurations even when automated tools are used