Mutation Testing in Java (Using Jumble)

Mutation testing is a white-box testing technique, which changes certain portions of the code to reveal possible faults. On a very high level, it is the process of rewriting the source code with some known or possible bugs or differences to the actual code in small ways in order to remove the redundancies in the source code. The goal of mutation testing is to evaluate the coverage of your tests. There are many mutation testing tool for Java.

1. Jumble
2. PIT
3. µJava
4. Jester

Jumble

This post deals with Jumble Mutation testing Tool in Java. Jumble is a mutation testing tool which changes the Java code at byte code level. It runs a unit test case and applies certain set of mutations.

• Passed Mutations: Changes in code which do not affect the final output
• Failed Mutations: Changes in code which change the final output

Score : It is the percentage of passed mutations out of the total mutations. It is used to assess the validity of our test suite.
Installation
Download Eclipse IDE and install Jumble to Eclipse.

Simple Code Snippet in Java

int index = 0;
while(true)
{
   index++;
   if (index == 10) 
        break;
}

Mutated Code in JAVA

int index = 0;
while (true)
{
   index++;
   if (index >= 10) 
        break;
}

Explanation: The mutant code above will pass the Jumble test because change in == to >= does not affect the output of the code. Execution will stop when index == 10 and since we are increasing value by 1 and index starts from 0 so the output will remain same.

Example with Jumble
The code written below has been tested with Jumble plugin in Eclipse. The code detects the first occurrence of a duplicate and returns the value to the calling function. The program is flawed in many ways which you can try figuring out.

// Java program to illustrate mutation Testing
// The code detects the first occurrence of a
// duplicate and returns the value to the calling
// function
package testPackage;

import java.util.Arrays;
import java.util.List;
public class SampProg
{
    protected int repeatedNumber(final List a)
    {
        int len = a.size(),i,dup = -1;
        int[] arr = new int[len];
        for (i=0; i<len; i++)
        {
            arr[i] = a.get(i);
        }

        Arrays.sort(arr);
        try
        {
            for (i=1; i<len; i++)
            {
                if(arr[i] == arr[i-1])
                {
                    dup = arr[i];
                    break;
                }
            }
        }
        catch(Exception e)
        {
            System.out.println(e.getMessage());
        }
        return dup;
    }
}

After writing the program we create test cases using JUnit in Java. The output obtained on performing Jumble Analysis is given below:

Mutating testPackage.SampProg
Tests: testPackage.SampProgTest
Mutation points = 11, unit test time limit 2.94s
M FAIL: (testPackage.SampProg.java:8): -1 -> 1
M FAIL: (testPackage.SampProg.java:10): 0 -> 1
.M FAIL: (testPackage.SampProg.java:10): negated conditional
M FAIL: (testPackage.SampProg.java:16): 1 -> 0
M FAIL: (testPackage.SampProg.java:18): 1 -> 0
M FAIL: (testPackage.SampProg.java:18): - -> +
M FAIL: (testPackage.SampProg.java:18): negated conditional
M FAIL: (testPackage.SampProg.java:16): += -> -=
M FAIL: (testPackage.SampProg.java:16): negated conditional
.
Jumbling took 7.595s
Score: 18%

Interpreting the Result

1. In the first line of M FAIL in the output -1 was changed to 1. In the main program, if there are no duplicates we expect the value to be returned as -1. but instead we get the value 1. We can fix the mutation by enabling checks for value mutation.
2. Next, we can see “negated conditional” on the 3rd, 7th and 9th line where condition has been negated. Here however we can’t fix the error for this program.
3. Other cases where the mutation failed to give proper output is when operators are changed to other operators. In this and many algorithmic codes, mutations related to operator changes cannot be fixed. However mutation testing gives brief idea in some mutations as to where the test case is weak or there is an error in the source code. This link gives mutations performed by jumble