How to know testcases where code fails?

In Competitive Programming, it’s quite often to write code that works on the given sample test cases but on submitting its end up with a WA(Wrong answer) verdict. This is where things get frustrating and not know on which test case the solution fails. In such situations there are a few things that a programmer can do:

1. Check manually for corner cases
2. Stress Testing

Checking for Corner Cases:

In some problem-statement, it has corner test cases where the programmer needs to add/edit some lines of code to make sure the program works fine for those cases as well. For example, in a problem where it needs to find the Minimum Positive Integer in an array and if a solution something like this:

int mn = arr[0]
for i : 1 -> arr_size:
    if (arr[i] < mn)
          mn = arr[i]
print mn

Explanation:
The above logic would not give an AC(Accepted) verdict and it is easy to figure out the problem here. One of the test cases where this code will fail is when the array is arr[] = {2, -1, 3, 4, 5}. The above algorithm Output is -1 But the correct Output is 2.
For finding the minimum positive integer, it needs to edit the code to make sure negative integers must not be included.

Stress Testing:

Stress Testing is a code testing technique that determines the robustness of code by testing beyond the limits of normal operation. Suppose a problem which has a naive solution(slow) and an optimal solution(fast) gets a TLE (Time Limit Exceeded) verdict on submitting the brute force solution, so come up with the optimal solution but on submitting it we get WA on some or all test cases.

Now the first thing to do here would be to think of some test cases where the solution may not work and check the solution on them. If unable to think of the test cases then there is one more way and that is stress testing. 

In stress testing, the idea is to generate random test cases and check the output of the brute force solution and the optimal solution. Though the brute force solution is slow still it’s correct while the optimal solution is faster but it’s wrong in some test cases. This allows us to check if the optimal solution’s output of some test case is correct or not without manually checking, it can simply check if the output of Naive Solution agrees with the Optimal Solution’s output. The checking is done automatically and also the correctness of code on thousands depends on the complexity of the Naive Solution of test cases in seconds, so the probability of finding the test case where our code fails becomes way high.

So, perform the below steps to check the solution on random input by running an infinite loop:

1. Generate random input (click here to know how)
2. Store output of brute force and optimal solution
3. If the two outputs are (equal) then print correct
4. Else print the input and break the loop

If the loop runs for some time without breaking i.e., all the outputs are correct then either check for larger input or try submitting your solution.

Example:

Problem Statement: Given an array arr[] of N positive Integers, the task is to find the maximum pairwise product of elements in an array.

Input: arr[] = [2, 3, 4, 9, 4, 7]
Output: 63 
Explanation:
The maximum pairwise product of the array is 9 * 7 = 63.

Input: arr[] = [-20, -50, 1, 2]
Output: 1000
Explanation:
The maximum pairwise product of the array is (-20) * (-50) = 1000.

Below is the implementation of stress testing for the above problem:

C++

// C++ program to illustrate the stress
// testing for the above problem
#include <bits/stdc++.h>
using namespace std;

// Function that implements the Naive
// Solution for the given problem
int maxProductNaive(int a[], int n)
{
    int ans;
    for (int i = 0; i < n; i++) {
        for (int j = i + 1; j < n; j++) {
            if (i == 0 && j == 1)
                ans = a[i] * a[j];
            else if (a[i] * a[j] > ans)
                ans = a[i] * a[j];
        }
    }
    return ans;
}

// Function that implements the Optimal
// Solution for the given problem
int maxProductOptimal(int a[], int n)
{
    // Sort the given array
    sort(a, a + n);

    // Find the maximum product
    int ans = a[n - 1] * a[n - 2];

    // Return the product
    return ans;
}

// Function that performs the
// Stress Testing
void test()
{
    // Seeding random number generator
    // to get uniques values
    srand(time(0));

    while (1) {

        // Generate values for n
        // from 2 to 10
        int n = rand() % 10 + 2;
        int a[n];

        // Iterate over array a[]
        for (int i = 0; i < n; i++) {

            // Subtracting -5 will
            // generate -ve integers
            a[i] = rand() % 10 - 5;
        }

        // Solution using Naive Approach
        int ans_brute
            = maxProductNaive(a, n);

        // Solution using Optimal Approach
        int ans_optimal
            = maxProductOptimal(a, n);

        // If ans is correct
        if (ans_brute == ans_optimal)
            cout << "Correct\n";

        // Else print the WA Test Case
        else {
            cout << "Wrong Answer\n";

            cout << n << endl;

            // Print the array a[]
            for (int i = 0; i < n; i++)
                cout << a[i] << " ";

            cout << "\nCorrect Output: "
                 << ans_brute << endl;

            cout << "Wrong Output: "
                 << ans_optimal << endl;
            break;
        }
    }
}

// Driver Code
int main()
{
    // Function Call for Stress Testing
    test();
    return 0;
}

Java

import java.util.Arrays;
import java.util.Random;

public class StressTesting {

    // Function that implements the Naive Solution for the given problem
    static int maxProductNaive(int[] arr) {
        int ans = Integer.MIN_VALUE;
        int n = arr.length;

        for (int i = 0; i < n; i++) {
            for (int j = i + 1; j < n; j++) {
                if (i == 0 && j == 1)
                    ans = arr[i] * arr[j];
                else if (arr[i] * arr[j] > ans)
                    ans = arr[i] * arr[j];
            }
        }
        return ans;
    }

    // Function that implements the Optimal Solution for the given problem
    static int maxProductOptimal(int[] arr) {
        // Sort the given array
        Arrays.sort(arr);

        // Find the maximum product
        int ans = arr[arr.length - 1] * arr[arr.length - 2];

        // Return the product
        return ans;
    }

    // Function that performs Stress Testing
    static void stressTest() {
        Random rand = new Random();

        while (true) {
            // Generate values for n from 2 to 10
            int n = rand.nextInt(9) + 2;
            int[] arr = new int[n];

            // Iterate over array arr
            for (int i = 0; i < n; i++) {
                // Subtracting 5 will generate negative integers
                arr[i] = rand.nextInt(11) - 5;
            }

            // Solution using Naive Approach
            int ansBrute = maxProductNaive(arr);

            // Solution using Optimal Approach
            int ansOptimal = maxProductOptimal(arr);

            // If ans is correct
            if (ansBrute == ansOptimal) {
                System.out.println("Correct");
            } else {
                System.out.println("Wrong Answer");

                System.out.println(n);

                // Print the array arr
                for (int i = 0; i < n; i++)
                    System.out.print(arr[i] + " ");

                System.out.println("\nCorrect Output: " + ansBrute);
                System.out.println("Wrong Output: " + ansOptimal);
                break;
            }
        }
    }

    // Driver Code
    public static void main(String[] args) {
        // Function Call for Stress Testing
        stressTest();
    }
}

Python3

import random

# Function that implements the Naive Solution for the given problem
def max_product_naive(arr):
    ans = float('-inf')
    n = len(arr)

    for i in range(n):
        for j in range(i + 1, n):
            if i == 0 and j == 1:
                ans = arr[i] * arr[j]
            elif arr[i] * arr[j] > ans:
                ans = arr[i] * arr[j]

    return ans

# Function that implements the Optimal Solution for the given problem
def max_product_optimal(arr):
    # Sort the given array
    arr.sort()

    # Find the maximum product
    ans = arr[-1] * arr[-2]

    # Return the product
    return ans

# Function that performs the Stress Testing
def test():
    # Seeding random number generator to get unique values
    random.seed()

    while True:
        # Generate values for n from 2 to 10
        n = random.randint(2, 10)
        arr = [random.randint(-5, 4) for _ in range(n)]

        # Solution using Naive Approach
        ans_brute = max_product_naive(arr)

        # Solution using Optimal Approach
        ans_optimal = max_product_optimal(arr)

        # If ans is correct
        if ans_brute == ans_optimal:
            print("Correct")
        else:
            print("Wrong Answer")

            print(f"n: {n}")

            # Print the array arr
            for i in arr:
              print(i, end=" ")
            print()

            print(f"Correct Output: {ans_brute}")
            print(f"Wrong Output: {ans_optimal}")

            break

# Driver Code
if __name__ == "__main__":
    # Function Call for Stress Testing
    test()

C#

using System;

class StressTesting
{
    // Function that implements the Naive Solution for the given problem
    static int maxProductNaive(int[] arr)
    {
        int ans = int.MinValue;
        int n = arr.Length;

        for (int i = 0; i < n; i++)
        {
            for (int j = i + 1; j < n; j++)
            {
                if (i == 0 && j == 1)
                    ans = arr[i] * arr[j];
                else if (arr[i] * arr[j] > ans)
                    ans = arr[i] * arr[j];
            }
        }
        return ans;
    }

    // Function that implements the Optimal Solution for the given problem
    static int maxProductOptimal(int[] arr)
    {
        // Sort the given array
        Array.Sort(arr);

        // Find the maximum product
        int ans = arr[arr.Length - 1] * arr[arr.Length - 2];

        // Return the product
        return ans;
    }

    // Function that performs Stress Testing
    static void stressTest()
    {
        Random rand = new Random();

        while (true)
        {
            // Generate values for n from 2 to 10
            int n = rand.Next(9) + 2;
            int[] arr = new int[n];

            // Iterate over array arr
            for (int i = 0; i < n; i++)
            {
                // Subtracting 5 will generate negative integers
                arr[i] = rand.Next(11) - 5;
            }

            // Solution using Naive Approach
            int ansBrute = maxProductNaive(arr);

            // Solution using Optimal Approach
            int ansOptimal = maxProductOptimal(arr);

            // If ans is correct
            if (ansBrute == ansOptimal)
            {
                Console.WriteLine("Correct");
            }
            else
            {
                Console.WriteLine("Wrong Answer");

                Console.WriteLine(n);

                // Print the array arr
                for (int i = 0; i < n; i++)
                    Console.Write(arr[i] + " ");

                Console.WriteLine("\nCorrect Output: " + ansBrute);
                Console.WriteLine("Wrong Output: " + ansOptimal);
                break;
            }
        }
    }

    // Driver Code
    public static void Main(string[] args)
    {
        // Function Call for Stress Testing
        stressTest();
    }
}

Javascript

// Function that implements the Naive Solution for the given problem
function maxProductNaive(a) {
    let ans;
    for (let i = 0; i < a.length; i++) {
        for (let j = i + 1; j < a.length; j++) {
            if (i === 0 && j === 1)
                ans = a[i] * a[j];
            else if (a[i] * a[j] > ans)
                ans = a[i] * a[j];
        }
    }
    return ans;
}

// Function that implements the Optimal Solution for the given problem
function maxProductOptimal(a) {
    // Sort the given array
    a.sort((x, y) => y - x);

    // Find the maximum product
    const ans = a[0] * a[1];

    // Return the product
    return ans;
}

// Function that performs the Stress Testing
function test() {
    while (true) {
        // Generate values for n from 2 to 10
        const n = Math.floor(Math.random() * 9) + 2;
        const a = [];

        // Iterate over array a[]
        for (let i = 0; i < n; i++) {
            // Subtracting -5 will generate negative integers
            a[i] = Math.floor(Math.random() * 10) - 5;
        }

        // Solution using Naive Approach
        const ansBrute = maxProductNaive(a);

        // Solution using Optimal Approach
        const ansOptimal = maxProductOptimal(a);

        // If ans is correct
        if (ansBrute === ansOptimal)
            console.log("Correct");
        else {
            console.log("Wrong Answer");

            console.log(n);

            // Print the array a[]
            console.log(a.join(" "));

            console.log("Correct Output: " + ansBrute);
            console.log("Wrong Output: " + ansOptimal);
            break;
        }
    }
}

// Driver Code
test();

Output

Wrong Answer
4
-4 -3 -3 1 
Correct Output: 12
Wrong Output: -3

Explanation:
The Optimal Solution works fine for positive integers but fails on negative integers like shown in the output of the code. Through stress test it was figured out the problem with the code. To generate and test our code for larger input, generate larger integers for n and a[I].

Summary: Stress testing will surely help in debugging the code but first should try to think of test case where the code may not work because its less complex to check for some simple test cases and if that doesn’t help then proceed with the Stress Testing.