What to do if the code doesn’t work?

This article will discuss the different routes one could take if their code fails to work in various scenarios.

If one has, seemingly at least, coded everything perfectly, tested solutions in the test cases included in the statements, but after the code’s submission into the system it fails in the test and there are no explicit error codes to help identify and solve the problem. This problem could be caused by a vast number of reasons but it is usually one of the following:

• Wrong Answer
• Time/Memory Limit Exceeded
• Failed (Runtime Error)

Although the approach may differ based on the reason, generally one can follow the steps below:

• Check for the Edge Cases: For example, if N = 1 or n is the maximum possible number satisfying the constraints. Check if the program behaves correctly given the input that hits constraints in all possible senses.
• Generate some general test cases: that you know the correct answer for. Don’t look at the code’s answer for these tests before figuring out what the answer should be with pen and paper. Otherwise, it is easy to convince yourself that it is correct and fail to see some silly error.
• The time limit exceeded the error, measuring how long the program works for the larger inputs. The following functions help measure the CPU time since the start of the program:
  • C++ (double)clock() / CLOCKS PER SEC with time included.
  • python time.clock() returns a floating-point value in seconds.
  • Java System.nanoTime() returns long values in nanoseconds.

Measure time for small tests, medium tests, and large tests. One of the following possible outcomes can be encountered:

• The program works for small and medium tests in time, but it is more than 10 times slower than needed (or hangs for the large tests). In that case, the program probably has complexity issues and we could try to figure out the problem using the following options:
  • Measure the time parts of the program take separately. For example, how much time reading the input/printing the output takes).
  • Compute the actual number of operations your algorithm and its parts do and see if it is expected.
  • Check if references have been passed to functions that only apply to C++, in Java, and in Python. It is always referenced).
• The program hangs on a small or a medium test. Check for an infinite loop/recursion. Add assertions (assert in c++, python, and java) on preconditions and postconditions of loops and functions and see if they fail. Use the debug output/debugger to see what code path leads to the hang.
• If it is a runtime error, the message could be an unknown signal, then it could be a good sign. It is one of the most informative reasons that means the program could crash due to one of the following factors:
  • A location is accessed in memory that doesn’t belong to the program. In C++, it can take two forms: trying to access a non-existing element of an array, trying to evaluate a null-pointer or a pointer that points to a location that doesn’t belong to the program.
  • Make an arithmetic error: division by zero, overflow of a floating-point number, etc.
  • (C++ specific) Stack size exceeded, which may be caused by infinite recursion or by creating a large object (such as an array) inside a function.

Generate different tests and run your program against them until it crashes:

• The ‘wrong answer‘ reason is probably the most challenging; many things can lead to it. To find a failing test, one of the following things can be done:
  • Find an alternative solution that may not be correct in terms of efficiency (and some cases don’t even work for some types of tests) but could be used to check if the main solution’s answer is correct.
  • Make the program crash if something is inconsistent. That means adding assertions for postconditions and preconditions of your functions and loops.

How to generate tests: The simplest way to generate a test is to write a program that prints a test to a text file. Below is an example to illustrate the same:

Program 1: Generating a test for the maximum pairwise distinct:

Python3

import sys
n = int(sys.argv[1])
print(n)
print(' '.join([str(i∗2)for i in range(n)])

The most cryptic thing here is probably sys.argv[1]. This is a getter for the first command-line argument. Now, how to use this to run the program? Copy the executable file or a python script to the same directory as the generating script and run the following commands:

python script.py 17 > input.txt
./your_program_name < input.txt

The use of Python3 is recommended over Python. It could help solve the issues in a few cases. Therefore, some tests can be automatically generated and the program can be run against them, but the following questions remain unanswered:

• How to randomize the tests?
• How to generate a lot of them?
• How to check the answers?

Generating random tests and running your program on them: The following technique consisting of 3 parts can be used:

• Test generator that accepts a seed as a command-line parameter as discussed in Program 2.
• An alternative solution.
• A script that repeatedly generates a test with the generator from (1) runs both the main solution and the model solution on the generated test and checks if the answers coincide in the program 2 or not.

Program 2: Generator that accepts a seed from the command line:

Python3

import random
import sys

# Input the number N
n = int(sys.argv[1])
myseed = int(sys.argv[2])
random.seed(myseed)

print(n)

# 1000 could also be moved to
# parameters instead of making it
# a hard constant in the code
print(' '.join([str(random.randint(1, 1000)) for i in range(n)])

Program 3: The actual script:

Python3

import random
import sys
import os

# Accept the number of tests as a
# command line parameter

tests = int(sys.argv[1])

# Accept the parameter for the
# tests as a command line parameter

n = int(sys.argv[2])
for i in range(tests):
    print("Test #" + str(i))

    # Run the generator gen.py with
    # parameter n and the seed i
os.system("python3 gen.py " + str(n) + " " + str(i) + " > input.txt")

# Run the model solution model.py
# Notice that it is not necessary
# that solution is implemented in
# python, you can as well run
# ./model < input.txt > model.txt
# for a C++ solution.

os.system("python3 model.py < input.txt > model.txt")

# Run the main solution

os.system("python3 main.py < input.txt > main.txt")

# Read the output of the
# model solution

with open('model.txt') as f:
    model = f.read()
print("Model: ", model)

# Read the output of the
# main solution :

with open('main.txt') as f:
    main = f.read()
print("Main: ", main)
if model != main:
    break

How to add assertions: In Java, Python and C++ assert expression, assert expression, and assert(expression); respectively, will produce a runtime error if the Boolean expression is false. One possible usage of assertions is verifying that the answer to the program is consistent. Another common use case is to verify that the intermediate steps of the program are consistent.