Testing in Python using Doctest module

Docstrings in Python are used not only for the description of a class or a function to provide a better understanding of the code and use but, also used for Testing purposes. Testing is a critical aspect of software development that ensures code functions as expected and guards against bugs. In Python, the Doctest module provides a simple yet powerful way to test code by embedding test cases within docstrings.

What is Doctest Module in Python

The Doctest Module finds patterns in the docstring that look like interactive shell commands. The input and expected output are included in the docstring, then the doctest module uses this docstring for testing the processed output. After parsing through the docstring, the parsed text is executed as Python shell commands and the result is compared with the expected outcome fetched from the docstring.

Why Choose Doctest in Python?

The reason for choosing Doctest in Python is that Doctest in Python is favored for its simplicity, seamlessly integrating documentation with testing. By embedding test cases within docstrings, promotes clear and concise code examples. It’s a choice unburdened by AI, relying solely on human-authored tests for verification, ensuring transparency and control over the testing process.

How to Use Doctest in Python ?

Here’s we will see how to use doctest in Python, here is a simple example of how to use doctest in Python.

1. import testmod from doctest to test the function.
2. Define our test function.
3. Provide a suitable docstring containing desired output on certain inputs.
4. Define the logic.
5. Call the testmod function with the name of the function to test and set verbose True as arguments.

Note: All the arguments are optional. The name of the function is explicitly passed to the testmod. It’s useful if there are multiple functions.

Implementation

In this example, below Python code tests the factorial function using doctests. The function calculates the factorial of a positive number recursively. Docstrings define input and expected output. Running testmod checks if the function behaves as expected for the provided test cases

Python3

# import testmod for testing our function 
from doctest import testmod 
  
# define a function to test 
def factorial(n): 
    ''' 
    This function calculates recursively and 
    returns the factorial of a positive number. 
    Define input and expected output: 
    >>> factorial(3) 
    6 
    >>> factorial(5) 
    120 
    '''
    if n <= 1: 
        return 1
    return n * factorial(n - 1) 
  
# call the testmod function 
if __name__ == '__main__': 
    testmod(name ='factorial', verbose = True) 

Output:

Trying:
    factorial(3)
Expecting:
    6
ok
Trying:
    factorial(5)
Expecting:
    120
ok
1 items had no tests:
    factorial
1 items passed all tests:
   2 tests in factorial.factorial
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Now, test for failure.

What if our Logic goes Wrong?

Below code, uses doctests to validate the factorial function. The function is designed to recursively compute the factorial of a positive integer, as documented in the docstring. However, there’s an issue with the recursive logic. The testmod function is called to verify the function’s correctness against the provided test cases.

Python3

# import testmod for testing our function 
from doctest import testmod 
  
# define a function to test 
def factorial(n): 
    ''' 
    This function calculates recursively and 
    returns the factorial of a positive number. 
    Define input and expected output: 
    >>> factorial(3) 
    6 
    >>> factorial(5) 
    120 
    '''
    if n <= 1: 
        return 1
    # wrong logic for factorial 
    return factorial(n - 1) 
  
# call the testmod function 
if __name__ == '__main__': 
    testmod(name ='factorial', verbose = True) 

Output:

Trying:
    factorial(3)
Expecting:
    6
**********************************************************************
File "woking_with_csv.py", line 33, in factorial.factorial
Failed example:
    factorial(3)
Expected:
    6
Got:
    1
Trying:
    factorial(5)
Expecting:
    120
**********************************************************************
File "woking_with_csv.py", line 35, in factorial.factorial
Failed example:
    factorial(5)
Expected:
    120
Got:
    1
1 items had no tests:
    factorial
**********************************************************************
1 items had failures:
   2 of   2 in factorial.factorial
2 tests in 2 items.
0 passed and 2 failed.
***Test Failed*** 2 failures.

Note: If ‘verbose’ is set to False(default), output will be shown in case of failure only, not in the case of success.

Examples to Understand Doctest Module

Below, are the example of testing in Python using Doctest Module in Python for better understanding.

Testing a Function with Outputs

When testing a function with complex outputs in Python using the Doctest module, it’s crucial to provide clear and detailed docstring examples that cover various scenarios. Ensure that the expected outputs are accurately documented for each input.

Python3

def complex_function(input_list):
    '''
    This function performs complex operations on the input list.
    Define input and expected output:
    >>> complex_function([1, 2, 3])
    [3, 4, 5]
    >>> complex_function([-1, 0, 1])
    [0, 1, 2]
    '''
    return [x + 2 for x in input_list]

if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

Output

Trying:
    complex_function([1, 2, 3])
Expecting:
    [3, 4, 5]
ok
Trying:
    complex_function([-1, 0, 1])
Expecting:
    [0, 1, 2]
**********************************************************************
File "__main__", line 7, in __main__.complex_function
Failed example:
    complex_function([-1, 0, 1])
Expected:
    [0, 1, 2]
Got:
    [1, 2, 3]
1 items had no tests:
    __main__
**********************************************************************
1 items had failures:
   1 of   2 in __main__.complex_function
2 tests in 2 items.
1 passed and 1 failed.
***Test Failed*** 1 failures.

Testing a Function with Multiple Test Cases

When testing a function with multiple test cases in Python using the Doctest module, ensure to cover various input scenarios and edge cases within the docstring examples. Each test case should be well-documented with its expected output. Here’s a code example illustrating testing a function with multiple test cases:

Python3

def multiple_test_cases(number):
    '''
    This function checks multiple test cases for a given number.
    Define input and expected output:
    >>> multiple_test_cases(0)
    'Zero'
    >>> multiple_test_cases(5)
    'Positive'
    >>> multiple_test_cases(-5)
    'Negative'
    >>> multiple_test_cases(100)
    'Large positive'
    >>> multiple_test_cases(-100)
    'Large negative'
    '''
    if number == 0:
        return 'Zero'
    elif number > 0:
        if number > 10:
            return 'Large positive'
        else:
            return 'Positive'
    else:
        if number < -10:
            return 'Large negative'
        else:
            return 'Negative'

if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

Output

Trying:
    complex_function([1, 2, 3])
Expecting:
    [3, 4, 5]
ok
Trying:
    complex_function([-1, 0, 1])
Expecting:
    [0, 1, 2]
**********************************************************************
File "__main__", line 7, in __main__.complex_function
Failed example:
    complex_function([-1, 0, 1])
Expected:
    [0, 1, 2]
Got:
    [1, 2, 3]
Trying:
    multiple_test_cases(0)
Expecting:
    'Zero'
ok
Trying:
    multiple_test_cases(5)
Expecting:
    'Positive'
ok
Trying:
    multiple_test_cases(-5)
Expecting:
    'Negative'
ok
Trying:
    multiple_test_cases(100)
Expecting:
    'Large positive'
ok
Trying:
    multiple_test_cases(-100)
Expecting:
    'Large negative'
ok
1 items had no tests:
    __main__
1 items passed all tests:
   5 tests in __main__.multiple_test_cases
**********************************************************************
1 items had failures:
   1 of   2 in __main__.complex_function
7 tests in 3 items.
6 passed and 1 failed.
***Test Failed*** 1 failures.

Limitations of Doctest Module

The Doctest module in Python has some limitations:

• Limited Test Coverage: Doctest relies heavily on docstrings, making it suitable mainly for testing small code snippets or simple functions. It may not cover all aspects of complex functions or classes.
• Readability Issues: Test cases embedded within docstrings can sometimes clutter the documentation, leading to reduced readability, especially for larger projects with numerous tests.
• Brittleness: Doctest tests are sensitive to changes in output formatting or minor code modifications, which can cause tests to fail unnecessarily. This brittleness makes maintenance challenging, especially in evolving codebases.

Here’s a code example illustrating these limitations:

In this example, below code showcases how Doctest tests might fail to cover all scenarios, clutter the docstring, and become brittle, especially when dealing with edge cases like division by zero.

Python3

def divide(a, b):
    """
    This function divides two numbers.
    Define input and expected output:
    >>> divide(4, 2)
    2
    >>> divide(10, 0)
    Traceback (most recent call last):
        ...
    ZeroDivisionError: division by zero
    """
    return a / b

if __name__ == "__main__":
    import doctest
    doctest.testmod(verbose=True)

Output

Trying:
    complex_function([1, 2, 3])
Expecting:
    [3, 4, 5]
ok
Trying:
    complex_function([-1, 0, 1])
Expecting:
    [0, 1, 2]
**********************************************************************
File "__main__", line 7, in __main__.complex_function
Failed example:
    complex_function([-1, 0, 1])
Expected:
    [0, 1, 2]
Got:
    [1, 2, 3]
Trying:
    divide(4, 2)
Expecting:
    2
**********************************************************************
File "__main__", line 5, in __main__.divide
Failed example:
    divide(4, 2)
Expected:
    2
Got:
    2.0
Trying:
    divide(10, 0)
Expecting:
    Traceback (most recent call last):
        ...
    ZeroDivisionError: division by zero
ok
Trying:
    multiple_test_cases(0)
Expecting:
    'Zero'
ok
Trying:
    multiple_test_cases(5)
Expecting:
    'Positive'
ok
Trying:
    multiple_test_cases(-5)
Expecting:
    'Negative'
ok
Trying:
    multiple_test_cases(100)
Expecting:
    'Large positive'
ok
Trying:
    multiple_test_cases(-100)
Expecting:
    'Large negative'
ok
1 items had no tests:
    __main__
1 items passed all tests:
   5 tests in __main__.multiple_test_cases
**********************************************************************
2 items had failures:
   1 of   2 in __main__.complex_function
   1 of   2 in __main__.divide
9 tests in 4 items.
7 passed and 2 failed.
***Test Failed*** 2 failures.

Frequently Asked Questions (FAQ’s)

1. How to Run Doctest in Python?

To run doctests in Python, you can include test cases in the docstrings of your functions or modules and use the doctest module. Execute python -m doctest <module_name>.py in your terminal to run the tests.

2. How Doctest Matches Expected and Actual Test Output?

Doctest matches expected and actual test output by parsing the docstrings of functions and modules for lines prefixed with >>>, which represent Python expressions. It then executes these expressions and compares their results with the expected output provided in the docstring. If there’s a match, the test passes; otherwise, it fails.