Test Oracles

Test Oracle is a mechanism, different from the program itself, that can be used to test the accuracy of a program’s output for test cases. Conceptually, we can consider testing a process in which test cases are given for testing and the program under test. The output of the two then compares to determine whether the program behaves correctly for test cases. This is shown in figure. 

Testing oracles are required for testing. Ideally, we want an automated oracle, which always gives the correct answer. However, often oracles are human beings, who mostly calculate by hand what the output of the program should be. As it is often very difficult to determine whether the behavior corresponds to the expected behavior, our “human deities” may make mistakes. Consequently, when there is a discrepancy, between the program and the result, we must verify the result produced by the oracle before declaring that there is a defect in the result. 

The human oracles typically use the program’s specifications to decide what the correct behavior of the program should be. To help oracle determine the correct behavior, it is important that the behavior of the system or component is explicitly specified and the specification itself be error-free. In other words actually specify the true and correct behavior. 

There are some systems where oracles are automatically generated from the specifications of programs or modules. With such oracles, we are assured that the output of the oracle conforms to the specifications. However, even this approach does not solve all our problems, as there is a possibility of errors in specifications. As a result, a divine generated from the specifications will correct the result if the specifications are correct, and this specification will not be reliable in case of errors. In addition, systems that generate oracles from specifications require formal specifications, which are often not generated during design.