Software Testing – Test Harness

A test harness is also called an automated test framework. In software testing, a test harness is a group of software and test data designed to test a program element by operating it under different situations and supervising its practices and results. A test harness is a software designed to ease integration testing. Here test stubs are common units of the software under progress and are restored by operating units as the software is developed. It is called top-down integration testing. Here, we will discuss the following topics:

1. What is Test Harness?
2. Features of Test Harness.
3. Why Use Test Harness?
4. How is the Test Harness Done?
5. Test Harness vs Test Framework.
6. Settings Where Test Harness is Utilized.
7. Test Harness Tools.
8. Benefits of Test Harness.
9. Limitations of Test Harness.

What is Test Harness?

The test harness is a collection of stubs, drivers, and other supporting tools that are required to automate test execution. Test harnesses are exterior to the software being under test and replicate resources or performance which are not available in a test environment. Suppose, when trying to design software that needs to combine with the software on a mainframe computer, but if no mainframe is there during the development phase, then a test harness should be created to use as a replacement.

• A test harness has two important parts, a test execution engine, and a test script repository.
• The test execution engine is the software that is used to perform the test and the test script repository is the location where test scripts and test cases are stored.
• It contains all information that is needed to compile and run a test like test cases, target deployment port, etc.
• Test harnesses are used in two main areas, automation testing, and integration testing.
• One of the benefits of test harnesses includes the automation of the testing process, support of debugging modes, etc. 

Features of Test Harness

• Support test automation: Test harnesses support the automation of tests. They can call functions with donated limits and compare the output to the estimated result. 
• The test harness is a holder to the developed code: It can be tested by an automation framework. It should permit particular tests to work, adapt a run-time condition, and provide a capacity to scan output.
• Test harness may be a portion of deliverable software: It is distinguished from the application source code and may be replicated on various projects. 
• Test harness replicates software operation: It will not have awareness of test suites, test cases, or test reports. Those things are given by a testing framework and corresponding automated testing tools. The test harness task is to arrange the right test matches. 
• Integrate test harness for composite frameworks: The test harness will normally be particular to a development environment like Java. But, integration test harnesses have been developed for use in more composite frameworks.

Why Use Test Harness?

• Automate testing process: Test Harness helps automate the testing procedures and thus increases the productivity of the system through automation.
• Execute test suites: Test cases and test suites execution.
• Print test results: The test harness is useful in generating test reports.
• Helps to measure code coverage: It helps developers to measure the cove coverage at a code level.
• Helps to simulate complex conditions: It helps to simulate and handle complex conditions that testers find difficult to handle.
• Support debugging: Test harness helps to support debugging activities.
• Analyze test results: A test harness helps to analyze test results.
• Enhance software quality: It helps to enhance the quality of the software components and applications.
• Increased productivity: Test harness helps to increase productivity through automation.

How is the Test Harness Done?

Test Harness is a procedure to do all the testing works, like test execution through test libraries and generating results. Below are steps that outline the procedure for the test harness:

1. Read test cases: Developers and testers need to create certain test scripts to manage specific test scenarios and test data. It starts with the preparation of data, a procedure to collect essential data.
2. Invokes applications: After the execution of the test harness, testers need to make sure that they can execute all tests and generate results through a single command.
3. Collects actual results: Results are collected from the execution.
4. Compare expected results: Expected results are compared with the actual test results to observe the discrepancies and report them.
5. Writes test reports: Test reports are created and submitted for further analyses.

 Test Harness vs Test Framework

Parameters	Test Harness	Test Framework
Definition	The test harness is a collection of stubs, drivers, and other supporting tools that are required to automate test execution.	A test framework is a set of procedures, processes, and environments through which the tester can design and implement automated tests.
Composition	It consists of information such as test cases, and the target deployment port that is required to compile and execute tests.	It consists of information such as test library, automation testing, testing tools, testing platform, etc.
Record and playback feature	The tester cannot use the record and playback feature.	The tester can manually record and playback scripts.
Examples	Automation testing and Integration testing.	Data-driven testing, Code-driven testing, Module driven testing, Behavior-driven testing, Keyword driven testing, Hybrid testing, and Modularity driven testing.
Example tools	JUnit, NUnit, etc.	Cucumber, Robot Framework, etc.

Settings Where Test Harness is Utilized

There are two settings where the test harness is utilized:

1. Automation Testing: In the world of automation testing, the Test Harness is defined as a framework and software system that contains the test scripts. It also includes the essential parameters required to run those test scripts. After obtaining the test results, testers differentiate them from other data and supervise the results.
2. Integration Testing: Integration Testing is defined as a process in which independent units and components are connected together and are tested to check whether they are behaving as expected or not. The purpose of integration testing is to identify defects when modules are integrated and communicate with each other. Integration testing can be done only when the software is unit tested and the modules are 100% ready for testing. But, this does not happen often because some modules meant for integration testing may not be ready at that time. In such cases, you can use test stubs and test drivers.

Test Harness Tools

Test Harness tools fall under automation software and test management software. Implementing a test harness with one tool is not possible. It can be done through a combination of tools like QTP, HP ALM, JUnit, and NUnit.

1. JUnit: JUnit is a unit testing framework. It is for the Java programming language. It is an open-source testing framework. It is essential in test-driven development. It is from the family of unit testing frameworks which is collectively known as xUnit that originated with SUnit.

Features of Junit:

• It is an open-source network.
• It provides Annotations, Assertions, and Test runners.
• It improves the quality of code.
• It supports automated test running.
• The results are easily analyzed.      

2. NUnit: It is an open-source unit testing framework. It is for the .NET Framework and Mono. Its purpose is the same as JUnit in the Java world. NUnit is one of several programs in the xUnit family.

Features of NUnit:

• Here, tests can be run from a console runner, in Visual Studio by a Test Adapter, or third-party runners.
• Parallel running of tests is also possible.
• .It provides support for data-driven tests.
• It supports multiple platforms like .NET Core, Xamarin Mobile, Compact Framework, and Silverlight.
• All test cases can be added to one or more categories, to permit careful operation.

Benefits of Test Harness

• Increases productivity: As the testing procedure is automated, productivity is increased.
• Automate testing process: A test harness helps to automate the testing process to the maximum extent.
• Support code debugging: Test harnesses are composed of drivers and stubs and thus are capable of support code debugging.
• Helps in early bug detection: Test harness helps developers in the early identification of the defect in the code.
• Improves software quality: It helps to enhance the overall quality of the software application.
• Helps to measure code coverage: It helps developers to measure the code coverage at the elementary level during application development.
• Cover complex test scenarios: Test harness helps to cover complex test scenarios which are difficult to handle using traditional testing approaches.
• Increases confidence in the robustness of the application: As the application is well-tested at the unit level so it helps to increase the confidence in the robustness of the application.

Limitations of Test Harness

• No support for record and play: Test harness tools provide no support for record and play features.
• Programming knowledge required: Testers using test harness tools are expected to have programming language knowledge such as Java, .NET, etc.
• Incurs cost: Test harness incurs the cost for writing automated test suites through skilled developers.