Test Tools and Automation- Strategies , Costs, Risks and Benefits
Test tools can be very useful. In fact, some tools are essential. For example- An incident tracking system. Generally, tools are meant to improve the efficiency and accuracy of testing. Hence, it is very much important that we carefully select tools and properly implement them as well. Many times, it has been observed that a good tool also does not get properly implemented and resulting in inefficiencies. Also, Ongoing maintenance of scripts and data is also very important to reap the benefits of test tools.
Most of the time, automation is equated with test execution automation. While it is true for the most part, but the other aspects of testing are also automated.
Test Automation strategies :
As a starting point, it should be clearly understood that a test tool is NOT equivalent to having an automation strategy. An expectation that getting a tool will solve the testing problems is not realistic. So, here are some of the important test automation strategies:
- First and Foremost –
Automate for the long term- It is not a good idea to automate a set of test cases that are going to be run only a couple of times. This will not give you a return on investment (ROI) with respect to the cost of the tool (if it is licensed) and effort for creating the scripts.
- Automate only those tests which are automatable i.e tests that can run unattended and human judgement is NOT required to interpret test results.
- Automate the tests which are error-prone if done manually- for Example, data load, regression tests.
- Only automate tests that have a valid business case i.e you should have an idea that how many times a test will be required to run between automation and till the application under test retires. If the test case is a one-off scenario that might be run rarely and is an edge scenario, there is no point to automate.
- Automate tests that require testing over multiple browsers/devices/OS/environment/hardware/configurations. It will give a good ROI.
- Automation test strategies should be tailored to the project to give the best outcomes. We can have a generic strategy, but it might not give optimal or best results. We should always tailor the strategy for automation.
- Automate tests where manual testing is impossible. Example — Performance tests.
- Automate tests which are maintainable over time: Many times, test automation is created and is successful for few runs only and over the time as the UI layout changes, test need change. However, due to various reasons, tests are not updated or the new team might not understand where to make the change. Hence, the script should be modular and easily understandable to make it maintainable.
Test automation Costs :
In any task, we need to identify costs, risks, and benefits. When it comes to costs- We can divide them in terms of initial costs and recurring costs. Initial costs for automation include-
- Evaluating and selecting the right tool. DO NOT select the tool and practice later.
- Purchasing the tool license or adapting to the open-source tool or developing your own custom tool.
- Learning or training to use the tool
- Integrating the tool with the overall test process and other tools in the process.
Recurring costs :
- Maintenance cost for scripts and tool
- Ongoing license fees, in case of licensed tools.
- Tool Support fees, if any.
- Ongoing training costs as teams are dynamic and team members keep changing.
- Continuous improvement in scripts based on technology upgrades.
It is always a good idea to do a POC before finalizing a tool. However, Please note that recurring costs are generally missed while doing POCs. Hence, be cautious if the cost is a driving factor in deciding a tool for your project/organization.
Test Automation Risks :
- Unrealistic expectation –
The first and foremost risk is Unrealistic expectations from the tool. This is fairly common especially in projects and organizations where there is limited prior automation experience. Test tools do not make a difficult test problem into an easy one.
- Underestimating time, cost, and effort required to introduce the tool –
There is always a sizable effort required to build test frameworks, integrate with other tools, plan and train people who are going to use the tool.
- Underestimating time and effort needed to achieve a positive return on automation –
A positive return on investment of automation means reduced overall test effort, reduced test execution period, increased coverage. it may take years for an organization to have positive returns on automation.
- Vendor issues –
Like the inability to provide technical support, inability to update the automation tools with changes in the software testing platform, a free tool is made licensed over the course of time.
- Underestimating the need for manual testing-
As automation experts, we should understand that not all tests can be automated. Hence, there is going to be an ongoing manual effort in general. Also, with the repetitive automated tests, we might end up having the issue called “Pesticide Paradox” where we do not find any new issues. Hence, we should have a careful mix of manual testers and automation.
- Maintainability of test scripts and data –
You could produce brittle, hard-to-maintain scripts and frameworks. This is a major blunder that organizations and projects do and later suffer as they are not able to get the return on automation investment. Hence, we should do a careful design of maintainable, robust, and modular frameworks such that test scripts and data could be reused and reduce the risk of lack of maintainability of scripts.
Test Automation Benefits :
- Saved time, Reduced effort, better coverage, better predictability of execution time are some obvious benefits of automation.
- Reliability, the automation testing tools are more reliable and consistent than human software testers because once robust scripts are designed, tests are not error-prone as such.
- More suitable for repetitive tests that are suitable for testing repetitive steps with different data sets, cross-browser tests, regression tests.
- Tests which cannot be done manually are possible with tools like performance tests. This helps in reducing risks.