If you want to make the introduction or expansion of test automation to your development and testing process a success, make sure you always start with the ‘why?’.
Since software development and delivery cycles are getting ever shorter in order to be able to answer to markets demanding high speed of delivery and continuous innovation, most organizations that produce software are relying on automated checks to perform a substantial part (or even all) of the testing process. Test automation has become the default answer to the question on how to perform tests or checks, really for most people involved in software testing.
In this article I want to explain why I think that blindly pulling the test automation card may not always result in better testing and therefore in better software.
One of the most important mistakes resulting in incorrect implementation of test automation is that the right question is not being asked first when deciding on a testing approach. All too often, the first question that comes to mind when new software is being developed is ‘how can we automate the tests for this piece of software?’. Instead, what in my opinion should have been the first question is ‘why should we automate the tests for this piece of software in the first place?’.
Don’t get me wrong, I am very much a fan of the smart application of test tools to improve and speed up the testing process, but simply diving in head first and automating everything in sight isn’t likely to yield long term success. Instead, I would like to suggest a simple three-question approach to the introduction and application of test automation as part of your testing activities.
Start with the ‘why?’
The first question that everybody thinking about introducing or expanding test automation activities with regards to their development process should ask themselves is: why do we want test automation in the first place? What is the value we want to add to our development process, to the products or services we offer and ultimately to our customers? The answer to the ‘why’ question can be different depending on the organization and the situation (tools are required to perform the required checks, manual testing resources are constrained, …).
However, here are two answers to this question that are definitely NOT correct:
- ‘We want 100% test automation coverage’. Besides the fact that there is no such thing as 100% test automation (only the checking part can be automated, the testing part is an exploratory activity that can only be performed by a human being), 100% automation is a useless metric in itself. Generally, there exists a sweet spot beyond which the effort required to create and maintain additional automated tests exceeds the gains achieved by introducing them. In most situations, this sweet spot will be somewhere (often way) south of 100%.
- ‘We are going to do more test automation because our management says so’. Unless said management can motivate their desire to increase test automation efforts by explaining what the added value will be, it might be better to default to ‘no, we’re not going to do that’. Remember that you’re the professional, so you might as well start behaving like one.
In general, answering the ‘why’ incorrectly or failing to answer it altogether will likely lead to automation that does not add the desired value to your development process, to your product or service and therefore to your customers.
Then define the ‘what?’
Only after we have determined the ‘why’ are we in the proper position to answer the next question: ‘exactly what (kind of) tests / checks are we going to automate?’. As everybody with even a passing interest in the field of test automation knows, some tests are better suited for automation than others. Repeatable checks that can be specified using some sort of algorithm or decision table can usually be performed by a computer (after a human person has instructed the computer how to do so, of course).
The stock example is the regression test, performed to see whether a change in functionality did not bring forth any unwanted side effects. Since these tests often cover basic use cases and scenarios, they tend to be pretty stable and therefore particularly suitable for automation. As another example, checks that do not involve a user interface (such as checks on the responses of a web service and checks on values in a database) are typically performed with the support of some sort of tool.
A different type of test that is particularly suitable for automation is the performance test. While you can theoretically invite a large number of real people to use your application in parallel and see how it behaves, a performance test is almost always performed with the use of a tool that spawns so-called virtual users to generate a certain amount of load on the system. Most performance tools are also able to measure a lot of different performance metrics and create elaborate reporting on the results.
On the other side of the spectrum, there are actual tests (as opposed to checks). Testing is an activity performed by human beings and involves evaluating the application under test by learning through exploration and experimentation. This is an activity that cannot be automated. Trying to do so will inherently fail, at least with the current set of tools available on the market.
So, to answer the question of ‘what are we going to automate?’, you need to clearly define what parts of the testing process consists of checks (that can be automated) and what part of those checks are worth automating. Note that this probably will not be completed overnight, but rather will be an ongoing learning and evaluation cycle.
Finally, answer the ‘how?’
Now that we have successfully answered the ‘why?’ and the ‘what?’ questions, it is time to take a look at the ‘how?’ of test automation. Some of the questions that should be asked in this stage of the test automation implementation process are:
- How do the automated checks fit into our development process? For example, are the checks to be made part of the continuous integration / continuous delivery pipeline? And if that is not the case, what is the trigger for running the automated checks?
- What is the scope of the checks we want to automate? Checks performed on the user interface level generally have a broader scope, but they also tend to require a longer time to develop as well as to execute. Also, they tend to require more maintenance as user interfaces are among the most dynamic and most often updated parts of an application. On the other hand, tests on the unit, API or database level run faster and are often more stable, but have a more limited scope.
- How are we going to implement our tests? Only after all of the previous questions have been answered is it time to find the tool that best suits your requirements. Selecting a tool first and then trying to fit it into an existing software development and testing process is a bit like buying a horse and then realizing you live in an apartment on the twentieth story of a Manhattan skyscraper: probably not such a good idea.
So, in conclusion: if you want to make the introduction or expansion of test automation to your development and testing process a success, make sure you always start with the ‘why?’.