Five steps from automation to automated QA

Jun 30, 2020 / by Henri Terho

What is the future of QA and how we can optimize the QA process to make it easier for ourselves to guarantee software that behaves in ways we want. Here are the five steps from only testing the system to understanding the usage of the software from human written tests towards measuring business KPIs and using these to evaluate the quality of the system.  


Step 1 - Test automation

Step 1 is the step that is now seeing more and more adoption in software companies. Test cases are being automated using tools such as Jenkins to run tests when new software versions are committed to version control and the deployment of these features to test servers is automated.

A good example of this would be a company using git as their version control system. When a commit is made, Jenkins triggers testing, which uses robot framework to run the tests. This data is then fed to the developer to see what the status of his/her commit was.

In this phase automated integration or deployment is not yet done and the release process might still include manual testing steps to verify the actual release. In this setup, the release decision is still totally in human control and no automation of the actual release decision has been done. The data used to test is mostly inside the test cases, no other data, for example static analysis or other quality gates are yet used. The steps and their key points are also outlined in table 1.

Step 2 – Continuous integration and deployment

All the systems inside the deployment pipe are connected to each other and their behavior automated. The system can automatically deploy new version to atleast a QA environment and each feature is tested automatically before its code is integrated into the main development branch.

Here data from all parts of the software development pipe are integrated into the QA pipeline. Static analysis with tools such as sonarqube is used to verify code quality, and for example company best practices analysis in the codebase can be used to standardize deployment. This pipeline can then deploy the new features to production without the need for human intervention.

All the data about system quality is collected just in the development process, not from the production environment. Automated testing focus is mostly in the technical quality of software, not in its functional, or business value. The deployment decision is automated from humans based on technical quality.

Step 3 – Defined product quality

This level could be characterized as being highly functional implementation of most DevOps practices. Comparing level 3 to level 2, the big difference is that level 3 starts to do optimization of testing based on the actual usage of the software in production, not just using development metrics. This shifts the role of the automated testing system away just from technical code quality validation more into the realm of validating business hypotheses. Jumping to this level requires a much clearer understanding of how your business goals link to your software development goals.

At this level we can start optimizing testing of the product based on the impact that different releases have on the actual quality of the product in use. For example these can be measured in web applications using google analytics or similar tools, analyzing usage logs and even billing data. All this data should be aggregated into a metric that can be used to measure the perceived level of the release.

This data can then be used to create A/B tests which try different approaches to optimize your chosen metrics. Here the decision on which features to actually incorporate into the product is automated. A/B testing in itself does not require ML or AI. This can be achieved with automation tools such as amazon fargate or other deployment tools, which can track assigned statistics and then scale the appropriate release. Also humans can still make these decisions based on the data gathered in the A/B testing, the next step is to automate this.

Step 4 – Optimized product quality

The main differentiator of step 4 is the automatic creation of A/B tests based on code variants. The acutal product quality metrics are defined by humans, but the devops system can create new variants, which can be A/B tested with small cohorts automatically and judged against the chosen metrics.

This ability to automatically create A/B variants has to be combined with a striong level of regression testing, which can be used to keep the core functionality of the software working, while enabling other versions to be automatically varied.

Because the system can create large amounts of varianst, here the analysis part of the product quality is automated using machine learning or AI systems, which can find patterns in the data that humans cannot easily find. Also identifying failing variants cannot be based on just deterministic testing sets, analysis of for example error frequencies in production on these variants can be used to identify failing cases.

These kind of systems have been developed by some larger players, but rarely shown to the public. Facebook and Netflix have given talks on this level of automation in their deployment pipelines and this level is pretty close to the state of the art currently. Facebook deployes a large number of variants each day and automatically A/B tests these features with small cohort sets, before automatically scaling the features to production if succesfull, or scaling them back if they fail based on data from production.

The steps 4 and 5 are the ones which will be boosted by the large growth in AI and ML and offer huge possibilities. The current implementations with facebook and netflix of these kind of rule based deployments are state of the art currently. Automating the creation of these rulesets is the next step in automating quality assurance.

Step 5 – Fully autonomous

Compared to self-managed level, the system itself attempts to infer the parameters and goals for the parameters to be optimized from all possible collected measurements. For example in paper and pulp industry optimizing paper mill production to maximize throughput with given quality parameters.

Possibly generating suggestions on key business KPIs impacting modifications to system for example generating end user experiments like optimizing recommender systems and generating UI A/B tests.

Fully autonomous prioritization implementation optimizes product business goal level off software artifact to maximise effect and return of investment on business goals. For example eCommerce solution maximising the profit/revenue/customer amount or combination of these and thus maximizing software value streams business value.


Looking at the 5 steps towards automated software QA, two important trends can be seen. First, the validation moves more towards abstract business goals from clear technical quality goals. The validation does not anymore look just into the fact that does this function work exactly in the way it should be, but is it actually achieving the business and value goals that have been assigned to the product.

Another pattern is the growing integration of data from all phases of the software life cycle. In the initial test automation steps, only data from testing is used and all other judgements are reserved for humans. In each step more data is collected and linked to all other steps to form a data lake. This data can then be used to incorporate even more analytics into the deployment pipeline to automate the validation of the software.

A third trend is automation of key decisions from humans. This I see as the most important change enabled by automating all the steps in software development and by getting data from all these steps. This is the next holy grail of software quality systems. A system which takes into account multiple levels of data from unit test results to actual usage of software by customers and helps us humans focus on creating value in the places it is needed the most. I do not see these kind of recommender systems taking our jobs, but augmenting us in ways that helps us make better decisions and lessen the amount of decisions we have to take in a hunch, reducing the decision fatigue of us all.

Read also the first part: Test automation stairway to heaven – Automated QA without humans in the loop.

Get started with test automaton right away and begin free trial:

Start Free Trial

Topics: Test Automation, Software Development, Artificial Intelligence, AI

Henri Terho

Written by Henri Terho