How to automate 18 109 test cases and keep maintenance effort under 15%

Mar 11, 2021 / by Antti Heimola

How to automate over 18 000 test cases in two years while keeping maintenance workload under 15 per cent - 14,3% to be exact. The system under test was a tax processing software. Despite several attempts, no one could successfully automate testing on this system earlier.

Background

Finnish Tax Administration started renewing its tax processing system. They realized early enough that manual testing only was not an option; a test automation solution was needed. The tax processing software GenTax by Fast Enterprises was selected due to its extensive configuration capabilities that were required to fulfil the specified requirements by the Finnish Tax Administration. Qentinel made a promise that automated test cases can be executed not only during the five-year release phases, but over the whole production phase, several decades. What made the promise even more challenging was the fact that there had been several GenTax test automation trials before, without success.

This case study contains the work from November 2015 to August 2018 which can be divided into three phases:

  • The initial test automation system development
  • Setting up the test automation process, infrastructure, and the team
  • Full scale test automation

Following sections will walk you through these phases.

Initial test automation system development - test automation challenges

Previous failed test automation trials made the solution provider sceptical about the test automation project at the Finnish Tax Administration. Fortunately, the main root causes were found and understood early in the project. There were three main issues to be resolved:

  • Unexpected application delays to be handled efficiently
  • Data table control and value verification
  • Reliable ways of locating user interface elements

Selenium tool is widely used in web application test automation. Unfortunately, it could not be used as is for GenTax test automation work, but it was a good reference for the GenTax test interface development work. Usability, reliability, and performance were identified as the key quality characteristics for the solution development, not to ignore the functionally, of course.

Test automation approach - Qentinel PaceWords

Qentinel had been using a keyword-based test automation technique before the GenTax test automation project. The same approach was also selected for the project due to test script readability requirement; non-technical people should be able to review test automation scripts. The keyword-based solution is using single-step keywords, like ClickText, TypeText, etc. Steps what a tester would use when testing an application manually. There are many other keyword driven test automation techniques, for example, Behaviour-Driven Development, ActionWords, etc., which normally perform some application specific operations or routines, instead of a single step.

The technique is called as Qentinel PaceWords. Some of the reasons why to use the approach:

  • They are not application specific, but reusability across different projects
  • Test scripts can be reviewed by non-technical people, like Subject Matter Experts (SME)
  • Easy to use, test automation with PaceWords does not require programming skills
  • Test case updating work is fast, i.e. low maintenance effort

During the development phase of the GenTax test automation solution there were two key test automation metrics: test case automation time and test case maintenance time. Target times were set to 30 minutes and 10 minutes in a defined scope which were achieved in four months. The solution was considered then as a “good enough”. Test cases were automated all the time along with the test automation system development, but after reaching the targets the focus was moved to ramping up the test automation team, robot infrastructure, and the test automation process improvement.

Setting up the test team, automation process, and infrastructure

Scaling up test automation has become a common practise along with continuous integration (CI). When scaling up a test automation team and process in a complex environment, it requires formal procedures to have an efficiently working team and a defined test automation process.

Setting up test automation required several training sessions when five new test automation engineers joined the team while two original members left the team. During the team ramp-up phase several new metrics were introduced, like the number commits, to follow the learning progress. Test case review process was also introduced to ensure common scripting conventions.

Test automation involving complex business processes requires subject matter experts to design test cases to be automated. It introduces process issues like how to specify, manage, and review thousands of test cases particularly when there are some 20-30 people involved in the process. It took several iterations to fine tune the process, for example, to have the right level of details in test case descriptions under the test management system.

One of the key items in a successful test automation project is not the test automation itself, but automation of things in general, which also had an important role in the test automation project. It is often more cost efficient to automate test environment and data management first than jumping into automating functional tests. You need to design the right solution right to be successful in test automation including infrastructure components.

As of August 2018, GenTax test automation solution consisted of 28 robots executing more than 500 hours of automated test cases daily basis controlled by an automatic pipeline. Scalable and automated infrastructure is the key in test automation projects including an automatic metric collection process, and dashboards proving real-time information for key stakeholders.

Full scale test automation

The test automation process and the team were fully functioning in June 2016, after some seven months of initial work. At that point there were already 720 regression test cases automated running in the CI environment. The team consisted of test automation engineers and software developers who were responsible for PaceWord development, maintenance, and CI based Robot environment. Some 15-20% was development work allocation. In addition to the core test automation team, there were also over 20 SMEs who were not only responsible for test case design but also responsible for reviewing and approving automated test cases.

Project in numbers

Some of the project numbers in August 2018:

Number of automated test cases 18 109
Number test case steps 709 526
Number of test suites 2109
Number of PaceWords covering 80% of all steps 19
Maintenance effort of total working hours 14.3%
Test execution hours per day 509
Number of tests automation Robots 28
Average test automation defect detection rate in a two-week sprint 11

 

There were many other metrics used in the project proving more detailed information, but the metrics shown here should provide some insight into the test automation work done at the Finnish Tax Administration.

Conclusion - Case GenTax test automation project

Being part of the GenTax test automation project, it was very rewarding to see the team growing from couple of individuals to efficiently cooperating teams. This all could not have been possible without the support and control from the Finnish Tax Administration project leads. Customer’s involvement is often underrated factor in test automation projects. Regular bi-weekly reviews had the major positive impact into the PaceWord driven test automation usability, reliability and efficiency, the defined key quality characteristics of the test automation solution.

Continuous improvement had also an important role in the project from the beginning, which would not have been successful without a good set of the metrics. There are often defined metrics in projects, but far too often they are not used that well. Metrics in GenTax test automation project guided the project from one phase to another, although the phases were not defined in detail in the beginning.

There are many web sources listing top benefits of the test automation, seldom in the form like Jarkko Levasma at Finnish Tax Administration wrote it: "Automating regression testing for our highly automated, legislation driven, and certainly very complex IT system has proven to be a vital part in maintaining correct and timely taxation in Finland. As many changes to the taxation rules are being introduced all the time it is crucial to know what you have and have not changed before moving changes into production. Knowing that we have not changed anything unintentionally is the main benefit of comprehensive test automation - certainly helps me sleep better at night."

As a conclusion referring to Hans Buwalda’s words back in 2002: “I have learned that testing and test automation remain very challenging areas and need the involvement of experienced specialists to do it right”. Nothing has changed since; it is still the same.


The Finnish Tax Administration case study was written as a part of the ITEA3 Testomat project – The Next Level of Test Automation. The project started in October 2017 and the case study was about to be published in a Testomat book/booklet late 2020. There have been unfortunate delays due to the COVID-19, therefore this early publication of the case study in the form of a blog. Thank you both Testomat and Finnish Tax Administration for approving this. See https://www.testomatproject.eu/ for more details of the project.

If you want to try out the keyword technique used in the Tax Administration case, there is a Free Trial version of Qentinel Pace available including the same PaceWords. Click just ‘Start Free Trial’

Start Free Trial

 

Topics: Software Testing, Test Automation, Software Development, Test design, Test architecture, RPA

Antti Heimola

Written by Antti Heimola