Accelerating DevOps with cloud-based test automation: A case study of doing what we preach

Jan 19, 2021 / by Esko Hannula

Qentinel Pace is Qentinel’s cloud-based test automation platform. It was built on three premises: scale, speed, and transparency. Because the scale of testing keeps growing test automation platforms must scale, too: from a single developer to a large organization, from small test volumes to huge volumes – and back. Because the software release times are shrinking, testing must be fast: time must not be wasted on patching test environments or maintaining cumbersome test cases. Because both speed and quality are important, decisions about releasing software or changing something in the software process must be fast and fact-based.

The team that wrote Qentinel Pace has its own product in active use: Qentinel Pace is used to test itself. Test volumes vary a lot in a release model where updates may be pushed out several times per day and new features are released every four weeks. The image below illustrates our daily test execution volumes in November 2020.


As you can see, both the execution times and the volumes vary a lot. Weekends are almost dead while a feature release causes a peak. Three hundred Qentinel Pace test minutes per day corresponds roughly to the daily work of 20 manual testers. While considerable, these numbers are small compared to many of our customers. Some test as much as quarter-a-million minutes per month. As we have all test infrastructure in the cloud an adequate capacity is always available and we pay for it only when we use it.

The tests themselves, as well as the test results, are also in the cloud. This means we are all always collaborating on the very same tests and have up-to-date test results always accessible. Tests cases are kept in version control so that different versions and branches can be tested effectively. Qentinel Pace integrates seamlessly with Git-based version control software. Because Qentinel Pace has a project level access control we can easily and securely invite our subcontractors on another continent to work with us.

The picture below is the test results view accessible to anyone in the project. Those who need to dig deeper have the test logs with screenshots and videos of failed tests behind just one click.


When regression tests get executed by a machine the testers can use their time on exploratory testing with new features. But there are still testing tasks where testers and developers may waste a lot of time: setting up and patching the testing environments and maintaining the regression test suite are probably the worst time-eaters. With Qentinel Pace the time required to set up a full-featured test environment is about 90 seconds.

Test design and maintenance still require human work but thanks to Qentinel’s PacewordsTM scripting technology and the interactive test development tools even a 50+ old CEO such as me can automate tests or maintain tests created by someone else. And yes, if you are a hardcore guy and loathe interactive tools just open your favorite editor and write the script directly.

Once in a while, I find something in Qentinel Pace that I believe could be a bug. I typically use the product itself to record a test case that can reproduce the suspicious behavior. Below is an excerpt of such a session.


As you can see, a PacewordsTM script is simple enough for anyone to follow: no xpaths, no code whatsoever, just keywords that follow the line of the thought of a human user. (If you want, you may write Python in Qentinel Pace, but you don’t need to.) I did not even need to write the script but recorded the whole thing using the LiveTesting feature of Qentinel Pace. It took 80 seconds.

If I need to touch my script the editor of Qentinel Pace helps me by suggesting what I’m likely to need next. It does this by analyzing the existing tests of the same project and applying some machine learning to guess my thoughts. In this case, the only change I made after the recording was to store my password in the secure vault of Qentinel Pace and modify the script to refer to it by the abstract name ${PASSWORD}. Now my secret is protected even when other people run the script or investigate the test logs.

I prefer to rely on facts when I need to know how close we are to being able to release or where I should look to find problems. I love chatting with the development team but, honestly, I’m not clever enough to understand how ready we are when they say things like “docker image still needs an update” or “platform API has too much load” or “well, it doesn’t suck right now”. Luckily, my product can provide me with data that helps me understand or, at least, make better questions. Below is the thing we call Quality Intelligence dashboard. It has a hierarchic, normalized KPI tree and a cause-effect map nicely place on the DevOps cycle. This snapshot tells that the current development release still is far from complete.


I usually look at the top-level metrics on the upper side of the dashboard first. If you are a DevOps virtuoso you may recognize that it follows the Flow Framework model. If everything is green, that’s all I need to know. I can easily trace the red light from release quality back to code quality and security. If I want to know more I click any of the metrics and get a graph showing the measurement history of that particular KPI. The dashboard is customizable. The default dashboard has those KPIs Qentinel Pace can calculate from its testing data. But we pull data also from other sources such as Git, Azure DevOps, TIOBE code analyzer, or Jira. There’s even an open API in which one can integrate any data source.

The best feedback always comes from customers that love the product and help us make it even better. The second best feedback is the way the Qentinel Pace team has managed to continuously improve their DevOps performance using their own tool.

If you want to try Qentinel Pace, you can start your test automation journey signing up for the free version.

Start Free Trial


Topics: DevOps, Qentinel Pace, Software Testing, Test Automation, Software Development

Esko Hannula

Written by Esko Hannula