Experiences in applying algorithms to common testing challenges

Oct 16, 2020 / by Kimmo Nupponen

Here’s a challenge. How would you test an application that is supposed to check that an email address provided is a proper one? Similarly, how would you thoroughly test an application that takes an IBAN number? Or personal identity number? Or a VAT number? Or business ID? How would you identify a complete enough test set for testing the application?

It’s easy to find a handful of valid data values such as email addresses and then figure out a few invalid ones. But how can you argue that those handful, or even tens or hundreds, of examples are enough to properly test the functionality?

Here’s another challenge. You have a system that accepts a configuration with a handful of different options each option assuming a few distinct values. How would you make sure that you properly test those various configurations? Sure, again it’s rather easy to find a collection of configurations, but how can you argue that your selection is even remotely enough?

I would argue that these types of testing challenges are more than common. We see them more than often and we just assume that testers will do decent enough job testing them.

These are types of challenges that we at Qentinel aim to answer with automation. We have turned challenges, just like ones mentioned above, into well-defined problems for which we have created algorithms that are based on well-founded research. Instead of asking testers to solve combinatorial problems as eluded earlier, we outsource the problem to the computer.

Let’s take that email address validation function as an example. As email address syntax can be described using a regular language, our algorithm takes that regular language and automatically creates test cases for the language. As your applications email (or f.ex. IBAN and ISBN, for that matter) validation should be based on this language, by automatically generating tests for all the “corner cases” of the language, we can make sure that we properly test the functionality. Rather common problem with different test generation approaches is that they may generate impractical number of tests and that’s why we rigorously optimise the tests being generated while making sure that we don’t miss those corner chttps://en.wikipedia.org/wiki/Regular_lases.

Test scripts in our case can be defined using an extremely intuitive and easy to understand notation which we call PaceWords. In order to for example type in something to an email textbox, you would have

TypeText    email   info@qentinel.com
VerifyText  Valid email!

Now the example above shows that we are testing the system with one particular email address. But what about those corner cases mentioned above? For those, we simply use a special identifier that triggers the test generation

TypeText    email   VALID_EMAIL_ADDRESS

And that’s all there is! Everything from now on is carried out by the algorithm! Simply by having the identifier there is enough to trigger fully automated method to test that your email validation indeed accepts valid email addresses.

What about invalid email addresses? The “negative” cases? Often those are more interesting, and we tend to find more issues around invalid cases. In order to perform test generation for invalid emails, you would simply write

TypeText     email   INVALID_EMAIL_ADDRESS
VerifyText   The entered email address is invalid!

Which would then trigger generation of carefully selected set of strings that all look like email addresses but in some respect are violating that email language.

What about that second challenge that was raised at the beginning? To answer that problem, we apply a combinatorial method, which is based on the premise that many errors in software can only arise from the interaction of two or more parameters. This is called interaction principle, which states that most software failures are induced by single “factor” faults or by a combinatorial effect of two factors, with progressively fewer failures induced by interactions between more factors. This is a practical hypothesis suggesting that if there is a fault that manifests with a specific setting of input variables, it is most likely caused by only a small subset of those variable values. This implies that software faults can be discovered by relatively simple and small tests.

While finding these simple all small set of combinations is computationally difficult algorithmic problem, it is next to impossible challenge for us humans. Therefore, we give users a simple interface where they can define the individual options while leaving the calculation of best possible set of combinations to the computer. For example

TypeTextBrowser     [Chrome, Firefox, Edge]
ClickText           [Submit, Cancel]

Where the individual options are presented using a list notation. Again, that’s all there is!

The key takeaway is that testing is difficult, and we humans are poor in combinatorics at least when compared to computers. Our time is valuable, and we should not be bothered with problems that an algorithm can easily solve.

Hope this inspired your interest and I welcome you to take a look how this works in practice by heading to www.qentinel.com.

You should also check out why Automated test design is a key to productivity.

Topics: Test Automation

Kimmo Nupponen

Written by Kimmo Nupponen