Automation testing: Seven tips for functional test design

Automated functional tests, or user interface (UI) tests, have a reputation for being hard to maintain and for not being powerful enough to actually find bugs. However, in most cases the reasons for this are not the fault of the test tools or the test frameworks, but can be traced to poor design of the individual tests themselves.

Here are seven functional test design tips to make UI tests more maintainable and more powerful.

Don't just click, check afterward

Many automated test tools include a feature that allows a set of actions to be recorded automatically and then played back. While such record/playback features are sometimes handy when creating tests, the results of pure record/playback actions tend to be poor tests. In particular, record/playback tests do not check the state of the application after manipulating elements in the application.

Clicking, typing, selecting, and other such functions all change the state of the application in some way. Good tests will check for proper results after manipulating elements in the application. If the automated test follows a link, have the test check that the resulting page is correct. If the test generates a report, have the test check that the content of the report is correct.

Wait, don't pause

Often an application will take some time before results are available for the test to check. This is particularly common with AJAX calls in Web browsers. It is tempting to simply have the test pause or sleep for some number of seconds before checking such a result, but pausing or sleeping is poor practice. If the application takes too long to return, then the test will generate a false failure. If the application returns more quickly, then the test is wasting time while it could be moving on.

Instead of pausing or sleeping, have the test wait for a particular aspect of the application to appear. Not only does this make the test less prone to false failures, but it also makes for a more powerful test, since the test is actually waiting for and checking the state of the application upon generating the aspect the test waits for.

Use discrete locators, not indexes

It is often tempting to have a test do something like "click the third link on the page" or "select the fifth element in the list." Instead of manipulating aspects of the application according to index, though, it is worth the effort to find or create unique identifiers for such elements.

If the order of the links change, or the order of the list changes, the test will go down an unexpected path, and maintaining such unpredictable tests is quite difficult.

Check sort order with regular expressions

It is often important to the user that aspects of the application appear in the correct order. Whether columns in tables or elements in a list, or text on the page itself, it is often important that automated tests check for the correct order of things.

Say there is a set of things that should appear in order called "one," "two." and "three." Tests can check the order of things using some sort of regular expressions. Here is an example using a simplified kind of regular expression called a "glob" that is available in Selenium and other automated test tools:

| getText | glob:one*two*three |

| click | sort_thing |

| getText | glob:three*two*one |

The first step of this test checks that the text "one" is followed by the text "two" followed by the text "three." The "*" character indicates that the test will allow any characters at all between "one" and "two" and "three." The second step of the test clicks something that causes "one" "two" and "three" to be sorted in reverse order, then the third step of the test checks that the sorting was actually successful.

Don't repeat yourself

As noted above, waiting for an element in the application to appear is a good practice. It is often the case that once the element appears, the test will want to manipulate that element, for instance by clicking. It is good practice to abstract common actions into their own methods or modules, then to call those actions from the tests as required. Here is an example of abstracting the wait-for-and-click action in the syntax of Fitnesse and Selenium:

!| scenario | Wait for and click | elementLocator |

| waitForElementPresent | @elementLocator |

| click | @elementLocator |

So from a test itself we need only write:

| open | www.foo.com |

| Wait for and click | link=Welcome to Foo! |

While this example saves only one line of typing, if 'Wait for and click' is performed hundreds or thousands of times in a test suite, that is a significant improvement in maintenance and readability. Other examples of actions to be abstracted to their own modules might be logging in, selecting all the elements of a list, checking for a set of errors, etc.

Don't use conditionals

Sometimes test environments can behave unpredictably. In such cases it is tempting to use a conditional in a test to say, for example, "if this element exists, click it, if it does not exist, do something else." There are a number of problems with this approach. One problem is similar to the problem caused by using indexes instead of specific locators: if the application being tested changes, the automated test could go down completely unpredicted and unknown paths, causing false failures (or worse, false successes) and making maintenance difficult. Another problem is that one branch of the conditional statement could (erroneously) disappear altogether, and the test would never show that a bug had been introduced.

Use Javascript to create reusable random data

Finally, below is a particular example of a good practice for certain kinds of test data specifically using Selenium and Fitnesse. In this example, the test needs to enter a Social Security Number that is unique, then check that that SSN has in fact been entered into the application:

| type; | ssn | javascript{RN =Math.floor(Math.random()*9999999);while (String(RN).length < 8) { RN=RN+'0';}} |

| $SSN= | getValue | ssn |

| click | link=Save |

| type; | search | $SSN |

| GET SEARCH RESULTS CONTAINING THE SSN |

Selenium will evaluate Javascript in-line. The first line of this test types into a field whose id value is "ssn" a random 9-digit number generated on the fly by evaluating the Javascript as an argument to the type() action. The second line uses a feature of Fitnesse to store that 9-digit number from the "ssn" field in a variable called "$SSN". Then the test types that same 9-digit number into a field whose id value is "search." This is an elegant and useful way within the test itself to handle test data required to be unique. The same approach should be available in any reasonable test tool or framework.

Good design for good testing

These are just a few examples to help make your automated tests both powerful and maintainable. Many other examples exist, and each automated test tool or framework will have good design practices unique to the tool as well.

The biggest complaints about automated functional tests are that they are hard to maintain, are not powerful and that they don't find bugs. But well-designed tests are not difficult to maintain; they are powerful in that they check the state of the application being tested for aspects of function important to the user and to the application itself, and well-designed automated tests absolutely find bugs.