Effective Test Automation in an Agile Environment

Five Common Mistakes and Their Solutions

The dynamically changing IT industry brings forth new objectives and new perspectives for automated testing in areas that were brought to life in the recent decade, such as cloud-based, SaaS applications, e-commerce, and so on. The last five years saw an immense growth in the number of agile and Scrum projects. Additionally, the IT market has changed significantly, not only with various new tools—including Selenium 2, Watir WebDriver, BrowserMob, and Robot Framework—but with approaches that have also completely changed. For example, more focus has been made on cloud-based test automation solutions both for performance testing and functional testing. Cloud-based testing of web applications is now replacing "classical" local deployments of testing tools.

Even though there are a vast number of benefits to automated testing, test automation can often fail. Some types of mistakes in test automation may include: Selecting the wrong automated tool; incorrectly using the tool; or setting the wrong time for test creation. It is also worth to pay special attention to the test automation framework and proper work scope division between the test automation and manual testing teams. The "Test Cases Selection" section of this article highlights many reasons why we must not automate certain test cases. Let’s take a closer look at the five most common mistakes of test automation in agile and their possible solutions.

1. Wrong Tool Selection
Even though the popular tool may contain a commendably rich feature set, and it's price may be affordable, the tool could have hidden problems that are not obvious at first glance. For example, there may be problems like insufficient support for the product, and a lack of reliability. This occurs in both commercial and open source tools.

Solution
When selecting a commercial test automation tool for a specific project, it is not enough to only consider the tool’s features and price; it`s best to analyze feedback and recommendations from people that have successfully used the tool on real projects. When selecting an open source freeware tool, the first thing to consider is the community support, because these tools are supported by their community only, and not by a vendor. The chances to correct arising issues with the tool are much higher if the community is strong. Looking at the number of posts in forums and blogs throughout the web is a good way to assess the actual size of the community. A couple good examples include: stackoverflow.com, answers.launchpad.net, www.qaautomation.net, and many other test automation forums and blogs which can be found by your search engine when you enter the name of the given tool in it.

In order to understand whether a test automation tool was selected properly, you should begin with answering a few questions:

Is your tool compatible with the application environment, technologies, and interfaces?
What is the cost of your chosen test automation tool?
How easy it is to write, execute, and maintain test scripts?
Is it possible to extend the tool with additional components?
How fast can a person learn the scripting language used by the tool?
Is your vendor ready to resolve tool-related issues? Is the community support strong enough?
How reliable is your test automation tool?

Answering these questions will provide a clear picture of the situation, and may help you to decide whether the advantages of this tool's usage outweigh the possible disadvantages.

2. Starting at the Wrong Time
It is a common mistake to begin test automation development too early, because the benefits almost never justify the losses of efforts for redevelopment of test automation scripts after the functionality of the application changes until the end of iteration. This is a particularly serious issue for GUI (Graphical User Interface) test automation, because it is much more likely that GUI automation scripts will be broken by development than any other types of automated tests, including like unit tests, performance tests, and API tests. Unfortunately, even after finishing the design phase you may still not know all the necessary technical details of the implementation, because the actual realization of the design selected could be achieved in a number of different ways. For GUI tests, technical details of the implementation always matter. Starting automation early may result in spending repeatable and meaningless efforts on redevelopment of the automated tests.

Solution
During the development phase, members of a Quality Assurance (QA) team should spend more time creating detailed manual test cases suitable for the test automation. If the manual test cases are detailed enough, they can be automated successfully after completion of the given feature. Of course, it`s not a bad idea to write automated tests earlier, but only in cases where you are 100 percent confident that further development within the current iteration will not disrupt your new tests.

3. Framework
Do you know what’s wrong with the traditional agile workflow? It seems to not encourage the inclusion of test automation framework development tasks, because they have zero user points. But it’s not a secret that any good and effective test automation requires both tools and framework. Even if you have already spent several thousands of dollars on a test automation tool, you still need a framework to be developed by your test automation engineers. Test automation framework should always be considered, and its development never underestimated. How does this fit into the agile process? Pretty easily, actually, and it's not as incompatible as it may seem.

Solution
How much time would you need to develop a test automation framework? In most cases it will take no longer than two weeks, which equals the usual agile iteration. Thus, the solution is for you to develop the test automation framework in the very first iteration. You are probably wondering if that means that the product will remain untested, but that is not the case, because it could be tested during the duration of that period. The workload increase on manual testers is probably unavoidable, but there is not much testing done during the first iterations—developers are more focused on backend development, usually covered by unit tests—so the process balances itself. The very first iteration will look like this: Start with both analyzing requirements and designing the test automation framework during the design phase; then develop, debug, and test it until the end of iteration.

4. Test Cases Selection
How do we select test cases for automation? That’s an interesting question and grounds for another common mistake–trying to automate all test cases. But automate them all" is hardly an answer if you are focused on quality and efficiency. Following this principle leads to useless efforts and money spent on test automation without bringing any real value to the product.

Solution
There are certain cases where it's better to automate and some cases where it doesn't make much sense to do so; it is the latter that always has the higher priority. You should perform automation when your test case is executed frequently enough—and takes time to run manually, you have a test that will run with different sets of data, or your test case needs to be run under many different platforms and system configurations.

On the other hand, test automation cannot be used for usability testing and the following instances: When the functionality of the application changes frequently; when the expenditures on test automation tools and the support of already existing tests are too high; when test automation doesn't provide enough advantages if compared to manual testing.

5. Test Automation vs. Manual Automation
A lack of coordination between your automated testing and manual testing subteams is another common mistake. This can lead to excessive efforts spent on testing and bad quality software. Why does this happen so often? In most cases, manual testing teams may not have enough technical skills to review automated test cases, so they prefer to hand-off this responsibility to the automated testing teams. This causes a different set of problems, including:

The test automation scripts are not testing what they should, and in the worst case scenario, they are testing something that is not even close to the requirements.
To ensure a successful test, test automation engineers can change test automation scripts to ignore certain verifications.
Automated tests can become unsynchronized with the manual test cases.
Some parts of the application under the test receive double coverage, while others are not covered at all.

Solution
In order to avoid these problems, it’s best to keep your whole QA team centralized and solid. The automated testing subteam should obtain the requirements from the same place as the manual testing subteam. The same list of test cases should be kept and supported for both subteams. Automated test cases should be presented in a format that is easy to understand for non-technical staff. There are many ways to achieve this, including using human-readable scenarios, keyword-driven frameworks, or just keeping the code clean while providing sufficient comments.

Conclusion
I have listed only the most common mistakes that could affect the efficiency of test automation for your project, resulting in its poor quality. It’s wise to pay more attention to the test automation activities, and to consider them an integral part of the quality assurance process of your project. If you take test automation seriously, you will be able to avoid most of the above-mentioned mistakes.

Agile testing quadrants: Guiding managers and teams in test strategies

Many software teams struggle with “fitting testing in” to the development lifecycle. For software managers and teams new to Agile development, the idea of planning and executing all the testing activities within short iterations and release cycles is daunting. I’m often asked questions such as: “When should we do performance testing? And who is going to do it? There aren’t any performance testing specialists on our cross-functional development team.” Substitute “user acceptance testing,” “exploratory testing,” “security testing,” or “usability testing” for “performance” – every Agile development organization faces similar challenges.

In my experience, a testing taxonomy such as the Agile testing quadrants (Figure 1) is a highly effective tool to help answer these questions.

Figure 1

How the quadrants work

The quadrants originated with Brian Marick's original posts on his Agile testing matrix. With his permission, Janet Gregory and I adapted this into the Agile testing quadrants, which form the heart of our Agile Testing book. The quadrants represent the many different purposes for different types of testing.

Tests on the left-hand quadrants help the team know what code to write, and know when they are done writing it. Tests on the right hand side help the team learn more about the code they’re written, and this learning often translates into new user stories and tests that feed back to the left-hand quadrants. Business stakeholders define quality criteria for the top two quadrants, while the bottom two quadrants relate more to internal quality and criteria.

The word “team” here includes both the customer and development teams. We need the business experts to provide examples that we turn into tests to drive development. We also need them to evaluate the resulting software as it is delivered in small increments, and give us fast feedback so we can make course corrections as we go.

The clouds at the quadrant corners denote whether tests in that quadrant generally require automation, manual testing or specialized tools. “Manual” doesn’t mean no specialized skills required – for instance, exploratory testing is a largely manual process but requires a high degree of expertise.

When to do which tests

The quadrant numbering system does not imply any order. You don’t work through the quadrants from Q1 to Q4 in a Waterfall style. Janet Gregory and I just chose an arbitrary numbering so that, in our book and when we are talking about the quadrants, we can say “Q1″ instead of “technology-facing tests that support the team.”

Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. However, I have worked on projects where we started out with performance testing (which is in Q4) on a spike of the architecture, because that was the most important criterion for the feature. If your customers are uncertain about their requirements, you might even do a spike and start with exploratory testing (Q3).

Q3 and Q4 testing pretty much require that some code be written and deployable, but most teams iterate through the quadrants rapidly, working in small increments. Write a test for some small chunk of a feature, write the code, once the test is passing, perhaps automate more tests for it, do exploratory testing on it, do security or load testing on it, whatever, then add the next small chunk and go through the whole process again.

How to use the quadrants

When you get the team together to plan a release or theme, go through each quadrant and identify which types of testing will be needed. Maybe this project doesn’t require usability testing, but reliability testing is critical. Talk with your customers about quality criteria. What absolutely has to work?

Next, figure out if the team (or teams) have people with the right skills to accomplish all the different types of testing, and if they already have the necessary hardware, software, data and test environments. If not, brainstorm ways to get what is needed in a timely manner.

Here are some examples. Does the team already have appropriate data for testing? If not, you may need a user story to obtain or create test data, or perhaps a business expert will arrange to provide it. If load testing is critical, but nobody on the team has experience with load testing, you could budget in time for programmers to experiment with developing a load testing harness, schedule time with a load testing expert from a different team within the company, or plan to contract with a load testing specialist. Identifying these issues early gives you time to find creative solutions.

If the team has decided to try a new practice such as using business-facing tests to drive development (known as “acceptance test-driven development (ATDD)” or “specification by example (SBE),” plan extra time for them to get up to speed with the practice. In the example of implementing ATDD, they need time to experiment with different approaches and create or adopt a testing framework. If nobody on the development team has experience with exploratory testing, you may want to plan training for that. The goal is to avoid suddenly getting stuck in mid-cycle due to lack of a particular testing skill, tool or infrastructure.

Repeat this process as you plan each iteration. For each user story, think through the testing requirements in each quadrant. How will tests be automated at various levels for a given story? When you start your planning by thinking about testing, you are likely to come up with technical implementations that make automation easier. If you lack some key testing component you may decide to postpone a theme or user story until that component can be obtained.

For example, my team had to rewrite the software to produce account statements for our financial services system into our new architecture. The legacy code had no automated regression tests, and the requirements were highly complex. We ran a huge risk of making mistakes that weren’t caught in testing, and mistakes on account statements are disastrous for the business. We decided to spend an entire sprint developing automated regression tests for the existing account statements, which required some creative experimentation. Armed with this safety net, we then developed the new code, confident that we had mitigated the risks.

It’s a tool, not a rule

Adapt and enhance the quadrants to suit your own needs. There are no hard and fast rules about which tests belong in which quadrant. Michael Hütterman added “outside-in, barrier-free, collaborative” to the middle of the quadrants in his book, Agile ALM. Mike Cohn’s Test Automation Pyramid is a good complement to help teams focus their automation efforts for maximum return on investment. Pascal Dufour integrates risk analysis with quadrants to decide what level of detail is needed in the specifications.

The quadrants are simply a taxonomy that helps teams plan their testing. Using the quadrants helps teams make sure they have all the people and resources they need to accomplish it. Think through them as you do your release, theme and iteration planning, so your whole team starts out by thinking about testing first. Focus on the purpose of each testing activity, and the quality you are building into your software.

Automation testing: Seven tips for functional test design

Automated functional tests, or user interface (UI) tests, have a reputation for being hard to maintain and for not being powerful enough to actually find bugs. However, in most cases the reasons for this are not the fault of the test tools or the test frameworks, but can be traced to poor design of the individual tests themselves.

Here are seven functional test design tips to make UI tests more maintainable and more powerful.

Don't just click, check afterward

Many automated test tools include a feature that allows a set of actions to be recorded automatically and then played back. While such record/playback features are sometimes handy when creating tests, the results of pure record/playback actions tend to be poor tests. In particular, record/playback tests do not check the state of the application after manipulating elements in the application.

Clicking, typing, selecting, and other such functions all change the state of the application in some way. Good tests will check for proper results after manipulating elements in the application. If the automated test follows a link, have the test check that the resulting page is correct. If the test generates a report, have the test check that the content of the report is correct.

Wait, don't pause

Often an application will take some time before results are available for the test to check. This is particularly common with AJAX calls in Web browsers. It is tempting to simply have the test pause or sleep for some number of seconds before checking such a result, but pausing or sleeping is poor practice. If the application takes too long to return, then the test will generate a false failure. If the application returns more quickly, then the test is wasting time while it could be moving on.

Instead of pausing or sleeping, have the test wait for a particular aspect of the application to appear. Not only does this make the test less prone to false failures, but it also makes for a more powerful test, since the test is actually waiting for and checking the state of the application upon generating the aspect the test waits for.

Use discrete locators, not indexes

It is often tempting to have a test do something like "click the third link on the page" or "select the fifth element in the list." Instead of manipulating aspects of the application according to index, though, it is worth the effort to find or create unique identifiers for such elements.

If the order of the links change, or the order of the list changes, the test will go down an unexpected path, and maintaining such unpredictable tests is quite difficult.

Check sort order with regular expressions

It is often important to the user that aspects of the application appear in the correct order. Whether columns in tables or elements in a list, or text on the page itself, it is often important that automated tests check for the correct order of things.

Say there is a set of things that should appear in order called "one," "two." and "three." Tests can check the order of things using some sort of regular expressions. Here is an example using a simplified kind of regular expression called a "glob" that is available in Selenium and other automated test tools:

| getText | glob:one*two*three |

| click | sort_thing |

| getText | glob:three*two*one |

The first step of this test checks that the text "one" is followed by the text "two" followed by the text "three." The "*" character indicates that the test will allow any characters at all between "one" and "two" and "three." The second step of the test clicks something that causes "one" "two" and "three" to be sorted in reverse order, then the third step of the test checks that the sorting was actually successful.

Don't repeat yourself

As noted above, waiting for an element in the application to appear is a good practice. It is often the case that once the element appears, the test will want to manipulate that element, for instance by clicking. It is good practice to abstract common actions into their own methods or modules, then to call those actions from the tests as required. Here is an example of abstracting the wait-for-and-click action in the syntax of Fitnesse and Selenium:

!| scenario | Wait for and click | elementLocator |

| waitForElementPresent | @elementLocator |

| click | @elementLocator |

So from a test itself we need only write:

| open | www.foo.com |

| Wait for and click | link=Welcome to Foo! |

While this example saves only one line of typing, if 'Wait for and click' is performed hundreds or thousands of times in a test suite, that is a significant improvement in maintenance and readability. Other examples of actions to be abstracted to their own modules might be logging in, selecting all the elements of a list, checking for a set of errors, etc.

Don't use conditionals

Sometimes test environments can behave unpredictably. In such cases it is tempting to use a conditional in a test to say, for example, "if this element exists, click it, if it does not exist, do something else." There are a number of problems with this approach. One problem is similar to the problem caused by using indexes instead of specific locators: if the application being tested changes, the automated test could go down completely unpredicted and unknown paths, causing false failures (or worse, false successes) and making maintenance difficult. Another problem is that one branch of the conditional statement could (erroneously) disappear altogether, and the test would never show that a bug had been introduced.

Use Javascript to create reusable random data

Finally, below is a particular example of a good practice for certain kinds of test data specifically using Selenium and Fitnesse. In this example, the test needs to enter a Social Security Number that is unique, then check that that SSN has in fact been entered into the application:

| type; | ssn | javascript{RN =Math.floor(Math.random()*9999999);while (String(RN).length < 8) { RN=RN+'0';}} |

| $SSN= | getValue | ssn |

| click | link=Save |

| type; | search | $SSN |

| GET SEARCH RESULTS CONTAINING THE SSN |

Selenium will evaluate Javascript in-line. The first line of this test types into a field whose id value is "ssn" a random 9-digit number generated on the fly by evaluating the Javascript as an argument to the type() action. The second line uses a feature of Fitnesse to store that 9-digit number from the "ssn" field in a variable called "$SSN". Then the test types that same 9-digit number into a field whose id value is "search." This is an elegant and useful way within the test itself to handle test data required to be unique. The same approach should be available in any reasonable test tool or framework.

Good design for good testing

These are just a few examples to help make your automated tests both powerful and maintainable. Many other examples exist, and each automated test tool or framework will have good design practices unique to the tool as well.

The biggest complaints about automated functional tests are that they are hard to maintain, are not powerful and that they don't find bugs. But well-designed tests are not difficult to maintain; they are powerful in that they check the state of the application being tested for aspects of function important to the user and to the application itself, and well-designed automated tests absolutely find bugs.

Testers: Put on Your End-user Hat Article by Paul Fratellone

Summary:

The more you know about the end-user, the more effective you will be as a tester. Here are some tips for adding value by thinking like your customer.

One of the biggest criticisms about testers and QA organizations is that they do not understand the business or the end-user. If you believe this to be true, it sends a clear message about not having a value-adding testing team of professionals. The more you know about the ultimate customer or end-user, the more you will become an effective, risked-based tester.

When I led QA teams in the past, I made "knowing your customer" a major performance criteria for my staff. To ensure this, I arranged field trips with business development to customer sites and had the testing team view how and why the end-users actually used the system or application. Upon returning from these field trips, the QA team started to modify how it approached end-to-end and user acceptance tests. It was so beneficial that the number of critical end-user defects dropped by more than 20 percent in a very short period of time.

This result inspired me to continue my learning. I took the product management certification course from Pragmatic Marketing and was certified in pragmatic product management in December 2009. From the course, I learned how understanding the following questions will increase the effectiveness of tests and testing teams (note: It is your responsibility to ensure you are adding value to the delivery of the product):

What problem or problems will this upgrade, enhancement, or new feature solve? This is the value proposition.
For whom do we solve that problem? This is the target market.
How will we measure success? This is the business result. What metrics will be needed to validate success has been attained?
What alternatives are out there? What is the competition doing? Is it a "blue ocean industry” or a "red ocean industry”?
Why are we best suited to pursue this? What is our differentiator?
Why now, and when does it need to be delivered? This is the market window or window of opportunity.
How will we deploy this? What will be our deployment strategy?
What is the preliminary estimated cost and benefit? Will there be a return on investment, customer satisfaction increase, or cost avoidance?

If you understand these high-level questions, you will ensure a higher level of end-user quality by designing and executing tests from an end-user’s perspective. Defining, quantifying, and weighing the many quality dimensions as perceived by your end-users, you will be able to approach testing in a very efficient and effective manner. Knowing what the user wants and needs to do with the system will enable a proactive mindset regarding requirements and feature reviews, acceptable behaviors, operational inconsistencies, interactions, and interoperability.

I have found the user manual to be a great source of knowledge for a test team. Granted, a newly developed application is devoid of a manual, as the manual gets developed along with the application. But, during my independent consulting years, I relied heavily on these manuals to gain an operational business perspective. Be careful, though, as they can be dated and may become stale depending upon how much the end-user relies upon them.

This perspective naturally leads to an understanding of where the potential risks are to the business:

What are the most common and critical areas of the functionality from the user’s point of view?
How accepting should the system be to incorrect input
Is the description complete enough to proceed to design, implement, and test the requirement processes?
What is an acceptable response time for the users? Are there stated performance requirements?
Which problems and risks may be associated with these requirements?
Are there limitations in the software or hardware?

Which omissions are assumed directly or indirectly?

Regardless of the delivery methodology you are involved with (iterative, agile, waterfall, etc.) the above will be applicable. A test strategy should be unaffected by the delivery method. It should be a relative constant, setting the goals, objectives, and approach. Your test strategy should articulate test governance and other standard operating procedures that the delivery team will be following, including but not limited to configuration, release, source code, defect, and test case management. Any test tools and test execution tools that will be deployed in the delivery of the application or system should also be identified. Clearly articulated expectations will be how the team will measure success and what defines "done." Without this, how will the team know when good is good enough? This document should also include a contingency plan for any regression that might take place. What severity and quantity of defects will be deemed acceptable, and how will the defects be handled with remaining deployments? Additionally, a test strategy needs to state what test metrics will be used and how they will be reported, communicated, and escalated when results display negative trends in the acceptance criteria.

The test plan should document the techniques and methods that will be employed to validate the system under test. The test plan should detail the estimates of the test cycles in relation to the delivery plan. A test estimation algorithm and how one derived these estimates are best guesses at this point, but be sure these assumptions are reviewed with the delivery team.

The same holds true for the development team. If the project is to be delivered in iterations, then naturally the team will jointly develop the estimated costs and duration. It is important to highlight to the delivery team the estimated defect rate for the delivery. As this rate is approached or exceeded, the impact to remaining deployments and regression can cause not only the timeline to skew but also costs to escalate. Feedback from these defect trends should trigger a re-estimation of remaining iterations or sprints, thereby increasing accuracy and confidence. The methods and techniques documented in the plan will support the estimation of costs and duration. Examples of these techniques include requirements-based, combinatorial, model, and scenario-based testing. Each technique has unique attributes that will be associated with the various levels of structural and functional testing. Test estimates will be challenged, so teams need to stay focused. Should all business-critical features and functions have more than 90 percent test case permutations covering the various combinations of valid, error, and user profiles? Do all the users view the feature set similarly and agree about what is critical to the operation of their needs and business?

Deciding at what level to stop testing is difficult, but there is the ever-present law of diminishing returns. Review the approach and risks with the business and product owners, and gain their insight into where there could be excessive testing and what is an acceptable level of risk. As a value-adding testing team, you must quantify the costs by articulating the number of test cases and how long it will take to deliver the quality the end-user is expecting. Understand where the greatest risks to the business exist—features frequently used by most or all customers, financial impacts from failure or errors, feature complexity, defect density, and historical data on problem areas or feature sets.

When it all comes together, the team can circle back to those quality dimensions highlighted earlier in this paper. Reliability, usability, and accuracy will manifest in the number of test cases and techniques used to satisfy the level of quality that the end-user is expecting and on which the business owner must plan to spend. Complete transparency enables the team to make sound business decisions and decide on appropriate levels of risk and tradeoffs when plans are not being met. The cost-risk-benefit equation of quality will be used as the team makes adjustments to content time and cost. There should be no surprises when the team is faced with the tough decisions that always arise during the deployment of software.

In the delivery of software, testers can wear many hats. Teams that are able to think like the end-user will make a significant contribution in ensuring that the test team is adding value and focused on meeting the client’s expectations.