Cloud testing: The cloud and your testing practice

The impact of the cloud on testing practices has grown with the cloud’s growing presence in the IT space. Testing practices are now dealing with several aspects of the cloud simultaneously -- three aspects of the cloud that directly impact our testing practices are: using the cloud to create scalable testing environments, non-functional testing of cloud-based solutions and functional testing of cloud-based solutions. In short, the cloud can be used:

* As a testing enabler
* For non-functional testing
* For functional testing (unit, integration, system, and regression)

While these are clearly distinct aspects of the cloud space and the discipline of testing, there are relationships between these aspects of the cloud that are being ignored or “glossed over” by both vendors and proponents of cloud-based computing -- specifically the non-functional risks around security/integrity and performance.

The cloud as a testing enabler

The cloud provides the opportunity to create scalable testing environments that can be easily ramped up or down given the immediate needs of your testing organization. Whether this type of scalable solution is an appropriate fit for your organization is dependent on the idle time of your development and testing infrastructure throughout the year. If most of your infrastructure is in use, or your infrastructure is inadequate, then a straight investment comparison of own versus rent/lease should be possible. On the other hand, if much of your development and test infrastructure remains idle throughout the year, then leveraging a cloud-based solution may alleviate your overall infrastructure cost.

There are other factors that must be considered before moving testing assets to a cloud-based solution. These factors do not involve the capacity of cloud-based testing solutions -- the capacity is certainly there, but instead the security and integrity of these solutions. The security of cloud-based solutions still remains problematic -- with major security incidents happening on regular basis. If your production environment has not moved to the cloud and you have not created obfuscated test data, then you are exposing your organization to significant security risk by moving testing to the cloud. The question to ask is, “Am I exposing my organization to additional risks by moving to a cloud-enabled testing solution?” If the answer is “yes,” then a proven risk mitigation plan needs to be put in place before moving testing assets to the cloud.


Non-functional testing of cloud-based solutions

As your IT organization and infrastructure moves into the cloud, non-functional testing becomes critical. Recent experience has shown us that most of the risks associated with the cloud come from non-functional requirements not being met or often not even being articulated and therefore not being tested or supported. The areas of proven vulnerability are performance, security, and disaster recovery/management -- recent examples being (April/May 2011):

* Amazon’s “glitch” (the last week of April) that caused numerous Websites hosts to crash or run very slowly.
* Sony of Japan revealed that hackers accessed 100-million PlayStation accounts including names, addresses, passwords, and possibility credit card details.
* Amazon Web Services (AWS) Virginia data centers in its US-East-1 region were down leaving many of its customers with no service and no service alternatives.

From disaster recovery/management perspective you need to ensure a plan has been put in place by the cloud provider to ensure interruptions in their service will be addressed. This is somewhat problematic since the key cloud providers have not yet addressed this issue -- witness Amazon’s “glitch” in late April. Your testing organization still needs to identify the risk, and test any recovery procedures presented by the cloud provider.

From a performance testing perspective the testing organization can apply traditional tools and techniques while ensuring the infrastructure of the cloud-based solution closely resembles (or is) production. There are additional factors that will need to be addressed, the most critical being:

* Addressing the loads that will be applied by other clients/customers of the cloud provider.
* Addressing the loads that will be applied against the internet providers (example: Cyber Monday).

From a security testing perspective, the testing practice will need to become much more aggressive than many test organizations have been in the past. Your business and your data now reside on a third party that has not yet presented a sound security solution -- with all transactions travelling over the Internet. The testing organization will have to address the security of the application presentation layer, the service layer, the data layer, and now the architecture/infrastructure to ensure security requirements have been met. In the past, security risks have often been mitigated by the nature of the architecture -- in-house applications on a secure network with little direct contact with the “world.” Now your applications will exist in the cloud, or cyberspace, with all the benefits and risks that provides.

Functional testing of cloud-based solutions

The test processes and technologies used to perform functional testing against cloud-based applications are not significantly different than traditional in-house applications. Adjustments do need to be made for the non-functional aspects of the application space that relate to security and data integrity. If testing involves production data then appropriate security and data integrity processes and procedures need to be in place and validated before functional testing begins.

The cloud and your testing practice

Awareness of the non-functional risks around the cloud are critical to successfully testing (functional and non-functional) or leveraging (test labs) the cloud. The responsibility for testing the non-functional aspects of a cloud-based application may reside within the testing practice or the infrastructure/security team. As a rule of thumb, if you are dealing with a public cloud or a provider-supplied cloud, then testing responsibility should stay within the testing practice. If you are dealing with a private cloud, then testing responsibility could be handled by the infrastructure/security team -- at least from a performance and security perspective. In either case, all parts of the IT organization need to work together to mitigate the risks presented by cloud-based solutions to the overall enterprise.