This post includes definition of Cloud Testing, Advantages of Cloud Testing, Limitations of Cloud Testing, Examples and Companies providing Cloud Testing services.
What is cloud testing?
Cloud testing is one of the newest forms of testing. Applications designed for cloud usage run remotely from the Internet. Many cloud applications will have large numbers of users, so performance testing and load testing are particularly important for cloud application. However, cloud applications are also subject to normal quality problems, security problems, and usability problems. As a general rule the applications intended to operate in the cloud will have a normal set of "ground" tests prior to "cloud" testing, which usually occur at about the same time as integration or system testing. [Via: The Economics of Software Quality By Capers Jones, Olivier Bonsignour, Jitendra Subramanyam]
Cloud Computing is a technology which gives computation, software, data and storage services over the network. It is not necessary for the user to know the physical location and configuration of the system which deliver these services to them. It is the use of any services offered commercially in the real time over the internet. What is Cloud Testing – It is software testing using Cloud Computing. [Via]
Advantages of cloud testing
Cloud-based testing has several unique advantages listed below.
Costs can be reduced by taking advantage of computing resource in the cloud – This refers to effectively using virtualized resources and shared cloud infrastructure to get rid of required computer resources and licensed software costs in a test laboratory. Cloud testing brings in the Pay As You Go, Pay Only For What You Need - Cloud Testing eliminates the cost of building and maintaining a test lab for load and performance testing. In other words Cloud is cheaper: As resources are used on demand.
Building a test lab used to be prohibitively costly for medium and large scale load tests. Cloud computing delivers bandwidth and computing power to achieve tests in the millions-of-users levels at commodity prices. Traditional approaches to test a software incurs high cost to simulate user activity from different geographic locations. Cloud Testing allows to run from multiple locations around the world - Cloud Testing environments are a world-wide phenomenon.
You can easily take advantage of scalable cloud system infrastructure to test and evaluate system (SaaS/ Cloud/Application) performance and scalability.
Test quality can be improved: 30% of defects (approx) are result of inaccurate configuration of test environments. Cloud-based testing service providers offer a standardized infrastructure and pre-configured software images that are capable of reducing such errors considerably.
Organizations get good time to market: Creating on-premise test environments can be time-consuming and cause project delays. With cloud-based testing, organizations no longer need to wait to buy and configure servers, or to license and install programs and testing tools.
Read Testing on the Cloud for more Key Benefits of Testing on the Cloud.
Limitations of cloud testing
Security of the data is the biggest disadvantage. Storing data in a cloud means the data is, in theory, accessible to anyone. Mostly because data and code may be stored in a remote location beyond an organization's legal and regulatory jurisdiction.
Yet another challenge is that some cloud providers offer only limited types of configurations, technology, servers and storage, networking and bandwidth, making it difficult to create real-time test environments.
Improper choice of cloud-based use and pricing options is another risk. While some vendors offer pay-as-you-go services, they are only cost-effective when the right plan and service provider are chosen for the anticipated needs (e.g. space vs. RAM vs. bandwidth). Costs can quickly spin out of control if resource estimates differ wildly from actual usage.
Integration testing in clouds - Although we have seen numerous published research papers addressing software integration testing issues and strategies, not much research results have been applied in the real engineering practice. One of the major reasons is the existing software and components are developed without enabling technology and solution to support and facilitate systematic software integration. In a cloud infrastructure, engineers must deal with integration of different SaaS and applications in/over clouds in a black-box view based on their provided APIs and connectivity protocols. This could cause a lot of extra integration costs and difficulties due to the following issues:
- There is a lack of well-defined validation methods and quality assurance standards to address the connectivity protocols, interaction interfaces, and service APIs provided by SaaS and clouds APIs.
- There is a lack of cost-effective integration solutions and framework to facilitate software application integration and assembly inside clouds and over clouds.
Infrastructure requirements: It is vital that Infrastructure requirements are rigorously set, because the very flexibility that the cloud offers for testing environments can itself be a risk if the requirements for those environments are inappropriate. Results will then be poor and negative perceptions of the cloud as a test environment will result from what was really an inattention to requirements around the infrastructure. Using a simple checklist will help reduce this risk to a minimum.
References:
Cloud Testing- Issues, Challenges, Needs and Practice
PushToTest
Wikipedia
Performance Testing in Cloud A pragmatic approach
Taking testing to the cloud
Cloud Testing: Benefits, Limitations & Challenges
As an example of cloud testing, you can read these two interesting posts: How the cloud has changed testing forever and An Example Demonstrating Cloud Test Automation (related to TestComplete).
Companies providing cloud testing services or involved in cloud testing:
1. SOASTA
2. Keynote
3. Monitis
4. Apicasystem
5. Compuware
6. PushToTest
7. Cloud Harmony
8. IBM
9. Loadstorm
10. Cloudsleuth