Automated Testing of Cloud Applications

PROJECT SUMMARY

Cloud computing has become a new computing paradigm where the cloud could provide both virtualized hardware and software resources that are hosted remotely and use-on-demand services model. One typically service model of cloud computing is Platform as a Service (PaaS). Such cloud platform services, such as Microsoft Azure platform, provide convenient infrastructures for conducting cloud and data-intensive computing. After deploying an application to the cloud, one can access and manage it from everywhere using a client application, such as the internet browsers, rather than running or storing the application locally.

With increasing adoption of cloud computing across various research and educational areas, quality assurance of cloud applications becomes an increasingly important task. One typical way of software quality assurance is developer testing: developers test their code as they write it, often in the form of unit testing. To reduce developer efforts in developer testing, an automatic testing tool such as Pex from Microsoft Research could be used to automatically generate test data for a unit under test such as a class in an Azure cloud application. However, the behavior of a unit in a cloud application is often dependent on the state of the cloud environment. Such environment dependency poses significant challenges for an automatic testing tool. To address these challenges, we develop a parameterized mock model to simulate various possible return values of API methods that interact with the Azure cloud environment. This model could enable an automated testing tool such as Pex to explore feasible execution paths in the cloud application under test, and generate test data and mock cloud states for achieving high structural coverage of the cloud application.

All the source code can be downloaded here.

PEOPLE

Facult

Tao Xie (Principal Investigator)

Students

Linghao Zhang (PhD Student)

Collaborators

Nikolai Tillmann, Peli de Halleux (Microsoft Research)

Empirical investigations

We surveyed 18 open source Azure applications mainly form codeplex and google code, among which 5 projects include unit tests. Most test cases, which are written for testing a unit that interacts with the cloud environment, begin with a manual environment-setup-preparation step and these test cases must run against local cloud environment simulator.

Testing Example with Stub Cloud Model

Now let us use an example illustrate how our approach works. In the simulated cloud queue storage, we use an instance of List<StubCloudQueue> named StubQueueList to store all the existing queues, and use an instance of List<CloudMessage> to represent a message queue. Initially, Pex would arbitrarily choose N (representing the total number of StubQueueList) to be 0, and there will exist no queue. So the execution would directly jump out of the loop at Line 4 and the path condition collected by Pex is “N == 0 && N >= 0”. Then, Pex next tries to come up with a new cloud state by flipping the condition “N == 0 && N >= 0”. By consulting the underlying constraint solver with the new path constraint, Pex chooses N to be 1 in the second run. Next, Pex would create a new StubCloudQueue and arbitrarily choose a name “/0” for this queue. Pex also chooses M (representing the total number CloudMessages in queue “/0”) to be 0. Then the execution would take the false branch at Line 8 because there is no message in StubCloudQueue “/0”. A new path condition collected by Pex in the second run is “M == 0 && StubQueueList.name == “/0” && N == 1”[1]. If the Depth-First-Search strategy is adopted, Pex would try to flip “M == 0” to “M != 0”. In the third run, the path constraint is “M != 0 && StubQueueList.name == “/0” && N == 1” and one queue with one message is created. The execution would take the false branch at Line 12, adding a new constraint “StubQueueList.name! = PhotoQueue”. Once Pex flips the constraint “StubQueueList.name! = PhotoQueue” in a certain run, Pex could create a new queue named “PhotoQueue”, and then the true branch of Line 12 can be covered. As Pex keeps exploring the unit under test, finally, all the feasible paths/blocks can be covered with various cloud states. If the feasible paths are infinity or too many, Pex would stop at a certain termination condition. Finally, our approach generates nine cloud states that cover all the blocks in Dispatch method after 194 runs. The details of these generated cloud states can be found in our project web site [10].

As we have discussed earlier, a naive implementation of a stub cloud model can easily cause false warnings because it fails to reflect the actual behaviors of the real cloud environment. In contrast, our cloud model avoids false warnings by simulating the basic behavior of the cloud storage. The unit test shown in Figure 2 would pass using our simulated cloud since the method “queue.GetMessage()” returns null. This return value would be the same when the unit is executed against the real cloud environment.

After Pex finishes exploring the unit under test with different cloud states, the Test Transformer transforms each cloud state to be a sequence of real cloud API methods. Suppose that one generated cloud state includes two queues named “PhotoQueue” and “PhotoCleanupQueue”, and each queue contains one CloudMessage “Msg1”. Such cloud state would be translated into the following method sequence:

1. var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;

2. CloudQueueClient queueClient = storageAccount.CreatCloudQueueClient();

3. var queue1 = queueClient.GetQueueReference(Constants.PhotoQueue).CreatIfNotExist();

4. queue1.addMesssage(new CloudQueueMessage(“Msg1”));

5. var queue2 = queueClient.GetQueueReference(Constants.PhotoCleanupQueue).CreatIfNotExist();

6. queue2.addMesssage(new CloudQueueMessage(“Msg1”));

The basic algorithm of Test Transformer is to traverse every storage item. Once a new item is visited, the Test Transformer records a pre-defined sequence of standard Azure API methods that create this item in the real storage. A transformed unit test could first call such method sequence to achieve the required cloud state, and then execute the unit under test followed by some assertions.

Analysis of Not Covered Blocks of lokad.Cloud project.

EVALUATIONS

We conducted experiments on 19 open source projects and focus on testing the program units that interact with the cloud environment.

RQ1: Whether our simulated cloud environment and mock API methods behave the same as the real cloud environment and API methods?

RQ2: How high is the percentage of structural coverage achieved by applying our approach?

RQ3: Whether the transformed general unit tests could run in real cloud environment.

RQ2:

We evaluate our approach on one open-source project PhluffyFotos from codeplex because it frequently use Azure APIs. We focus on testing the units that interact with the cloud environment.

In total, our approach achieved 76.9% block coverage. Since the Azure Table Service is an extension of ADO.net data services, we also mocked some of the ADO.net data services API methods.

In addition, we also apply our approach on other two open source projects.

All Evaluation Subjects (source code is also available for testing)

RQ3: Each generated tests could be transformed into program inputs and a sequences of Azure API methods that construct a certain cloud state.

This is a list of generated tests for Dispatch method, Let's take test #9 as an example. In order to cover a certain path of the method under test, our approach automatically

generate a cloud state that includes two queues (Queue "photos" and Queue "purgephoto"), and each queue contains one message.

Our approach transform this state into a sequence of real cloud API method calls that construct the same cloud state.

The transformed method sequences is as follow: