C2B2 logo icon

Auto scaling with Google App Engine Part 3

This is the third in a series of blogs on auto scaling with Google AP Engine. In Part 1 I described how to deploy a simple application and in Part 2 I did some basic load testing and monitored the scaling process. In the last post of this series I’m going to use a slightly more realistic application interface and testing process.

The application I’ll use for this testing returns a Person as XML (incidentally a Harry Potter character). 

Good REST style web services carefully use the URL to represent the resource, however, previously in Part 1 when using the WSInvoker class, I was passing in values for processType as query parameters. This is typical of an HTML form post to a web application and not terribly RESTful. So for my PersonService I will expose a GET request of the form /rest/persons/{id} where id is the id of the Person. The first path I used was as below.


public class PersonService {

  @Produces( { MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON })
  public Person getPerson(@PathParam("id") String id) {
    log.debug("PersonService.getPerson() APPLICATION_XML id: " + id);
    return getPersonFromDB(id);



I tested access from a browser to http://localhost:8888/rest/persons/1 and all seemed fine as below.



I then created a project in SOAPUI 4 as below.

This is based on the WADL as below, note that the resource base is http://localhost:8888/rest/ and the resource path is /persons/{id}.
Then when you look at the request that SOAPUI generates as below it ends up with // in the path which returns a HTTP 404 when you send a request in.


This tripped me up initially until I changed the path to:

  public class PersonService {

omitting the leading slash. I presume this interacts with the path specification in the web.xml for Jersey as below:

    <servlet-name>Jersey REST Service</servlet-name>



However, I was getting diverted to I didn’t investigate further. Okay we are back on track now and when I create the SOAPUI project from the application.wadl it generates a getPerson request when I can set 1 as the parameter and get the response below. 

  Similarly I can use the URL http://localhost:8888/rest/persons/1 and get the response.


As you can see the Person class is very simple and gets serialized to XML using in built support for JAXB. All that is required is the @XmlRootElement annotation to the class as below.

package uk.co.c2b2.scaledemo;
import javax.xml.bind.annotation.XmlRootElement;




public class Person {
       String firstName;
       String lastName;
       String id;
       public String getFirstName() {
              return firstName;
       public void setFirstName(String firstName) {
              this.firstName = firstName;
       public String getLastName() {
              return lastName;
       public void setLastName(String lastName) {
              this.lastName = lastName;
       public String getId() {
              return id;
       public void setId(String id) {
              this.id = id;
       public String toString() {
return "Person [firstName=" + firstName + ", id=" + id + ", lastName=" + lastName + "]";



I build a simple HashMap of persons as below for this testing. I plan to move this to Cassandra or a combination of MySQL with Infinispan for future testing. 


       private static void buildPersonDB(){
              log.debug("Building person data ...");
              Person p1 = new Person();
              persons.put("1", p1);
              Person p2 = new Person();
              persons.put("2", p2);
              Person p3 = new Person();
              persons.put("3", p3);                   
              log.debug("Built: " + persons);


Everything seems to be working as expected now so I’m going to deploy to the Google App Engine and then start using an Amazon EC2 desktop instead of working locally. First the deployment which is simply a click of the jet engine icon.



Then confirming the dialogue below.


Watching the progress monitor then hopefully getting the console output below.
Closing update: new version is ready to start serving.
Uploading index definitions.

Deployment completed successfully



Now we can give it a quick test at http://c2b2autoscale.appspot.com/rest/persons/1 from a browser and get Harry Potter back.








I move to the Centos desktop on the Amazon EC2 environment I’m using and import the WADL into SOAPUI, generate some requests, name them, create a test suite and an associated load test. I use the thread strategy to climb from 1 thread to a maximum of 15 and the end result looks like below.







This is really just a warm up run but looking at the results we can see that the maximum for getting ID 2 and 3, Hermione and Ron are around 4.2 seconds but for Harry at ID 1 its only 1.3. They all have similar averages at around 193ms. Looking in the App Engine dashboard we can see the numbe of instances has not grown to 7.









Then by looking at the logs we can see the warmup requests casing a new process to be started for the application. It also gives a warning that the request used a high amount of CPU.
























We started with one instance and ended up with seven and there are six requests logged in the period 11:17:30 to 11:19:45 which caused a new process to start. That all seems logical what I don’t understand is how we end up with seven instances and possibly only one active. Obviously the capacity needs to precede the demand but it seems a bit extreme. I read up on instances at http://code.google.com/appengine/docs/adminconsole/instances.html and the section on How Applications Scale which says: 














App Engine applications are powered by any number of dynamic instances at any given time, depending on the volume of requests received by your application. As requests for your application increase, so do the number of dynamic instances powering it.

Each dynamic instance has its own queue for incoming requests. App Engine monitors the number of requests waiting in each instance's queue. If App Engine detects that queues for an application are getting too long due to increased load, it automatically creates a new instance of the application to handle that load.

So it’s the number of requests in the instance queue which triggers the creation of new instances. I wonder if using a higher load will change the ratio of number of active instances to the number of instances created, let’s see. I increase the number of maximum threads to 25 and also increase the test run time to 5 minutes instead of 3 and get the results as below.



























The average response time across all 3730 requests is about 210ms with a maximum of 4785ms and a minimum of 128ms. Looking at the number of requests / second in the App Engine dashboard I’ve got up to about 50.












And the number of instances has grown to 14 over the period of the tests.





















Exporting the results to excel and plotting the load versus the response time I get the graph below. This shows that at the given load the response time does not increase, albeit with a couple of outliers. 

























So this has been a quick run through of using Google App Engine, a REST web service developed using Jersey and some load injection with SOAPUI from Amazon's cloud. I havent really delved in but I hope you've found it interesting please let me know.

In the next series I want to test Amazon's Elastic Beanstalk service in a similar manner then we'll have some fun comparing the two.

Pete Raymond

















If you enjoyed this post, why not subscribe to our feed?





Enter your email address: