Abstract.

Matt Brasier, Senior Consultant at Enterprise Java professional services firm C2B2, provides an in-depth case study of clustering iPoint portal with Terracotta.  This case study describes how a cluster of iPoint portal instances sharing a single database was built, using iPoint 2.4.2, Java 6.0, Apache Tomcat, Hibernate, Terracotta and MySQL to provide a scalable architecture for iPoint deployments and demonstrates Terracotta being used on an existing, complex, real-world application.

Introduction.

iPoint portal is an open source collaboration portal that supports JSR168 portlets and comes with a large number of portlets out of the box. iPoint deploys as a WAR file and stores its data in a database, using Hibernate as its persistence engine. By default it uses OSCache as a Hibernate cache.

Previous attempts to cluster iPoint portal using OSCache clustering, based on JGroups have not been completely successful, due to the complex nature of the object relationships in the iPoint data model.

Terracotta is an open source tool that allows objects in memory to be shared between Java virtual machines. It provides an integration with Hibernate based on its integration with EhCache. Terracotta advertises that it integrates with many applications with little or no code changes required. Terracotta also provides a method of clustering users’ HTTP session data between a set of Apache Tomcat instances.

The objective of this exercise was to build a clustered set of iPoint portal instances, running on Apache Tomcat and using Hibernate with EhCache and Terracotta to provide a clustered view of the database. The clustering of users’ HTTP sessions is not in scope for this exercise, so the Terracotta integration with Tomcat was not tested.

Method.

The first step was to create a cluster testing environment. For this exercise, we created two clones of an existing VMWare virtual machine that uses Ubuntu Linux, running iPoint 2.4 on Tomcat 6 and Java 6.

Install Terracotta

The first step was to download the Terracotta distribution from http://www.terracotta.org/ and install it into /usr/local. This was as simple as downloading the gzipped tar archive,  extracting it to the desired location, and creating a symlink for /usr/local/terracotta (to make scripts easier to use).

Configure iPoint to Use One Database

The next step was to get the two iPoint instances talking to the same database. After shutting down both tomcat instances, we modified the context.xml (/usr/local/tomcat/webapps/iPoint/META-INF/context.xml) on both servers to contain the IP address of machine1, rather than localhost. This took care of the Tomcat database connectivity configuration.

Allowing remote connections to MySQL is rather more complex. There is a good guide for doing so at http://www.cyberciti.biz/tips/how-do-i-enable-remote-access-to-mysql-database-server.html.  We copied the /usr/local/mysql/support-files/my-small.cnf to /usr/local/mysql/data/my.cnf, then edited my.cnf to add the bind-address. Finally we logged on to the MySQL instance and granted all permissions for the user from the remote IP address.

Modify the Hibernate Configuration

By default, iPoint is configured to use OSCache for its Hibernate configuration. As Terracotta only integrates with EhCache, it was necessary to switch Hibernates caching configuration to use EhCache rather than OSCache. To get EhCache working we needed to use a newer version of EhCache (as of this writing, it is recommend that you use version 1.3.  We tried 1.4 Beta but without any luck). EHCache can be downloaded from http://ehcache.sourceforge.net/.  We deleted the ehcache1.1.jar from /usr/local/tomcat/webapps/iPoint/WEB-INF/lib and copied in ehcache1.3.jar downloaded from SourceForge.

The Hibernate cache configuration file in iPoint is contained within ipoint.jar, which is located in the same WEB-INF/lib directory as the ehcache jar. To modify the configuration you need to extract the jar file, modify com/c2b2/ipoint/model/open.hibernate.caching.xml, add com/c2b2/ipoint/model/ehcache.xml configuration file and re-jar the archive (If you are following these steps, then before re-jarring the archive, be sure to review the section at the end about code changes). Alternatively you could modify the open.hibernate.caching.xml file before performing a build, as ant will build the ipoint.jar and deploy the web application.   See Listing 1 for the full hibernate caching configuration file.

open.hibernate.caching.xml

As well as modifying open.hibernate.caching.xml, it is necessary to add the ehcache.xml configuration file (referred to in the Hibernate config).  See Listing 2 for the full EHCache configuration file.

ehcache.xml

Unfortunately, Terracotta needs access to the Hibernate session factory, which it does by initialising Hibernate, and, in doing so, Terracotta expects the Hibernate configuration to be called hibernate.cfg.xml and to exist in the root of the web application. We were forced to create a hibernate.cfg.xml file in the root of the web application.

The hibernate.cfg.xml should contain the connection settings and dialect for Hibernate in addition to the settings from open.hibernate.caching.cfg.xml. This means that we had to have two copies of the Hibernate configuration, one open.hibernate.caching.xml and one hibernate.cfg.xml. Hopefully Terracotta will soon fix the product so that it is possible to pass it a pre-initialised Hibernate session factory, which will remove the need to add the hibernate.cfg.xml to the root of the web application.  See Listing 3 for the full hibernate configuration file.

hibernate.cfg.xml

Finally, we downloaded the jsr107cache-1.0.jar file from http://repo1.maven.org/maven2/net/sf/jsr107cache/jsr107cache/1.0/ and placed it in WEB-INF/lib. This jar is required by the EhCache and Terracotta integration.

Configure Terracotta

The first step is to create tc-config.xml, which is the Terracotta configuration file. This file tells Terracotta where the server should run, where log files should be stored and which classes should be instrumented for storage in the shared memory space. Initially we instrumented just the com.c2b2.ipoint.model..* classes, but it turns out that the class com.c2b2.ipoint.processing.RecentlyViewedManager also requires instrumentation. The resulting tc-config.xml file is presented below:

tc-config.xml

Start Terracotta Server

Starting the Terracotta server is done using a script in the bin directory of the Terracotta install. The necessary command line was:

/usr/local/terracotta/bin/start-tc-server.sh –f /usr/local/terracotta/tc-config.xml

Modify Tomcat Start Script

The Terracotta client works by instrumenting classes as they are loaded and by providing pre-instrumented versions of a number of the core Java classes. In order to get Tomcat to run as a Terracotta client, it is necessary to pass a number of additional arguments to the JVM used to start the Tomcat instance. This can be done easily using the JAVA_OPTS environment variable. See http://www.terracotta.org/confluence/display/docs1/Setting+Up+a+Tomcat+Web+Cluster for more information on setting up Tomcat.

The Tomcat servers should be configured to connect to the Terracotta server to download the Terracotta configuration, rather than having their own copy. The resulting fragment from the start script is:

tomcat start fragment

Modify iPoint Code

This step is required because iPoint makes use of a class from the java.util.concurrent package that cannot be used in Terracotta shared memory. The class com.c2b2.ipoint.processing.RecentlyViewedManager contains the reference, which can be replaced with synchronized access to a java.util.LinkedList instead. This change was made and the iPoint jar was re-built (remembering to keep the updated open.hibernate.caching.xml).

This change has been committed to the iPoint Subversion repository, so if you are building using the latest code from the repository, or are using a version of iPoint newer than 2.4.2, you will not need to make this change.

Outcome

Once all these changes were made, both Tomcat instances were started. As Tomcat starts, a message is printed indicating that the Terracotta client has initialised and connected to the server. The first time that Terracotta was run on each machine, it created a ‘boot jar’ which contains instrumented versions of core Java library classes that are loaded by the null loader and must, therefore, be instrumented at compile time. This was done automatically with no extra configuration required.  We were able to connect to the portal using a web browser, and navigate and make changes.  Any changes made on one instance were instantly reflected in the other instance.

Conclusion

Terracotta provides a very powerful and neat solution to the problem of handling clustered caches. The product integrated easily with EHCache and Hibernate with little configuration required, and there was plenty of excellent information on the web to help us  develop a good understanding of Terracotta, how it works and the kinds of problems it solves. Using Terracotta to enable the clustering of iPoint portal is the best approach we have seen so far and the one that we will be recommending to our customers.

It is also clear, from this brief investigation, that Terracotta has many applications outside the clustered caching that we used it for.