In this article on how to configure WebLogic for production environments, I look at the reasons why running it in its out-of-the-box configuration isn't a suitable way forward, and explore some of the things you can do to tune and configure WebLogic server to improve the security, manageability, performance and reliability of your applications.
What's wrong with the out-of-the-box WebLogic configuration?
There isn’t anything wrong as such with the configuration of WebLogic from a default installation - it's just that it is set up for use in a development environment where security is fairly lax, and the workload of the server is fairly small.
The reasons for having these defaults are fairly sensible, as most projects will go into development environments first - and if WebLogic doesn’t perform well in development (because it's configured for a large production environment) it is unlikely that a project would continue using WebLogic server.
There is also not really a sensible set of prerequisite “defaults” for a production environment, it depends very much on what your application does, and how it is written, so any default “production” settings in WebLogic would still require tuning to get the most out of them.
So, we're faced with a number of areas where we need to consider what our application needs, and how WebLogic is best configured, and the two I'm going to look at first are Security and Manageability.
Security is usually the first thing that comes to minds when consider configuring a system for production use - and some of the more obvious examples of things to configure fall into this category.
For example - by default, WebLogic ships with a set of “demo” keystores, and if you use SSL then these will therefore be the keystores used as part of your security measures. In a production environment, this represents a significant risk, because the passwords and trust chain of these keystores are well known - and since SSL is used to control communication between the node manager and servers, it would be possible for someone to spoof node manager commands to your servers, thereby shutting them down or obtaining their log files.
A more serious issue is ensuring that you have two administrator accounts with strong passwords. The administrator user has access to all features in the WebLogic domain, including the ability to deploy code - so if this user is compromised, then there is very little that an attacker cannot accomplish.
With deployment rights, it would be a trivial matter to obtain data from your database, delete the contents of your database, or even get shell access to the machine that WebLogic is running on - so you can see why having a strong password is important.
And I mention having two accounts, because as we've probably all experienced at one time or another, managing strong passwords can be a challenge for some people! It's quite possible that someone will write it down wrong after changing it, or change it and then go on holiday, leaving you locked out of the ability to administer your servers.
Areas key areas where you would want to consider security configuration for your production system would include:
- Administrator accounts and passwords
- SSL and PKI certificates
- JMS security
- Remote JDBC
- Server to server communication
- Node manager to server communication
Manageability refers to ways that we can make the environment easier to manage - and the biggest of these is ensuring that the servers are under node manager control. The WebLogic node manager is a daemon process that runs in the background, and is responsible for starting and stopping WebLogic server instances.
It has the advantage that it provides a number of ways to start and stop the servers, such as via the WebLogic web console, or using a WLST script. However there two additional features of node manager control that make it really worth using.
The first is its ability to detect when a server it started has died (without it receiving a stop command), and restart it, and the second is its ability to detect when a server has become unhealthy and then proceed to restart it. With these two options it is possible for the node manager to keep your WebLogic infrastructure available even when there are known issues such as memory leaks or JVM bugs affecting it.
The key areas to consider when ensuring your WebLogic domain is manageable include:
- Use of node manager to manage servers
- Use of scripts to make configuration changes
- Log level tuning
- Log rotation
- Use of the domain log broadcaster
- Use of a domain admin port
- Server health monitoring tuning
So now, let's move on and look at the areas of WebLogic performance and peliability...
Performance is a key concern for any production deployment, but performance tuning a WebLogic server domain is very specific to the application running on it. This being said, there are a number of key areas where tuning occurs, and a number of common mistakes that cause performance problems - so it is often possible to make some fairly generic recommendations.
The first thing to understand when considering the performance of an application running in WebLogic server is that it sits at the peak of a pyramid of components, Performance issues can arise at any of these levels, and so tuning needs to incorporate all of them.
In this article, we are mostly focussing on configuration from the JVM layer up.
One thing to be aware of with performance tuning, is that you can only tune parts of an application that create bottlenecks. If your application is not IO bound and you make a change that improves IO, you aren’t going to see a performance improvement with your application. It is therefore important that any programme of performance tuning is accompanied by performance tests and monitoring to identify where the improvements are needed.
With all of this in mind, there are a number of places where WebLogic performance problems commonly occur, and we can discuss how some of those can be mitigated.
JVM memory settings
The first of these is having the wrong JVM memory settings. Java memory management and garbage collection tuning is a complex topic, so I won’t go into it in detail here, but I will say that it is usual for a WebLogic application to have around 2GB to 8GB of heap allocated, with 25-50% of that in the new generation. Heaps larger than 8GB will lead to long pauses when full garbage collection occurs, and in general, smaller heap sizes lead to faster applications - so you should focus on having a heap large enough to handle your workload, but small enough not to cause long garbage collections.
Another area to consider within performance is that of threading. As with memory, threads represent a resource that can run out - yet too many threads also cause performance problems. WebLogic’s thread pools are self-tuning, so you don’t need to set things like number of threads, but what you can do is create work managers. A work manger is a ring-fenced set of threads (a percentage of the total number) that only process tasks associated with that work manager. This allows you to ensure that different parts of your application (such as the user interface and batch processing) are working with separate sets of threads, and that one alone cannot starve the others of thread resources.
Areas where performance tuning can be applied to a WebLogic domain include:
- Memory pool sizes and total size
- Garbage collector
- Threads and work managers
- JDBC connection pool sizes and configuration
Reliability of a system is about ensuring that it is available when needed, and in the terms of our tuning, means that the system shouldn’t crash, or require regular restarts. A lot of the tuning in this area is therefore about making sure that things fail gracefully, rather than bringing parts of the system down (or even the whole system).
A good place to start with this is transaction timeouts. When changing transaction timeouts, it is important to ensure that timeouts happen at the finer-grained levels first, thus allowing consistency to be maintained. For example, in the situation of an EJB which accesses a database and a JMS queue, the timeout in the database should be the lowest number of seconds, followed by the JMS connection factory, then the EJB timeout, and finally the global XA timeout.
This allows higher levels to receive the timeout from the fine-grained resources, and process them correctly. If the global JTA timeout occurred first, WebLogic would kill the transaction without the database or JMS queue having a chance to correctly clean up their resources.
Another area where reliability can be tuned is in JDBC connection pool testing. This feature allows WebLogic to test a database connection (by executing an SQL statement against it) before giving it to an application component to use, or on a periodic basis. This ensures that invalid connections (caused by them being timed out by a firewall, or closed by the database, etc.) are not given to applications. This prevents the applications producing errors when they try and use an invalid connection.
There is also a very specific and useful use-case for periodic connection checking. It is not uncommon for there to be a firewall between the application servers and the database servers, and during long quiet periods, such as overnight, the firewall will close any inactive connections (such as all of the connections in the JDBC pool). When a user first logs in in the morning, they will find that the application takes a long time to do so. This is because the connection has been closed - but the application doesn’t know this until it tries to use it, and gets a TCP timeout.
By setting periodic connection checking, the connection remains “in use” overnight and the firewall doesn’t close it, thus preventing this issue.
There are a number of areas that should be considered for configuration to make sure the application is reliable:
- Transaction timeouts
- JDBC connection checking
- JDBC connection reclaiming
- Log rotation
- JMS delivery settings
- JMS dead letter handling
Hopefully you can see from this short article that we are only touching the surface of the number of tasks that need to be performed to get your WebLogic Server infrastructure ready for production use. The good news is that most of these tasks can be automated and scripted, and there are a number of ways that you can build a WebLogic domain to a known configuration, so that you don’t manually have to make changes to your environment - however that is a topic for another post!
It isn’t possible to tell you every change you need to make to your domain, as the requirements of each application and domain are very individual, but hopefully this article has given you some ideas, and convinced you not to just accept the out-of-the-box defaults.