C2B2 logo icon

Monitoring GlassFish over ssh with tmux

Steve examines a monitoring solution for GlassFish production using terminal multiplexer, tmux

Often as a consultant in C2B2 you get asked to work out why a production GlassFish application is running slowly. You ask the customer what monitoring they have in place and the usual answer is none. So what do you do? Well this post will give you an idea of how to knock up some rapid usable monitoring of a GlassFish server when all you've got is SSH. If you set this up in front of somebody then they will think you are a total Linux ninja!

To set up our console we are going to use a couple of linux tools which may need to be installed on the host. If they aren't there then you will have to embarrass the local Linux sysadmin by asking how he survives without them. The main one you need is tmux and the other is watch. tmux allows you to split a terminal screen into multiple terminals and watch allows you to run a command periodically.

Platform Monitoring

So login to your GlassFish host over ssh, fire up tmux and split the window into 4 "panes"
$ tmux
$ <Ctrl>-b %   # splits screen horizontally
$ <Ctrl>-b "   # splits screen vertically
$ <Ctrl>-b o   # navigates between panes choose the left one
$ <Ctrl>-b "   # Split the left hand pane vertically
Now use <Ctrl>-b o to navigate so the top left pane is active, and we'll set up some system monitoring.
In the top left pane run htop (this is an enhanced top but you can just run top if htop is not available)

In the next down run iftop (this provides network monitoring)

In the top right run dstat (does disk and network io monitoring)
In the final screen run tail -f /var/log/syslog (to trace the syslog)

GlassFish Monitoring

We'll now create another window and set up some GlassFish specific monitoring.
$ <Ctrl>-b c   # gets a new window
Now split the window into 4 panes as before.
In the top left pane we are going to configure JVM Garbage Collection Monitoring:
$ jps    # find the Java processes running
$ jstat -gcutil -h 10 <pid> 1000 # where pid is the GlassFish pid
In the top right pane we will tail the log - for example:
$ tail -f domain1/logs/server.log

In the bottom right we will setup periodic thread dumps using watch:

$ watch -n 10 jstack <pid> # where pid is the Glassfish pid

Then in the final pane we will run some asadmin monitoring
$ asadmin monitor --interval=5 --type=webmodule server
This sets up monitoring of the servlet container and lists number of active sessions, servlets etc. There are many other monitors in glassfish and using watch you can run many other asadmin commands periodically.


Navigating Between Windows

Now you have two windows, you can navigate between them using <Ctrl>-b n (for next) and <Ctrl>-b p for previous. You can also name windows in the status bar below using <Ctrl>-b , and then the name - obviously you can create more windows for more monitors.

Disconnect and Reconnect

The other cool thing about tmux is that once you have this running and configured you can just disconnect from the server and reconnect at will. 
To disconnect;
$ <Ctrl>-b d   # disconnects from tmux
This puts tmux into the background and now you can log out the server and when you log back in to reattach to the tmux session use;
 $ tmux attach-session
Combining tmux with watch to run commands periodically and using other linux and GlassFish command line monitoring tools it is very easy to set up a fairly sophisticated real time view of what is going on in a GlassFish server even when all you have is a raw headless ssh connection.
As an aside, if it's available on your system byobu is a nice front end to tmux.