DevOps Toolsets Part 2 – How to Install and configure Nagios to Monitor Weblogic using Ansible

C2B2 Support uses Nagios to monitor Weblogic and JBoss environments for several of its customers. With a remit to install and configure Nagios at a low cost with minimum infrastructure, Principal Consultant Alan Fryer describes how he solved this using Ansible.

Following on from my look at how Chef has evolved, I'll turn my attention to other toolsets used by our customers. Several of which use Nagios to monitor their current infrastructure - covering servers, networks, operating systems and file systems, but needed help in monitoring their WebLogic and JBoss servers for which we used the Nagios jmxeval plugin to collect JMX metrics. This required the plugin to be installed on many servers in a consistent and repeatable manner.  


Nagios jmxeval Plugin

The Nagios jmxeval plugin can be used to collect the metrics from any Java process that implements JMX MBeans, and so is ideal for monitoring middleware servers such as WebLogic, JBoss, Tomcat, GlassFish and Payara. The jmxeval plugin uses XML configuration files that define metrics to be collected, and the thresholds Nagios uses for raising warning and critical alerts. 

We created a set of template XML configuration files to collect metrics for WebLogic and JBoss, covering the typical things you would need to monitor on the Java middleware stack, such as datasources, JMS destinations, JVM Heap utilisation and garbage collection.  

The templates, together with the jmxeval plugin, need to be installed on all the hosts to be monitored, then services created for each jmxeval check and associated to the respective hosts using the Nagios web UI.  Fortunately, this process can be automated by importing a configuration file defining the host services to be added/updated into the Nagios Server and executing the bash script reconfigure_nagios.sh to apply the configuration - see Nagios Automated Host Management for more details. After much debate within the team, it was decided to use Ansible to install and configure the jmxeval plugin.   


Why Ansible?

As this is what was being rolled out on customer sites, we had less control on what could be installed on the hosts to be monitored, so the solution needed to be lightweight with little or no dependencies on software being installed to control the process. 

Out of all the DevOps Automation tools we considered, Ansible was chosen as it requires no software to be installed on the hosts being managed. All that is required is for Ansible to be installed on one server, the ‘Ansible Controller’, which is used to execute tasks or playbooks on the remote servers it manages using SSH. The playbooks are written in the data serialization language YAML and uses a very simple syntax in a human readable form - so is very easy to learn.


What is Ansible?

Ansible is an open source automation tool that can be used to configure systems, deploy applications and orchestrate more advanced tasks. Ansible manages remote hosts using an agentless architecture, using OpenSSH and WinRM to run the code on the remote hosts. Like other Automation tools, Ansible models your infrastructure as code using Playbooks and configuration data defined in the Inventory and Variables. The code is organised into Playbooks - which are Ansible’s configuration, deployment and orchestration language, describing the policy you want your remote systems to enforce. 

Playbooks are defined in a YAML format, using a minimum syntax. Each playbook is composed of one or more ‘plays’ in a list, mapping a group of hosts to well defined roles, represented by what Ansible defines as a task. A task is nothing more than a call to an Ansible module. Modules, also referred to as “task plugins” are the ones that do the actual work in Ansible, they are what gets executed in each playbook task. Modules should be idempotent, and will only make changes if they detect that the current state is different from the desired final state. When used in Ansible playbooks, modules can trigger ‘change events’ notifying ‘handlers’ to run additional tasks. 


Code walk through

Let’s run through the main code written for the Ansible playbook I created to install and configure the Nagios jmxeval plugin code. The playbook is organised with the following directory structure:

The playbook uses roles to group the vars_files, tasks, files, templates and handlers, allowing them to be shared with other users. Two roles have been created, one for the installing the Nagios jmxeval plugin and the other for configuring the hosts and services on the Nagios Server.  

The playbook’s root folder contains two files, the ‘inventory’ and ‘site.yml’ files. The ‘inventory’ file is used to define the servers to be managed and associates them to a Group. Variables can also be defined for a Host or a Group, the code snippet below creates two groups, ‘nagioshosts’ and ‘nagiosserver’. 

ansible-nagios-c2b2/inventory

[nagioshosts]
host1.c2b2.co.uk 
host2.c2b2.co.uk

[nagiosserver]
nagiosserver.c2b2.co.uk 

The site.yml file is the master playbook file which maps the top-level tasks ‘Install jmxeval plugin’ and ‘Configure Nagios’ and maps them to the respective host groups and roles.

ansible-nagios-c2b2/site.yml

# This playbook deploys the jmxeval plugin and configures Nagios.
  - name: Install jmxeval plugin
    hosts: nagioshosts

    roles:
      - jmxeval

  - name: Configure Nagios
    hosts: nagiosserver

    roles:
      - nagios

jmxeval Role

This role contains the following folders:

files – contains files that are copied from the task to the respective folders on the Nagios host being managed. The files include the jmxeval plugin tar file that needs to be unpacked on the host and the template XML configuration files created to collect the JMX metrics for the associated WebLogic and JBoss resources being monitored.

templates – contains the bash script used to execute the jmxeval plugin. This file is created from an Ansible template which is processed by the Jina2 templating language, substituting placeholders with the appropriate variables.

tasks – contains the main.yml file which defines the tasks executed on the Nagios hosts being managed.

The following code is contained in the main.yml file and installs the jmxeval plugin on the Nagios hosts defined in the inventory and performs the following tasks:

Install Java 1.8 and a number dependent packages using the yum module. The dependencies are defined using the with_items: option so the call to yum will iterate over the list of packages defined

- name: Install Java 1.8 and some basic dependencies
  yum: name={{item}} state=present
  with_items:
   - unzip
   - java-1.8.0-openjdk
   - libselinux-python
   - libsemanage-python

Copy the plugin tar file from the playbook to the Nagios host and extract the file to the libexec directory of the Nagios Remote Plugin Executor installation.

- name: Copy tar file to destination
  copy: 
    src={{tar_file}} 
    dest={{nagios_libexec_dir}} 
    mode=0755

- name: Extract tar file
  unarchive: 
    dest={{nagios_libexec_dir}} 
    src={{tar_path}} 
    creates={{tmp_unpack_dir}} 
    copy=no

Create the directory for containing the XML configuration files in the jmxeval directory created when the tar file is unpacked. This uses file module and reads the directory path from the variable jmxeval_conf_dir which is defined in the group_vars of the playbook

- name: Create jmxeval conf directory
  file: 
    path={{jmxeval_conf_dir}} 
    mode=0775 
    recurse=yes 
    state=directory

Copy the predefined XML configuration files used by jmxeval to the folder create above. This uses the copy module, this again reads the respective directory path from the variable defined in group_vars

- name: Copying jmxeval configuration files
  copy: 
    src={{item}}
    dest={{jmxeval_conf_dir}}
  with_fileglob:
     - files/conf/*

Create the bash script used to execute the jmxeval plugin in the libexec directory of the Nagios Remote Plugin Executor installation. This is done with the template module, and references a template defined in the templates folder. The Jina2 template processing language substitutes placeholders with variables defined in the playbook’s group_vars

- name: Add check_jmxeval script
  template:
    src=jboss/check_jmxeval.j2
    dest={{nagios_libexec_dir}}/check_jmxeval
    mode=750


Nagios Role

This role implements a task, handler and template to configure the services on the Nagios Server to monitor the Weblogic/JBoss resources for each host. This is done by creating hostservice.cfg configuration file for each host which is imported into the Nagios Server and executing the Nagios reconfigure script to apply the configuration.

The main.yml script defines the tasks to configure the services on the Nagios Server and performs the following tasks:

Create the Nagios command object to run the jmxeval plugin with its associated parameters. Append a ‘define command {}’  block to the end of commands.cfg file. First check if the check_jmxeval command has already been added, use the shell module to cat the contents of the file and register the output to the variable user_accts. 

- name: Check if file has check_jmxeval
  shell: cat {{nagios_base_dir}}/etc/commands.cfg
  ignore_errors: yes
  register: user_accts

The blockinfile module is used to insert a text block defining the command into the configuration file if it does not exist (uses the when condition to check if string ‘check_jmxeval’ exists in the results of the cat on the configuration file). 

- name: insert check_jmxeval command to commands.cfg
  notify: reconfigure nagios
  become: yes
  become_user: "{{nagios_user}}"
  blockinfile:
    dest: "{{nagios_base_dir}}/etc/commands.cfg"
    regexp: "^# END OF FILE"
    insertafter: "^}"
    owner: "{{nagios_user}}"
    group: "{{nagios_group}}"
    block: |
      define command {
       command_name check_jmxeval
       command_line $USER1$/check_jmxeval $USER1$/jmxeval/conf/$ARG1$ 
                    --set host=localhost --set port=$ARG2$ $ARG3$
      }
  when: user_accts.stdout.find('check_jmxeval') == -1

Add a hostservice configuration file for each host being configured. The file is created from a template hostservices.js. This defines a static list of standard server metrics to be monitored together with custom jmxeval ones defined in the host_vars file for the host. 

- name: Add host 
  notify: reconfigure nagios
  become: yes
  become_user: "{{nagios_user}}"
  template:
    src=hostservices.j2
    dest={{nagios_base_dir}}/etc/import/{{item}}.cfg
    owner={{nagios_user}}
    group={{nagios_group}}
  with_inventory_hostnames: nagioshosts

The template uses the following block to create service definition block looping over the jmx_services hash variable defined in the host variables file created for that host. 

{% for service in hostvars[item].jmx_services %}
define service {
  host_name			{{item}}
  service_description		{{service.description}}
  display_name			{{service.display_name}}
  use				local-service
  check_command			{{service.command}}
  register			1
  }
{% endfor %}

The host variables files are contained in the host_vars folder and use the hostname as the files name, the variables

jmx_user: admin
jmx_pwd: Password01%
jmx_creds: --set username={{jmx_user}} --set password={{jmx_pwd}} 
  
jmx_services:
  - {description: 'JMS Queues', display_name: 'jms-queues', command: 'check_jmxeval$ARGS'}
  - {description: 'Datasorces', display_name: 'datasources', command: 'check_jmxeval$ARGS}

The two above make changes to configuration files so use the notify option which executes the handler ‘reconfigure nagios’.  The handler uses the shell module to execute the reconfigure_nagios.sh bash script to apply the configuration changes on the Nagios server. 

- name: reconfigure nagios
  become: true
  become_user: root
  shell: /usr/local/nagiosxi/scripts/reconfigure_nagios.sh
  args:
    chdir: /usr/local/nagiosxi/scripts/


Conclusion

Ansible, compared to other automation tools, adopts a very simple approach using an agentless architecture to manage remote hosts over SSH for Unix based systems and WinRM for Windows.  It is very easy to pick up, and within a few hours you should be happily creating playbooks. The configuration data can be defined on the file system in Inventory, host_vars, group_vars and playbook files which makes it easy to package and execute playbooks on remote hosts without any dependent software being installed on the hosts being managed. A perfect fit for rolling out Nagios monitoring configuration to our customers.