For this chapter, we are going to work with Nagios as our best choice, considering the performance and the simplicity of its setup and configuration. As we have already mentioned, Nagios is open source software that can be installed on multiple Linux distributions. In our case, we will be installing it on CentOS 7. It is a network, infrastructure, and server-monitoring tool. It will be monitoring switches, applications, and services. It has an alerting feature that helps inform users about all issues that occur while the infrastructure is being monitored. It also alerts the user if the issues have been fixed. Other than monitoring, Nagios also has the ability to identify system or network issues that could cause problems, with real-time problem notification. Furthermore, it has some security features, by virtue of which it can identify security breaches in the infrastructure.

In this section, we are going to install Nagios on a machine. It will act as our monitoring server. We need a test client to have it monitored. The client will have some common services; we will try to mess with them a little to test the Nagios notification service.

Let’s talk a bit about the things we need before we start our monitoring server installation.

First, we need to have the Linux Apache MySQL PHP ( LAMP) services installed on our machine. Since Nagios will be accessible via the web interface, having a web server installed is something obvious. For a more detailed and secure web server installation, you can go back and check out Chapter 3, Linux for Different Purposes.

Nagios won’t be installed from the CentOS 7 package manager. We have to download it and then compile it, so we need basic compiling tools and a downloading tool to download the Nagios source code archive. We will install these using Yum, the CentOS package manager:

$ sudo yum install gcc cpp glibc glibc-common glibc-devel glibc-headers gd gd-devel kernel-headers libgomp libmpc mpfr make net-snmp openssl-devel xinetd

We wait until the installation is done and then proceed to the next step of the preparation.

In order to run the Nagios process, we need to create a Nagios user and give it a password:

$ sudo useradd nagios
$ sudo passwd Really_Secure_Password

We need to make sure that we are using well-secured passwords while creating any.

Next, we create a new group called nagcmd to allow external commands to be submitted through the web interface once it’s up-and-running . Then, we need to add both Nagios and Apache to this group:

$ sudo groupadd nagcmd
$ sudo usermod -a -G nagcmd nagios
$ sudo usermod -a -G nagcmd apache

We move on to the final step, which is downloading the source archive for the latest version of Nagios. To do the downloading, we will be using Wget, a tool that we have already installed.

During this tutorial, we will be using Nagios 4:

$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.1.1.tar.gz

After downloading the latest Nagios stable version, we need to extract it. Well, since Nagios will be installed at the position where we are going to extract its source, we are going to put it in an appropriate location. We have a choice between /usr/local and /opt, so we need to copy the source package file there and then extract it. For this example, we will just go with /usr/local:

$ sudo cp nagios-4.1.1.tar.gz /usr/local/
$ cd /usr/local/
$ sudo tar xzvf nagios-4.1.1.tar.gz

After extracting the archive, there will be a new folder created, holding the named Nagios and including the corresponding version. We need to go inside the folder to start compiling it:

$ cd nagios-4.1.1/

Just before we start the compiling process, we need to run the configuration script that will help run the compiling process with no error by configuring it to use the available compiling tools that we have installed previously:

$ sudo ./configure --with-command-group=nagcmd

This configuration process has the option to set up the latest created group as the one that will be running the internal commands.

Now, we are actually able to start the compiling process:

$ sudo make all

This command can take a lot of time depending on the machine’s processing power.

After doing this, we proceed to the installation phase. We need to install Nagios, its initialization scripts, some sample configuration files, and the Nagios web interface:

$ sudo make install
$ sudo make install-commandmode
$ sudo make install-init
$ sudo make install-config
$ sudo make install-webconf

Before moving on the next step, we need to set up our Nagios administrator user and password to access the web interface:

$ sudo htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

Then, we type in the password twice to have our web interface administrator well created and configured.

After Nagios has been installed, we can add some useful plugins. First, we need to download the latest stable source version of those plugins. We need to go to the /usr/local folder and download the plugin’s source archive there. This step installs everything there well organized for future diagnostics:

$ cd /usr/local

Then, we start the download using
Wget:

$ sudo wget http://nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz

After completing the download, we can start extracting the archive using the same command:

$ sudo tar xzvf nagios-plugins-2.1.1.tar.gz

Then, we enter the directory we just created:

$ cd nagios-plugins-2.1.1/

Again, we need to compile the source files. Just before compiling, we need to run the configuration script with some useful options, as follows:

$ sudo ./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl

For the configuration option, we set the user and group Nagios as the default to access and use the plugins. Also, we use OpenSSL to secure the plugin usage.

Then, we start compiling the plugins:

$ sudo make

After that, we can start the installation:

$ sudo make install

Once this command is executed with no errors, we can say that our Nagios Plugins are well installed. We can move on to set up the Nagios Remote Plugin Executor ( NRPE). This is a Nagios agent that simplifies remote system monitoring using scripts that are hosted on remote systems. We need to download, configure, compile, and install it in the same way. We first need to find the latest stable version of the source package, and then we download it to /usr/local:

$ cd /usr/local/
$ sudo wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz

Next, we extract it at the same location, and go inside the folder to start the compilation:

$ sudo tar xzvf nrpe-2.15.tar.gz
$ cd  nrpe-2.15/

We start by running the NRPE configuration script. We define the user and the group using the Nagios process and the security tools:

$ sudo ./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu

Then, we run the compiling command, followed by the installation commands:

$ sudo make all
$ sudo make install
$ sudo make install-xinetd
$ sudo make install-plugin
$ sudo make install-daemon
$ sudo make install-daemon-config

Next, we configure the xinetd startup script:

$ sudo nano /etc/xinetd.d/nrpe

We need to look for the line that starts with only_from and then, add the IP address of the monitoring server. It can be a public or a private address depending on where we want to make the server accessible from:

only_from = 127.0.0.1 10.0.2.1

Then, we save the file to give only our Nagios server the capacity to communicate with NRPE. After that, we add the following line to define the port number for the NRPE service:

$ sudo echo "nrpe 5666/tcp # NRPE" >> /etc/services

To have this configuration active and running, we need to restart xinetd to launch NRPE:

$ sudo service xinetd restart

Now, we have our Nagios monitoring server officially installed. We can proceed with the configuration steps. We go to the Nagios main configuration file and activate the folder that will store all the configuration files:

$ sudo nano /usr/local/nagios/etc/nagios.cfg

Then, we uncomment the following line, save the file, and exit:

cfg_dir=/usr/local/nagios/etc/servers

We create the configuration folder that will store the configuration file for each machine that will be monitored:

$ sudo mkdir /usr/local/nagios/etc/servers

Then, we move on to configure the Nagios contacts file to set the e-mail address associated with the Nagios administrator. Usually, it is used to receive alerts:

$ sudo nano /usr/local/nagios/etc/objects/contacts.cfg

Now, we need to change the administrator e-mail address. To do so, we need to type in the right one after the email option:

email                           packtadmin@packt.co.uk        ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******

Then, we save the file and exit it.

Now, we proceed to the check_nrpe command configuration. We start by adding a new command to our Nagios server:

$ sudo nano /usr/local/nagios/etc/objects/commands.cfg

We add the following lines at the end:

define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

We save the file and exit to allow the new command to become usable.

Now, we go ahead and configure the access restriction to IP addresses that can access the Nagios web interface:

$ sudo nano /etc/httpd/conf.d/nagios.conf

We need to comment these two lines:

Order allow,deny
Allow from all

Next, we uncomment the following three lines:

#  Order deny,allow
#  Deny from all
#  Allow from 127.0.0.1

We can always add any network or address to allow it to have access to the monitoring server:

Allow from 127.0.0.1 10.0.2.0/24

We can always check whether there is any configuration error in the Nagios configuration file using the following command:

$ /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Just before starting Nagios, we need to make the Nagios CGI accessible by changing SELinux actions from enforcing mode to permissive:

$ sudo nano /etc/selinux/config

Then, we change this line to look like the following:

SELINUX=permissive

Now, we can restart the Nagios service and add it to the startup menu. We also need to restart the Apache service:

$ sudo systemctl start nagios.service
$ sudo systemctl enable nagios.service
$ sudo systemctl restart httpd.service

We can now access the Nagios server, but still we need to be allowed to try accessing it from the server itself, or from a machine that is connected to the network that is allowed to access the server. So, we go to the web browser and type http://Nagios_server_IP_Address/nagios. Then, we type the admin username, nagiosadmin, and its password, which has already been defined earlier, to get access to the Nagios interface.

Now, we move on to our client server—the one that we want to monitor using Nagios. First, we need to install the required packages. For CentOS 7, we need to have the EPEL repository installed in order to get the required packages:

$ sudo yum install epel-release

Now, we can install the Nagios plugins and NRPE:

$ sudo yum install nrpe nagios-plugins-all openssl

Let’s start by updating the NRPE configuration file:

$ sudo nano /etc/nagios/nrpe.cfg

We have to find the line that starts with allowed_hosts and add the IP address of our monitoring server:

allowed_hosts=127.0.0.1,10.0.2.1

Then, we save and exit the file. To complete the configuration, we need to start the NRPE service and add it to the startup menu:

$ sudo systemctl start nrpe.service
$ sudo systemctl enable nrpe.service

Once we are done configuring the host that we want to monitor, we go to the Nagios server to add it to the configuration folder.

On the Nagios server, we need to create a file with the name of the machine. We can take the machine hostname or put something that indicates the role of the machine or any other indication:

$ sudo nano /usr/local/nagios/etc/servers/packtserver1.cfg

Then, we add the following lines, replacing host_name with the client hostname replacing the alias value with a short description of the server’s main job, and finally replacing address with the server IP address:

define host {
        use                             linux-server
        host_name                       packtserver1
        alias                           Packt Apache server
        address                         10.0.2.12
        max_check_attempts              5
        check_period                    24x7
        notification_interval           30
        notification_period             24x7
}

With this configuration saved, Nagios will only monitor whether the host is up or down. To make it do more, we need to add some services to monitor, such as HTTP and SSH. Also, we are adding the option to check whether the server is active. We need to open the same file and define a service block for each service that we want to monitor:

$ sudo nano /usr/local/nagios/etc/servers/packtserver1.cfg

define service {
        use                             generic-service
        host_name                       packtserver1
        service_description             SSH
        check_command                   check_ssh
        command_line  $USER1$/check_ssh $ARG1$ $HOSTADDRESS$
        notifications_enabled           0
}

define service {
        use                             generic-service
        host_name                       packtserver1
        service_description             HTTP
        check_command                   check_http
    command_line  $USER1$/check_http -I $HOSTADDRESS$ $ARG1$
        notifications_enabled           0
}

define service {
        use                             generic-service
        host_name                       packtserver1
        service_description             PING
        check_command                   check_ping!100.0,20%!500.0,60%
}

Then, we save the file and reload the Nagios service:

$ sudo systemctl reload nagios.service

We will see the new server on the host list and its services on the services list. To test whether Nagios is doing its job, we disable the SSH service:

$ sudo systemctl stop sshd.service

Then, on the web interface, we can see how the service will go down from green to red. The red signal means that the test for that service has failed or has returned nothing, which means that the service is rather disabled or inaccessible. An error notification e-mail will be received by the Nagios administrator.

Source: https://thenullterminator.wordpress.com/

After that, we try the second test, to start the service:

$ sudo systemctl start sshd.service

To indicate that the service is back, another e-mail is received with the new status, where all its information will turn to green, as shown in the following screenshot:

Source: https://www.digitalocean.com/

Now, after setting up the first server, we can go ahead and add all the machines, including the switches, printers, and workstations that we need to monitor. Also, to be more practical, we should add only those services that we care about. So, if we have a server that runs a number of services and we will be using only two of them, it is pointless to add all of them and overload the server dashboard and the mailbox of the administrator with things that we don’t care about, which are later treated as spam.

Now, we will configure the NRPE daemon to receive information from the clients about their status. First, at the Nagios server, we edit the Xinetd NRPE configuration file to add which IP address the server should listen from:

$ sudo nano /etc/xinetd.d/nrpe

We need to add the IP address after the only_from option:

only_from       = 127.0.0.1 10.0.2.1

Then, we need to add the NRPE service to the system services:

$ sudo nano /etc/services

We add the following line at the end of the file:

nrpe 5666/tcp # NRPE

To have it submitted, we restart the Xinetd service:

$ sudo systemctl restart Xinetd

Then, we go to the client and make these modifications:

$ sudo /usr/lib/nagios/plugins/check_users -w 5 -c 10
$ sudo /usr/lib/nagios/plugins/ check_load -w 15,10,5 -c 30,25,20
$ sudo /usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda1

These three commands are used to activate the Nagios agent to send information about server load and disk storage. In our case, our disk is defined as sda1. We can check the naming of the disk using the lsblk command.