Changing the from e-mail address in Nagios

This is going to be another short one.  I recently had a need to change the FROM address for e-mail from our Nagios installation. E-mail was coming from nagios@host.domain.com, which is non-route-able  from outside our network.

The change is very simple. Change the two command lines in your Nagios commands.cfg dealing with notify by e-mail. The command names are "notify-host-by-email" and "notify-service-by-email".

By default these lines read:
 /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

To change  the from address you append " -- -f nagios@domain.com", that is without the quotes of course.  So the new lines look like:

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$ -- -f nagios@domain.com

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$  -- -f nagios@domain.com

To explain it a little what we appended was a space followed by two dashes, which forces mail to pass those along to sendmail. Send mails sees the addition of  a dash followed by the letter f and then another space and the e-mail address you want to send from.

On my system this is an alias for myself so that all replies come to my inbox.

All of this works on CentOS which is what I'm currently running.  It should work on other distributions as well, but I haven't the time to verify that.

NRPE: Unable to read output and sudo

Thank you to Andrea Leofreddi over at cyberz.org for the blog post Nagios nrpe and sudo: “NRPE: Unable to read output”. This was a tremendous help back when I first started working with my md-raid device and Nagios. I found this entry again while working on my very own plug-in for Nagios, check_supervisorctl.sh.

In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty

For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh

Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}

Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:

[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!

Strange Nagios Error Solved

This morning I added some new services to a server, but they wouldn't move out of "Pending" status. The error I received was
"Feb 9 09:27:36 nagios: Warning: Check result queue contained results for service '' on host '', but the service could not be found! Perhaps you forgot to define the service in your config files?"

I stopped the nagios service and ran ps -ef | grep nagios. To my surprise there was still a Nagios instance running. This means two things. First the init script that comes with Nagios is borked and doesn't correctly check for running nagios instances. Second, I somehow started a second Nagios instance.

I thought something was up in the first place because every other refresh or so of the Nagios web view I would either see the three pending services or I wouldn't. This was my first clue that something was borked. I then went to tail /var/log/messages and saw the error message. Then I started investigating the issue with the help of Google. Once I saw that there where two instances of Nagios things started to make sense.

I killed the second Nagios instance and any children processes (in my case ndo2db) and then restarted nagios via the init script.

Once I had everything up and running (but only one instance) I was able to successfully check my new services.

Documentation, not always the How To Dos

Documentation for me is often just How to do something, but I have been forgetting the WHY part of the equation. For instance, my documentation says to do our weekly maintenance window after 21:30 and to only reboot one particular server after 21:45, but I didn't say why and had forgotten myself. So I started doing the maintenance earlier and at one point rebooted the one particular server at 21:30. This caused the last of our production cron jobs to not run, and thus a customer didn't get their batch for the day.

Now this could have been avoided a few different ways. Since that first time was a mistake any thing that happens after it is a failure on my part, and thus can't happen again.

1. I could have followed our procedure to the T.
2. Read my e-mail to see that the cron job hadn't run yet.
3. Run the cron job by hand after the server restart.
4. Did all of the precursor work and waited for the cron job to run.
5. Did all of the precursor work and run the cron job by hand.

I have chosen to go with option 1, follow our procedure to the T. We started doing the weekly maintenance for a few reasons and it was to be done after 9:30, with that one particular server last to ensure this didn't happen. Now once again I have my WHY, and I have it written down so if I question it again I know why.


This is just one instance where the WHY is critical, but there are others. So please when you are writing documentation, and you should be please include the WHY you do something the way you do it. It also helps train the new guy, or your replacement.

Enabling Ping responses in Windows 7

Out of the box if you Ping a Windows 7 host you will recieve "Request Timed Out". This is because the Windows 7 firewall is blocking ICMP echo requests. If this is causing you problems and you need to open up access to ping requests it is fairly simple.

1. Go to the "Windows Firewall" in the Control Panel.
2. On the left hand side click "Advanced Settings"
3. Click on "Inbound Rules"
4. Right click on "Inbound Rules" and choose "New Rule"
5. Select "Custom (Custom Rule)" and press "Next"
6. Select the (default) "All Programs" and press "Next"
7. Change the Protocol type from "Any" to "ICMPv4"
8. Unless you want to restrict the ping response choose the option "Any IP address". (This is for your adapter.)
9. Unless you want to restrict which hosts can ping you choose "Any IP address".
10. On the Action screen ensure it is an "Allowed" connection.
11. Leave all three check boxes check on the Profile screen.
12. On the final screen, Name, give it a meaningful name such as "Echo Ping Request"
13. Finally click finish.
    

CHECK_NRPE: Error - Could not complete SSL handshake

Recently I have been auditing our servers versus what we check in nagios versus what we need to do when a system is rebooted during routine maintenance. I found that two of our servers have been left out of nagios monitoring even though they are on my maintenance checklist, and they have software that I still start by hand (I KNOW IT'S NOT A BEST PRACTICE!).

Both servers had NRPE installed and configured (mostly), one of them even had a configuration file on the nagios server but it wasn't enabled (hostname.disabled instead of hostname.cfg). The other server needed a configuration file, but even then it wasn't working, so here are my troubleshooting steps:

1) Check to see that NRPE is compiled and installed. [It was]
2) Check that NRPE was listening (netstat -an | grep 5666) [It was]
3) Check that NRPE was listed in /etc/services [It wasn't]
4) Check the NRPE config file (/etc/xinetd.d/nrpe) for "only_from = 127.0.0.1 192.168.100.31" [It wasn't]

So I added the service definition to /etc/services and the nagios server IP to the only from line and restarted xinetd (service xinetd restart) and I was finally able to connect from my nagios server.

All of this is on CentOS 5.5 for both the server and the client.

NDO2DB daemon startup script

I would like to thank Chris over at http://sysengineers.wordpress.com for the excellent post on how to daemonize NDO2DB. His post NDO2DB startup script for RH (EL) / OEL does an excellent job getting everything working. I had to make one small change to the script as I kept my ndo2db executable as ndo2db-3x instead of just ndo2db.

For me this fixes a problem where I have to remember to manually start the ndo2db service after a server restart, which hasn't been happening the past few times. The second way I am going to fix this problem is by creating a checklist of things that need to happen during a server restart / boot up sequence for each PC. This will also reduce the effort needed for when I move the rack later this year. The third way I will be checking this is by adding a check to Nagios to make sure this is running at all times.

This also illustrates my general way of checking / double checking things. One go off of a checklist. The double check is to verify that Nagios is seeing the same thing.

Mounting a LVM volume in Ubuntu (Live CD)

A while back my testing server crashed. This was no surprise to anyone as it was just a (Very) old workstation. However it was running my nagios install in a production setting. I had been meaning to move it to a proper server, but just hadn't gotten around to it. To make matters worse, I didn't back any of it up. So thankfully it was only the motherboard that failed and not the HDD.

I mounted the HDD in another PC I had sitting around and booted it using Ubuntu Live CD.

First, boot Ubuntu.

Second, install the needed tools:

$ sudo apt-get install lvm2

Third, load the modules to do our task:

$ sudo modprobe dm-mod

Fourth, scan the system for LVM volumes. Look for the volumes you want to mount. Typically this will be VolGroup00:

$ sudo vgscan

Fifth, we need to activate the volume(s):

$ sudo vgtchange -ay VolGroup00

Sixth, Look for the logical volume containing the root file system. Typically this will be LogVol00:

$ sudo lvs

Seventh, create the directory to mount the drive:

$ sudo mkdir /mnt/restore

Eighth, Mount the volume to the directory you just created.

$ sudo mount /dev/VolGroup00/LogVol00 /mnt/restore -o ro,user

Ninth, Copy your files off of the drive.

$ cp /mnt/restore/some/dir/and/path /some/dir/and/path

Tenth, Setup what ever backup means you have on the new server!

All in all this wasn't a terrible thing, it could have been much worse. I have since moved the nagios setup to a virtual machine and am backing it up nightly.

Nagios: check_http, using the --invert-regex option

Some times you want to check that something is running or working correctly and you work out tests for that. Other times you want to know when something is broken and throwing error messages. This is about the latter, a proper HTTP 200 code is great and all, but what if the page is just showing "Too Many Connections" instead of your home page? My old check_http command for this server used to look like, well, check_http. I didn't check anything about it specifically, just that it was returning a 200 code.

Today however I knew I needed something more in depth. Our database server lost its local network connection, but still was available over the public IP, which is what I test against. Once we re-directed the SQL requests to the public IP address of the server everything started working again, until we ran across "Too Many Connections". The database server kept all of the "local" connections open and thus we ate up the rest.

So, how to test for this scenario? After reading through the man pages of check_http I saw this little gem "--invert-regex Return CRITICAL if found, OK if not". This I knew was exactly what I was looking for! If it sees our error codes it will go Critical! Now to put this gem into practice. Here is where the man pages fall short. There is no explanation on HOW to use this, just that it exists. I tried the obvious to me "check_http -H hostname.com -w 3 -c 5 --invert-regex 'Some string'", but that didn't work. OK, lets try "check_http -H hostname.com -w 3 -c 5 --invert-regex='Some string'" nope that errored out with " option `--invert-regex' doesn't allow an argument".

Third times the charm right?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string' --invert-regex '"

# HTTP OK HTTP/1.1 200 OK - 0.355 second response time |time=0.354966s;3.000000;5.000000;0.000000 size=12975B;;;0

Yes, as it turns out third time is the charm. So that got me thinking some more. How can I ensure that the page is rendering correctly, and if it isn't fail but in a specific way?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string I want in my page' -r 'Some string I don't want to see' --invert-regex '"

You can add more than one -r to the check_http command and it will require all of them to be present for the test to pass, and if one of them fails then it will go critical! Perfect!

If you have any more insight into using the check_http command in Nagios I want to hear about it. We are always running into new failure scenarios that we didn't anticipate and I want to know about them before one of my users tells me about it.

Spiceworks releases 3.5!

Just found out that Spiceworks has been updated to 3.5. I never got the chance I wanted to beta test 3.5, but now I don't have to worry about it. Looking forward to the Network Bandwidth Analyzer and the Nagios integration.

I plan on doing a backup of my database tonight so I can do the update after hours. Wish me luck.

nagstamon: A Nagios system tray monitor

First a little bit about my setup. While its true that I love nagios I don't always want to wait for e-mails to come before I know about a problem. I run a dual PC setup with Synergy2 to connect the two monitors and use only one keyboard and mouse. I keep the second monitor on my helpdesk tickets and nagios service detail page. I also use the Nagios Checker for Firefox on my second screen, but since I run Chrome as my main browser I cannot use that option on my main screen.

Today I found nagstamon and instantly fell in love with it! Configuration is easy. Below you can see what settings I am using for optimal performance in my opinion. Of course nagios-server is my actual FQDN for my nagios server and I don't use nagiosadmin to login to my nagios server.

If you notice on the last tab "Executables" I keep putty in the Windows directory so I don't have to update my path. This allows me to run putty right from the command line anywhere I am once I am at a command prompt.

Also if you don't choose to put NagStaMon in the system tray it will float in it's own tiny window like you see below. Look near the upper left corner by the Firefox Icon.

Using nagios to check PHP script execution time

We want to record the execution time of some of our PHP scripts.  I want to use nagios to do this but it is proving rather difficult.

Initially I was hoping I could provide some credentials to nagios for it to use and log into our web site, run a set of scripts, and then report back to us how long each script took.  Since our site does not use basic authentication I cannot do it the intial way I wanted.  

To work around the issues we are having and hopefully NOT have to write our own nagios plug-in here is what I am proposing.

Create a test directory that uses basic authentication so it is protected from the general public and has a set of scripts that we run with the execution time posted near the bottom.  We then use the standard check_http plug-in from nagios to search for that string and parse it as normal. 

Hopefully this approach to things won't take too long to develop and prototype.  Also I hope to report back with a follow up on what I actually did (with code examples) to get this working.

Nagios check_mysql_connections example

I had a little bit of trouble getting the check_mysql_connections working but once I figured it out on the command line I had to get it working in nagios.  Still being a nagios noob I had a little trouble but also got it working in nagios.

The line below will work from the command line.

[user@host libexec]# ./check_mysql_connections -w 75 -c 90 -H 555.555.555.555 -u YOUR_USER -p YOUR_PASS

In the host file I setup the service as 

define service{

        host_name                       db1.mrswi.com

        service_description             MySQL Connections

        check_command                   check_mysql_connections! -w 75 -c 90 -H 555.555.555.555 -u YOUR_USER -p YOUR_PASS

        check_period                    24x7

        max_check_attempts              2

        notification_interval           30

        notification_options            c,w

        contact_groups                  mrsitis

        }