Changing the from e-mail address in Nagios

This is going to be another short one.  I recently had a need to change the FROM address for e-mail from our Nagios installation. E-mail was coming from nagios@host.domain.com, which is non-route-able  from outside our network.

The change is very simple. Change the two command lines in your Nagios commands.cfg dealing with notify by e-mail. The command names are "notify-host-by-email" and "notify-service-by-email".

By default these lines read:
 /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

To change  the from address you append " -- -f nagios@domain.com", that is without the quotes of course.  So the new lines look like:

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$ -- -f nagios@domain.com

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$  -- -f nagios@domain.com

To explain it a little what we appended was a space followed by two dashes, which forces mail to pass those along to sendmail. Send mails sees the addition of  a dash followed by the letter f and then another space and the e-mail address you want to send from.

On my system this is an alias for myself so that all replies come to my inbox.

All of this works on CentOS which is what I'm currently running.  It should work on other distributions as well, but I haven't the time to verify that.

NRPE: Unable to read output and sudo

Thank you to Andrea Leofreddi over at cyberz.org for the blog post Nagios nrpe and sudo: “NRPE: Unable to read output”. This was a tremendous help back when I first started working with my md-raid device and Nagios. I found this entry again while working on my very own plug-in for Nagios, check_supervisorctl.sh.

In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty

For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh

Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}

Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:

[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!

ImageMagick convert -geometry weirdness

I don't know if this is really documented anywhere, but when using the ImageMagick program convert with the -geometry or -resize flags it also converts to 16bit color, at least when used against 1bit (Black and White) image.

The reason this is important to note is that using -compress Group4 will not work on these images because they aren't 1bit color images and they need to be. To work around this limitation you need to force the color depth with -monochrome. The downside to all of this is time.

Ripping an Image file from a PDF takes roughly .206 seconds for an 8.5x11 PDF with pdfimages (useful if a PDF has OCR embedded in it.)
Converting the resultant PBM file without -geometry 1700x2200! -monochrome takes roughly .194 seconds.
Converting the resultant PBM file with -geometry 1700x2200! -monochrome takes roughly 4.594 seconds. This is an increase of 10 - 10.5 % adding in the geometry.

*all of the above numbers are from one file, but testing of different files showed similar results. All tests where done at 200 DPI.

Here are some examples of the commands we are running for anyone curious.
convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 -geometry 1700x2200! -monochrome ${outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 -geometry 2550x3300! -monochrome ${outputTiffRoot}_300.tiff

Installing Backup Exec System Restore on CentOS 5.5 and First backup / restore

As I said in my review I would telling you how to install this on CentOS 5.5, so here it is.

Normally one would simply run the Symantec_Backup_Exec_System_Recovery.bin with ./Symantec_Backup_Exec_System_Recovery.bin. But on a non-supported distribution / kernel that won't work.

However all is not lost! You can install this with a little shell-fu.
Instead of ./Symantec_Backup_Exec_System_Recovery.bin run
./Symantec_Backup_Exec_System_Recovery.bin --noexec --target .

This will extract the contents of the binary to the current directory. Next you need to install a few RPM packages, but you need to know what version of kernel you are running (for me it was either PAE or non-PAE). So I ran uname -a to see what version of the kernel that machine was running.

rpm -Uhv SymSnap/symbdsnap-1.0.1-36146.2.6.18_8.el5PAE.i686.rpm
rpm -Uhv symmount-1.0.1-36352.i686.rpm
rpm -Uhv besr-1.0.1-36352.i686.rpm

Once all of these packages are installed I was able to do my first backup. And by first backup I mean first three (/boot, /, and SWAP). While I probably don't need to do all three (/boot and /) I don't want to have to create a SWAP system after a restore. So I ran all three.

besr -b/dev/sda1 -d /Storage/backups/hostname/boot.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol00 -d /Storage/backups/hostname/Vol00Log00.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol01 -d /Storage/backups/hostname/Vol00Log01.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard

Now in /Storage/backups/hostname I have three files
boot.v2i
Vol00Log00.v2i
Vol00Log01.v2i

If I needed to restore these I would use the Backup Exec System Recovery Windows based LiveCD. During the restore I would choose all three of these in the order shown above.

After the restore process was finished I need to boot using the gparted LiveCD. Once in I would open up a terminal and install grub on to my sda hard drive.

#grub
#root (hd0,0)
#setup (hd0)
#quit

Once grub is installed on sda I needed to update fstab to no longer look for / on the /dev/mapper/VolGroup00-LogVol00 and swap to not be on /dev/mapper/VolGroup00-LogVol01.

Backup Exec System Recovery Review

I asked a question over on Serverfault about backup software that supported Windows and Linux and did Bare-Metal Recovery and the response I got was to try Backup Exec System Recovery, which so far has been pretty good.

The bad stuff:

1) Restore CD creation on CentOS is impossible from what I have tried and can tell. They really do mean Redhat Enterprise Linux only.
2) Bare-metal recovery with MBR restore doesn't work. Which means I have to install grub after a restore.
3) LVM support is once again not there or well hidden.
4) Centralized management. It is a separate 1GB download as an Addon. I did the download and install, but now I can't find the Management console. Setting up each of our servers isn't too big of a problem, but I really wanted one window to view for backups.
5) There is no built in scheduler with Linux.
6) There is no incremental backup in Linux.

The good stuff:
1) It works on CentOS with a little bit of hacker-y to install (more on that in another post).
2) It is easy to install and configure on Windows, even though it requires a reboot. (BOO!)
3) It is pretty quick to get a backup started.
4) The backup procedure is different on Windows than on Linux. Windows is GUI based 100%. (I haven't looked to see if I can configure a backup from the command line.) While Linux is 100% CLI.
5) The Windows based restore CD is a pretty useful tool with out limits (unlike the Acronis Boot Disk).
6)Restores are quick and pretty simple to do.

All in all after trying three different products and looking at half a dozen I think this may be the winner.

CHECK_NRPE: Error - Could not complete SSL handshake

Recently I have been auditing our servers versus what we check in nagios versus what we need to do when a system is rebooted during routine maintenance. I found that two of our servers have been left out of nagios monitoring even though they are on my maintenance checklist, and they have software that I still start by hand (I KNOW IT'S NOT A BEST PRACTICE!).

Both servers had NRPE installed and configured (mostly), one of them even had a configuration file on the nagios server but it wasn't enabled (hostname.disabled instead of hostname.cfg). The other server needed a configuration file, but even then it wasn't working, so here are my troubleshooting steps:

1) Check to see that NRPE is compiled and installed. [It was]
2) Check that NRPE was listening (netstat -an | grep 5666) [It was]
3) Check that NRPE was listed in /etc/services [It wasn't]
4) Check the NRPE config file (/etc/xinetd.d/nrpe) for "only_from = 127.0.0.1 192.168.100.31" [It wasn't]

So I added the service definition to /etc/services and the nagios server IP to the only from line and restarted xinetd (service xinetd restart) and I was finally able to connect from my nagios server.

All of this is on CentOS 5.5 for both the server and the client.

Acronis Backup and Recovery Advanced Server 10

Getting up and running with Acronis Backup and Recovery Advanced Server 10 is pretty quick, especially for the demo. It took under an hour to install, configure and start backing up a Windows 2008 Standard server. Getting the Backup agent installed on our Linux servers took a little bit more work because of the SnapAPI kernel modules. After a little bit of digging around in the Acronis knowledge base I was able to resolve all of the issues I had getting the Acronis Backup Agent installed on our CentOS 5.4 servers. If we go with Acronis Backup and Recovery Advanced Server 10 I will need to add some additional lines to our Linux Post-Install script to add in the kernel-devel package and the additional RPMs that the agent needs. (DKMS and SnapAPI) both of which are already on our Storage drive.

With our second trial run here we where able to resolve most of the outstanding issues we had from the first trial. Namely excessive recovery times and recovering to dis-similar hardware. I have yet to truly attempt a Linux restore to dis-similar hardware, but I have the base system recovered and the instructions, so I will be attempting one later this week (with results to follow). Restoring our Windows machines to either dis-similar hardware or a VM is pretty straight forward with the Universal Restore CD. The main problem holding us back in those scenarios is not having the drivers readily available for post install. This can be remedied by always going to VM and installing the VM tools as they also contain the drivers for the system.

Each backup policy allows us to modify settings for the backups such as Encryption, Compression, and throttling resources. The encryption can have a separate encryption key and varying levels of encryption from none to AES-128, AES-192, and finally AES-256. The automated backups can do a simple backup plan with full and incremental that are stored as a file on the storage system. The policy also allows us to run custom commands before and after the backup so can could do a virus scan, or shred temp files or what ever we desired.

One of the strangest problems I came across was while doing the restores to VM, if started from the VMs Console after picking the Disk to recover to the process could take hours to complete. Whereas if I did the same restore on to physical hardware the process would take seconds. The work around is to boot the VM from the Universal Restore CD and use the Management Console to connect to the VM. Once connected to the VM from the Management console I was able to start the restore in a matter of minutes, more akin to a physical machine.

Something to remember when setting up the backup policy for Linux machines that utilize LVM is to back up the disk as a whole. You don't want to backup the LVM by its self because after the restore it will fail to boot.

All in all I feel that Acronis would make a wonderful addition to our eco-system. It fulfills all of the requirements of our backup scheme, except the web based access, which is not uncommon for Windows based software.

NDO2DB daemon startup script

I would like to thank Chris over at http://sysengineers.wordpress.com for the excellent post on how to daemonize NDO2DB. His post NDO2DB startup script for RH (EL) / OEL does an excellent job getting everything working. I had to make one small change to the script as I kept my ndo2db executable as ndo2db-3x instead of just ndo2db.

For me this fixes a problem where I have to remember to manually start the ndo2db service after a server restart, which hasn't been happening the past few times. The second way I am going to fix this problem is by creating a checklist of things that need to happen during a server restart / boot up sequence for each PC. This will also reduce the effort needed for when I move the rack later this year. The third way I will be checking this is by adding a check to Nagios to make sure this is running at all times.

This also illustrates my general way of checking / double checking things. One go off of a checklist. The double check is to verify that Nagios is seeing the same thing.

Installing Acronis Backup and Recovery 10 Linux Agent on CentOS 5.4

I know it has been a while since I have posted anything. I have been busy testing out backup and recovery software. The latest one is Acronis Backup and Recovery 10 Advanced Server. The management server is very easy to setup and configure. I had some issues however while installing the Agent for Linux. But with a little research I was able to find what I needed and get everything installed.

Step number one is to make sure you have the Kernel Development package for your kernel. The easiest way to find out which version you need is to run the command "uname -r". On one server I needed the "kernel-devel" package and on another I needed the "kernel-PAE-devel" package. So if you don't have them already install the correct one for your server.

After this is installed you can install another prerequisite package Dynamic Kernel Module Support Framework or DKMS for short. I got mine off of DAG's repo at http://dag.wieers.com/rpm/packages/dkms/. Once this is downloaded install the package.

The last prerequisite is an updated SnapAPI module. Thanks to the Acronis KB I was able to find the updated package at http://kb.acronis.com/sites/default/files/content/2009/10/4371/snapapi26_modules-0.7.47-1.noarch.rpm.

Now that we have all of the packages installed we can install the Agent for Linux. The install is rather painless, except the License Key that you have to type in, by hand, every time for the trial edition. Maybe its only a pain point for me because I had to do it so many times while trying to get it installed.

TL;DR

#uname -r
#yum install kernel-devel
#wget http://dag.wieers.com/rpm/packages/dkms/dkms-2.0.17.6-1.rh9.rf.noarch.rpm
#rpm -Uhv http://dag.wieers.com/rpm/packages/dkms/dkms-2.0.17.6-1.rh9.rf.noarch.rpm
#wget http://kb.acronis.com/sites/default/files/content/2009/10/4371/snapapi26_modules-0.7.47-1.noarch.rpm
#rpm -Uhv snapapi26_modules-0.7.47-1.noarch.rpm
#./AcronisAgentLinux.i686

Mounting a LVM volume in Ubuntu (Live CD)

A while back my testing server crashed. This was no surprise to anyone as it was just a (Very) old workstation. However it was running my nagios install in a production setting. I had been meaning to move it to a proper server, but just hadn't gotten around to it. To make matters worse, I didn't back any of it up. So thankfully it was only the motherboard that failed and not the HDD.

I mounted the HDD in another PC I had sitting around and booted it using Ubuntu Live CD.

First, boot Ubuntu.

Second, install the needed tools:

$ sudo apt-get install lvm2

Third, load the modules to do our task:

$ sudo modprobe dm-mod

Fourth, scan the system for LVM volumes. Look for the volumes you want to mount. Typically this will be VolGroup00:

$ sudo vgscan

Fifth, we need to activate the volume(s):

$ sudo vgtchange -ay VolGroup00

Sixth, Look for the logical volume containing the root file system. Typically this will be LogVol00:

$ sudo lvs

Seventh, create the directory to mount the drive:

$ sudo mkdir /mnt/restore

Eighth, Mount the volume to the directory you just created.

$ sudo mount /dev/VolGroup00/LogVol00 /mnt/restore -o ro,user

Ninth, Copy your files off of the drive.

$ cp /mnt/restore/some/dir/and/path /some/dir/and/path

Tenth, Setup what ever backup means you have on the new server!

All in all this wasn't a terrible thing, it could have been much worse. I have since moved the nagios setup to a virtual machine and am backing it up nightly.

Bash: Finding files between two dates in the current directory

Today my boss asked me for a bash command (or script) to find some files between two dates.

Thanks to Jadu Saikia over at Unstableme his post UNIX BASH scripting: Find Files between two dates, I had a starting point.

This will find all files between the two dates (20071019 & 20071121) in this case.

find . -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'

Now, if you want just PGP files you would do:

find *.pgp -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'

The second request that my boss was looking for with this is the file size, something that was being left out by awk. So we can fix that by updating the command to:

find *.pgp -type f -exec ls -lh --time-style=full-iso {} \; | awk '{print $6,$NF,$5}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20090624 && $1<= 20090901 {print $2,$3}'

We added in a $5 to the first awk command, and the final one had $3 added to it. Also I like human readable file sizes so I added -h to the ls command.

Copy Directory Structure Only

This is a simple one liner for copying a Directory structure, and not the contents.

find * -type d -exec mkdir /new_directory/\{\} \;

Now there are a few caveots to this of course, but they are simple.

First the /new_directory/ has to exist.

Second, you have to run the command from within the directory that you want to copy the structure from.

For example, if I need to copy the structure of /Storage to /newStorage I would:

mkdir /newStorage

cd /Storage

find * -type d -exec mkdir /newStorage/\{\} \;

Failed to modify password entry for user while adding user with smbpasswd

I was trying to create a user today in samba and was getting the error "Failed to modify password entry for user [USER]".

I read over the man pages for smbpasswd and saw that I needed to add the -n switch because this user will not have a password. So again I tried to add the user with "smbpasswd -a -n [USER]" and got the same error.

A quick Google search lead me to a newb mistake on my part... I didn't have the user I was trying to add in my UNIX password file. A quick "useradd [USER]" followed by the same "smbpasswd -a -n [USER]" and I was all set.

Granted there are other reasons why this process might fail, but for me this was the reason.

mail command returns fseek Invalid argument, panic temporary file seek

Today I logged into one of our older servers, haven't logged into it for a while and saw that there was new mail (as always given the number of cron jobs running).

Below is the output of my command.

[root@server /]# mail 

"/var/spool/mail/root": 1832 messages 1777 new 1832 unread 

fseek: Invalid argument 

panic: temporary file seek

After a little bit of searching online I found two possible and simple solutions.

If you want to read the mail try using mutt instead of mail. It doesn't have a problem with the 2GB file size of the mail box that mail did.

If you don't care about the old stuff you can run:

rm -f /var/spool/mail/root

to remove the file and then

cat /dev/null > /var/spool/mail/root

to recreate a blank file.  

I ended up reading the mail I wanted and then blowing the file away.  But before I could recreate it there was already a new file with 62MB of mail in it.  

Excluding Directories from updatedb on CentOS 5

Running the updatedb command will update the slocate database.  However if you want to exclude certain directories for any reason, such as not wanting to include a huge NFS file store or something to that affect you have two options.

1) use the -e switch with a comma seperated list of directories to not index. (updatedb -e /Storage,/home)

2)edit the /etc/updatedb.conf file.

•  vi /etc/updatedb.conf
  
• find the PRUNEPATHS section and add the directories to the list seperated by spaces. (PRUNEPATHS = "/afs /media /net /sfs /tmp /udev /var/spool/cups /var/spool/squid /var/tmp /Storage /home")

Also you can read up on updatedb by reading the man pages.

man updatedb

memcached init.d statup scripts for CentOS 5.2.

While working to setup memcached on my CentOS servers I came across these scripts.  They are the typical startup, restart, shutdown, status scripts I am sure you are used to using.  

I found them on both http://www.dev411.com/wiki/Memcached_startup_files_for_Red_Hat_(RHEL) and http://www.vbseo.com/blogs/danny-bembibre/daemon-scripts-memcached-44/.  

Both sites have the same script contents but I feel that vbseo.com has the more complete instructions.