Installing and Configuring ZendServer Community Edition (CE) on CentOS 5 / 6

The quick and the dirty:

Download the Zend Server (DEB/RPM Installer Script) from zend.com. (An account is required).

un-pack the the tarball (tar -xzfv ZendServer-5.5.0-RepositoryInstaller-linux.tar.gz)

Run ./install_zs.sh 5.3 ce or ./install_zs.sh 5.2 ce depending on which version of PHP you want to run.

Edit your iptables (you are running iptables right?) vi /etc/sysconfig/iptables

Add in a line for the ZendServer lighthttpd server (-A INPUT -m state --state NEW -m tcp -p tcp --dport 10081 -j ACCEPT)

Restart iptables (/sbin/service iptables restart)

Visit (http://YOURSERVERHERE.com:10081/ZendServer/) in a web browser to accept the EULA and set a password.

Alternatively run (/usr/local/zend/bin/zs-setup accept-eula) and (/usr/local/zend/bin/zs-setup set-password YOURSECUREPASSWORD )

If you need to restart Zend Server run /sbin/service zend-server restart.  This will restart both apache (httpd) and the LightHTTPD Zend Server gui.

Some important notes before heading off into the wonderful world of Zend Server:
Be sure that your distribution's PHP isn't installed as well as the Zend Server, including the CLI, as it will mess with running php from the command line and who knows what else.

The php binary is located at /usr/local/zend/bin/php, which can be verified by running which php.  As such if you need to run php from cron be sure to add this to your path. (I have PATH=$PATH:$HOME/bin:/usr/local/zend/bin in my ~/.bash_profile).

If you need to modify a setting in php.ini you will find it at /usr/local/zend/etc/php.ini.  Remember to restart zend server for any changes to take effect.

Beyond those things there isn't too much difference between running zend server and running php from your distribution.

System Activity Report (sar) and You

sar is an acronym for System Activity Report. It takes a snapshot of the system periodically. On most distributions it comes with the sysstat package. On Redhat and derived distributions the package will install a set of cron jobs in /etc/cron.d/sysstat. There are two cron jobs to take note of. The first one runs every ten minutes as root. It runs the script /usr/lib/sa/sa1 -S Disk 1 1. This script saves its output in report files. The files are written to /var/log/sa/sar[dd], where [dd] is the two digit date for today's date. (e.g. Today is 8/26/2011. The log file is /var/log/sa/sar26)

The second cron job runs at 23:53. This cron job summarizes the days activity. Both of the reports are saved as binary data, so normal tools are useless here.

There are many flags to use with sar when running it interactivly. Some of the flags have additional atguments that are required when used. One example of thses flags is -n which al requires an additional argement suach as DEV or NFS ro IP. This specific example has eightteen (18) potential arguments, not including ALL.

With sar liberal use of the man pages are highly suggested. Not only are the flags and any arguments documented, but the headers for each one and what they represent are explained as well. This comes in handy if you get overzealous with flags and aren't quite sure what you're looking at.

I'm only going to cover some of the most notable flags, what they show and their headers. Be careful however as some flags exist in both upper and lower case and report vastly different metrics. One example of the is -b which reports on I/O transfer but -B reports on paging stats.

First off the plate is -b which as I've already stated reports on I/O transfer stats and has the following headers:
    tps: Transfers per second to a physical device.
    rtps: Read transfers per second to a physical device.
    wtps: Write transfers per second to a physical device.
    bread/s: Blocks (since kernel 2.4 = sectors = 512 bytes) read from devices per second.
    bwrtn/s: Blocks written to devices per second.
Example output from a production server running: sar -b 1 1
  Linux 2.6.18-274 el5PAE (server.domain.com) 8/26/2011
  09:23:57 PM    tps             rtps         wtps          bread/s     bwrtn/s
  09:23:58 PM    7829.00    133.00    7696.00    1528.00    76960.00
  Average:           7829.00    133.00    7696.00    1528.00    76960.00

-B will report paging stats. Some metrics / headers are only available in kernels 2.5 and newer.
    pgpgin/s: Kilobytes paged in from disk per second.
    pgpgout/s: Kilobytes paged out from disk per second.
    magflt/s: Major faults per second (hits to disk, this is a bad thing...)

-c Process creation stats.
    proc/s: Processes created per second.

-d Activity for each block device.
    tps: Transfers per second
    rd_sec/s: Sectors (512 bytes) read from block device per second.
    wr_sec/s: Sectors (512 bytes) written to block device per second.
    avgrq-sz: Average # of sectors
    avgqu-sz: Average queue length
    await: Average time in milliseconds for queue + servicing request.
    svctm: Average servicing time.
    %util: CPU percentage while I/O requests where issued. Close to 100% = device saturation.
 
-n DEV Network interface stats.
    IFACE: Interface Name
    rxpck/s: Packets received per second.
    tcpck/s: Packets sent per second.
    rxbyt/s: Bytes received per second.
    txbyt/s: Bytes sent per second.
    rxcmp/s: Compressed packets received per second.
    txcmp/s: Compressed packets sent per second.
    rxmcst/s: Multicast packets received per second.

-P ALL Per processor (or core) stats

-p Print pretty device names
    Shows block devices as sda instead of dev8-0. Has no effect on Network device names.
-A same as: -bBcdqrRuvwWy -I SUM -I XALL -n ALL -P ALL

After you install the sysstat package you really need to let it run for a while and gather stats to see the real beauty of it all.  However you can run it interactively if required. When running sar interactively the syntax is sar -FLAGS Interval Duration. (e.g. sar -b 2 60) runs I/O stats every two seconds for a minute.  This is very hand to run if you're troubleshooting a slow system or watching it under load.

Changing the from e-mail address in Nagios

This is going to be another short one.  I recently had a need to change the FROM address for e-mail from our Nagios installation. E-mail was coming from nagios@host.domain.com, which is non-route-able  from outside our network.

The change is very simple. Change the two command lines in your Nagios commands.cfg dealing with notify by e-mail. The command names are "notify-host-by-email" and "notify-service-by-email".

By default these lines read:
 /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

To change  the from address you append " -- -f nagios@domain.com", that is without the quotes of course.  So the new lines look like:

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/        Time: $LONGDATETIME$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$ -- -f nagios@domain.com

and

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE        $\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$  -- -f nagios@domain.com

To explain it a little what we appended was a space followed by two dashes, which forces mail to pass those along to sendmail. Send mails sees the addition of  a dash followed by the letter f and then another space and the e-mail address you want to send from.

On my system this is an alias for myself so that all replies come to my inbox.

All of this works on CentOS which is what I'm currently running.  It should work on other distributions as well, but I haven't the time to verify that.

Yum 5.x Local repository and the new Continuous Release Repo

A few months back, possibly a year ago I started running my own local mirror of the CentOS YUM repository. The main benefit is speed in updating all of my servers. I also was able to setup a local repo of just tools that we use but aren't available in the Base CentOS repos. (I've since started adding the EPEL repository to servers that needed it.) For those curious I mirror the os, updates, and now the cr repos.

A few weeks ago the CentOS announce list and twitter feed introduced yet another repo the Continuous Release repo or cr for short. This new repo contains security and bug fixes from the upstream 5.7 release and I imagine will continue with further releases (5.8 & 5.9). CentOS has also promised a 6.0 Continuous Release repo, but have yet to deliver on that.


First off create the directories that will become the repositories.
mkdir -pv /Storage/yumRepo/{5,5.4,5.5/{os,updates,cr}/i386
mkdir -pv /Storage/yumRepo/local/el5/i386


So with out further ado, lets get into the guts of actually getting everything pulled down, synced and a repository created.

Below is the script I use to keep my mirror in sync with the closest mirror that offers rsync:

      1 #!/bin/sh
      2
      3 rsync="rsync -avrt --bwlimit=796"
      4
      5 mirror=rsync://centos.mirrors.tds.net/CentOS
      6
      7 verlist="5 5.5 5.6 5.7"
      8 archlist="i386"
      9 baselist="os updates cr"
     10 local=/Storage/yumRepo/centos
     11
     12 for ver in $verlist
     13 do
     14  for arch in $archlist
     15  do
     16   for base in $baselist
     17   do
     18     remote=$mirror/$ver/$base/$arch/
     19     $rsync $remote $local/$ver/$base/$arch/
     20     createrepo -v --update $local/$ver/$base/$arch
     21   done
     22  done
     23 done
     24 hardlink /Storage/yumRepo/centos

Now, lets break this script down line by line.  Line one is obvious, it tells the program loader what interpreter to use when running this file. Line three sets up the rsync command I'll be using.  In my case I need to limit the bandwidth usage to something sane, even though I run it off peak hours. Line five sets the mirror I'm running against. Line seven sets the versions of CentOS I'm interested in.  I really could trim this back to just 5 and 5.7 now, but I'm going to leave it in there for now. Line eight tells me I'm only going to sync i386 architect and not the x64 architect. I'm doing this because as of right now we only run x86. Line nine gives me which repositories in the i386 architect I'm going to mirror. Line ten sets up where on the local server I'm going to store everything.

Lines twelve through twenty three run rsync against all of the variations of repository and architect type for each version. In my case it would be twelve different syncs. If I where also running x64 architect it would double to twenty four. Line nineteen is the actual rsync command while line twenty creates the necessary repository files for yum to use.

Line twenty four is rather new to this script and requires you have the hardlink command installed (yum install hardlink). The hardlink command will run through everyfile in the directory specified and compare it to every other file in the directory.  It is looking for the same contents with the same permissions, but it can have a different filename.  If it finds a match it will hardlink one of them to the other, thus reducing the space required to store two copies. My only wish is that I had some hard stats for how much disk space this is saving me on the repository alone.


Now create the repository file for Yum.
#vi /etc/yum.repos.d/MyRepo.repo
This file contains:
[base]
name=CentOS-$releasever - Base
baseurl=repo.mydomain.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
protect=1
enabled=1

[update]
name=CentOS-$releasever - Updates
baseurl=http://repo.mydomain.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
protect=1
enabled=1

[CompanyName]
name=CentOS-$releasever - Local
baseurl=http://repo.mydomain.com/local/el$releasever/$basearch
enabled=1
gpgcheck=0
protect=0
enabled=0


We need to also tell Apache about these directories and how to serve them out over HTTP.
I changed the DocumentRoot to
DocumentRoot "/Storage/yumRepo"
then I changed the Default directory to

Options +Indexes +FollowSymLinks
AllowOverride All
Allow from all

Restart apache.
#service httpd restart


Populate Company Local repo with custom RPMs.
On every server we have installed Webmin. In the past this was installed either by hand or via our install script, but that had to be updated by hand with each update. So I will move the latest into our Company Local repo.
#mv /Storage/rpmDownloads/webmin-1.510.rpm /Storage/yumRepo/local/el5/i386/

You can move as many files into here as you want, including built from source RPMs that you may have made. After you move in any custom RPMs you have to update the Repository.
#createrepo -v --update /Storage/yumRepo/local/el5/i386/

Comparing ext3 to ext4 benchmarks

Below is the output from bonnie++, installed from rpmforge on CentOS 5.5, running against a RAID 0 (Linux Software RAID) of two Seagate Barracuda LP ST32000542AS, formatted with ext3.
#bonnie++ -d /mnt/SOMEDRIVE -n 32:64:4:4 -q | bon_csv2html > /StorageServer/RAID0.ext3.html

Here is the same command but running on the same server, but this time running CentOS 5.6 with the same Linux software RAID 0 on two Seagate Barracuda LP ST32000542AS, but formatted with ext4.

We can see that Sequential Output Per Character is slightly higher, Block K/Sec is higher and with significantly lower latency, but Rewrite is lower and with higher latency on ext4. (Higher is better)

Sequential input on both Per Caharacter and Block K/Sec is also slightly higher and with lower latency on ext 4. Random seeks are also slightly higher on ext4. (Higher is better)

Sequential creates are higher, but require more CPU, while Sequential Reads are lower using about the same CPU. Sequential Deletes are also slower on ext4 than they where on ext3.

Where ext4 really seems to shine is on the Random Create section. Creating random files almost doubled per second on ext4 vs. ext3, all while using similar CPU. Random Reads where so fast on ext4 that I need to retest to get accurate results (shown by +++++ on the results). Random Deletes where about 25% faster on ext4 when compared to ext3 again while using less CPU, though not by much (1%).

Now these being benchmarks they give a nice indication about what kinds of performance I may be able to have, but I need to do real world testing. Hopefully I have some meaningful results after I finish upgrading all of my servers and filesystems. I will also benchmark our RAID 5 system before and after the conversion.

For our purposes it appears that ext4 is the better filesystem given our setup and procedures.

Converting an ext3 filesystem to ext4 on CentOS 5.6

If you recently installed CentOS 5.6 or updated to CentOS 5.6 you can now utilize the ext4 filesystem.

To do the conversion or create a new ext4 filesystem you need the e4fsprogs tool kit from yum.

#yum -y install e4fsprogs

I did this conversion on a test system with a non-root filesystem to avoid any possible problems. I also backed up the filesystem just in case something went terribly wrong.

First you need to un-mount the filesystem, as it cannot be in use.

#cd /; umount /dev/VolGroup00/LogVol00

Now you can run the tune4fs command to convert the filesystem to ext4.
#tune4fs -O extents,uninit_bg,dir_index /dev/VolGroup00/LogVol00

Now that the filesystem is ext4 it is no longer able to be mounted as ext3, so change its entry in the fstab.

#vi /etc/fstab

Now because we used the option uninit_bg we need to run fsck on the new ext4 filesystem. This is a good idea to do anyways, but is a requirement here.

#e4fsck -fDC0 /dev/VolGroup00/LogVol00

e4fsck will complain about "One or more block group descriptor checksums are invalid", this is totally normal.

Before doing these steps on the root filesystem (/) I would recommend you read over the Ext4 Howto on kernel.org. I re-wrote the steps for more accurate information pertaining specifically to CentOS 5.6 and my system.

As always:

THE INFORMATION IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE INFORMATION PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW THE AUTHOR WILL BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE INFORMATION TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

NRPE: Unable to read output and sudo

Thank you to Andrea Leofreddi over at cyberz.org for the blog post Nagios nrpe and sudo: “NRPE: Unable to read output”. This was a tremendous help back when I first started working with my md-raid device and Nagios. I found this entry again while working on my very own plug-in for Nagios, check_supervisorctl.sh.

In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty

For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh

Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}

Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:

[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!

Adding a vCPU to a Windows 2008 R2 guest

Last night I had to add an additional vCPU to a Windows 2008 R2 guest. I tried and tried to research doing this, but all I was able to find was how to Hot add a vCPU, which I didn't care to do. Also all of the VMWare documentation said to view a certain PDF to see what OSes could even take a Hot CPU addition, but I was unable to find anywhere in the PDF what OSes could handle it and which ones couldn't.

So I installed the newest patches (being patch Tuesday and all) and I shut down the server. I then went from one vCPU to two vCPUs. I ignored the warning that adding CPUs to an already installed system may cause it to be unstable.

So far the server has been stable and everything worked like I had hoped. I don't know if this would work on any of our Windows 2008 servers, or for that matter the CentOS 5.5 servers, but all of them already have two vCPUs.

If you are wondering why I needed to do this, well, it is because our VoIP recording solution uses MySQL, and for some reason it has been using 100% of the CPU for weeks now. This despite a mostly empty process list in MySQL.

Strange Nagios Error Solved

This morning I added some new services to a server, but they wouldn't move out of "Pending" status. The error I received was
"Feb 9 09:27:36 nagios: Warning: Check result queue contained results for service '' on host '', but the service could not be found! Perhaps you forgot to define the service in your config files?"

I stopped the nagios service and ran ps -ef | grep nagios. To my surprise there was still a Nagios instance running. This means two things. First the init script that comes with Nagios is borked and doesn't correctly check for running nagios instances. Second, I somehow started a second Nagios instance.

I thought something was up in the first place because every other refresh or so of the Nagios web view I would either see the three pending services or I wouldn't. This was my first clue that something was borked. I then went to tail /var/log/messages and saw the error message. Then I started investigating the issue with the help of Google. Once I saw that there where two instances of Nagios things started to make sense.

I killed the second Nagios instance and any children processes (in my case ndo2db) and then restarted nagios via the init script.

Once I had everything up and running (but only one instance) I was able to successfully check my new services.

ImageMagick convert -geometry weirdness

I don't know if this is really documented anywhere, but when using the ImageMagick program convert with the -geometry or -resize flags it also converts to 16bit color, at least when used against 1bit (Black and White) image.

The reason this is important to note is that using -compress Group4 will not work on these images because they aren't 1bit color images and they need to be. To work around this limitation you need to force the color depth with -monochrome. The downside to all of this is time.

Ripping an Image file from a PDF takes roughly .206 seconds for an 8.5x11 PDF with pdfimages (useful if a PDF has OCR embedded in it.)
Converting the resultant PBM file without -geometry 1700x2200! -monochrome takes roughly .194 seconds.
Converting the resultant PBM file with -geometry 1700x2200! -monochrome takes roughly 4.594 seconds. This is an increase of 10 - 10.5 % adding in the geometry.

*all of the above numbers are from one file, but testing of different files showed similar results. All tests where done at 200 DPI.

Here are some examples of the commands we are running for anyone curious.
convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 -geometry 1700x2200! -monochrome ${outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 -geometry 2550x3300! -monochrome ${outputTiffRoot}_300.tiff

Documentation, not always the How To Dos

Documentation for me is often just How to do something, but I have been forgetting the WHY part of the equation. For instance, my documentation says to do our weekly maintenance window after 21:30 and to only reboot one particular server after 21:45, but I didn't say why and had forgotten myself. So I started doing the maintenance earlier and at one point rebooted the one particular server at 21:30. This caused the last of our production cron jobs to not run, and thus a customer didn't get their batch for the day.

Now this could have been avoided a few different ways. Since that first time was a mistake any thing that happens after it is a failure on my part, and thus can't happen again.

1. I could have followed our procedure to the T.
2. Read my e-mail to see that the cron job hadn't run yet.
3. Run the cron job by hand after the server restart.
4. Did all of the precursor work and waited for the cron job to run.
5. Did all of the precursor work and run the cron job by hand.

I have chosen to go with option 1, follow our procedure to the T. We started doing the weekly maintenance for a few reasons and it was to be done after 9:30, with that one particular server last to ensure this didn't happen. Now once again I have my WHY, and I have it written down so if I question it again I know why.


This is just one instance where the WHY is critical, but there are others. So please when you are writing documentation, and you should be please include the WHY you do something the way you do it. It also helps train the new guy, or your replacement.

Installing Backup Exec System Restore on CentOS 5.5 and First backup / restore

As I said in my review I would telling you how to install this on CentOS 5.5, so here it is.

Normally one would simply run the Symantec_Backup_Exec_System_Recovery.bin with ./Symantec_Backup_Exec_System_Recovery.bin. But on a non-supported distribution / kernel that won't work.

However all is not lost! You can install this with a little shell-fu.
Instead of ./Symantec_Backup_Exec_System_Recovery.bin run
./Symantec_Backup_Exec_System_Recovery.bin --noexec --target .

This will extract the contents of the binary to the current directory. Next you need to install a few RPM packages, but you need to know what version of kernel you are running (for me it was either PAE or non-PAE). So I ran uname -a to see what version of the kernel that machine was running.

rpm -Uhv SymSnap/symbdsnap-1.0.1-36146.2.6.18_8.el5PAE.i686.rpm
rpm -Uhv symmount-1.0.1-36352.i686.rpm
rpm -Uhv besr-1.0.1-36352.i686.rpm

Once all of these packages are installed I was able to do my first backup. And by first backup I mean first three (/boot, /, and SWAP). While I probably don't need to do all three (/boot and /) I don't want to have to create a SWAP system after a restore. So I ran all three.

besr -b/dev/sda1 -d /Storage/backups/hostname/boot.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol00 -d /Storage/backups/hostname/Vol00Log00.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol01 -d /Storage/backups/hostname/Vol00Log01.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard

Now in /Storage/backups/hostname I have three files
boot.v2i
Vol00Log00.v2i
Vol00Log01.v2i

If I needed to restore these I would use the Backup Exec System Recovery Windows based LiveCD. During the restore I would choose all three of these in the order shown above.

After the restore process was finished I need to boot using the gparted LiveCD. Once in I would open up a terminal and install grub on to my sda hard drive.

#grub
#root (hd0,0)
#setup (hd0)
#quit

Once grub is installed on sda I needed to update fstab to no longer look for / on the /dev/mapper/VolGroup00-LogVol00 and swap to not be on /dev/mapper/VolGroup00-LogVol01.

Backup Exec System Recovery Review

I asked a question over on Serverfault about backup software that supported Windows and Linux and did Bare-Metal Recovery and the response I got was to try Backup Exec System Recovery, which so far has been pretty good.

The bad stuff:

1) Restore CD creation on CentOS is impossible from what I have tried and can tell. They really do mean Redhat Enterprise Linux only.
2) Bare-metal recovery with MBR restore doesn't work. Which means I have to install grub after a restore.
3) LVM support is once again not there or well hidden.
4) Centralized management. It is a separate 1GB download as an Addon. I did the download and install, but now I can't find the Management console. Setting up each of our servers isn't too big of a problem, but I really wanted one window to view for backups.
5) There is no built in scheduler with Linux.
6) There is no incremental backup in Linux.

The good stuff:
1) It works on CentOS with a little bit of hacker-y to install (more on that in another post).
2) It is easy to install and configure on Windows, even though it requires a reboot. (BOO!)
3) It is pretty quick to get a backup started.
4) The backup procedure is different on Windows than on Linux. Windows is GUI based 100%. (I haven't looked to see if I can configure a backup from the command line.) While Linux is 100% CLI.
5) The Windows based restore CD is a pretty useful tool with out limits (unlike the Acronis Boot Disk).
6)Restores are quick and pretty simple to do.

All in all after trying three different products and looking at half a dozen I think this may be the winner.

CHECK_NRPE: Error - Could not complete SSL handshake

Recently I have been auditing our servers versus what we check in nagios versus what we need to do when a system is rebooted during routine maintenance. I found that two of our servers have been left out of nagios monitoring even though they are on my maintenance checklist, and they have software that I still start by hand (I KNOW IT'S NOT A BEST PRACTICE!).

Both servers had NRPE installed and configured (mostly), one of them even had a configuration file on the nagios server but it wasn't enabled (hostname.disabled instead of hostname.cfg). The other server needed a configuration file, but even then it wasn't working, so here are my troubleshooting steps:

1) Check to see that NRPE is compiled and installed. [It was]
2) Check that NRPE was listening (netstat -an | grep 5666) [It was]
3) Check that NRPE was listed in /etc/services [It wasn't]
4) Check the NRPE config file (/etc/xinetd.d/nrpe) for "only_from = 127.0.0.1 192.168.100.31" [It wasn't]

So I added the service definition to /etc/services and the nagios server IP to the only from line and restarted xinetd (service xinetd restart) and I was finally able to connect from my nagios server.

All of this is on CentOS 5.5 for both the server and the client.

Acronis Backup and Recovery Advanced Server 10

Getting up and running with Acronis Backup and Recovery Advanced Server 10 is pretty quick, especially for the demo. It took under an hour to install, configure and start backing up a Windows 2008 Standard server. Getting the Backup agent installed on our Linux servers took a little bit more work because of the SnapAPI kernel modules. After a little bit of digging around in the Acronis knowledge base I was able to resolve all of the issues I had getting the Acronis Backup Agent installed on our CentOS 5.4 servers. If we go with Acronis Backup and Recovery Advanced Server 10 I will need to add some additional lines to our Linux Post-Install script to add in the kernel-devel package and the additional RPMs that the agent needs. (DKMS and SnapAPI) both of which are already on our Storage drive.

With our second trial run here we where able to resolve most of the outstanding issues we had from the first trial. Namely excessive recovery times and recovering to dis-similar hardware. I have yet to truly attempt a Linux restore to dis-similar hardware, but I have the base system recovered and the instructions, so I will be attempting one later this week (with results to follow). Restoring our Windows machines to either dis-similar hardware or a VM is pretty straight forward with the Universal Restore CD. The main problem holding us back in those scenarios is not having the drivers readily available for post install. This can be remedied by always going to VM and installing the VM tools as they also contain the drivers for the system.

Each backup policy allows us to modify settings for the backups such as Encryption, Compression, and throttling resources. The encryption can have a separate encryption key and varying levels of encryption from none to AES-128, AES-192, and finally AES-256. The automated backups can do a simple backup plan with full and incremental that are stored as a file on the storage system. The policy also allows us to run custom commands before and after the backup so can could do a virus scan, or shred temp files or what ever we desired.

One of the strangest problems I came across was while doing the restores to VM, if started from the VMs Console after picking the Disk to recover to the process could take hours to complete. Whereas if I did the same restore on to physical hardware the process would take seconds. The work around is to boot the VM from the Universal Restore CD and use the Management Console to connect to the VM. Once connected to the VM from the Management console I was able to start the restore in a matter of minutes, more akin to a physical machine.

Something to remember when setting up the backup policy for Linux machines that utilize LVM is to back up the disk as a whole. You don't want to backup the LVM by its self because after the restore it will fail to boot.

All in all I feel that Acronis would make a wonderful addition to our eco-system. It fulfills all of the requirements of our backup scheme, except the web based access, which is not uncommon for Windows based software.

NDO2DB daemon startup script

I would like to thank Chris over at http://sysengineers.wordpress.com for the excellent post on how to daemonize NDO2DB. His post NDO2DB startup script for RH (EL) / OEL does an excellent job getting everything working. I had to make one small change to the script as I kept my ndo2db executable as ndo2db-3x instead of just ndo2db.

For me this fixes a problem where I have to remember to manually start the ndo2db service after a server restart, which hasn't been happening the past few times. The second way I am going to fix this problem is by creating a checklist of things that need to happen during a server restart / boot up sequence for each PC. This will also reduce the effort needed for when I move the rack later this year. The third way I will be checking this is by adding a check to Nagios to make sure this is running at all times.

This also illustrates my general way of checking / double checking things. One go off of a checklist. The double check is to verify that Nagios is seeing the same thing.

Installing Acronis Backup and Recovery 10 Linux Agent on CentOS 5.4

I know it has been a while since I have posted anything. I have been busy testing out backup and recovery software. The latest one is Acronis Backup and Recovery 10 Advanced Server. The management server is very easy to setup and configure. I had some issues however while installing the Agent for Linux. But with a little research I was able to find what I needed and get everything installed.

Step number one is to make sure you have the Kernel Development package for your kernel. The easiest way to find out which version you need is to run the command "uname -r". On one server I needed the "kernel-devel" package and on another I needed the "kernel-PAE-devel" package. So if you don't have them already install the correct one for your server.

After this is installed you can install another prerequisite package Dynamic Kernel Module Support Framework or DKMS for short. I got mine off of DAG's repo at http://dag.wieers.com/rpm/packages/dkms/. Once this is downloaded install the package.

The last prerequisite is an updated SnapAPI module. Thanks to the Acronis KB I was able to find the updated package at http://kb.acronis.com/sites/default/files/content/2009/10/4371/snapapi26_modules-0.7.47-1.noarch.rpm.

Now that we have all of the packages installed we can install the Agent for Linux. The install is rather painless, except the License Key that you have to type in, by hand, every time for the trial edition. Maybe its only a pain point for me because I had to do it so many times while trying to get it installed.

TL;DR

#uname -r
#yum install kernel-devel
#wget http://dag.wieers.com/rpm/packages/dkms/dkms-2.0.17.6-1.rh9.rf.noarch.rpm
#rpm -Uhv http://dag.wieers.com/rpm/packages/dkms/dkms-2.0.17.6-1.rh9.rf.noarch.rpm
#wget http://kb.acronis.com/sites/default/files/content/2009/10/4371/snapapi26_modules-0.7.47-1.noarch.rpm
#rpm -Uhv snapapi26_modules-0.7.47-1.noarch.rpm
#./AcronisAgentLinux.i686

Mounting a LVM volume in Ubuntu (Live CD)

A while back my testing server crashed. This was no surprise to anyone as it was just a (Very) old workstation. However it was running my nagios install in a production setting. I had been meaning to move it to a proper server, but just hadn't gotten around to it. To make matters worse, I didn't back any of it up. So thankfully it was only the motherboard that failed and not the HDD.

I mounted the HDD in another PC I had sitting around and booted it using Ubuntu Live CD.

First, boot Ubuntu.

Second, install the needed tools:

$ sudo apt-get install lvm2

Third, load the modules to do our task:

$ sudo modprobe dm-mod

Fourth, scan the system for LVM volumes. Look for the volumes you want to mount. Typically this will be VolGroup00:

$ sudo vgscan

Fifth, we need to activate the volume(s):

$ sudo vgtchange -ay VolGroup00

Sixth, Look for the logical volume containing the root file system. Typically this will be LogVol00:

$ sudo lvs

Seventh, create the directory to mount the drive:

$ sudo mkdir /mnt/restore

Eighth, Mount the volume to the directory you just created.

$ sudo mount /dev/VolGroup00/LogVol00 /mnt/restore -o ro,user

Ninth, Copy your files off of the drive.

$ cp /mnt/restore/some/dir/and/path /some/dir/and/path

Tenth, Setup what ever backup means you have on the new server!

All in all this wasn't a terrible thing, it could have been much worse. I have since moved the nagios setup to a virtual machine and am backing it up nightly.

Nagios: check_http, using the --invert-regex option

Some times you want to check that something is running or working correctly and you work out tests for that. Other times you want to know when something is broken and throwing error messages. This is about the latter, a proper HTTP 200 code is great and all, but what if the page is just showing "Too Many Connections" instead of your home page? My old check_http command for this server used to look like, well, check_http. I didn't check anything about it specifically, just that it was returning a 200 code.

Today however I knew I needed something more in depth. Our database server lost its local network connection, but still was available over the public IP, which is what I test against. Once we re-directed the SQL requests to the public IP address of the server everything started working again, until we ran across "Too Many Connections". The database server kept all of the "local" connections open and thus we ate up the rest.

So, how to test for this scenario? After reading through the man pages of check_http I saw this little gem "--invert-regex Return CRITICAL if found, OK if not". This I knew was exactly what I was looking for! If it sees our error codes it will go Critical! Now to put this gem into practice. Here is where the man pages fall short. There is no explanation on HOW to use this, just that it exists. I tried the obvious to me "check_http -H hostname.com -w 3 -c 5 --invert-regex 'Some string'", but that didn't work. OK, lets try "check_http -H hostname.com -w 3 -c 5 --invert-regex='Some string'" nope that errored out with " option `--invert-regex' doesn't allow an argument".

Third times the charm right?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string' --invert-regex '"

# HTTP OK HTTP/1.1 200 OK - 0.355 second response time |time=0.354966s;3.000000;5.000000;0.000000 size=12975B;;;0

Yes, as it turns out third time is the charm. So that got me thinking some more. How can I ensure that the page is rendering correctly, and if it isn't fail but in a specific way?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string I want in my page' -r 'Some string I don't want to see' --invert-regex '"

You can add more than one -r to the check_http command and it will require all of them to be present for the test to pass, and if one of them fails then it will go critical! Perfect!

If you have any more insight into using the check_http command in Nagios I want to hear about it. We are always running into new failure scenarios that we didn't anticipate and I want to know about them before one of my users tells me about it.

Bash: Finding files between two dates in the current directory

Today my boss asked me for a bash command (or script) to find some files between two dates.

Thanks to Jadu Saikia over at Unstableme his post UNIX BASH scripting: Find Files between two dates, I had a starting point.

This will find all files between the two dates (20071019 & 20071121) in this case.

find . -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'

Now, if you want just PGP files you would do:

find *.pgp -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'

The second request that my boss was looking for with this is the file size, something that was being left out by awk. So we can fix that by updating the command to:

find *.pgp -type f -exec ls -lh --time-style=full-iso {} \; | awk '{print $6,$NF,$5}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20090624 && $1<= 20090901 {print $2,$3}'

We added in a $5 to the first awk command, and the final one had $3 added to it. Also I like human readable file sizes so I added -h to the ls command.