Comparing ext3 to ext4 benchmarks

Below is the output from bonnie++, installed from rpmforge on CentOS 5.5, running against a RAID 0 (Linux Software RAID) of two Seagate Barracuda LP ST32000542AS, formatted with ext3.
#bonnie++ -d /mnt/SOMEDRIVE -n 32:64:4:4 -q | bon_csv2html > /StorageServer/RAID0.ext3.html

Here is the same command but running on the same server, but this time running CentOS 5.6 with the same Linux software RAID 0 on two Seagate Barracuda LP ST32000542AS, but formatted with ext4.

We can see that Sequential Output Per Character is slightly higher, Block K/Sec is higher and with significantly lower latency, but Rewrite is lower and with higher latency on ext4. (Higher is better)

Sequential input on both Per Caharacter and Block K/Sec is also slightly higher and with lower latency on ext 4. Random seeks are also slightly higher on ext4. (Higher is better)

Sequential creates are higher, but require more CPU, while Sequential Reads are lower using about the same CPU. Sequential Deletes are also slower on ext4 than they where on ext3.

Where ext4 really seems to shine is on the Random Create section. Creating random files almost doubled per second on ext4 vs. ext3, all while using similar CPU. Random Reads where so fast on ext4 that I need to retest to get accurate results (shown by +++++ on the results). Random Deletes where about 25% faster on ext4 when compared to ext3 again while using less CPU, though not by much (1%).

Now these being benchmarks they give a nice indication about what kinds of performance I may be able to have, but I need to do real world testing. Hopefully I have some meaningful results after I finish upgrading all of my servers and filesystems. I will also benchmark our RAID 5 system before and after the conversion.

For our purposes it appears that ext4 is the better filesystem given our setup and procedures.

Converting an ext3 filesystem to ext4 on CentOS 5.6

If you recently installed CentOS 5.6 or updated to CentOS 5.6 you can now utilize the ext4 filesystem.

To do the conversion or create a new ext4 filesystem you need the e4fsprogs tool kit from yum.

#yum -y install e4fsprogs

I did this conversion on a test system with a non-root filesystem to avoid any possible problems. I also backed up the filesystem just in case something went terribly wrong.

First you need to un-mount the filesystem, as it cannot be in use.

#cd /; umount /dev/VolGroup00/LogVol00

Now you can run the tune4fs command to convert the filesystem to ext4.
#tune4fs -O extents,uninit_bg,dir_index /dev/VolGroup00/LogVol00

Now that the filesystem is ext4 it is no longer able to be mounted as ext3, so change its entry in the fstab.

#vi /etc/fstab

Now because we used the option uninit_bg we need to run fsck on the new ext4 filesystem. This is a good idea to do anyways, but is a requirement here.

#e4fsck -fDC0 /dev/VolGroup00/LogVol00

e4fsck will complain about "One or more block group descriptor checksums are invalid", this is totally normal.

Before doing these steps on the root filesystem (/) I would recommend you read over the Ext4 Howto on kernel.org. I re-wrote the steps for more accurate information pertaining specifically to CentOS 5.6 and my system.

As always:

THE INFORMATION IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE INFORMATION PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW THE AUTHOR WILL BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE INFORMATION TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Update to Promiscous Mode on vSphere 4

A few months back I wrote about Enabling Promiscuous Mode on vSphere 4. Well, I have learned some more since then. There isn't much information out there about recording a VoIP stream in a virtual machine environment. I hope to change that today.

First thing is first. If you are using VLANS on your physical network set your Virtual Machine Port Group to be a trunk. Also set the port on the physical switch to be a trunk instead of an access port.

Second thing, if you are using the E1000 NIC driver in your guest OS, turn off VLAN support. The screen shot is from Windows 2008 R2 64-bit, but it is similar in every other Windows OS.







Performance Best Practices for vSphere 4

NRPE: Unable to read output and sudo

Thank you to Andrea Leofreddi over at cyberz.org for the blog post Nagios nrpe and sudo: “NRPE: Unable to read output”. This was a tremendous help back when I first started working with my md-raid device and Nagios. I found this entry again while working on my very own plug-in for Nagios, check_supervisorctl.sh.

In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty

For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh

Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}

Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:

[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!

Adding a vCPU to a Windows 2008 R2 guest

Last night I had to add an additional vCPU to a Windows 2008 R2 guest. I tried and tried to research doing this, but all I was able to find was how to Hot add a vCPU, which I didn't care to do. Also all of the VMWare documentation said to view a certain PDF to see what OSes could even take a Hot CPU addition, but I was unable to find anywhere in the PDF what OSes could handle it and which ones couldn't.

So I installed the newest patches (being patch Tuesday and all) and I shut down the server. I then went from one vCPU to two vCPUs. I ignored the warning that adding CPUs to an already installed system may cause it to be unstable.

So far the server has been stable and everything worked like I had hoped. I don't know if this would work on any of our Windows 2008 servers, or for that matter the CentOS 5.5 servers, but all of them already have two vCPUs.

If you are wondering why I needed to do this, well, it is because our VoIP recording solution uses MySQL, and for some reason it has been using 100% of the CPU for weeks now. This despite a mostly empty process list in MySQL.

Strange Nagios Error Solved

This morning I added some new services to a server, but they wouldn't move out of "Pending" status. The error I received was
"Feb 9 09:27:36 nagios: Warning: Check result queue contained results for service '' on host '', but the service could not be found! Perhaps you forgot to define the service in your config files?"

I stopped the nagios service and ran ps -ef | grep nagios. To my surprise there was still a Nagios instance running. This means two things. First the init script that comes with Nagios is borked and doesn't correctly check for running nagios instances. Second, I somehow started a second Nagios instance.

I thought something was up in the first place because every other refresh or so of the Nagios web view I would either see the three pending services or I wouldn't. This was my first clue that something was borked. I then went to tail /var/log/messages and saw the error message. Then I started investigating the issue with the help of Google. Once I saw that there where two instances of Nagios things started to make sense.

I killed the second Nagios instance and any children processes (in my case ndo2db) and then restarted nagios via the init script.

Once I had everything up and running (but only one instance) I was able to successfully check my new services.

ImageMagick convert -geometry weirdness

I don't know if this is really documented anywhere, but when using the ImageMagick program convert with the -geometry or -resize flags it also converts to 16bit color, at least when used against 1bit (Black and White) image.

The reason this is important to note is that using -compress Group4 will not work on these images because they aren't 1bit color images and they need to be. To work around this limitation you need to force the color depth with -monochrome. The downside to all of this is time.

Ripping an Image file from a PDF takes roughly .206 seconds for an 8.5x11 PDF with pdfimages (useful if a PDF has OCR embedded in it.)
Converting the resultant PBM file without -geometry 1700x2200! -monochrome takes roughly .194 seconds.
Converting the resultant PBM file with -geometry 1700x2200! -monochrome takes roughly 4.594 seconds. This is an increase of 10 - 10.5 % adding in the geometry.

*all of the above numbers are from one file, but testing of different files showed similar results. All tests where done at 200 DPI.

Here are some examples of the commands we are running for anyone curious.
convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 -geometry 1700x2200! -monochrome ${outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 -geometry 2550x3300! -monochrome ${outputTiffRoot}_300.tiff