Installing Backup Exec System Restore on CentOS 5.5 and First backup / restore

As I said in my review I would telling you how to install this on CentOS 5.5, so here it is.

Normally one would simply run the Symantec_Backup_Exec_System_Recovery.bin with ./Symantec_Backup_Exec_System_Recovery.bin. But on a non-supported distribution / kernel that won't work.

However all is not lost! You can install this with a little shell-fu.
Instead of ./Symantec_Backup_Exec_System_Recovery.bin run
./Symantec_Backup_Exec_System_Recovery.bin --noexec --target .

This will extract the contents of the binary to the current directory. Next you need to install a few RPM packages, but you need to know what version of kernel you are running (for me it was either PAE or non-PAE). So I ran uname -a to see what version of the kernel that machine was running.

rpm -Uhv SymSnap/symbdsnap-1.0.1-36146.2.6.18_8.el5PAE.i686.rpm
rpm -Uhv symmount-1.0.1-36352.i686.rpm
rpm -Uhv besr-1.0.1-36352.i686.rpm

Once all of these packages are installed I was able to do my first backup. And by first backup I mean first three (/boot, /, and SWAP). While I probably don't need to do all three (/boot and /) I don't want to have to create a SWAP system after a restore. So I ran all three.

besr -b/dev/sda1 -d /Storage/backups/hostname/boot.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol00 -d /Storage/backups/hostname/Vol00Log00.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard
besr -b /dev/mapper/VolGroup00-LogVol01 -d /Storage/backups/hostname/Vol00Log01.v2i -use-aes-encryption standard -p SoM3SupErSecureP@ssW0rd -compress Standard

Now in /Storage/backups/hostname I have three files
boot.v2i
Vol00Log00.v2i
Vol00Log01.v2i

If I needed to restore these I would use the Backup Exec System Recovery Windows based LiveCD. During the restore I would choose all three of these in the order shown above.

After the restore process was finished I need to boot using the gparted LiveCD. Once in I would open up a terminal and install grub on to my sda hard drive.

#grub
#root (hd0,0)
#setup (hd0)
#quit

Once grub is installed on sda I needed to update fstab to no longer look for / on the /dev/mapper/VolGroup00-LogVol00 and swap to not be on /dev/mapper/VolGroup00-LogVol01.

Backup Exec System Recovery Review

I asked a question over on Serverfault about backup software that supported Windows and Linux and did Bare-Metal Recovery and the response I got was to try Backup Exec System Recovery, which so far has been pretty good.

The bad stuff:

1) Restore CD creation on CentOS is impossible from what I have tried and can tell. They really do mean Redhat Enterprise Linux only.
2) Bare-metal recovery with MBR restore doesn't work. Which means I have to install grub after a restore.
3) LVM support is once again not there or well hidden.
4) Centralized management. It is a separate 1GB download as an Addon. I did the download and install, but now I can't find the Management console. Setting up each of our servers isn't too big of a problem, but I really wanted one window to view for backups.
5) There is no built in scheduler with Linux.
6) There is no incremental backup in Linux.

The good stuff:
1) It works on CentOS with a little bit of hacker-y to install (more on that in another post).
2) It is easy to install and configure on Windows, even though it requires a reboot. (BOO!)
3) It is pretty quick to get a backup started.
4) The backup procedure is different on Windows than on Linux. Windows is GUI based 100%. (I haven't looked to see if I can configure a backup from the command line.) While Linux is 100% CLI.
5) The Windows based restore CD is a pretty useful tool with out limits (unlike the Acronis Boot Disk).
6)Restores are quick and pretty simple to do.

All in all after trying three different products and looking at half a dozen I think this may be the winner.

CHECK_NRPE: Error - Could not complete SSL handshake

Recently I have been auditing our servers versus what we check in nagios versus what we need to do when a system is rebooted during routine maintenance. I found that two of our servers have been left out of nagios monitoring even though they are on my maintenance checklist, and they have software that I still start by hand (I KNOW IT'S NOT A BEST PRACTICE!).

Both servers had NRPE installed and configured (mostly), one of them even had a configuration file on the nagios server but it wasn't enabled (hostname.disabled instead of hostname.cfg). The other server needed a configuration file, but even then it wasn't working, so here are my troubleshooting steps:

1) Check to see that NRPE is compiled and installed. [It was]
2) Check that NRPE was listening (netstat -an | grep 5666) [It was]
3) Check that NRPE was listed in /etc/services [It wasn't]
4) Check the NRPE config file (/etc/xinetd.d/nrpe) for "only_from = 127.0.0.1 192.168.100.31" [It wasn't]

So I added the service definition to /etc/services and the nagios server IP to the only from line and restarted xinetd (service xinetd restart) and I was finally able to connect from my nagios server.

All of this is on CentOS 5.5 for both the server and the client.

Spiceworks Update 4.7.51900 Now Available

Spiceworks just released an update to their fantastic software and the patch notes have something I have been looking forward to for a while, yet it is really simple, LOG ROTATION!

Spiceworks 4.7.51900 is now available. It includes the following changes:

• Application startup performance improvement
• If a device is HP and scannable, and product number comes back as "Not Found", the message "All your warranties will be scanned soon" is displaying - fixed
• Power Manager and Performance Monitor plugins cannot coexist because of scheduler conflict - fixed
• Daily rotation of log files to reduce performance impact from large log files
• Server freezes after permission denied error in log - fixed
• Query inventory devices to locate IPs for VLAN data
• Undo action link keeps getting added when you close/reopen a ticket on the portal - fixed

Full Patch Notes

Acronis Backup and Recovery Advanced Server 10

Getting up and running with Acronis Backup and Recovery Advanced Server 10 is pretty quick, especially for the demo. It took under an hour to install, configure and start backing up a Windows 2008 Standard server. Getting the Backup agent installed on our Linux servers took a little bit more work because of the SnapAPI kernel modules. After a little bit of digging around in the Acronis knowledge base I was able to resolve all of the issues I had getting the Acronis Backup Agent installed on our CentOS 5.4 servers. If we go with Acronis Backup and Recovery Advanced Server 10 I will need to add some additional lines to our Linux Post-Install script to add in the kernel-devel package and the additional RPMs that the agent needs. (DKMS and SnapAPI) both of which are already on our Storage drive.

With our second trial run here we where able to resolve most of the outstanding issues we had from the first trial. Namely excessive recovery times and recovering to dis-similar hardware. I have yet to truly attempt a Linux restore to dis-similar hardware, but I have the base system recovered and the instructions, so I will be attempting one later this week (with results to follow). Restoring our Windows machines to either dis-similar hardware or a VM is pretty straight forward with the Universal Restore CD. The main problem holding us back in those scenarios is not having the drivers readily available for post install. This can be remedied by always going to VM and installing the VM tools as they also contain the drivers for the system.

Each backup policy allows us to modify settings for the backups such as Encryption, Compression, and throttling resources. The encryption can have a separate encryption key and varying levels of encryption from none to AES-128, AES-192, and finally AES-256. The automated backups can do a simple backup plan with full and incremental that are stored as a file on the storage system. The policy also allows us to run custom commands before and after the backup so can could do a virus scan, or shred temp files or what ever we desired.

One of the strangest problems I came across was while doing the restores to VM, if started from the VMs Console after picking the Disk to recover to the process could take hours to complete. Whereas if I did the same restore on to physical hardware the process would take seconds. The work around is to boot the VM from the Universal Restore CD and use the Management Console to connect to the VM. Once connected to the VM from the Management console I was able to start the restore in a matter of minutes, more akin to a physical machine.

Something to remember when setting up the backup policy for Linux machines that utilize LVM is to back up the disk as a whole. You don't want to backup the LVM by its self because after the restore it will fail to boot.

All in all I feel that Acronis would make a wonderful addition to our eco-system. It fulfills all of the requirements of our backup scheme, except the web based access, which is not uncommon for Windows based software.

NDO2DB daemon startup script

I would like to thank Chris over at http://sysengineers.wordpress.com for the excellent post on how to daemonize NDO2DB. His post NDO2DB startup script for RH (EL) / OEL does an excellent job getting everything working. I had to make one small change to the script as I kept my ndo2db executable as ndo2db-3x instead of just ndo2db.

For me this fixes a problem where I have to remember to manually start the ndo2db service after a server restart, which hasn't been happening the past few times. The second way I am going to fix this problem is by creating a checklist of things that need to happen during a server restart / boot up sequence for each PC. This will also reduce the effort needed for when I move the rack later this year. The third way I will be checking this is by adding a check to Nagios to make sure this is running at all times.

This also illustrates my general way of checking / double checking things. One go off of a checklist. The double check is to verify that Nagios is seeing the same thing.

Helpful Tech Support!

I don't often gush about good experiences, but I recently had an issue with Spiceworks, which ended up being a configuration problem on my end. Not only did the Support Engineer help me with my issue but they also saw another configuration issue and made some recommendations on how to fix it. Additionally they told me about a bug in the software that I was not aware of that has been affecting the performance!

Also the support rep was able to do all of this and not make me feel like an idiot!

TL;DR
Had a problem. It was fixed. Also fixed two other issues I didn't know I had!