Adding a vCPU to a Windows 2008 R2 guest

Last night I had to add an additional vCPU to a Windows 2008 R2 guest. I tried and tried to research doing this, but all I was able to find was how to Hot add a vCPU, which I didn't care to do. Also all of the VMWare documentation said to view a certain PDF to see what OSes could even take a Hot CPU addition, but I was unable to find anywhere in the PDF what OSes could handle it and which ones couldn't.

So I installed the newest patches (being patch Tuesday and all) and I shut down the server. I then went from one vCPU to two vCPUs. I ignored the warning that adding CPUs to an already installed system may cause it to be unstable.

So far the server has been stable and everything worked like I had hoped. I don't know if this would work on any of our Windows 2008 servers, or for that matter the CentOS 5.5 servers, but all of them already have two vCPUs.

If you are wondering why I needed to do this, well, it is because our VoIP recording solution uses MySQL, and for some reason it has been using 100% of the CPU for weeks now. This despite a mostly empty process list in MySQL.

Strange Nagios Error Solved

This morning I added some new services to a server, but they wouldn't move out of "Pending" status. The error I received was
"Feb 9 09:27:36 nagios: Warning: Check result queue contained results for service '' on host '', but the service could not be found! Perhaps you forgot to define the service in your config files?"

I stopped the nagios service and ran ps -ef | grep nagios. To my surprise there was still a Nagios instance running. This means two things. First the init script that comes with Nagios is borked and doesn't correctly check for running nagios instances. Second, I somehow started a second Nagios instance.

I thought something was up in the first place because every other refresh or so of the Nagios web view I would either see the three pending services or I wouldn't. This was my first clue that something was borked. I then went to tail /var/log/messages and saw the error message. Then I started investigating the issue with the help of Google. Once I saw that there where two instances of Nagios things started to make sense.

I killed the second Nagios instance and any children processes (in my case ndo2db) and then restarted nagios via the init script.

Once I had everything up and running (but only one instance) I was able to successfully check my new services.

ImageMagick convert -geometry weirdness

I don't know if this is really documented anywhere, but when using the ImageMagick program convert with the -geometry or -resize flags it also converts to 16bit color, at least when used against 1bit (Black and White) image.

The reason this is important to note is that using -compress Group4 will not work on these images because they aren't 1bit color images and they need to be. To work around this limitation you need to force the color depth with -monochrome. The downside to all of this is time.

Ripping an Image file from a PDF takes roughly .206 seconds for an 8.5x11 PDF with pdfimages (useful if a PDF has OCR embedded in it.)
Converting the resultant PBM file without -geometry 1700x2200! -monochrome takes roughly .194 seconds.
Converting the resultant PBM file with -geometry 1700x2200! -monochrome takes roughly 4.594 seconds. This is an increase of 10 - 10.5 % adding in the geometry.

*all of the above numbers are from one file, but testing of different files showed similar results. All tests where done at 200 DPI.

Here are some examples of the commands we are running for anyone curious.
convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 {outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 200 -compress Group4 -geometry 1700x2200! -monochrome ${outputTiffRoot}_200.tiff

convert ${outputTiffRoot}-000.p*m -density 300 -compress Group4 -geometry 2550x3300! -monochrome ${outputTiffRoot}_300.tiff

Gawker Media Account Database hack

I'm sure you have heard about the release of about 1.5 million username / password combinations (encrypted with DES). I have gotten e-mails from a few web companies saying that I should re-set my password. These companies include LinkedIn and Blizzard (for my World of Warcraft account).

Since my roommate didn't get an e-mail from Blizzard (as he doesn't have an account on any Gawker Media website) and I did, I can only assume that Blizzard downloaded the hacked account database and compared it to their account database. Any matches where to get this e-mail.

I honestly hope this is what happened and that Blizzard and LinkedIn didn't just randomly send out password reset e-mails. In this case the most responsible thing to do is download the file and cross reference it with your own data.

If you are wondering where you affected by this breach visit http://www.didigetgawkered.com/.

Yesterday I read both an Analysis of the hack and an Analysis of the Analysis. I am fairly certain that my password was unique in the database, but I do know that I used to use that password all over the place. A few months ago I started changing password and using KeePass to store them securely. I disagree with Wikidsystem's Analysis of the Analysis.

Yes, I was just as owned as the person using something insecure such as letmein or password. But the "owning" wasn't because of a weak password. Yes, I do have to copy and paste my passwords, but that doesn't make me a loser. I treat all of my online identities the same, as they are a representation of ME. The real losers are the ones that use any sort of username / password combinations on multiple sites. If you don't care that some sites might get hacked with the same username / password then fine, but I do. I want any potential fallout to be minimal.

Also, for things I truly care about, if a two-factor authentication mechanism is available I use it. The other thing more developers need to account for is LONGER passwords. My pseudo-random password generator generates long passwords, sometimes too long for an account. Please make the password field huge and don't store it in plain text. I hate having to cut down a password from 30+ characters to 8 because that is the longest your application will allow.

You see I don't care if my password is 30+ characters, because I don't need to remember it! Thats what I have KeePass for!

Enabling Promiscuous Mode on vSphere 4

We are in desperate need of a testing server. The testing server needs to be able to listen to the network, through port mirroring, to record VoIP calls. In the physical world this is no problem, but we don't have the budget for another physical box, especially for something as non-critical as testing a program upgrade (even though the program it self is very critical to our business).

A year ago the software vendor said that they don't support virtual environments and their product doesn't work in a virtual environment. During discussions this year however they told the truth that they simply don't know if it will work, and thus "don't support" it. I am calling it lazy. It doesn't take much these days to get a VM host going and configured for testing things out.

Well we have done the leg work for them. I have confirmed that vSphere does allow for promiscuous mode, even from the physical network. To be honest I was a little shocked to see this worked.

Here are the steps to enable promiscuous mode on your vSphere host and the guest VM.

1. Enable promiscuous mode on the virtual adapter.
   
2. Enable promiscuous mode on the vSwitch.
3. Enable promiscuous mode for the guest.
4. Enable port mirroring on the physical switch (not covered here).
5. Test capturing network data.

Now that we have the general synopsis of the procedure we can begin.

We enable promiscuous mode on the virtual adapter by logging into the vSphere Client, going to the VM Host, clicking on the "Configuration" tab, then on "Networking" in the "Hardware" section on the left. Now click on "Properties..." for the Virtual Switch.


Now enabling promiscuous mode on the vSwitch is pretty simple. Click on the "vSwitch" on the "Ports" tab, as shown below and then click on the "Edit..." button.



Now that we are editing the vSwitch properties click on the "Security" tab. Change the option for Promiscuous mode to "Accept" if it isn't already and hit "OK".


To enable promiscuous mode for the guest we need to break down to the command line. I used putty to SSH into my vSphere host, which I had previously setup. Now you need to edit the .vmx file of the guest that will be listening to the network.

# vi /vmfs/volumes/datastore1/testServer/testServer.vmx

I did a search for ethernet, so the promiscuous mode configuration would be with the rest of the ethernet config. Add the following line to the configuration file:

ethernet0.noPromisc = "FALSE"



And save when you are done. I rebooted my testServer just as a precaution, but I'm not certain it is required.

At this point everything is configured on the VM side of things. Make sure you have port mirroring enabled on your physical switch and give it a test. In my environment I commonly use Wireshark. I did my testing by pinging a server on a mirrored port.

Using PSExec to Defragment your PCs

Way back in 2000 / 2001 I was an IT intern. My job was to do the really manual processes for a small department with in a much larger company. One of those was monthly defrags of all PCs and Laptops (if they where available). I had to go to each and every PC, log on, and start the defrag process manually. I believe this to be my first moment of "I should automate this". Problem was I didn't know how or really where to start. I knew script was the answer, but just couldn't get things to work. I believe I eventually created a scheduled task on each machine (by hand) to do this for us.

Fast forward 10 years and I am still in need of the same thing, as we don't have Vista or Windows 7 deployed. But now I have a much better understanding of what needs to happen and even better I know where to start!

PSExec, which is part of the System Internals PsTools suite is my answer these days. A simple script, run from my PC (still manually for the time being) is able to handle defragmenting all of our PCs.

psexec @C:\Updates\Comps\AppPCs.txt -n 10 -c -f -d JkDefragCMD.exe

I use the files with PC names to speed up re-deployment of everything. The -n 10 flag tells psexec t0 wait 10 seconds before it times out the PC, instead of 60 (I believe this is the default). -c copies the file (jkdefragcmd.exe) to the remote system. The flag -f forces copying, even if the file exists. The flag -d doesn't wait for the process to terminate, this is as asynchronously as I can do.

My next step is to hook this up to task scheduler and have it run the first Sunday or something.

Documentation, not always the How To Dos

Documentation for me is often just How to do something, but I have been forgetting the WHY part of the equation. For instance, my documentation says to do our weekly maintenance window after 21:30 and to only reboot one particular server after 21:45, but I didn't say why and had forgotten myself. So I started doing the maintenance earlier and at one point rebooted the one particular server at 21:30. This caused the last of our production cron jobs to not run, and thus a customer didn't get their batch for the day.

Now this could have been avoided a few different ways. Since that first time was a mistake any thing that happens after it is a failure on my part, and thus can't happen again.

1. I could have followed our procedure to the T.
2. Read my e-mail to see that the cron job hadn't run yet.
3. Run the cron job by hand after the server restart.
4. Did all of the precursor work and waited for the cron job to run.
5. Did all of the precursor work and run the cron job by hand.

I have chosen to go with option 1, follow our procedure to the T. We started doing the weekly maintenance for a few reasons and it was to be done after 9:30, with that one particular server last to ensure this didn't happen. Now once again I have my WHY, and I have it written down so if I question it again I know why.


This is just one instance where the WHY is critical, but there are others. So please when you are writing documentation, and you should be please include the WHY you do something the way you do it. It also helps train the new guy, or your replacement.