Documentation, not always the How To Dos

Documentation for me is often just How to do something, but I have been forgetting the WHY part of the equation. For instance, my documentation says to do our weekly maintenance window after 21:30 and to only reboot one particular server after 21:45, but I didn't say why and had forgotten myself. So I started doing the maintenance earlier and at one point rebooted the one particular server at 21:30. This caused the last of our production cron jobs to not run, and thus a customer didn't get their batch for the day.

Now this could have been avoided a few different ways. Since that first time was a mistake any thing that happens after it is a failure on my part, and thus can't happen again.

1. I could have followed our procedure to the T.
2. Read my e-mail to see that the cron job hadn't run yet.
3. Run the cron job by hand after the server restart.
4. Did all of the precursor work and waited for the cron job to run.
5. Did all of the precursor work and run the cron job by hand.

I have chosen to go with option 1, follow our procedure to the T. We started doing the weekly maintenance for a few reasons and it was to be done after 9:30, with that one particular server last to ensure this didn't happen. Now once again I have my WHY, and I have it written down so if I question it again I know why.


This is just one instance where the WHY is critical, but there are others. So please when you are writing documentation, and you should be please include the WHY you do something the way you do it. It also helps train the new guy, or your replacement.

Nagios: check_http, using the --invert-regex option

Some times you want to check that something is running or working correctly and you work out tests for that. Other times you want to know when something is broken and throwing error messages. This is about the latter, a proper HTTP 200 code is great and all, but what if the page is just showing "Too Many Connections" instead of your home page? My old check_http command for this server used to look like, well, check_http. I didn't check anything about it specifically, just that it was returning a 200 code.

Today however I knew I needed something more in depth. Our database server lost its local network connection, but still was available over the public IP, which is what I test against. Once we re-directed the SQL requests to the public IP address of the server everything started working again, until we ran across "Too Many Connections". The database server kept all of the "local" connections open and thus we ate up the rest.

So, how to test for this scenario? After reading through the man pages of check_http I saw this little gem "--invert-regex Return CRITICAL if found, OK if not". This I knew was exactly what I was looking for! If it sees our error codes it will go Critical! Now to put this gem into practice. Here is where the man pages fall short. There is no explanation on HOW to use this, just that it exists. I tried the obvious to me "check_http -H hostname.com -w 3 -c 5 --invert-regex 'Some string'", but that didn't work. OK, lets try "check_http -H hostname.com -w 3 -c 5 --invert-regex='Some string'" nope that errored out with " option `--invert-regex' doesn't allow an argument".

Third times the charm right?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string' --invert-regex '"

# HTTP OK HTTP/1.1 200 OK - 0.355 second response time |time=0.354966s;3.000000;5.000000;0.000000 size=12975B;;;0

Yes, as it turns out third time is the charm. So that got me thinking some more. How can I ensure that the page is rendering correctly, and if it isn't fail but in a specific way?

"check_http -H hostname.com -w 3 -c 5 -r 'Some string I want in my page' -r 'Some string I don't want to see' --invert-regex '"

You can add more than one -r to the check_http command and it will require all of them to be present for the test to pass, and if one of them fails then it will go critical! Perfect!

If you have any more insight into using the check_http command in Nagios I want to hear about it. We are always running into new failure scenarios that we didn't anticipate and I want to know about them before one of my users tells me about it.

How to: Use chkconfig, or keeping your services running on a new run level

OK, so I didn't take my own advice and made sure that all of my services where set to the run on the correct run levels before I switched run levels and rebooted the servers.

chkconfig is a command-line tool for updating the /etc/rc[0-6].d directories.

So I ran chkconfig --list | more to see what was running and in what run levels. Next I ran chkconfig --levels 235 service on to tell the service to start when it enters that run level. For example:

chkconfig --levels 235 named on

Now since I have already rebooted the server, and I know this service didn't start on it's own I had to:

service named start

If I had been smarter and done all of this before hand but wanted to see if named was running or not I would have run:

service named status

I understand that all of this is basics, but some people learn from others mistakes, so hopefully I can save one person from doing the same thing.

Other useful chkconfig switches:

chkconfig --help (Used to display the help dialog)

chkconfig --add [Service Name] (chkconfig --add mysqld) (Adds a service to the chkconfig list)

chkconfig --del [Service Name] (chkconfig --del mysqld) (Deletes a service from the chkconfig list)

chkconfig --level [2,3,4,5][Service Name][on, off, reset] (chkconfig --levels 235 httpd on)(Sets the run levels a service should start in.)

chkconfig can also manage xinetd scripts via /etc/xinetd.d.

Oh, and while you are at it run chkconfig --list | more to review what services are running on your server, you might be suprised. For instance I had bluetooth support running, but not one computer in my company has bluetooth support, so I disabled it (chkconfig --levels 2345 off) and (service bluetooth stop).

Failed to send SMTP (2AFC) for CA BrightStor ARCServe Back

This error was getting thrown after every backup my system does. BrightStor used to send me emails when it finished a backup job but at some point in time it started failing.

The simple fix for this is to add in a hosts entry for the SMTP server. Even though I specified an IP address for my SMTP server this fix worked.

To add in a hosts record open Notepad or whatever text editor you love and then open the file "C:\Windows\System32\Drivers\Etc\hosts". Notice there is no extension on this file.

If you have your \Windows directory located on a partition other than "C:" use "%windir%\System32\Drivers\Etc\hosts".

How to: Recreate Outlook Web Access Virtual Directory in Exchange 2007

Last week after another round of Microsoft update I rebooted our Microsoft Exchange 2007 server. After the reboot Outlook Web Access was no longer working. I could see in Event Viewer a bunch of errors relating to .Net 1.1 and a few other DLL files relating to authentication.

After a lot of looking around online I figured out a few things.
You cannot install .Net 1.1 on the OWA server. OWA requires IIS to be running in 64-bit mode (x64), while .Net requires 32-bit mode (x86). So I un-installed the .Net framework since I didn't need it any how and rebooted.

Things still where not working, so I tried to delete the Virtual Directory right from the IIS admin screen. That didn't work so I restored the Virtual Directory from a back up I had made right before (Trust me this is always a good idea!).

I also found out that with good examples and the proper references that the new PowerShell interface for Exchange is very powerful. However I am still looking forward to Exchange 2007 SP1 when Microsoft brings some things missing from the GUI back.

Today I found the proper way to recreate the Virtual Directory.
Step one is to view all of the Virtual Directories on the server.
Get-OwaVirtualDirectory will display a list of all the Exchange Virtual Directories on that server. The one you are looking for here is "OWA (Default Web Site)" on a default install.

Step two is to remove the default OWA Virtual Directory with:
Remove-OwaVirtualDirectory -identity "owa (Default Web Site)"

The next step is to recreate the OWA Virtual Directory with:
New-OWAVirtualDirectory -OWAVersion "Exchange2007" -Name "owa (Default Web Site)"

Now go into the Exchange 2007 Management Console and setup the correct internal and external URLs and any other settings you may have changed the first time around. One of the changes I recommend is on the "Authentication" tab the Logon Format for Forms based authentication to "User name only" and then setting the domain. This will make it easier for your users to login by not having to remember the domain or typing domain\username.

Microsoft Exchange TCP/IP Ports List

This is just going to be a list of all the TCP & UDP IP ports that you could possibly need to use (or open in your firewall(s)).

TCP port 25 - SMTP
TCP port 26 - SSL secured SMTP
TCP port 110 - Post Office Protocol v. 3 (POP3)
TCP port 995 - SSL secured POP3
TCP port 143 - Internet Message Access Protocol v. 4 (IMAP4)
TCP port 993 - SSL secured IMAP4
TCP port 80 - HTTP for Outlook Web Access
TCP port 443 - SSL secured HTTP for Outlook Web Access (HTTPS)
TCP/UDP port 389 - Lightweight Directory Access Protocol (LDAP)
TCP port 3268 - LDAP communications with an Active Directory Global Catalog Server
TCP port 119 - Network News Transfer Protocol (NNTP)
TCP port 563 - SSL secured NNTP
TCP port 135 - Remote Procedure Protocol (RPC) however it will also use ports 1024 and up as needed.
TCP/UDP port 53 - Domain Name System (DNS)
TCP port 102 - Mail Transfer Agent (MTA) *Only used in X.400 connections.
TCP/UDP port 88 - KERBEROS
Additionally if you are secureing any servers such as front-end / back-end servers with IPSEC you will need to allow IP protocol 51 for the Authenitcation Header, IP protocol 50, TCP/UDP port 88 for KERBEROS, and UDP port 500 for Key Exchanges.

That's a pretty decent sized list "JUST" for one network service. On the plus side most of those are not needed in a default setup as POP3 and IMAP4 are rarely used in a pure Microsoft network.

Also hopefully this will help me remember some of the less used ports and let me ace my test!

-EDIT: Thanks to the commenter I updated this post to have the correct Global Catalog server port.

How to Use Nslookup to Verify MX record configuration

As I have been installing, configuring, and testing our Microsoft Exchange Server 2007 I have had to test the DNS system to ensure our domain is registared and working properly. This is something that all system administrators should know how to do, but the exact syntax is often forgotten.

1. Open a Command Prompt (Start, Run, cmd, [Enter])
2. Type nslookup.
3. Server (The DNS server that you want to use. This is optional.)
4. Set q=mx (This sets the query type to "MX" {Mail Exchanger}.)
5. (The domain name you want to lookup the mail exchanger to.)

This operation can be done on Windows XP, Windows Server 2003, and Windows Vista. There are similar options if you are doing this from Linux.

1. Open up a terminal (Differs by distribution)
2. Type nslookup.
3. Set type=mx (This sets the query type to "MX" {Mail Exchanger}.)
4. (The domain name you want to lookup the mail exchanger to.)

MC MCSE: A resource for aspiring MCSEs

I just found MC MCSE yesterday while looking for tips to pass my up and coming 70-284 exam. They offer:
Free Practice Exams - 44 free practice tests with over 4,300 total questions.
Free Study Guides - 110 study guides and tutorials to help you pass your exams.
Find Training Materials - Find the best books and training products for your studies.
Free Tech Magazines - Free subscriptions to ComputerWorld, eWeek and many more.
I took my first free test yesterday and got a 76% on it, passing, but not excellent. I find that most of the questions are well written but there where a few that needed some work. (Much like this blog I know!)

ESEUTIL for Microsoft Exchange, Part 1 The Basics

ESEutil (Extensible Storage Engine utility) is the sister EXE to NTDSutil (New Technology Directory Service I believe). There are three sides to this utility that any Microsoft Exchange administrator should know how to use. The harmless checking options, the hairier defragmentation option, and finally the scariest restore options.

On a side note it should be mentioned to any new Microsoft Exchange administrators that ESEUTIL is not a part of the PATH system variable, and as such must be run from the Exchange/bin folder. If you find yourself using ESEUTIL frequently I would suggest adding it to your path as I have.

First there is the mostly harmless options /k, /mh, and /cc. These flags will do various things that Microsoft Exchange will normally do for you, such as re-run log files after you remount a store. The /mh flag will check the store to see if it was properly shut down and along the way will tell you when the last time it was backed up. This is an easy way to start learning how to use ESEUTIL properly.

Example: “eseutil /mh "d:\program files\exchsrvr\mdbdata\priv1.edb" (Assuming Exchange 2003 is installed on the d:\ drive.)”

This command should return “State: Clean Shutdown” within its output. Also you can use /mh to see how many /r commands have been run against it, with anything greater than zero being your number.

There are other simple flags to use such as /ml to check log files, /mm to dump the metadata (not truly useful to most administrators, but interesting to see none the less.), /mk to see information about the checkpoint file (.chk files).

ESEUTIL /k will run a similar function to running a checksum against a file to verify its integrity. It will only VERIFY the database, not repair as that is what /p and /r are for. This is most often used when Microsoft Exchange has been shut down improperly. On another side note it worth mentioning that hundreds of uninitialized pages are normal, but bad checksums or wrong page numbers are bad. Also you can verify log files with /k in the same way you verify stores.

Example: “eseutil /k "d: \program files\exchsrvr\mdbdata\priv1.edb" “

Just as easy to run is ESEUTIL with the /d flag. This will defragment the .edb file and recover lost disk space. This is much akin to running jetpack against the DHCP.mdb file to defragment the DHCP database in Microsoft Windows Server 2003 and not so much like defragmenting you hard drive with Windows built in defragmenter. You must dismount the store in order to run eseutil /d, but you do not need to stop the Information Store service to do so.

The last function of ESEUTIL, and by far the scariest function set is the restore options. The /r and /p flags can really muck up a Microsoft Exchange store quickly. These options will require their own blog entry so stay tuned for more.

A quick over view of all the options for Microsoft Exchange’s ESEUTIL.
Eseutil /cc Performs a hard recovery after a database restore.
Eseutil /d Performs an offline compaction of a database.
Eseutil /g Verifies the integrity of a database.
Eseutil /k Verifies the checksums of a database.
Eseutil /m Generates formatted output of various database file types. e.g. /mh
Eseutil /p Repairs a corrupted or damaged database.
Eseutil /r Performs soft recovery to bring a single database into a consistent or clean shutdown state.
Eseutil /y Copies a database, streaming file, or log file.

A simple way to Secure Microsoft Exchange 2003

One of the easiest ways to secure Microsoft Exchange 2003 is to not run Microsoft Exchange 2003 on an Active Directory Domain controller. This way IF your Microsoft Exchange 2003 Server is compramised you will not have to worry about your Active Directory data being compramised as well.

Small Business Server users should take note of this as the default install of Small Business Server has Microsoft Exchange 2003 Server running on your Active Directory Domain Controller with Microsoft SQL Server running as well. Also more than likely this same server is also running your Active Directory DNS server, DHCP server, WINS server and in some cases even your public Web Site running on IIS and an FTP server also running on IIS.

Implementing OpenDNS to enhance Enterprise DNS Resolution

We use OpenDNS as our forwarders here for a few reasons. One they are much faster at what they do than our ISP. Two they don’t drop requests for resolution like our ISP does. Three it is much faster than using Root Hints to resolve and cache manually. Four by combining resolution requests from around the country (globe soon) we can all get better resolution.

Setting this up for an Active Directory enterprise is easy. For each of your DNS servers of which you should have at least two, open up the DNS Management Console, right click on the server and choose Properties. Go to the Forwarders tab and check the box for “Enable Forwarders”. Now add in two IP address for the OpenDNS servers. The IP addresses that you want are 208.67.222.222 and 208.67.220.220. Now whenever your server has a resolution request that it cannot fulfill it will look to the OpenDNS servers instead of using root hints to resolve the query.