sar is an acronym for System Activity Report. It takes a snapshot of the system periodically. On most distributions it comes with the sysstat package. On Redhat and derived distributions the package will install a set of cron jobs in /etc/cron.d/sysstat. There are two cron jobs to take note of. The first one runs every ten minutes as root. It runs the script /usr/lib/sa/sa1 -S Disk 1 1. This script saves its output in report files. The files are written to /var/log/sa/sar[dd], where [dd] is the two digit date for today's date. (e.g. Today is 8/26/2011. The log file is /var/log/sa/sar26)
The second cron job runs at 23:53. This cron job summarizes the days activity. Both of the reports are saved as binary data, so normal tools are useless here.
There are many flags to use with sar when running it interactivly. Some of the flags have additional atguments that are required when used. One example of thses flags is -n which al requires an additional argement suach as DEV or NFS ro IP. This specific example has eightteen (18) potential arguments, not including ALL.
With sar liberal use of the man pages are highly suggested. Not only are the flags and any arguments documented, but the headers for each one and what they represent are explained as well. This comes in handy if you get overzealous with flags and aren't quite sure what you're looking at.
I'm only going to cover some of the most notable flags, what they show and their headers. Be careful however as some flags exist in both upper and lower case and report vastly different metrics. One example of the is -b which reports on I/O transfer but -B reports on paging stats.
First off the plate is -b which as I've already stated reports on I/O transfer stats and has the following headers:
tps: Transfers per second to a physical device.
rtps: Read transfers per second to a physical device.
wtps: Write transfers per second to a physical device.
bread/s: Blocks (since kernel 2.4 = sectors = 512 bytes) read from devices per second.
bwrtn/s: Blocks written to devices per second.
Example output from a production server running: sar -b 1 1
Linux 2.6.18-274 el5PAE (server.domain.com) 8/26/2011
09:23:57 PM tps rtps wtps bread/s bwrtn/s
09:23:58 PM 7829.00 133.00 7696.00 1528.00 76960.00
Average: 7829.00 133.00 7696.00 1528.00 76960.00
-B will report paging stats. Some metrics / headers are only available in kernels 2.5 and newer.
pgpgin/s: Kilobytes paged in from disk per second.
pgpgout/s: Kilobytes paged out from disk per second.
magflt/s: Major faults per second (hits to disk, this is a bad thing...)
-c Process creation stats.
proc/s: Processes created per second.
-d Activity for each block device.
tps: Transfers per second
rd_sec/s: Sectors (512 bytes) read from block device per second.
wr_sec/s: Sectors (512 bytes) written to block device per second.
avgrq-sz: Average # of sectors
avgqu-sz: Average queue length
await: Average time in milliseconds for queue + servicing request.
svctm: Average servicing time.
%util: CPU percentage while I/O requests where issued. Close to 100% = device saturation.
-n DEV Network interface stats.
IFACE: Interface Name
rxpck/s: Packets received per second.
tcpck/s: Packets sent per second.
rxbyt/s: Bytes received per second.
txbyt/s: Bytes sent per second.
rxcmp/s: Compressed packets received per second.
txcmp/s: Compressed packets sent per second.
rxmcst/s: Multicast packets received per second.
-P ALL Per processor (or core) stats
-p Print pretty device names
Shows block devices as sda instead of dev8-0. Has no effect on Network device names.
-A same as: -bBcdqrRuvwWy -I SUM -I XALL -n ALL -P ALL
After you install the sysstat package you really need to let it run for a while and gather stats to see the real beauty of it all. However you can run it interactively if required. When running sar interactively the syntax is sar -FLAGS Interval Duration. (e.g. sar -b 2 60) runs I/O stats every two seconds for a minute. This is very hand to run if you're troubleshooting a slow system or watching it under load.