What is normal for a server? In support we get that question from time to time. The problem is that normal varies from server to server. A load average of 200 is probably not normal but a load of 5 to 10 very well could be normal, depending on the server's application. What to do?
Baselining to the rescue. The idea behind baselining is to get performance numbers on your application when things are "normal" so that you have solid math to indicate when things are not "normal".
What makes a good baseline? Things like RAM use (overall, per process, rate of change), number and types of processes running, processor usage, disk usage (total, per app), disk speed and network utilization are all good OS metrics. You can also get metrics from your application. E-mails per hour, web page generation time, and number of users logged in are good to know.
You can capture OS metrics using tools like top, free, ps and iostat on Linux. Actually if you have iostat you probably have 'sar' which is great for performance history. Sar has a process that runs every few minutes and records various OS counters including processor info, RAM use, disk I/O and the like.
For the Windows people you have Task Manager and Performance Monitor. Task Manager is pretty simple and gives mostly an overview. Perfmon is really where its at on Windows. Using PerfMon you can track dozens of performance counters on disk, proc, memory, the network and even application specific metrics if you are running apps like MS Exchange that support them.
As with most tasks related to being the lord and master of a server, performance monitoring isn't a one time thing. As you make changes to the system you have to run new baselines. Between changes you should run your performance routines periodically to see how things are changing. It is much easier to look into an issue if you spot it earlier rather than later.
Go forth and make sure all your baselines are belong to you!
*bonus cool points for those who knew the title of this blog was also the title of a "Roswell" episode.