The video below documents the installation of a new Cummins two-megawatt diesel generator for a pod in our DAL05 data center. You see the crane prepare for the work by installing counter-balance weights, and work starts with the team placing a utility transformer on its pad outside our generator yard. A truck pulls up with the generator base in tow, and you watch the base get positioned and lowered into place. The base looks so large because it also serves as the generator’s 4,000 gallon “belly” fuel tank. After the base is installed, the generator is trucked in, and it is delicately picked up, moved, lined up and lowered onto its base. The last step you see is the generator housing being installed over the generator to protect it from the elements. At this point, the actual “installation” is far from over — we need to hook everything up and test it — but those steps don’t involve the nostalgia-inducing heavy machinery you probably came to this post to see:

When we talk about the “megawatt” capacity of a generator, we’re talking about the bandwidth of power available for use when the generator is operating at full capacity. One megawatt is one million watts, so a two-megawatts generator could power 20,000 100-watt light bulbs at the same time. This power can be sustained for as long as the generator has fuel, and we have service level agreements to keep us at the front of the line to get more fuel when we need it. Here are a few other interesting use-cases that could be powered by a two-megawatt generator:

• 1,000 Average Homes During Mild Weather
• 400 Homes During Extreme Weather
• 20 Fast Food Restaurants
• 3 Large Retail Stores
• 2.5 Grocery Stores
• A SoftLayer Data Center Pod Full of Servers (Most Important Example!)

Every SoftLayer facility has an n+1 power architecture. If we need three generators to provide power for three data center pods in one location, we’ll install four. This additional capacity allows us to balance the load on generators when they’re in use, and we can take individual generators offline for maintenance without jeopardizing our ability to support the power load for all of the facility’s data center pods.

Those of you who are in the fondly remember Tonka trucks and CAT crane toys are the true target audience for this post, but even if you weren’t big into construction toys when you were growing up, you’ll probably still appreciate the work we put into safeguarding our facilities from a power perspective. You don’t often see the “outside the data center” work that goes into putting a new SoftLayer data center pod online, so I thought it’d give you a glimpse. Are there an topics from an operations or facilities perspectives that you also want to see?

-Robert

It seems silly to prepare for a failure in a security plan we spend time and effort creating, but if we stick our heads in the sand and tell ourselves that we’re secure, we won’t be prepared in the unlikely event of something happening. Every attempt to mitigate risks and stop threats in their tracks will be circumvented by the one failure, threat or disaster you didn’t cover in your risk management plan. When that happens, accurate event logging will help you record what happened, respond to the event (if it’s still in progress) and have the information available to properly safeguard against or prevent similar threats in the future.

Like any other facet of security, “event logging” can seem overwhelming and unforgiving if you’re looking at hundreds of types of events to log, each with dozens of variations and options. Like we did when we looked at securing servers, let’s focus our attention on a few key areas and build out what we need:

Which events should you log?
Look at your risk assessment and determine which systems are of the highest value or could cause the most trouble if interrupted. Those systems are likely to be what you prioritized when securing your servers, and they should also take precedence when it comes to event logging. You probably don’t have unlimited compute and storage resources, so you have to determine which types of events are most valuable for you and how long you should keep records of them — it’s critical to have your event logs on-hand when you need them, so logs should be retained online for a period of time and then backed up offline to be available for another period of time.

Your goal is to understand what’s happening on your servers and why it’s happening so you know how to respond. The most common audit-able events include successful and unsuccessful account log-on events, account management events, object access, policy change, privilege functions, process tracking and system events. The most conservative approach actually involves logging more information/events and keeping those logs for longer than you think you need. From there, you can evaluate your logs periodically to determine if the level of auditing/logging needs to be adjusted.

Where do you store the event logs?
Your event logs won’t do you any good if they are stored in a space that is insufficient for the amount of data you need to collect. I recommend centralizing your logs in a secure environment that is both readily available and scalable. In addition to the logs being accessible when the server(s) they are logging are inaccessible, aggregating and organize your logs in a central location can be a powerful tool to build reports and analyze trends. With that information, you’ll be able to more clearly see deviations from normal activity to catch attacks (or attempted attacks) in progress.

How do you protect your event logs?
Attacks can come from both inside and out. To avoid intentional malicious activity by insiders, separation of duties should be enforced when planning logging. Learn from The X Files and “Trust no one.” Someone who has been granted the ‘keys to your castle’ shouldn’t also be able to disable the castle’s security system or mess with the castle’s logs. Your network engineer shouldn’t have exclusive access to your router logs, and your sysadmin shouldn’t be the only one looking at your web server logs.

Keep consistent time.
Make sure all of your servers are using the same accurate time source. That way, all logs generated from those servers will share consistent time-stamps. Trying to diagnose an attack or incident is exceptionally more difficult if your web server’s clock isn’t synced with your database server’s clock or if they’re set to different time zones. You’re putting a lot of time and effort into logging events, so you’re shooting yourself in the foot if events across all of your servers don’t line up cleanly.

Read your logs!
Logs won’t do you any good if you’re not looking at them. Know the red flags to look for in each of your logs, and set aside time to look for those flags regularly. Several SoftLayer customers — like Tech Partner Papertrail — have come up with innovative and effective log management platforms that streamline the process of aggregating, searching and analyzing log files.

It’s important to reiterate that logging — like any other security endeavor — is not a ‘one size fits all’ model, but that shouldn’t discourage you from getting started. If you aren’t logging or you aren’t actively monitoring your logs, any step you take is a step forward, and each step is worth the effort.

Thanks for reading, and stay secure, my friends!

-Matthew

If you’ve configured an infrastructure to accommodate big data, you know how much of a pain it can be: You choose your hardware, you configure it to run NoSQL, you install an open source NoSQL project that you think will meet your needs, and you keep tweaking your environment to optimize its performance. Assuming you have the resources (and patience) to get everything running efficiently, you’ll wind up with the horizontally scalable database infrastructure you need to handle the volume of content you and your users create and consume. SoftLayer and 10gen are making that process a whole lot easier.

Our new MongoDB solutions take the time and guesswork out of configuring a big data environment. We give you an easy-to-use system for designing and ordering everything you need. You can start with a single server or roll out multiple servers in a single replica set across multiple data centers, and in under two hours, an optimized MongoDB environment is provisioned and ready to be used. I stress that it’s an “optimized” environment because that’s been our key focus. We collaborated with 10gen engineers on hardware and software configurations that provide the most robust performance for MongoDB, and we incorporated many of their MongoDB best practices. The resulting “engineered servers” are big data powerhouses:

From each engineered server base configuration, you can customize your MongoDB server to meet your application’s needs, and as you choose your upgrades from the base configuration, you’ll see the thresholds at which you should consider upgrading other components. As your data set’s size and the number of indexes in your database increase, you’ll need additional RAM, CPU, and storage resources, but you won’t need them in the same proportions — certain components become bottlenecks before others. Sure, you could upgrade all of the components in a given database server at the same rate, but if, say, you update everything when you only need to upgrade RAM, you’d be adding (and paying for) unnecessary CPU and storage capacity.

Using our new Solution Designer, it’s very easy to graphically design a complex multi-site replica set. Once you finalize your locations and server configurations, you’ll click “Order,” and our automated provisioning system will kick into high gear. It deploys your server hardware, installs CentOS (with OS optimizations to provide MongoDB performance enhancements), installs MongoDB, installs MMS (MongoDB Monitoring Service) and configures the network connection on each server to cluster it with the other servers in your environment. A process that may have taken days of work and months of tweaking is completed in less than four hours. And because everything is standardized and automated, you run much less risk of human error.

One of the other massive benefits of working so closely with 10gen is that we’ve been able to integrate 10gen’s MongoDB Cloud Subscriptions into our offering. Customers who opt for a MongoDB Cloud Subscription get additional MongoDB features (like SSL and SNMP support) and support direct from the MongoDB authority. As an added bonus, since the 10gen team has an intimate understanding of the SoftLayer environment, they’ll be able to provide even better support to SoftLayer customers!

You shouldn’t have to sacrifice agility for performance, and you shouldn’t have to sacrifice performance for agility. Most of the “big data” offerings in the market today are built on virtual servers that can be provisioned quickly but offer meager performance levels relative to running the same database on bare metal infrastructure. To get the performance benefits of dedicated hardware, many users have chosen to build, roll out and tweak their own configurations. With our MongoDB offering, you get the on-demand availability and flexibility of a cloud infrastructure with the raw power and full control of dedicated hardware.

If you’ve been toying with the idea of rolling out your own big data infrastructure, life just got a lot better for you.

-Duke

Every risk has different probabilities of occurrence, potential damages, and prevention costs, and the risk management process helps us balance the costs and benefits of various security methods. When it comes to securing a home, the most effective protection comes by using layers of different methods … To prevent a home invasion, you might lock your door, train your dog to make intruders into chew toys and have an alarm system installed. Even if an attacker can get a key to the house and bring some leftover steaks to appease the dog, the motion detectors for the alarm are going to have the police on their way quickly. (Or you could violate every HOA regulation known to man by digging a moat around the house, filling with sharks with laser beams attached to their heads, and building a medieval drawbridge over the moat.)

I use the example of securing a house because it’s usually a little more accessible than talking about “server security.” Server security doesn’t have to be overly complex or difficult to implement, but its stigma of complexity usually prevents systems administrators from incorporating even the simplest of security measures. Let’s take a look at the easiest steps to begin securing your servers in the context of their home security parallels, and you’ll see what I’m talking about.

Keep “Bad People” Out: Have secure password requirements.

Passwords are your keys and your locks — the controls you put into place that ensure that only the people who should have access get it. There’s no “catch all” method of keeping the bad people out of your systems, but employing a variety of authentication and identification measures can greatly enhance the security of your systems. A first line of defense for server security would be to set password complexity and minimum/maximum password age requirements.

If you want to add an additional layer of security at the authentication level, you can incorporate “Strong” or “Two-Factor” authentication. From there, you can learn about a dizzying array of authentication protocols (like TACACS+ and RADIUS) to centralize access control or you can use active directory groups to simplify the process of granting and/or restricting access to your systems. Each layer of authentication security has benefits and drawbacks, and most often, you’ll want to weigh the security risk against your need for ease-of-use and availability as you plan your implementation.

Stay Current on your “Good People”: When authorized users leave, make sure their access to your system leaves with them.

If your neighbor doesn’t return borrowed tools to your tool shed after you gave him a key when he was finishing his renovation, you need to take his key back when you tell him he can’t borrow any more. If you don’t, nothing is stopping him from walking over to the shed when you’re not looking and taking more (all?) of your tools. I know it seems like a silly example, but that kind of thing is a big oversight when it comes to server security.

Employees are granted access to perform their duties (the principle of least privilege), and when they no longer require access, the “keys to the castle” should be revoked. Auditing who has access to what (whether it be for your systems or for your applications) should be continual.

You might have processes in place to grant and remove access, but it’s also important to audit those privileges regularly to catch any breakdowns or oversights. The last thing you want is to have a disgruntled former employee wreak all sorts of havoc on your key systems, sell proprietary information or otherwise cost you revenue, fines, recovery efforts or lost reputation.

Catch Attackers: Monitor your systems closely and set up alerts if an intrusion is detected.

There is always a chance that bad people are going to keep looking for a way to get into your house. Maybe they’ll walk around the house to try and open the doors and windows you don’t use very often. Maybe they’ll ring the doorbell and if no lights turn on, they’ll break a window and get in that way.

You can never completely eliminate all risk. Security is a continual process, and eventually some determined, over-caffeinated hacker is going to find a way in. Thinking your security is impenetrable makes you vulnerable if by some stretch of the imagination, an attacker breaches your security (see: Trojan Horse). Continuous monitoring strategies can alert administrators if someone does things they shouldn’t be doing. Think of it as a motion detector in your house … “If someone gets in, I want to know where they are.” When you implement monitoring, logging and alerting, you will also be able to recover more quickly from security breaches because every file accessed will be documented.

Minimize the Damage: Lock down your system if it is breached.

A burglar smashes through your living room window, runs directly to your DVD collection, and takes your limited edition “Saved by the Bell” series box set. What can you do to prevent them from running back into the house to get the autographed posted of Alf off of your wall?

When you’re monitoring your servers and you get alerted to malicious activity, you’re already late to the game … The damage has already started, and you need to minimize it. In a home security environment, that might involve an ear-piercing alarm or filling the moat around your house even higher so the sharks get a better angle to aim their laser beams. File integrity monitors and IDS software can mitigate damage in a security breach by reverting files when checksums don’t match or stopping malicious behavior in its tracks.

These recommendations are only a few of the first-line layers of defense when it comes to server security. Even if you’re only able to incorporate one or two of these tips into your environment, you should. When you look at server security in terms of a journey rather than a destination, you can celebrate the progress you make and look forward to the next steps down the road.

Now if you’ll excuse me, I have to go to a meeting where I’m proposing moats, drawbridges, and sharks with laser beams on their heads to SamF for data center security … Wish me luck!

-Matthew

Just reading that you need to invest in “Risk Management” probably makes you shudder. Admittedly, when a business owner has to start quantifying and qualifying potential areas of business risk, the process can seem daunting and full of questions … “What kinds of risks should I be concerned with?” “Once I find a potential risk, should I mitigate it? Avoid it? Accept it?” “How much do I need to spend on risk management?”

When it comes to risk management in hosting, the biggest topics are information security, backups and disaster recovery. While those general topics are common, each business’s needs will differ greatly in each area. Because risk management isn’t a very “cookie-cutter” process, it’s intimidating. It’s important to understand that protecting your business from risks isn’t a destination … it’s a journey, and whatever you do, you’ll be better off than you were before you did it.

Because there’s not a “100% Complete” moment in the process of risk management, some people think it’s futile — a gross waste of time and resources. History would suggest that risk management can save companies millions of dollars, and that’s just when you look at failures. You don’t see headlines when businesses effectively protect themselves from attempted hacks or when sites automatically fail over to a new server after a hardware failure.

It’s unfortunate how often confidential customer data is unintentionally released by employees or breached by malicious attackers. Especially because those instances are often so easily preventable. When you understand the potential risks of your business’s confidential data in the hands of the wrong people (whether malicious attackers or careless employees), you’ll usually take action to avoid quantifiable losses like monetary fines and unquantifiable ones like the loss of your reputation.

More and more, regulations are being put in place to holding companies accountable for protecting their sensitive information. In the healthcare industry businesses have to meet the strict Health Insurance Portability and Accountability Act (HIPAA) regulations. Sites that accept credit card payments online are required to operate in Payment Card Industry (PCI) Compliance. Data centers will spend hours (and hours and hours) achieving and maintaining their SSAE 16 certification. These rules and requirements are not arbitrarily designed to be restrictive (though they can feel that way sometimes) … They are based on best practices to ultimately protect businesses in those industries from risks that are common throughout the respective industry.

Over the coming months, I’ll discuss ways that you as a SoftLayer customer can mitigate and manage your risk. We’ll talk about security and backup plans that will incrementally protect your business and your customers. While we won’t get to the destination of 100% risk-mitigated operations, we’ll get you walking down the path of continuous risk assessment, identification and mitigation.

Stay tuned!

-Matthew

The question everyone asks is why this kind of thing still happens. To a certain extent, the reality is that most entrepreneurs don’t know what they don’t know. I spoke with an woman who was going to be featured on BBC’s Dragons’ Den, and she said that the traffic from the show’s viewers crippled most (if not all) of the businesses that were presented on the program. She needed to safeguard from that happening to her site, and she didn’t know how to do that.

Fortunately, it’s pretty easy to keep sites and applications online with on-demand infrastructure and auto-scaling tools. Unfortunately, most business owners don’t know how easy it is, so they don’t take advantage of the resources available to them. Preparing a website, game or application for its own success doesn’t have to be expensive or time consuming. With pay-for-what-you-use pricing and “off the shelf” cloud management solutions, traffic-caused outages do NOT have to happen.

First impressions are extremely valuable, and if I wasn’t really interested in that conference or the new product Dutch Cowboys blogged about, I’d probably never go back to those sites. Most Internet visitors would not. I cringe to think about the potential customers lost.

Businesses spend a lot of time and energy on user experience and design, and they don’t think to devote the same level of energy on their infrastructure. In the 90′s, sites crashing or slowing was somewhat acceptable since the interwebs were exploding beyond available infrastructure’s capabilities. Now, there’s no excuse.

If you’re launching a new site, product or application, how do you get started?

The first thing you need to do is understand what resources you need and where the potential bottlenecks are when hundreds, thousands or even millions of people want to what you’re launching. You don’t need to invest in infrastructure to accommodate all of that traffic, but you need to know how you can add that infrastructure when you need it.

One of the easiest ways to prepare for your own success without getting bogged down by the bits and bytes is to take advantage of resources from some of our technology partners (and friends). If you have a PHP, Ruby on Rails or Node.js applications, Engine Yard will help you deploy and manage a specialized hosting environment. When you need a little more flexibility, RightScale‘s cloud management product lets you easily manage your environment in “a single integrated solution for extreme efficiency, speed and control.” If your biggest concern is your database’s performance and scalability, Cloudant has an excellent cloud database management service.

Invest a little time in getting ready for your success, and you won’t need to play catch-up when that success comes to you. Given how easy it is to prepare and protect your hosting environment these days, outages should go the way of the 8-track player.

-@jpwisler

Today everyone is trying to scramble and figure out how this “new” IT shift will work itself out. Our friends over at Parallels had a similar ambitious undertaking—trying to automate and enable a complete gamut of hosting and cloud services. This created a framework for our partnership. We worked with their engineering and sales teams, starting back in 2005, which resulted in Parallels Plesk Panel being offered as an option on every SoftLayer server. That was just the beginning. We are now deploying Parallels Automation for hosting partners and have plans to integrate with their Application Packaging Standard offering. Plans to integrate with other products like Parallels Cloud Server are also on the horizon. It all comes down to helping hosting companies and other joint customers thrive and succeed.

To find out more about our partnership and how it can help streamline your entry into cloud computing click here. We are also the only “Diamond” sponsor at the Parallels Summit 2012 APAC in Singapore this year. We share a heritage and understanding with Parallels borne from a need to simplify and solve IT problems on a broad scale. Now that’s what I call a likeminded partnership.

-@gkdog

In the months that followed the shelving of SOPA and PIPA, many of the hosting companies that were active in the fight were invited to join a new coalition that would focus on proposed legislation that affects Internet infrastructure providers … The Internet Infrastructure Coalition (or “i2Coalition”) was born. i2Coalition co-founder and Board Chair Christian Dawson explains the basics:

SoftLayer is proud to be a Charter Member of i2Coalition, and we’re excited to see how many vendors, partners, peers and competitors have joined us. Scrolling the ranks of founding members is a veritable “Who’s who?” of the companies that make up the “nuts and bolts” of the Internet.

The goal of i2Coalition is to facilitate public policy education and advocacy, develop market-driven standards formed by consensus and give the industry a unified voice. On the i2Coalition’s Public Policy page, that larger goal is broken down into focused priorities, with the first being

“In all public policy initiatives of the i2Coalition will be to encourage the growth and development of the Internet infrastructure industry and to protect the interests of members of the Coalition consistent with this development.”

Another huge priority worth noting is the focus on enabling and promoting the free exercise of human rights — including freedom of speech, freedom of assembly and the protection of personal privacy. Those rights are essential to fostering effective Internet advancement and to maintain a free and open Internet, and SoftLayer is a strong supporter of that platform.

If you operate in the hosting or Internet infrastructure space and you want to be part of the i2Coalition, we encourage you to become a member and join the conversation. When policymakers are talking about getting “an Internet” from their staff members, we know that there are plenty of opportunities to educate and provide context on the technical requirements and challenges that would result from proposed legislation, and the Internet Infrastructure Coalition is well equipped to capitalize on those opportunities.

-@toddmitchell

If you’ve ever looked at building your own private cloud in the past, the “days, weeks or months” timeline isn’t very surprising — you have to get the hardware provisioned, the software installed and the network configured … and it all has to work together. Hearing that SoftLayer Private Clouds can be provisioned in “hours” probably seems too good to be true to administrators who have tried building a private cloud in the past, so I thought I’d put it to the test by ordering a private cloud and documenting the experience.

At 9:30am, I walked over to Phil Jackson‘s desk and asked him if he would be interested in helping me out with the project. By 9:35am, I had him convinced (proof), and the clock was started.

When we started the order process, part of our work is already done for us:

To guarantee peak performance of the CloudPlatform management server, SoftLayer selected the hardware for us: A single processor quad core Xeon 5620 server with 6GB RAM, GigE, and two 2.0TB SATA II HDDs in RAID1. With the management server selected, our only task was choosing our host server and where we wanted the first zone (host server and management server) to be installed:

For our host server, we opted for a dual processor quad core Xeon 5504 with the default specs, and we decided to spin it up in DAL05. We added (and justified) a block of 16 secondary IP addresses for our first zone, and we submitted the order. The time: 9:38am.

At this point, it would be easy for us to game the system to shave off a few minutes from the provisioning process by manually approving the order we just placed (since we have access to the order queue), but we stayed true to the experiment and let it be approved as it normally would be. We didn’t have to wait long:

At 9:42am, our order was approved, and the pressure was on. How long would it take before we were able to log into the CloudStack portal to create a virtual machine? I’d walked over to Phil’s desk 12 minutes ago, and we still had to get two physical servers online and configured to work with each other on CloudPlatform. Luckily, the automated provisioning process took on a the brunt of that pressure.

Both server orders were sent to the data center, and the provisioning system selected two pieces of hardware that best matched what we needed. Our exact configurations weren’t available, so a SBT in the data center was dispatched to make the appropriate hardware changes to meet our needs, and the automated system kicked into high gear. IP addresses were assigned to the management and host servers, and we were able to monitor each server’s progress in the customer portal. The hardware was tested and prepared for OS install, and when it was ready, the base operating systems were loaded — CentOS 6 on the management server and Citrix XenServer 6 on the host server. After CentOS 6 finished provisioning on the management server, CloudStack was installed. Then we got an email:

At 11:24am, less than two hours from when I walked over to Phil’s desk, we had two servers online and configured with CloudStack, and we were ready to provision our first virtual machines in our private cloud environment.

We log into CloudStack and added our first instance:

We configured our new instance in a few clicks, and we clicked “Launch VM” at 11:38am. It came online in just over 3 minutes (11:42am):

I got from “walking to Phil’s desk” to having a multi-server private cloud infrastructure running a VM in exactly two hours and twelve minutes. For fun, I created a second VM on the host server, and it was provisioned in 31.7 seconds. It’s safe to say that the claim that SoftLayer takes “hours” to provision a private cloud has officially been confirmed, but we thought it would be fun to add one more wrinkle to the system: What if we wanted to add another host server in a different data center?

From the “Hardware” tab in the SoftLayer portal, we selected “Add Zone” to from the “Actions” in the “Private Clouds” section, and we chose a host server with four portable IP addresses in WDC01. The zone was created, and the host server went through the same hardware provisioning process that our initial deployment went through, and our new host server was online in < 2 hours. We jumped into CloudStack, and the new zone was created with our host server ready to provision VMs in Washington, D.C.

Given how quick the instances were spinning up in the first zone, we timed a few in the second zone … The first instance was online in about 4 minutes, and the second was running in 26.8 seconds.

By the time I went out for a late lunch at 1:30pm, we’d spun up a new private cloud infrastructure with geographically dispersed zones that launched new cloud instances in under 30 seconds. Not bad.

Don’t take my word for it, though … Order a SoftLayer Private Cloud and see for yourself.

-@khazard

5 Ways to Minimize Downtime During Summer Vacation

It’s a fact of life that everything runs smoothly until you’re out of contact, away from the Internet or on holiday. However, you can’t be available 24/7 on the chance that something breaks; instead, there are several things you can do to ensure that when things go wrong, the problem can be managed and resolved quickly. To help you set up your own “get back up” plan, we’ve come up with a checklist of the top five things you can do to prepare for an ill-timed issue.

1. Monitoring

How will you know when things break? Using a tool like Server Density — which combines availability monitoring from locations around the world with internal server metrics like disk usage, Apache and MySQL — means that you can be alerted if your site goes down, and have the data to find out why.

Surprisingly, the most common problems we see are some that are the easiest to fix. One problem that happens all too often is when a customer simply runs out of disk space in a volume! If you’ve ever had it happen to you, you know that running out of space will break things in strange ways — whether it prevents the database from accepting writes or fails to store web sessions on disk. By doing something as simple as setting an alert to monitor used disk space for all important volumes (not just root) at around 75%, you’ll have proactive visibility into your server to avoid hitting volume capacity.

Additionally, you should define triggers for unusual values that will set off a red flag for you. For example, if your Apache requests per second suddenly drop significantly, that change could indicate a problem somewhere else in your infrastructure, and if you’re not monitoring those indirect triggers, you may not learn about those other problems as quickly as you’d like. Find measurable direct and indirect relationships that can give you this kind of early warning, and find a way to measure them and alert yourself when something changes.

2. Dealing with Alerts

It’s no good having alerts sent to someone who isn’t responding (or who can’t at a given time). Using a service like Pagerduty allows you to define on-call rotations for different types of alerts. Nobody wants to be on-call every hour of every day, so differentiating and channeling alerts in an automated way could save you a lot of hassle. Another huge benefit of a platform like Pagerduty is that it also handles escalations: If the first contact in the path doesn’t wake up or is out of service, someone else gets notified quickly.

3. Tracking Incidents

Whether you’re the only person responsible or you have a team of engineers, you’ll want to track the status of alerts/issues, particularly if they require escalation to different vendors. If an incident lasts a long time, you’ll want to be able to hand it off to another person in your organization with all of the information they need. By tracking incidents with detailed notes information, you can avoid fatigue and prevent unnecessary repetition of troubleshooting steps.

We use JIRA for this because it allows you to define workflows an issue can progress along as you work on it. It also includes easy access to custom fields (e.g. specifying a vendor ticket ID) and can be assigned to different people.

4. Understanding What Happened

After you have received an alert, acknowledged it and started tracking the incident, it’s time to start investigating. Often, this involves looking at logs, and if you only have one or two servers, it’s relatively easy, but as soon as you add more, the process can get exponentially more difficult.

We recommend piping them all into a log search tool like (fellow Tech Partners Marketplace participant) Papertrail or Loggly. Those platforms afford you access to all of your logs from a single interface with the ability to see incoming lines in real-time or the functionality to search back to when the incident began (since you’ve clearly monitored and tracked all of that information in the first three steps).

5. Getting Access to Your Servers

If you’re traveling internationally, access to the Internet via a free hotspot like the ones you find in Starbucks isn’t always possible. It’s always a great idea to order a portable 3G hotspot in advance of a trip. You can usually pick one up from the airport to get basic Internet access without paying ridiculous roaming charges. Once you have your connection, the next step is to make sure you can access your servers.

Both iPhone and Android have SSH and remote desktop apps available which allow you to quickly log into your servers to fix easy problems. Having those tools often saves a lot of time if you don’t have access to your laptop, but they also introduce a security concern: If you open server logins to the world so you can login from the dynamic IPs that change when you use mobile connectivity, then it’s worth considering a multi-factor authentication layer. We use Duo Security for several reasons, with one major differentiator being the modules they have available for all major server operating systems to lock down our logins even further.

You’re never going to escape the reality of system administration: If your server has a problem, you need to fix it. What you can get away from is the uncertainty of not having a clearly defined process for responding to issues when they arise.

-David Mytton, ServerDensity