May 18, 2009

Special Ops: The “SEALs” of SoftLayer

When you think about a Special Operations Unit, you probably think of TV shows like, “The Unit”, or maybe you have the Military Channel and have seen the reality TV show, “Navy SEALs: BUD/s training”, or maybe you are one of the 7 people that saw that 1980s movie starring Charlie Sheen…………naaahhh. Anyway, whether it is secretive missions in Iraq or taking out pirates, real Special Ops Teams are very well trained individually and as a team. It takes a desire on the part of the individual to be the best at what he does and a desire to be an intricate part of a highly skilled, successful team.

I have been at SoftLayer for over 2 years now, and I particularly enjoy how our support team has come together in much the same way as a military special ops team. No, most of us do not wear our hair “high and tight”, and, unlike Navy SEALs, there are various piercings about the face of several of my teammates adding to an already very “distinctive” style of dress (There is a very loose dress code in the support department). But, the focused hours of training put into being the best at our craft is very similar to a special ops team.

I remember an occasion during my time here at SoftLayer when we had a sudden outage in which a switch failed. Any major data center will have a piece of equipment fail eventually, but the difference comes in how it is handled. Monitoring alerts went off and the team jumped into action. The managers and shift leads were instantly organizing, although the rest of us already knew what to do as training had prepared us. Each of us took a group of servers and checked for network connectivity in order to localize the issue. We fielded phone calls while the switch was being replaced. Tickets were answered quickly and grouped according to information needed by the specific customer. Verbal, IM, and email communication was flying and everyone knew the status from moment to moment. The switch was replaced and the event was concluded. Customers received the information they needed to pass on to their customers and peace was restored. I am amazed by the speed and efficiency with which this situation was handled. And, our customers were very happy with our speed especially considering an outage is never good news.

We would all like for everything to always work perfectly, but knowing that a highly efficient, highly competent, well-trained, focused, “special ops” team is ready at a moment’s notice, whatever the mission, is very comforting to our customers when financial success is on the line. Secretly, I have always wanted to be part of a special ops team, I just never imagined it would happen at a technology company called SoftLayer.

May 15, 2009

Disaster Recovery Plan

A few days ago I was reading a news story about a man who just lost everything to a fire. One of the comments he made was that he had never thought to plan for something like this; it was the type of thing that happened to other people but never to me. I started thinking about how true that statement was. Many people just never think it will happen to them.

This type of situation happens every day in the IT field. There is some sort of disaster causing a server to crash or simply stop working all together, the drives on the server are completely corrupted and the data is just gone. The question is; when this happens to you, will you be prepared? Thankfully, there are steps each person can take to limit the pain and downtime a situation like this can cause. Like any other disaster recovery plan, the more you are willing to put into it, the more protection you will have when disaster strikes.

This is where SoftLayer comes in. Here at SoftLayer we understand the importants of providing our customers the means to create a good disaster recovery plan that meets their needs. We understand that a detailed disaster recovery plan will include things such as backups and replication. Our services such as NAS and EVault are perfect solutions for performing and managing the backups for you server. When looking into replication, we offer services such as iSCSI replication, Raids, local and global loadbalancing which will provide our customer with the tools to replicate not only their data across multiple locations but their servers as well. Above all, we provide our private network to securely transfer this data to the many locations without impacting the traffic on your public network.

We can only hope that on the day disaster strikes, everyone has some plan in place to deal with it. There is nothing more frustrating in this industry then the loss of crucial data that in many instances cannot be recovered.p

May 13, 2009

The Data Center is Full of Surprises

After having been in the IT industry in some form or fashion for the last decade or so, I’ve learned that no matter how well you prepare yourself for disaster, you never seem to be surprised by certain issues that present themselves. Yes, ladies and gentlemen, I’m talking about the many surprises our friend Mr. Murphy can throw at us. I’m sure many a tech will anecdotally speak of the time where their server borked on them, their backups failed despite numerous backup audits, and they were up the infamous creek (I’m only assuming at least a few readers are nodding right now). Sometimes painful lessons are the best times to learn, but it’s a bad day when it happens on a production server.

Working in the SoftLayer data center, we take incredible measures to protect our customer’s servers. In a sense, we try to keep Mr. Murphy away. From the biggies (like redundant power and MASSIVE cooling units) to the routine (such as the data center walkthroughs, and proactive RAID alerts), we do our best to keep the servers in the data center running smoothly, and free of surprises.

Beyond the punches our friend Mr. Murphy can throw at us now and again, it’s nice to know there are a few good surprises in store for you, too. You might be surprised at the great deals our SLales team can provide. You might also be surprised at not only the amazing features such as the new Cloudlayer™ Storage, but the incredible rate we keep bringing new features to the table. I’ve also seen customer’s surprise when we rescue their server from the brink of disaster, or when we are able to provide a few tweaks to give THEIR business the edge it needs.

Furthermore, our people keep the data center interesting. SoftLayer sees no shortage of antics. There’s John’s fully automatic Nerf gun. There’s also plenty of jokes played at the expense of someone unfortunate enough to leave their workstation unlocked (call it “security training” – favorite backgrounds include the Care Bears and My Little Pony). We also have that one hardware tech who likes to hide around corners or sneak up behind you, and scare the life out of you while you’re focused on the task at hand.

With so many surprises, SoftLayer continues to be a very interesting place to work, and most certainly a place where one would never get bored!

May 11, 2009

Has Your Life Changed Due to Social Networking?

Social Networking in my eyes has been around for years since the old bulletin board system days, though now with the advancement of technology and the internet, we see it on a much larger scale. I remember in 2004 I told myself I would never join Myspace. That turned out to be wrong in October of 2005 I joined and connected with friends from the past.

Then came facebook, I once again refused to become a member of another social networking site, though in 2008 I joined to help diagnose one of our customers issues as they hosted an application on the site. I then later found additional friends that had one or the other, so I started using both to stay connected (Still all wondering why no one just uses email like it was intended for instead of just for notifications from your favorite social sites).

I then joined a more professional social networking site, LinkedIn, and became connected with former co-workers, contractors, professors, students, and industry people. Then came twitter, which I just signed up for a few months ago and haven’t really gotten into it, as I feel it is the same as status updates on facebook and myspace, while thinking to myself, do I really need to update my friends with what I am doing every hour on all three. No I don’t think so, but I have friends that do!

My blackberry has five main applications that are used on it: Email; Browser; SMS messages; Myspace; and Facebook. All get checked about every 30 minutes so I can stay in the know about everything from work, to friends. I rarely use personal email for things anymore as most of my friends contact me on one of the social networking sites. I have often thought about deleting one or the other if not both. I find myself telling myself, why it’s like when you disconnected text messaging from your cell phone in 2000 because you hated it, you’ll sign right back up for it as it has became an essential in my life.

I can’t say I couldn’t live without social networking sites, but it would require me to reorganize how I organize my already busy life. My answer is yes, social networking has changed my life and how I interact with people. Social Networking can be a very vital tool, if you are a restaurant / bar, band, sales person, etc. You can use Social Networking to expose yourself to that many more people and reach out!

Now I’m off to install TwitterBerry and see if that makes me like twitter anymore, ask me in three months.

May 8, 2009

Interview with the Printer

SL: Hey, The elevator was acting strangely this morning. I wanted your opinion on a few things.
Printer: *whir*

SL: Excellent, I’m glad to hear your enthusiasm. I’ve been doing a lot of thinking lately about how the web hosting industry seems to be weathering the storm of the global economic downturn quite well. It seems regardless of the bank bailouts, failed mortgages, and credit crises, there is still a high demand for social networking applications, online shopping and exchange of information via the many forums available. Furthermore, with the reliability of our redundant links, businesses are finding it more affordable to outsource their IT assets, and host with us.

SL: I Get it. By hosting here, you can also begin the transition to a paperless business. Something to the effect of a tech taking an X-Ray, uploading it to an SL server through the private uplink, and instantly having it available to a doctor thousands of miles away. And that’s just one possibility out of the endless uses for a server here. We have a ton of space available, and by design, we use approximately a square foot per server in our Datacenter… How’s that for space efficiency?

SL: Yea, it gets pretty toasty in there, but luckily we have environmental controls in place to mitigate the heat put out by the thousands of servers in the pods. We also have monitoring in place to notify us of any possible situations. As uptime is vital in the web hosting industry, we have a number of features available both internally and externally. We have 24/7/365 monitoring, automatic reboots, and a highly intuitive customer portal. Not to mention the best technicians in the industry
Printer: *beep*

SL: Now that’s just rude. We have staff from every facet of IT working in our NOC every day of the year. With the highly skilled staff holding years upon years of experience, there are few issues that can’t be solved quickly and efficiently.
Printer: PAPER_JAM

SL: Doubtful. We continue to innovate in efficiency and features. Now you’re acting just like the elevator.

SL: Ugh, you’re just as bad as the Elevator. What does that mean, anyway?

May 6, 2009

Always Use Protection

When it comes to managing a server remember you can never be to careful. In this day and age we face a lot of things that can damage and even take a server to its knees here’s a few things for everyone to consider.


This is a must on systems open to the net now days. There are always nasty little things floating around looking to take your server apart from the OS out. For windows servers there are a multitude of choices and I’ll just mention a few that can help protect your goods. You can use several programs such as avast (which offers a free edition), ClamWin (open source), Kaspersky , and Panda just to name a few. I would suggest before installing any of these you check links such as to name one that provides a list of several choices and their compatibility. You may also want to read reviews that compare the available options and give you an idea of what to expect from them. This will allow you to make an informed choice on which one works best for you. Now with linux there are also several options for this including the well known clamav which from personal experience works really well and can be installed on a variety of linux disro’s(aka distributions). It’s very simple to use and may prevent you from headache later on down the road.


Firewalls are a double edged sword but are most defiantly needed. When it comes to firewalls you can protect yourself from quite a bit of headache however if setup to strict you can block positive traffic and even yourself from reaching your server but in the long run a defiant way to help protect your server from unwanted visitors. A lot of firewalls also have modules and add-ons that further assist in protecting you and securing your server. If in doubt it’s always a good idea to have a security company do an audit and even a security hardening session with your server to make sure you are protected the best way possible.


This is probably one of the most important this you can do to secure your server. Use strong passwords (no using password or jello is not a secure password even if it is in all caps) and if you are worried about not being able to come up with a secure one there are several password generators on the web that can come up with secure ones to assist. Passwords should contain caps letters, numbers, symbols, and should be at minimum 8 – 10 characters (the more the better). It’s the easy to remember and easy read passwords that get you into trouble.

Armed with this information and so much more on security that can be located on the web using the great and all powerful Google should be a good start to making sure you don’t have to worry about data loss and system hacks. Also remember no matter how secure you think you are make regular backups of all your important data as if you server could crash at any time.

May 4, 2009

Paradigm Shift

From the beginning of my coming of age in the IT industry, It’s been one thing – Windows. As a system administrator in a highly mobile Windows environment, you learn a thing or two to make things tick, and to make them keep ticking. I had become quite proficient with the Active Directory environment, and was able to keep a domain going. While windows is a useful enterprise-grade server solution, it’s certainly not the only solution. Unfortunately when I made my departure from that particular environment, I hadn’t had much exposure to the plethora of options available to an administrator.

Then Along comes SoftLayer, and opens my eyes to an array of new (well, at least to me) operating systems. Now, I had begun my ‘new’ IT life, with exposure to the latest and greatest, to include Windows, as well as virtualization software such as Xen and Virtuozzo, and great open source operating systems such as CentOS, and FreeBSD. With the new exposure to all these high-speed technologies, I felt that maybe it was time for me to let the de-facto home operating system take a break, and kick the tires on a new installation.

I can say that while switching to open source was a bit nerve racking, it ended up being quick and painless, and I’m not looking back. I’ve lost a few hours of sleep here and there trying to dive in and learn a thing or two about the new operating system, as well as making some tweaks to get it just like I like it. The process was certainly a learning experience, and I’ve become much more familiar with an operating system that, at first, can seem rather intimidating. I went through a few different distributions till I settled on one that’s perfect for what I do (like reading the InnerLayer, and finishing the multitude of college papers).

The only problem with always reloading a PC is you have to sit there and watch it. It doesn’t hurt to have a TV and an MP3 player sitting around while you configure everything and get the reload going, but you still have to be around to make sure everything goes as planned. Imagine this… You click a button, and check back in a few. Sound Familiar? Yep, it would have been nice to have an automated reload system much like we have here at SoftLayer. Not to mention, if something goes awry, there’s the assurance that someone will be there to investigate and correct the issue. That way, I can open a cold one, and watch the game, or attend to other matters more important than telling my computer my time zone.

May 1, 2009

What A Cluster

When you think about all the things that have to go right all the time where all the time is millions of times per second for a user to get your content it can be a little... daunting. The software, the network, the hardware all have to work for this bit of magic we call the Internet to actually occur.

There are points of failure all over the place. Take a server for example: hard drives can fail, power supplies can fail, the OS could fail. The people running servers can fail.. maybe you try something new and it has unforeseen consequences. This is simply the way of things.

Mitigation comes in many forms. If your content is mostly images you could use something like a content delivery network to move your content into the "cloud" so that failure in one area might not take out everything. On the server itself you can do things like redundant power supplies and RAID arrays. Proper testing and staging of changes can help minimize the occurrence of software bugs and configuration errors impacting your production setup.

Even if nothing fails there will come a time when you have to shut down a service or reboot an entire server. Patches can't always update files that are in use, for example. One way to work around this problem is to have multiple servers working together in a server cluster. Clustering can be done in various ways, using Unix machines, Windows machines and even a combination of operating systems.

Since I've recently setup a Windows 2008 cluster that is we're going to discuss. First we need to discuss some terms. A node is a member of a cluster. Nodes are used to host resources, which are things that a cluster provides. When a node in a cluster fails another node takes over the job of offering that resource to the network. This can be done because resources (files, IPs, etc) are stored on the network using shared storage, which is typically a set of SAN drives to which multiple machines can connect.

Windows clusters come in a couple of conceptual forms. Active/Passive clusters have the resources hosted on one node and have another node just sitting idle waiting for the first to fail. Active/Active clusters on the other hand host some resources on each node. This puts each node to work. The key with clusters is that you need to size the nodes such that your workloads can still function even if there is node failure.

Ok, so you have multiple machines, a SAN between them, some IPs and something you wish to serve up in a highly available manner. How does this work? Once you create the cluster you then go about defining resources. In the case of the cluster I set up my resource was a file share. I wanted these files to be available on the network even if I had to reboot one of the servers. The resource was actually combination of an IP address that could be answered by either machine and the iSCSI drive mount which contained the actual files.

Once the resource was established it was hosted on NodeA. When I rebooted NodeA though the resource was automatically failed over to NodeB so that the total interruption in service was only a couple of seconds. NodeB took possession of the IP address and the iSCSI mount automatically once it determined that NodeA had gone away.

File serving is a really basic example but you can clustering with much more complicated things like the Microsoft Exchange e-mail server, Internet Information Server, Virtual Machines and even network services like DHCP/DNS/WINs.

Clusters are not the end of service failures. The shared storage can fail, the network can fail, the software configuration or the humans could fail. With a proper technical staff implementing and maintaining them, however, clusters can be a useful tool in the quest for high availability.

April 29, 2009

Musician's Mind

One thing I have noticed about my SoftLayer family is the number of musicians here. I spent nine years as a musician, working the bar/festival circuits all over the midwest. When I arrived at SoftLayer, people jokingly asked if I was joining the SoftLayer band due to my previous experience. Just looking at the Operations Management Team, most of us have written/performed and many continue to. The more I thought about it, I think that this is a good thing.

There have been a number of scientific research projects about the academic performance of children and teens who are involved in a music program at school. The mixed left/right brain activity of music lends itself to problem solving and critical thinking as well as creativity which are required by many jobs. A musician's mind is capable of working complex geometric patterns into physical movements in coordination with muscle memory. Once more advanced levels are reached, a musician is capable of not only composing music, but also improvisation. These aspects are similar to many necessary thought processes used in the office.

I'd like to think that these mental processes allow me to think better on my feet, deal with change, and have a global view of the projects that I am involved with. Maybe that's why our Inventory, Hardware, & 2/5 Datacenter Managers are musicians. Not only that, but our Director of Operations is an avid musician. If you count the actual system admins who are musical, then you would be adding another 3 people.

So if all the research is correct, this may have something to do with the quality of operations here at SoftLayer. I wonder if being a musician had anything to do with my interview and its result. I know that it would catch my attention knowing that an applicant had experience with music or any other analytical/creative endeavor, especially if it has been shown to improve overall performance or intelligence.

Maybe I should write a SoftLayer song - an anthem to our Datacenter or a love song about the management network and IPMI?

April 27, 2009

Into the Cloud

You'll see the word "cloud" bouncing around quite a bit in IT nowadays. If you have been following The Inner Layer you'll have seen it a few times here as well. A cloud service is just something that is hosted on the Internet. Typically in a cloud scenario you are not actually doing the hosting but rather using hosted resources someone else is providing. Usually you'll hear it in terms of computing and storage.

This is going to be a brief article on a cloud storage product we are doing here at SoftLayer called CloudLayer™ Storage.

CloudLayer™ Storage is a WebDAV based system which uses client software on the local machine in order to redirect filesystem operations to the storage repository here at SoftLayer. In Windows you end up with a drive letter; on a Unix system you end up with a mount point. In both cases when you create folders and save files on those locations the actions actually happen on our storage repository here. Because the files are located with us you are able to access them wherever you are. Through the CloudLayer™ web UI you're able to also set up files for sharing with others so even if where you are never changes there is still value to using the cloud storage.

Even using a cloud storage system you must maintain proper backups. Hardware, software and human error all happens. Tied in with that concept of "errors happen" ... if you have files on CloudLayer™ that you need for a presentation, download them ahead of time. You don't want to be caught without files simply because the Internet connection at your hotel decided to take a nap the morning of your event.

Now what of security? Well, the connection to the CloudLayer™ endpoint here at SoftLayer is done via an encrypted session so people cannot snoop on your transmissions. This does mean you need to allow 443/tcp outbound communications via your firewall but since that is the normal HTTPS port I'd imagine you already have it open. Within CloudLayer™ you can control with whom you share your files.

Since CloudLayer™ is a filesystem redirected over the Internet the performance you get will be dependent on your local connection speed. Its best to treat CloudLayer™ Storage as simply a file repository. If you find you need some kind of off-machine storage for running applications on your server you could look into our iSCSI product.

So, dear readers, go forth and have a cloudy day.


Subscribe to introductions