Posts Tagged 'Planning'

May 1, 2009

What A Cluster

When you think about all the things that have to go right all the time where all the time is millions of times per second for a user to get your content it can be a little... daunting. The software, the network, the hardware all have to work for this bit of magic we call the Internet to actually occur.

There are points of failure all over the place. Take a server for example: hard drives can fail, power supplies can fail, the OS could fail. The people running servers can fail.. maybe you try something new and it has unforeseen consequences. This is simply the way of things.

Mitigation comes in many forms. If your content is mostly images you could use something like a content delivery network to move your content into the "cloud" so that failure in one area might not take out everything. On the server itself you can do things like redundant power supplies and RAID arrays. Proper testing and staging of changes can help minimize the occurrence of software bugs and configuration errors impacting your production setup.

Even if nothing fails there will come a time when you have to shut down a service or reboot an entire server. Patches can't always update files that are in use, for example. One way to work around this problem is to have multiple servers working together in a server cluster. Clustering can be done in various ways, using Unix machines, Windows machines and even a combination of operating systems.

Since I've recently setup a Windows 2008 cluster that is we're going to discuss. First we need to discuss some terms. A node is a member of a cluster. Nodes are used to host resources, which are things that a cluster provides. When a node in a cluster fails another node takes over the job of offering that resource to the network. This can be done because resources (files, IPs, etc) are stored on the network using shared storage, which is typically a set of SAN drives to which multiple machines can connect.

Windows clusters come in a couple of conceptual forms. Active/Passive clusters have the resources hosted on one node and have another node just sitting idle waiting for the first to fail. Active/Active clusters on the other hand host some resources on each node. This puts each node to work. The key with clusters is that you need to size the nodes such that your workloads can still function even if there is node failure.

Ok, so you have multiple machines, a SAN between them, some IPs and something you wish to serve up in a highly available manner. How does this work? Once you create the cluster you then go about defining resources. In the case of the cluster I set up my resource was a file share. I wanted these files to be available on the network even if I had to reboot one of the servers. The resource was actually combination of an IP address that could be answered by either machine and the iSCSI drive mount which contained the actual files.

Once the resource was established it was hosted on NodeA. When I rebooted NodeA though the resource was automatically failed over to NodeB so that the total interruption in service was only a couple of seconds. NodeB took possession of the IP address and the iSCSI mount automatically once it determined that NodeA had gone away.

File serving is a really basic example but you can clustering with much more complicated things like the Microsoft Exchange e-mail server, Internet Information Server, Virtual Machines and even network services like DHCP/DNS/WINs.

Clusters are not the end of service failures. The shared storage can fail, the network can fail, the software configuration or the humans could fail. With a proper technical staff implementing and maintaining them, however, clusters can be a useful tool in the quest for high availability.

November 13, 2008

Size Isn't Everything

A couple days ago, I took my daughter to her favorite store. We picked up a fair amount and on the way to the car she asked a simple question, or so I thought. “Why did they only fill these bags half way”. Confused I looked at the bags and realized she was holding a bag which had a large stuffed bear in it and was looking at a bag less than half full of canned food.

Being the person I am, rather than attempt to explain this to her I wanted to let her try and figure it out for herself so she would understand it better. When we got home, I filled the rest of the bag with cans and had her try and pick it up, as I expected the bag broke in her hands. I explained to her that the cans were much heavier then the bag. She still doesn’t quite understand the concept that the bag has 2 limits, size and weight but she is starting to understand this concept.

I thought about this story this morning when I started working on a project of determining how many containers a Virtuozzo server could handle based on its system requirements. Just like the bag, a Virtuozzo system has multiple limitations that need to be observed, the size of the containers as well as their “weight”. In this situation “weight” would be the drain on overall system resources. When attempting to determine how many containers a system can handle, you need to take into account not only how many will fit size wise, but also how much of the overall system resources each container will require.

It turns out this question is much easier to ask then to answer. You can take a small server such as a dual core with 4GB of RAM and put 20 or even 30 containers onto the server and have it run flawlessly when those containers are small and do not require much in the way of system recourses. At the same time however I can take a quad proc quad core with 64GB of RAM and grind it to a halt with 1 or 2 containers.

At the end of the day, I have found that you can make just about anything work, but before you attempt to determine what hardware you will need to run a Virtuozzo server, it’s a good idea to have an estimate of what you expect the containers to be doing. What could be worse than spending hours configuring a server and getting it online only to watch it grind to a halt because there are just too many containers completely saturating your system resources?

-Mathew

Categories: 
January 14, 2008

Growth is a Good Thing. No Really.

The high-pitched whine of a drill sends a shiver down my spine. I jump a little in my seat at a loud bang followed by shuffling feet and mumbled voices. I involuntarily cower at the unmistakable sound of a saw blade spinning—gaining momentum—biting. Nope, I'm not sitting in a theater watching Eli Roth's next installment in the Hostel franchise. In fact, I'm at the office.

That's right. I'm sitting at my desk. Sitting at my desk and trying hard to ignore the plethora of singing power tools and crooning contractors who for the last two months have been busy putting up dry wall, wiring electrical outlets, installing locks, and occasionally setting off the fire alarm. It's the sound of growth. And at the risk of conjuring up images of bad 80's haircuts, guys in jeans way-too-tight, and shirts where the collars just wouldn't seem to stay down-- one might dare refer to the ruckus as "growing pains".

Make no mistake about it, growing is painful. Take it from me. I think I was 19 before I managed to grow enough facial hair to require the use of a razor. Combine that tidbit of info with the fact that I had every 8-bit computer known to man proudly on display in my room right next to my impressive collection of latex Hollywood style monster masks and you'll start to get the picture. Growing requires a lot of work and allows almost no planning as humans have a habit of blossoming in their own sweet time. Companies are no different.

So while management did everything possible to make the required building expansion as unobtrusive as possible, well, it's still construction work within earshot of a whole team of developers, technicians, and engineers. That's just the way it is. And while I may complain about the noise and distractions now and again, there is also something very comforting about knowing that I am working at a place that is growing. Growing phenomenally, in a time when not all technology companies are fairing so well.

When the dust settles there will be a lot of new space.

More space means a lot of new hires. More space means more opportunity for existing employees. And yes, more space means more work for everyone involved. Having worked for three failed ventures in as many years, I can tell you I am more than happy to be putting my time and effort and energies into something that is successful; something that continues to be more successful every day. It feels good to be on the winning team for a change. Hearing what some of the other engineers here are saying I don't think I'm alone in that sentiment.

That's not to say I'll miss the noise when the construction is all said and done. Which in case you are interested sounds to be winding down. As for SoftLayer, well something tells me we are just getting started.

-William

Categories: 
Subscribe to planning