Posts Tagged 'Systems'

March 9, 2011

Building a Data Center | Part 2: The Absence of Heat

As you walk down the cold aisle in a data center, you might be curious about how all that cold air gets there. Like the electrical system, data center cooling travels a path through the data center that relies on many integrated systems working together to achieve the desired result.

To start, I should give a crash course in Heating, Ventilating, and Air Conditioning (HVAC). The most important thing to understand in HVAC theory is that cold is the absence of heat. When you say you're cooling a space, you're not adding cold air, rather you are removing heat. Heat is removed in a cycle called the refrigerant cycle. The refrigerant cycle is present in all air conditioning systems and is made up of four main components:

  • Refrigerant: Refrigerants are engineered chemicals developed to have very specific boiling and condensation temperatures. They come in many different flavors with cryptic names like 410a, R22, and water.
  • Compressor: Compresses refrigerant, turning it from a warm liquid to hot gas. This compression assists in the movement of heat and refrigerant within the system.
  • Evaporator: Evaporators are heat exchangers (devices built for efficient heat transfer from one medium to another), so an evaporator passes heat from the air to the refrigerant.
  • Condenser: Condensers are also heat exchangers. The condenser releases trapped heat in refrigerant outside the space being cooled.

This is a very simplified explanation of the refrigerant cycle components, and I only mention these four components or steps because they are common to all HVAC systems regardless of size or type, and you can apply them to any data center cooling system (or even a residential system).

Cooling System

Which came first - the chicken or the egg? Like the old analogy, figuring out a starting point for our cooling cycle isn't easy to do, so let's start at the source of heat. Your server uses electrical energy to process the information in the CPU, turn the spindle of hard drives, and light up the pretty little LEDs on the chassis. All that conversion of electrical energy to useful work creates heat as a side effect. Remember that we have to remove heat to cool something, so a server's cooling system acts like a heat exchanger, extracting heat from its components and passing that heat to cooler air entering the front of the server. That heat is rejected from the back of the servers into the hot aisle.

When the heat is exhausted into the hot aisle, it is pulled to the evaporator, which we call different things depending on how they perform their function. CRACs – Computer Room Air Conditioners – and AHU – Air Handling Units – are a few of the common terms we use. Regardless of what they are called, they perform the same function by removing heat from the hot aisle (called return air) and supply cooled air to the cold aisle. This completes the first step in the cycle.

Now that the heat was removed from the server room and passed to the refrigerant, it must go somewhere and that is where the compressor comes in. Warm liquid refrigerant is compressed into a hot gas and this compression of refrigerant forces it to travel to the condenser where the heat is absorbed into the outside air. This allows the cooled refrigerant to condense and return to the evaporator to start the process all over again. And again, this part of the cycle is accomplished different ways depending on the type of equipment installed.

If a CRAC unit is installed, the evaporator and compressor are on the computer room floor, a remote condenser will in placed outside, and fans will extract the heat from the refrigerant. In areas where AHUs are used, only the evaporator will be typically be on the raised floor. These systems use a remote compressor and condenser to send chilled water to the AHU’s on the raised floor. Also, these chilled water systems actually have two separate refrigerant systems in place, isolating the inside and outside portions of the refrigerant cycle. They are used in larger denser data centers because they allow for more efficient control of temperatures in the data center.

Like I said, this is a simplified explanation of the cooling of a data center but it lays the ground work for a more in-depth look at your specific systems.

-John

January 26, 2011

Time for an Oil Change?

<Fade In>
Man driving into Jiffy Lube, car sputtering and smoking.
Attendant: "Looks like you need an oil change buddy."
Buddy: "Yep, I think so. I was here last week and I think they used the wrong oil!"
Attendant: "Nah, we wouldn't do that. In fact we only have one kind of oil here and that's SAS 70."
Buddy: "Well, that's odd; I am told that I need SSAE 16 for mine to work right."
<Mass Confusion>

Welcome to my world! We have SAS 70 today, but soon we will have the new synthetic, non abrasive, engine-cleaning SSAE 16. Sounds fun right? I sure hope so.

Why the change? Good question. When SAS 70 first appeared in the early 90s, the world's economies weren't quite as intertwined as they are today. It was much harder to do business globally than it is now. (I think the "fad" called the internet has a little something to do with that but I could be wrong!) Now that the oceans have shrunk to a more manageable size, there is a need for the standards that companies use worldwide to match more closely. The goal of the U.S. Statement on Standards for Attestation Engagements 16 (SSAE 16) is to meet a more uniform reporting standard.

What's the difference? It's an "attestation" not an "audit." Google and thefreedictionary.com define attestation as "To affirm to be correct, true, or genuine," and audit as "an inspection, correction, and verification of business accounts." Though they are closely related, they mean different things.

What stay's the same? The focus will still be on controls at service organizations when the controls are relevant to their user entities' internal control over financial reporting. (For some reason, servers tend to have quite a bit to do with that!) There will still be a Type 1 and Type 2 with similar scopes in format. The reports will look very similar but they should be a bit more descriptive. The report will still be used in the same methods and by the same type of user.

What Changes? SSAE 16 is now an attestation and not really an audit. The service auditor will still provide an opinion but it will align itself more closely with existing international attestation standards.

  • Written Management Assertion - Management will be required to provide an assertion, to be included in the report, stating the system is fairly represented, suitably designed and implemented and the related controls were suitably designed to achieve the stated control objectives, and that the controls operated effectively throughout the period. The report will reference that management is responsible for preparing the system description, providing the stated services, specifying the control objectives, identifying the risks, selecting the criteria and designing, implementing and documenting controls that are suitably designed and operating effectively. The auditor's opinion remains in the role of providing assurance, not as the entity responsible for the communication.
  • System Description - The more inclusive description must detail the services covered, classes of transactions, events other than transactions, report preparation processes, control objectives and related controls, complementary user controls and other relevant aspects of the organization's control environment, risk assessment process, information and communication systems, control activities and monitoring controls. (I think an accountant came up with all of that!)

There are quite a few other differences but I think these are the big headliners. SoftLayer is committed to making this change and having it available for our customers that require it. Our normal SAS 70 schedule is Nov. 1 – Oct. 31 but we will be accelerating the process to have the SSAE 16 in place as soon as possible.

We are continuously looking at other compliance, reporting, audits and certifications. If you have any that would help you and your business, let us know.

-Skinman

Categories: 
January 13, 2011

API Basics: What is the API?

Over the past few years, there has been a dramatic increase in the use of APIs, or Application Programming Interfaces. An API allows a developer to create software that directly interfaces with another system. A simple example would be an online retail site that drop ships products from a distributor. Without an API, the retail site administrator might have to manually update stock availability and product information, but with an API, the retail site administrator can link directly to the distributor's database to display real-time stock and product changes.

SoftLayer has taken this concept to a whole new level. Most APIs are added to an existing system allowing the developer to interact with a small part of the overall architecture. SoftLayer took a different approach. We built our entire system on the API. This means anything you can do from our Portal, you can do via the API.

The breadth of the API can be seen in its current use cases. On one end of the usage spectrum, customers simply incorporate a few API calls to keep track of their hosting account details like bandwidth usage and invoice information. On the other end, users have built the API into complete management systems that allow control over all aspects of their SoftLayer experience - essentially a custom, personalized portal.

What gives an API its true power is its ability to automate. In a standard system, you can plans for any situation, but someone needs to see the situation and react to it. With the API, the system can automatically react. For example, an administrator could design a system that requires a minimum number of IP addresses for every X number of clients on a server. If a new client is added to the server and a new IP address is needed, the administrator's application can use the API to automatically request an additional block of IPs to be provisioned to the server.

This level of automation can be taken even further: A system is in place that needs to dynamically scale based on traffic. With the power of the API and the use of Cloud Instances, this can be achieved. By designing a system to monitor the current traffic trends, when certain thresholds are reached, the system can automatically request additional instances to be provisioned or excess instances to be removed from the pool and terminated. This system would function without any manual intervention to recognize the issue and react to it. Obviously, the real-time automation saves money because instances are turned up and turned down as they are needed, and it ensures your systems stay up and running. It might even let your systems administrator get a full night of sleep.

There is no limit to the potential applications of the SoftLayer API. Whether you are an individual just starting to program or have a team of developers with years of experience, the API has the tools that you will need to get started.

-Mathew

February 26, 2010

Hero or Failure?

You’re hired, welcome to the company! All you techies out there have heard that before. Then for the first couple of weeks you get the luxury of, “just take a look around the network and see what you see, make a note of what is good and what needs some work”. You make a few notes during your two week honeymoon period and then you hit the ground running. You make changes to a few of the server configs to speed them up, and you notice that there are a couple of hard drives in the server farm that are showing they are about to fail and you make a note to get that fixed. Everyone on the team hails your progress, smarts, and work ethic and thinks they have made the right choice. Even though the in-house gear is a little old you have made changes that made things faster and more redundant in your first month. Great Job! You are on your way to the Information Systems Hero title.

Everything is going along great at about the 8 month point. You have made a few key decisions along the way and have some of your gear outsourced now. All the ancient hardware onsite has been retired and liquidated and just a few core machines remain. You still have a large storage device and a tape robot onsite for your backups and you keep the tape library safely offsite. All is good in the department.

If you want to be the Hero skip to the word HERO / If you want to be a failure please skip 2 paragraphs to the word FAILURE

HERO

You have a free day or two in which nothing pressing needs to be addressed and you decide to look into the backup rotation and type. After spending a little time looking at it and not feeling comfortable you make the decision to create a secondary backup into the cloud as a test. After a little setup and tinkering you finish up and go on with your daily tasks.

A few months later your onsite storage device hard fails and there is massive data loss. A new system is delivered the same day and once the setup is complete the tapes are delivered and the restore process starts. Three hours into the restore a bad tape is encountered and again you are faced with massive data loss. The entire group is now in panic mode. It suddenly hits you that you setup a test backup offsite. What are the odds that it is still functioning and you will be able to get the data? With help from the entire department you get the network right and the data transfer starts. About one hour later the data is restored and your employees are happy not to mention your boss. You are now an IT hero.

FAILURE

A few months later your onsite storage device hard fails and there is massive data loss. A new system is delivered the same day and once the setup is complete the tapes are delivered and the restore process starts. Three hours into the restore a bad tape is encountered and again you are faced with massive data loss. The entire group is now in panic mode. After many attempts at trying to repair the damaged tape and having multiple experts look at the failed storage device. You and your team realize that 5 days of data will be lost and have to be recreated. Not a great day for your team. You are now an IT failure.

Moral of the story?

Use the tools the world provides to stay ahead of the curve. All it takes is one mistake to be a failure.

June 17, 2009

Problem Solving

Quite often my friends who are not really that internet savvy ask me what I do at work, I think back to the time in the first grade when my teacher Mrs. Hyde told me: “ Bill you’re going to be a great problem solver when you get older, your problem solving skills are already at a fourth grade level.” Now you’re probably reading this wondering how problem solving problems in the first grade have anything to do with my job. It is, as she told me, all about how you think. She told me I was an outside the box thinker.

My co-workers and I deal with a network of 20,000+ servers, and 5500+ customers, in over 110 different countries, and support over 15 different operating systems. That leads to an almost infinite combination of language, hardware, and software options. When our customers submit an issue for us to work on, it is always different than the time before – whether that is a ticket from the same customer or a ticket on a similar topic. We have a very diverse range of customers using our servers for a number of things, so not every server in here is doing the same thing. In order to be good at supporting our customers, SoftLayer’s management, in my opinion, has hired some of the best problem solvers around the world to address all of our customer issues. So that is what I am: I am a problem solver! Otherwise known as a Customer Systems Administrator. We’re required to know a broad range of technologies and have the passion to learn the new ones as they come along. I think that is why I chose to work in the field that I work in, it is always changing. I tried moving over to telecommunications engineering a few years ago, but got bored with is as it was the same issues day in and day out on the equipment. Working here at SoftLayer is wonderful as there is never a dull moment.

May 31, 2008

Response to On Site Development

On May 14th my buddy Shawn wrote On Site Development. Aside from the ambiguous title (I originally thought it was an article on web site development, rather than the more appropriate on-site development), there were a number of things that I felt could be expanded upon. I started by simply commenting on his post, but the comment hit half a page and I had to admit to myself that I was, in fact, writing an entire new post.

Updating the computer systems in these restaurants is a question of scale. Sure, it seems cheap to update the software on the 6 computers in a local fast food restaurant. However, a certain “largest fast-food chain in the world” has 31,000+ locations (according to Wikipedia). Now I know how much I would charge to update greasy fast-food computers, and if you multiply that by 31,000, you get a whole lot of dollars. It just doesn’t scale well enough to make it worthwhile. The bottom line is, the companies do cost-benefit analysis on all projects, and the cost of re-doing the messed up orders is apparently less than the cost of patching the software on a quarter million little cash registers and kitchen computers.

It's the same logic that lead to Coke being sold for 5 cents for more than 60 years, spanning two world wars and the great depression without fluctuating in price. The vast majority of Coca-Cola during that time period was sold from vending machines. These vending machines only accepted nickels, and once a nickel was inserted, a Coke came out. That’s it. Nothing digital, no multi-coin receptacles, just insert nickel…receive Coke. The cost of replacing 100,000 vending machines was far higher than the profits they would get by increasing the price of coke slightly. Only after World War II, when industrialization and the suburb were really taking off, did Coca-Cola start to phase out their existing vending machine line and replace it with machines capable of charging more than 5 cents per bottle.

Of course, we all know how coke machines operate now. Computerized bill changers, many of them hooked up to the internet, allow Coke to charge upwards of $3 for a 20oz beverage on a hot day at a theme park. Coke even attempted (in 2005) to fluctuate the price of Coke based on local weather conditions. People would want a Coke more on a hot summer day, so why not charge more for it? (Because the public backlash was severe to the point where boycotts were suggested the very same day Coke announced their new plan, but that’s another story.)

The fast food problem Shawn mentioned, as well as the vending machine problem, is why so many companies are moving onto the web. Online retail is exploding at a rate that can be described as a “barely controlled Bubble.” To tie back in with my comments on the fast food restaurant, this means that all your customers see the exact same website, written by the exact same piece of code. Want to change the way orders are displayed? Well simply alter the order display page, and every customer in every country from now on will see that new display format.

This doesn’t just apply to retail, however. Many companies are moving towards web-based internal pages. When I got my mortgage, the load officer entered all my information into a web form on their intranet. This is brilliant, because it takes away all the cost of synchronizing the employee computers with the software, it removes the time needed for upgrades, and (most importantly) it means developers don’t have to come into the office at 4am to ensure that upgrades go smoothly before the start of the business day. So any of you business owners out there that have had to deal with the nightmare of upgrading antiquated POS software on dozens, hundreds, or hundreds of thousands of computers, consider making everything a web site.

SoftLayer has geographically diverse data centers, so your stores can always log in to a nearby servers to cut down on latency, and we allow for VPN access, distributed databases, and real-time backups, making a web-based solution preferable to even the hard coded local systems that many stores use now.

-Daniel

Categories: 
May 23, 2007

Who is SamF?

Since this is my first blog post, I thought I would take the time to introduce myself and explain my role here at SoftLayer. That way, if you wind up reading any future posts, your first question won’t be “who is this guy and why do I care?”

Like many of you, I’ve been in this business for quite some time. My first job in the industry was back in 1992 when I was working with the CIS department at Texas A&M helping to manage the university Gopher system. I remember going around campus to the various departments helping to convince people that putting information online in Gopher was the end-all/be-all for sharing information. Of course, that evangelizing didn't last long. Shortly after going to GopherCon '94 in Minnesota, our attention started to shift to the Mosaic browser and HTTP protocol. From there, things just steamrolled.

After A&M, I went to work for Oracle Corp where we started work on an online learning website. The goal was to take all Oracle related CBT courses and find ways to put them online under one site. This was before such things were designed for the web and it meant working with the various vendors and all the different CBT formats to find ways to get them online.

Next was an ISP / shared hosting company named Catalog.com (now known as Webhero.com). We provided all the typical Internet services including dial up access, DSL, shared hosting, domain name registration, online storefronts as well as hosting for some extremely large enterprise organizations. We did a lot with that company and it still continues on today with a pretty solid product offering and services.

From there, it was into the enterprise datacenter hosting and dedicated server hosting markets. Now it's all about SoftLayer and the services we can provide customers with our latest and greatest infrastructure.

As COO at SoftLayer, I am basically in charge of day to day operations including support, facilities management, internal systems infrastructure and anything else that gets dreamed up on a daily basis. What's the funnest part of my job? Every bit of it! I love the daily challenges in the support group. Facilities planning and forecasting allow me to really dig into the numbers. And, since I originally started out as a developer and system administrator, I love being involved with internal systems. Now at this point, I’ve got to be honest; we've got some really good people here at SoftLayer that do all of the dirty work (the actual fun stuff), but I do get to stay involved in all of it. However, because these guys are so good at what they do, I don't have to lose sleep over any one particular thing – instead, I get to stay involved in every piece of it. Maybe in future posts I’ll explain how we determine the number of chassis fans that go inside each server (over 35,000 chassis fans in production so far) or how many different types of SAS and SATA cables we need with how many different types of connectors (so many of differing types that it eventually became cheaper and more efficient to just have them custom made), where to put all of these servers, etc.

I guess the point of all that was to introduce myself and to let you know - having been in the industry for so long now and having dealt with everything from Gopher to dial up access to enterprise hosting to being in the dedicated server market now for quite a while, I feel I have a pretty decent understanding of what our customers are looking for and what their pain points are. While overall operations are critical for everyone, enterprise customers running CRM apps, file servers and domain controllers view things from a different standpoint than someone running a personal mail server or even a large shared hosting or VPS business. As I read through tickets on a daily basis, I try to put myself back in the customers’ shoes to make sure that the services we provide cover the needs of all the different types of customers we have. Having been a customer or provider at pretty much every level, I certainly understand the challenges many of you face on a regular basis. It’s our job to help you overcome as many of those as possible.

We have a lot of really cool things going on at SoftLayer and I hope to share some of those in future posts. In my next post, I’ll tell you all about Truck Day at SoftLayer.

-SamF

Subscribe to systems