Posts Tagged ‘management’

Recently on one of our technical forums I contributed to a discussion about the Windows operating system. One of our director’s saw the post and thought it might be of interest to readers of the InnerLayer as well. The post focused on the pros and cons of Windows 2008 from the viewpoint of a systems / driver engineer (aka me). If you have no technical background, or interest in Microsoft operating system offerings, what follows probably will not be of interest to you—just the same, here is my two cents.

Microsoft is no different than any other developer when it comes to writing software–they get better with each iteration. There is not a person out there who would argue that the world of home computers would have been better off if none of us ever progressed beyond MS-DOS 1.0. Not to say there is anything wrong with MS-DOS. I love it. And still use it occasionally doing embedded work. But my point is that while there have certainly been some false starts along the way (can you say BOB), Microsoft’s operating systems generally get better with each release.

So why not go out and update everything the day the latest and greatest OS hits the shelves? Because as most of you know, there are bugs that have to get worked out. To add to that, the more complex the OS gets, the more bugs there are and the more time it takes to shake those bugs out. Windows Server 2008 is no different. In my experience there are still a number of troublesome issues with W2K8 that need to be addressed. Just to name a few:

• UAC (user access control) – these are the security features that give us so much headache. I’m not saying we don’t need the added security. I’m just saying this is a new arena for MS and they still have a lot to learn. After clicking YES, I REALLY REALLY REALLY WANT TO INSTALL SAID APPLICATION for the 40th time in a day, most administrators will opt to disable UAC, thereby thwarting the added security benefits entirely. If I were running this team at MS I’d require all my developers to take a good hard look at LINUX.
• UMD (user mode drivers) – the idea of running a device driver, or a portion of a device driver, in the restricted and therefore safe user memory of the kernel is a great idea in terms of improving OS reliability. I’ve seen numbers suggesting that as many as 90% of hard OS failures are caused by faulty third-party drivers mucking around in kernel mode. However implementing user mode drivers adds some new complexities if hardware manufacturers don’t want to take a performance hit and from my experience not all hardware vendors are up to speed yet.
• Driver Verification – this to me is the most troublesome and annoying issue right now with the 64-bit only version of W2K8. Only kernel mode software that has been certified in the MS lab is allowed to execute on a production boot of the OS. Period. Since I am writing this on the SoftLayer blog, I am assuming most of you are not selecting hardware and drivers to run on your boxes. We are handling that for you. But let me tell you it’s a pain in the butt to only run third party drivers that have been through the MS quality lab. Besides not being able to run drivers we have developed in house it is impossible for us to apply a patch from even the largest of hardware vendors without waiting on that patch to get submitted to MS and then cleared for the OS. A good example was a problem we ran into with an Intel Enet driver. Here at SoftLayer we found a bug in the driver and after a lot of back and forth with Intel’s Engineers we had a fix in hand. But that fix could not be applied to the W2K8 64-bit boxes until weeks later when the fix finally made it from Intel to MS and back to Intel and us again. Very frustrating.

Okay, so now that you see some of the reasons NOT to use MS Windows Server 2008 what are some of the reasons it’s at least worth taking a look at? Well here are just a few that I know of from some of the work I have done keeping up to speed with the latest driver model.

• Improved Memory Management – W2K8 issues fewer and larger disk I/O’s than its 2003 predecessor. This applies to standard disk fetching, but also paging and even read-aheads. On Windows 2003 it is not uncommon for disk writes to happen in blocks < 64KB and certainly never more than 64KB (this was a limitation that had been in place since Windows NT). On W2K8 the memory manager frequently handles writes as large as a full MB. Furthermore, writing related data in large blocks (especially during paging) significantly reduces fragmentation vastly improving read back times as well.
• Improved Data Reliability – Everyone knows how painful disk corruption can be. And everyone knows taking a server offline on a regular basis to run chkdsk and repair disk corruption is slow. One of the ideal improvements in terms of administering a websever is that W2K8 employs a technology called NTFS self-healing. This new feature built into the file system detects disk corruption on the fly and quarantines that sector, allowing system worker-threads to execute chkdsk like repairs on the corrupted area without taking the rest of the volume offline.
• Scalability – The W2K8 kernel introduces a number of streamlining factors that greatly enhance system wide performance. A minor but significant change to the operating system’s low level timer code, combined with new I/O completion handling, and more efficient thread pool, offer marked improvement on load-heavy server applications. I have read documentation supporting claims that the minimization in CPU synchronization alone results directly in a 30% gain on the number of concurrent Windows 2008 users over 2003. That’s not to say once you throw in all the added security and take the user mode driver hit you won’t be looking at 2003 speeds. I’m just pointing out hard kernel-level improvements that can be directly quantified by multiplying your resources against the number of saved CPU cycles.

Alright, no need to beat a dead horse. My hope was if nothing else to muddy the waters a bit. The majority of posts I read on our internal forums seemed to recommend avoiding W2K8 like the plague. I’m only suggesting while it is certainly not perfect, there are some benefits to at least taking it for a test drive. Besides, with SoftLayer’s handy dandy portal driven OS deployment, in the amount of time it took you to read all my rambling you might have already installed Windows Server 2008 and tried it out for yourself. Okay, maybe that’s a bit of an exaggeration. But still…you get the idea!

-William

Ok, the title of this blog may sound funny but you would be surprised how many phone calls I get about that very subject. Sure it’s not that specific case every time, sometimes it’s a software issue, other times hardware. But in the end not being able to access your server is the worst feeling in the world.

Enter KVM over IP. (Also known as Keyboard-Video-Mouse)

Yes boys and girls, this wonderful feature provided on all mid to high-performance multi-core servers can be your best friend in a time of need. While on a routine support call, a customer of mine stated the server was blocking not only himself but a lot of his customers. I kept a level head and told him it was no problem. He paused for a moment then let me know just how big a deal it was, while he was explaining I promptly used the KVM to login to his server and shutdown the firewall. All of a sudden he stopped talking and said “It’s working!”, “What did you do?” I explained to him how KVM works just as if you were hooking up a console to your server, and can be used even if your public Ethernet cable is unplugged. I went on to show him where it was in his home portal and how all of this was given to him for free. Also I explained the issue had been fixed from my desk without ever having access to either the public or private ports on his server. The customer had never heard of such a feature and was amazed at how easy it was to use.

The beauty of KVM over IP is it removes the one thing many server owners dread, not being able to be in the data center when issues arise with their standard connection methods (RDP, SSH). With KVM over IP we are giving the customer a solution to that problem. Via KVM you can login to the management interface card, which in most cases resides on an entirely different network, and within seconds you will have access to your terminal as if you were standing right there in the datacenter!!! Not only can you connect to your server, you can manually power it on/off and also reboot your server all within the same management screen. Beyond server access you can monitor temperature readings as well as fan speeds in the server. The KVM card is a HUGE tool in any Softlayer customers’ toolbox and one that we in the Operations Team use often.

Here at Softlayer we are always thinking about how to make your business easier to run, whether it be implementing global services such as CDN, or making sure our customers have basic access to their server in the event of a crisis. Since starting my career here at Softlayer and learning of the KVM feature I’ve made it a point to inform the customer of the KVM interface along with all features that are offered to them (and believe me they never stop coming!) so be sure and check our announcements page because you never know what we will come out with next!

-Romeo

So…I was just promoted to a management position after serving SoftLayer’s customers as a CSA for 15 months. “Things look different from up here!” Moohahahaaa. Anyway, I find it to be extremely interesting to see how our support works from a new perspective. When you are in the trenches as a CSA, it is very hard to see the big picture as you are on the phone, working diligently on a ticket (or ten tickets), or completing a plethora of other tasks from the moment you arrive for your shift until an hour or so after your shift officially ends most days. You become so involved in the specific issues you are working and the customers with whom you are dealing, you are hard-pressed to step back and see the effect of your team’s efforts on the customers who were served during your shift. Honestly, some days you are simply glad to get to go home and rest your brain for a while. Other days, you leave with a great sense of satisfaction in the fact that you conquered several difficult issues and made a lot of customers, as well as their customers, very happy.

As a manager, you have the privilege of seeing the different talents, abilities, and areas of specialized knowledge of a team of great technicians come together to create an outstanding support department for our customers. As we support such a tremendously wide range of issues, it would be impossible for any one tech, or even two techs, to have all the answers. But, here at SoftLayer, the egos seem to be left at the door and there is a meeting of the minds and a cooperation among peers that may rival the acropolis (well, maybe not…different clothing anyway ). But seriously, the techs do band together to find resolutions to difficult issues and therefore, a customer can rest assured that the issue is being dealt with by a consortium of scholars, if you will.

Before becoming a part of the management team, I frequently heard my co-workers talk about how nice it was to work in a place that was free from the drudgery of politics in which most work-places are buried. The management team has done a very good job of keeping it down to business and absent of drama and red tape. The opportunities to advance are many as I can attest. Among the CSA teams, there is a sense of purpose and camaraderie that foster the great support that our customer’s enjoy. Of course, it is not mount Olympus (speaking of drama) as no place is perfect, but if you stopped one of SoftLayer’s CSA’s on the street, I’m sure you would find someone who enjoys going to work and serving their customers. The excitement of a fast-growing company with many opportunities for hard-working technicians makes for a positive, success-driven, committed environment. SoftLayer just keeps getting better and better, both for the customers, and the employees.

-David

A couple weeks ago, I made a quick stop by my friendly neighborhood drug store to pick up some film for my camera. When I came out of the store, I hopped in my car, turned the key, and then… BOOM! That’s right—there was an explosion. Nothing earth shattering, but alarming none-the-less. The explosion was certainly loud enough to turn a few heads. And it gave me a bit of a scare as there was a moment where I found myself wondering if my ex wife had finally saved up enough money to have me taken out.

After giving the smoke a few minutes to clear out, and my heart a few beats to find its way out of my throat and back into my chest, I got out and looked under the hood. Not because I know anything about what makes a car tick, but because looking under hoods is what guys do. Especially when people are watching. In fact, another guy nearby came over and looked under the hood too. And as soon as he opened his mouth I realized that like me, he was only looking under the hood because that is what guys do in these situations.

“I don’t think your battery is supposed to have a big crack down the middle like that,” he said to me.

“Me neither,” I mumbled. I suppressed the urge to ask Mr. Obvious where he went to automotive school.

“You try it again?” he asked me.

For a second I thought he was joking. Then I realized he wasn’t and more importantly that I didn’t have any better ideas. So I hunted around the engine block until I found where the top of the battery case had been propelled, just sort of laid it over the smoking remnants of the battery, then slid back into the car and tried the key. It didn’t start. When I turned the key all the lights came on, all the dials and gauges swung wildly from side to side, and then everything went dark. Lights out. I tried again. But this time there weren’t even any lights. My sporty Mazda 6 might as well have turned into a pumpkin for all the good it was going to be getting me home. So much for zoom, zoom, zoom.

“You might need to call for a tow,” said Nostradamus now standing by my car window with his hands in his pockets.

“Thanks again,” I said unenthusiastically, “I got it from here.” I hoped he’d take the hint.

He did.

I waited till I saw him drive off then tried the key one more time. Nothing. So I broke down and called the towing service. Rather than calling a friend or a taxi, I opted to walk the two miles home from the drug store. During which I had some time to think. It occurred to me that the idea I might start up my car and drive it to the nearest shop for diagnostics after it just got done blowing up in my face was pretty outlandish. And yet, that is exactly what we offer with the servers we sell at SoftLayer.

Lights Out Management (LOM) or Out Of Band Management (OOB) as it is sometimes called is a feature we include with all of our servers at SoftLayer. If you’re a current customer, you have probably noticed the “management ip address” noted for your servers in our portal. That’s exactly what I’m referring to. And while LOM is the stuff of science fiction in automobiles, in our world class servers it’s a reality. That’s right, with our OOB offerings you can:

• Cycle the power on a server even if the operating system has crashed, locked, or otherwise blown up.
• Start up a dead server with the push of a button.
• Get critical readings of system health indicators like processor temperature and fan speeds, regardless of what operating system or software you have installed on that server.
• Manipulate system BIOS and perform diagnostics remotely with full video, keyboard, and mouse support.
• And a whole slew of other things that will make your life much much easier.

In essence, SoftLayer’s OOB management features are the next best thing to driving to one of our data centers and plugging a keyboard into your server. Maybe even better, since you don’t have to fight the traffic. It’s the sort of thing a system administrator dreams of. The sort of thing that sets SoftLayer apart from the myriad of other hosting companies out there.

As for my car, two days and two hundred dollars later it was back in my driveway. Apparently the mechanic who worked on my vehicle went to the same school of engineering as the Good Samaritan who provided me so much insight two days prior. The diagnosis, according to the invoice, was: cracked battery. Then just to give me a warm and fuzzy written in big block letters at the bottom of the page was:

IF THIS HAPPENS AGAIN BRING IT BACK IN

I can’t wait till the automotive industry catches up to SoftLayer.

-William

I have a friend who worked at an internationally recognized fast food restaurant. It relies heavily on it’s computer systems to operate. Being a programmer (I’ve always preferred Computer Alchemist), I’ve always been fascinated with the command and control programs used at these restaurants, and have drilled many family members and friends to describe (in the most non-specific non-job-endangering way) what they do with those computers to do their job.

At the fast food restaurant, every order is entered into the cash register at the front of the line, processed by the computer systems on site, the orders are relayed to the kitchen in realtime and displayed on monitors, and various thermal ticket printers spit out meal tickets to be affixed to the various food items to describe their state (Double Cheeseburger, No Pickles, No Onions, Add Secret Sauce). It’s an amazing dance of software.

But I noticed that my friend the grill cook was constantly complaining about the system. Apparently, the order processing is really real time. When the guy up front presses the “double cheeseburger” button, it immediately lights up in the back but it doesn’t alert you that this is an order in progress! So, if the cooks are in a rush and just preparing orders as fast as possible, they might already have the pickles on the burger before the lady up front presses the “No Pickles” button, updating the display. Also, updated items are NOT IDENTIFIED by the system, so if the burger is already wrapped and the burger man is already moving to the next little square, it might not be caught! (Keep this in mind: most order “errors” at this internationally recognized fast food restaurant are caused by slow order takers, or slow customers, or customers who change their mind at the end of an order (Oh, could you make that burger with no onions?). Remember, if the guys in the back are rushing, get your order together in your head before you speak. If you say “Double Cheeseburger, no pickles, no onions, add secret sauce” just like that, it’ll pop into the system the right way!)

My question, of course, was “Why not rewrite the cash registers to be more awesome?” See, the old registers tied into a small computer in the back room that tied into the monitors in the kitchen. All state was held on that one machine, so obviously as soon as it was updated, it would appear on the monitors. But I couldn’t figure out why they kept using this stupid backwards system. It’s 2008! When I worked in the local greasepit in my small town, we used paper tickets we were more efficient than this fast food restaurant’s system! So why haven’t they updated? Simple: the people who can fix it aren’t there to use it.

Aha! That’s what makes SoftLayer so awesome. See, our programming staff is right here, in the same office as everyone else (except for those people who work in our datacenters, but they have our internal IM, email, and phone numbers and know they can contact us at any time.). If something breaks, or if something is built in a backwards or strange way, we know immediately and can turn around and fix it. In fact, we have a whole lot of programmers, as a ratio of developers to normal people. This high ratio allows us to have the best control portal in the world, and to add features quickly. See, if only developers worked at that fast food restaurant, and saw how the system was used, and were allowed to make changes (another awesome bit about SoftLayer: management is open to change. You’d be surprised how many obvious changes simply are not allowed by management in some dev shops…), then it would be a much easier place for everyone involved. And maybe I’d get my Double Cheeseburger right. Without the onions.

-Shawn

What language do we speak here at Softlayer? What language is spoken across the industry? Is it the same, or does everyone have their own code for translation?

It seems that in the “on demand datacenter industry”, “hosting industry”, “dedicated server industry” (or whatever you prefer to call it) each company or provider has its own idea and way of projecting who they are. These projections are seldom in line with one another and have a slight difference only to give some idea of separation.

The biggest grey area that I have seen and something that gets distorted is the idea of managed services and just the term “management” in general seems to have lost any kind of universal meaning. The thing that I run into most is when a customer asks us if we are a managed company. I find this to be a loaded question knowing what I know of the industry and other providers. The reason is that you can call your service anything you want to and even come up with clever and creative names for it, but at the end of the day creative marketing doesn’t get results when the rubber meets the road.

It is imperative that the correct expectations are set so that customers aren’t lead astray and find themselves in a situation that they were unprepared for because they were disillusioned by gimmicky wording. Softlayer has the reputation of being an honorable company and I am proud to be a part of that. We do not consider ourselves to be a managed service, but we do offer support and help in many situations. We have support staff here 24 hours every day of the year that can help you, or possibly help point you in a good direction for you to be able to help yourself. We offer OS updates and patches at no additional charge automatically. We strive to be as upfront and fair on everything from the bottom to the top, so I feel that it is my duty to explain this situation in more detail. In fact, all of our sales representatives and many of our customers have felt this same way. What I have found is that this upfront and honest explanation is a rarity and that many companies are not as forthcoming as they could be or should be when discussing “managed services”.

There are a lot of companies who provide the exact same services as we do and call this being “fully managed”. There are others who promise the moon and a shoe shine with their service and then just come up short on the efforts. Now, this may not be entirely their fault because they may have excellent intentions, but a poor and impractical business model. However, intentions don’t get results and customers are the ones who pay most for the misrepresentation. The worst situation is when someone pays a huge premium for a service that is overpromised and then severely under delivered. When I am able to talk to someone who has been in this situation they really appreciate the options and control that they have over every aspect of their service when choosing Softlayer as their provider. The only satisfaction I can get is helping people that have been taken advantage of find a provider that they know they can count on and exactly what they can expect.

This elaboration of services extends beyond server management. No matter what buzz words a company may want to use to describe your company (“largest”, “best”, “heroic”, “ultimate”, really just fill in the blank on this if you’ve been around long enough.) the main thing that matters in this industry is functionality. I am confident in saying that no other company can offer anything close to what Softlayer can provide. Softlayer provides options and capabilities which are unparalleled in the industry in order to give customers complete control over their hardware and thus their own business. There are some that have tried to copy our model and others who have tried to produce a stop-gap solution between what they offer and what we offer, but they have failed. Is this, perhaps, a key reason why we have been able to sustain our high level of growth and remain stable? Possibly. Is this a sign that the best is yet to come with Softlayer? Definitely.

If you want to talk about situations or projects you may have coming up, I would be happy to speak with you and help come up with a solution that will maximize your businesses potential. In fact my entire team is here for this specific purpose.

-Doug

As I have started to settle into my new role with SoftLayer, we have spent a lot of time meeting with various vendors and partners to discuss our overall vision and plan for 2008. In almost every one of those meetings we get to the same question: “What does a typical SoftLayer customer look like?” Or, the other version of that: “What is SoftLayer’s target customer?”

You would think this should be an easy question to answer. After all, we have over 4,500 loyal customers that rely on us each and every day to deliver on-demand, world class IT infrastructure. Surely, there must be some common thread among these customers. Being responsible for “Strategy & Marketing” I decided to look into this to come up with a standard reply to that question. The standard ways to do this from a marketing text book (i.e. “in theory”) perspective include:

• Industry – financial, manufacturing, retail, distribution, etc.
• Geography – typically regions within a country, or countries themselves
• Customer Size – normally based on revenue or employees
• “Retail” or “Wholesale” – are we selling to the final consumer of our products or to a reseller

The next step — look at our customer database and start to build up a profile based on those criteria. A relatively simple process, but the problem we found was that the four metrics above did not adequately define any of our customers. Some examples:

• Industry — we serve all possible combinations of traditional and new industry classifications; from large manufacturing, to Web 2.0 start ups and no single segment is more than 5% of our business
• Geography – we have customers in over 100 countries. Even in the US our customers come from every corner and every state in the country
• Customer Size – 1 employee to 50,000+ employees and everything in between; $0 in revenue to $10 billion and more
• Retail and Wholesale – almost an even split between the two groups

Being inquisitive by nature, I could not let this end with an answer like: “we have a very diverse set of customers that represent all industries, all geographies, and all customer size categories.” It did lead me down a path to start asking customers some questions like:

• Is IT infrastructure a critical component of your business?
• Do you need highly scalable IT to adjust for seasonality or growth in your business?
• Do you want a simple and flexible management tool to allow complete control of your data center infrastructure?
• Are enterprise grade solutions of value to your business, but something you cannot afford?
• Are you looking for innovative solutions to help drive your business forward?
• Do you value standards based processes and controls?

To steal a quote from a very, very distant relative… Eureka! While this might not be as significant a discovery as the wheel, fire, or the Archimedes’ screw, it did finally bring some clarity to our little customer debate. The vast majority of our customers answered “yes” to many or all of those questions. It also led me to understand what our customers do not want from us:

• Specialized application support
• Highly custom solutions that scale poorly
• Up-front fees and long term contract commitments

The net result is that our customers are segmented very differently than traditional methods would suggest. They are clustered around a common need that spans across all demographics. The customers that come to us are looking for a very special thing – the SoftLayer approach to IT management. If you belong to a company that can resonate with the questions above, you have come to the right place.

-@gkdog

I’m spoiled. Really, really spoiled. I have a test lab full of servers to play with about thirty paces away from my office. Most of them have KVM over IP on a daughtercard. When I need to jam an OS on a server or manage to lock myself out by screwing up a network config, do you think I stand up and take a short walk? Nope. I fire up the KVM/IP and take care of business from my comfy office chair.

Let’s see how old the audience is. Raise your hand if you ever had to yell into a phone telling a datacenter tech what to type.

“’S’ as in Sam, ‘H’ as in Harry, ‘O’ as in Oscar, ‘W’ as in Wally, SPACE, ‘D’ as in David, ‘E’ as in Edward, ‘V’ as in Victor, ‘I’ as in Isabel, ‘C’ as in Charlie, ‘E’ as in Edward, ENTER” (extra credit to whoever can name the OS without using a search engine or reading ahead).

For some of you this is a recent event, but there will come a day when our IT generation can regale the youngsters with stories of “When I first started in IT, we didn’t have this fancy KVM stuff you kids have today…”.

KVM over IP isn’t exactly brand new. It has been around for a few years starting with external devices hanging off the back of the server. But it is becoming much more common to find daughtercards from your favorite motherboard manufacturer with this capability. The motherboard suppliers have already added other server control technologies like IPMI and iAMT to the motherboard. I wonder how long until KVM over IP makes the jump from the optional daughtercard to coming standard on the motherboard? I’ll bet we’ll see it before you can spell VMS.

-@nday91