customer-service

February 13, 2008

The Usage Of Complex Algorithms For Password Generation

Passwords are difficult. On the first hand, you want to create a password that uncrackable by anyone, lest they be teenage hackers or CSI experts with magical hacking tools. On the other hand, the password has be rememberable by you yourself, lest only teenage hakcers and CSI experts with magical hacking tools are able to access your data.

So, how do you make passwords?

One of the more secure ways are to use a random letter generator, like random.org, to build random strings, pick one, and memorize it. It's pretty secure (random.org uses real random noise to produce it's random numbers)and with seven random alphanumeric characters, the password search space is about 2.2 trillion combinations! But are you really going to remember "QRSr0Fu" or "W96TUON" two weeks from now? (My generated set had "myELlRK" which I might be able to remember...) If you type your password every hour or so, you might remember this by muscle memory pretty quick. Just in time to have to change it, I bet.

Another way is to take a word or phrase, turn some letters into |33+sp34k, and you get something more random, but much more rememberable. So, for example, "minivan" becomes "m1n1v4n!" and "washington" becomes "w4sh1ngt0n!?!" These are actually quite rememberable; the use of non-standard characters disallows the use of rainbow tables and dictionary attacks, so they're much less suseptable to cracking. However, what happens when you forget the "!", or that "Washington" gets "?!?" or that you did NOT turn "t" into "+"? You could end up going through a few cycles trying to "guess" your own password. Again, if you use it all the time, you'll learn by muscle memory. And this lets you come up with some cool passwords, like "c4p+41nK1rk". How can you beat that?

My favorite way, however, lets you write your password down in plain sight. I tend to cycle through passwords, and if you're anything like me you have two online banking passwords, four credit card or loan company passwords, a work domain password, 6 email passwords, a home log in password, etc, etc, etc. If you take the easy way out and use the same password everywhere, you end up making kittens and security experts cry. If, however, you have a completely separate randomized combination for each account, your brain will get stuck in an infinite loop. Using this method, you get to write down your passwords and tack them to the wall. Or put 'em on a sticky note. In plain sight. Email them to yourself without a care. It uses a special type of encryption to keep your password safe. Not AES or DES or TEA or other TLAs. I call this "Hippocampy Encryption" (named in honor of the part of the brain that does memory type activities).

The key is to write down a set of clues that will tell you (but only you) what your password is. You can add symbols to help you remember what kind of encoding to use for your password. Here's a password I just made up right now as an example:


Shawn's rival ^
shout your home team
Esirpretne
Sam.

Because everything on this note is simply a hint for your specific brain to recall a password, it's specific to you. Hints don't even have to have anything to do with the subject. The hint "Red October" could tell you the word "fortworth", whereas for me, I'd be trying "R4M1US", "M1SSL3S", "jackryan", "TomClancy", etc. You can string three or four hints together for a password. Note, these create long passwords, and your coworkers may start to believe that you have a superhuman capacity for memorizing long strings of randomized data. Do not do anything to dissuade them from this belief. And, because the hints point to common words and numbers already lodged in your grey matter, you may be suprised just how fast you type in that 20 character long password. Compared to my speed on 7 character random strings, it's blazing.

And due to the pattern matching ability of your brain, remembering the passwords are easy. Lets say you've written your clue on the back of one of your business cards, so you have it handy if you need it. After a few days, just SEEING a business card will bring your new password to the front of your mind. After a while, you'll stop needing your hint sheet, as you'll just remember the password. And when it comes time to change your password, shred your card and your postit, post a new one (in a different color if you can, helps the brain), and give yourself a few days. Unlike scrawling your random digits on a paper or card, even if somebody stole your "Hippocampically Encoded" card, they would have to REALLY know you (or be a really good guesser) to get the password. Even with your card, you've reduced them to brute searching. And if your card/note turns up missing, it takes about 30 seconds to whip up a new hint sheet. Not only is your attacker brute forcing your hint sheet, but it's the wrong hint sheet anyway!

So... have you guessed my password above? It's GARYkemp!1071Max. 'Course, you'd only know that if you knew that I played Pokemon and left my rival's name at default, that I decided that "^" meant "Make it all uppercase", that my home team is the Kemp High School (and that I was talking high school football), that by "Shout" I meant "give it an exclamation point", but that the whole word should be lower case (because the hint is), that Esirpretne is "Enterprise" backwards, and that I meant to make the serial numbers backwards (but not the NCC part), and that by Sam (a very common name) I meant "Give me the name of Sam's partner in that incredibly funny cartoon by Steve Purcell, Sam and Max: Freelance Police." The period is just decoration. If you did guess it, contact the NSA. I hear they're hiring people like you.

-Zoey

February 11, 2008

Spares at the Ready

In Steve's last post he talked about the logic of outsourcing. The rationale included the cost of redundant internet connections, the cost of the server, UPS, small AC, etc. He covers a lot of good reasons to get the server out of the broom closet and into a real datacenter. However, I would like to add one more often over looked component to that argument: the Spares Kit.

Let's say that you do purchase your own server and you set it up in the broom closet (or a real datacenter for that matter) and you get the necessary power, cooling and internet connectivity for it. What about spare parts?

If you lose a hard drive on that server, do you have a spare one available for replacement? Maybe so - that's a common part with mechanical features that is liable to fail - so you might have that covered. Not only do you have a spare drive, the server is configured with some level of RAID so you're probably well covered there.

What if that RAID card fails? It happens - and it happens with all different brands of cards.

What about RAM? Do you keep a spare RAM DIMM handy or if you see failures on one stick, do you just plan to remove it and run with less RAM until you can get more on site? The application might run slower because it's memory starved or because now your memory is not interleaved - but that might be a risk you are willing to take.

How about a power supply? Do you keep an extra one of those handy? Maybe you keep a spare. Or, you have dual power supplies. Are those power supplies plugged into separate power strips on separate circuits backed up by separate UPSs?

What if the NIC on the motherboard gets flaky or goes out completely? Do you keep a spare motherboard handy?

If you rely on out of band management of your server via an IPMI, Lights Out or DRAC card - what happens if that card goes bad while you're on vacation?

Even if you have all necessary spare parts for your server or you have multiple servers in a load balanced configuration inside the broom closet; what happens if you lose your switch or your load balancer or your router or your... What happens if that little AC you purchased shuts down on Friday night and the broom closet heats up all weekend until the server overheats? Do you have temperature sensors in the closet that are configured to send you an alert - so that now you have to drive back to the office to empty the water pail of the spot cooler?

You might think that some of these scenarios are a bit far fetched but I can certainly assure you that they're not. At SoftLayer, we have spares of everything. We maintain hundreds of servers in inventory at all times, we maintain a completely stocked inventory room full of critical components, and we staff it all 24/7 and back it all up with a 4 hour SLA.

Some people do have all of their bases covered. Some people are willing to take a chance, and even if you convince your employer that it's ok to take those chances, how do you think the boss will respond when something actually happens and critical services are offline?

-SamF

February 8, 2008

Outsource It: Part II

Wow, I like all of this feedback guys! Really! I had been chewing on that blog for a while. I was basically trying to decide how to write it and apparently the format worked and got some juices flowing on our forums. I was going to post this on the Forums but I think it is a bit too long and isn't using the forums standards. So here is my follow up to TheRabbit in Blog format.

A bit about me; I am an old guy (shh don't tell the guys I play Racquetball with) and I have been in LOTS of different companies of various sizes and types of business. Back when the internet was young and dial-up was the name of the game, I played in that field. In fact, I see a lot of familiar faces here every day. They all stayed in that field and honed their skills and are the guts behind SoftLayer today.

I went out into the world to see what it was all about. I decided I wanted to be technical and since I was a Windows guy it would have to be Microsoft. So I took the tests and got my MCSE and then worked for Alliance Data Systems, a Cargo Airline, A college in Dallas, Cement Company, and a small Outsourced IT company, then I met back up with these guys and here I sit.

So I used some of my experiences with all of those places to write the last blog. Here are a few of those experiences so you can see where it came from.

Alliance Data Systems had great DC's and lots of cash, they didn't need to outsource because they spent the money to do things correctly and had their own raised floor DC's and connectivity, etc. It was a cool place to work and I learned quite a bit. They did things right.

Cargo Airline - Well they tried. We built out a new office building at the airport and we had an office with no carpet, and extra cooling for our server "room". We had some old boat anchor HP equipment and a single IBM server for the JD Edwards accounting box and boy was it slow. We were using Windows 2000 with AD and DHCP to hand out IP's. Funny story, we merged with a really "smart" software company and part of the merger was that the powers from that company got the reigns and could run our IS department. Maybe they are reading this... (evil grin) - So the first thing they did was pulled DHCP out of the mix and went all static IP's because they were easier to track. "You can just enter them in a spreadsheet!" I was told. "Then you know that a 10.x.1.x is accounting, and a 10.x.2.x is sales, etc, etc." I still laugh about that decision today. Ok, back to the real subject. This company didn't spend the kind of money needed to have a good core of systems, and network and therefore the applications suffered. Most of the apps they wrote or used were Web apps and could have been housed in an outsourced facility.

College in Dallas - Believe it or not, the college had some pretty cool DC's on the Campus. They were secure and if I forgot my jacket I froze my butt off. They used Compaq 1u's like sliced bread. Server after server for student access, student records and it was all Citrix apps that students and faculty could connect to. To me it SCREAMED outsource. Think of the electric bills they paid to freeze my butt off, think of the purchasing department that had to buy all those machines. Think of how much they paid me to un-box those servers and rack them, and cable them, and install the OS from CD, and install Citrix and the apps. Then the accounting department had to track them and make sure they were paid for and depreciate them. Granted, even if they outsourced them the purchasing group still has to order them online and the accounting department has to give us a Visa but that is the extent of it. We have Truck days of joy and do all the manual labor for you and we automate the OS install. Then it is just down to the Tech installing Citrix and the apps from the comfort of his desk remotely.

Cement Company, one of my favorite places to work. I was in charge of the Citrix farm, Exchange and RightFax. Oh what fun. They had over 40 home built apps that ran on Citrix. We had 3 DC's, Dallas, Midlothian, and Virginia. They were Top of the line! If you were a rat and liked chewing through cables and you are into Liebert cooling systems from the early 60's! Ok, it might not have been the 60's but they were old. The DC in Midlothian was the best. We finally boarded up the windows facing west because we figured a lot of the extra heat was due to the Texas sun baking them. Ok, funny story #2. While un-boxing and racking a few Dell 1U servers (again they paid me a pretty good salary for my Citrix and Exchange skills, and here I am un-boxing and racking again) my helper decided that it was time to drop test a Dell. I was behind the rack and there was really nothing I could do except watch this brand new Dell server go crashing to the floor from above his head. After reseeding all the cards, CPU, and memory, we crossed our fingers and it fired up. It was a bit warped and bent but we strategically jammed it in between 2 straight servers and it took some of the flex out of the bent box and it worked great, might even still be working today. As you can tell some outsourcing by them would be good as well; Even if it is just the Development and test systems. We lined up like ants at a sugar sack begging for servers for Dev and Test but they were NEVER in the budget. Another great point I think, Capital Expense vs Monthly Expense. For a huge company it is MUCH easier to get them to sign off on a monthly expense.

Outsourced IT - Here is the one that wins it all. My job was to go around Dallas to small and medium sized businesses and be their IT guy. My main focus of course was Citrix and Exchange but you just never knew what you were going to walk in on. One plumbing company had their servers in a barn. An auto parts supplier had theirs in the back of a storage building behind the restroom. Use your imagination. But the ones that got me the most were Doctors offices. Broom Closets, Office Managers offices, just in the hall out in the open, you name it and I saw servers there. Most of the offices already had a T1 in place so connectivity wasn't the real issue. An interesting point is that I always had to sign a Hipaa form to be legal to work on the systems. It amazed me that these systems were so accessible to anyone that might have had access to the office. I wonder if the maid service had to sign Hipaa forms since the servers were right in the open. Sometimes right behind the trash cans. 90% of the medical software I came in contact with was WEB software which is easily outsourceable. And the number 1 complaint I heard from office managers and Doctors was, "I want to connect from home. Can you help me?" So of course we would setup remote access. But it never failed. During Storms they would lose power or connectivity. Or the building power would drop for construction, or a car would hit a pole. There were always issues. I swayed a few high tech Docs to finally consider and try outsourcing and they loved it. A few even use thin clients in the office now and everything happens in a DC. They love it.

I still say that no matter what size business you have OUTSOURCE IT! Maybe not all of it, but for DEV and Test, a hot site AD controller, Web App Servers, Giant DB Servers that live behind those web app servers, Web Farms...etc be the ball and give it a try. We won't argue!

-Skinman

February 6, 2008

Big Business is Messing with my Caffeine Fix!

Most of the posts here have a technical spin, and well deservedly, but this one is a little different. This is my version of an open letter to the CEO of Starbucks and any other CEO out there who is messing with my daily life by putting the idea of "conquering the world", over the needs of the people that will get them there -- the consumer. So here goes my rant:

Dear Mr. Starbuck's (aka Howard Schultz),
As a frequent patron of your fine establishment for many years, covering many locations in multiple states and multiple countries, I need to ask a personal favor. PLEASE stop messing with me, with the idea of me being anyone other than 'big business'. See I live in Dallas, Texas and this past weekend I had the urge to actually forego my normal $5 cup o' java at any of the 50 Starbucks within the 4 mile radius of my house and actually go to a local establishment that has some great beans. I wasn't going for a cup, but I was going for an actual bag of beans that I might be able to take back to the house and brew a random cup of sissy coffee (the flavored coffees that actually make hair fall off your chest, rather than put it on there like the SENOMA blend from the aforementioned Starbucks). BTW, for you Dallas'er's my preference for my random sissy coffee fix is a Cafe Brazil.

Location aside, I drive past the 50 Starbucks in route to the closest Cafe Brazil which is about 5 miles away. My coffee of choice is the 'Snickerdoodle', so I was thinking I will big bag it and get a pound, maybe two, to ensure my fix is completely covered. I walk in to the aroma o' joy that comes along with a coffee house. To an addict of caffeine, it's kind of like Vick's to a cold! All employees eye me and my girlfriend as we smile our way to the counter. As we are walking up, something just isn't right and we can tell immediately something is amiss. Where are the bean's that all of the other locations have? Where are the grinders? Being sure they are in the back or on the other side of the restaurant, I say with confidence, I want the biggest bag of beans I can get my hands on. The response, without a bat of an eye, was "not here sir, Starbucks forbids it!" WHAT THE $!%$? Again, the CSR at the counter say Starbucks told the landlord that they forbid anyone else in the shopping square to sell Coffee Beans to- go and went as far to tell me that he could not even pour me a cup of the coffee in a to-go cup, as per this was also forbid by Starbucks. This disappointment was seen in both of our faces and the CSR could tell that tears may be near, so the obligatory 'sorry' was thrown out with a 'can we do anything to make this right' comment?

This is unacceptable to me. As the loyal readers of theinnerlayer and all of the employees of Softlayer can attest, Caffeine in any form is like the blood through the veins of this company. Pound for pound, employee for employee, I would challenge the caffeine intake of Softlayer against any other company in the world. Pot after pot of, yes Starbuck's, is brewed hourly, if not minutely. Literally, cases of Monster are brought in weekly to support the efforts here. With the new JAVA Monster, the numbers may just fly right off the charts. Hence, the frustration

Mr. Starbucks, as one of the founders here at Softlayer I can tell you that all of us think about dominating our segment of the world, planned for it and expect it. Surrounding myself with the smartest people I have ever been around gives me a comforting feeling that all of these goals will be achieved. With the support of these smart people I refer to, we all have a standing order internally that to get to our stated goals; the idea of alienating customers by self serving goals has to nipped in the bud. We are a services company to the masses which means we believe that natural competition is healthy and that continuing to strive to build the bigger, better solution, customers will always be the winner in the equation. If we believed that exclusionary practices and pure heavy weight domination was the proper way to win, we would have thrown our money that way, but the open market allows us to stay at the top of our game, remain cutting edge and push for innovation and automation that will allow us to grow our customer base because we have a better solution for the customer, not because we don't allow the customer to have any other option. I/we may be a small fish in the pond so to speak, but I think you might be able to learn something from my statements. I'm off to get my fill of caffeine, but not sure Starbucks will be my first choice for the next short while.

Sincerely,
One un-caffeinated, unhappy customer
(Sean Charnock)

Categories: 
February 5, 2008

A Look Back Before Moving Forward and the Phenomenon of "_aaS"

Hi! I'm George, one of the newest additions to the SoftLayer team. I joined the company for a few reasons:

  1. The People - one of the best teams that I have ever met, and now, have the pleasure of working with
  2. Vision - as someone recently said, how often do you get a chance for a "do over" without being Bill Murray in Groundhog Day?
  3. Industry - how cool is it to be able to play with new technology and help shape the way people will do business
  4. Acronyms and Buzzwords - as anyone that has worked in the telecom or technology industries can attest, this is the best place to be if you want to assemble new words based simply on the first letter of each.

Even before the Internet, (did that time really exist?), telecom hardware vendors, service providers and others in this little universe loved to create acronyms to make technology sound complicated. And of course, it created employment for thousands of people, which according to many of those techies, offered no real value, to help translate all of this into something that the consumer and investment community would buy... Welcome to the world of marketing in the Internet Age!

One thing that all of these people that came to be known as "Gurus" or an even buzzier buzzword - Evangelists - learned was to standardize on at least part of the acronym. Out of this concept was born:

  • _AN (G, L, M, S, W)
  • _EO (G, L, VL, S)
  • _DSL (H, A, S, V)
  • _SP (A, M, S)
  • _2_ (B2B, B2C, P2P, M2M)

And each of these led to spin-off acronyms, like DSLAM, FRAD, ATM (the network not the cash machine), and my favorite - BE - which said a lot about what we were all doing back then. Acronyms became the patents of the original dot.com era (which we did not actually call Web 1.0, but more on that later). Of course, we also learned different naming systems and adjusted the English language to suit this purpose. Capitalization rules were thrown out along with the baby and bathwater. Capitals now appeared in the middle of all company names (yes, we even did it here...) and products thanks to another phenomenon of the era - the mandatory use of compound words.

Best of all, the digital age gave birth to an industry designed to make all of us look like techies - the acronym dictionary. Many of the generation that graduated into the telecom and Internet revolution of the early 1990s (including yours truly) built careers on the ability to string acronyms together to define the future of networking and ultimately, life. The common toolkit for all of us: Newton's Telecom Dictionary and a fluffy cloud graphic from the MS Office clip art files - that was generally used by "advanced" gurus and not something to be thrown around by anyone with less than 12 months of experience in the dot.com trenches.

The web generation also taught the masses about versions and a fundamental rule in software: avoid buying version 1.0 and never launch a product called version 0. Even the web generation of telecom -mobile carriers - figured this one out. 3G has long been touted as the utopia of communications, but we had to get through 2.5G first. Never mind that v1 and v2 seemed to actually work. Thus, versioning took its rightful place at the left-hand of acronyms.
Jumping ahead a decade or so, we are now firmly in the grips of Web 2.0 and a new set of buzzwords and acronyms. If you don't have an avatar living on a virtual street in a virtual world that spends virtual time stuck in virtual traffic driving to a virtual job, you probably are like me - part of the 1.0 generation. I too used an Apple IIe in school, but for me it was grade 10, not grade 1! And then there was the Newton which did not exactly fit into a palm, but that story will have to wait...

Like all things, buzzwords and acronyms have evolved as well. We are now using words like "grid", "utility" and "always on" to reflect the way we are connected and work. And of course we all need to be Irish for more than 1 day per year. Don't get me wrong. SoftLayer agrees that green is good. We have turned our facilities green in search of the "green". Basically, we believe the best way to help our customers with access to technology on a real-time basis is by being as operationally efficient as possible. So, we have forced ourselves to be "green" in everything that we do. But, I digress...

My favorite current acronym root is "_aaS". We can thank the failure of ASP (part of Web 1.5) and its lesser known cousin - AIP - to catch on for this latest iteration. The good news is that almost everything that we can think of can be called an "X as a Service". Our internal top 10 list includes:

  • Architecture as a Service
  • Communication as a Service
  • Data Center as a Service
  • Hardware as a Service
  • Network as a Service
  • Platform as a Service
  • Software as a Service
  • Storage as a Service
  • Virtualization as a Service
  • Widgets as a Service (the favorite of our Starbucks fan)

But, don't take my word for it. Whatever as a Service will be hard to avoid. Now that I think of it, MaaS (Monster as a Service) might be a big money maker around here.

I know we would all like to see Service as a Service, but that may need to wait until Web 4.0 when we have all been virtualized. Personally, I can't wait to see what comes next. Given our place in the universe, I think that SoftLayer will have some influence and that's why I'm here...

-@gkdog

February 4, 2008

Nnet Strikes Back

I'm not going to tell you my name for two reasons: First, I don't want a million tickets assigned to me asking if I'm crazy. Second, if I am crazy, I don't want anyone knowing it's me.

I'm not a writer myself, so I asked Shawn to write this up for me. He's a programmer, and more important a Trekkie, so he's likely to understand (and more important, believe) this story. Besides, he's written a few humorous, slightly preposterous posts for this blog, and that's very, very important.

Unlucky as I am, I was the first person to notice something strange going on. I'm a datacenter tech for the company (but I'm not going to tell you WHICH datacenter), and my job... well, I'm the power guy. I make rounds in the datacenter, checking breakers and power panels, keep an eye on voltages in the portal, that kind of thing. No power issues at the datacenter? That's because of me. So, I'm perusing the tickets and keeping an eye on things, like I should.

As I was answering a particularly interesting ticket, I received an IM from a datacenter engineer I hadn't met yet. That's not surprising; we're growing like crazy here, and I don't always get the "Welcome a new employee" email before I find myself working with the guy or gal. I finished my ticket and opened up the IM window. It was from "Nnet," and the contents caused me to leap out of my seat:

"The power strips on the new racks (205, 206, 207) are drawing too much current; it will pop the breakers in 52 minutes, 12 seconds."

I had just CHECKED those racks. I walked down to the server room, muttering about some whippersnapper of a new engineer playing a trick on me. I was going on vacation in a week, and I did NOT want any power issues; I was training another engineer to take the console while I was gone, and if anything happened during testing I would surely be called in. Anyway, I walked into the server room and checked the gauges on the power panel.

And they were drawing almost a full five amps too much. If we had turned on the third rack, the whole aisle would have gone down. That wouldn't have been too bad; no servers were hooked up. This is exactly why we test the power before we put servers in.

I and the rack crew worked for about an hour rewiring the racks, starting from the third rack. Sure enough, about 52 minutes later, rack 205 shut down. Mentally thanking "Nnet" for finding this (and more importantly, not tinkering with it before letting me know!), we got the racks wired more efficiently (they're supposed to be on separate breakers, but the electrician labeled the wires wrong), reset the breakers, and had absolutely no issues for the rest of the day.

I got back and thanked "Nnet" for finding that issue. The next day, I got to thinking about how "Nnet" had saved my vacation (I would have spent all week tracing wires to figure out what had happened), and I wanted to invite him or her to lunch. So I IMmed "Nnet" with an invitation. An hour went by with no response, but it's not too strange to have a datacenter tech away from their desk for a couple hours. So I sent an email to Nnet.

The email bounced back.

-"Mystery Author"
Maybe HR hadn't set up the email yet? So I called them up to see what was up with Nnet's email address.

That's when HR told me that nobody with the last name "Net" had been hired (I thought "Net" was a strange name for a tech, but it's not the strangest last name I've ever heard). I called the networking department to ask how I could receive a company IM from somebody who doesn't work here? They researched it and couldn't find any incoming links through our firewalls or any of the internal logs. Stranger yet, the Jabber server indeed DOES have an account for "Nnet", but the engineer who runs the server swears that he never set that up.

We were discussing this back and forth when one of the developers walked by, overhearing our conversation. He laughed, and when we asked why, he told us that he was reading a book about the human brain, and that the brain is made up of million of millions of neurons all interconnected with each other; that these interconnected neurons work together to create intelligence.

Could that be true? Absolutely not. It's preposterous. Sure, we've got tens of thousands of computers around here, dual cores and quad cores running various operating systems and applications, all connected by an incredibly fast private network...

...could it be?

The engineers are all completely sure that one of the datacenter techs must be playing a joke, and they're currently tracking it down. But I'm not too convinced. "Nnet" knew which power strips were having trouble in a room keycarded to open only for me and a hand full of other techs. And they all swear they didn't send it.

That's when I talked to Shawn. He told me that there's a lot of technically minded people out there who read fantastic science fiction stories and come up with solutions... even knowing that the tech is impossible, they can find a way to solve the problem. So we hatched up this idea to write out a fantastic blog post, an interesting narrative of my predicament.
Then we'd post it to the blog and watch for any discussion on the customer forums. Our customers are really smart, and they like solving problems. Maybe somebody out there has an idea of how we can figure out what's going on around here.

So here's the story. A completely fantastic modern day science fiction story about a sentient datacenter.

Preposterous!

...any ideas?

February 1, 2008

I Outsourced It

Have you ever wanted to tell your CIO that? His response might be, "you outsourced what?? You respond, "it!" With a perplexed look he asks again, "You outsourced what, it?" Again you respond with, "All of it." His reaction at that point could go either way. In most CIO type heads today, they can't grasp the savings associated with outsourcing and even the ones that DO understand would then have to go to the CEOs office and inform him or her that all of the company's valuable data will now be housed in a safe and secure facility off-site on dedicated servers... or "Hosted IT" even. Stop reading and go tell your CEO that right now. I'll wait...go ahead.

Ok, I see that you are back, are you still employed? We are hiring if you need a new job resumes@softlayer.com

Ok, really, how do you think that conversation would go? I have had that same conversation with ex-bosses and owners of small and medium sized businesses in the past and most of the time they don't go very well. Granted they were a few years ago so hopefully times are changing.

I have been told a few times, "no, I don't want to pay $300 per month for a server we don't own and put my data on it! That is ridiculous, just go buy me a new $3,500 server and we will put it in our local Datacenter, Server Room, Broom Closet, Bathroom, Office Managers office..." well you get my drift.

"But Sir, with this outsourced server we could easily have off-site backups, more processing power, some cool redundancy and it will not annoy everyone in the office with the loud fans and heat generation. And when we have a power outage in the office and everyone goes home for the day, they will be able to work from home because the server will still be online. Oh yeah, and our company website and email will still be functioning as well."

"Are you insane? Those challenges are so easy to overcome. We will simply add a small air conditioner to the broom closet and buy a big UPS system that will keep the server alive in the event of another power outage, and we can hire a service to come by every morning to pick up tapes and deliver them to an offsite bunker. Instead of a single connection to the internet we can buy two and have redundant connections also."

"Sir, I am no accountant, but by the time you pay someone to keep up with the depreciation of a new server, buy and install a small A/C unit and UPS unit, pay for a 2nd internet connection that will sit idle and pay a service to DRIVE here daily I really think the outsourced server would be cheaper. Not to mention in the event of data loss we could get the data restored to the server much quicker than waiting on a service to physically bring it to us." An interesting note here is, I don't care what kind of offsite data bunker you have, the Monster in Cloverfield IS going to destroy it so think multiple copies of data in multiple cities!

"Well I have made my decision; we will not be outsourcing my very valuable data - Hackers might get it, it is more secure here, so leave my office. Before you go could you please try to get my printer working again, and I am getting this annoying pop-up about spyware and it seems that my ITunes files have lost their license and I used to have a folder called Docs on my desktop with everyone's salary in it that is missing and my PDA will not sync...(zzzzzzz) -- OUTSOURCE IT!

-Skinman

January 31, 2008

SL Headquarters

Life at our Corporate Headquarters can get a bit weird at times but is never short on fun. Every day I get to pass the Ferrari Dealership of Dallas on the way to work, often times there are a few whizzing on by my Honda, much to my envy.

When I get into the office where Corporate is located in North Dallas I sit in my cube with Superman as my sidekick and go to work. Sitting on my row is Daniel the gadget guy, Doug who never stops talking (ever), and Mary who manages to receive at least one package a day from various vendors even if she is not here - how she does that I will never know. In the next row sits Amanda from The 'Ville, Laude the Kung-Fu Legend and Patty Mac-Patty Mac. While wheeling and dealing sales on our services we are always yelling over each other at what, when and where we are selling for coordination of product placement and inventory updates, it's very much like the floor at the NYMEX, and I wouldn't have it any other way.

So there we sit, 12+ hours a day in our nice massage chairs shouting at each other like heathens. Did I mention the story on the massage chairs? Daniel being the gadget guy that he is was always raving about how cool it would be to get office chairs that massage you while you sit there and work. So Lance made it one of our Sales incentives that month that if we hit, we all get massage chairs. We of course hit our team goal and now there is a general low humming from the chairs that resonate from the Sales section of the office - along with the yelling and screaming of course. More from Amanda on our cool new chairs.

Now we share this office building with several other companies and I would estimate that there are around 100+ people at any given time in this building. The restroom here has two stalls - you do the math. It gets quite interesting at times but at least they have elevator music playing in there. I guess.

After my day is over I like to go around and talk to co-workers to catch up on things. Then it's off past the dealership to my home for some more clicking away at my keyboard to tie up some loose ends from work and some much needed rest. And tomorrow I get to see some more Ferraris - how cool is that?

-Michael # 1

January 30, 2008

That's SMART

My grandmother used to say an ounce of prevention is worth a pound of cure. Usually this was her polite way of telling me to pick my skateboard up off the stairs before she stepped on it and broke her neck or to put a sheet of newspaper over her antique kitchen table before I began refueling my model airplane. All very sound advice looking back. And now here I find myself repeating the same adage some twenty years later in the context of predicting mechanical drive failure. An ounce of prevention is worth a pound of cure.

Hard disk drive manufacturers recognized both the reality and the advantages of being able to predict normal hard disk failures associated with drive degradation sometime around 2003. This led a number of leading hard disk makers to collaborate on a standard which eventually became known as SMART. This acronym stands for Self-Monitoring, Analysis and Reporting Technology and when used properly is a formidable weapon in any system administrator's arsenal.

The basic concept is that firmware on the hard disk itself will record and report key "attributes" of that drive which when monitored and analyzed over time can be used to predict and avoid catastrophic hard disk failures. Anyone who has been around computers for more than a day knows the terrible feeling that manifests in the pit of your stomach when it becomes apparent that your server or workstation will not boot because the hard disk has cratered. Luckily, we ALL of course back up our hard drives daily! Right?

All kidding aside even with a recent back up just the task of restoring and getting your system back in working order is a serious hassle and it’s not something you get the luxury of scheduling if the machine is critical to operations and failed in the middle of your work day or worse yet, the middle of your beauty sleep. That is where SMART comes in. When properly used SMART data can give “clues” that a drive is reaching a failure point--prior to it failing. This in turns means you can schedule a drive cloning and replacement within your next regular maintenance window. Really aside from a hard disk that lasts forever what more could an administrator ask for?

SMART drive data has been described as a jigsaw puzzle. That's because it takes monitoring a myriad of data points consistently over time to be able to put together a picture of your hard disk health. The idea is that an administrator regularly records and analyzes characteristics about the installed spinning media and looks for early warning signs that something is going wrong. While different drives have different data points, some of the key and most common attributes are:

  • head flying height
  • data throughput performance
  • spin-up time
  • re-allocated sector count
  • seek error rate
  • seek time performance
  • spin try recount
  • drive calibration retry count

These items are considered typical drive health indicators and should be base-lined at drive installation and then monitored for significant degradation. While the experts still disagree on the exact value of SMART data analysis, I have seen sources that claim at least 30% of drive failures can be detected some 60 days prior to the actual failure through the monitoring of SMART data.

Of course not all drive failures can be predicted. Plus some failures are caused by factors other than drive degradation. Consider drives damaged by power surges or drives that are dropped in shipping as good examples of drive failures that cannot normally be detected through SMART monitoring. However in my humble opinion even one hard disk failure prevented over the course of my career is something to celebrate--unless you happen to own stock in McNeil Consumer Healthcare, a.k.a. the distributors of Tylenol!

So what does this have to do with SoftLayer? Well I am certainly not claiming that SoftLayer is going to predict all your hard drive disasters so there is no reason for you to back up your data. In fact, I recommend not just backing it up but backing it up in geographically disparate locations (did I mention we have data centers in Dallas and Seattle?). What I do mean to share is that technologies like SMART data are just one of the many ways SoftLayer is currently investigating to improve what is already the best hosting company in the business.

I should know. I was tasked with writing the low-level software to extract this data. That’s right. SoftLayer has engineers working at the application layer, down at the device driver layer, and everywhere in between. If that doesn’t give you a warm fuzzy about your hosting company, I don’t know what will.

-William

January 29, 2008

Where is All the Noise?

Earlier in the week, Shawn Boles wrote a post that included some nostalgic feelings for old-fashioned computers. As I read it, I was thinking of my own experiences with the grandfathers of our desktop computers. I vividly remember getting the floppy disks out of the big dusty box, sliding them into the drive, and listening as the drive began its slow, crunching march towards the data I needed. That’s when it hit me. I can’t hear my computer! Sure, there’s a cooling fan in it, and if I stick my head under my desk I can hear not only the fan, but my coworkers quietly asking each other why my head is under my desk in the middle of the day. Other than that, there’s nothing! No drive heads moving, no crunching, no system beeping, no modem noises, and certainly no death rattles.

That’s right, death rattles. Those of you unfamiliar with the dinosaurs we used to call computers don’t know about the death rattles. The large floppy drives were the worst offenders. Every faulty sector on those disks caused a loud grinding sound that you could FEEL, indicating that the pencil-sized drive head was none too happy with encountering the data equivalent of a marching band at a funeral. Now that we’ve moved away from media that comes in physical contact with the drive head, we’ve moved away from the death rattles. However, hard drives can occasionally produce them, even to this day. When a hard drive goes bad, it emits this high-pitched chirping sound, akin to what you would expect a robotic sparrow to make.

When I was in high school, my best friend’s dad was a programmer. One day we were working on a bad hard drive for a friend and had a brilliant idea. We took a microphone and recorded the sounds the hard drive was making. With some Mp3 editing tools, we added roughly 10 minutes of silence in front of that sound. Then, we made the sound into his dad’s windows startup sound. So upon turning on his computer, he would work for 10 minutes and then hear the DEATH RATTLE. Watching his pasty frame lunge for the power button was hilarious for a good 24 hours after it happened.

Browsing computer message boards (yes, I’m a nerd, that’s why I work here as a developer) I occasionally come across people who are still cursed by dial-up modems, and they invariably ask how they can disable the noises their modem makes when it connects. I want to scream to them “DON’T! You’ll destroy your last audible connection to your machine!” The noises a modem makes are iconic, and a sufficiently trained guru can tell the connection speed simply by listening to the connection noises. From the industrial clanking of the 2400 baud to the sci-fi whooshing of the 56k, each remaining hardware noise is precious.

Precious to humans anyway. When I was younger I saved all my money for weeks to buy a 56k modem. I painstakingly researched (at 24,000 baud) all the different kinds of modems I could purchase, and finally decided on a US Robotics 56k modem. I arrived home from my triumphant shopping trip and promptly tore the cover off the family computer to install my new toy. Once installed, I powered up the computer, installed the drivers, and attempted to connect. Sweet, sweet screeching poured out of the modem, to my delight! However, my dog was also in the room. He had, until this point, tolerated the noises the computer had been making in “his” room. However, how there was a new noise. A new, screechier noise. And it was coming from an OPEN COMPUTER CASE. That’s right, in my excitement, I had left the case cover off just in case I had to tinker some more. My dog got up, walked across the room, and promptly ripped the modem right out of the case with one mighty chomp. He threw it on the ground, chomped it once more and then, satisfied that it would no longer disturb his rest, flopped back down onto his bed in the corner. I still cannot hear a connection sequence without bringing up that memory.

Of course, we don’t have to worry about these things at SoftLayer (dogs OR noises). We developers work on silent machines with no rattles or screeches, while the servers in the data center are attended to by dozens of employees, aware of any slight problem. The noise in the data center is substantial. Each employee wears hearing protection, and even with that the noise of thousands of cooling fans will get to you. Each time the developers have to spend a morning in the data center installing new servers, we spend the rest of the day shouting “WHAT?!” to each other. But computers will continue to get quieter, and our connection to them will continue to be less and less physical. Our customers know this already: none of them have even seen their servers, they rely on our skilled datacenter staff to monitor their hardware for them, day and night. But there are no death rattles, no screeches, just cooling fans.

The last of our noisy connection to our computers is starting to fade. Already we’ve stopped using magnetic media like floppy disks, and the noisy CD and DVD drives of yesteryear are being replaced with silent models. Even the hard drives that caused my friend and I so much joy are being replaced with mechanically simple solid state drives, which run much faster and have no noise at all. Come to think of it, I think I can learn to live in the new silent computing future.

-Daniel

Categories: 

Pages

Subscribe to customer-service