We hosted our first quarterly networking event in Singapore yesterday, and as I spoke with the partners, prospects, customers and SLayers in attendance, I heard some incredible stories about struggles with scaling infrastructure and how SoftLayer has revolutionized the way our customers look at their physical and virtual infrastructure. As we talked about our experiences, one of my own "war stories" came to mind, and I got to share it:
On on a Sunday afternoon in March 2002, an earthquake hit Taiwan. It measured 6.8 on the Richter scale, and it shook buildings across the island, flattening some of them and wreaking general havoc in cities. Beyond the visible damage it caused, it took out the fiber landing stations in Taiwan, cutting off Asia Pacific Internet traffic from the US. Typically when a fiber cable system is cut, telcos will scramble to re-route their traffic to the next available path, but because North Asia was crippled by the quake, all Internet traffic in Asia was being routed through Australia, causing major congestion down under, resulting in virtually zero Internet connectivity to the rest of the world.
At that time, I was VP of Sales for a leading Singapore-based hosting company. I received a call on my sales hotline at 7am on the morning after the earthquake. The caller was the CEO of a major gaming company in Australia, and he sounded desperate. All his servers — hosted in the US at the time — were unreachable, and he had been calling hosting companies all over Asia to buy some dedicated servers to host the game for his Asian customers. While I couldn't help him when it came to getting connectivity to his servers in the US, I thought it would be easy to accommodate his request for hardware based in Asia.
I asked him what server configurations he needed, and he detailed 20 identical servers that needed to be up and running for his gaming application within 24 hours, highlighting that he was losing thousands in revenue by the day. He explained that the projected revenue loss would exponentially increase to thousands per hour if the game remained offline for 24 hours more. He gave me his RAM, hard disk, OS and Database requirements, and he added, "We need all of them to be on Woodcrest!"
I remember vividly saying, "Woodcrest what? Oh, yes, yes, we have those!" I told him I'd get back to him, hung up the phone and went straight to our provisioning manager. We stock to provide 20 servers, but we didn't have any Woodcrest CPUs. There was no way we could locate, rack and provision the requested servers 24 hours ... The best we could commit to was 10 days. Obviously, that wasn't going to work, but I wasn't discouraged. I was going to solve the problem.
I managed to scrape together 20 Woodcrest CPUs from different local electronics retailers, and after wrangling cheques from the finance department and getting the CEO to apply pressure the provisioning manager, I was able to "fast-track" the servers to a four-day provisioning time. When all was said and done, he was able to bring his game back online after losing out on 8 days of business. Despite the losses, being able to turn around that kind of order that "quickly" made me pretty proud.
10 years later, I can't believe how much things have changed.
SoftLayer automates almost all of the manual processes, and we're able to provision a dedicated servers in 2-4 hours. While that's a pretty impressive feat, it's even more amazing when you consider that we can bring up 20, 50 or 100 dedicated servers in the same time frame. Just look at what OMGPOP was able to do when their "Draw Something" app was downloaded 36 million times. That's what automation is all about. Anything that we can automate, we automate, and that makes for an unbeatable user experience.
If someone came to us today with the an urgent order similar to the one I dealt with in 2002, the entire interaction above would boil down to, "What specs do you need? *typing* Here's your order number. You can expect the machines to be provisioned within 4 hours." We'd be off the phone by about 7:20am, and by noon, all of the servers would be online and hosting the game. The craziest part is that we're not even satisfied with that turnaround time yet. Our commitment is to continue to innovate, automate and empower our customers through our customer portal and APIs, and because our goals are to get better and serve our customers faster, the carrot will always be in front of us ... the same way UPS has a philosophy of "constructive dissatisfaction."
I want to thank everyone who came to our networking event yesterday. I hope you learned a little something about SoftLayer because I certainly learned a lot about our customers in the dozens of conversations I had. If you weren't able to attend and want to see what you missed, we posted a few pictures on Flickr: SoftLayer Singapore - Quarterly Networking Event - March 28, 2012
Do you have any infrastructure horror stories from the past like mine?