Author Archive: Kevin Hazard

December 15, 2011

Fighting SPAM and Abuse on a Global Network

For better or worse, one of the most engaging posts on the SoftLayer Blog is "We are a No-Spam Network," written by Jacob Linscott in June 2007. When it was posted, it celebrated a completely clear Spamhaus listing page – quite an accomplishment for a large hosting provider (for reasons I'll illustrate below). Since the post was published, it has become a hotbed of conversation about any and all abuse-related issues. Google "SoftLayer SPAM," and you'll see the post show up as the second result, so a lot of Internet passers-by will come across the post and use the comment section as a platform to share abuse-related concerns they have for us.

That engagement is a double-edge sword: It's good because we hear the concerns people have. It's bad because the post was meant to be a celebration of the continuous work that the abuse department does, and uninitiated visitors seem to consider it a unilateral claim that we've beaten spam once and for all. In the course of responding to comments on that post, I shared an analogy to convey what it's like to run abuse for a large hosting provider:

Scenario

Let's say you're the security manager for a huge mall. This mall has 100,000 stores with people walking in and out 24x7x365. In this scenario, there are "good guys" and "bad guys" who walk into and out of the mall, and every person looks exactly the same. Some of those people are store owners while others are customers of those stores. As the security manager for the mall, you want to maintain the safest, most well-maintained mall in the world, so when you find bad guys walking in and out of your mall, you do everything you can to kick them out and keep them out. Sometimes those bad guys are store owners who attract and send the wrong crowd; sometimes they are bad guy customers of a good guy store owner.

How would you manage your mall? It's not possible to differentiate whether a store owner will be a good guy or a bad guy when they're applying to lease space in your mall, so you can't "keep the bad guys out" in that regard. You can't have a security team of 100,000 people monitoring what's happening in those 100,000 stores, much less have someone individually check the millions of visitors streaming in and out of the stores. What's a security manager to do?

If you look at how Las Vegas casinos address that concern, it's clear that your best bet is to install security cameras and have a team monitoring them all the time. You might not be able to watch everything at the same time, but you can document what's happening around your mall and respond if you notice something unusual (or if someone calls in to report that they've seen bad guys coming from a store in your mall).

That's the position we're in.

SoftLayer Abuse Team

SoftLayer's network is the mall, the stores are servers, the store owners are our customers (who are often responsible for several "stores"), and the good guys and bad guys are traffic into and out of the network. We try to differentiate good guys and bad guys, but even if we know that all good guys have purple eyes and all bad guys have neon green eyes, it's still difficult to look 26,000+ store owners in the eye every day as they're walking into and out of the mall.

We staff a team of people intent on clearing the bad guys from our mall, and we know that even though good guy store owners may inadvertently host their own bad guy customers, they want to remove those customers from their store as well, so they appreciate us helping them pinpoint those customers so they can be removed.

We keep an eye on our security cameras and get our security guards to the stores where bad guys are reported as quickly as possible. If no one reports that the people coming out of store #73,403 are all bad guys, it's hard for us to know that they aren't good guys ... Which is why we encourage anyone and everyone to report abuse-related concerns to abuse@softlayer.com so we can mobilize our security force.

As Edmund Burke once said, "When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle." Or more colloquially, "All that is necessary for the triumph of evil is that good men do nothing."

Given that illustration, the abuse team deserves a LOT of credit for the work they do behind the scenes. They are constantly investigating reports and working with customers to get remove any and all content that violate SoftLayer's MSA, and too often, that can be a thankless job. Fighting abuse is an ongoing process, and while the nature of the beast might suggest the overall war will never be won, we're always getting faster and stronger, so the individual battles are easier and easier to win.

-@khazard

September 8, 2011

Boston Startup Scene - WebInnovatorsGroup

We love startups and entrepreneurship communities that help startups become successful. Startups are usually all about innovation and approaching existing problems in a new way ... And if you're familiar with SoftLayer's "Innovate or Die" motto, you know that we're cut from the same cloth. We've partnered with incubators like Tech Wildcatters to provide up-and-coming companies with a year of $1,000/mo hosting credits along with a little SoftLayer expertise sprinkled in for good measure, and we are happy to support community partners like non-profits and user groups where new ideas are born every day.

Given our commitment to the startup community, when we heard that a sponsorship opened up for the September 13 WebInnovatorsGroup quarterly meeting, we jumped on the chance to get involved. WebInno events are fueled by a long-standing community of Internet and mobile entrepreneurs founded by David Beisel, and while I could tell you everything I know about what they're doing in Boston, the best person to hear from is David himself:

Boston + Entrepreneurs + Technology + Beer ... It was a no-brainer for us to be a Gold Sponsor of WebInno31.

Visit WebInnovatorsGroup.com to learn more about the WebInno community or head straight to the WebInno31 registration form to reserve your spot at Royal Sonesta Cambridge on Tuesday, September 13, at 6:30pm.

-Kevin

P.S. If you have a startup community or an ongoing event like WebInno that SoftLayer can be involved with, leave a comment on this blog or let us know on Twitter: @SoftLayer

August 18, 2011

Subtract Server. Add Humor.

Once in a blue moon, a SoftLayer customer has to cancel a server. Sometimes their business is growing and they're moving up to more powerful hardware, sometimes they need to consolidate their equipment to cut their costs, and sometimes their reason can't really be categorized. In this case, a happy customer with a few dozen servers decided he needed to shut one down, and the explanation he gave would clearly fall into the third category:

Initial Ticket

Customer
I would like to cancel this server on August 20th, 2011, but not before that date. Anytime on this date will be okay.

We no longer have a need for this server and would like to cancel it before our next billing period. Thank you for your help in this matter. Please send me an email when this server has been canceled on August 20th, 2011.

She's been with us for a long time, but things just aren't working out ... She's become a gold digger. It's her, not me. Please let her down easy. I don't like punking out and having someone do my dirty work, but I'm afraid she might be violent. Diamond rings hurt when you get hit with them.

SoftLayer
I'm sorry to hear things did not work out for the two of you. While your safety is important to us, I must ask that you end this relationship via official channels.

Please submit an official cancellation request by going to Sales --> Cancel Server and proceeding through the cancellation steps. The server will be reclaimed at the end of your billing cycle on August 22nd.

Please let us know if you have any questions.

Customer
She always tried to make it hard for me to break up with her. Done!
 
SoftLayer
Glad to hear things went smoothly. Things don't always do, but we knew you could pull through it. :-)
 

Official Cancellation Request

Customer
Word to your moms I came to drop bombs, I got more rhymes than the Bible's got Psalms.
 
SoftLayer
Thanks for your unique note, definitely was a nice break from the norm.

We're glad to continue being part of your success!

Please contact us should future needs arise.

Customer
Thanks, it was a subtle reminder to get out your seat and jump around.
 

Let this be a lesson to all of you: Get out your seat and jump around.

-@khazard

August 4, 2011

Blood, Sweat and Tears: The Server Challenge

When you're walking down the aisles of an expo hall at a technical conference, what do you expect to see? Stacks of collateral? Maybe a few giveaway T-shirts? A fancy switch-ball or two? How about a crowd of people watching as a fellow attendee slams hard drive trays into a server enclosure and frantically plugs in network cables as a digital clock times them?

Cynical attendees might look at the Server Challenge and think of it as a gimmicky way to draw a crowd to our booth, but when you step up to the server enclosure to compete, you're getting a crash course in SoftLayer's business (along with an exciting tangible experience).

Before your first attempt, you'll learn that SoftLayer is a hosting provider and that you'll be reassembling a miniature version of the larger server racks we have filling data centers around the country (soon to be around the world). You see that one of SoftLayer's biggest differentiators is our network configuration: A public network, a private network and an out-of-band management network connection to every SoftLayer server for free ... And when the clock starts, we can share even more of the SoftLayer story.

Our goal is to let you experience SoftLayer while you're just hearing about other companies. As it turns out, the experience draws people in:

One of the coolest parts of pulling together that time lapse video from OSCON was seeing the reactions on the faces of the participants when they finished. The challenge sparks a surge of adrenaline, so when competitors stop the clock, they expectantly check to see how they fare against the conference's Top 10 times.

In the last conference alone, no fewer than five other companies (who don't even have a connection with the hosting industry) approached us to ask how they could build their own Server Challenge. Needless to say, the Server Challenge is becoming a SoftLayer conference staple ... And we're looking forward to the hottest competition ever at HostingCon 2011 next week!

Between your study of server schematics and your dissection of the winning run's strategy from the end of the OSCON video, make sure you click through to George's HostingCon preview so you can learn where to find SoftLayer in San Diego.

-@khazard

P.S. Space is limited for the HostingCon Party, so if you'll be in town, make sure to let us know so we can give you a promo code for free admission!

June 23, 2011

IPv6 - Blocks, Slashes and Big Numbers

IPv4 addresses are 32-bit while IPv6 addresses are 128-bit. Customers can get a /64 allocation of IPv6 addresses provisioned to a single SoftLayer server. A /64 block of IPv6 addresses contains 18,446,744,073,709,551,616 distinct addresses. The entire IPv4 address space is 4,294,967,296 distinct addresses.

It's easy to get lost in a sea of numbers when you start talking about IPv4 and IPv6 address space. With the exhaustion of IPv4 address space and the big push toward IPv6, everyone's talking about address blocks, usage justification and dual stack compatibility, but all of those conversations presuppose a certain understanding of why IP addresses are the way they are. Someone can say, "The IPv6 pool is exponentially larger than the IPv4 pool," but that statement needs a little context when you hear that providers like SoftLayer are provisioning a free /64 IPv6 allocation of 18,446,744,073,709,551,616 addresses to a single server. If the entire IPv4 pool on the Internet is 4,294,967,296 addresses and we're giving away that many IPv6 addresses to a single server, a simple question logically follows:

MattCodes

Are the Internet authorities being irresponsible when they're allowing such huge numbers of IPv6 addresses to be assigned to individual servers without a demonstrated need for that many addresses? Will this "wastefulness" lead to another IP address pool depletion in our lifetime? These questions are completely legitimate, and they're much easier to explain in a visualized format than they are if we answered them line-by-line in text:

The video duration might seem intimidating, especially if you consider that all 15 minutes are spent talking about IP addresses (Woohoo!), but there's a lot of information, and we did our best to break it down to simple pieces that logically follow each other to help you get the full picture of the world of IP addresses. We explain what CIDR Slash (/) Notation (where you see IP address blocks written as "192.0.2.0/24"), and we offer a simple trick to calculate the number of distinct addresses available in a given IPv4 block. There's a fair amount of witty (and not-witty) banter and at least one use of the word "ridonkulous," so if you enjoyed the DC Construction video commentary, you'll get a kick out of this one too.

Toward the end of the video, we speak directly to why SoftLayer is able to give a /64 of IPv6 addresses to every server and what that means for the future of the IPv6 space.

Fun Fact: SoftLayer IP Address Space*

  • IPv4: 872,448 Addresses
  • IPv6 (/32): 79,228,162,514,264,337,593,543,950,336 Addresses

*Does not include IP space assigned to The Planet

Did the video help you wrap your mind around the differences between IPv4 and IPv6? Do you have any more questions about the differences between the two or how SoftLayer is approaching them?

-@khazard

Categories: 
May 12, 2011

Follow 750 Servers from Truck to DC Rack

What do you call the day after you finish building a new data center server room and cabling the server racks in it? If you're an employee at SoftLayer, you call it Truck Day.

Last week, a few of the folks from marketing were invited to celebrate in the Truck Day festivities for Pod 2 in DAL05 (SR02.DAL05), and I jumped at the opportunity. I don't go anywhere without at least one camera on-hand to document and share what's going on with the SoftLayer community, and Truck Day wasn't an exception ... In fact, I had three different cameras going at all times!

The truck arrived at around 7 a.m. with a few dozen pallets of servers, and about forty employees from all around the company immediately jumped into action. As the pallets moved from the loading dock to the inventory room, people were unboxing servers and piling them on carts. When a cart was full, it was whisked to the data center and unloaded. The data center techs plugged in each of the servers to confirm its configuration and stacked it with matching configurations in designated areas around the data center. By the time one cart got back to the inventory room, another was on its way to the data center, so very little time was lost.

Back in 2007, SamF did a great job of explaining the process, so I won't reinvent the wheel. Instead, I'll let you see the activities as they were captured by the three cameras I toted along:

To give you an idea of how fast all of this was done, each the time lapse cameras set up in the data center and in the inventory room captured images every five seconds. When the video was compiled, the frame rate was set to 20 frames per second, so each second of time lapse video is the equivalent of 100 seconds of work. In a matter of just a few hours, we received, inventoried, racked, cabled and started selling around 750 servers in a brand new data center pod. Competitors: Be afraid. Be very afraid. :)

Pictures from DAL05 Pod 2 Truck Day have been posted on our Flickr Account: http://sftlyr.com/8g

In the past three weeks, we brought three different data center pods online in three different parts of the country: On April 25, it was our first server room in San Jose (SJC01); on May 2, the second server room in DAL05; and on May 10, our second server room in WDC01. As far as I know, we don't have a new pod planned for next month, but given how quickly the operations team has been building data center space, I wouldn't be surprised to get a call asking me to come in a little early to help unload servers in a new data center next week.

-@khazard

Music Credit: The background track in the video is "Your Coat" from SoftLayer's very own Chris Interrante. Keep an eye out for his soon-to-be released EP: OVERDRAFT.

May 6, 2011

Cabling a SoftLayer Server Rack

A few weeks ago, SamF posted "Before They Were SoftLayer Data Centers," a virtual scrapbook from the San Jose data center construction process, and based on the surge of traffic we saw to the post, our customers loved it. It's incredible to see an open warehouse-looking space transformed into an enterprise data center environment, and there's more amazingness where that came from.

In addition to the pre-"Truck Day" pictures we posted on the blog and in the San Jose DC Construction album on Flickr, we trained a video camera on a row in the data center to capture the cabling process.

What's so interesting about plugging in cables?

Consider the fact that each of the network switches we use in a rack has at least 48 ports. Now consider that each rack has two public network switches, two private network switches and one out-of-band management network switch that need to be connected to every SoftLayer server in the rack. That's 240 pre-measured network cables that need to be labeled and routed to specific heights in each rack ... without getting tangled and knotted up (see: behind your TV or under your computer desk).

The cabling process is so precise that if a single cable is out of place, the zip-tie on an entire bundle will be cut, and the process is started from scratch. The process is time-consuming, but the results speak for themselves:

SoftLayer Server Rack

Without further ado, here's the SJ data center team in action. The video is playing at 20x normal speed, and given the amount of time it takes to complete the cabling process for each rack, we enlisted the help of Spongebob SquarePants in our use of the "Two Hours Later" cut:

Impressed? Amazed?

Just wait until you see the time-lapse from Truck Day.

-Kevin

April 6, 2011

Take Me Out to the Ball Game

If you didn't read the title to this post in the singsong seventh-inning stretch tune, the rest of this post probably won't be for you. For those of you who just got to "Buy me some peanuts and Cracker Jack," as the song kept playing in your head, you're going to love the news we have to share. We'll wait for you to finish belting out "At the old ball game!" first, though.

[Pausing here for everyone to finish the song.]

Now that everyone's back together, I want you to make sure you don't lose any of that late-inning adrenaline because you might need it at the end of this post.

SoftLayer is all about customer experience. Just ask Skinman. If you're a SoftLayer employee and you don't have "the customer" in the top slot of your "work priorities" list, you'll either need to update that list quickly or update your résumé. This post isn't about THE SoftLayer customer experience, though ... It's about A SoftLayer customer experience.

THE SoftLayer customer experience is all about automation, efficiency, service and innovation. A SoftLayer customer experience uses the term in a much more general sense: It's any opportunity we have to give back to our customers in the form of events, contests, and in this case, baseball tickets! If you're a SoftLayer customer, you're entitled to more fun than our competitors' customers ... And if that's not in our terms of service, it probably should be. :-)

SoftLayer Baseball

Throughout the 2011 Major League Baseball season, SoftLayer will be giving away tickets to Texas Rangers home games in Arlington, Texas! We're going to keep you guessing about how/when/where we'll be giving them away, but if you keep your eye on the SoftLayer Blog, follow @SoftLayer on Twitter, subscribe to SoftLayerTube on YouTube and "Like" us on Facebook, you'll be the first to hear.

We're pretty sure customers in the DFW area are going to be the most excited, since they can root for the home team, but as the season progresses, the net may be cast significantly wider ... Reaching out to customers in other parts of the country (world?) who love SoftLayer and want to catch a game while they're in town for a data center tour. But let's not get too far ahead of ourselves just yet. Let's give away our first set of tickets!

Texas Rangers v. Anaheim Angels

  • Date: Monday, April 18, 2011
  • Time: 7:05pm
  • Location: Rangers Ballpark in Arlington
  • Seats: 2 - Section 26 (Lower Level, behind Home Plate!)
  • Transportation: You're responsible for transportation to/from the park

How to Enter
Since our first giveaway doesn't include transportation to/from the game, the primary pool of participants will be customers who live within driving distance (or happen to be in the DFW area on April 18). Entry into the competition is simple: Comment on this post about why you love SoftLayer.

When you're entering your email, please use a contact address associated with your SoftLayer account. Submissions will be accepted from now until 10 a.m. CDT on Thursday, April 14, so get to writing! We'll have a quick internal vote for all of the submissions after removing your contact information to obscure on which account goes with which response. If your submission wins, we'll email you on Thursday to arrange for ticket delivery ... You'll have the whole weekend to get excited about the game!

Play Ball!

-@khazard

March 23, 2011

SoftLayer Rocked SxSW 2011

South by Southwest 2011 is over. Phew. The chaotic buzz of sessions, trade show booths, concerts, happy hours and parties has subsided, so we can finally take a little time to look back at our experience in Austin last week. Our most talked about contributions to the 2011 SxSW community were our SxSL (South by SoftLayer) event at Iron Cactus on Monday and the SoftLayer Server Challenge on the trade show floor.

SxSL

If you've ever been to a SoftLayer soiree, you know that we know how to get down with our bad selves to throw a good party, and SxSL was no exception. The Cactus Room at Iron Cactus was a perfect venue to unwind after the first day on the trade show floor, and it proved to be a great setting for many interesting conversations about hosting, cloud computing and SoftLayer's plans for world domination ... err ... growth.

As you can see from a few of the pictures we took at the event, Snappy - HostGator's mascot - made an appearance, and he was quite the popular guy. He made so many friends at the party, he actually followed us back to the office in Dallas.

Server Challenge

If you followed the link to the Server Challenge at the top of this post, there's no need to reintroduce the competition, so we'll dive straight into how it went. Before I tell you what I think, listen to what @ipbrian had to say about it on Twitter: "Congrats to @SoftLayer for having the best contest and booth at #sxswi. I have NEVER wanted to repeat visit a vendor more."

Brian was a fierce competitor who pushed the limits of how fast our rack of servers could be reassembled, and prior to SxSW, he'd never heard of SoftLayer. As he hurried to reattach network cables, he experienced what we do in our data centers, and that experience is worth more than any piece of collateral we could have given him. That experience was our goal in designing the challenge, and based on our first show with it, we're confident that our goal is being met.

Some attendees saw the blazing times on our Server Challenge leader board as inspiration to complete an "Eye of the Tiger"-speed assembly while others - like the two squirrels from getacorn.com in the video below - knew they might not win the iPad 2 for being the fastest at SxSW but wanted to try anyway:

In addition to the official Server Challenge competition, we were happy to take part in Das IronGeek for the second year in a row. Das IronGeek put six press and bloggers through a series of five technology-related challenges to test their "geekiness," and the SoftLayer Server Challenge was the final "make it or break it" event to determine the champion. All of the competitors stared down the SoftLayer server rack and had a blast completing the challenge. Joshua Baer bested the other participants to become the 2011 Das IronGeek Champion, and if I were him, I'd be showing my kids the 2011 Das IronGeek wrap up for years to come.

As the trade show wound down on Thursday, our booth had a last surge of Server Challenge participants looking to reach the top of the leader board to with an iPad 2, but as you learned from Highlander, there can be only one. That "one" at South by Southwest was Erik Wagner from Netbiscuits with an amazing time of 1:08.8. When he recorded that time, we knew it would be tough to beat, so we had him complete it one more time on camera to show future generations of Server Challenge participants where the bar has been set. Even with the additional pressure of being on camera, he recorded a faster time than any other participant:

We have a few tweaks and improvements planned for our next Server Challenge competition, and we're excited to see how attendees at other shows respond.

As I write about SxSL and the Server Challenge, I'm reminded of stories about the popularity of the good ol' SoftLayer switch-ball and the hallway war we may or may not have been responsible for supplying with foam missile ammunition, but those stories will have to wait for another post. I'm still tired from SxSW sleep deprivation, and I need a nap.

-@khazard

March 14, 2011

SoftLayer SxSW Server Challenge

As the doors open to the trade show at South by Southwest (SxSW) Interactive 2011, SoftLayer is poised and ready to greet attendees with a brand new "Server Challenge" in booth 400. In previous iterations, the Server Challenge involved reassembling a single server with about a dozen components. In this challenge, we're going bigger. Literally.

In SoftLayer's booth, we have a 12U rack loaded with five servers and three network switches, and the challenge is to put it all back together as fast as possible. Check it out:

If you're roaming the trade show aisles, swing by and try your hand! Fastest time wins an iPad 2.

-@khazard

Subscribe to Author Archive: %