business

November 20, 2012

Community Development: Catalysing European Startups

SoftLayer works hard and plays hard. A few weeks ago, I traveled to Dallas for the first "Global Catalyst Summit"* where the community development teams in Europe, Asia and the United States all came together under one roof to learn, strategize and bond. What that really means is that we all experienced a week of hardcore information flow and brutal fun.

The onboarding process to become a part of the SoftLayer's Community Development (Catalyst) team is pretty rigorous, and traveling to Dallas from Amsterdam for the training made it even more intense. In short order, I learned about the roots of the Catalyst program and why SoftLayer is so interested in investing in helping startups succeed. I got the low-down on the hundreds of companies that are taking advantage of the program right now, and I was inspired by the six incredible people who focus exclusively on the Catalyst program at SoftLayer ... And Big Tex:

SoftLayer Community Development Team and Big Tex

When the whirlwind week of orientation and training came to an end, I came to a solid conclusion: I am working at SoftLayer for a reason. I believe SoftLayer has the most kick-ass global on-demand technology platform out there, and our focus on innovation and automation is reflected in everything we do. On top of that, we give that platform to startups to help springboard their success. I get to work with a community of world-changers. Needless to say, that's an amazing conclusion to come to.

As a member of the Catalyst team in EMEA (Europe, Middle East, Africa), I can provide signficant resources to entrepreneurs who are building awesome new applications and technologies that are making a difference locally, regionally and globally. Anna Bofill Bert and I work out of SoftLayer's Amsterdam office, and we are fully dedicated to helping startup and developer communities in our region.

As a review exercise and a way to educate the audience that may be unfamiliar with Catalyst, I thought I'd bullet out a few of the main ideas:

What is Catalyst?

The SoftLayer Catalyst Startup Program provides:

  • A generous monthly hosting credit toward dedicated, cloud or hybrid compute environments for a FULL YEAR (Ideal for dev-ops/next generation startup compute applications who want high performance from the start).
  • Direct connection to highest level programming team at SoftLayer — Our Innovation Team. Participating companies get help and advice from the people that are writing the book on highly scalable, global infrastructure environments.
  • Connection to the SoftLayer Marketing and PR Team for help getting spreading the word around the world about all the cool stuff participating startups are doing.

We reach startups by listening to them and meeting needs that all of them express. We are telling the SoftLayer story, networking, making friends, drinking too much and travelling like mad. In the course of a month, we went to Lean Start Up Machine in Rotterdam, Structure Europe in Amsterdam, Pioneers Festival in Vienna, HowToWeb in Bucharest and we managed to complete a quick tour of startup communities in Spain.

Like our peers on the US team, we partner with incubators and accelerators to make sure that when startups look for help getting started, they also find SoftLayer. We're already working with partners like Springboard, Seedcamp, GameFounders, Startup Sauna, the INLEA Foundation and Tetuan Valley, and the list of supported communities seems to grow daily. When the portfolio companies in each of these organizations are given access to the Catalyst program, that means SoftLayer's Catalyst customer base is growing pretty phenomenally as well.

What I actually like most about how we help startups is the mentorship and office hours we provide participating companies as well. SoftLayer was founded by ten guys in a living room in 2005, and we've got hundreds of millions of dollars in annual revenue as of 2012. That success is what the SoftLayer team is excited to share insights about.

Hustling is a major part of startup culture, so it's only fitting that I feel like I had to hustle through this blog to get all of my thoughts down. Given that SoftLayer EMEA is a bit of a startup itself, I'm happy to be practicing what we preach. If you'd like more information about Catalyst or you want to apply, please feel free to hit me up: esampson@softlayer.com

We want to be part of your company's success story.

-@EmilyBlitz

*Note: As an homage to Big Tex after the fire, we referred to our meeting as the "Global Catalyst Summit with Big Tex" at the Texas State Fair. We hope to see you back in action in 2013, Big Tex!

November 19, 2012

How It's Made (and Won): The Server Challenge II

Every year, we attend more than fifty trade shows and conferences around the world. We want to spread the word about SoftLayer and connect with each conference's technical audience (also known as future SoftLayer customers). That goal is pretty straightforward on paper, but when it comes to executing on it, we're faced with the same challenge as all of our fellow exhibitors: How do we get our target audience to the our booth?

Walk down any aisle of an expo hall, and you'll see collateral and swag beckoning to attendees like a candy bar at the grocery store register. Some exhibitors rely on Twitter to monitor an event's hashtag and swoop in at every opportunity to reach the show's influential attendees. Other exhibitors might send out emails to their clients and prospects in the area to invite them to the show. We see value in each of those approaches, but what we found to be most effective was to bring a SoftLayer data center to our booth ... or at least a piece of one.

The Server Challenge has come a long way over the years. Its meager beginnings involved installing RAM and hard drive cables in a tower server. Shortly thereafter, a rack-mount server replaced the tower server, but you were still tasked with "inside the server" challenges. As we started looking for ways to tell the bigger SoftLayer story with the Server Challenge, we moved to miniature server rack, and the competition really started to pick up steam. This year, we made it our goal to take the Server Challenge to the next level, and when Supermicro stepped in to sponsor the next iteration of the the competition, we started thinking BIG.

Why use a miniature version of a SoftLayer rack when we could use a full-size version? Why have a standalone screen when rack-mount monitors can make the display part of the unit? Why rely on speakers behind the booth to pump "Eye of the Tiger" while attendees are competing when we could easily build those into the next version of the challenge? What was initially intended to be a "tweak" of the first Server Challenge became a complete overhaul ... Hence the new "Server Challenge II" moniker.

Harkening back to the 8-bit glory days of Pac Man and Space Invaders, the Server Challenge II uses a full-size 42U server rack with vintage arcade-style branding, a built-in timer and speakers that blast esoteric video game music. The bread and butter of the challenge is the actual server hardware, though ... Supermicro provided two new 2U servers to replace the previous version's five 1U servers, and we installed the same Cisco (public and private networks) and SMC (out-of-band management network) switches you see in SoftLayer's pods.

Server Challenge II

We had two instances of the original Server Challenge (one in the US, one in Amsterdam), so in order for the Server Challenge II to be bigger and better, we had to increase that total to five — one instance in Europe, one in Asia and three in the United States. Things might get a little crazier logistically, but as a potential conference attendee, it means you're even more likely to encounter the Server Challenge II if you attend any events with us.

The Server Challenge II's Internal Debut

The first instance of the Server Challenge II made its debut at GDC Online in Austin, and we immediately knew we had a hit. By the time the rack got back to our office, we had to get it ready for its next destination (Cloud Expo West), but before we sent it on its way, we gave it an official internal debut ... and raised some money for the American Heart Association in the process.

Server Challenge II at SoftLayer

SLayers at the SoftLayer HQ in Dallas could pay $3 for one attempt or $5 for two attempts to reach the top of the Server Challenge II leader board. Needless to say, it was competitive. If you click on the image above, you'll notice that our fearless leader, Lance Crosby, stopped by and gave tips to (and/or heckled) a few participants. Unsurprisingly, one of our very talented Server Build Technicians — Ellijah Fleites — took home a MacBook Air and bragging rights as SoftLayer champion with a record time of 1:03.79 ... But records are made to be broken.

In Two Places at Once

Immediately after the AHA fundraiser, we crated up the rack and sent it along to Cloud Expo West in Santa Clara. A few days later, we put the finishing touches on the second Server Challenge II rack, and because we got it done quickly, we were able to get it shipped to the other side of the country for ad:tech NYC. We would finally have the competition running in two places at the exact same time!

We weren't disappointed.

On both coasts, the retro style of the Server Challenge II lured some fantastic competitors (excellent!), and started a lot of great conversations (even better!). Here are the final leader boards from the shows:

Server Challenge II
Server Challenge II

You probably notice that the times in the ad:tech leader board are a little higher than the times in the Cloud Expo leader board, and our team figured out why that was in the middle of the second day of the conference ... The way we bound the network cables differed slightly between the two instances, and we were using different switches to time the competition (one that required only one hand to activate/deactivate, the other requiring both hands). In order to have an "apples-to-apples" comparison between all of our shows, we're going to make sure everything is consistent with all of the instances, and we plan on keeping a running list of fastest overall challenge times ... and maybe even a "World Championship" one day.

Given the early success of the Server Challenge II, you can bet that it's not going anywhere any time soon. If we have multiple shows running the challenge at one time, we might even fire up a video chat where you can compete against an attendee at a completely different conference ... so be prepared.

In the next year, we'll have all five of the Server Challenge II instances in rotation across three continents, and with the popularity of the competition growing by leaps and bounds after every show, we hope by next holiday season, a home version of the Server Challenge II is at the top of every wish list on the planet. :-)

For now, though, I'll just leave you with a glimpse at the action from Cloud Expo West (click for more pictures from the show):

Cloud Expo West

-Raleigh

November 16, 2012

Going Global: Domo Arigato, Japan

I'm SoftLayer's director of international operations, so I have the unique pleasure of spending a lot of time on airplanes and in hotels as I travel between Dallas, Amsterdam, Singapore and wherever else our event schedule dictates. In the past six months, I've spent most of my time in Asia, and I've tried to take advantage of the opportunity relearn the culture to help shape SoftLayer Asia's business.

To really get a sense the geographic distance between Dallas and Singapore, find a globe and put one index finger on Dallas and put your other index finger on Singapore. To travel from one location to the other, you fly to the other side of the planet. Given the space considerations, our network map uses a scaled-down representative topology to show our points of presence in a single view, and you get a sense of how much artistic license was used when you actually make the trip to Singapore.

Global Network

The longest currently scheduled commercial flight on the planet takes you from Singapore to Newark in a cool 19 hours, but I choose to maintain my sanity rather than set world records for amount of time spent in a metal tube. I usually hop from Dallas to Tokyo (a mere 14 hours away) where I spend a few days, and I get on another plane down to Singapore.

The break between the two legs of the trip serves a few different purposes ... I get a much needed escape from the confines of an airplane, I'm able to spend time in an amazing city (where I lived 15 years ago), and I can use the opportunity to explore the market for SoftLayer. Proximity and headcount dictated that we spend most of our direct marketing and sales time focusing on the opportunities radiating from Singapore, so we haven't been able to spend as much time as we'd like in Japan. Fortunately, we've been able organically grow our efforts in the country through community-based partnerships and sponsorships, and we owe a great deal of our success to our partners in the region and our new-found friends. I've observed from our experience in Japan that the culture breeds two contrasting business realities that create challenges and opportunities for companies like SoftLayer: Japan is insular and Japan is global.

When I say that Japan is insular, I mean that IT purchases are generally made in the realm of either Japanese firms or foreign firms that have spent decades building reputation in market. Becoming a trusted part of that market is a time-consuming (and expensive) endeavor, and it's easy for a business to be dissuaded as an outsider. The contrasting reality that Japanese businesses also have a huge need for global reach is where SoftLayer can make an immediate impact.

Consider the Japanese electronics and the automobile industries. Both were built internally before making the leap to other geographies, and over the course of decades, they have established successful brands worldwide. Japanese gaming companies, social media companies and vibrant start-up communities follow a similar trend ... only faster. The capital investment required to go global is negligible compared to their forebears because they don't need to build factories or put elaborate logistics operations in place anymore. Today, a Japanese company with a SaaS solution, a game or a social media experience can successfully share it with the world in a matter minutes or hours at minimal cost, and that's where SoftLayer is able to immediately serve the Japanese market.

The process of building the SoftLayer brand in Asia has been accelerated by the market's needs, and we don't take that for granted. We plan to continue investing in local communities and working with our partners to become a trusted and respected resource in the market, and we are grateful for the opportunities those relationships have opened for us ... Or as Styx would say, "Domo Arigato, Mr. Roboto."

-@quigleymar

November 14, 2012

Risk Management: Securing Your Servers

How do you secure your home when you leave? If you're like most people, you make sure to lock the door you leave from, and you head off to your destination. If Phil is right about "locks keeping honest people honest," simply locking your front door may not be enough. When my family moved into a new house recently, we evaluated its physical security and tried to determine possible avenues of attack (garage, doors, windows, etc.), tools that could be used (a stolen key, a brick, a crowbar, etc.) and ways to mitigate the risk of each kind of attack ... We were effectively creating a risk management plan.

Every risk has different probabilities of occurrence, potential damages, and prevention costs, and the risk management process helps us balance the costs and benefits of various security methods. When it comes to securing a home, the most effective protection comes by using layers of different methods ... To prevent a home invasion, you might lock your door, train your dog to make intruders into chew toys and have an alarm system installed. Even if an attacker can get a key to the house and bring some leftover steaks to appease the dog, the motion detectors for the alarm are going to have the police on their way quickly. (Or you could violate every HOA regulation known to man by digging a moat around the house, filling with sharks with laser beams attached to their heads, and building a medieval drawbridge over the moat.)

I use the example of securing a house because it's usually a little more accessible than talking about "server security." Server security doesn't have to be overly complex or difficult to implement, but its stigma of complexity usually prevents systems administrators from incorporating even the simplest of security measures. Let's take a look at the easiest steps to begin securing your servers in the context of their home security parallels, and you'll see what I'm talking about.

Keep "Bad People" Out: Have secure password requirements.

Passwords are your keys and your locks — the controls you put into place that ensure that only the people who should have access get it. There's no "catch all" method of keeping the bad people out of your systems, but employing a variety of authentication and identification measures can greatly enhance the security of your systems. A first line of defense for server security would be to set password complexity and minimum/maximum password age requirements.

If you want to add an additional layer of security at the authentication level, you can incorporate "Strong" or "Two-Factor" authentication. From there, you can learn about a dizzying array of authentication protocols (like TACACS+ and RADIUS) to centralize access control or you can use active directory groups to simplify the process of granting and/or restricting access to your systems. Each layer of authentication security has benefits and drawbacks, and most often, you'll want to weigh the security risk against your need for ease-of-use and availability as you plan your implementation.

Stay Current on your "Good People": When authorized users leave, make sure their access to your system leaves with them.

If your neighbor doesn't return borrowed tools to your tool shed after you gave him a key when he was finishing his renovation, you need to take his key back when you tell him he can't borrow any more. If you don't, nothing is stopping him from walking over to the shed when you're not looking and taking more (all?) of your tools. I know it seems like a silly example, but that kind of thing is a big oversight when it comes to server security.

Employees are granted access to perform their duties (the principle of least privilege), and when they no longer require access, the "keys to the castle" should be revoked. Auditing who has access to what (whether it be for your systems or for your applications) should be continual.

You might have processes in place to grant and remove access, but it's also important to audit those privileges regularly to catch any breakdowns or oversights. The last thing you want is to have a disgruntled former employee wreak all sorts of havoc on your key systems, sell proprietary information or otherwise cost you revenue, fines, recovery efforts or lost reputation.

Catch Attackers: Monitor your systems closely and set up alerts if an intrusion is detected.

There is always a chance that bad people are going to keep looking for a way to get into your house. Maybe they'll walk around the house to try and open the doors and windows you don't use very often. Maybe they'll ring the doorbell and if no lights turn on, they'll break a window and get in that way.

You can never completely eliminate all risk. Security is a continual process, and eventually some determined, over-caffeinated hacker is going to find a way in. Thinking your security is impenetrable makes you vulnerable if by some stretch of the imagination, an attacker breaches your security (see: Trojan Horse). Continuous monitoring strategies can alert administrators if someone does things they shouldn't be doing. Think of it as a motion detector in your house ... "If someone gets in, I want to know where they are." When you implement monitoring, logging and alerting, you will also be able to recover more quickly from security breaches because every file accessed will be documented.

Minimize the Damage: Lock down your system if it is breached.

A burglar smashes through your living room window, runs directly to your DVD collection, and takes your limited edition "Saved by the Bell" series box set. What can you do to prevent them from running back into the house to get the autographed posted of Alf off of your wall?

When you're monitoring your servers and you get alerted to malicious activity, you're already late to the game ... The damage has already started, and you need to minimize it. In a home security environment, that might involve an ear-piercing alarm or filling the moat around your house even higher so the sharks get a better angle to aim their laser beams. File integrity monitors and IDS software can mitigate damage in a security breach by reverting files when checksums don't match or stopping malicious behavior in its tracks.

These recommendations are only a few of the first-line layers of defense when it comes to server security. Even if you're only able to incorporate one or two of these tips into your environment, you should. When you look at server security in terms of a journey rather than a destination, you can celebrate the progress you make and look forward to the next steps down the road.

Now if you'll excuse me, I have to go to a meeting where I'm proposing moats, drawbridges, and sharks with laser beams on their heads to SamF for data center security ... Wish me luck!

-Matthew

November 8, 2012

Celebrating the First Anniversary of SoftLayer Going Global

In October, SoftLayer's data center in Singapore (SNG01) celebrated its first birthday, and our data center in Amsterdam (AMS01) turned one year old this week as well. In twelve short months, SoftLayer has completely transformed into a truly global operation with data centers and staff around the world. Our customer base has always had an international flavor to it, and our physical extension into Europe and Asia was a no-brainer.

At the end of 2011, somewhere in the neighborhood of 40% of our revenue was generated by companies outside of North America. Since then, both facilities have been fully staffed, and we've ratcheted up support in local startup communities through the Catalyst program. We've also aggressively promoted SoftLayer's global IaaS (Infrastructure-as-a-Service) platform on the trade show circuit, and the unanimous response has been that our decision to go global has been a boon to both our existing and new customers.

This blog is filled with posts about SoftLayer's culture and our SLayers' perspectives on what we're doing as a company, and that kind of openness is one of the biggest reasons we've been successful. SoftLayer's plans for global domination included driving that company culture deep into the heart of Europe and Asia, and we're extremely proud of how both of our international locations show the same SLayer passion and spirit. In Amsterdam, our office is truly pan-European — staffed by employees who hail from the US, Croatia, Greece, France, the Netherlands, Poland, Spain, Sweden, Ireland and England. In Singapore, the SoftLayer melting pot is filled with employees from the US, Singapore, Malaysia, Indonesia and New Zealand. The SoftLayer culture has flourished in the midst of that diversity, and we're a better company for it.

All of this is not to say the last year has not been without challenges ... We've logged hundreds of thousands of air miles, spent far too many nights in hotels and juggled 13-hour and 6-hour time zone difference to make things work. Beyond these personal challenges, we've worked through professional challenges of how to make things happen outside of North America. It seems like everything is different — from dealing with local vendors to adjusting to the markedly different work cultures that put bounds around how and when we work (I wish I was Dutch and had as many vacation days...) — and while some adjustments have been more difficult than others, our team has pulled through and gotten stronger as a result.

As we celebrate our first anniversary of global operations, I reflect on a few of the funny "light bulb" moments I've experienced. From seeing switch balls get the same awed looks at trade shows on three different continents to realizing how to effectively complete simple tasks in the Asian business culture, I'm ecstatic about how far we've come ... And how far we're going to go.

To infinity and beyond?

-@quigleymar

November 6, 2012

Tips and Tricks - Pure CSS Sticky Footers

By now, if you've seen my other blog posts, you know that I'm fascinated with how much JavaScript has evolved and how much you can do with jQuery these days. I'm an advocate of working smarter, not harder, and that maxim knows no coding language limits. In this post, I want to share a pure CSS solution that allows for "sticky" footers on a web page. In comparing several different techniques to present this functionality, I found that all of the other routes were overkill when it came to processing time and resource usage.

Our objective is simple: Make the footer of our web page stay at the bottom even if the page's content area is shorter than the user's browser window.

This, by far, is one of my *favorite* things to do. It makes the web layout so much more appealing and creates a very professional feel. I ended up kicking myself the very first time I tried to add this functionality to a project early in my career (ten years ago ... already!?) when I found out just how easy it was. I take solace in knowing that I'm not alone, though ... A quick search for "footer stick bottom" still yields quite a few results from fellow developers who are wrestling with the same frustrating experience I did. If you're in that boat, fear no more! We're going to your footers in shape in a snap.

Here's a diagram of the problem:

CSS Footer

Unfortunately, a lot of people try to handle it with setting a fixed height to the content which would push the footer down. This may work when YOU view it, but there are several different browser window heights, resolutions and variables that make this an *extremely* unreliable solution (notice the emphasis on the word "extremely" ... this basically means "don't do it").

We need a dynamic solution that is able to adapt on the fly to the height of a user's browser window regardless if the resize it, have Firebug open, use a unique resolution or just have a really, really weird browser!

Let's take a look at what the end results should look like:

CSS Footer

To make this happen, let's get our HTML structure in place first:

<div id="page">
 
      <div id="header"> </div>
 
      <div id="main"> </div>
 
      <div id="footer"> </div>
 
</div>

It's pretty simple so far ... Just a skeleton of a web page. The page div contains ALL elements and is immediately below the

tags in the page code hierarchy. The header div is going to be our top content, the main div will include all of our content, and the footer div is all of our copyrights and footer links.

Let's start by coding the CSS for the full page:

Html, body {
      Padding: 0;
      Margin: 0;
      Height: 100%;
}

Adding a 100% height allows us to set the height of the main div later. The height of a div can only be as tall as the parent element encasing it. Now let's see how the rest of our ids are styled:

#page {
      Min-height: 100%;
      position:relative;
}
 
#main {
      Padding-bottom: 75px;   /* This value is the height of your footer */
}
 
#footer {
      Position: absolute;
      Width: 100%;
      Bottom: 0;
      Height: 75px;  /* This value is the height of your footer */
}

These rules position the footer "absolutely" at the bottom of the page, and because we set #page to min-height: 100%, it ensures that #main is exactly the height of the browser's viewing space. One of the best things about this little trick is that it's compliant with all major current browsers — including Firefox, Chrome, Safari *AND* Internet Explorer (after a little tweak). For Internet Explorer to not throw a fit, we need concede that IE doesn't recognize min-height as a valid property, so we have to add Height: 100%; to #page:

#page {
      Min-height: 100%;  /* for all other browsers */
      height: 100%;  /* for IE */
      position:relative;
}

If the user does not have a modern, popular browser, it's still okay! Though their old browser won't detect the magic we've done here, it'll fail gracefully, and the footer will be positioned directly under the content, as it would have been without our little CSS trick.

I can't finish this blog without mentioning my FAVORITE perk of this trick: Should you not have a specially designed mobile version of your site, this trick even works on smart phones!

-Cassandra

November 5, 2012

O Canada! - Catalyst, Startups and "Coming Home"

I was born and raised in Brockville, Ontario, and I've always been a proud Canadian. In 2000, I decided to leave my homeland to pursue career options south of the 49th parallel, so I became an active participant in Canada's so-called "brain drain." It's never easy starting over, but I felt that my options were limited in Canada and that I wouldn't find many opportunities to make an impact on a global stage.

Fast-forward to 2012. Early in the year, we were introduced to GrowLab — a leading Vancouver based accelerator — by our friends at East Side Games Studio. They seemed to have a lot of incredible stuff going on, so I planned an exploratory mission of sorts ... In June, I'd visit a few Canadian cities with an open mind to see what, if anything, had changed. With the Catalyst Program's amazing success in the US, I hoped we could hunt down one or two Canadian startups and accelerators to help out.

I was very pleasantly surprised at what I found: A vibrant, thriving Canadian community of entrepreneurs that seemed to match or exceed the startup activity I've seen in Silicon Valley, Boulder, Boston, New York, Amsterdam, Hong Kong, and Dubai. How times have changed! Investing in the Canadian startup scene was a no-brainer.

Canada Approved

The Catalyst team hit the ground running and immediately started working with GrowLab and several other incredible organizations like Communitech, Ryerson University Digital Media Zone (DMZ), Innovation Factory, Extreme Startups and the Ontario Network of Excellence (ONE).

We'll enroll startups participating in those organizations into the Catalyst Program, and we'll provide infrastructure credits (for servers, storage and networking), executive mentoring, engineering resources and limited financial support. SoftLayer wants to become the de facto Infrastructure as a Service (IaaS) provider for Canadian startups and startups worldwide, so this is a huge first step onto the international stage. More importantly — and on a personal level — I'm excited that we get to help new companies in Canada make a global impact with us.

As a Canadian expat, having the opportunity to give something back means a great deal to me. I see an incredible opportunity to nurture and help some of these Canadian startups take flight. SoftLayer is still an entrepreneurial company at heart, and we have a unique perspective on what it takes to build and scale the next killer app or game, so we feel especially suited to the task.

One of the Canadian entrepreneurs we've been working with sent us this great video produced by the Vancouver-based GROW Conference about entrepreneurship, and it immediately resonated with me, so I wanted to be sure to include it in this post:

We've already started working with dozens entrepreneurs in Vancouver, Toronto, Hamilton and Waterloo who embody that video and have kindred spirits to my own. SoftLayer has a few Canadian ex-pats on our team, and as Catalyst moves into Canada officially, we're all extremely proud of our heritage and the opportunity we have to help.

Some have called our foray into the Canadian market an "international expansion" of sorts, I think of it more as a "coming home party."

-@gkdog

Canada Approved

November 2, 2012

The Trouble with Open DNS Resolvers

In the last couple of days, there's been a bit of buzz about "open DNS resolvers" and DNS amplification DDoS attacks, and SoftLayer's name has been brought up a few times. In a blog post on October 30, CloudFlare explained DNS Amplification DDoS attacks and reported the geographic and network sources of open DNS resolvers that were contributing to a 20Gbps attack on their network. SoftLayer's AS numbers (SOFTLAYER and the legacy THEPLANET-AS number) show up on the top ten "worst offenders" list, and Dan Goodin contacted us to get a comment for a follow-up piece on Ars Technica — Meet the network operators helping to fuel the spike in big DDoS attacks.

While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren't "helping to fuel the spike in big DDoS attacks"). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that's not the case.

The majority of SoftLayer's infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they'd like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we've got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it's easier for those customers and customers-of-customers and customers-of-customers-of-customers to use "defaults" instead of looking for, learning and implementing "best practices."

It's all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It's usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet ... It's a question of whether the DNS owner has any idea that their "configuration" is vulnerable to be abused in this way.

SoftLayer's network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game ... Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the "abusive" traffic is nearly impossible for even the DNS owner to differentiate from "regular" traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you're not sure whether it's configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google's Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you're just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare's post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don't want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

October 30, 2012

Startup Series: YouNoodle

In the startup world, the resources you have are almost as important as your vision and your ability to execute. That simple idea fueled the creation of Catalyst, and it's a big component of our incredible success. We're taking the complexity (and cost) out of the hosting decision for the coolest startups we meet, and by doing so, those startups have the freedom to focus on their applications. But that's only the beginning.

In addition to providing infrastructure, my team and I also try to introduce Catalyst participants to investors, incubators, accelerators and other startup founders. By building a strong network of experienced peers, entrepreneurs have a HUGE advantage as they're building their businesses. The difficulty in making those introductions is that it's such a labor-intensive process ... Or I guess I should say that it *was* a labor-intensive process. Then we found YouNoodle.

YouNoodle is an online network for entrepreneurs that was founded in 2010 in San Francisco, California. The 18-person startup is built to connect entrepreneurs with people, startups, competitions and groups based on what's relevant to each entrepreneur's mission. What the Catalyst team has been doing in a labor-intensive fashion, YouNoodle has automated and streamlined! We had to meet these folks.

YouNoodle

We heard that YouNoodle was putting together a start-up crawl during one of their immersion programs — they bring international entrepreneurs to Silicon Valley to learn best practices and make connections in the US market — and we jumped at an opportunity to provide the beer and sandwiches at one of the stops. If you've ever worked at a startup before, you know that the way to an entrepreneur's heart is through his/her stomach, so we hoped it would be "love at first bite."

We chatted with the YouNoodle team, and they showed us the recently released 2.0 version of Podium, the SaaS platform they built to manage the selection process for entrepreneurial competitions and challenges from organizations like Start-Up Chile, The Next Web, Intel, NASA and seven out of the top ten universities around the world. Basically, Podium enables the most talented individuals and innovative startups to rise to the top and get the opportunities they deserve.

YouNoodle was an obvious fit for Catalyst, and Catalyst was an obvious fit for YouNoodle. Other Catalyst participants could join the thriving community of entrepreneurs that YouNoodle has built, and YouNoodle could take advantage of the power of SoftLayer's hosting platform. And by helping support YouNoodle, Catalyst gets to indirectly help even more entrepreneurs and startups ... Very "meta!"

Over the past two years, YouNoodle has managed over 400 competitions which have received entries from more than 28,000 entrepreneurs around the world. They're a key player in the acceleration of global entrepreneurship, and they share our vision of breaking down the geographic barriers to innovation. And with the momentum they've got now, it's clear that they're just getting started.

If you have a second, head over to YouNoodle.com to check out the fresh, easy-to-use interface they launched to help users discover, get inspired by and connect with like-minded individuals on a global scale.

-@PaulFord

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don't have to be technically savvy to understand what's going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I'm particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you've "taken care of the problem," and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: "How did the issue occur?" "What did you do to resolve the issue?" "What steps are being taken to secure the server in order to prevent further abuse?"

Who cares how the issue happened if it's resolved now, right? Didn't I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it's analogy time:

You're sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It's not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don't see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients') data. In the last installment of "Tips from the Abuse Department," Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that "something's wrong."

At a crucial point like this, it's important to get the water out of the boat AND prevent the vessel from taking on any more water. You won't be sailing smoothly unless both are done as quickly as possible.

Let's look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be "done" with the issue, but that long-time client (who's not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it's quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client "fixes" the problem by deleting the files/directories. Invariably, they'd get compromised again in the same way when the domain is restored, and you'd hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it's not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you'll be using your server resources more efficiently, and (best of all) you won't have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post ...

Pages

Subscribe to business