The Server Challenge II “Hall of Fame” was particularly competitive at Interop 2013. Only 8 seconds separated our first place finisher from tenth place:

Jim Chrapowicz recorded the competition-winning time of 58.40 seconds (after a 5-second penalty for not closing one of the latches), edging out the second place time by a razor-thin margin of less than two tenths of a second. For his Server Challenge II heroics, Jim is being rewarded with the MacBook Air grand prize, and everyone who made the top ten list will be receiving $25 iTunes gift cards. Here’s video of the winning completion:

Take a look at some of the other action from the show floor:

About the Server Challenge II

The Server Challenge II is a race to reassemble a scaled-down version of a SoftLayer server rack. Participants are tasked with repopulating the drive bays of two 2U Supermicro servers and plugging 18 network cables into network switches. The competition provides conference attendees with a fun opportunity to get hands-on with the servers and network gear that fuel SoftLayer’s global cloud infrastructure platform. For more information about the Server Challenge II, check out “Server Challenge II: How SoftLayer Saves the World.”

About SoftLayer

SoftLayer operates a global cloud infrastructure platform built for Internet scale. Spanning 13 data centers in the United States, Asia and Europe and a global footprint of network points of presence, SoftLayer’s modular architecture provides unparalleled performance and control, with a full-featured API and sophisticated automation controlling a flexible unified platform that seamlessly spans physical and virtual devices, and a global network for secure, low-latency communications. With 100,000 devices under management, SoftLayer is the largest privately held Infrastructure-as-a-Service (IaaS) provider in the world with a portfolio of leading-edge customers from Web startups to global enterprises. For more information, visit softlayer.com.

About Supermicro

Supermicro, the leading innovator in high-performance, high-efficiency server technology is a premier provider of advanced server Building Block Solutions for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and Embedded Systems worldwide. Supermicro is committed to protecting the environment through its “We Keep IT Green” initiative and provides customers with the most energy-efficient, environmentally-friendly solutions available on the market. For more information, visit supermicro.com.

This isn’t the first time the advertising-focused crowd at ad:tech has seen the Server Challenge, but with the competition’s new retro arcade game design, it was much more of a focal point this year than it has been in years past … And it didn’t hurt that we were in an awesome location right at the entrance of the expo floor:

Given the fact that most people who stopped at our booth were drawn to us as part of a crowd around the Server Challenge, the first question we heard was subtly different than the “What does SoftLayer do?” question we’re used to answering at ad:tech. This year, most of my conversations started with an attendee asking, “What in the world does this game have to do with SoftLayer?” Luckily, the graphic on the front of the Server Challenge with three simple objectives provides a great outline for the competition’s relevance to our business:

1. Load the Data
2. Connect the Network
3. Save the World

1. Load the Data

Game Application: Insert all 24 of the drive trays into the drive bays of two Supermicro servers.
SoftLayer Significance: We have more than 100,000 Supermicro servers in our 13 data centers around the world. When you walk into one of our facilities in Dallas, Houston, Seattle, Washington, D.C., San Jose, Amsterdam or Singapore, you’ll see racks filled with servers just like the ones in the Server Challenge II, and those servers are loaded up with the hard drives you choose when you order from us.

2. Connect the Network

Game Application: Connect the 18 network cables into the three network switches.
SoftLayer Significance: The three different colors of network cables are the same colors you’ll see in our data centers. The red cables carry public network traffic, the blue cables carry private network traffic, and the green cables carry out-of-band management network traffic. This is a huge differentiator for SoftLayer because those three physical networks allow for much greater flexibility for our customers. While the public network is serving public traffic to your websites, games and apps, you could be running an off-site backup of your database over the private network (where you don’t incur bandwidth charges), and you can manage your server over SSL, PPTP and IPSEC connections via the out-of-band management network carried by the green cables.

3. Save the World

Game Application: Win a MacBook Air!
SoftLayer Significance: SoftLayer provides the flexible, scalable platform on which you can build your application, run your game or push an advertising campaign. The fact that all of our servers are racked, networked and ready for your order means that we’re ready to “Save the World” for you by provisioning on-demand bare metal cloud servers and virtual cloud computing instances.

At least four or five times per show, I hear attendees talking about how the Server Challenge is the most fun game at the conference (even at GDC … where the entire expo hall is filled with gaming companies). While it draws crowds for being fun, the best part of the competition is that it helps us tell our story and creates memories at the same time. When Server Challenge competitors hear that their companies need a new server, they’re going to have a flashback to stepping up to a SoftLayer server rack and learning what makes SoftLayer the best choice as a cloud hosting provider. With the crowds we see at every show, that means we’ve got a lot of future customers:

Thanks to all of the ad:tech attendees who took on the Server Challenge II this year. The show actually had one of the most dramatic conclusions of any we’ve ever had before! Yuki Matsumoto broke the one-minute mark early on Day 2 of the expo with his first attempt of the day, and John Li managed to squeak by him with a time of 0:58.05 less than five minutes before the show floor closed:

Yuki had one shot at redemption as the last competitor of the show, but he wasn’t able to beat John’s 58-second completion, so the MacBook Air went to John Li! Keep practicing your server-building skills and come look for SoftLayer (and the Server Challenge) in an expo hall near you!

-Summer

To get the ball rolling, we need to download CSF and install it on our server. In this post, we’re working with a CentOS 6.0 32-bit server, so our (root) terminal commands would look like this to download and install CSF:

$ wget http://www.configserver.com/free/csf.tgz #Download CSF using wget.
$ tar zxvf csf.tgz #Unpack it.
$ yum install perl-libwww-perl #Make sure perl modules are installed ...
$ yum install perl-Time-HiRes  #Otherwise it will generate an error.
$ cd csf
$ ./install.sh #Install CSF.
 
#MAKE SURE YOU HAVE YOUR BAND-AID READY
 
$ /etc/init.d/csf start #Start CSF. (Note: You can also use '$ service csf start')

Once you start CSF, you can see a list of the default rules that load at startup. CSF defaults to a DROP policy:

$ iptables -nL | grep policy
Chain INPUT (policy DROP)
Chain FORWARD (policy DROP)
Chain OUTPUT (policy DROP)

Don’t ever run “iptables -F” unless you want to lock yourself out. In fact, you might want to add “This server is running CSF – do not run ‘iptables -F’” to your /etc/motd, just as a reminder/warning to others.

CSF loads on startup by default. This means that if you get locked out, a simple reboot probably won’t fix the problem. Runlevels 2, 3, 4, and 5 are all on:

$ chkconfig --list | grep csf
csf             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Some features of CSF will not work unless you have certain iptables modules installed. I believe they are installed by default in CentOS, but if you custom-built your iptables, they might not all be installed. Run this script to see if all modules are installed:

$ /etc/csf/csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
 
RESULT: csf should function on this server

As I mentioned, this is the default iptables installation on a minimal CentOS 6.0 image, so chances are good that these modules are already installed on your system. It never hurts to check, though.

The CSF Configuration File

The primary CSF configuration is stored in the well-documented /etc/csf/csf.conf file. CSF is extremely configurable, so there are a lot of options to read over. Let’s take a look over some of the more important features:

Testing

TESTING = "1"
TESTING_INTERVAL = "5"

This TESTING cron job runs every “5″ minutes so you don’t lock yourself out when you’re testing your rules. When you are satisfied with your rules (and confident that you won’t lock yourself out), you can set TESTING to “0″.

Globally Allowed Ports

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995"
 
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,53,80,110,113,443"
 
# Allow incoming UDP ports
UDP_IN = "20,21,53"
 
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "20,21,53,113,123"

Incoming Ping Requests

# Allow incoming PING
ICMP_IN = "1"

Allowing ping is usually a good option for diagnostic purposes, so I don’t recommend turning it off. Disallowing ping is an example of “security through obscurity,” and it will not typically dissuade your attackers.

Ethernet Device

ETH_DEVICE = ""
ETH6_DEVICE = ""

Here, you can configure iptables to ONLY use one Ethernet adapter. You might want to only guard your public network adapter in some situations.

IP Limit in Permanent “Deny” File

DENY_IP_LIMIT = "200"

A higher number here will obviously screen out more IP addresses in csf.deny, but higher numbers also may cause slowdowns.

IP Limit in Temporary “Deny” File

DENY_TEMP_IP_LIMIT = "100"

Similar to DENY_IP_LIMIT, the DENY_TEMP_IP_LIMIT represents the maximum number of IPs that can be stored in the temporary ban list.

SMTP Blocking

SMTP_BLOCK = "0"

When set to “1″, SMTP_BLOCK does not completely block outbound SMTP, but it does block it for most users. This will prevent malicious scripts and compromised users from making outbound connections from unauthorized mail clients on the server. SMTP_BLOCK doesn’t stop those scripts from running, but it does stop them from functioning. Mail sent through the proper channels will still be delivered normally.

Allowing SMTP on localhost

SMTP_ALLOWLOCAL = "1"

Custom Mail Port Designation

SMTP_PORTS = "25,465,587"

Allowing SMTP Access to Users/Groups

SMTP_ALLOWUSER = ""
SMTP_ALLOWGROUP = "mail,mailman"

SYN Flood Protection

SYNFLOOD = "0"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

Per the documentation, you should only enable SYN flood protection (SYNFLOOD= “1″) if you are currently under a SYN flood attack.

Concurrent Connections Limit

CONNLIMIT = "22;5,80;20"
PORTFLOOD = "22;tcp;5;300,80;tcp;20;5

These options allow you to add customized DoS protection. CONNLIMIT handles the number of concurrent connections, and in this example, we’re limiting port 22 to 5 connections and port 80 to 20 connections.

PORTFLOOD on deals with connections per second. In this example, we’re limiting the TCP connection on port 22 to 5 connections/second with a quiet period of 300 seconds before the connection is unblocked. Additonally, we’re limiting the TCP connection on port 80 to 20 connections/second with a quiet period of 5 seconds before the connection is unblocked.

Check the readme.txt file for more information about the syntax.

Logging to Syslog

SYSLOG = "0"

When enabled, this option logs lfd (Login Failure Daemon) messages to syslog as well as to /var/log/lfd.log.

Dropping v. Rejecting Packets

DROP = "DROP"

This configuration allows you to either DROP or REJECT packets. REJECT tells the sender that the packet has been blocked by the firewall. DROP just drops the packet and does not send a response. I like DROP better for regular use, but REJECT might be more helpful if you need to diagnose a connectivity issue.

Logging Dropped Connections

DROP_LOGGING = "1"

This option logs dropped connections to syslog. I don’t see any reason to turn this off unless your hard drive is getting full.

Port Exceptions When Logging Dropped Connections

DROP_NOLOG = "67,68,111,113,135:139,445,500,513,520"

These ports are specifically blocked from being logged either to conserve hard drive space or make the log file easier to read.

“Watch Mode”

WATCH_MODE = "0"

If you are ever stuck trying to troubleshoot a large ruleset, you might consider turning this option on. You can use it to track the actions to watched IP addresses to see where they are getting blocked or accepted.

Login Failure Daemon Alert

LF_ALERT_TO = ""
LF_ALERT_FROM = ""
LF_ALERT_SMTP = ""

You can specify an email address to report errors from the Login Failure Daemon, which tracks and automatically blocks brute force login attempts.

Permanent Blocks and NetBlocks

LF_PERMBLOCK = "1"
LF_PERMBLOCK_INTERVAL = "86400"
LF_PERMBLOCK_COUNT = "4"
LF_PERMBLOCK_ALERT = "1"
LF_NETBLOCK = "0"
LF_NETBLOCK_INTERVAL = "86400"
LF_NETBLOCK_COUNT = "4"
LF_NETBLOCK_CLASS = "C"
LF_NETBLOCK_ALERT = "1"

These settings control the permanent block and netblock blocking. You probably don’t need to touch these settings, but you might want some additional security or less security depending on your company needs. If something gets permablocked, it will require your intervention to clear it, which might create downtime for your clients. Likewise, if a legitimate IP address happens to be part of a netblock which has an attacking IP address on it, it will get blocked if you have that feature turned on. A class C network encompasses 256 IP addresses. You can set this to class B or A, but that could block thousands or millions of IP addresses, respectively. Unless you find yourself under constant attack, I would advise you to leave that LF_NETBLOCK off.

Additional Protection During Updates

# Safe Chain Update. If enabled, all dynamic update chains (GALLOW*, GDENY*,
# SPAMHAUS, DSHIELD, BOGON, CC_ALLOW, CC_DENY, ALLOWDYN*) will create a new
# chain when updating, and insert it into the relevant LOCALINPUT/LOCALOUTPUT
# chain, then flush and delete the old dynamic chain and rename the new chain.
#
# This prevents a small window of opportunity opening when an update occurs and
# the dynamic chain is flushed for the new rules.
SAFECHAINUPDATE = "0"

Activating this option will increase your system resource usage and will require more rules to be running at one time, but it provides an additional layer of protection during updates. Without this option turned on, your rules will be flushed for a short amount of time, leaving your server vulnerable.

Multi-Server Deployment Options

LF_GLOBAL = "0"
GLOBAL_ALLOW = ""
GLOBAL_DENY = ""
GLOBAL_IGNORE = ""

Like APF, you can configure global lists for multiple server deployments. You’ll need to specify a URL of the text file with the IP addresses for the global lists.

SPAMHAUSE Blocklist

LF_SPAMHAUS = "0"

This option enables the SPAMHAUS blocklist. Specify the number of seconds between refreshes. Recommended setting is 86400 (1 day).

Blocking TOR Exit IP Addresses

LF_TOR = "0"

Enabling this option will block TOR exit IP addresses. If you are not familiar with TOR, it is a completely anonymous proxy network. This could block some legitimate users who are trying to protect their anonymity, so I would recommend only turning this on if you are already under attack from a TOR exit address.

Blocking Bogon Addresses

LF_BOGON = "0"
LF_BOGON_URL = "http://www.cymru.com/Documents/bogon-bn-agg.txt"
LF_BOGON_SKIP = ""

Blocking bogon addresses (addresses that should not be possible) is usually a good decision. To enable, set the number of seconds between refreshes. I recommend enabling this option and setting the refresh at 86400 (1 day). If you do so, be sure to add your private network adapters to the skip list.

Country-Specific Access to Your Server

CC_DENY = ""
CC_ALLOW = ""

With these options, you can block or allow entire countries from accessing your server. To do so, enter the country codes in a comma separated list. Even though this generates a lot of additional rules, it’s valuable to some sysadmins.

CC_ALLOW_FILTER = ""

Alternatively, you can set your server to exclusively accept traffic from a list of country codes. All other countries not listed will have their traffic dropped. There are many other settings related to these options that I don’t have time to cover in this blog.

Blocking Login Failures

LF_TRIGGER = "0"

This enables blocking of login failures (per service). There are a lot of great customization options in this section.

Scanning Directories for Malicious Files

LF_DIRWATCH = "300"

This feature scans /tmp and /dev/shm for potentially malicious files and alerts you to their presence based on the interval you designate. You can also have CSF automatically quarantine malicious files with this option:

LF_DIRWATCH_DISABLE = "0"

Distributed Attack Protection

LF_DISTATTACK = "0"

By enabling this option, you activate additional protection against distributed attacks.

Blocking Based on Abusive Email Usage

LT_POP3D = "0"
LT_IMAPD = "0"

If a user checks email too many times per hour (more than the non-zero value specified), the user’s IP address is blocked.

Email Alert Following Block

LT_EMAIL_ALERT = "1"

This will send you email when something is blocked. I’d recommend leaving it on.

Blocking IP Addresses Based on Number of Connections

CT_LIMIT = "0"

This feature tracks connections and blocks the IP if the number of connections is too high. Use caution because if you enable this option and set this value too low, it will block legitimate traffic.

Application-Level Protection

PT_LIMIT = "60"

This feature provides application level protection against malicious scripts that take a long time to execute.

Blocking Port Scanners

PS_INTERVAL = "300"
PS_LIMIT = "10"

Enabling HTML User Interface for CSF

UI = "0"

CSF has a built-in HTML user interface. You can enable this by setting UI = “1″. There are a list of prerequisites for this option in the readme.txt.

Notifying Blocked IP Addresses

MESSENGER = "0"

This option will notify blocked IP addresses when they have been blocked by the firewall.

Port Knocking

PORTKNOCKING = ""

CSF supports port knocking, which is a technique that provides an additional layer of security. See http://www.portknocking.org/ for details.

Allow and Deny Lists

As we walked through the CSF configuration file, you saw that I referenced the csf.deny file, so it should come as no surprise that CSF also includes csf.allow to customize “allow” rules as well. If you are familiar with APF, these files have a very similar syntax … Each entry is made up of the same four components: protocol|flow|port|IP. The only real difference being that APF uses the colon as a delimiter while CSF uses the pipe:

#APF Version
tcp:in:d=48000_48020:s=10.0.0.0/8
 
#CSF Version
tcp|in|d=48000_48020|s=10.0.0.0/8

Fortunately, replacing your colon with a pipe is a minimally invasive procedure that can be automated with a tool like vi.

CSF Command Line Tool

The command line tool for CSF is much more robust than the one for APF:

$ csf --help
csf: v5.79 (cPanel)
 
ConfigServer Security & Firewall
(c)2006-2013, Way to the Web Limited (http://www.configserver.com)
 
Usage: /usr/sbin/csf [option] [value]
 
Option              Meaning
-h, --help          Show this message
-l, --status        List/Show iptables configuration
-l6, --status6      List/Show ip6tables configuration
-s, --start         Start firewall rules
-f, --stop          Flush/Stop firewall rules (Note: lfd may restart csf)
-r, --restart       Restart firewall rules
-q, --startq        Quick restart (csf restarted by lfd)
-sf, --startf       Force CLI restart regardless of LF_QUICKSTART setting
-a, --add ip        Allow an IP and add to /etc/csf.allow
-ar, --addrm ip     Remove an IP from /etc/csf.allow and delete rule
-d, --deny ip       Deny an IP and add to /etc/csf.deny
-dr, --denyrm ip    Unblock an IP and remove from /etc/csf.deny
-df, --denyf        Remove and unblock all entries in /etc/csf.deny
-g, --grep ip       Search the iptables rules for an IP match (incl. CIDR)
-t, --temp          Displays the current list of temp IP entries and their TTL
-tr, --temprm ip    Remove an IPs from the temp IP ban and allow list
-td, --tempdeny ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP ban list. ttl is how long to
                    blocks for (default:seconds, can use one suffix of h/m/d).
                    Optional port. Optional direction of block can be one of:
                    in, out or inout (default:in)
-ta, --tempallow ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP allow list (default:inout)
-tf, --tempf        Flush all IPs from the temp IP entries
-cp, --cping        PING all members in an lfd Cluster
-cd, --cdeny ip     Deny an IP in a Cluster and add to /etc/csf.deny
-ca, --callow ip    Allow an IP in a Cluster and add to /etc/csf.allow
-cr, --crm ip       Unblock an IP in a Cluster and remove from /etc/csf.deny
-cc, --cconfig [name] [value]
                    Change configuration option [name] to [value] in a Cluster
-cf, --cfile [file] Send [file] in a Cluster to /etc/csf/
-crs, --crestart    Cluster restart csf and lfd
-w, --watch ip      Log SYN packets for an IP across iptables chains
-m, --mail [addr]   Display Server Check in HTML or email to [addr] if present
-lr, --logrun       Initiate Log Scanner report via lfd
-c, --check         Check for updates to csf but do not upgrade
-u, --update        Check for updates to csf and upgrade if available
-uf                 Force an update of csf
-x, --disable       Disable csf and lfd
-e, --enable        Enable csf and lfd if previously disabled
-v, --version       Show csf version

The command line tool will also tell you if the testing mode is enabled (which is a very useful feature). If TESTING were enabled, we’d see this line at the bottom of the output:

*WARNING* TESTING mode is enabled - do not forget to disable it in the configuration

Did you make it all the way through?! Great! I know it’s a lot to take in, but it’s not terribly complicated when we break it down and understand how each piece works. Next time, I’ll be back with some tips on integrating CSF into cPanel.

-Mark

Walk down any aisle of an expo hall, and you’ll see collateral and swag beckoning to attendees like a candy bar at the grocery store register. Some exhibitors rely on Twitter to monitor an event’s hashtag and swoop in at every opportunity to reach the show’s influential attendees. Other exhibitors might send out emails to their clients and prospects in the area to invite them to the show. We see value in each of those approaches, but what we found to be most effective was to bring a SoftLayer data center to our booth … or at least a piece of one.

The Server Challenge has come a long way over the years. Its meager beginnings involved installing RAM and hard drive cables in a tower server. Shortly thereafter, a rack-mount server replaced the tower server, but you were still tasked with “inside the server” challenges. As we started looking for ways to tell the bigger SoftLayer story with the Server Challenge, we moved to miniature server rack, and the competition really started to pick up steam. This year, we made it our goal to take the Server Challenge to the next level, and when Supermicro stepped in to sponsor the next iteration of the the competition, we started thinking BIG.

Why use a miniature version of a SoftLayer rack when we could use a full-size version? Why have a standalone screen when rack-mount monitors can make the display part of the unit? Why rely on speakers behind the booth to pump “Eye of the Tiger” while attendees are competing when we could easily build those into the next version of the challenge? What was initially intended to be a “tweak” of the first Server Challenge became a complete overhaul … Hence the new “Server Challenge II” moniker.

Harkening back to the 8-bit glory days of Pac Man and Space Invaders, the Server Challenge II uses a full-size 42U server rack with vintage arcade-style branding, a built-in timer and speakers that blast esoteric video game music. The bread and butter of the challenge is the actual server hardware, though … Supermicro provided two new 2U servers to replace the previous version’s five 1U servers, and we installed the same Cisco (public and private networks) and SMC (out-of-band management network) switches you see in SoftLayer’s pods.

We had two instances of the original Server Challenge (one in the US, one in Amsterdam), so in order for the Server Challenge II to be bigger and better, we had to increase that total to five — one instance in Europe, one in Asia and three in the United States. Things might get a little crazier logistically, but as a potential conference attendee, it means you’re even more likely to encounter the Server Challenge II if you attend any events with us.

The Server Challenge II’s Internal Debut

The first instance of the Server Challenge II made its debut at GDC Online in Austin, and we immediately knew we had a hit. By the time the rack got back to our office, we had to get it ready for its next destination (Cloud Expo West), but before we sent it on its way, we gave it an official internal debut … and raised some money for the American Heart Association in the process.

SLayers at the SoftLayer HQ in Dallas could pay $3 for one attempt or $5 for two attempts to reach the top of the Server Challenge II leader board. Needless to say, it was competitive. If you click on the image above, you’ll notice that our fearless leader, Lance Crosby, stopped by and gave tips to (and/or heckled) a few participants. Unsurprisingly, one of our very talented Server Build Technicians — Ellijah Fleites — took home a MacBook Air and bragging rights as SoftLayer champion with a record time of 1:03.79 … But records are made to be broken.

In Two Places at Once

Immediately after the AHA fundraiser, we crated up the rack and sent it along to Cloud Expo West in Santa Clara. A few days later, we put the finishing touches on the second Server Challenge II rack, and because we got it done quickly, we were able to get it shipped to the other side of the country for ad:tech NYC. We would finally have the competition running in two places at the exact same time!

We weren’t disappointed.

On both coasts, the retro style of the Server Challenge II lured some fantastic competitors (excellent!), and started a lot of great conversations (even better!). Here are the final leader boards from the shows:

You probably notice that the times in the ad:tech leader board are a little higher than the times in the Cloud Expo leader board, and our team figured out why that was in the middle of the second day of the conference … The way we bound the network cables differed slightly between the two instances, and we were using different switches to time the competition (one that required only one hand to activate/deactivate, the other requiring both hands). In order to have an “apples-to-apples” comparison between all of our shows, we’re going to make sure everything is consistent with all of the instances, and we plan on keeping a running list of fastest overall challenge times … and maybe even a “World Championship” one day.

Given the early success of the Server Challenge II, you can bet that it’s not going anywhere any time soon. If we have multiple shows running the challenge at one time, we might even fire up a video chat where you can compete against an attendee at a completely different conference … so be prepared.

In the next year, we’ll have all five of the Server Challenge II instances in rotation across three continents, and with the popularity of the competition growing by leaps and bounds after every show, we hope by next holiday season, a home version of the Server Challenge II is at the top of every wish list on the planet.

For now, though, I’ll just leave you with a glimpse at the action from Cloud Expo West (click for more pictures from the show):

-Raleigh

Most spam tickets we create on customer accounts pinpoint spam sent from a compromised or exploited server. Our direct customer didn’t send the phishing email, malware distribution, pharmacy advertisement or pornographic spam, but that activity came from their account. While they’re accountable for the abusive behavior coming from their server, in many cases, they don’t know that there’s a problem until we post an abuse ticket on their account. These servers are targeted and compromised by common techniques and exploits that could have been easily avoided, but they aren’t very well known outside the world of abuse.

To protect yourself from a spammer, you need to think like a spammer. You need to understand how someone might try to exploit your environment so that you can prevent them from doing so. As you’re looking at ways to secure your server proactively, make sure you target these five exploits in particular:

1. User Auth Login

This is by far the most common exploit to used to send spam. This method involves a person or script using the credentials of a user to send spam through a domain’s mail server. The majority of these incidences are caused by malware on a client PC that obtains the login and password for a domain user and uses that information to log on and send mail from the client PC through the server. Often, these spam messages are sent through a botnet command structure.

When an account is compromised, simply changing the password for the compromised user on the server usually won’t stop the abuse. We see quite a few accounts that continue to send spam after an initial abuse ticket results in a password change. Most servers that are sending spam with this method are found to only be sending a small amount of spam at any given time to avoid detection. The low volume of spam that is being sent per server is made up for by the fact that there are thousands of servers being used for the same spamming campaigns.

In order to stop the User Auth Login exploit, a customer needs to clean all of the malicious software (malware) from their environments. To prevent future User Auth Login compromises, users should be made aware of the potential dangers of untrusted software, and if they believe their machines are infected, they need to know what to do.

2. Tell-a-friend Exploitation

The User Auth Login technique is the most common method employed by spammers, but the “tell-a-friend” script exploitation isn’t far behind when it comes to volume of affected servers. This spamming method find websites that use scripts to invite users to refer friends to a page or product. Spammers will use the ‘Your Message’ field in one of these scripts to input their own content and links, and they’ll push the actual page referral link to the bottom of the message. When these site scripts aren’t secure, the spammer will use them to send hundreds or thousands of messages.

To avoid having your website fall victim to this type of spam, be very wary of any widget or script you add. If you need to add Facebook, Twitter and email “share” functionality to your site, make sure you incorporate a tell-a-friend script that does not allow for customizable messages or does not accept input of more than one email address. Also, users won’t need the “cc” or “bcc” fields, so you can be sure those are axed as well. If you can’t find a good “share” script that you’re comfortable with from a security perspective, it might be a good idea to remove that functionality to avoid exploitation.

3. Uploaded Mailers

Spam sent via an uploaded third party mailer can sometimes prove difficult for admins to locate. An uploaded third party mailer could be capable of creating it’s own outbound SMTP connection, and that would allow a program to bypass the existing MTA on the server and render any legitimate mail logs useless for investigation. Another challenge is that a php mailer can be uploaded to a location within a user’s web content, and that mailer is run by the user ‘nobody’ (the default Apache user).

We strongly suggest configuring your server to have the mail headers show the script’s user (that’s not the Apache default user) and the location the script is running from on the server. Many times, these kinds of mailers are maliciously uploaded after a user’s FTP password is been compromised, so be sure your FTP login information is secure.

4. Software Exploits

The “software exploits” category casts a huge shadow. Every piece of software on a server — from mail servers, content management systems and control panels to the operating system itself — can be targeted by hackers. They probe servers to find security vulnerabilities and weak coding, and when they find a vulnerability, they take control.

The hacker who found the software vulnerability might not actually take advantage of the exploit immediately. That user may sell access to other entities for their use, and that use often ends up being spam. In addition to having strong firewall rules and access restrictions, you should update and maintain the current stable versions of all software on your servers.

5. WordPress Exploits

WordPress exploits would technically fall under the “Software Exploits” category, but I’m breaking it out into its own category simply due to the volume of spam issues that are the result of exploiting this particular piece of software. The first step to protecting against spam being sent through this source is to make sure you have the latest version of WordPress installed. With that done, be sure to research the latest security plugins for that version and install any that are applicable to your environment.

These five techniques are not the only ones used by spammers to take advantage of your environment, but they are some of the most common. To protect yourself from becoming a source of spam, make your servers a more difficult target to exploit. To stop spam, you need to know spam. Now that you know spam, it’s time to stop it. Ask questions, test your environment regularly and watch your logs for any unexplained usage.

-Andrew

HostingCon brings the hosting industry together every year, and the conference winds up being surprisingly similar to classic 80′s “coming of age” movies:

• “Geeks” are among the main characters.
• There’s always a “funny guy.”
• At some point, the geeks attend a party.
• The characters learn more about themselves and others over the course of the movie.
• As the credits roll, everyone is inspired … Ready to take on the world.

With that in mind, HostingCon 2012 in Boston was a veritable John Hughes flick. There was no shortage of geeks, we hung out with one of the funniest people in the country, we threw a massive party, and we learned a ton. Without a doubt, attendees returned home with their intensity and enthusiasm cranked up to eleven (another 80′s reference).

The expo hall was abuzz with activity — albeit after a lull in the morning following the aptly named “Host Me All Night Long” party — and we enjoyed the opportunity to catch up with current partners and customers while meeting and speaking with soon-to-be partners and customers. While running a highly competitive Server Challenge, we were still able to dive deeper into partnerships, the build v. buy decision, branding, and launching a product when attendees visited our booth after hearing from our team in conference sessions and panels, and those conversations are what keep us coming back to HostingCon every year.

As a “veteran” of the hosting industry (assuming seven years of experience qualifies me), I’ve learned a great deal about the dynamics of the hosting industry from events like HostingCon over the years. On one hand, many of the attendees are “competitors,” and on the other hand, we’re all trying to make the industry better (since “a rising tide lifts all boats”). As a great example, look at the Internet Infrastructure Coalition (i2C), a trade association of companies with the shared goal and purpose of representing the industry in Washington, D.C., and beyond.

As it turns out, that unity flew out the door when attendees stood face-to-rack with the Server Challenge, though. Unlike our experiences at more general “technology” conferences, the components in our competition needed no introduction, and participants were particularly driven to best their peers … not only for the iPad, but for the pride of owning the Server Challenge title at HostingCon:

1. Darin Goldman – 0:59.28
2. Devon Hillard – 1:01.58
3. Ijan Kruizinga – 1:01.83
4. Jon Basha – 1:03.02
5. Sean Whitley – 1:03.06

As you saw in the video, Darin Goldman had the luxury of not needing his second attempt on the final day of the conference to secure a victory, but we were glad he let us record his “Breakfast Club” fist-pump to share with the world.

Don’t stop believing (in hosting).

-@khazard

P.S. I recorded the first few minutes of Ralphie May‘s set, but the adult language-ness of the content makes it a little more difficult to share with the world.

What about the customers who aren’t able to visit us, though? We can post pictures, share stats, describe our architecture and show you diagrams of our facilities, but those mediums can’t replace the experience of an actual data center tour. In the interest of bridging the “data center tour” gap for customers who might not be able to visit SoftLayer in person (or who want to show off their infrastructure), we decided to record a video data center tour.

If you’ve seen “professional” video data center tours in the past, you’re probably positioning a pillow on top of your keyboard right now to protect your face if you fall asleep from boredom when you hear another baritone narrator voiceover and see CAD mock-ups of another “enterprise class” facility. Don’t worry … That’s not how we roll:

Josh Daley — whose role as site manager of DAL05 made him the ideal tour guide — did a fantastic job, and I’m looking forward to feedback from our customers about whether this data center tour style is helpful and/or entertaining.

If you want to see more videos like this one, “Like” it, leave comments with ideas and questions, and share it wherever you share things (Facebook, Twitter, your refrigerator, etc.).

-@khazard

That’s pretty intimidating … Even for tech businesses.

Shifts in technology don’t have to be as drastic and obvious as a “printed newspaper v. social news site” comparison for them to be disruptive. Even subtle advances can wind up making or breaking a business. In fact, many of today’s biggest and most successful tech companies are scrambling to adapt to two simple “game changers” that seem terribly significant:

• IPv6
• “The Cloud”

IPv6

A quick search of the SoftLayer Blog reminds me that Lance first brought up the importance of IPv6 adoption in October 2007:

ARIN has publically announced the need to shift to IPv6 and numerous articles have outlined the D-Day for IPv4 space. Most experts agree, its coming fast and that it will occur sometime in 2010 at the current pace (that’s about two years for those counting). IPv6 brings enough IP space for an infinite number of users along with improved security features and several other operational efficiencies that will make it very popular. The problem lies between getting from IPv4 to IPv6.

When IPv4 exhaustion was just a blip on the horizon, many businesses probably thought, “Oh, I’ll get around to it when I need to. It’s not a problem yet.” When IANA exhausted the IPv4 pool, they probably started picking up the phone and calling providers to ask what plans they had in place. When some of the Internet’s biggest websites completed a trial transition to IPv6 on World IPv6 Day last year, those businesses started feeling the urgency. With today’s World IPv6 Launch, they know something has to be done.

Regardless of how conservative providers get with IPv4 space, the 4,294,967,296 IPv4 addresses in existence will not last much longer. Soon, users will be accessing an IPv6 Internet, and IPv4-only websites will lose their opportunity to reach those users. That’s a “game changer.”

“The Cloud”

The other “game changer” many tech businesses are struggling with these days is the move toward “the cloud.” There are a two interesting perspectives in this transition: 1) The challenge many businesses face when choosing whether to adopt cloud computing, and 2) The challenges for businesses that find themselves severing as an integral (sometimes unintentional) part of “the cloud.” You’ve probably seen hundreds of blog posts and articles about the first, so I’ll share a little insight on the second.

When you hear all of the hype about cloud computing and cloud storage offering a hardware-agnostic Utopia of scalable, reliable power, it’s easy to forget that the building blocks of a cloud infrastructure will usually come from vendors that provided a traditional hosting resources. When a computing instance is abstracted from a hardware device, it’s opens up huge variations in usage. It’s possible to have dozens of public cloud instances using a single server’s multi-proc, multi-core resources at a given time. If a vendor prices a piece of software on a “per server” basis, how do they define a “server” when their users are in the cloud? It can be argued that a cloud computing instance with a single core of power is a “server,” and on the flip-side, it’s easy to define a “server” as the hardware object on which many cloud instances may run. I don’t know that there’s an easy way to answer that question, but what I do know is that applying “what used to work” to “what’s happening now” isn’t the right answer.

The hardware and software providers in the cloud space who are able to come up with new approaches unencumbered by the urge to continue “the way we’ve always done it” are going to be the ones that thrive when technology “game changers” emerge, and the providers who dig their heels in the dirt or try to put a square peg into a round hole will get the short end of the “adapt or die” stick.

We’ve tried to innovate and take a fresh look at every opportunity that has come our way, and we do our best to build relationships with agile companies that we see following suit.

I guess a better way to position the decision at the beginning of this post would be to add a little tweak: “Innovate, adapt or die.” How you approach technology “game changers” will define your business’s success.

-@gkdog

But what if we wanted to show what automation actually looks like?

It seems like a silly question to ask. If our automated platform is powered by software built by the SoftLayer development team, there’s no easy way to show what that automation looks like … At least not directly. While the bits and bytes aren’t easily visible, the operational results of automation are exceptionally photogenic. Let’s take a look at a few examples of what automation enables to get an indirect view of what it actually looks like.

Example: A New Server Order

A customer orders a dedicated server. That customer wants a specific hardware configuration with a specific suite of software in a specific data center, and it needs to be delivered within four hours. What does that usually look like from an operations perspective?

If you want to watch those blinking lights for two or three hours, you’ll have effectively watched a new server get provisioned at SoftLayer. When an order comes in, the automated provisioning system will find a server matching the order’s hardware requirements in the requested data center facility, and the software will be installed before it is handed over to the the customer.

Example: Server Reboot or Operating System Reload

A customer needs to reboot a server or install a new operating system. Whether they want a soft reboot, a hard reboot with a full power cycle or a blank operating system install, the scene in the data center will look eerily familiar:

Gone are the days of server build technicians wheeling a terminal over to every server that needs work done. From thousands of miles away, a customer can remotely “unplug” his or her server via the rack’s power strip, initiate a soft reboot or reinstall an operating system. But what if they want even more accessibility?

Example: What’s on the Screen?

When remotely rebooting or power cycling a server isn’t enough, a customer might want someone in the data center to wheel over to their server in the rack to look at any of the messages that can only be read with a monitor attached. This would generally happen behind the server, but for the sake of this example, we’ll just watch the data center technician pass in front of the servers to get to the back:

Yeah, you probably could have seen that one coming.

Because KVM over IP is included on every server, physical carts carrying “keyboard, video and mouse” are few and far between. By automating customers’ access to their server and providing as much virtual access as we possibly can, we’re able to “get out of the way” of our technical users and only step in to help when that help is needed.

I could go on and on with examples of cloud computing upgrades and downgrades, provisioning a firewall or adding a load balancers, but I’ll practice a little restraint. If you want the full effect, you can scroll up and watch the blinking lights a little while longer.

Automation looks like what you don’t see. No humanoid robots or needlessly complex machines (that I know of) … Just a data center humming along with some beautiful flashing server lights.

-Duke

P.S. If you want to be able to remotely bask in the glow of some blinking server lights, bookmark the larger-sized SoftLayer Rack animated gif … You could even title the bookmark, “Check on the Servers.”

In the first post of this series, we talked about the three key deciding factors every cloud customer has to consider, and we set up a Venn diagram to distinguish the surprisingly broad range of unique priorities customers can have:

Because every customer will prioritize a cloud’s cost, technology and hosting provider a little differently (for completely valid reasons), we mapped out seven distinct “zones” to differentiate some of the basic market segments, or “personas,” of cloud hosting buyers. That post was intended to set the stage for a larger discussion on how customers choose their cloud providers and how cloud providers choose their customers, and we’re just scratching the surface. We’re tackling a pretty big topic here, so as Bill Cosby famously says, “I told you that story to tell you this one.”

As a hosting provider, SoftLayer can’t expect to be all things for all people. It’s impossible to offer a quad-core hex-proc dedicated server for a price that will appeal to a customer in the market for a $49/mo dedicated server.

To better illustrate SoftLayer’s vision in the cloud market, we need to take that generic cost v. technology v. hosting provider diagram and give it the “Three Bars” treatment:

We’re much more interested in living and breathing the Zone 5 “Technology” space rather than the traditional Zone 2 “Hosting Provider” space. That’s why in the past two months, you’ve seen announcements about our launch of the latest Intel Processors, HPC computing with NVidia GPUs, searchable OpenStack Object Storage, and an innovative “Flex Image” approach to bluring the lines between physical and virtual servers. We choose to pursue the cloud customers who make their buying decisions in Zone 3.

That’s a challenging pursuit … It’s expensive to push the envelope in technology, customers primarily interested in technology/performance have demanding needs and expectations, and it’s easier to make mistakes when you’re breaking new ground. The majority of the hosting industry seems to have an eye on the buyer in Zone 1 because they believe the average hosting customer is only interested in the bottom line … That hosting is more or less a commodity, so the focus should be on some unverifiable qualitative measure of support or the next big special that’ll bring in new orders.

As you may have seen recently, GigaOm posted a lovely article that references several high-profile companies in our 25,000+ customer family. We like to say that SoftLayer builds the platform on which our customers build the future, and that short post speaks volumes about the validity of that statement. Our goal is to provide the most powerful, scalable and seamlessly integrated IT infrastructure for the most innovative companies in the world. Innovate or Die isn’t just our company motto … It’s our hope for our customers, as well.

We might miss out on your business if you want a $49/mo dedicated server, but if you’re looking to change the world, we’ve got you covered.

-@khazard