Posts Tagged 'Servers'

March 19, 2013

iptables Tips and Tricks: CSF Configuration

In our last "iptables Tips and Tricks" installment, we talked about Advanced Policy Firewall (APF) configuration, so it should come as no surprise that in this installment, we're turning our attention to ConfigServer Security & Firewall (CSF). Before we get started, you should probably run through the list of warnings I include at the top of the APF blog post and make sure you have your Band-Aid ready in case you need it.

To get the ball rolling, we need to download CSF and install it on our server. In this post, we're working with a CentOS 6.0 32-bit server, so our (root) terminal commands would look like this to download and install CSF:

$ wget http://www.configserver.com/free/csf.tgz #Download CSF using wget.
$ tar zxvf csf.tgz #Unpack it.
$ yum install perl-libwww-perl #Make sure perl modules are installed ...
$ yum install perl-Time-HiRes  #Otherwise it will generate an error.
$ cd csf
$ ./install.sh #Install CSF.
 
#MAKE SURE YOU HAVE YOUR BAND-AID READY
 
$ /etc/init.d/csf start #Start CSF. (Note: You can also use '$ service csf start')

Once you start CSF, you can see a list of the default rules that load at startup. CSF defaults to a DROP policy:

$ iptables -nL | grep policy
Chain INPUT (policy DROP)
Chain FORWARD (policy DROP)
Chain OUTPUT (policy DROP)

Don't ever run "iptables -F" unless you want to lock yourself out. In fact, you might want to add "This server is running CSF - do not run 'iptables -F'" to your /etc/motd, just as a reminder/warning to others.

CSF loads on startup by default. This means that if you get locked out, a simple reboot probably won't fix the problem. Runlevels 2, 3, 4, and 5 are all on:

$ chkconfig --list | grep csf
csf             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Some features of CSF will not work unless you have certain iptables modules installed. I believe they are installed by default in CentOS, but if you custom-built your iptables, they might not all be installed. Run this script to see if all modules are installed:

$ /etc/csf/csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
 
RESULT: csf should function on this server

As I mentioned, this is the default iptables installation on a minimal CentOS 6.0 image, so chances are good that these modules are already installed on your system. It never hurts to check, though.

The CSF Configuration File

The primary CSF configuration is stored in the well-documented /etc/csf/csf.conf file. CSF is extremely configurable, so there are a lot of options to read over. Let's take a look over some of the more important features:

Testing

TESTING = "1"
TESTING_INTERVAL = "5"

This TESTING cron job runs every "5" minutes so you don't lock yourself out when you're testing your rules. When you are satisfied with your rules (and confident that you won't lock yourself out), you can set TESTING to "0".

Globally Allowed Ports

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995"
 
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,53,80,110,113,443"
 
# Allow incoming UDP ports
UDP_IN = "20,21,53"
 
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "20,21,53,113,123"

Incoming Ping Requests

# Allow incoming PING
ICMP_IN = "1"

Allowing ping is usually a good option for diagnostic purposes, so I don't recommend turning it off. Disallowing ping is an example of "security through obscurity," and it will not typically dissuade your attackers.

Ethernet Device

ETH_DEVICE = ""
ETH6_DEVICE = ""

Here, you can configure iptables to ONLY use one Ethernet adapter. You might want to only guard your public network adapter in some situations.

IP Limit in Permanent "Deny" File

DENY_IP_LIMIT = "200"

A higher number here will obviously screen out more IP addresses in csf.deny, but higher numbers also may cause slowdowns.

IP Limit in Temporary "Deny" File

DENY_TEMP_IP_LIMIT = "100"

Similar to DENY_IP_LIMIT, the DENY_TEMP_IP_LIMIT represents the maximum number of IPs that can be stored in the temporary ban list.

SMTP Blocking

SMTP_BLOCK = "0"

When set to "1", SMTP_BLOCK does not completely block outbound SMTP, but it does block it for most users. This will prevent malicious scripts and compromised users from making outbound connections from unauthorized mail clients on the server. SMTP_BLOCK doesn't stop those scripts from running, but it does stop them from functioning. Mail sent through the proper channels will still be delivered normally.

Allowing SMTP on localhost

SMTP_ALLOWLOCAL = "1"

Custom Mail Port Designation

SMTP_PORTS = "25,465,587"

Allowing SMTP Access to Users/Groups

SMTP_ALLOWUSER = ""
SMTP_ALLOWGROUP = "mail,mailman"

SYN Flood Protection

SYNFLOOD = "0"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

Per the documentation, you should only enable SYN flood protection (SYNFLOOD= "1") if you are currently under a SYN flood attack.

Concurrent Connections Limit

CONNLIMIT = "22;5,80;20"
PORTFLOOD = "22;tcp;5;300,80;tcp;20;5

These options allow you to add customized DoS protection. CONNLIMIT handles the number of concurrent connections, and in this example, we're limiting port 22 to 5 connections and port 80 to 20 connections.

PORTFLOOD watches the number of connections per a given number of seconds. In this example, we're limiting the TCP connection on port 22 to 5 connections/second with a quiet period of 300 seconds before the connection is unblocked. Additonally, we're limiting the TCP connection on port 80 to 20 connections/second with a quiet period of 5 seconds before the connection is unblocked.

Check the readme.txt file for more information about the syntax.

Logging to Syslog

SYSLOG = "0"

When enabled, this option logs lfd (Login Failure Daemon) messages to syslog as well as to /var/log/lfd.log.

Dropping v. Rejecting Packets

DROP = "DROP"

This configuration allows you to either DROP or REJECT packets. REJECT tells the sender that the packet has been blocked by the firewall. DROP just drops the packet and does not send a response. I like DROP better for regular use, but REJECT might be more helpful if you need to diagnose a connectivity issue.

Logging Dropped Connections

DROP_LOGGING = "1"

This option logs dropped connections to syslog. I don't see any reason to turn this off unless your hard drive is getting full.

Port Exceptions When Logging Dropped Connections

DROP_NOLOG = "67,68,111,113,135:139,445,500,513,520"

These ports are specifically blocked from being logged either to conserve hard drive space or make the log file easier to read.

"Watch Mode"

WATCH_MODE = "0"

If you are ever stuck trying to troubleshoot a large ruleset, you might consider turning this option on. You can use it to track the actions to watched IP addresses to see where they are getting blocked or accepted.

Login Failure Daemon Alert

LF_ALERT_TO = ""
LF_ALERT_FROM = ""
LF_ALERT_SMTP = ""

You can specify an email address to report errors from the Login Failure Daemon, which tracks and automatically blocks brute force login attempts.

Permanent Blocks and NetBlocks

LF_PERMBLOCK = "1"
LF_PERMBLOCK_INTERVAL = "86400"
LF_PERMBLOCK_COUNT = "4"
LF_PERMBLOCK_ALERT = "1"
LF_NETBLOCK = "0"
LF_NETBLOCK_INTERVAL = "86400"
LF_NETBLOCK_COUNT = "4"
LF_NETBLOCK_CLASS = "C"
LF_NETBLOCK_ALERT = "1"

These settings control the permanent block and netblock blocking. You probably don't need to touch these settings, but you might want some additional security or less security depending on your company needs. If something gets permablocked, it will require your intervention to clear it, which might create downtime for your clients. Likewise, if a legitimate IP address happens to be part of a netblock which has an attacking IP address on it, it will get blocked if you have that feature turned on. A class C network encompasses 256 IP addresses. You can set this to class B or A, but that could block thousands or millions of IP addresses, respectively. Unless you find yourself under constant attack, I would advise you to leave that LF_NETBLOCK off.

Additional Protection During Updates

# Safe Chain Update. If enabled, all dynamic update chains (GALLOW*, GDENY*,
# SPAMHAUS, DSHIELD, BOGON, CC_ALLOW, CC_DENY, ALLOWDYN*) will create a new
# chain when updating, and insert it into the relevant LOCALINPUT/LOCALOUTPUT
# chain, then flush and delete the old dynamic chain and rename the new chain.
#
# This prevents a small window of opportunity opening when an update occurs and
# the dynamic chain is flushed for the new rules.
SAFECHAINUPDATE = "0"

Activating this option will increase your system resource usage and will require more rules to be running at one time, but it provides an additional layer of protection during updates. Without this option turned on, your rules will be flushed for a short amount of time, leaving your server vulnerable.

Multi-Server Deployment Options

LF_GLOBAL = "0"
GLOBAL_ALLOW = ""
GLOBAL_DENY = ""
GLOBAL_IGNORE = ""

Like APF, you can configure global lists for multiple server deployments. You'll need to specify a URL of the text file with the IP addresses for the global lists.

SPAMHAUSE Blocklist

LF_SPAMHAUS = "0"

This option enables the SPAMHAUS blocklist. Specify the number of seconds between refreshes. Recommended setting is 86400 (1 day).

Blocking TOR Exit IP Addresses

LF_TOR = "0"

Enabling this option will block TOR exit IP addresses. If you are not familiar with TOR, it is a completely anonymous proxy network. This could block some legitimate users who are trying to protect their anonymity, so I would recommend only turning this on if you are already under attack from a TOR exit address.

Blocking Bogon Addresses

LF_BOGON = "0"
LF_BOGON_URL = "http://www.cymru.com/Documents/bogon-bn-agg.txt"
LF_BOGON_SKIP = ""

Blocking bogon addresses (addresses that should not be possible) is usually a good decision. To enable, set the number of seconds between refreshes. I recommend enabling this option and setting the refresh at 86400 (1 day). If you do so, be sure to add your private network adapters to the skip list.

Country-Specific Access to Your Server

CC_DENY = ""
CC_ALLOW = ""

With these options, you can block or allow entire countries from accessing your server. To do so, enter the country codes in a comma separated list. Even though this generates a lot of additional rules, it's valuable to some sysadmins.

CC_ALLOW_FILTER = ""

Alternatively, you can set your server to exclusively accept traffic from a list of country codes. All other countries not listed will have their traffic dropped. There are many other settings related to these options that I don't have time to cover in this blog.

Blocking Login Failures

LF_TRIGGER = "0"

This enables blocking of login failures (per service). There are a lot of great customization options in this section.

Scanning Directories for Malicious Files

LF_DIRWATCH = "300"

This feature scans /tmp and /dev/shm for potentially malicious files and alerts you to their presence based on the interval you designate. You can also have CSF automatically quarantine malicious files with this option:

LF_DIRWATCH_DISABLE = "0"

Distributed Attack Protection

LF_DISTATTACK = "0"

By enabling this option, you activate additional protection against distributed attacks.

Blocking Based on Abusive Email Usage

LT_POP3D = "0"
LT_IMAPD = "0"

If a user checks email too many times per hour (more than the non-zero value specified), the user's IP address is blocked.

Email Alert Following Block

LT_EMAIL_ALERT = "1"

This will send you email when something is blocked. I'd recommend leaving it on.

Blocking IP Addresses Based on Number of Connections

CT_LIMIT = "0"

This feature tracks connections and blocks the IP if the number of connections is too high. Use caution because if you enable this option and set this value too low, it will block legitimate traffic.

Application-Level Protection

PT_LIMIT = "60"

This feature provides application level protection against malicious scripts that take a long time to execute.

Blocking Port Scanners

PS_INTERVAL = "300"
PS_LIMIT = "10"

Enabling HTML User Interface for CSF

UI = "0"

CSF has a built-in HTML user interface. You can enable this by setting UI = "1". There are a list of prerequisites for this option in the readme.txt.

Notifying Blocked IP Addresses

MESSENGER = "0"

This option will notify blocked IP addresses when they have been blocked by the firewall.

Port Knocking

PORTKNOCKING = ""

CSF supports port knocking, which is a technique that provides an additional layer of security. See http://www.portknocking.org/ for details.

Allow and Deny Lists

As we walked through the CSF configuration file, you saw that I referenced the csf.deny file, so it should come as no surprise that CSF also includes csf.allow to customize "allow" rules as well. If you are familiar with APF, these files have a very similar syntax ... Each entry is made up of the same four components: protocol|flow|port|IP. The only real difference being that APF uses the colon as a delimiter while CSF uses the pipe:

#APF Version
tcp:in:d=48000_48020:s=10.0.0.0/8
 
#CSF Version
tcp|in|d=48000_48020|s=10.0.0.0/8

Fortunately, replacing your colon with a pipe is a minimally invasive procedure that can be automated with a tool like vi.

CSF Command Line Tool

The command line tool for CSF is much more robust than the one for APF:

$ csf --help
csf: v5.79 (cPanel)
 
ConfigServer Security & Firewall
(c)2006-2013, Way to the Web Limited (http://www.configserver.com)
 
Usage: /usr/sbin/csf [option] [value]
 
Option              Meaning
-h, --help          Show this message
-l, --status        List/Show iptables configuration
-l6, --status6      List/Show ip6tables configuration
-s, --start         Start firewall rules
-f, --stop          Flush/Stop firewall rules (Note: lfd may restart csf)
-r, --restart       Restart firewall rules
-q, --startq        Quick restart (csf restarted by lfd)
-sf, --startf       Force CLI restart regardless of LF_QUICKSTART setting
-a, --add ip        Allow an IP and add to /etc/csf.allow
-ar, --addrm ip     Remove an IP from /etc/csf.allow and delete rule
-d, --deny ip       Deny an IP and add to /etc/csf.deny
-dr, --denyrm ip    Unblock an IP and remove from /etc/csf.deny
-df, --denyf        Remove and unblock all entries in /etc/csf.deny
-g, --grep ip       Search the iptables rules for an IP match (incl. CIDR)
-t, --temp          Displays the current list of temp IP entries and their TTL
-tr, --temprm ip    Remove an IPs from the temp IP ban and allow list
-td, --tempdeny ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP ban list. ttl is how long to
                    blocks for (default:seconds, can use one suffix of h/m/d).
                    Optional port. Optional direction of block can be one of:
                    in, out or inout (default:in)
-ta, --tempallow ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP allow list (default:inout)
-tf, --tempf        Flush all IPs from the temp IP entries
-cp, --cping        PING all members in an lfd Cluster
-cd, --cdeny ip     Deny an IP in a Cluster and add to /etc/csf.deny
-ca, --callow ip    Allow an IP in a Cluster and add to /etc/csf.allow
-cr, --crm ip       Unblock an IP in a Cluster and remove from /etc/csf.deny
-cc, --cconfig [name] [value]
                    Change configuration option [name] to [value] in a Cluster
-cf, --cfile [file] Send [file] in a Cluster to /etc/csf/
-crs, --crestart    Cluster restart csf and lfd
-w, --watch ip      Log SYN packets for an IP across iptables chains
-m, --mail [addr]   Display Server Check in HTML or email to [addr] if present
-lr, --logrun       Initiate Log Scanner report via lfd
-c, --check         Check for updates to csf but do not upgrade
-u, --update        Check for updates to csf and upgrade if available
-uf                 Force an update of csf
-x, --disable       Disable csf and lfd
-e, --enable        Enable csf and lfd if previously disabled
-v, --version       Show csf version

The command line tool will also tell you if the testing mode is enabled (which is a very useful feature). If TESTING were enabled, we'd see this line at the bottom of the output:

*WARNING* TESTING mode is enabled - do not forget to disable it in the configuration

Did you make it all the way through?! Great! I know it's a lot to take in, but it's not terribly complicated when we break it down and understand how each piece works. Next time, I'll be back with some tips on integrating CSF into cPanel.

-Mark

November 19, 2012

How It's Made (and Won): The Server Challenge II

Every year, we attend more than fifty trade shows and conferences around the world. We want to spread the word about SoftLayer and connect with each conference's technical audience (also known as future SoftLayer customers). That goal is pretty straightforward on paper, but when it comes to executing on it, we're faced with the same challenge as all of our fellow exhibitors: How do we get our target audience to the our booth?

Walk down any aisle of an expo hall, and you'll see collateral and swag beckoning to attendees like a candy bar at the grocery store register. Some exhibitors rely on Twitter to monitor an event's hashtag and swoop in at every opportunity to reach the show's influential attendees. Other exhibitors might send out emails to their clients and prospects in the area to invite them to the show. We see value in each of those approaches, but what we found to be most effective was to bring a SoftLayer data center to our booth ... or at least a piece of one.

The Server Challenge has come a long way over the years. Its meager beginnings involved installing RAM and hard drive cables in a tower server. Shortly thereafter, a rack-mount server replaced the tower server, but you were still tasked with "inside the server" challenges. As we started looking for ways to tell the bigger SoftLayer story with the Server Challenge, we moved to miniature server rack, and the competition really started to pick up steam. This year, we made it our goal to take the Server Challenge to the next level, and when Supermicro stepped in to sponsor the next iteration of the the competition, we started thinking BIG.

Why use a miniature version of a SoftLayer rack when we could use a full-size version? Why have a standalone screen when rack-mount monitors can make the display part of the unit? Why rely on speakers behind the booth to pump "Eye of the Tiger" while attendees are competing when we could easily build those into the next version of the challenge? What was initially intended to be a "tweak" of the first Server Challenge became a complete overhaul ... Hence the new "Server Challenge II" moniker.

Harkening back to the 8-bit glory days of Pac Man and Space Invaders, the Server Challenge II uses a full-size 42U server rack with vintage arcade-style branding, a built-in timer and speakers that blast esoteric video game music. The bread and butter of the challenge is the actual server hardware, though ... Supermicro provided two new 2U servers to replace the previous version's five 1U servers, and we installed the same Cisco (public and private networks) and SMC (out-of-band management network) switches you see in SoftLayer's pods.

Server Challenge II

We had two instances of the original Server Challenge (one in the US, one in Amsterdam), so in order for the Server Challenge II to be bigger and better, we had to increase that total to five — one instance in Europe, one in Asia and three in the United States. Things might get a little crazier logistically, but as a potential conference attendee, it means you're even more likely to encounter the Server Challenge II if you attend any events with us.

The Server Challenge II's Internal Debut

The first instance of the Server Challenge II made its debut at GDC Online in Austin, and we immediately knew we had a hit. By the time the rack got back to our office, we had to get it ready for its next destination (Cloud Expo West), but before we sent it on its way, we gave it an official internal debut ... and raised some money for the American Heart Association in the process.

Server Challenge II at SoftLayer

SLayers at the SoftLayer HQ in Dallas could pay $3 for one attempt or $5 for two attempts to reach the top of the Server Challenge II leader board. Needless to say, it was competitive. If you click on the image above, you'll notice that our fearless leader, Lance Crosby, stopped by and gave tips to (and/or heckled) a few participants. Unsurprisingly, one of our very talented Server Build Technicians — Ellijah Fleites — took home a MacBook Air and bragging rights as SoftLayer champion with a record time of 1:03.79 ... But records are made to be broken.

In Two Places at Once

Immediately after the AHA fundraiser, we crated up the rack and sent it along to Cloud Expo West in Santa Clara. A few days later, we put the finishing touches on the second Server Challenge II rack, and because we got it done quickly, we were able to get it shipped to the other side of the country for ad:tech NYC. We would finally have the competition running in two places at the exact same time!

We weren't disappointed.

On both coasts, the retro style of the Server Challenge II lured some fantastic competitors (excellent!), and started a lot of great conversations (even better!). Here are the final leader boards from the shows:

Server Challenge II
Server Challenge II

You probably notice that the times in the ad:tech leader board are a little higher than the times in the Cloud Expo leader board, and our team figured out why that was in the middle of the second day of the conference ... The way we bound the network cables differed slightly between the two instances, and we were using different switches to time the competition (one that required only one hand to activate/deactivate, the other requiring both hands). In order to have an "apples-to-apples" comparison between all of our shows, we're going to make sure everything is consistent with all of the instances, and we plan on keeping a running list of fastest overall challenge times ... and maybe even a "World Championship" one day.

Given the early success of the Server Challenge II, you can bet that it's not going anywhere any time soon. If we have multiple shows running the challenge at one time, we might even fire up a video chat where you can compete against an attendee at a completely different conference ... so be prepared.

In the next year, we'll have all five of the Server Challenge II instances in rotation across three continents, and with the popularity of the competition growing by leaps and bounds after every show, we hope by next holiday season, a home version of the Server Challenge II is at the top of every wish list on the planet. :-)

For now, though, I'll just leave you with a glimpse at the action from Cloud Expo West (click for more pictures from the show):

Cloud Expo West

-Raleigh

October 23, 2012

Tips from the Abuse Department: Know Spam. Stop Spam.

As an abuse administrator, I'm surrounded by spam on a daily basis. When someone sends an abuse-related complaint to our abuse@softlayer.com contact address, it gets added to our ticket queue, and our Abuse SLayers take time to investigate and follow up with the customers whose servers violate our acceptable use policy. The majority of those abuse-related submissions are reporting spam coming from our network, and in my interaction with customers, I've noticed that spam (and the source of spam) is widely misunderstood.

Most spam tickets we create on customer accounts pinpoint spam sent from a compromised or exploited server. Our direct customer didn't send the phishing email, malware distribution, pharmacy advertisement or pornographic spam, but that activity came from their account. While they're accountable for the abusive behavior coming from their server, in many cases, they don't know that there's a problem until we post an abuse ticket on their account. These servers are targeted and compromised by common techniques and exploits that could have been easily avoided, but they aren't very well known outside the world of abuse.

To protect yourself from a spammer, you need to think like a spammer. You need to understand how someone might try to exploit your environment so that you can prevent them from doing so. As you're looking at ways to secure your server proactively, make sure you target these five exploits in particular:

1. User Auth Login

This is by far the most common exploit to used to send spam. This method involves a person or script using the credentials of a user to send spam through a domain's mail server. The majority of these incidences are caused by malware on a client PC that obtains the login and password for a domain user and uses that information to log on and send mail from the client PC through the server. Often, these spam messages are sent through a botnet command structure.

When an account is compromised, simply changing the password for the compromised user on the server usually won't stop the abuse. We see quite a few accounts that continue to send spam after an initial abuse ticket results in a password change. Most servers that are sending spam with this method are found to only be sending a small amount of spam at any given time to avoid detection. The low volume of spam that is being sent per server is made up for by the fact that there are thousands of servers being used for the same spamming campaigns.

In order to stop the User Auth Login exploit, a customer needs to clean all of the malicious software (malware) from their environments. To prevent future User Auth Login compromises, users should be made aware of the potential dangers of untrusted software, and if they believe their machines are infected, they need to know what to do.

2. Tell-a-friend Exploitation

The User Auth Login technique is the most common method employed by spammers, but the "tell-a-friend" script exploitation isn't far behind when it comes to volume of affected servers. This spamming method find websites that use scripts to invite users to refer friends to a page or product. Spammers will use the 'Your Message' field in one of these scripts to input their own content and links, and they'll push the actual page referral link to the bottom of the message. When these site scripts aren't secure, the spammer will use them to send hundreds or thousands of messages.

To avoid having your website fall victim to this type of spam, be very wary of any widget or script you add. If you need to add Facebook, Twitter and email "share" functionality to your site, make sure you incorporate a tell-a-friend script that does not allow for customizable messages or does not accept input of more than one email address. Also, users won't need the "cc" or "bcc" fields, so you can be sure those are axed as well. If you can't find a good "share" script that you're comfortable with from a security perspective, it might be a good idea to remove that functionality to avoid exploitation.

3. Uploaded Mailers

Spam sent via an uploaded third party mailer can sometimes prove difficult for admins to locate. An uploaded third party mailer could be capable of creating it's own outbound SMTP connection, and that would allow a program to bypass the existing MTA on the server and render any legitimate mail logs useless for investigation. Another challenge is that a php mailer can be uploaded to a location within a user's web content, and that mailer is run by the user 'nobody' (the default Apache user).

We strongly suggest configuring your server to have the mail headers show the script's user (that's not the Apache default user) and the location the script is running from on the server. Many times, these kinds of mailers are maliciously uploaded after a user's FTP password is been compromised, so be sure your FTP login information is secure.

4. Software Exploits

The "software exploits" category casts a huge shadow. Every piece of software on a server — from mail servers, content management systems and control panels to the operating system itself — can be targeted by hackers. They probe servers to find security vulnerabilities and weak coding, and when they find a vulnerability, they take control.

The hacker who found the software vulnerability might not actually take advantage of the exploit immediately. That user may sell access to other entities for their use, and that use often ends up being spam. In addition to having strong firewall rules and access restrictions, you should update and maintain the current stable versions of all software on your servers.

5. WordPress Exploits

WordPress exploits would technically fall under the "Software Exploits" category, but I'm breaking it out into its own category simply due to the volume of spam issues that are the result of exploiting this particular piece of software. The first step to protecting against spam being sent through this source is to make sure you have the latest version of WordPress installed. With that done, be sure to research the latest security plugins for that version and install any that are applicable to your environment.

These five techniques are not the only ones used by spammers to take advantage of your environment, but they are some of the most common. To protect yourself from becoming a source of spam, make your servers a more difficult target to exploit. To stop spam, you need to know spam. Now that you know spam, it's time to stop it. Ask questions, test your environment regularly and watch your logs for any unexplained usage.

-Andrew

July 30, 2012

Don't Stop Believing (in Hosting)

If 80's movies have taught me anything, it's that any good story needs to have a video montage with Journey playing in the background. With that in mind, I'll start this blog post with a glimpse of HostingCon 2012:

HostingCon brings the hosting industry together every year, and the conference winds up being surprisingly similar to classic 80's "coming of age" movies:

  • "Geeks" are among the main characters.
  • There's always a "funny guy."
  • At some point, the geeks attend a party.
  • The characters learn more about themselves and others over the course of the movie.
  • As the credits roll, everyone is inspired ... Ready to take on the world.

With that in mind, HostingCon 2012 in Boston was a veritable John Hughes flick. There was no shortage of geeks, we hung out with one of the funniest people in the country, we threw a massive party, and we learned a ton. Without a doubt, attendees returned home with their intensity and enthusiasm cranked up to eleven (another 80's reference).

The expo hall was abuzz with activity — albeit after a lull in the morning following the aptly named "Host Me All Night Long" party — and we enjoyed the opportunity to catch up with current partners and customers while meeting and speaking with soon-to-be partners and customers. While running a highly competitive Server Challenge, we were still able to dive deeper into partnerships, the build v. buy decision, branding, and launching a product when attendees visited our booth after hearing from our team in conference sessions and panels, and those conversations are what keep us coming back to HostingCon every year.

As a "veteran" of the hosting industry (assuming seven years of experience qualifies me), I've learned a great deal about the dynamics of the hosting industry from events like HostingCon over the years. On one hand, many of the attendees are "competitors," and on the other hand, we're all trying to make the industry better (since "a rising tide lifts all boats"). As a great example, look at the Internet Infrastructure Coalition (i2C), a trade association of companies with the shared goal and purpose of representing the industry in Washington, D.C., and beyond.

As it turns out, that unity flew out the door when attendees stood face-to-rack with the Server Challenge, though. Unlike our experiences at more general "technology" conferences, the components in our competition needed no introduction, and participants were particularly driven to best their peers ... not only for the iPad, but for the pride of owning the Server Challenge title at HostingCon:

  1. Darin Goldman - 0:59.28
  2. Devon Hillard - 1:01.58
  3. Ijan Kruizinga - 1:01.83
  4. Jon Basha - 1:03.02
  5. Sean Whitley - 1:03.06

As you saw in the video, Darin Goldman had the luxury of not needing his second attempt on the final day of the conference to secure a victory, but we were glad he let us record his "Breakfast Club" fist-pump to share with the world.

Fist Pump

Don't stop believing (in hosting).

-@khazard

P.S. I recorded the first few minutes of Ralphie May's set, but the adult language-ness of the content makes it a little more difficult to share with the world.

Categories: 
July 27, 2012

SoftLayer 'Cribs' ≡ DAL05 Data Center Tour

The highlight of any customer visit to a SoftLayer office is always the data center tour. The infrastructure in our data centers is the hardware platform on which many of our customers build and run their entire businesses, so it's not surprising that they'd want a first-hand look at what's happening inside the DC. Without exception, visitors to a SoftLayer data center pod are impressed when they walk out of a SoftLayer data center pod ... even if they've been in dozens of similar facilities in the past.

What about the customers who aren't able to visit us, though? We can post pictures, share stats, describe our architecture and show you diagrams of our facilities, but those mediums can't replace the experience of an actual data center tour. In the interest of bridging the "data center tour" gap for customers who might not be able to visit SoftLayer in person (or who want to show off their infrastructure), we decided to record a video data center tour.

If you've seen "professional" video data center tours in the past, you're probably positioning a pillow on top of your keyboard right now to protect your face if you fall asleep from boredom when you hear another baritone narrator voiceover and see CAD mock-ups of another "enterprise class" facility. Don't worry ... That's not how we roll:

Josh Daley — whose role as site manager of DAL05 made him the ideal tour guide — did a fantastic job, and I'm looking forward to feedback from our customers about whether this data center tour style is helpful and/or entertaining.

If you want to see more videos like this one, "Like" it, leave comments with ideas and questions, and share it wherever you share things (Facebook, Twitter, your refrigerator, etc.).

-@khazard

June 6, 2012

Today's Technology "Game Changers": IPv6 and Cloud

"Game Changers" in technology force a decision: Adapt or die. When repeating rifles gained popularity in the late 1800s, a business of manufacturing muzzle-loading or breech-loading rifles would have needed to find a way to produce a repeating rifle or it would have lost most (if not all) of it's business to Winchester. If a fresh-faced independent musician is hitting it big on the coffee shop scene in 2012, she probably won't be selling out arenas any time soon if she refuses to make her music available digitally. Just ask any of the old-timers in the print media industry ... "Game Changers" in technology can be disastrous for an established business in an established industry.

That's pretty intimidating ... Even for tech businesses.

Shifts in technology don't have to be as drastic and obvious as a "printed newspaper v. social news site" comparison for them to be disruptive. Even subtle advances can wind up making or breaking a business. In fact, many of today's biggest and most successful tech companies are scrambling to adapt to two simple "game changers" that seem terribly significant:

  • IPv6
  • "The Cloud"

IPv6

A quick search of the SoftLayer Blog reminds me that Lance first brought up the importance of IPv6 adoption in October 2007:

ARIN has publically announced the need to shift to IPv6 and numerous articles have outlined the D-Day for IPv4 space. Most experts agree, its coming fast and that it will occur sometime in 2010 at the current pace (that's about two years for those counting). IPv6 brings enough IP space for an infinite number of users along with improved security features and several other operational efficiencies that will make it very popular. The problem lies between getting from IPv4 to IPv6.

When IPv4 exhaustion was just a blip on the horizon, many businesses probably thought, "Oh, I'll get around to it when I need to. It's not a problem yet." When IANA exhausted the IPv4 pool, they probably started picking up the phone and calling providers to ask what plans they had in place. When some of the Internet's biggest websites completed a trial transition to IPv6 on World IPv6 Day last year, those businesses started feeling the urgency. With today's World IPv6 Launch, they know something has to be done.

World IPv6 Launch Day

Regardless of how conservative providers get with IPv4 space, the 4,294,967,296 IPv4 addresses in existence will not last much longer. Soon, users will be accessing an IPv6 Internet, and IPv4-only websites will lose their opportunity to reach those users. That's a "game changer."

"The Cloud"

The other "game changer" many tech businesses are struggling with these days is the move toward "the cloud." There are a two interesting perspectives in this transition: 1) The challenge many businesses face when choosing whether to adopt cloud computing, and 2) The challenges for businesses that find themselves severing as an integral (sometimes unintentional) part of "the cloud." You've probably seen hundreds of blog posts and articles about the first, so I'll share a little insight on the second.

When you hear all of the hype about cloud computing and cloud storage offering a hardware-agnostic Utopia of scalable, reliable power, it's easy to forget that the building blocks of a cloud infrastructure will usually come from vendors that provided a traditional hosting resources. When a computing instance is abstracted from a hardware device, it's opens up huge variations in usage. It's possible to have dozens of public cloud instances using a single server's multi-proc, multi-core resources at a given time. If a vendor prices a piece of software on a "per server" basis, how do they define a "server" when their users are in the cloud? It can be argued that a cloud computing instance with a single core of power is a "server," and on the flip-side, it's easy to define a "server" as the hardware object on which many cloud instances may run. I don't know that there's an easy way to answer that question, but what I do know is that applying "what used to work" to "what's happening now" isn't the right answer.

The hardware and software providers in the cloud space who are able to come up with new approaches unencumbered by the urge to continue "the way we've always done it" are going to be the ones that thrive when technology "game changers" emerge, and the providers who dig their heels in the dirt or try to put a square peg into a round hole will get the short end of the "adapt or die" stick.

We've tried to innovate and take a fresh look at every opportunity that has come our way, and we do our best to build relationships with agile companies that we see following suit.

I guess a better way to position the decision at the beginning of this post would be to add a little tweak: "Innovate, adapt or die." How you approach technology "game changers" will define your business's success.

-@gkdog

May 30, 2012

What Does Automation Look Like?

Innovation. Automation. Innovation. Automation. Innovation. Automation. That's been our heartbeat since SoftLayer was born on May 5, 2005. The "Innovation" piece is usually the most visible component of that heartbeat while "Automation" usually hangs out behind the scenes (enabling the "Innovation"). When we launch a new product line like Object Storage, add new functionality to the SoftLayer API, announce a partnership with a service provider like RightScale, or simply receive and rack the latest and greatest server hardware from our vendors, our automated platform allows us to do it quickly and seamlessly. Because our platform is built to do exactly what it's supposed to without any manual intervention, it's easily overlooked.

But what if we wanted to show what automation actually looks like?

It seems like a silly question to ask. If our automated platform is powered by software built by the SoftLayer development team, there's no easy way to show what that automation looks like ... At least not directly. While the bits and bytes aren't easily visible, the operational results of automation are exceptionally photogenic. Let's take a look at a few examples of what automation enables to get an indirect view of what it actually looks like.

Example: A New Server Order

A customer orders a dedicated server. That customer wants a specific hardware configuration with a specific suite of software in a specific data center, and it needs to be delivered within four hours. What does that usually look like from an operations perspective?

SoftLayer Server Rack

If you want to watch those blinking lights for two or three hours, you'll have effectively watched a new server get provisioned at SoftLayer. When an order comes in, the automated provisioning system will find a server matching the order's hardware requirements in the requested data center facility, and the software will be installed before it is handed over to the the customer.

Example: Server Reboot or Operating System Reload

A customer needs to reboot a server or install a new operating system. Whether they want a soft reboot, a hard reboot with a full power cycle or a blank operating system install, the scene in the data center will look eerily familiar:

SoftLayer Server Rack

Gone are the days of server build technicians wheeling a terminal over to every server that needs work done. From thousands of miles away, a customer can remotely "unplug" his or her server via the rack's power strip, initiate a soft reboot or reinstall an operating system. But what if they want even more accessibility?

Example: What's on the Screen?

When remotely rebooting or power cycling a server isn't enough, a customer might want someone in the data center to wheel over to their server in the rack to look at any of the messages that can only be read with a monitor attached. This would generally happen behind the server, but for the sake of this example, we'll just watch the data center technician pass in front of the servers to get to the back:

SoftLayer Server Rack

Yeah, you probably could have seen that one coming.

Because KVM over IP is included on every server, physical carts carrying "keyboard, video and mouse" are few and far between. By automating customers' access to their server and providing as much virtual access as we possibly can, we're able to "get out of the way" of our technical users and only step in to help when that help is needed.

I could go on and on with examples of cloud computing upgrades and downgrades, provisioning a firewall or adding a load balancers, but I'll practice a little restraint. If you want the full effect, you can scroll up and watch the blinking lights a little while longer.

Automation looks like what you don't see. No humanoid robots or needlessly complex machines (that I know of) ... Just a data center humming along with some beautiful flashing server lights.

-Duke

P.S. If you want to be able to remotely bask in the glow of some blinking server lights, bookmark the larger-sized SoftLayer Rack animated gif ... You could even title the bookmark, "Check on the Servers."

April 23, 2012

Choosing a Cloud: Which Cloud Chooses You?

It's not easy to choose a cloud hosting provider.

In the first post of this series, we talked about the three key deciding factors every cloud customer has to consider, and we set up a Venn diagram to distinguish the surprisingly broad range of unique priorities customers can have:

Cloud Customer Zones

Because every customer will prioritize a cloud's cost, technology and hosting provider a little differently (for completely valid reasons), we mapped out seven distinct "zones" to differentiate some of the basic market segments, or "personas," of cloud hosting buyers. That post was intended to set the stage for a larger discussion on how customers choose their cloud providers and how cloud providers choose their customers, and we're just scratching the surface. We're tackling a pretty big topic here, so as Bill Cosby famously says, "I told you that story to tell you this one."

As a hosting provider, SoftLayer can't expect to be all things for all people. It's impossible to offer a quad-core hex-proc dedicated server for a price that will appeal to a customer in the market for a $49/mo dedicated server.

To better illustrate SoftLayer's vision in the cloud market, we need to take that generic cost v. technology v. hosting provider diagram and give it the "Three Bars" treatment:

SoftLayer Venn Diagram

We're much more interested in living and breathing the Zone 5 "Technology" space rather than the traditional Zone 2 "Hosting Provider" space. That's why in the past two months, you've seen announcements about our launch of the latest Intel Processors, HPC computing with NVidia GPUs, searchable OpenStack Object Storage, and an innovative "Flex Image" approach to bluring the lines between physical and virtual servers. We choose to pursue the cloud customers who make their buying decisions in Zone 3.

That's a challenging pursuit ... It's expensive to push the envelope in technology, customers primarily interested in technology/performance have demanding needs and expectations, and it's easier to make mistakes when you're breaking new ground. The majority of the hosting industry seems to have an eye on the buyer in Zone 1 because they believe the average hosting customer is only interested in the bottom line ... That hosting is more or less a commodity, so the focus should be on some unverifiable qualitative measure of support or the next big special that'll bring in new orders.

As you may have seen recently, GigaOm posted a lovely article that references several high-profile companies in our 25,000+ customer family. We like to say that SoftLayer builds the platform on which our customers build the future, and that short post speaks volumes about the validity of that statement. Our goal is to provide the most powerful, scalable and seamlessly integrated IT infrastructure for the most innovative companies in the world. Innovate or Die isn't just our company motto ... It's our hope for our customers, as well.

We might miss out on your business if you want a $49/mo dedicated server, but if you're looking to change the world, we've got you covered. :-)

-@khazard

April 20, 2012

Choosing a Cloud: Cost v. Technology v. Hosting Provider

If you had to order a new cloud server right now, how would choose it?

I've worked in the hosting industry for the better part of a decade, and I can safely say that I've either observed or been a part of the buying decision for a few thousand hosting customers — from small business owners getting a website online for the first time to established platforms that are now getting tens of millions of visits every day. While each of those purchasers had different requirements and priorities, I've noticed a few key deciding factors that are consistent in a all of those decisions:

The Hosting Decision

How much will the dedicated server or cloud computing instance cost? What configuration/technology do I need (or want)? Which hosting provider should I trust with my business?

Every website administrator of every site on the Internet has had to answer those three questions, and while they seem pretty straightforward, they end up overlapping, and the buying decision starts to get a little more complicated:

The Hosting Decision

The natural assumption is that everyone will choose a dedicated server or cloud computing instance that falls in the "sweet spot" where the three circles overlap, right? While that makes sense on paper, hosting decisions are not made in a vacuum, so you'll actually see completely valid hosting decisions targeting every spot on that graph.

Why would anyone choose an option that wouldn't fit in the sweet spot?

That's a great question, and it's a tough one to answer in broad strokes. Let's break the chart down into a few distinct zones to look at why a user would choose a server in each area:

The Hosting Decision

Zone 1

Buyers choosing a server in Zone 1 are easiest to understand: Their budget takes priority over everything else. They might want to host with a specific provider or have a certain kind of hardware, but their budget doesn't allow for either. Maybe they don't need their site to use the latest and greatest hardware or have it hosted anywhere in particular. Either way, they choose a cloud solely based on whether it fits their budget. After the initial buying decision, if another server needs to be ordered, they might become a Zone 4 buyer.

Zone 2

Just like Zone 1 buyers, Zone 2 buyers are a pretty simple bunch as well. If you're an IT administrator at a huge enterprise that does all of your hosting in-house, your buying decision is more or less made for you. It doesn't matter how much the solution costs, you have to choose an option in your data center, and while you might like a certain technology, you're going to get what's available. Enterprise users aren't the only people deciding to order a server in Zone 2, though ... It's where you see a lot of loyal customers who have the ability to move to another provider but prefer not to — whether it's because they want their next server to be in the same place as their current servers, they value the capabilities of a specific hosting provider (or they just like the witty, interesting blogs that hosting provider writes).

Zone 3

As with Zone 1 and Zone 2, when a zone doesn't have any overlapping areas, the explanation is pretty easy. In Zone 3, the buying decision is being made with a priority on technology. Buyers in this area don't care what it costs or where it's hosted ... They need the fastest, most powerful, most scalable infrastructure on the market. Similar to Zone 1 buyers, once Zone 3 buyers make their initial buying decision, they might shift to Zone 5 for their next server or cloud instance, but we'll get to that in a minute.

Zone 4

Now we're starting to overlap. In Zone 4, a customer will be loyal to a hosting provider as long as that loyalty doesn't take them out of their budget. This is a relatively common customer ... They'll try to compare options apples-to-apples, and they'll make their decision based on which hosting provider they like/trust most. As we mentioned above, if a Zone 1 buyer is adding another server to their initial server order, they'll likely look to add to their environment in one place to make it easier to manage and to get the best performance between the two servers.

Zone 5

Just like the transitional Zone 1 buyers, when Zone 3 buyers look to build on their environment, they'll probably become Zone 5 buyers. When your initial buying decision is based entirely on technology, it's unusual to reinvent the wheel when it comes to your next buying decision. While there are customers that will reevaluate their environment and choose a Zone 3 option irrespective of where their current infrastructure is hosted, it's less common. Zone 5 users love having he latest and greatest technology, and they value being able to manage it through one provider.

Zone 6

A Zone 6 buyer is usually a Zone 1 buyer that has specific technology needs. With all the options on the table, a Zone 6 buyer will choose the cloud environment that provides the latest technology or best performance for their budget, regardless of the hosting provider. As with Zone 1 and Zone 3 buyers, a Zone 6 buyer will probably become a Zone 7 buyer if they need to order another server.

Zone 7

Zone 7 buyers are in the sweet spot. They know the technology they want, they know the price they want to pay, and they know the host they want to use. They're able to value all three of their priorities equally, and they can choose an environment that meets all of their needs. After Zone 6 buyers order their first server(s), they're going to probably become Zone 7 buyers when it comes time for them to place their next order.

As you probably noticed, a lot of transitioning happens between an initial buying decision and a follow-up buying decision, so let's look at that quickly:

The Hosting Decision

Regardless of how you make your initial buying decision, when it's time for your next server or cloud computing instance, you have a new factor to take into account: You already have a cloud infrastructure at a hosting provider, so when it comes time to grow, you'll probably want to grow in the same place. Why? Moving between providers can be a pain, managing environments between several providers is more difficult, and if your servers have to work together, they're generally doing so across the public Internet, so you're not getting the best performance.

Where does SoftLayer fit in all of this? Well beyond being a hosting provider that buyers are choosing, we have to understand buyers are making their buying decisions, and we have to position our business to appeal to the right people with the right priorities. It's impossible to be all things for all people, so we have to choose where to invest our attention ... I'll leave that post for another day, though.

If you had to choose a zone that best describes how you made (or are currently making) your buying decision, which one would it be?

-@khazard

April 9, 2012

Scaling SoftLayer

SoftLayer is in the business of helping businesses scale. You need 1,000 cloud computing instances? We'll make sure our system can get them online in 10 minutes. You need to spin up some beefy dedicated servers loaded with dual 8-core Intel Xeon E5-2670 processors and high-capacity SSDs for a new application's I/O-intensive database? We'll get it online anywhere in the world in under four hours. Everywhere you look, you'll see examples of how we help our customers scale, but what you don't hear much about is how our operations team scales our infrastructure to ensure we can accommodate all of our customers' growth.

When we launch a new data center, there's usually a lot of fanfare. When AMS01 and SNG01 came online, we talked about the thousands of servers that are online and ready. We meet huge demand for servers on a daily basis, and that presents us with a challenge: What happens when the inventory of available servers starts dwindling?

Truck Day.

Truck Day not limited to a single day of the year (or even a single day in a given month) ... It's what we call any date our operations team sets for delivery and installation of new hardware. We communicate to all of our teams about the next Truck Day in each location so SLayers from every department can join the operations team in unboxing and preparing servers/racks for installation. The operations team gets more hands to speed up the unloading process, and every employee has an opportunity to get first-hand experience in how our data centers operate.

If you want a refresher course about what happens on a Truck Day, you can reference Sam Fleitman's "Truck Day Operations" blog, and if you want a peek into what it looks like, you can watch Truck Day at SR02.DAL05. I don't mean to make this post all about Truck Day, but Truck Day is instrumental in demonstrating the way SoftLayer scales our own infrastructure.

Let's say we install 1,000 servers to officially launch a new pod. Because each pod has slots for 5,000 servers, we have space/capacity for 3,000-4,000 more servers in the server room, so as soon as more server hardware becomes available, we'll order it and start preparing for our next Truck Day to supplement the pod's inventory. You'd be surprised how quickly 1,000 servers can be ordered, and because it's not very easy to overnight a pallet of servers, we have to take into account lead time and shipping speeds ... To accommodate our customers' growth, we have to stay one step ahead in our own growth.

This morning in a meeting, I saw a pretty phenomenal bullet that got me thinking about this topic:

Truck Day — 4/3 (All Sites): 2,673 Servers

In nine different data center facilities around the world, more than 2,500 servers were delivered, unboxed, racked and brought online. Last week. In one day.

Now I know the operations team wasn't looking for any kind of recognition ... They were just reporting that everything went as planned. Given the fact that an accomplishment like that is "just another day at SoftLayer" for those guys, they definitely deserve recognition for the amazing work they do. We host some of the most popular platforms, games and applications on the Internet, and the DC-Ops team plays a huge role in scaling SoftLayer so our customers can scale themselves.

-@gkdog

Subscribe to servers