To get the ball rolling, we need to download CSF and install it on our server. In this post, we’re working with a CentOS 6.0 32-bit server, so our (root) terminal commands would look like this to download and install CSF:

$ wget http://www.configserver.com/free/csf.tgz #Download CSF using wget.
$ tar zxvf csf.tgz #Unpack it.
$ yum install perl-libwww-perl #Make sure perl modules are installed ...
$ yum install perl-Time-HiRes  #Otherwise it will generate an error.
$ cd csf
$ ./install.sh #Install CSF.
 
#MAKE SURE YOU HAVE YOUR BAND-AID READY
 
$ /etc/init.d/csf start #Start CSF. (Note: You can also use '$ service csf start')

Once you start CSF, you can see a list of the default rules that load at startup. CSF defaults to a DROP policy:

$ iptables -nL | grep policy
Chain INPUT (policy DROP)
Chain FORWARD (policy DROP)
Chain OUTPUT (policy DROP)

Don’t ever run “iptables -F” unless you want to lock yourself out. In fact, you might want to add “This server is running CSF – do not run ‘iptables -F’” to your /etc/motd, just as a reminder/warning to others.

CSF loads on startup by default. This means that if you get locked out, a simple reboot probably won’t fix the problem. Runlevels 2, 3, 4, and 5 are all on:

$ chkconfig --list | grep csf
csf             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Some features of CSF will not work unless you have certain iptables modules installed. I believe they are installed by default in CentOS, but if you custom-built your iptables, they might not all be installed. Run this script to see if all modules are installed:

$ /etc/csf/csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
 
RESULT: csf should function on this server

As I mentioned, this is the default iptables installation on a minimal CentOS 6.0 image, so chances are good that these modules are already installed on your system. It never hurts to check, though.

The CSF Configuration File

The primary CSF configuration is stored in the well-documented /etc/csf/csf.conf file. CSF is extremely configurable, so there are a lot of options to read over. Let’s take a look over some of the more important features:

Testing

TESTING = "1"
TESTING_INTERVAL = "5"

This TESTING cron job runs every “5″ minutes so you don’t lock yourself out when you’re testing your rules. When you are satisfied with your rules (and confident that you won’t lock yourself out), you can set TESTING to “0″.

Globally Allowed Ports

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995"
 
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,53,80,110,113,443"
 
# Allow incoming UDP ports
UDP_IN = "20,21,53"
 
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "20,21,53,113,123"

Incoming Ping Requests

# Allow incoming PING
ICMP_IN = "1"

Allowing ping is usually a good option for diagnostic purposes, so I don’t recommend turning it off. Disallowing ping is an example of “security through obscurity,” and it will not typically dissuade your attackers.

Ethernet Device

ETH_DEVICE = ""
ETH6_DEVICE = ""

Here, you can configure iptables to ONLY use one Ethernet adapter. You might want to only guard your public network adapter in some situations.

IP Limit in Permanent “Deny” File

DENY_IP_LIMIT = "200"

A higher number here will obviously screen out more IP addresses in csf.deny, but higher numbers also may cause slowdowns.

IP Limit in Temporary “Deny” File

DENY_TEMP_IP_LIMIT = "100"

Similar to DENY_IP_LIMIT, the DENY_TEMP_IP_LIMIT represents the maximum number of IPs that can be stored in the temporary ban list.

SMTP Blocking

SMTP_BLOCK = "0"

When set to “1″, SMTP_BLOCK does not completely block outbound SMTP, but it does block it for most users. This will prevent malicious scripts and compromised users from making outbound connections from unauthorized mail clients on the server. SMTP_BLOCK doesn’t stop those scripts from running, but it does stop them from functioning. Mail sent through the proper channels will still be delivered normally.

Allowing SMTP on localhost

SMTP_ALLOWLOCAL = "1"

Custom Mail Port Designation

SMTP_PORTS = "25,465,587"

Allowing SMTP Access to Users/Groups

SMTP_ALLOWUSER = ""
SMTP_ALLOWGROUP = "mail,mailman"

SYN Flood Protection

SYNFLOOD = "0"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

Per the documentation, you should only enable SYN flood protection (SYNFLOOD= “1″) if you are currently under a SYN flood attack.

Concurrent Connections Limit

CONNLIMIT = "22;5,80;20"
PORTFLOOD = "22;tcp;5;300,80;tcp;20;5

These options allow you to add customized DoS protection. CONNLIMIT handles the number of concurrent connections, and in this example, we’re limiting port 22 to 5 connections and port 80 to 20 connections.

PORTFLOOD on deals with connections per second. In this example, we’re limiting the TCP connection on port 22 to 5 connections/second with a quiet period of 300 seconds before the connection is unblocked. Additonally, we’re limiting the TCP connection on port 80 to 20 connections/second with a quiet period of 5 seconds before the connection is unblocked.

Check the readme.txt file for more information about the syntax.

Logging to Syslog

SYSLOG = "0"

When enabled, this option logs lfd (Login Failure Daemon) messages to syslog as well as to /var/log/lfd.log.

Dropping v. Rejecting Packets

DROP = "DROP"

This configuration allows you to either DROP or REJECT packets. REJECT tells the sender that the packet has been blocked by the firewall. DROP just drops the packet and does not send a response. I like DROP better for regular use, but REJECT might be more helpful if you need to diagnose a connectivity issue.

Logging Dropped Connections

DROP_LOGGING = "1"

This option logs dropped connections to syslog. I don’t see any reason to turn this off unless your hard drive is getting full.

Port Exceptions When Logging Dropped Connections

DROP_NOLOG = "67,68,111,113,135:139,445,500,513,520"

These ports are specifically blocked from being logged either to conserve hard drive space or make the log file easier to read.

“Watch Mode”

WATCH_MODE = "0"

If you are ever stuck trying to troubleshoot a large ruleset, you might consider turning this option on. You can use it to track the actions to watched IP addresses to see where they are getting blocked or accepted.

Login Failure Daemon Alert

LF_ALERT_TO = ""
LF_ALERT_FROM = ""
LF_ALERT_SMTP = ""

You can specify an email address to report errors from the Login Failure Daemon, which tracks and automatically blocks brute force login attempts.

Permanent Blocks and NetBlocks

LF_PERMBLOCK = "1"
LF_PERMBLOCK_INTERVAL = "86400"
LF_PERMBLOCK_COUNT = "4"
LF_PERMBLOCK_ALERT = "1"
LF_NETBLOCK = "0"
LF_NETBLOCK_INTERVAL = "86400"
LF_NETBLOCK_COUNT = "4"
LF_NETBLOCK_CLASS = "C"
LF_NETBLOCK_ALERT = "1"

These settings control the permanent block and netblock blocking. You probably don’t need to touch these settings, but you might want some additional security or less security depending on your company needs. If something gets permablocked, it will require your intervention to clear it, which might create downtime for your clients. Likewise, if a legitimate IP address happens to be part of a netblock which has an attacking IP address on it, it will get blocked if you have that feature turned on. A class C network encompasses 256 IP addresses. You can set this to class B or A, but that could block thousands or millions of IP addresses, respectively. Unless you find yourself under constant attack, I would advise you to leave that LF_NETBLOCK off.

Additional Protection During Updates

# Safe Chain Update. If enabled, all dynamic update chains (GALLOW*, GDENY*,
# SPAMHAUS, DSHIELD, BOGON, CC_ALLOW, CC_DENY, ALLOWDYN*) will create a new
# chain when updating, and insert it into the relevant LOCALINPUT/LOCALOUTPUT
# chain, then flush and delete the old dynamic chain and rename the new chain.
#
# This prevents a small window of opportunity opening when an update occurs and
# the dynamic chain is flushed for the new rules.
SAFECHAINUPDATE = "0"

Activating this option will increase your system resource usage and will require more rules to be running at one time, but it provides an additional layer of protection during updates. Without this option turned on, your rules will be flushed for a short amount of time, leaving your server vulnerable.

Multi-Server Deployment Options

LF_GLOBAL = "0"
GLOBAL_ALLOW = ""
GLOBAL_DENY = ""
GLOBAL_IGNORE = ""

Like APF, you can configure global lists for multiple server deployments. You’ll need to specify a URL of the text file with the IP addresses for the global lists.

SPAMHAUSE Blocklist

LF_SPAMHAUS = "0"

This option enables the SPAMHAUS blocklist. Specify the number of seconds between refreshes. Recommended setting is 86400 (1 day).

Blocking TOR Exit IP Addresses

LF_TOR = "0"

Enabling this option will block TOR exit IP addresses. If you are not familiar with TOR, it is a completely anonymous proxy network. This could block some legitimate users who are trying to protect their anonymity, so I would recommend only turning this on if you are already under attack from a TOR exit address.

Blocking Bogon Addresses

LF_BOGON = "0"
LF_BOGON_URL = "http://www.cymru.com/Documents/bogon-bn-agg.txt"
LF_BOGON_SKIP = ""

Blocking bogon addresses (addresses that should not be possible) is usually a good decision. To enable, set the number of seconds between refreshes. I recommend enabling this option and setting the refresh at 86400 (1 day). If you do so, be sure to add your private network adapters to the skip list.

Country-Specific Access to Your Server

CC_DENY = ""
CC_ALLOW = ""

With these options, you can block or allow entire countries from accessing your server. To do so, enter the country codes in a comma separated list. Even though this generates a lot of additional rules, it’s valuable to some sysadmins.

CC_ALLOW_FILTER = ""

Alternatively, you can set your server to exclusively accept traffic from a list of country codes. All other countries not listed will have their traffic dropped. There are many other settings related to these options that I don’t have time to cover in this blog.

Blocking Login Failures

LF_TRIGGER = "0"

This enables blocking of login failures (per service). There are a lot of great customization options in this section.

Scanning Directories for Malicious Files

LF_DIRWATCH = "300"

This feature scans /tmp and /dev/shm for potentially malicious files and alerts you to their presence based on the interval you designate. You can also have CSF automatically quarantine malicious files with this option:

LF_DIRWATCH_DISABLE = "0"

Distributed Attack Protection

LF_DISTATTACK = "0"

By enabling this option, you activate additional protection against distributed attacks.

Blocking Based on Abusive Email Usage

LT_POP3D = "0"
LT_IMAPD = "0"

If a user checks email too many times per hour (more than the non-zero value specified), the user’s IP address is blocked.

Email Alert Following Block

LT_EMAIL_ALERT = "1"

This will send you email when something is blocked. I’d recommend leaving it on.

Blocking IP Addresses Based on Number of Connections

CT_LIMIT = "0"

This feature tracks connections and blocks the IP if the number of connections is too high. Use caution because if you enable this option and set this value too low, it will block legitimate traffic.

Application-Level Protection

PT_LIMIT = "60"

This feature provides application level protection against malicious scripts that take a long time to execute.

Blocking Port Scanners

PS_INTERVAL = "300"
PS_LIMIT = "10"

Enabling HTML User Interface for CSF

UI = "0"

CSF has a built-in HTML user interface. You can enable this by setting UI = “1″. There are a list of prerequisites for this option in the readme.txt.

Notifying Blocked IP Addresses

MESSENGER = "0"

This option will notify blocked IP addresses when they have been blocked by the firewall.

Port Knocking

PORTKNOCKING = ""

CSF supports port knocking, which is a technique that provides an additional layer of security. See http://www.portknocking.org/ for details.

Allow and Deny Lists

As we walked through the CSF configuration file, you saw that I referenced the csf.deny file, so it should come as no surprise that CSF also includes csf.allow to customize “allow” rules as well. If you are familiar with APF, these files have a very similar syntax … Each entry is made up of the same four components: protocol|flow|port|IP. The only real difference being that APF uses the colon as a delimiter while CSF uses the pipe:

#APF Version
tcp:in:d=48000_48020:s=10.0.0.0/8
 
#CSF Version
tcp|in|d=48000_48020|s=10.0.0.0/8

Fortunately, replacing your colon with a pipe is a minimally invasive procedure that can be automated with a tool like vi.

CSF Command Line Tool

The command line tool for CSF is much more robust than the one for APF:

$ csf --help
csf: v5.79 (cPanel)
 
ConfigServer Security & Firewall
(c)2006-2013, Way to the Web Limited (http://www.configserver.com)
 
Usage: /usr/sbin/csf [option] [value]
 
Option              Meaning
-h, --help          Show this message
-l, --status        List/Show iptables configuration
-l6, --status6      List/Show ip6tables configuration
-s, --start         Start firewall rules
-f, --stop          Flush/Stop firewall rules (Note: lfd may restart csf)
-r, --restart       Restart firewall rules
-q, --startq        Quick restart (csf restarted by lfd)
-sf, --startf       Force CLI restart regardless of LF_QUICKSTART setting
-a, --add ip        Allow an IP and add to /etc/csf.allow
-ar, --addrm ip     Remove an IP from /etc/csf.allow and delete rule
-d, --deny ip       Deny an IP and add to /etc/csf.deny
-dr, --denyrm ip    Unblock an IP and remove from /etc/csf.deny
-df, --denyf        Remove and unblock all entries in /etc/csf.deny
-g, --grep ip       Search the iptables rules for an IP match (incl. CIDR)
-t, --temp          Displays the current list of temp IP entries and their TTL
-tr, --temprm ip    Remove an IPs from the temp IP ban and allow list
-td, --tempdeny ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP ban list. ttl is how long to
                    blocks for (default:seconds, can use one suffix of h/m/d).
                    Optional port. Optional direction of block can be one of:
                    in, out or inout (default:in)
-ta, --tempallow ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP allow list (default:inout)
-tf, --tempf        Flush all IPs from the temp IP entries
-cp, --cping        PING all members in an lfd Cluster
-cd, --cdeny ip     Deny an IP in a Cluster and add to /etc/csf.deny
-ca, --callow ip    Allow an IP in a Cluster and add to /etc/csf.allow
-cr, --crm ip       Unblock an IP in a Cluster and remove from /etc/csf.deny
-cc, --cconfig [name] [value]
                    Change configuration option [name] to [value] in a Cluster
-cf, --cfile [file] Send [file] in a Cluster to /etc/csf/
-crs, --crestart    Cluster restart csf and lfd
-w, --watch ip      Log SYN packets for an IP across iptables chains
-m, --mail [addr]   Display Server Check in HTML or email to [addr] if present
-lr, --logrun       Initiate Log Scanner report via lfd
-c, --check         Check for updates to csf but do not upgrade
-u, --update        Check for updates to csf and upgrade if available
-uf                 Force an update of csf
-x, --disable       Disable csf and lfd
-e, --enable        Enable csf and lfd if previously disabled
-v, --version       Show csf version

The command line tool will also tell you if the testing mode is enabled (which is a very useful feature). If TESTING were enabled, we’d see this line at the bottom of the output:

*WARNING* TESTING mode is enabled - do not forget to disable it in the configuration

Did you make it all the way through?! Great! I know it’s a lot to take in, but it’s not terribly complicated when we break it down and understand how each piece works. Next time, I’ll be back with some tips on integrating CSF into cPanel.

-Mark

Don’t worry … it’s really quit simple. If you can use FTP (File Transfer Protocol), you can handle this bit of server management. Depending on the exact problem, we might instruct you to free up space by removing files in one of the following directories:

• /var/log
• /usr/local/cpanel
• /usr/local/apache/logs
• /usr/local/apache/domlogs

The reason these directories are usually overlooked is because they are not accessible by normal FTP users — users who only upload website content. When you upload website content to the server via FTP, the FTP user is limited to the directory structure for that website. Directories starting with “/var” and “/usr” cannot be accessed by these non-root users (The “root” user can access anything). And while root is a powerful user, for the sake of security, it is not normally allowed to log in over FTP because FTP is not secure … That’s where SFTP (Secure File Transfer Protocol) comes in.

Most FTP clients support SFTP, so you don’t have to learn a new environment to securely access any file on the server. Every FTP client is different, but I’ll illustrate with FileZilla because it’s free and available on Mac, Windows and Linux. If you don’t already have an FTP client, I highly recommend FileZilla. Because there are a few ways to use FileZilla to get an SFTP connection, I can share different options for you to try:

Quick Connect

The Quick Connect bar is the quickest way to connect to your server. Start FileZilla and look immediately under the toolbar for the Quick Connect bar:

Enter the hostname (IP address or domain name), “root” in the Username field, the root password in the Password field, and “22″ in the port field. Remember, port 22 is for SFTP, the same as SSH. Click the Quickconnect button to connect.

Using the Site Manager

The Site Manager lets you save your login details. Start FileZilla and you’ll see the following:

To open the Site Manager, click the left-most icon in tool bar or go to File >> Site Manager in the menu.

Enter an IP address or domain name for your server in the Host field, and select “SFTP” as your protocol. You’ll enter the root user’s login information, and you’re ready to connect by clicking the “Connect” button or you can click the “OK” button to save and close the dialog box.

If you just saved your settings and the Site Manager is not open, click the Site Manager icon again. From there, you can select the site under the “Select Entry” box, and you just have to click “Connect” to initiate the SFTP connection with your saved settings.

If you see a pop-up that warns of an “Unknown host key,” clicking the “Always trust this host, add this key to the cache” option will prevent this interruption from showing in the future. Once you click “OK” to complete the connection, your FileZilla screen should look like this:

Notice the “Remote site” section on the middle right of the FileZilla screen:

This area in FileZilla is the directory and file listing of the server. Navigate the server’s file structure here, and click “/” to access the top of the folder structure. You should see the “/usr” and “/var” directories, and you can explore the filesystem to delete the files technical support recommended to create space!

Message Log

If you have a problem connecting to your server by FTP or SFTP, the open area below the Quickconnect bar is the Message Log. If you can copy and paste this text into a ticket, you’ll help technical support troubleshoot your connection problems. Below is an example log of a successful FTP session:

Status: Connecting to server.example.com...
Response:   fzSftp started
Command:    open "root@server.example.com" 22
Command:    Trust new Hostkey: Once
Command:    Pass: **********
Status: Connected to server.example.com
Status: Retrieving directory listing...
Command:    pwd
Response:   Current directory is: "/root"
Command:    ls
Status: Listing directory /root
Status: Calculating timezone offset of server...
Command:    mtime ".lesshst"
Response:   1326387703
Status: Timezone offsets: Server: -21600 seconds. Local: -21600 seconds. Difference: 0 seconds.
Status: Directory listing successful

And here’s an example of a failed connection:

Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection timed out
Error:  Could not connect to server
Status: Waiting to retry...
Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection attempt interrupted by user

If you have any questions, leave them in a comment below. Enjoy your new-found SFTP powers!

-Lyndell

Every profession has their collection of acronyms. The little expressions serve as a verbal and written short hand for people who toil daily with the topics of their trade. I’m proud to confess that I’ve been using these minute medleys of letters for over twelve years. Given that I work on the Internet, I’ve been exposed to hundreds of acronyms in the fields of technology, business and management, and in my experience, I’ve had to break through several acronym “barriers” to get in the know. Because I happen to interact with customers every day as the manager of SoftLayer’s technical support department, I’ve encountered a few “Can you tell means?” responses, so I thought I’d write a quick blog post to clarify some of the common acronyms you may see in the SoftLayer vernacular.

Within support we have our CSTs (customer support technicians) and CSAs (customer support admins) who, with the help of SBTs (server build technicians), manage our massive fleet of servers. SBTs are the hands and eyes of our data centers, working closely with the hardware to ensure your server is online and operating in peak condition. The CSTs and CSAs are focused on the software and services that power your websites and applications.

Beyond employee title acronyms, you’ll probably see a collection of terms that describe the products and services that we manage. In support, we receive questions about accessing servers or CCIs (cloud computing instances) using KVM (Keyboard, Video and Mouse) or IPMI (Intelligent Platform Management Interface) through our VPN (Virtual Private Network). Once connected to our back-end network through a SSL (Secure Socket Layer), PPTP (Point-to-Point Tunnel Protocol) or IPSEC (Internet Protocol Security) VPN, you have access to services such as DNS (Domain Name Service), NAS (Network Attached Storage) or iSCSI (Internet Small Computer System Interface). Finally, while discussing our network, I often refer to http://www.softlayer.com/diagrams/pod-network-diagram/dal05 to show the difference between a VER (VPN Edge Router) and a BCS (Back-end Customer Switch).

If you run across an acronym you don’t understand in a ticket, please let us know so we can share its full meaning … By using these shortened terms, our team can provider faster service (and you can read their responses quicker). I know that seeing all the bold TLAs above may seem a little off-putting initially, but as you have a chance to read them in the context of some of the other acronyms you already know, I hope you have an “Aha!” moment … Like finding the Rosetta Stone or the Code of Hammurabi. Given the quick glance at the terms above, if you want to learn more about one of the TLAs in particular, leave a comment below, and we’ll respond in another comment with details.

CBNO

-Chris

I’m not a graphics specialist or a GUI designer. I just don’t have that in my DNA. I focus more on the technical side of things working on better ways to deliver content to the user. And in the purely technical area, the best thing to do to improve the user experience is to improve the delivery speed to the user.

There are a lot of tools out there that can be used to speed up delivery. CDN, for example, is an awesome way to get static content to an end user and is very scalable. But what about scaling out the application itself?

Traditionally, a simple Layer-4 Load Balancer has been a staple component of scalable applications. This type of Load Balancing can provide capacity during traffic peaks as well as increase availability. The application runs on several servers and the load balancer uses some simple methods (least connections, round robin, etc) to distribute the load. For a lot of applications this is sufficient to get content reliably and quickly to the end user. SoftLayer offers a relatively inexpensive load-balancing service for our customers that can provide this functionality.

There is another, more sophisticated, tool that can be used to manage internet application traffic. That is the “Application Delivery Controller” (obligatory Wikipedia link: http://en.wikipedia.org/wiki/Application_Delivery_Controller) or “Load Balancer on Steroids”. This class of traffic manager can act in Layer-7, the data layer. These devices can make decisions based on the actual content of the data packets, not just the source and destination.

And an ADC can do more than load balance. It can act as a Web Application Firewall to protect your data. It can speed up your application using SSL Offloading, Content Caching, TCP Optimization, and more. This type of device is very smart and very configurable and will help in the delivering the application to the end user.

At SoftLayer we have seen our customers achieve a lot of success with our Layer-4 Load Balancer product. But we are always looking for other tools to help our customers. We always have admired the advanced functionality in the appliance-based Application Delivery Controllers on the market. Finding a way to get this enterprise-grade technology to our customers in an affordable manner was problematic. When Citrix announced that they were going to create a version of their NetScaler product that didn’t require an appliance we were thrilled. With the announcement of the NetScaler VPX we finally thought we had found the right product that we could use to affordably provision this advanced technology on-demand to our customers.

SoftLayer is VERY excited to partner with Citrix to provide the NetScaler VPX Application Delivery Controller to our customers. Our customers can order a NetScaler VPX, and in a matter of minutes be managing the delivery of their online applications using one of the most sophisticated tools on the market. Citrix does a better job of promoting the product than I do, so here is the link to their site: http://citrix.com/English/ps2/products/product.asp?contentID=21679&ntref=hp_nav_US.

Remember, it’s all about the experience of the user at the other end of the wire. Find the right tools to manage that experience and you are most of the way there. Oh yeah, and find a good graphics designer too. That helps. So does good content.

-@nday91

Knowledge gained by one tech in tackling and conquering a specific issue is shared with all for the betterment of the team. Like a gladiator in the bowels of the Roman Coliseum of old preparing for his fight, the SoftLayer Support technician must be ready to do battle. Disciplined cross-training is the order of the day; mental and physical preparation is key. A technician must enter the halls of a SoftLayer datacenter ready to conquer whatever comes through the gates! It is truly a battle worthy of the Coliseum.

You might ask how a day in the SoftLayer NOC resolving technical issues compares to a battle fought in the Roman Coliseum. Well, if you measure a “battle” by the excitement and tension in the air … the blood, the sweat, and yes, at times, the tears, the pain of defeat, and the celebration of victory, then SoftLayer Technical Support technicians are definitely involved in a true battle worthy of the Roman Coliseum on a daily basis.

Picture, if you will, a well-trained, focused individual walking into the Dallas Infomart with his security badge in hand. He is not there to pass the time or participate in some mind-numbing repetitive task. He is there to do battle with a beast named Technology. With a strategic plan in mind, he enters the elevator preparing himself mentally for what surely awaits upon entering his cubicle. As he opens the door to the NOC, he is greeted by his fellow “warriors”. Some are weary from battle, yet have a sense of satisfaction about them as a Roman Soldier of old looking across the battlefield at his conquered foe.

The stories of a multitude of battles won, and maybe even a few lost, are recounted. The technical warrior packs some sustenance from the chow line (the loaded NOC break room refrigerator), and settles into his chariot he likes to call a cubicle, pulling out his weapons, a keyboard and mouse, and bringing up the battlefield onscreen. He begins with the speed of a cheetah typing more and more furiously as each ticket darts to and fro trying to elude him. The warrior is undaunted. He will not be defeated today. Yes, he may need to look to his comrades in arms for assistance in flanking the enemy, but in the end, as a team of highly trained warriors, they will prevail.

This day will not be without its casualties, but the warrior must always repeat to himself, “I will not let our customer’s down. The enemy (technical issues) will not prevail…not on my watch.”

As did the citizens of Rome, I take great pride in our warriors and the superior way in which they continue to win battles for the glory of our customers. The inspiration for this writing came from a recent victory in which a warrior named Stefanus (Steve) stood in victory after wrestling with a beast of an issue, which he finally destroyed while the customer rejoiced and his wealth increased. All the warriors: Krishenus, Jamesus, and Samuel gathered around Stefanus to congratulate him on his victory. Of course, they all knew that the ultimate victory was enjoyed by the SoftLayer customer.

-David

Microsoft is no different than any other developer when it comes to writing software–they get better with each iteration. There is not a person out there who would argue that the world of home computers would have been better off if none of us ever progressed beyond MS-DOS 1.0. Not to say there is anything wrong with MS-DOS. I love it. And still use it occasionally doing embedded work. But my point is that while there have certainly been some false starts along the way (can you say BOB), Microsoft’s operating systems generally get better with each release.

So why not go out and update everything the day the latest and greatest OS hits the shelves? Because as most of you know, there are bugs that have to get worked out. To add to that, the more complex the OS gets, the more bugs there are and the more time it takes to shake those bugs out. Windows Server 2008 is no different. In my experience there are still a number of troublesome issues with W2K8 that need to be addressed. Just to name a few:

• UAC (user access control) – these are the security features that give us so much headache. I’m not saying we don’t need the added security. I’m just saying this is a new arena for MS and they still have a lot to learn. After clicking YES, I REALLY REALLY REALLY WANT TO INSTALL SAID APPLICATION for the 40th time in a day, most administrators will opt to disable UAC, thereby thwarting the added security benefits entirely. If I were running this team at MS I’d require all my developers to take a good hard look at LINUX.
• UMD (user mode drivers) – the idea of running a device driver, or a portion of a device driver, in the restricted and therefore safe user memory of the kernel is a great idea in terms of improving OS reliability. I’ve seen numbers suggesting that as many as 90% of hard OS failures are caused by faulty third-party drivers mucking around in kernel mode. However implementing user mode drivers adds some new complexities if hardware manufacturers don’t want to take a performance hit and from my experience not all hardware vendors are up to speed yet.
• Driver Verification – this to me is the most troublesome and annoying issue right now with the 64-bit only version of W2K8. Only kernel mode software that has been certified in the MS lab is allowed to execute on a production boot of the OS. Period. Since I am writing this on the SoftLayer blog, I am assuming most of you are not selecting hardware and drivers to run on your boxes. We are handling that for you. But let me tell you it’s a pain in the butt to only run third party drivers that have been through the MS quality lab. Besides not being able to run drivers we have developed in house it is impossible for us to apply a patch from even the largest of hardware vendors without waiting on that patch to get submitted to MS and then cleared for the OS. A good example was a problem we ran into with an Intel Enet driver. Here at SoftLayer we found a bug in the driver and after a lot of back and forth with Intel’s Engineers we had a fix in hand. But that fix could not be applied to the W2K8 64-bit boxes until weeks later when the fix finally made it from Intel to MS and back to Intel and us again. Very frustrating.

Okay, so now that you see some of the reasons NOT to use MS Windows Server 2008 what are some of the reasons it’s at least worth taking a look at? Well here are just a few that I know of from some of the work I have done keeping up to speed with the latest driver model.

• Improved Memory Management – W2K8 issues fewer and larger disk I/O’s than its 2003 predecessor. This applies to standard disk fetching, but also paging and even read-aheads. On Windows 2003 it is not uncommon for disk writes to happen in blocks < 64KB and certainly never more than 64KB (this was a limitation that had been in place since Windows NT). On W2K8 the memory manager frequently handles writes as large as a full MB. Furthermore, writing related data in large blocks (especially during paging) significantly reduces fragmentation vastly improving read back times as well.
• Improved Data Reliability – Everyone knows how painful disk corruption can be. And everyone knows taking a server offline on a regular basis to run chkdsk and repair disk corruption is slow. One of the ideal improvements in terms of administering a websever is that W2K8 employs a technology called NTFS self-healing. This new feature built into the file system detects disk corruption on the fly and quarantines that sector, allowing system worker-threads to execute chkdsk like repairs on the corrupted area without taking the rest of the volume offline.
• Scalability – The W2K8 kernel introduces a number of streamlining factors that greatly enhance system wide performance. A minor but significant change to the operating system’s low level timer code, combined with new I/O completion handling, and more efficient thread pool, offer marked improvement on load-heavy server applications. I have read documentation supporting claims that the minimization in CPU synchronization alone results directly in a 30% gain on the number of concurrent Windows 2008 users over 2003. That’s not to say once you throw in all the added security and take the user mode driver hit you won’t be looking at 2003 speeds. I’m just pointing out hard kernel-level improvements that can be directly quantified by multiplying your resources against the number of saved CPU cycles.

Alright, no need to beat a dead horse. My hope was if nothing else to muddy the waters a bit. The majority of posts I read on our internal forums seemed to recommend avoiding W2K8 like the plague. I’m only suggesting while it is certainly not perfect, there are some benefits to at least taking it for a test drive. Besides, with SoftLayer’s handy dandy portal driven OS deployment, in the amount of time it took you to read all my rambling you might have already installed Windows Server 2008 and tried it out for yourself. Okay, maybe that’s a bit of an exaggeration. But still…you get the idea!

-William

Have you ever wished you could copy data to your server without bandwidth concerns, or wondered how to reboot a server that is no longer responding? Ever wonder what the RescueLayer is and who it rescues? What is NAS, iSCSI, a firewall, how do I load balance? What is the CDNLayer and how can it help me? These are just a few of things we can help you better understand. This will allow you the leisure to ponder the ways to make your server more profitable. No more wasting time creating support tickets when you can do it yourself fast and easy. We can show you how. Here is a little more about the team:

What do we do for fun – Our hobbies include Aviation, Camping, Music, and Automotive Engineering. Some like the outdoors, golf, Karate, poker, etc and some spend their off hours with the family and kids. Smarts – The technical abilities in the group range from Automation to Xen and everything in between including disaster recovery, portal and backend database design, server administration, load balancing, hosting in general (the what to do’s and what not to do’s), ASP and PHP development, developing online collaborations sites, MMO gaming, and LAMP stacks to Windows. We try to be as well rounded as we can. The group has years and years of experience in the hosting, internet, ISP, and system administration arena and we are here and ready to help.

We are STAT! The SoftLayer Technical Assurance Team, pleasure to meet you. How can we help?

-John