Posts Tagged 'Traffic'

October 14, 2013

Product Spotlight: Vyatta Network Gateway Appliance

In the wake of our recent Vyatta network gateway appliance product launch, I thought I'd address some of the most common questions customers have asked me about the new offering. With inquiries spanning the spectrum from broad and general to detailed and specific, I might not be able to cover everything in this blog post, but at the very least, it should give a little more context for our new network gateway offering.

To begin, let's explore the simplest question I've been asked: "What is a network gateway?" A network gateway provides tools to manage traffic into and out of one or more VLANs (Virtual Local Area Networks). The network gateway serves a customer-configurable routing device that sits in front of designated VLANs. The servers in those VLANs route through the network gateway appliance as their first hop instead of Front-end Customer Routers (FCR) or Back-end Customer Routers (BCR). From an infrastructure perspective, SoftLayer's network gateway offering consists of a single server, and in the future, the offering will be expanded to multi-server configurations to support high availability needs and larger clustered configurations.

The general function of a network gateway may seem a little abstract, so let's look at a couple real world use cases to see how you can put that functionality to work in your own cloud environment.

Example 1: Complex Traffic Management
You have a multi-server cloud environment and a complex set of firewall rules that allow certain types of traffic to certain servers from specific addresses. Without a network gateway, you would need to configure multiple hardware and software firewalls throughout your topology and maintain multiple rules sets, but with the network gateway appliance, you streamline your configuration into a single point of control on both the public and private networks.

After you order a gateway appliance in the SoftLayer portal and configure which VLANs route through the appliance, the process of configuring the device is simple: You define your production, development and QA environments with distinct traffic rules, and the network gateway handles the traffic segmentation. If you wanted to create your own VPN to connect your hosted environment to your office or in-house data center, that configuration is quick and easy as well. The high-touch challenge of managing several sets of network rules across multiple devices is simplified and streamlined.

Example 2: Creating a Static NAT
You want to create a static NAT (Network Address Translation) so that you can direct traffic through a public IP address to an internal IP address. With the IPv4 address pool dwindling and new allocations being harder to come by, this configuration is becoming extremely popular to accommodate users who can't yet reach IPv6 addresses. This challenge would normally require a significant level of effort of even the most seasoned systems administrator, but with the gateway appliance, it's a painless process.

In addition to the IPv4 address-saving benefits, your static NAT adds a layer of protection for your internal web servers from the public network, and as we discussed in the first example, your gateway device also serves as a single configuration point for both inbound and outbound firewall rules.

If you have complex network-related needs, and you want granular control of the traffic to and from your servers, a gateway appliance might be the perfect tool for you. You get the control you want and save yourself a significant amount of time and effort configuring and tweaking your environment on-the-fly. You can terminate IPSec VPN tunnels, execute your own network address translation, and run diagnostic commands such as traffic monitoring (tcpdump) on your global environment. And in addition to that, your gateway serves as a single point of contact to configure sophisticated firewall rules!

If you want to learn more about the gateway appliance, check out KnowledgeLayer or contact our friendly sales team directly with your questions: sales@softlayer.com

-Ben

April 29, 2013

Web Development - Installing mod_security with OWASP

You want to secure your web application, but you don't know where to start. A number of open-source resources and modules exist, but that variety is more intimidating than it is liberating. If you're going to take the time to implement application security, you don't want to put your eggs in the wrong basket, so you wind up suffering from analysis paralysis as you compare all of the options. You want a powerful, flexible security solution that isn't overly complex, so to save you the headache of making the decision, I'll make it for you: Start with mod_security and OWASP.

ModSecurity (mod_security) is an open-source Apache module that acts as a web application firewall. It is used to help protect your server (and websites) from several methods of attack, most common being brute force. You can think of mod_security as an invisible layer that separates users and the content on your server, quietly monitoring HTTP traffic and other interactions. It's easy to understand and simple to implement.

The challenge is that without some advanced configuration, mod_security isn't very functional, and that advanced configuration can get complex pretty quickly. You need to determine and set additional rules so that mod_security knows how to respond when approached with a potential threat. That's where Open Web Application Security Project (OWASP) comes in. You can think of the OWASP as an enhanced core ruleset that the mod_security module will follow to prevent attacks on your server.

The process of getting started with mod_security and OWASP might seem like a lot of work, but it's actually quite simple. Let's look at the installation and configuration process in a CentOS environment. First, we want to install the dependencies that mod_security needs:

## Install the GCC compiler and mod_security dependencies ##
$ sudo yum install gcc make
$ sudo yum install libxml2 libxml2-devel httpd-devel pcre-devel curl-devel

Now that we have the dependencies in place, let's install mod_security. Unfortunately, there is no yum for mod_security because it is not a maintained package, so you'll have to install it directly from the source:

## Get mod_security from its source ##
$ cd /usr/src
$ git clone https://github.com/SpiderLabs/ModSecurity.git

Now that we have mod_security on our server, we'll install it:

## Install mod_security ##
$ cd ModSecurity
$ ./configure
$ make install

And we'll copy over the default mod_security configuration file into the necessary Apache directory:

## Copy configuration file ##
$ cp modsecurity.conf-recommended /etc/httpd/conf.d/modsecurity.conf

We've got mod_security installed now, so we need to tell Apache about it ... It's no use having mod_security installed if our server doesn't know it's supposed to be using it:

## Apache configuration for mod_security ##
$ vi /etc/httpd/conf/httpd.conf

We'll need to load our Apache config file to include our dependencies (BEFORE the mod_security module) and the mod_security file module itself:

## Load dependencies ##
LoadFile /usr/lib/libxml2.so
LoadFile /usr/lib/liblua5.1.so
## Load mod_security ##
LoadModule security2_module modules/mod_security2.so

We'll save our configuration changes and restart Apache:

## Restart Apache! ##
$ sudo /etc/init.d/httpd restart

As I mentioned at the top of this post, our installation of mod_security is good, but we want to enhance our ruleset with the help of OWASP. If you've made it this far, you won't have a problem following a similar process to install OWASP:

## OWASP ##
$ cd /etc/httpd/
$ git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git
$ mv owasp-modsecurity-crs modsecurity-crs

Just like with mod_security, we'll set up our configuration file:

## OWASP configuration file ##
$ cd modsecurity-crs
$ cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_config.conf

Now we have mod_security and the OWASP core ruleset ready to go! The last step we need to take is to update the Apache config file to set up our basic ruleset:

## Apache configuration ##
$ vi /etc/httpd/conf/httpd.conf

We'll add an IfModule and point it to our new OWASP rule set at the end of the file:

<IfModule security2_module>
    Include modsecurity-crs/modsecurity_crs_10_config.conf
    Include modsecurity-crs/base_rules/*.conf
</IfModule>

And to complete the installation, we save the config file and restart Apache:

## Restart Apache! ##
$ sudo /etc/init.d/httpd restart

And we've got mod_security installed with the OWASP core ruleset! With this default installation, we're leveraging the rules the OWASP open source community has come up with, and we have the flexibility to tweak and enhance those rules as our needs dictate. If you have any questions about this installation or you have any other technical blog topics you'd like to hear from us about, please let us know!

-Cassandra

January 29, 2013

iptables Tips and Tricks: APF (Advanced Policy Firewall) Configuration

Let's talk about APF. APF — Advanced Policy Firewall — is a policy-based iptables firewall system that provides simple, powerful control over your day-to-day server security. It might seem intimidating to be faced with all of the features and configuration tools in APF, but this blog should put your fears to rest.

APF is an iptables wrapper that works alongside iptables and extends its functionality. I personally don't use iptables wrappers, but I have a lot of experience with them, and I've seen that they do offer some additional features that streamline policy management. For example, by employing APF, you'll get several simple on/off toggles (set via configuration files) that make some complex iptables configurations available without extensive coding requirements. The flip-side of a wrapper's simplicity is that you aren't directly in control of the iptables commands, so if something breaks it might take longer to diagnose and repair. Before you add a wrapper like APF, be sure that you know what you are getting into. Here are a few points to consider:

  • Make sure that what you're looking to use adds a feature you need but cannot easily incorporate with iptables on its own.
  • You need to know how to effectively enable and disable the iptables wrapper (the correct way ... read the manual!), and you should always have a trusted failsafe iptables ruleset handy in the unfortunate event that something goes horribly wrong and you need to disable the wrapper.
  • Learn about the basic configurations and rule changes you can apply via the command line. You'll need to understand the way your wrapper takes rules because it may differ from the way iptables handles rules.
  • You can't manually configure your iptables rules once you have your wrapper in place (or at least you shouldn't).
  • Be sure to know how to access your server via the IPMI management console so that if you completely lock yourself out beyond repair, you can get back in. You might even go so far as to have a script or set of instructions ready for tech support to run, in the event that you can't get in via the management console.

TL;DR: Have a Band-Aid ready!

APF Configuration

Now that you have been sufficiently advised about the potential challenges of using a wrapper (and you've got your Band-Aid ready), we can check out some of the useful APF rules that make iptables administration a lot easier. Most of the configuration for APF is in conf.apf. This file handles the default behavior, but not necessarily the specific blocking rules, and when we make any changes to the configuration, we'll need to restart the APF service for the changes to take effect.

Let's jump into conf.apf and break down what we see. The first code snippit is fairly self-explanatory. It's another way to make sure you don't lock yourself out of your server as you are making configuration changes and testing them:

# !!! Do not leave set to (1) !!!
# When set to enabled; 5 minute cronjob is set to stop the firewall. Set
# this off (0) when firewall is determined to be operating as desired.
DEVEL_MODE="1"

The next configuration options we'll look at are where you can make quick high-level changes if you find that legitimate traffic is being blocked and you want to make APF a little more lenient:

# This controls the amount of violation hits an address must have before it
# is blocked. It is a good idea to keep this very low to prevent evasive
# measures. The default is 0 or 1, meaning instant block on first violation.
RAB_HITCOUNT="1"
 
# This is the amount of time (in seconds) that an address gets blocked for if
# a violation is triggered, the default is 300s (5 minutes).
RAB_TIMER="300"
# This allows RAB to 'trip' the block timer back to 0 seconds if an address
# attempts ANY subsiquent communication while still on the inital block period.
RAB_TRIP="1"
 
# This controls if the firewall should log all violation hits from an address.
# The use of LOG_DROP variable set to 1 will override this to force logging.
RAB_LOG_HIT="1"
 
# This controls if the firewall should log all subsiqent traffic from an address
# that is already blocked for a violation hit, this can generate allot of logs.
# The use of LOG_DROP variable set to 1 will override this to force logging.
RAB_LOG_TRIP="0"

Next, we have an option to adjust ICMP flood protection. This protection should be useful against some forms of DoS attacks, and the associated rules show up in your INPUT chain:

# Set a reasonable packet/time ratio for ICMP packets, exceeding this flow
# will result in dropped ICMP packets. Supported values are in the form of:
# pkt/s (packets/seconds), pkt/m (packets/minutes)
# Set value to 0 for unlimited, anything above is enabled.
ICMP_LIM="30/s"

If you wanted to add more ports to block for p2p traffic (which will show up in the P2P chain), you'll update this code:

# A common set of known Peer-To-Peer (p2p) protocol ports that are often
# considered undesirable traffic on public Internet servers. These ports
# are also often abused on web hosting servers where clients upload p2p
# client agents for the purpose of distributing or downloading pirated media.
# Format is comma separated for single ports and an underscore separator for
# ranges (4660_4678).
BLK_P2P_PORTS="1214,2323,4660_4678,6257,6699,6346,6347,6881_6889,6346,7778"

The next few lines let you designate the ports that you want to have closed at all times. They will be blocked for INPUT and OUTPUT chains:

# These are common Internet service ports that are understood in the wild
# services you would not want logged under normal circumstances. All ports
# that are defined here will be implicitly dropped with no logging for
# TCP/UDP traffic inbound or outbound. Format is comma separated for single
# ports and an underscore separator for ranges (135_139).
BLK_PORTS="135_139,111,513,520,445,1433,1434,1234,1524,3127"

The next important section to look at deals with conntrack. If you get "conntrack full" errors, this is where you'd increase the allowed connections. It's not uncommon to need more connections than the default, so if you need to adjust that value, you'd do it here:

# This is the maximum number of "sessions" (connection tracking entries) that
# can be handled simultaneously by the firewall in kernel memory. Increasing
# this value too high will simply waste memory - setting it too low may result
# in some or all connections being refused, in particular during denial of
# service attacks.
SYSCTL_CONNTRACK="65536"

We've talked about the ports we want closed at all times, so it only makes sense that we'd specify which ports we want open for all interfaces:

# Common inbound (ingress) TCP ports
IG_TCP_CPORTS="22"
# Common inbound (ingress) UDP ports
IG_UDP_CPORTS=""
# Common outbound (egress) TCP ports
EG_TCP_CPORTS="21,25,80,443,43"
# Common outbound (egress) UDP ports
EG_UDP_CPORTS="20,21,53"

And when we want a special port allowance for specific users, we can declare it easily. For example, if we want port 22 open for user ID 0, we'd use this code:

# Allow outbound access to destination port 22 for uid 0
EG_TCP_UID="0:22"

The next few sections on Remote Rule Imports and Global Trust are a little more specialized, and I encourage you to read a little more about them (since there's so much to them and not enough space to cover them here on the blog). An important feature of APF is that it imports block lists from outside sources to keep you safe from some attackers, so the Remote Rule Imports can prove to be very useful. The Global Trust section is incredibly useful for multi-server deployments of APF. Here, you can set up your global allow/block lists and have them all pull from a central location so that you can make a single update to the source and have the update propogated to all servers in your configuration. These changes are synced to the glob_allow/deny.rules files, and they will be downloaded (and overwritten) on a regular basis from your specified source, so don't make any manual edits in glob_allow/deny.rules.

As you can see, apf.conf is no joke. It has a lot of stuff going on, but it's very straightforward and documented well. Once we've set up apf.conf with the configurations we need, it's time to look at the more focused allow_hosts.rules and deny_hosts.rules files. These .rules files are where where you put your typical firewall rules in place. If there's one piece of advice I can give you about these configurations, it would be to check if your traffic is already allowed or blocked. Having multiple rules that do the same thing (possibly in different places) is confusing and potentially dangerous.

The deny_hosts.rules configuration will look just like allow_hosts.rules, but it's performing the opposite function. Let's check out an allow_hosts.rules configuration that will allow the Nimsoft service to function:

tcp:in:d=48000_48020:s=10.0.0.0/8
tcp:out:d=48000_48020:d=10.0.0.0/8

The format is somewhat simplistic, but the file gives a little more context in the comments:

# The trust rules can be made in advanced format with 4 options
# (proto:flow:port:ip);
# 1) protocol: [packet protocol tcp/udp]
# 2) flow in/out: [packet direction, inbound or outbound]
# 3) s/d=port: [packet source or destination port]
# 4) s/d=ip(/xx) [packet source or destination address, masking supported]
# Syntax:
# proto:flow:[s/d]=port:[s/d]=ip(/mask)

APF also uses ds_hosts.rules to load the DShield.org blocklist, and I assume the ecnshame_hosts.rules does something similar (can't find much information about it), so you won't need to edit these files manually. Additionally, you probably don't need to make any changes to log.rules, unless you want to make changes to what exactly you log. As it stands, it logs certain dropped connections, which should be enough. Also, it might be worth noting that this file is a script, not a configuration file.

The last two configuration files are the preroute.rules and postroute.rules that (unsurprisingly) are used to make routing changes. If you have been following my articles, this corresponds to the iptables chains for PREROUTING and POSTROUTING where you would do things like port forwarding and other advanced configuration that you probably don't want to do in most cases.

APF Command Line Management

As I mentioned in the "points to consider" at the top of this post, it's important to learn the changes you can perform from the command line, and APF has some very useful command line tools:

[root@server]# apf --help
APF version 9.7 <apf@r-fx.org>
Copyright (C) 2002-2011, R-fx Networks <proj@r-fx.org>
Copyright (C) 2011, Ryan MacDonald <ryan@r-fx.org>
This program may be freely redistributed under the terms of the GNU GPL
 
usage /usr/local/sbin/apf [OPTION]
-s|--start ......................... load all firewall rules
-r|--restart ....................... stop (flush) &amp; reload firewall rules
-f|--stop........ .................. stop (flush) all firewall rules
-l|--list .......................... list all firewall rules
-t|--status ........................ output firewall status log
-e|--refresh ....................... refresh &amp; resolve dns names in trust rules
-a HOST CMT|--allow HOST COMMENT ... add host (IP/FQDN) to allow_hosts.rules and
                                     immediately load new rule into firewall
-d HOST CMT|--deny HOST COMMENT .... add host (IP/FQDN) to deny_hosts.rules and
                                     immediately load new rule into firewall
-u|--remove HOST ................... remove host from [glob]*_hosts.rules
                                     and immediately remove rule from firewall
-o|--ovars ......................... output all configuration options

You can use these command line tools to turn your firewall on and off, add allowed or blocked hosts and display troubleshooting information. These commands are very easy to use, but if you want more fine-tuned control, you'll need to edit the configuration files directly (as we looked at above).

I know it seems like a lot of information, but to a large extent, that's all you need to know to get started with APF. Take each section slowly and understand what each configuration file is doing, and you'll master APF in no time at all.

-Mark

January 9, 2012

iptables Tips and Tricks – Troubleshooting Rulesets

One of the most time consuming tasks with iptables is troubleshooting a problematic ruleset. That will not change no matter how much experience you have with it. However, with the right mindset, this task becomes considerably easier.

If you missed my last installment about iptables rule processing, here's a crash course:

  1. The rules start at the top, and proceed down, one by one unless otherwise directed.
  2. A rule must match exactly.
  3. Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

There are essentially two things that you will be troubleshooting with iptables ... Either it's not accepting traffic and it should be OR it's accepting traffic and it shouldn't be. If the server is intermittently blocking or accepting traffic, that may take some additional troubleshooting, and it may not even be related to iptables.

Keep in mind what you are looking for, and don't jump to any conclusions. Troubleshooting iptables takes patience and time, and there shouldn't be any guesswork involved. If you have a configuration of 800 rules, you should expect to need to look through every single rule until you find the rule that is causing your problems.

Before you begin troubleshooting, you first need to know some information about the traffic:

  1. What is the source IP address or range that is having difficulty connecting?
  2. What is the destination IP address or website IP?
  3. What is the port or port range affected, or what type of traffic is it (TCP, ICMP, etc.)?
  4. Is it supposed to be accepted or blocked?

Those bits of information should be all you need to begin troubleshooting a buggy ruleset, except in some rare cases that are outside the scope of this article.

Here are some things to keep in mind (especially if you did not program every rule by hand):

  • iptables has three built in chains. These are for INPUT – the traffic coming in to the server, OUTPUT – the traffic coming out of the server, and FORWARD – traffic that is not destined to or coming from the server (usually only used when iptable is acting as a firewall for other servers). You will start your troubleshooting at the top of one of these three chains, depending on the type of traffic.
  • The "target" is the action that is taken when the rule matches. This may be another custom chain, so if you see a rule with another chain as the target that matches exactly, be sure to step through every rule in that chain as well. In the following example, you will see the BLACKLIST2 sub-chain that applies to traffic on port 80. If traffic comes through on port 80, it will be diverted to this other chain.
  • The RETURN target indicates that you should return to the parent chain. If you see a rule that matches with a RETURN target, stop all your troubleshooting on the current chain, and return the rule directly after the rule that referenced the custom chain.
  • If there are no matching rules, the chain policy is applied.
  • There may be rules in the "nat," "mangle" or "raw" tables that are blocking or diverting your traffic. Typically, all the rules will be in the "filter" table, but you might run into situations where this is not the case. Try running this to check: iptables -t mangle -nL ; iptables -t nat -nL ; iptables -t raw -nL
  • Be cognisant of the policy. If the policy is ACCEPT, all traffic that does not match a rule will be accepted. Conversely, if the policy is DROP or REJECT, all traffic that does not match a rule will be blocked.
  • My goal with this article is to introduce you to the algorithm by which you can troubleshoot a more complex ruleset. It is intentionally left simple, but you should still follow through even when the answer may be obvious.

Here is an example ruleset that I will be using for an example:

Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010

Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0

Here is the problem: Your server is accepting SSH traffic to anyone, and you wish to only allow SSH to your IP – 111.111.111.111. We know that this is inbound traffic, so this will affect the INPUT chain.

We are looking for:

source IP: any
destination IP: any
protocol: tcp
port: 22

Step 1: The first rule denotes any source IP and and destination IP on destination port 80. Since this is regarding port 22, this rule does not match, so we'll continue to the next rule. If the traffic here was on port 80, it would invoke the BLACKLIST2 sub chain.
Step 2: The second rule denotes any source IP and any destination IP on destination port 50. Since this is regarding port 22, this rule does not match, so let's continue on.
Step 3: The third rule denotes any source IP and any destination IP on destination port 53. Since this is regarding port 22, this rule does not match, so let's continue on.
Step 4: The fourth rule denotes any source IP and any destination IP on destination port 22. Since this is regarding port 22, this rule matches exactly. The target ACCEPT is applied to the traffic. We found the problem, and now we need to construct a solution. I will be showing you the Redhat method of doing this.

Do this to save the running ruleset as a file:

iptables-save > current-iptables-rules

Then edit the current-iptables-rules file in your favorite editor, and find the rule that looks like this:

-A INPUT -p tcp --dport 22 -j ACCEPT

Then you can modify this to only apply to your IP address (the source, or "-s", IP address).

-A INPUT -p tcp -s 111.111.111.111 --dport 22 -j ACCEPT

Once you have this line, you will need to load the iptables configuration from this file for testing.

iptables-restore < current-iptables-rules

Don't directly edit the /etc/sysconfig/iptables file as this might lock you out of your server. It is good practice to test a configuration before saving to the system configuration files. This way, if you do get locked out, you can reboot your server and it will be working. The ruleset should look like this now:

Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 111.111.111.111 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010

Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0

The policy of "DROP" will now block any other connection on port 22. Remember, the rule must match exactly, so the rule on port 22 now *ONLY* applies if the IP address is 111.111.111.111.

Once you have confirmed that the rule is behaving properly (be sure to test from another IP address to confirm that you are not able to connect), you can write the system configuration:

service iptables save

If this troubleshooting sounds boring and repetitive, you are right. However, this is the secret to solid iptables troubleshooting. As I said earlier, there is no guesswork involved. Just take it step by step, make sure the rule matches exactly, and follow it through till you find the rule that is causing the problem. This method may not be fast, but it's reliable. You'll look like an expert in no time.

-Mark

January 5, 2012

iptables Tips and Tricks - Rule Processing

As I mentioned in "iptables Tips and Tricks - Port Redirection," iptables is probably a complete mystery to a lot of users, and one the biggest hurdles is understanding the method by which it filters traffic ... Once you understand this, you'll be able to tame the beast.

When I think of iptables, the best analogy that comes to mind is a gravity coin sorting bank with four rules and one policy. If you're not familiar with a gravity coin sorting bank, each coin is starts at the same place and slides down an declined plane until it can fall into it's appropriate tube:

iptables Rule Sorter

As you can see, once a coin starts down the path, there are four rules – each one "filtering traffic" based on the width of the coin in millimeters (Quarter = 25mm, Nickel = 22mm, Penny = 20mm, Dime = 18mm). Due to possible inconsistencies in the coins, the tube widths are slightly larger than the official sizes of each coin to prevent jamming. At the end of the line, if a coin didn't fit in any of the tubes, it's dropped out of the sorter.

As we use this visualization to apply to iptables, there are three important things to remember:

  1. The rules start at the top, and proceed down, one by one unless otherwise directed.
  2. A rule must match exactly.
  3. Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

Let's jump back to the coin sorter. What would happen if you introduced a 23mm coin (slightly larger than a nickel)? What would happen if you introduced a 17mm coin (smaller than a dime)? What would happen if you dropped in a $1 coin @ 26.5mm?

In the first scenario, the coin would enter into the rule processing by being dropped in at the top. It would first pass by the dime slot, which requires a diameter of less than 18mm. It passes by the pennies slot as well, which requires less than 20mm. It continues past the nickels slot, which requires 22mm or less. It will then be "accepted" into the quarters slot, and there will be no further "processing" on the coin.

The iptables rules might look something like this:

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width

It's important to remember that once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it. In the second scenario (17mm coin), the coin would only be processed through the first rule; the other 3 rules would not be used even though the coin would meet their rules as well. Just because a port or and IP address is allowed somewhere in a chain, if a matching rule has dropped the packet, no further rules will be processed.

The final scenario (26.5mm coin) outlines a situation where none of the rules match, and this indicates that the policy will be used. In the coin bank example, it would be physically dropped off the side of the bank. iptables keeps a tally of the number of packets dropped and the corresponding size of the data. You can view this data by using the "iptables -vnL" command.

Chain OUTPUT (policy ACCEPT 3418K packets, 380M bytes)

cPanel even uses this tally functionality to track bandwidth usage (You may have seen the "acctboth" chain - this is used for tracking usage per IP).

So there you have it: iptables is just like a gravity coin sorting bank!

-Mark

December 26, 2011

iptables Tips and Tricks - Port Redirection

One of the most challenging and rewarding aspects of Linux administration is the iptables firewall. To the unenlightened, this can be a confusing black box that breaks your web server and blocks your favorite visitors from viewing your content at the most inconvenient times. This blog is the first in a series aimed at clarifying this otherwise mysterious force at work in your server.

Nothing compares with the frustration of trying to make a program listen on a different port – like if you wanted to configure your mail client to listen on port 2525. Many times, configuring a program the hard way (some would say the "correct" way) using configuration files may not be worth your time and effort ... Especially if the server is running on a control panel that does not natively support this functionality.

Fortunately, iptables offers an elegant solution:

iptables -t nat -A PREROUTING -p tcp --dport 2525 -j REDIRECT --to-ports 25

What this does:

  1. This specifies -t nat to indicate the nat table. Typically rules are added to the "filter" table (if you do not specify another table), and this is where the majority of the traffic is handled. In this case, however, we require the use of the nat table.
  2. This rules appends (-A), which means to add the rule at the bottom of the list.
  3. This rule is added to the PREROUTING chain.
  4. For the tcp protocol (-p tcp)
  5. The destination port (--dport) is 2525 - this is the port that the client is trying to access on your server.
  6. The traffic is jumped (-j) to the REDIRECT action. This is the action that is taken when the rule matches.
  7. The port is redirected to port 25 on the server.

As you can see, by changing the protocol to either tcp or udp or by adjusting the dport number and the to-ports number, you can redirect any port incoming to any listening port on the server. Just remember that the dport is the port the client machine is trying to connect to (the port they configure in the mail client, for example).

But check this out: Say for example you have a website (shocking, I know). You don't have a load balancer or a firewall set up, but you want to split off your email traffic to a second server to reduce strain on your web server. Essentially, you want to take incoming port 25 and redirect it ... to ANOTHER SERVER. With iptables, you can make this work:

iptables -t nat -A PREROUTING -p tcp -d 123.123.123.123 --dport 25 -j DNAT --to-destination 10.10.10.10:25

What this does:

  1. It specifies a destination (-d) IP address. This is not needed, but if you want to limit the email redirection to a single address, this is how you can do it.
  2. It is jumped to DNAT, which stands for destination nat.
  3. The destination and port are specified as arguments on to-destination

As you can see, this forwards all traffic on port 25 to an internal IP address.

Now, say you want to redirect from a different incoming port to a port on another server:

iptables -t nat -A PREROUTING -p tcp --dport 5001 -j DNAT --to-destination 10.10.10.10:25
iptables -t nat -A POSTROUTING -p tcp --dport 25 -j MASQUERADE

In this example, the incoming port is different, so we need to change it back to the standard port on the way back out through the primary server.

If you would like further reading on this topic, I recommend this great tutorial:
http://www.karlrupp.net/en/computer/nat_tutorial

Remember, when you are modifying your running configuration of iptables, you will still need to save your changes in order for it to persist on reboot. Be sure to test your configuration before saving it with "service iptables save" so that you don't lock yourself out.

-Mark

December 1, 2010

Every Cloud Has a Silver Lining

Last week, Netflix made headlines when the company announced that it was moving most of its operations to Amazon Web Services' Elastic Compute Cloud. The news was greeted with enthusiasm and was seen as further justification of the public cloud. Rightly so: the fact that Netflix generates up to 20% of US traffic during peak times, and that this traffic is moving to the public cloud would seem justification to me. This is a great piece of advertising for the cloud (much better than Microsoft's "to the cloud" campaign), and by proxy a great piece of advertising for SoftLayer.

So why did Netflix make the move? Economics - plain and simple. It is less expensive to move to the cloud than it is to continue supporting everything via internal Netflix DCs. In a cloud model, peak traffic loads dictate Netflix's economics - they pay for peaks, but only when they occur. When traffic drops off, Netflix enjoys the resultant cost savings, plus they relieve themselves of a considerable management burden. The argument is straightforward.

All of this has brought me back to consideration of the "private" cloud (which is arguably not the cloud at all, but I digress) and the value that it offers. The industry definition of a private cloud is a cloud implementation that is internal to (still owned and operated by) a single enterprise. SoftLayer defines a private cloud a little differently: SoftLayer remains the IAAS provider, but we ensure that a customer is on a single node (i.e. server). This conversation will stick with the industry definition.

So, what are the impacts of the private cloud across an enterprise?

In theory, a private cloud would give individual departments or discrete project teams the ability to better manage cost. As with a public cloud, a project team would be able to take advantage of the cost savings that come with paying only for what they use. However, this means that change is necessary in corporate accounting functions given that systems now need to be managed based on a "pay as you go model" versus a cost center model. This means a fundamental change in IT philosophy, as they now need to bill departments working on a variable use model - all of a sudden they have to think more like a business unit with a P&L to manage.

The cloud provides the ability to quickly spin up and scale. In the SoftLayer world, this translates to availability in anywhere from 2-4 hours. This should mean an increase in operational efficiency across departments using the cloud. Projects can start and end quickly without concern for lengthy implementation or teardown windows. That said I am not sure this increase in efficiency is meaningful when balanced against an IT department that must build and support a cloud infrastructure that has to account for the operation across the entire enterprise. The impact is potentially great at a micro level, but wanes when you consider the larger picture.

From a planning point of view, IT must now consider what a variable use model means in practical terms. Different departments will experience different peaks and valleys based on different workloads. In all likelihood, these peaks will not align on the calendar, nor will they be consistent month over month. In addition, my assumption is that deployment of a cloud will engender unanticipated usage patterns given the supposed cost and operational flexibility that the cloud delivers within the enterprise. The challenge will be to balance these needs against the delivery of a service that will adhere to QoS promises and associated internal service level agreements. ( And I think they will have to exist. If IT moves beyond a cost center, and internal organizations are trying to budget based upon forecasted compute use, it only makes sense that IT will be held up against external providers. Indeed, I would expect some rogue departments to go off the reservation to external providers based on cost considerations alone.)

My guess is that the IT response to planning will be predictable - over-engineer the private cloud to make sure that it is bullet proof. This might work, but it will be expensive and will paradoxically lead to underutilization of what is an over planned resource - something the cloud is supposed to mitigate. This approach is also likely to lead to IT bloat as capable internal resources are likely to be thin, driving a round of hiring to ensure expertise is on hand to manage the cloud.

In addition, I would assume that some applications will continue to be supported (for a variety of reasons - security, I/O challenges), thus adding more cost to the equation.

The arguments against the private cloud are numerous and ought to give the enterprise pause for thought. Regardless, I am willing to bet that private cloud implementations will accelerate in the enterprise. Many companies are supported by IT organizations that are strong and well entrenched within the corporate culture. Part of the fight will be based around losing budget, headcount and perceived power if the decision is made to go to an external IAAS provider. All of the usual rubrics will be used: security, quality of service, performance and on the list goes. In essence, these are the same arguments that have been made in the past when a decision to outsource anything has come to the fore. Does it make sense to outsource everything? The answer is a resounding no, but the argument for looking to the public cloud, or SoftLayer's private cloud is strong.

-@gkdog

October 8, 2010

From Zero to Ten in 10

Our second Dallas data center went live 10 days ago and we are already pushing 10 GB of sustainable traffic out the door. I have spent some time in the DC with some of our ops guys, and the place is impressive.

A terrific amount of computing power sits in row after row of server racks, driving a diverse array of business to more than 110 countries. Each rack features powerful processors, lots of RAM and heaps of storage. There is very little that our customers are unable to do over Softlayer’s infrastructure. And if they need more, SoftLayer can add additional servers very quickly to meet this demand. I wish the rest of our business were as simple as this.

Despite the state of the art infrastructure that sits in the DC, it remains a challenge to meet the needs of our customers. Why? Network, that’s why. SoftLayer’s challenge will be to continuously stay ahead of our customers’ demands, primarily in the network. If the network is unable to support the traffic that is pushed across our DC, everything comes tumbling down.

To a degree, we are victims of our own success. As we add servers to racks, we are placing increasing demand on the network. The more successful we are, the more pressure we place on the network.

Consider the following statistics:

  • When SoftLayer went live five years ago, we used two carriers and pushed 20 Gbps out the door.
  • Four years ago, this had gone up to four carriers and eight 10 Gbps links.
  • In January 2009 we pushed about 70 Gbps of sustained traffic. And this doubled for President Obama’s inauguration.
  • Today we use over ten carriers, with over 1000 Gbps of capacity.
  • In addition to the needs that our customers drive, we cannot forget to consider DDOS attacks as DDOS attacks add significant load to the network. We consistently absorb and successfully defend attacks of 5 Gbps, 10 Gbps or more and the peaks have grown by a factor of ten since SoftLayer went live.

The trend revealed is significant – in five years the amount of traffic sustained over our network has increased by more than ten times. And it shows little signs of slowing down.

Suffice to say, we spend a significant amount of time designing our networks to ensure that we are able to handle the traffic loads that are generated – we have to. Aggressively overbuilding the network brings us some short term pain, but if we are going to stay ahead of demand it is simply good business (and it makes sure our customers are happy). The new DC in Dallas is a great example of how we stay ahead of the game.

Each server has 5 NICs – 2 x 1 Gbps (bonded) for the public network, 2 x 1 Gbps (bonded) for the private network and one for management. The net of this is that customers can push 2 Gbps to the internet assuming server processors can handle the load.

-@quigleymar

September 9, 2010

Popularity, Are You Ready?

Instantly Becoming Popular, Are You Ready?

As an amateur filmmaker, I strive to get my footage seen by as many people as possible. Shameless plug time - http://vimeo.com/14159857 . Most of the time, it is only 1 or 2 people per week but when a guy sells $50,000 for $200 people want to watch! Have you clicked the link yet?. My normal traffic is usually handled by a virtual machine without issues. But what happens when you instantly become popular?

This happened to me this weekend. A newer video I posted last week became popular. It went from 85 plays on Friday, to 5,007 on Saturday, to 22,136 on Sunday. On Monday, I had an all time high of 62,397 plays on Monday. The average steamed video file size was 53MB. Some were bigger (actually viewing it in Full HD mode), and some were smaller (mobile versions that were streamed). Over these 4 days, 4.75TB of my video was transferred across the internet, with 70% of this done on Monday.

How do you survive such a burst in popularity? Good backend design, and having a provider which can scale quickly to offset these issues. Here at SoftLayer, we can spin up your custom cloud instance templates in just a few minutes to help relieve that sudden increase in traffic. Still not enough? How about using our CDN to move your content files closer to the end user? At SoftLayer, the sky is the limit on the possibilities of how we can help you survive instantly becoming popular.

Categories: 
September 15, 2009

Managing Your Traffic in the Modern Era

Over the past 10 years, I’ve run or helped run all sizes of web sites and internet applications. I’ve seen everything from single-page brochure web sites to horizontally scaled interactive portals. And what I’ve learned is that it is all about the end-user experience.

I’m not a graphics specialist or a GUI designer. I just don’t have that in my DNA. I focus more on the technical side of things working on better ways to deliver content to the user. And in the purely technical area, the best thing to do to improve the user experience is to improve the delivery speed to the user.

There are a lot of tools out there that can be used to speed up delivery. CDN, for example, is an awesome way to get static content to an end user and is very scalable. But what about scaling out the application itself?

Traditionally, a simple Layer-4 Load Balancer has been a staple component of scalable applications. This type of Load Balancing can provide capacity during traffic peaks as well as increase availability. The application runs on several servers and the load balancer uses some simple methods (least connections, round robin, etc) to distribute the load. For a lot of applications this is sufficient to get content reliably and quickly to the end user. SoftLayer offers a relatively inexpensive load-balancing service for our customers that can provide this functionality.

There is another, more sophisticated, tool that can be used to manage internet application traffic. That is the “Application Delivery Controller” (obligatory Wikipedia link: http://en.wikipedia.org/wiki/Application_Delivery_Controller) or “Load Balancer on Steroids”. This class of traffic manager can act in Layer-7, the data layer. These devices can make decisions based on the actual content of the data packets, not just the source and destination.

And an ADC can do more than load balance. It can act as a Web Application Firewall to protect your data. It can speed up your application using SSL Offloading, Content Caching, TCP Optimization, and more. This type of device is very smart and very configurable and will help in the delivering the application to the end user.

At SoftLayer we have seen our customers achieve a lot of success with our Layer-4 Load Balancer product. But we are always looking for other tools to help our customers. We always have admired the advanced functionality in the appliance-based Application Delivery Controllers on the market. Finding a way to get this enterprise-grade technology to our customers in an affordable manner was problematic. When Citrix announced that they were going to create a version of their NetScaler product that didn’t require an appliance we were thrilled. With the announcement of the NetScaler VPX we finally thought we had found the right product that we could use to affordably provision this advanced technology on-demand to our customers.

SoftLayer is VERY excited to partner with Citrix to provide the NetScaler VPX Application Delivery Controller to our customers. Our customers can order a NetScaler VPX, and in a matter of minutes be managing the delivery of their online applications using one of the most sophisticated tools on the market. Citrix does a better job of promoting the product than I do, so here is the link to their site: http://citrix.com/English/ps2/products/product.asp?contentID=21679&ntref=hp_nav_US.

Remember, it’s all about the experience of the user at the other end of the wire. Find the right tools to manage that experience and you are most of the way there. Oh yeah, and find a good graphics designer too. That helps. So does good content.

-@nday91

Subscribe to traffic