Posts Tagged 'Policy'

January 29, 2013

iptables Tips and Tricks: APF (Advanced Policy Firewall) Configuration

Let's talk about APF. APF — Advanced Policy Firewall — is a policy-based iptables firewall system that provides simple, powerful control over your day-to-day server security. It might seem intimidating to be faced with all of the features and configuration tools in APF, but this blog should put your fears to rest.

APF is an iptables wrapper that works alongside iptables and extends its functionality. I personally don't use iptables wrappers, but I have a lot of experience with them, and I've seen that they do offer some additional features that streamline policy management. For example, by employing APF, you'll get several simple on/off toggles (set via configuration files) that make some complex iptables configurations available without extensive coding requirements. The flip-side of a wrapper's simplicity is that you aren't directly in control of the iptables commands, so if something breaks it might take longer to diagnose and repair. Before you add a wrapper like APF, be sure that you know what you are getting into. Here are a few points to consider:

  • Make sure that what you're looking to use adds a feature you need but cannot easily incorporate with iptables on its own.
  • You need to know how to effectively enable and disable the iptables wrapper (the correct way ... read the manual!), and you should always have a trusted failsafe iptables ruleset handy in the unfortunate event that something goes horribly wrong and you need to disable the wrapper.
  • Learn about the basic configurations and rule changes you can apply via the command line. You'll need to understand the way your wrapper takes rules because it may differ from the way iptables handles rules.
  • You can't manually configure your iptables rules once you have your wrapper in place (or at least you shouldn't).
  • Be sure to know how to access your server via the IPMI management console so that if you completely lock yourself out beyond repair, you can get back in. You might even go so far as to have a script or set of instructions ready for tech support to run, in the event that you can't get in via the management console.

TL;DR: Have a Band-Aid ready!

APF Configuration

Now that you have been sufficiently advised about the potential challenges of using a wrapper (and you've got your Band-Aid ready), we can check out some of the useful APF rules that make iptables administration a lot easier. Most of the configuration for APF is in conf.apf. This file handles the default behavior, but not necessarily the specific blocking rules, and when we make any changes to the configuration, we'll need to restart the APF service for the changes to take effect.

Let's jump into conf.apf and break down what we see. The first code snippit is fairly self-explanatory. It's another way to make sure you don't lock yourself out of your server as you are making configuration changes and testing them:

# !!! Do not leave set to (1) !!!
# When set to enabled; 5 minute cronjob is set to stop the firewall. Set
# this off (0) when firewall is determined to be operating as desired.
DEVEL_MODE="1"

The next configuration options we'll look at are where you can make quick high-level changes if you find that legitimate traffic is being blocked and you want to make APF a little more lenient:

# This controls the amount of violation hits an address must have before it
# is blocked. It is a good idea to keep this very low to prevent evasive
# measures. The default is 0 or 1, meaning instant block on first violation.
RAB_HITCOUNT="1"
 
# This is the amount of time (in seconds) that an address gets blocked for if
# a violation is triggered, the default is 300s (5 minutes).
RAB_TIMER="300"
# This allows RAB to 'trip' the block timer back to 0 seconds if an address
# attempts ANY subsiquent communication while still on the inital block period.
RAB_TRIP="1"
 
# This controls if the firewall should log all violation hits from an address.
# The use of LOG_DROP variable set to 1 will override this to force logging.
RAB_LOG_HIT="1"
 
# This controls if the firewall should log all subsiqent traffic from an address
# that is already blocked for a violation hit, this can generate allot of logs.
# The use of LOG_DROP variable set to 1 will override this to force logging.
RAB_LOG_TRIP="0"

Next, we have an option to adjust ICMP flood protection. This protection should be useful against some forms of DoS attacks, and the associated rules show up in your INPUT chain:

# Set a reasonable packet/time ratio for ICMP packets, exceeding this flow
# will result in dropped ICMP packets. Supported values are in the form of:
# pkt/s (packets/seconds), pkt/m (packets/minutes)
# Set value to 0 for unlimited, anything above is enabled.
ICMP_LIM="30/s"

If you wanted to add more ports to block for p2p traffic (which will show up in the P2P chain), you'll update this code:

# A common set of known Peer-To-Peer (p2p) protocol ports that are often
# considered undesirable traffic on public Internet servers. These ports
# are also often abused on web hosting servers where clients upload p2p
# client agents for the purpose of distributing or downloading pirated media.
# Format is comma separated for single ports and an underscore separator for
# ranges (4660_4678).
BLK_P2P_PORTS="1214,2323,4660_4678,6257,6699,6346,6347,6881_6889,6346,7778"

The next few lines let you designate the ports that you want to have closed at all times. They will be blocked for INPUT and OUTPUT chains:

# These are common Internet service ports that are understood in the wild
# services you would not want logged under normal circumstances. All ports
# that are defined here will be implicitly dropped with no logging for
# TCP/UDP traffic inbound or outbound. Format is comma separated for single
# ports and an underscore separator for ranges (135_139).
BLK_PORTS="135_139,111,513,520,445,1433,1434,1234,1524,3127"

The next important section to look at deals with conntrack. If you get "conntrack full" errors, this is where you'd increase the allowed connections. It's not uncommon to need more connections than the default, so if you need to adjust that value, you'd do it here:

# This is the maximum number of "sessions" (connection tracking entries) that
# can be handled simultaneously by the firewall in kernel memory. Increasing
# this value too high will simply waste memory - setting it too low may result
# in some or all connections being refused, in particular during denial of
# service attacks.
SYSCTL_CONNTRACK="65536"

We've talked about the ports we want closed at all times, so it only makes sense that we'd specify which ports we want open for all interfaces:

# Common inbound (ingress) TCP ports
IG_TCP_CPORTS="22"
# Common inbound (ingress) UDP ports
IG_UDP_CPORTS=""
# Common outbound (egress) TCP ports
EG_TCP_CPORTS="21,25,80,443,43"
# Common outbound (egress) UDP ports
EG_UDP_CPORTS="20,21,53"

And when we want a special port allowance for specific users, we can declare it easily. For example, if we want port 22 open for user ID 0, we'd use this code:

# Allow outbound access to destination port 22 for uid 0
EG_TCP_UID="0:22"

The next few sections on Remote Rule Imports and Global Trust are a little more specialized, and I encourage you to read a little more about them (since there's so much to them and not enough space to cover them here on the blog). An important feature of APF is that it imports block lists from outside sources to keep you safe from some attackers, so the Remote Rule Imports can prove to be very useful. The Global Trust section is incredibly useful for multi-server deployments of APF. Here, you can set up your global allow/block lists and have them all pull from a central location so that you can make a single update to the source and have the update propogated to all servers in your configuration. These changes are synced to the glob_allow/deny.rules files, and they will be downloaded (and overwritten) on a regular basis from your specified source, so don't make any manual edits in glob_allow/deny.rules.

As you can see, apf.conf is no joke. It has a lot of stuff going on, but it's very straightforward and documented well. Once we've set up apf.conf with the configurations we need, it's time to look at the more focused allow_hosts.rules and deny_hosts.rules files. These .rules files are where where you put your typical firewall rules in place. If there's one piece of advice I can give you about these configurations, it would be to check if your traffic is already allowed or blocked. Having multiple rules that do the same thing (possibly in different places) is confusing and potentially dangerous.

The deny_hosts.rules configuration will look just like allow_hosts.rules, but it's performing the opposite function. Let's check out an allow_hosts.rules configuration that will allow the Nimsoft service to function:

tcp:in:d=48000_48020:s=10.0.0.0/8
tcp:out:d=48000_48020:d=10.0.0.0/8

The format is somewhat simplistic, but the file gives a little more context in the comments:

# The trust rules can be made in advanced format with 4 options
# (proto:flow:port:ip);
# 1) protocol: [packet protocol tcp/udp]
# 2) flow in/out: [packet direction, inbound or outbound]
# 3) s/d=port: [packet source or destination port]
# 4) s/d=ip(/xx) [packet source or destination address, masking supported]
# Syntax:
# proto:flow:[s/d]=port:[s/d]=ip(/mask)

APF also uses ds_hosts.rules to load the DShield.org blocklist, and I assume the ecnshame_hosts.rules does something similar (can't find much information about it), so you won't need to edit these files manually. Additionally, you probably don't need to make any changes to log.rules, unless you want to make changes to what exactly you log. As it stands, it logs certain dropped connections, which should be enough. Also, it might be worth noting that this file is a script, not a configuration file.

The last two configuration files are the preroute.rules and postroute.rules that (unsurprisingly) are used to make routing changes. If you have been following my articles, this corresponds to the iptables chains for PREROUTING and POSTROUTING where you would do things like port forwarding and other advanced configuration that you probably don't want to do in most cases.

APF Command Line Management

As I mentioned in the "points to consider" at the top of this post, it's important to learn the changes you can perform from the command line, and APF has some very useful command line tools:

[root@server]# apf --help
APF version 9.7 <apf@r-fx.org>
Copyright (C) 2002-2011, R-fx Networks <proj@r-fx.org>
Copyright (C) 2011, Ryan MacDonald <ryan@r-fx.org>
This program may be freely redistributed under the terms of the GNU GPL
 
usage /usr/local/sbin/apf [OPTION]
-s|--start ......................... load all firewall rules
-r|--restart ....................... stop (flush) &amp; reload firewall rules
-f|--stop........ .................. stop (flush) all firewall rules
-l|--list .......................... list all firewall rules
-t|--status ........................ output firewall status log
-e|--refresh ....................... refresh &amp; resolve dns names in trust rules
-a HOST CMT|--allow HOST COMMENT ... add host (IP/FQDN) to allow_hosts.rules and
                                     immediately load new rule into firewall
-d HOST CMT|--deny HOST COMMENT .... add host (IP/FQDN) to deny_hosts.rules and
                                     immediately load new rule into firewall
-u|--remove HOST ................... remove host from [glob]*_hosts.rules
                                     and immediately remove rule from firewall
-o|--ovars ......................... output all configuration options

You can use these command line tools to turn your firewall on and off, add allowed or blocked hosts and display troubleshooting information. These commands are very easy to use, but if you want more fine-tuned control, you'll need to edit the configuration files directly (as we looked at above).

I know it seems like a lot of information, but to a large extent, that's all you need to know to get started with APF. Take each section slowly and understand what each configuration file is doing, and you'll master APF in no time at all.

-Mark

September 17, 2012

Joining the Internet Infrastructure Coalition

In January, we posted a series of blogs about legislation in the U.S. House of Representatives and Senate that would have had a serious impact on the hosting industry. We talked about SOPA and PIPA, and how those proposed laws would "break the Internet" as we know it. The hosting industry rallied together to oppose the passage of those bills, and in doing so, we proved to be a powerful collective force.

In the months that followed the shelving of SOPA and PIPA, many of the hosting companies that were active in the fight were invited to join a new coalition that would focus on proposed legislation that affects Internet infrastructure providers ... The Internet Infrastructure Coalition (or "i2Coalition") was born. i2Coalition co-founder and Board Chair Christian Dawson explains the basics:

SoftLayer is proud to be a Charter Member of i2Coalition, and we're excited to see how many vendors, partners, peers and competitors have joined us. Scrolling the ranks of founding members is a veritable "Who's who?" of the companies that make up the "nuts and bolts" of the Internet.

The goal of i2Coalition is to facilitate public policy education and advocacy, develop market-driven standards formed by consensus and give the industry a unified voice. On the i2Coalition's Public Policy page, that larger goal is broken down into focused priorities, with the first being

"In all public policy initiatives of the i2Coalition will be to encourage the growth and development of the Internet infrastructure industry and to protect the interests of members of the Coalition consistent with this development."

Another huge priority worth noting is the focus on enabling and promoting the free exercise of human rights — including freedom of speech, freedom of assembly and the protection of personal privacy. Those rights are essential to fostering effective Internet advancement and to maintain a free and open Internet, and SoftLayer is a strong supporter of that platform.

If you operate in the hosting or Internet infrastructure space and you want to be part of the i2Coalition, we encourage you to become a member and join the conversation. When policymakers are talking about getting "an Internet" from their staff members, we know that there are plenty of opportunities to educate and provide context on the technical requirements and challenges that would result from proposed legislation, and the Internet Infrastructure Coalition is well equipped to capitalize on those opportunities.

-@toddmitchell

May 25, 2012

Tear Down the (Immigration) Wall ... Or at Least Install a Door

A few years ago, I went through a nightmare trying to get to permanent resident status in the United States. My file sat in a box for over a year, was lost, re-submitted and FINALLY rushed through by Ted Kennedy's office. And I was on a "fast track" due to a long record of published research and employment history. I had the means to pay lawyers and the time to repeat the filing and wait for a decision. If I didn't have the means or the time to wait for the process to complete, I don't know where I'd be, but in all likelihood, it wouldn't be here. It's no surprise that immigration reform is high on my list of priorities, and given SoftLayer's involvement in the USCIS Entrepreneurs in Residence program along with Lance's appointment to a Bloomberg committee focused on immigration reform, it's clear I'm not alone.

The bi-partisan Partnership for a New American Economy recently published a very interesting report — Not Coming to America: Why the US is Falling Behind in the Global Race for Talent — that speaks to a lot of the challenges plaguing the current US immigration policy. Because of those challenges, "the future of America's position as the global magnet for the world's most talented and hardest-working is in jeopardy." Here are a few of the projected economic realities of not reforming immigration laws to keep up with other countries:

SHORTAGE OF WORKERS IN INNOVATION INDUSTRIES: Jobs in science, technology, engineering, and math ("STEM" fields) are increasing three times faster than jobs in the rest of the economy, but American students are not entering these innovative fields in sufficient numbers. As a result, by 2018, we face a projected shortfall of 230,000 qualified advanced-degree STEM workers.

SHORTAGE OF YOUNG WORKERS: The US population is aging, baby boomers are retiring en masse, and the growth in the US labor force has slowed to historic lows of less than 1 percent. We cannot continue to produce the GDP growth the nation has come to expect without dramatic increases in productivity or welcoming more working age immigrants.

A STALLED ECONOMY: The US has faced years of stunted economic growth. History shows that new businesses are the biggest drivers of job creation, yet the most recent US Census data show that the number of business startups has hit a record low.

This concern isn't unique to the United States. With a global focus on innovation and technology, countries around the world are actively competing for the best and the brightest. In Canada, a report a few weeks ago spoke to Canada's need to double in size in the next few decades or risk losing relevance and becoming just another resource-rich colony. The nation's response? It's ready to open its doors to more immigrants.

The same applies to the United States ... It just may take longer.

Go back to how this country was built, and apply that to today. The biggest difference: The "skilled trades" we talk about in the most general sense are no longer carpenters like my grandfather but highly educated programmers, engineers and researchers. The idea isn't to replace the programmers, engineers and researchers in the US, rather it's to meet the existing unmet needs for programmers, engineers and researchers.

In all of SoftLayer's efforts to affect change in the US immigration policy, we have to make clear that our goal is not to drop the walls simply to add more permanent residents. It's about lowering many of the current artificial barriers that might prevent the next Fortune 500 founder from starting his or her business in the United States. If you don't think that's a serious concern, I'd point to a pretty surprising stat in the "Not Coming to America" report: "Today, more than 40 percent of America's Fortune 500 companies were founded by an immigrant or a child of an immigrant."

Immigration drives the economy. It's not a drain on the economy. Every country needs more smart people because smart people create new ideas, new ideas become new businesses, and new businesses create new jobs.

Because this is a politically charged issue, it's one I know many people don't necessarily agree with. Along with immigration, we have to look at how the education system can empower young people like my son to become the programmers, engineers and researchers that the US will need, and we have to be intentional about not simply adding permanent residents for the sake of adding permanent residents. If you have any thoughts one way or the other, I'd encourage you to share them with us here in a blog comment or link us to any of the resources you've found interesting in researching and discussing the topic.

-@gkdog

March 5, 2012

iptables Tips and Tricks - Not Locking Yourself Out

The iptables tool is one of the simplest, most powerful tools you can use to protect your server. We've covered port redirection, rule processing and troubleshooting in previous installments to this "Tips and Tricks" series, but what happens when iptables turns against you and locks you out of your own system?

Getting locked out of a production server can cost both time and money, so it's worth your time to avoid this. If you follow the correct procedures, you can safeguard yourself from being firewalled off of your server. Here are seven helpful tips to help you keep your sanity and prevent you from locking yourself out.

Tip 1: Keep a safe ruleset handy.

If you are starting with a working ruleset, or even if you are trying to troubleshoot an existing ruleset, take a backup of your iptables configuration before you ever start working on it.

iptables-save > /root/iptables-safe

Then if you do something that prevents your website from working, you can quickly restore it.

iptables-restore

Tip 2: Create a cron script that will reload to your safe ruleset every minute during testing.

This was pointed out to my by a friend who swears by this method. Just write a quick bash script and set a cron entry that will reload it back to the safe set every minute. You'll have to test quickly, but it will keep you from getting locked out.

Tip 3: Have the IPMI KVM ready.

SoftLayer-pod servers* are equipped with some sort of remote access device. Most of them have a KVM console. You will want to have your VPN connection set up, connected and the KVM window up. You can't paste to and from the KVM, so SSH is typically easier to work with, but it will definitely cut down on the downtime if something does go wrong.

*This may not apply to servers that were originally provisioned under another company name.

Tip 4: Try to avoid generic rules.

The more criteria you specify in the rule, the less chance you will have of locking yourself out. I would liken this to a pie. A specific rule is a very thin slice of the pie.

iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -d 123.123.123.123 -j DROP

But if you block port 22 from any to any, it's a very large slice.

iptables -A INPUT -p tcp --dport 22 -j DROP

There are plenty of ways that you can be more specific. For example, using "-i eth0" will limit the processing to a single NIC in your server. This way, it will not apply the rule to eth1.

Tip 5: Whitelist your IP address at the top of your ruleset.

This may make testing more difficult unless you have a secondary offsite test server, but this is a very effective method of not getting locked out.

iptables -I INPUT -s <your IP> -j ACCEPT

You need to put this as the FIRST rule in order for it to work properly ("-I" inserts it as the first rule, whereas "-A" appends it to the end of the list).

Tip 6: Know and understand all of the rules in your current configuration.

Not making the mistake in the first place is half the battle. If you understand the inner workings behind your iptables ruleset, it will make your life easier. Draw a flow chart if you must.

Tip 7: Understand the way that iptables processes rules.

Remember, the rules start at the top of the chain and go down, unless specified otherwise. Crack open the iptables man page and learn about the options you are using.

-Mark

January 9, 2012

iptables Tips and Tricks – Troubleshooting Rulesets

One of the most time consuming tasks with iptables is troubleshooting a problematic ruleset. That will not change no matter how much experience you have with it. However, with the right mindset, this task becomes considerably easier.

If you missed my last installment about iptables rule processing, here's a crash course:

  1. The rules start at the top, and proceed down, one by one unless otherwise directed.
  2. A rule must match exactly.
  3. Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

There are essentially two things that you will be troubleshooting with iptables ... Either it's not accepting traffic and it should be OR it's accepting traffic and it shouldn't be. If the server is intermittently blocking or accepting traffic, that may take some additional troubleshooting, and it may not even be related to iptables.

Keep in mind what you are looking for, and don't jump to any conclusions. Troubleshooting iptables takes patience and time, and there shouldn't be any guesswork involved. If you have a configuration of 800 rules, you should expect to need to look through every single rule until you find the rule that is causing your problems.

Before you begin troubleshooting, you first need to know some information about the traffic:

  1. What is the source IP address or range that is having difficulty connecting?
  2. What is the destination IP address or website IP?
  3. What is the port or port range affected, or what type of traffic is it (TCP, ICMP, etc.)?
  4. Is it supposed to be accepted or blocked?

Those bits of information should be all you need to begin troubleshooting a buggy ruleset, except in some rare cases that are outside the scope of this article.

Here are some things to keep in mind (especially if you did not program every rule by hand):

  • iptables has three built in chains. These are for INPUT – the traffic coming in to the server, OUTPUT – the traffic coming out of the server, and FORWARD – traffic that is not destined to or coming from the server (usually only used when iptable is acting as a firewall for other servers). You will start your troubleshooting at the top of one of these three chains, depending on the type of traffic.
  • The "target" is the action that is taken when the rule matches. This may be another custom chain, so if you see a rule with another chain as the target that matches exactly, be sure to step through every rule in that chain as well. In the following example, you will see the BLACKLIST2 sub-chain that applies to traffic on port 80. If traffic comes through on port 80, it will be diverted to this other chain.
  • The RETURN target indicates that you should return to the parent chain. If you see a rule that matches with a RETURN target, stop all your troubleshooting on the current chain, and return the rule directly after the rule that referenced the custom chain.
  • If there are no matching rules, the chain policy is applied.
  • There may be rules in the "nat," "mangle" or "raw" tables that are blocking or diverting your traffic. Typically, all the rules will be in the "filter" table, but you might run into situations where this is not the case. Try running this to check: iptables -t mangle -nL ; iptables -t nat -nL ; iptables -t raw -nL
  • Be cognisant of the policy. If the policy is ACCEPT, all traffic that does not match a rule will be accepted. Conversely, if the policy is DROP or REJECT, all traffic that does not match a rule will be blocked.
  • My goal with this article is to introduce you to the algorithm by which you can troubleshoot a more complex ruleset. It is intentionally left simple, but you should still follow through even when the answer may be obvious.

Here is an example ruleset that I will be using for an example:

Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010

Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0

Here is the problem: Your server is accepting SSH traffic to anyone, and you wish to only allow SSH to your IP – 111.111.111.111. We know that this is inbound traffic, so this will affect the INPUT chain.

We are looking for:

source IP: any
destination IP: any
protocol: tcp
port: 22

Step 1: The first rule denotes any source IP and and destination IP on destination port 80. Since this is regarding port 22, this rule does not match, so we'll continue to the next rule. If the traffic here was on port 80, it would invoke the BLACKLIST2 sub chain.
Step 2: The second rule denotes any source IP and any destination IP on destination port 50. Since this is regarding port 22, this rule does not match, so let's continue on.
Step 3: The third rule denotes any source IP and any destination IP on destination port 53. Since this is regarding port 22, this rule does not match, so let's continue on.
Step 4: The fourth rule denotes any source IP and any destination IP on destination port 22. Since this is regarding port 22, this rule matches exactly. The target ACCEPT is applied to the traffic. We found the problem, and now we need to construct a solution. I will be showing you the Redhat method of doing this.

Do this to save the running ruleset as a file:

iptables-save > current-iptables-rules

Then edit the current-iptables-rules file in your favorite editor, and find the rule that looks like this:

-A INPUT -p tcp --dport 22 -j ACCEPT

Then you can modify this to only apply to your IP address (the source, or "-s", IP address).

-A INPUT -p tcp -s 111.111.111.111 --dport 22 -j ACCEPT

Once you have this line, you will need to load the iptables configuration from this file for testing.

iptables-restore < current-iptables-rules

Don't directly edit the /etc/sysconfig/iptables file as this might lock you out of your server. It is good practice to test a configuration before saving to the system configuration files. This way, if you do get locked out, you can reboot your server and it will be working. The ruleset should look like this now:

Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 111.111.111.111 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010

Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0

The policy of "DROP" will now block any other connection on port 22. Remember, the rule must match exactly, so the rule on port 22 now *ONLY* applies if the IP address is 111.111.111.111.

Once you have confirmed that the rule is behaving properly (be sure to test from another IP address to confirm that you are not able to connect), you can write the system configuration:

service iptables save

If this troubleshooting sounds boring and repetitive, you are right. However, this is the secret to solid iptables troubleshooting. As I said earlier, there is no guesswork involved. Just take it step by step, make sure the rule matches exactly, and follow it through till you find the rule that is causing the problem. This method may not be fast, but it's reliable. You'll look like an expert in no time.

-Mark

January 5, 2012

iptables Tips and Tricks - Rule Processing

As I mentioned in "iptables Tips and Tricks - Port Redirection," iptables is probably a complete mystery to a lot of users, and one the biggest hurdles is understanding the method by which it filters traffic ... Once you understand this, you'll be able to tame the beast.

When I think of iptables, the best analogy that comes to mind is a gravity coin sorting bank with four rules and one policy. If you're not familiar with a gravity coin sorting bank, each coin is starts at the same place and slides down an declined plane until it can fall into it's appropriate tube:

iptables Rule Sorter

As you can see, once a coin starts down the path, there are four rules – each one "filtering traffic" based on the width of the coin in millimeters (Quarter = 25mm, Nickel = 22mm, Penny = 20mm, Dime = 18mm). Due to possible inconsistencies in the coins, the tube widths are slightly larger than the official sizes of each coin to prevent jamming. At the end of the line, if a coin didn't fit in any of the tubes, it's dropped out of the sorter.

As we use this visualization to apply to iptables, there are three important things to remember:

  1. The rules start at the top, and proceed down, one by one unless otherwise directed.
  2. A rule must match exactly.
  3. Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

Let's jump back to the coin sorter. What would happen if you introduced a 23mm coin (slightly larger than a nickel)? What would happen if you introduced a 17mm coin (smaller than a dime)? What would happen if you dropped in a $1 coin @ 26.5mm?

In the first scenario, the coin would enter into the rule processing by being dropped in at the top. It would first pass by the dime slot, which requires a diameter of less than 18mm. It passes by the pennies slot as well, which requires less than 20mm. It continues past the nickels slot, which requires 22mm or less. It will then be "accepted" into the quarters slot, and there will be no further "processing" on the coin.

The iptables rules might look something like this:

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width ACCEPT all --- 0.0.0.0/00.0.0.0/0width

It's important to remember that once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it. In the second scenario (17mm coin), the coin would only be processed through the first rule; the other 3 rules would not be used even though the coin would meet their rules as well. Just because a port or and IP address is allowed somewhere in a chain, if a matching rule has dropped the packet, no further rules will be processed.

The final scenario (26.5mm coin) outlines a situation where none of the rules match, and this indicates that the policy will be used. In the coin bank example, it would be physically dropped off the side of the bank. iptables keeps a tally of the number of packets dropped and the corresponding size of the data. You can view this data by using the "iptables -vnL" command.

Chain OUTPUT (policy ACCEPT 3418K packets, 380M bytes)

cPanel even uses this tally functionality to track bandwidth usage (You may have seen the "acctboth" chain - this is used for tracking usage per IP).

So there you have it: iptables is just like a gravity coin sorting bank!

-Mark

May 26, 2011

IPv4 Justification Changes in IMS

When IANA ran out of IPv4 blocks in February, the belts tightened yet another notch in the IPv4 world. Regional registries such as ARIN have changed how they are allocating additional IPv4 blocks to ISPs, including performing more stringent reviews to ensure efficient utilization of current allocations and reducing future allocation sizes to more fairly distribute the remaining addresses. In turn, ISPs such as SoftLayer have to more closely monitor how we assign our IPv4 space to ensure efficient utilization by our customers. In order to maximize this limited resource, we have to make changes to the way customers order and receive additional IPv4 IPs.

Being down the line from ARIN, SoftLayer's networking department has to ensure we abide by all updated ARIN policies when we issue IPv4 addresses from our available pool in order to remain eligible to receive additional resource allocations. In addition to policy compliance, we are also focused on IPv4 conservation methods such as those referenced in RFC2050. Accordingly, we've made improvements to the way that we handle IPv4 requests to better streamline the process of collecting and reviewing usage and justification details.

Every SoftLayer server comes with one public IPv4 address, and until recently was assigned 4 additional IPs (/30) statically routed to that server. Our first step of IP conservation took place earlier this year, when we ceased the practice of automatically assigning the 4 additional IPs, and only issued them by customer request.

As we move forward, were trying to be as transparent with our customers about the IPv4 justification process as we can, so we're letting you know that additional justification requirements have been imposed on all ISPs by ARIN, and the best way we can meet those requirements is to have our customers follow the same guidelines. Being SoftLayer, we're doing what we can to automate and streamline the IPv4 justification process where possible, and are therefore implementing changes in the ordering system. Beginning on June 1, you will now be presented with a brief questionnaire whenever requesting additional IPv4 addresses. We must collect the requester's contact details, number of IPs that are expected to be used immediately and within the next year, as well as a brief description of how the IPs will be used. What happens next depends on the information you have provided as well as the current IP usage on your account: either the request is automatically approved and fulfilled, or a ticket is spawned for additional manual review by one of the folks in our networking team.

When a ticket is spawned for our network folks, we will get back to you within one business day, and let you know whether we need additional details from you in order to consider the request. These manual reviews will take additional time, depending on the size and complexity of request, as well as the quality of information provided. Please be prepared that there will be some cases where we will not be able to approve a request, such as when name-based virtualization can be used for IP conservation, or when services such as SEO, email campaigns, or VPN termination are the intended use for the additional IPv4 addresses.

If you're familiar with SoftLayer's previous method of ordering additional IPv4 addresses, you might feel like we're making you jump through hoops. These "hoops" are not intended to make the process arbitrarily more difficult. Rather, they're being put in place specifically to make sure we're in the best position possible to meet customer demands with our current IPv4 allocations, and get additional allocations from ARIN before they are completely depleted.

The ISPs that have no problems giving away IPv4 addresses right now without more stringent review are also going to be the providers that have the hardest time getting additional IPv4 blocks from ARIN, and therefore will have a hard time fulfilling future customer needs. At SoftLayer we are going by the philosophy that the more cautiously we approach IPv4 depletion by justifying our customers' needs for each IPv4 address, the longer our IPv4 pool will last to meet those needs.

This is a good opportunity to mention IPv6 as the long-term solution for IPv4 address depletion. The sooner that the public Internet becomes fully IPv6-enabled, the better off the resource constraint will become. The SoftLayer network is enabled for IPv6, and you may place an order for 18.4 quintillion free IPv6 addresses for your server via our web portal today!

-Dani

August 6, 2009

Punishing Success

Let’s say you worked for years to become a world class athlete. As a kid, you were in the gym while other athletes were at the movies. You were in the weight room on Saturday nights when no one else was there. You shunned pizza and soda in favor of grilled fish and fresh fruit. By the time Letterman hit the evening airwaves, you were well into restorative sleep. You were out the door for your morning runs while other athletes snoozed. As a result of all this, now you perform at an elite level and are very successful at your sport. Suddenly, you find that there are people who have a vested interest in helping you maximize your athletic potential. Your coaches, your managers, and companies who pay you to endorse their products all want to see you do your best. Why? Because doing your best helps them be more successful.

So, they provide you with all the things you need to maximize your potential. You get the best training gear and training regimens. You get the best nutrition. You get the right amount of rest. All these things help you maximize your potential. Thus the relationship is a nice symbiotic cycle – the more success you experience, the more success your coaches, managers, and endorsement companies experience. Win-win. Makes sense, right?

So, imagine the silliness if your coaches, managers, etc., made the decision that because you were so fortunate in your success that you had to “give back” almost half your resources to train the athletes who loafed, stayed out late, partied and gorged on pizza. Because you’re such a hard-working and smart athlete, you don’t need all those resources to participate adequately in your sport, they rationalize. Consequently, you don’t hit your potential, your coaches and managers don’t distinguish themselves, and endorsing companies don’t call you. You then feel that you’ve been punished for your hard work and success.

Sadly, much of our government policy falls under this flawed logic. The IRS just released their latest income tax stats for the year 2007. For that year the top 1% of earners paid 40.4% of all income taxes collected. We all know that right now we’re coming out of a recession and we really folks to invest in businesses and hire people to get the economy moving. So how do the 2007 numbers compare to, say, the 1980’s? During the ‘80’s, we managed to shake off the “stagflation” of the ‘70’s and get the economy rolling again. It was during this time that many technology juggernaut companies were spawned – Microsoft being a good example. So, how much of the income taxes in the ‘80’s were paid by the top 1% of earners? The average for the 10 years from 1980-1989 was 22.2%.

Let’s do some quick math. $1.116 trillion in income taxes was collected in 2007. Of that, $455.3 billion was paid by the top 1% of earners. If they paid 22.2% as in the ‘80’s, they would have paid $247.8 billion in taxes, and right now we’d have $207.5 billion MORE dollars invested in our economy. That would be quite a stimulus package! Our current policy punishes success and chokes off fuel from our economic engines while we’re trying to climb out of a worse recession than we had in the ‘70’s. Not smart.

Some may think that this would simply mean that our government deficit would be $207.5 billion higher. This is not the case at all. These folks that make up that top 1% didn’t get there by being lazy or not putting their money to work. I know some folks in that group, and they WANT to put their money to work! I know one gentleman who had to be told some legal docs for a deal could not be prepared over the weekend because Christmas was on that weekend. These folks are like the world class athlete I mentioned above – by and large they’re disciplined and hard-working. Their money will build new businesses and create more jobs, and the government will collect far more revenue from this new economic activity than it would give up in collections from these top 1% folks. Think about it – how many of us have ever been hired by a “poor” person? Instead of punishing economic success, we should encourage it!

Bottom line, if government policy were to make sense, it would encourage these folks to maximize their economic potential and find the correct balance of revenue to collect and yet still promote economic growth. What would we prefer? That the government collects 50% of $1 trillion or 30% of $2 trillion? Hint: 30% of $2 trillion is a WAY better deal.

At SoftLayer, we think very differently about things. We simply do not punish our customers for succeeding. We empower them to be more successful – why? Because if our customers succeed, we succeed. We get this.

Can we prove this? Perhaps a look at how customers vote with their feet is an indicator. For the past few months, SoftLayer has seen the lowest percentage of customers terminating business with us in our history. If we punished our customers for their success, they would go elsewhere.

Subscribe to policy