startup-series

April 16, 2013

iptables Tips and Tricks - Track Bandwidth with iptables

As I mentioned in my last post about CSF configuration in iptables, I'm working on a follow-up post about integrating CSF into cPanel, but I thought I'd inject a simple iptables use-case for bandwidth tracking. You probably think about iptables in terms of firewalls and security, but it also includes a great diagnostic tool for counting bandwidth for individual rules or set of rules. If you can block it, you can track it!

The best part about using iptables to track bandwidth is that the tracking is enabled by default. To see this feature in action, add the "-v" into the command:

[root@server ~]$ iptables -vnL
Chain INPUT (policy ACCEPT 2495 packets, 104K bytes)

The output includes counters for both the policies and the rules. To track the rules, you can create a new chain for tracking bandwidth:

[root@server ~]$ iptables -N tracking
[root@server ~]$ iptables -vnL
...
Chain tracking (0 references)
 pkts bytes target prot opt in out source           destination

Then you need to set up new rules to match the traffic that you wish to track. In this scenario, let's look at inbound http traffic on port 80:

[root@server ~]$ iptables -I INPUT -p tcp --dport 80 -j tracking
[root@server ~]$ iptables -vnL
Chain INPUT (policy ACCEPT 35111 packets, 1490K bytes)
 pkts bytes target prot opt in out source           destination
    0   0 tracking    tcp  --  *  *   0.0.0.0/0        0.0.0.0/0       tcp dpt:80

Now let's generate some traffic and check it again:

[root@server ~]$ iptables -vnL
Chain INPUT (policy ACCEPT 35216 packets, 1500K bytes)
 pkts bytes target prot opt in out source           destination
  101  9013 tracking    tcp  --  *  *   0.0.0.0/0        0.0.0.0/0       tcp dpt:80

You can see the packet and byte transfer amounts to track the INPUT — traffic to a destination port on your server. If you want track the amount of data that the server is generating, you'd look for OUTPUT from the source port on your server:

[root@server ~]$ iptables -I OUTPUT -p tcp --sport 80 -j tracking
[root@server ~]$ iptables -vnL
...
Chain OUTPUT (policy ACCEPT 26149 packets, 174M bytes)
 pkts bytes target prot opt in out source           destination
  488 3367K tracking    tcp  --  *  *   0.0.0.0/0        0.0.0.0/0       tcp spt:80

Now that we know how the tracking chain works, we can add in a few different layers to get even more information. That way you can keep your INPUT and OUTPUT chains looking clean.

[root@server ~]$ iptables –N tracking
[root@server ~]$ iptables –N tracking2
[root@server ~]$ iptables –I INPUT –j tracking
[root@server ~]$ iptables –I OUTPUT –j tracking
[root@server ~]$ iptables –A tracking –p tcp --dport 80 –j tracking2
[root@server ~]$ iptables –A tracking –p tcp --sport 80 –j tracking2
[root@server ~]$ iptables -vnL
 
Chain INPUT (policy ACCEPT 96265 packets, 4131K bytes)
 pkts bytes target prot opt in out source           destination
 4002  184K tracking    all  --  *  *   0.0.0.0/0        0.0.0.0/0
 
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source           destination
 
Chain OUTPUT (policy ACCEPT 33751 packets, 231M bytes)
 pkts bytes target prot opt in out source           destination
 1399 9068K tracking    all  --  *  *   0.0.0.0/0        0.0.0.0/0
 
Chain tracking (2 references)
 pkts bytes target prot opt in out source           destination
 1208 59626 tracking2   tcp  --  *  *   0.0.0.0/0        0.0.0.0/0       tcp dpt:80
  224 1643K tracking2   tcp  --  *  *   0.0.0.0/0        0.0.0.0/0       tcp spt:80
 
Chain tracking2 (2 references)
 pkts bytes target prot opt in out source           destination

Keep in mind that every time a packet passes through one of your rules, it will eat CPU cycles. Diverting all your traffic through 100 rules that track bandwidth may not be the best idea, so it's important to have an efficient ruleset. If your server has eight processor cores and tons of overhead available, that concern might be inconsequential, but if you're running lean, you could conceivably run into issues.

The easiest way to think about making efficient rulesets is to think about eating the largest slice of pie first. Understand iptables rule processing and put the rules that get more traffic higher in your list. Conversely, save the tiniest pieces of your pie for last. If you run all of your traffic by a rule that only applies to a tiny segment before you screen out larger segments, you're wasting processing power.

Another thing to keep in mind is that you do not need to specify a target (in our examples above, we established tracking and tracking2 as our targets). If you're used to each rule having a specific purpose of either blocking, allowing, or diverting traffic, this simple tidbit might seem revolutionary. For example, we could use this rule:

[root@server ~]$ iptables -A INPUT

If that seems a little bare to you, don't worry ... It is! The output will show that it is a rule that tracks all traffic in the chain at that point. We're appending the data to the end of the chain in this example ("-A") but we could also insert it ("-I") at the top of the chain instead. This command could be helpful if you are using a number of different chains and you want to see the exact volume of packets that are filtered at any given point. Additionally, this strategy could show how much traffic a potential rule would filter before you run it on your production system. Because having several of these kinds of commands can get a little messy, it's also helpful to add comments to help sort things out:

[root@server ~]$ iptables -A INPUT -m comment --comment "track all data"
 
[root@server ~]$ iptables -vnL
Chain INPUT (policy ACCEPT 11M packets, 5280M bytes)
 pkts bytes target prot opt in out source           destination
   98  9352        all  --  *  *   0.0.0.0/0        0.0.0.0/0       /* track all data */

Nothing terribly complicated about using iptables to count bandwidth, right? If you have iptables rulesets and you want to get a glimpse at how your traffic is being affected, this little trick could be useful. You can rely on the information iptables gives you about your bandwidth usage, and you won't be the only one ... cPanel actually uses iptables to track bandwidth.

-Mark

April 15, 2013

The Heart of SoftLayer: People

When I started working for SoftLayer as a software engineer intern, I was skeptical about the company's culture. I read many of the culture posts on the blog, and while they seemed genuine, I was still a little worried about what the work atmosphere would be for a lowly summer intern. Fast-forward almost a year, and I look back on my early concerns and laugh ... I learned quickly that the real heart of SoftLayer is its employees, and the day-to-day operations I observed in the office consistently reinforced that principle.

It's easy to think about SoftLayer as a pure technology company. We provide infrastructure as a service capabilities for businesses with on-demand provisioning and short-term contracts. Our data centers, portal, network and APIs get the spotlight, but those differentiators wouldn't exist without the teams of employees that keep improving them on a daily basis. By focusing on the company culture and making sure employees are being challenged (but not overwhelmed), SoftLayer was indirectly improving the infrastructure we provide to customers.

When I walked into the office for my first day of work, I imagined that I'd be working in a cramped, dimly lit room in the back of the building where I'd be using hand-me-down hardware. When I was led to a good-sized, well-lit room and given a Core i3 laptop with two large monitors and a full suite of software, I started realizing how silly my worries were. I had access to the fully stocked break room, and within about a week, I felt like part of a community rather than a stale workplace.

My coworkers not only made me feel welcome but would frequently go out of their way to make sure I am comfortable and have the resources I needed to succeed. While the sheer amount of new information and existing code was daunting, managers assigned projects that were possible to complete and educational. I was doing useful work building and improving a complex production system rather than the busy work offered by many other employers' internship programs. I learned several new techniques and solidified my understanding of software engineering theory through practice. The open-door policy and friendly people around me not only created a strong sense of community but also allowed more efficient problem solving.

You may have noticed early in this post that I joined the company on a summer internship and that I also told you it's been about a year since I started. While summers in Texas feel long, they don't actually last a full year ... After my internship, I was offered a part-time position as a software engineer, and I'm going to be full-time when I graduate in May.

It's next to impossible to find a company that realizes the importance of its employees and wants to provide an environment for employees to succeed. The undeniable runaway success of the company is proof that SoftLayer's approach to taking care of employees is working.

-John

April 12, 2013

Catalyst at SXSW 2013: Mentorship and Meaningfulness

In the Community Development group, our mission is simple: Create the industry's most substantially helpful startup program that assists participants in a MEANINGFUL way. Meaningfulness is a subjective goal, but when it comes to fueling new businesses, numbers and statistics can't tell the whole story. Sure, we could run Catalyst like some of the other startup programs in the infrastructure world and gauge our success off of the number of partners using the hosting credits we provide, but if we only focused on hosting credits, we'd be leaving a significant opportunity on the table.

SoftLayer is able to offer the entrepreneurial community so much more than cloud computing instances and powerful servers. As a startup ourselves not so long ago, our team knows all about the difficulties of being an entrepreneur, and now that we're able to give back to the startup community, we want to share battle stories and lessons learned. Mentorship is one of the most valuable commodities for entrepreneurs and business founders, and SoftLayer's mentors are in a unique position to provide feedback about everything from infrastructure planning to hiring your first employees to engaging with your board of advisors to negotiating better terms on a round of funding.

The Catalyst team engages in these kinds discussions with our clients every day, and we've had some pretty remarkable success. When we better understand a client's business, we can provide better feedback and insight into the infrastructure that will help that business succeed. In other words, we build meaningful relationships with our Catalyst clients, and as a result, those clients are able to more efficiently leverage the hosting credits we provide them.

The distinction between Catalyst and other startup programs in the hosting industry has never been so apparent than after South by Southwest (SXSW) in Austin this year. I had the opportunity to meet with entrepreneurs, investors, and industry experts who have been thirsting for a program like Catalyst for years, and when they hear about what we're doing, they know they've found their oasis. I had a chance to sit down with Paul Ford in the Catalyst Startup Lounge at SXSW to talk about the program and some of the insights and feedback we'd gotten at the show:

Paul was quick to point out that being a leader in the startup community has more impact when you provide the best technology and pair that with a team that can deliver for startups what they need: meaningful support.

Later, I had an impromptu coffee with one of the world's largest, most prestigious Silicon Valley-based venture capital firms — probably THE most respected venture capital firm in the world, actually. As we chatted about the firm's seed-funding practices, the investment partner told me, "There is no better insurance policy for an infrastructure company than what SoftLayer is doing to ensure success for its startup clients." And I thought that was a pretty telling insight.

That simple sentence drove home the point that success in a program like Catalyst is not guaranteed by a particular technology, no matter how innovative or industry-leading that technology may be. Success comes from creating value BEYOND that technology, and when I sat down with George Karidis, he shared a few insights how the Catalyst vision came to be along with how the program has evolved to what it is today:

Catalyst is special. The relationships we build with entrepreneurs are meaningful. We've made commitments to have the talented brainpower within our own walls to be accessible to the community already. After SXSW, I knew I didn't have to compare what we were doing from what other programs are doing because that would be like comparing apples and some other fruit that doesn't do nearly as much for you as apples do.

I was told once on the campaign trail for President Clinton in '96 that so long as you have a rock-solid strategy, you cannot be beaten if you continue to execute on that strategy. Execute, Execute, Execute. If you waiver and react to the competition, you're dead in the water. With that in mind, we're going to keep executing on our strategy of being available to our Catalyst clients and actively helping them solve their problems. The only question that remains is this:

How can we help you?

-@JoshuaKrammes

April 10, 2013

Plivo: Tech Partner Spotlight

We invite each of our featured SoftLayer Tech Marketplace Partners to contribute a guest post to the SoftLayer Blog, and this week, we're happy to welcome Mike Lauricella from Plivo. Plivo is an open communications and messaging platform with advanced features, simple APIs, easy management and volume pricing.

Company Website: http://www.plivo.com/
Tech Partners Marketplace: http://www.softlayer.com/marketplace/plivo

Bridging the Gap Between the Web and Telephony

Businesses face a fundamental challenge in the worlds of telephony and messaging: Those worlds move too slowly, require too much telecom knowledge and take too long to adopt. As a result, developers often forgo phone and SMS functionality in their applications because the learning curves are so steep, and the dated architecture seems like a foreign language. Over the last twenty years, the web has evolved a lot faster than telephony, and that momentum only widens the gap between the "old" telecom world and the "new" Internet world. Plivo was created to bridge that gap and make telephony easy for developers to understand and incorporate into their applications with simple tools and APIs.

I could bore you to tears by describing the ins and outs of what we've learned about telephony and telecom since Plivo was founded, but I'd rather show off some of the noteworthy ways our customers have incorporated Plivo in their own businesses. After all, seeing those real-world applications is much more revealing about what Plivo does than any description of the nuts and bolts of our platform, right?

Conferencing Solution
The purest use-cases for Plivo are when our customers can simply leverage powerful telephony functionality. A perfect example is a conferencing solution one of our customers created to host large-scale conferences with up to 200 participants. The company integrated the solution into their product and CRM so that sales reps and customers could jump on conference calls quickly. With that integration, the executive management team can keep track of all kinds of information about the calls ... whether they're looking to find which calls resulted in closed sales or they just want to see the average duration of a conference call for a given time frame.

Call Tracking
Beyond facilitate conference calls quickly and seamlessly, many businesses have started using Plivo's integration to incorporate call tracking statistics in their environments. Call tracking is big business because information about who called what number, when they called, how long they talked and the result of the call (sale, no sale, follow up) can determine whether the appropriate interaction has taken place with prospects or customers.

Two Factor Authentication
With ever-increasing concerns about security online, we've seen a huge uptick in developers that come to Plivo for help with two factor authentication for web services. To ensure that a new site registrant is a real person who has provided a valid phone number (to help cut down on potential fraud), they use Plivo to send text messages with verification codes to those new registrant.

Mass Alert Messaging
Because emergencies can happen at any time, our customers have enlisted Plivo's functionality to send out mass alerts via phone calls and SMS messages when their customers are affected by an issue and need to be contacted. These voice and text messages can be sent quickly and easily with our automated tools, and while no one ever wants to deal with an emergency, having a solid communication lifeline provides some peace of mind.

WebRTC
An emerging new standard for communications is WebRTC — open project that enables web browsers with Real-Time Communications (RTC) capabilities. WebRTC make communications a feature of the Web without plugins or complex SIP clients. Plivo already supports WebRTC, and even though the project is relatively young, it's already being used in some amazing applications.

These use-cases are only the tip of the iceberg when it comes to how our customers are innovating on our platform, but I hope it helps paint a picture of the kinds of functionality Plivo enables simply and quickly. If you've been itching to incorporate telephony into your application, before you spending hours of your life poring over complex telecom architecture requirements, head over to plivo.com to see how easy we can make your life. We offer free developer accounts where you can start to make calls to other Plivo users and other SIP endpoints immediately, and we'd love to chat with you about how you can leverage Plivo to make your applications communicate.

If you have any questions, feel free to drop us a note at hello@plivo.com, and we'll get back to you with answers.

-Mike Lauricella, Plivo

This guest blog series highlights companies in SoftLayer's Technology Partners Marketplace.
These Partners have built their businesses on the SoftLayer Platform, and we're excited for them to tell their stories. New Partners will be added to the Marketplace each month, so stay tuned for many more come.
April 5, 2013

Server Challenge II Soliloquy: GDC 2013

This guest blog comes to us from one of the most popular members of the SoftLayer trade show team: The SoftLayer + Supermicro Server Challenge II. You've seen our coverage of conference attendees competing to win a MacBook Air, but you've never gotten the story from the Server Challenge's perspective ... until now. We secretly recorded the Server Challenge's introspective reflections on the competition at GDC 2013 to share with the world.

To compete, or not to compete, that is the question ... Or at least the question I see most conference attendees struggle with when they see me. Some people light up with excitement at the sight of me while others turn away in fear, and I've even noticed a few of them start shaking uncontrollably as they recount the years of toil they survived in data centers when they managed server hardware for a living. I don't take it personally, though ... which is fitting because I'm not a person.

I am just a simple server rack with an ambitious purpose. I was made to give conference attendees a tangible, server-related experience when they visit SoftLayer's booth, and I can humbly say that I've served that role faithfully and successfully. As attendees step up before me, they may have never touched a server in their lives, but by the time they finish their first attempt at the competition, that naivete is completely vanquished ... Some even spend hours asking questions and studying strategy about how to most effectively install drive trays and plug in network cables. In fact, I wouldn't be surprised if a few of the people reading this post are doing so in preparation for their next attempt.

When I was chosen as one of only a few server racks to don the Server Challenge II moniker, I knew my life would be difficult and dangerous. Luckily, I've been equipped with Supermicro servers that have proven to be even more resilient and durable than my creators would have hoped. While hard drive bays are designed to be hot-swappable, no one could have ever expected those bays would be swapped thousands of times by (often careless) conference attendees, but I haven't needed a single server to be replaced, and my hard drive trays have also held up remarkably well. As I was sleeping last night, I had a flashback to GDC in San Francisco:

It's dizzying to have flashbacks of time-lapse pictures, but those pictures painted a pretty accurate picture of what a single day of competition looks like for me. It's clear that I'm serving my purpose when I see crowds of attendees looking on as competitors set amazingly fast times. As I hear the conversations about strategies and techniques that might shave tenths or hundredths of seconds off the next attempt, I stand a little taller and play my 8-bit music a little louder.

I am the Server Challenge II ... Who's next?

-The Server Challenge II

April 1, 2013

SoftLayer Mobile: Now a Universal iOS Application

Last month, we put SoftLayer Mobile HD out to pasture. That iPad-specific application performed amazingly, and we got a lot of great feedback from our customers, so we doubled-down on our efforts to support iPad users by merging SoftLayer Mobile HD functionality with our standard SoftLayer Mobile app to provide a singular, universal application for all iOS devices.

By merging our two iOS applications into a single, universal app, we can provide better feature parity, maintain coherent architecture and increase code reuse and maintainability because we're only working with a single feature-rich binary app that provides a consistent user experience on the iPhone and the iPad at the same. Obviously, this meant we had to retool much of the legacy iPhone-specific SoftLayer Mobile app in order to provide the same device-specific functionality we had for the iPad in SoftLayer Mobile HD, but I was surprised at how straightforward that process ended up being. I thought I'd share a few of the resources iOS includes that simplify the process of creating a universal iOS application.

iOS supports development of universal applications via device-specific resource loading and device-specific runtime checks, and we leveraged those tools based on particular situations in our code base.

Device-specific resource loading allows iOS to choose the appropriate resource for the device being used. For example, if we have two different versions of an image called SoftLayerOnBlack.png to fit either an iPhone or an iPad, we simply call one SoftLayerOnBlack~iphone.png and call the other one SoftLayerOnBlack~ipad.png. With those two images in our application bundle, we let the system choose which image to use with a simple line of code:

UIImage* image = [UIImage imageNamed: @"SoftLayerOnBlack.png"];

In addition to device-specific resource loading, iOS also included device-specific runtime checks. With these runtime checks, we're able to create conditional code paths depending on the underlying device type:

if (UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad) {
    // The device is an iPad running iOS 3.2 or later.
} else {
    // The device is an iPhone or iPod touch.
}

These building blocks allow for a great deal of flexibility when it comes to creating a universal iOS application. Both techniques enable simple support based on what device is running the application, but they're used in subtly different ways. With those device-specific tools, developers are able to approach their universal applications in a couple of distinct ways:

Device-Dependent View Controller:
If we want users on the iPhone and iPad applications to have the same functionality but have the presentation tailored to their specific devices, we would create separate iPhone and iPad view controllers. For example, let's look at how our Object Storage browser appears on the iPhone and the iPad in SoftLayer Mobile:

Object Storage - iPhoneObject Storage - iPad

We want to take advantage of the additional real estate the iPad provides, so at runtime, the appropriate view controller is be selected based on the devices' UI context. The technique would look a little like this:

@implementation SLMenuController
...
 
- (void) navigateToStorageModule: (id) sender {
UIViewController<SLApplicationModule> *storageModule = nil;
    if (UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad) {
        storageModule = [SLStorageModule_iPad storageModule];
    } else {
        storageModule = [SLStorageModule storageModule];
    }
    [self navigateToModule: storageModule];
}
...
@end

"Universal" View Controller
In other situations, we didn't need for the viewing experience to differ between the iPhone and the iPad, so we used a single view controller for all devices. We don't compromise the user experience or presentation of data because the view controller either re-scales or reconfigures the layout at runtime based on screen size. Take a look at the "About" module on the iPhone and iPad:

About Module - iPhoneAbout Module - iPad

The code for the universal view controller of the "About" module looks something like this:

@implementation SLAboutModuleNavigationViewController
…
 
- (id) init {
    self = [super init];
    if (self) {
      _navigationHidden = YES;
_navigationWidth = [[UIScreen mainScreen] bounds].size.width * 0.5;
    }
    return self;
}@end

There are plenty of other iOS features and tricks in the universal SoftLayer Mobile app. If you've got a SoftLayer account and an iOS devices, download the app to try it out and let us know what you think. If you were a SoftLayer Mobile HD user, do you notice any significant changes in the new app from the legacy app?

-Pawel

P.S. If you're not on iOS but you still want some SoftLayer love on your mobile device, check out the other SoftLayer Mobile Apps on Android and Windows Phone.

March 26, 2013

Should My Startup Join an Accelerator/Incubator Program?

As part of my role at SoftLayer, I have the opportunity and privilege to mentor numerous entrepreneurs and startup teams when they partner with us through our Catalyst program. One question I hear often is, "Should I join an accelerator?" My answer: "That all depends." Let's look at the five lessons entrepreneurs should learn before they decide to join a startup accelerator or incubator program.

Lesson 1: The founders must be committed to the success of their venture.
Joining an accelerator or incubator comes with some strings attached — startups give up between 6 to 10 percent of their equity in exchange for some cash and structured program that usually lasts around three months. Obviously, this kind of commitment should not be taken lightly.

Too often, startups join accelerator programs before they are ready or mature enough as a team. Sometimes, a company's idea isn't fully baked, so they end up spending as much time "creating" their business as they do "accelerating" it. As a result, that company isn't able to leverage an accelerator's resources efficiently throughout the entire program ... The founders need to establish a vision for the business, begin laying the groundwork for the company's products and services, and be 100% committed to the accelerator program before joining. If you can't say with confidence that your startup meets all three of those requirements, don't do it. Take care of those three points and proceed to the next lesson.

Lesson 2: Be prepared to leverage what you are given.
Many startups join accelerator and incubator programs with unrealistic expectations. Participation in these programs — even the most exclusive and well-known ones — by no means guarantees that you'll raise additional money or have a successful exit. These programs provide startups with office space, free cloud services, and access to mentors, investors, recruiters and media ... Those outstanding services provide participating startups with a distinct competitive advantage, but they don't serve up success on a silver platter. If you aren't ready work tirelessly to leverage the benefits of a startup program, don't bother.

Lesson 3: Take advice and criticism well; mentors are trying to help.
"Mentorship" is very tough to qualify, and criticism is difficult to take ... Especially if you're 100% committed to your business and you don't want to be told that you've done something wrong. Mentors in these startup programs have "been there and done that," and they wouldn't be in a mentorship position if they weren't looking out for your best interest and the ultimate success of your company.

Look programs that take mentorship seriously and can provide a broad range of expertise from strategy to marketing and business development to software architecture to building and scaling IT infrastructure. Then be intentional about listening to the people around you.

Lesson 4: Do your research and make an informed decision.
With the proliferation of startups globally, we're also seeing an evolution in the accelerator ecosystem. There are a number accelerators being positioned to help support founders with ideas on a global, regional and local basis, but it's important to evaluate a program's vision with its execution of that vision. Not all startup programs are created equal, and some might not offer the right set of resources and opportunities for your team. When you're giving up equity in your company, you should have complete confidence that the accelerator or incubator you join will deliver on its side of the deal.

Lesson 5: Leverage the network and community you will meet.
When you've done your homework, applied and been accepted to the perfect startup program, meet everyone you can and learn from them. One of the most tangible benefits of joining an accelerator is the way you can fast track a business idea while boosting network contacts. Much in the way someone chooses a prestigious college or joins a fraternity, some of the most valuable resources you'll come across in these programs are the people you meet. In this way, accelerators and incubators are becoming a proxy for undergrad and graduate school ... The appeal for promising entrepreneurs is simple: Why wait to make a dent in the universe? Today, more people are going to college and fewer are landing well-paying jobs after graduation, so some of the world's best and brightest are turning to these communities and foregoing the more structured "higher education" process.

Even if your startup is plugging along smoothly, a startup accelerator or incubator program might be worth a look. Venture capitalists often trust programs like TechStars and 500 Startups to filter or vet early stage companies. If your business has the stamp of approval from one of these organizations, it's decidedly less risky than a business idea pitched by a random entrepreneur.

If you understand each of these lessons and you take advantage of the resources and opportunities provided by startup accelerators and incubators, the sky is the limit for your business. Now get to work.

Class dismissed.

-@gkdog

March 22, 2013

Social Media for Brands: Monitor Twitter Search via Email

If you're responsible for monitoring Twitter for conversations about your brand, you're faced with a challenge: You need to know what people are saying about your brand at all times AND you don't want to live your entire life in front of Twitter Search.

Over the years, a number of social media applications have been released specifically for brand managers and social media teams, but most of those applications (especially the free/inexpensive ones) differentiate themselves only by the quality of their analytics and how real-time their data is reported. If that's what you need, you have plenty of fantastic options. Those differentiators don't really help you if you want to take a more passive role in monitoring Twitter search ... You still have to log into the application to see your fancy dashboards with all of the information. Why can't the data come to you?

About three weeks ago, Hazzy stopped by my desk and asked if I'd help build a tool that uses the Twitter Search API to collect brand keywords mentions and send an email alert with those mentions in digest form every 30 minutes. The social media team had been using Twilert for these types of alerts since February 2012, but over the last few months, messages have been delayed due to issues connecting to Twitter search ... It seems that the service is so popular that it hits Twitter's limits on API calls. An email digest scheduled to be sent every thirty minutes ends up going out ten hours late, and ten hours is an eternity in social media time. We needed something a little more timely and reliable, so I got to work on a simple "Twitter Monitor" script to find all mentions of our keyword(s) on Twitter, email those results in a simple digest format, and repeat the process every 30 minutes when new mentions are found.

With Bear's Python-Twitter library on GitHub, connecting to the Twitter API is a breeze. Why did we use Bear's library in particular? Just look at his profile picture. Yeah ... 'nuff said. So with that Python wrapper to the Twitter API in place, I just had to figure out how to use the tools Twitter provided to get the job done. For the most part, the process was very clear, and Twitter actually made querying the search service much easier than we expected. The Search API finds all mentions of whatever string of characters you designate, so instead of creating an elaborate Boolean search for "SoftLayer OR #SoftLayer OR @SoftLayer ..." or any number of combinations of arbitrary strings, we could simply search for "SoftLayer" and have all of those results included. If you want to see only @ replies or hashtags, you can limit your search to those alone, but because "SoftLayer" isn't a word that gets thrown around much without referencing us, we wanted to see every instance. This is the code we ended up working with for the search functionality:

def status_by_search(search):
    statuses = api.GetSearch(term=search)
    results = filter(lambda x: x.id > get_log_value(), statuses)
    returns = []
    if len(results) > 0:
        for result in results:
            returns.append(format_status(result))
 
        new_tweets(results)
        return returns, len(returns)
    else:
        exit()

If you walk through the script, you'll notice that we want to return only unseen Tweets to our email recipients. Shortly after got the Twitter Monitor up and running, we noticed how easy it would be to get spammed with the same messages every time the script ran, so we had to filter our results accordingly. Twitter's API allows you to request tweets with a Tweet ID greater than one that you specify, however when I tried designating that "oldest" Tweet ID, we had mixed results ... Whether due to my ignorance or a fault in the implementation, we were getting fewer results than we should. Tweet IDs are unique and numerically sequential, so they can be relied upon as much as datetime (and far easier to boot), so I decided to use the highest Tweet ID from each batch of processed messages to filter the next set of results. The script stores that Tweet ID and uses a little bit of logic to determine which Tweets are newer than the last Tweet reported.

def new_tweets(results):
    if get_log_value() < max(result.id for result in results):
        set_log_value(max(result.id for result in results))
        return True
 
 
def get_log_value():
    with open('tweet.id', 'r') as f:
        return int(f.read())
 
 
def set_log_value(messageId):
    with open('tweet.id', 'w+') as f:
        f.write(str(messageId))

Once we culled out our new Tweets, we needed our script to email those results to our social media team. Luckily, we didn't have to reinvent the wheel here, and we added a few lines that enabled us to send an HTML-formatted email over any SMTP server. One of the downsides of the script is that login credentials for your SMTP server are stored in plaintext, so if you can come up with another alternative that adds a layer of security to those credentials (or lets you send with different kinds of credentials) we'd love for you to share it.

From that point, we could run the script manually from the server (or a laptop for that matter), and an email digest would be sent with new Tweets. Because we wanted to automate that process, I added a cron job that would run the script at the desired interval. As a bonus, if the script doesn't find any new Tweets since the last time it was run, it doesn't send an email, so you won't get spammed by "0 Results" messages overnight.

The script has been in action for a couple of weeks now, and it has gotten our social media team's seal of approval. We've added a few features here and there (like adding the number of Tweets in an email to the email's subject line), and I've enlisted the help of Kevin Landreth to clean up the code a little. Now, we're ready to share the SoftLayer Twitter Monitor script with the world via GitHub!

SoftLayer Twitter Monitor on GitHub

The script should work well right out of the box in any Python environment with the required libraries after a few simple configuration changes:

  • Get your Twitter Customer Secret, Access Token and Access Secret from https://dev.twitter.com/
  • Copy/paste that information where noted in the script.
  • Update your search term(s).
  • Enter your mailserver address and port.
  • Enter your email account credentials if you aren't working with an open relay.
  • Set the self.from_ and self.to values to your preference.
  • Ensure all of the Python requirements are met.
  • Configure a cron job to run the script your desired interval. For example, if you want to send emails every 10 minutes: */10 * * * * <path to python> <path to script> 2>&1 /dev/null

As soon as you add your information, you should be in business. You'll have an in-house Twitter Monitor that delivers a simple email digest of your new Twitter mentions at whatever interval you specify!

Like any good open source project, we want the community's feedback on how it can be improved or other features we could incorporate. This script uses the Search API, but we're also starting to play around with the Stream API and SoftLayer Message Queue to make some even cooler tools to automate brand monitoring on Twitter.

If you end up using the script and liking it, send SoftLayer a shout-out via Twitter and share it with your friends!

-@SoftLayerDevs

March 20, 2013

Learntrail: Tech Partner Spotlight

We invite each of our featured SoftLayer Tech Marketplace Partners to contribute a guest post to the SoftLayer Blog, and this week, we're happy to welcome Daniel Hamilton, CTO of Learntrail. Learntrail is a learning management system for creating, assigning, and tracking e-learning programs. It helps you train your employees and develop a more effective workforce.

The Power of Great People

In 1995, Peter Drucker, one of the founding fathers of modern-day management, shared a profoundly simple idea: "People are our greatest asset." Today, almost two decades later, that quote is reiterated in one form or another by the top executives at the largest companies in the world. You can have the best product, a stellar marketing plan and the perfect vision, but without a great team of people to execute with those tools, your company isn't going anywhere.

In an online world now driven by innovation, it's easy to want to substitute "technology" for "people" as a business's greatest asset, but I'd argue that Peter Drucker's quote is as true now as it was in 1995. Think about it in terms of keeping your webiste online. Your server's hardware — a powerful CPU, ample storage space, tons of RAM and a fast network connection — might dictate how your website runs when everything is going smoothly, but when your traffic spikes over the holidays or an article on your blog goes viral, your ability to respond quickly to keep your website operational will be dictated by the quality of your server admins and support staff.

While good companies focus on improving their products, great companies focus on improving their people. In 2010, Google approached the challenge of improving its people by creating GoogleEDU — a program designed to formalize the process of educating employees in new skills, strategies and perspectives. Beyond building a stronger team of smarter individuals, Google is clearly investing in its employees, and that investment goes a long way to engender loyalty and job satisfaction.

What if your business doesn't happen to have Google's resources or a $269 billion market cap? That's the problem Learntrail set out to solve. Our platform was designed to make it easy for businesses to create stunning, full-featured multimedia courses that can be monitored and tracked in detail with a few clicks.

Learntrail Chalkboard

You can bring your new-hire orientation program online, centralize training documents for new products, or create simple lessons about company-specific procedures through a sleek, easy-to-use portal. You’ll also get real-time reports about your team’s progress, so you'll know exactly how your training is being used by your employees. To prove how confident we are that Learntrail will meet your needs, we have a risk-free, no credit card required 14-day trial that lets you kick the tires and get a feel for how Learntrail can work for your business.

Your people are your greatest asset.

-Daniel Hamilton, Learntrail

This guest blog series highlights companies in SoftLayer's Technology Partners Marketplace.
These Partners have built their businesses on the SoftLayer Platform, and we're excited for them to tell their stories. New Partners will be added to the Marketplace each month, so stay tuned for many more come.
March 19, 2013

iptables Tips and Tricks: CSF Configuration

In our last "iptables Tips and Tricks" installment, we talked about Advanced Policy Firewall (APF) configuration, so it should come as no surprise that in this installment, we're turning our attention to ConfigServer Security & Firewall (CSF). Before we get started, you should probably run through the list of warnings I include at the top of the APF blog post and make sure you have your Band-Aid ready in case you need it.

To get the ball rolling, we need to download CSF and install it on our server. In this post, we're working with a CentOS 6.0 32-bit server, so our (root) terminal commands would look like this to download and install CSF:

$ wget http://www.configserver.com/free/csf.tgz #Download CSF using wget.
$ tar zxvf csf.tgz #Unpack it.
$ yum install perl-libwww-perl #Make sure perl modules are installed ...
$ yum install perl-Time-HiRes  #Otherwise it will generate an error.
$ cd csf
$ ./install.sh #Install CSF.
 
#MAKE SURE YOU HAVE YOUR BAND-AID READY
 
$ /etc/init.d/csf start #Start CSF. (Note: You can also use '$ service csf start')

Once you start CSF, you can see a list of the default rules that load at startup. CSF defaults to a DROP policy:

$ iptables -nL | grep policy
Chain INPUT (policy DROP)
Chain FORWARD (policy DROP)
Chain OUTPUT (policy DROP)

Don't ever run "iptables -F" unless you want to lock yourself out. In fact, you might want to add "This server is running CSF - do not run 'iptables -F'" to your /etc/motd, just as a reminder/warning to others.

CSF loads on startup by default. This means that if you get locked out, a simple reboot probably won't fix the problem. Runlevels 2, 3, 4, and 5 are all on:

$ chkconfig --list | grep csf
csf             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Some features of CSF will not work unless you have certain iptables modules installed. I believe they are installed by default in CentOS, but if you custom-built your iptables, they might not all be installed. Run this script to see if all modules are installed:

$ /etc/csf/csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
 
RESULT: csf should function on this server

As I mentioned, this is the default iptables installation on a minimal CentOS 6.0 image, so chances are good that these modules are already installed on your system. It never hurts to check, though.

The CSF Configuration File

The primary CSF configuration is stored in the well-documented /etc/csf/csf.conf file. CSF is extremely configurable, so there are a lot of options to read over. Let's take a look over some of the more important features:

Testing

TESTING = "1"
TESTING_INTERVAL = "5"

This TESTING cron job runs every "5" minutes so you don't lock yourself out when you're testing your rules. When you are satisfied with your rules (and confident that you won't lock yourself out), you can set TESTING to "0".

Globally Allowed Ports

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995"
 
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,53,80,110,113,443"
 
# Allow incoming UDP ports
UDP_IN = "20,21,53"
 
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "20,21,53,113,123"

Incoming Ping Requests

# Allow incoming PING
ICMP_IN = "1"

Allowing ping is usually a good option for diagnostic purposes, so I don't recommend turning it off. Disallowing ping is an example of "security through obscurity," and it will not typically dissuade your attackers.

Ethernet Device

ETH_DEVICE = ""
ETH6_DEVICE = ""

Here, you can configure iptables to ONLY use one Ethernet adapter. You might want to only guard your public network adapter in some situations.

IP Limit in Permanent "Deny" File

DENY_IP_LIMIT = "200"

A higher number here will obviously screen out more IP addresses in csf.deny, but higher numbers also may cause slowdowns.

IP Limit in Temporary "Deny" File

DENY_TEMP_IP_LIMIT = "100"

Similar to DENY_IP_LIMIT, the DENY_TEMP_IP_LIMIT represents the maximum number of IPs that can be stored in the temporary ban list.

SMTP Blocking

SMTP_BLOCK = "0"

When set to "1", SMTP_BLOCK does not completely block outbound SMTP, but it does block it for most users. This will prevent malicious scripts and compromised users from making outbound connections from unauthorized mail clients on the server. SMTP_BLOCK doesn't stop those scripts from running, but it does stop them from functioning. Mail sent through the proper channels will still be delivered normally.

Allowing SMTP on localhost

SMTP_ALLOWLOCAL = "1"

Custom Mail Port Designation

SMTP_PORTS = "25,465,587"

Allowing SMTP Access to Users/Groups

SMTP_ALLOWUSER = ""
SMTP_ALLOWGROUP = "mail,mailman"

SYN Flood Protection

SYNFLOOD = "0"
SYNFLOOD_RATE = "100/s"
SYNFLOOD_BURST = "150"

Per the documentation, you should only enable SYN flood protection (SYNFLOOD= "1") if you are currently under a SYN flood attack.

Concurrent Connections Limit

CONNLIMIT = "22;5,80;20"
PORTFLOOD = "22;tcp;5;300,80;tcp;20;5

These options allow you to add customized DoS protection. CONNLIMIT handles the number of concurrent connections, and in this example, we're limiting port 22 to 5 connections and port 80 to 20 connections.

PORTFLOOD watches the number of connections per a given number of seconds. In this example, we're limiting the TCP connection on port 22 to 5 connections/second with a quiet period of 300 seconds before the connection is unblocked. Additonally, we're limiting the TCP connection on port 80 to 20 connections/second with a quiet period of 5 seconds before the connection is unblocked.

Check the readme.txt file for more information about the syntax.

Logging to Syslog

SYSLOG = "0"

When enabled, this option logs lfd (Login Failure Daemon) messages to syslog as well as to /var/log/lfd.log.

Dropping v. Rejecting Packets

DROP = "DROP"

This configuration allows you to either DROP or REJECT packets. REJECT tells the sender that the packet has been blocked by the firewall. DROP just drops the packet and does not send a response. I like DROP better for regular use, but REJECT might be more helpful if you need to diagnose a connectivity issue.

Logging Dropped Connections

DROP_LOGGING = "1"

This option logs dropped connections to syslog. I don't see any reason to turn this off unless your hard drive is getting full.

Port Exceptions When Logging Dropped Connections

DROP_NOLOG = "67,68,111,113,135:139,445,500,513,520"

These ports are specifically blocked from being logged either to conserve hard drive space or make the log file easier to read.

"Watch Mode"

WATCH_MODE = "0"

If you are ever stuck trying to troubleshoot a large ruleset, you might consider turning this option on. You can use it to track the actions to watched IP addresses to see where they are getting blocked or accepted.

Login Failure Daemon Alert

LF_ALERT_TO = ""
LF_ALERT_FROM = ""
LF_ALERT_SMTP = ""

You can specify an email address to report errors from the Login Failure Daemon, which tracks and automatically blocks brute force login attempts.

Permanent Blocks and NetBlocks

LF_PERMBLOCK = "1"
LF_PERMBLOCK_INTERVAL = "86400"
LF_PERMBLOCK_COUNT = "4"
LF_PERMBLOCK_ALERT = "1"
LF_NETBLOCK = "0"
LF_NETBLOCK_INTERVAL = "86400"
LF_NETBLOCK_COUNT = "4"
LF_NETBLOCK_CLASS = "C"
LF_NETBLOCK_ALERT = "1"

These settings control the permanent block and netblock blocking. You probably don't need to touch these settings, but you might want some additional security or less security depending on your company needs. If something gets permablocked, it will require your intervention to clear it, which might create downtime for your clients. Likewise, if a legitimate IP address happens to be part of a netblock which has an attacking IP address on it, it will get blocked if you have that feature turned on. A class C network encompasses 256 IP addresses. You can set this to class B or A, but that could block thousands or millions of IP addresses, respectively. Unless you find yourself under constant attack, I would advise you to leave that LF_NETBLOCK off.

Additional Protection During Updates

# Safe Chain Update. If enabled, all dynamic update chains (GALLOW*, GDENY*,
# SPAMHAUS, DSHIELD, BOGON, CC_ALLOW, CC_DENY, ALLOWDYN*) will create a new
# chain when updating, and insert it into the relevant LOCALINPUT/LOCALOUTPUT
# chain, then flush and delete the old dynamic chain and rename the new chain.
#
# This prevents a small window of opportunity opening when an update occurs and
# the dynamic chain is flushed for the new rules.
SAFECHAINUPDATE = "0"

Activating this option will increase your system resource usage and will require more rules to be running at one time, but it provides an additional layer of protection during updates. Without this option turned on, your rules will be flushed for a short amount of time, leaving your server vulnerable.

Multi-Server Deployment Options

LF_GLOBAL = "0"
GLOBAL_ALLOW = ""
GLOBAL_DENY = ""
GLOBAL_IGNORE = ""

Like APF, you can configure global lists for multiple server deployments. You'll need to specify a URL of the text file with the IP addresses for the global lists.

SPAMHAUSE Blocklist

LF_SPAMHAUS = "0"

This option enables the SPAMHAUS blocklist. Specify the number of seconds between refreshes. Recommended setting is 86400 (1 day).

Blocking TOR Exit IP Addresses

LF_TOR = "0"

Enabling this option will block TOR exit IP addresses. If you are not familiar with TOR, it is a completely anonymous proxy network. This could block some legitimate users who are trying to protect their anonymity, so I would recommend only turning this on if you are already under attack from a TOR exit address.

Blocking Bogon Addresses

LF_BOGON = "0"
LF_BOGON_URL = "http://www.cymru.com/Documents/bogon-bn-agg.txt"
LF_BOGON_SKIP = ""

Blocking bogon addresses (addresses that should not be possible) is usually a good decision. To enable, set the number of seconds between refreshes. I recommend enabling this option and setting the refresh at 86400 (1 day). If you do so, be sure to add your private network adapters to the skip list.

Country-Specific Access to Your Server

CC_DENY = ""
CC_ALLOW = ""

With these options, you can block or allow entire countries from accessing your server. To do so, enter the country codes in a comma separated list. Even though this generates a lot of additional rules, it's valuable to some sysadmins.

CC_ALLOW_FILTER = ""

Alternatively, you can set your server to exclusively accept traffic from a list of country codes. All other countries not listed will have their traffic dropped. There are many other settings related to these options that I don't have time to cover in this blog.

Blocking Login Failures

LF_TRIGGER = "0"

This enables blocking of login failures (per service). There are a lot of great customization options in this section.

Scanning Directories for Malicious Files

LF_DIRWATCH = "300"

This feature scans /tmp and /dev/shm for potentially malicious files and alerts you to their presence based on the interval you designate. You can also have CSF automatically quarantine malicious files with this option:

LF_DIRWATCH_DISABLE = "0"

Distributed Attack Protection

LF_DISTATTACK = "0"

By enabling this option, you activate additional protection against distributed attacks.

Blocking Based on Abusive Email Usage

LT_POP3D = "0"
LT_IMAPD = "0"

If a user checks email too many times per hour (more than the non-zero value specified), the user's IP address is blocked.

Email Alert Following Block

LT_EMAIL_ALERT = "1"

This will send you email when something is blocked. I'd recommend leaving it on.

Blocking IP Addresses Based on Number of Connections

CT_LIMIT = "0"

This feature tracks connections and blocks the IP if the number of connections is too high. Use caution because if you enable this option and set this value too low, it will block legitimate traffic.

Application-Level Protection

PT_LIMIT = "60"

This feature provides application level protection against malicious scripts that take a long time to execute.

Blocking Port Scanners

PS_INTERVAL = "300"
PS_LIMIT = "10"

Enabling HTML User Interface for CSF

UI = "0"

CSF has a built-in HTML user interface. You can enable this by setting UI = "1". There are a list of prerequisites for this option in the readme.txt.

Notifying Blocked IP Addresses

MESSENGER = "0"

This option will notify blocked IP addresses when they have been blocked by the firewall.

Port Knocking

PORTKNOCKING = ""

CSF supports port knocking, which is a technique that provides an additional layer of security. See http://www.portknocking.org/ for details.

Allow and Deny Lists

As we walked through the CSF configuration file, you saw that I referenced the csf.deny file, so it should come as no surprise that CSF also includes csf.allow to customize "allow" rules as well. If you are familiar with APF, these files have a very similar syntax ... Each entry is made up of the same four components: protocol|flow|port|IP. The only real difference being that APF uses the colon as a delimiter while CSF uses the pipe:

#APF Version
tcp:in:d=48000_48020:s=10.0.0.0/8
 
#CSF Version
tcp|in|d=48000_48020|s=10.0.0.0/8

Fortunately, replacing your colon with a pipe is a minimally invasive procedure that can be automated with a tool like vi.

CSF Command Line Tool

The command line tool for CSF is much more robust than the one for APF:

$ csf --help
csf: v5.79 (cPanel)
 
ConfigServer Security &amp; Firewall
(c)2006-2013, Way to the Web Limited (http://www.configserver.com)
 
Usage: /usr/sbin/csf [option] [value]
 
Option              Meaning
-h, --help          Show this message
-l, --status        List/Show iptables configuration
-l6, --status6      List/Show ip6tables configuration
-s, --start         Start firewall rules
-f, --stop          Flush/Stop firewall rules (Note: lfd may restart csf)
-r, --restart       Restart firewall rules
-q, --startq        Quick restart (csf restarted by lfd)
-sf, --startf       Force CLI restart regardless of LF_QUICKSTART setting
-a, --add ip        Allow an IP and add to /etc/csf.allow
-ar, --addrm ip     Remove an IP from /etc/csf.allow and delete rule
-d, --deny ip       Deny an IP and add to /etc/csf.deny
-dr, --denyrm ip    Unblock an IP and remove from /etc/csf.deny
-df, --denyf        Remove and unblock all entries in /etc/csf.deny
-g, --grep ip       Search the iptables rules for an IP match (incl. CIDR)
-t, --temp          Displays the current list of temp IP entries and their TTL
-tr, --temprm ip    Remove an IPs from the temp IP ban and allow list
-td, --tempdeny ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP ban list. ttl is how long to
                    blocks for (default:seconds, can use one suffix of h/m/d).
                    Optional port. Optional direction of block can be one of:
                    in, out or inout (default:in)
-ta, --tempallow ip ttl [-p port] [-d direction]
                    Add an IP to the temp IP allow list (default:inout)
-tf, --tempf        Flush all IPs from the temp IP entries
-cp, --cping        PING all members in an lfd Cluster
-cd, --cdeny ip     Deny an IP in a Cluster and add to /etc/csf.deny
-ca, --callow ip    Allow an IP in a Cluster and add to /etc/csf.allow
-cr, --crm ip       Unblock an IP in a Cluster and remove from /etc/csf.deny
-cc, --cconfig [name] [value]
                    Change configuration option [name] to [value] in a Cluster
-cf, --cfile [file] Send [file] in a Cluster to /etc/csf/
-crs, --crestart    Cluster restart csf and lfd
-w, --watch ip      Log SYN packets for an IP across iptables chains
-m, --mail [addr]   Display Server Check in HTML or email to [addr] if present
-lr, --logrun       Initiate Log Scanner report via lfd
-c, --check         Check for updates to csf but do not upgrade
-u, --update        Check for updates to csf and upgrade if available
-uf                 Force an update of csf
-x, --disable       Disable csf and lfd
-e, --enable        Enable csf and lfd if previously disabled
-v, --version       Show csf version

The command line tool will also tell you if the testing mode is enabled (which is a very useful feature). If TESTING were enabled, we'd see this line at the bottom of the output:

*WARNING* TESTING mode is enabled - do not forget to disable it in the configuration

Did you make it all the way through?! Great! I know it's a lot to take in, but it's not terribly complicated when we break it down and understand how each piece works. Next time, I'll be back with some tips on integrating CSF into cPanel.

-Mark

Pages

Subscribe to startup-series