Posts Tagged 'Open Source'

April 29, 2013

Web Development - Installing mod_security with OWASP

You want to secure your web application, but you don't know where to start. A number of open-source resources and modules exist, but that variety is more intimidating than it is liberating. If you're going to take the time to implement application security, you don't want to put your eggs in the wrong basket, so you wind up suffering from analysis paralysis as you compare all of the options. You want a powerful, flexible security solution that isn't overly complex, so to save you the headache of making the decision, I'll make it for you: Start with mod_security and OWASP.

ModSecurity (mod_security) is an open-source Apache module that acts as a web application firewall. It is used to help protect your server (and websites) from several methods of attack, most common being brute force. You can think of mod_security as an invisible layer that separates users and the content on your server, quietly monitoring HTTP traffic and other interactions. It's easy to understand and simple to implement.

The challenge is that without some advanced configuration, mod_security isn't very functional, and that advanced configuration can get complex pretty quickly. You need to determine and set additional rules so that mod_security knows how to respond when approached with a potential threat. That's where Open Web Application Security Project (OWASP) comes in. You can think of the OWASP as an enhanced core ruleset that the mod_security module will follow to prevent attacks on your server.

The process of getting started with mod_security and OWASP might seem like a lot of work, but it's actually quite simple. Let's look at the installation and configuration process in a CentOS environment. First, we want to install the dependencies that mod_security needs:

## Install the GCC compiler and mod_security dependencies ##
$ sudo yum install gcc make
$ sudo yum install libxml2 libxml2-devel httpd-devel pcre-devel curl-devel

Now that we have the dependencies in place, let's install mod_security. Unfortunately, there is no yum for mod_security because it is not a maintained package, so you'll have to install it directly from the source:

## Get mod_security from its source ##
$ cd /usr/src
$ git clone https://github.com/SpiderLabs/ModSecurity.git

Now that we have mod_security on our server, we'll install it:

## Install mod_security ##
$ cd ModSecurity
$ ./configure
$ make install

And we'll copy over the default mod_security configuration file into the necessary Apache directory:

## Copy configuration file ##
$ cp modsecurity.conf-recommended /etc/httpd/conf.d/modsecurity.conf

We've got mod_security installed now, so we need to tell Apache about it ... It's no use having mod_security installed if our server doesn't know it's supposed to be using it:

## Apache configuration for mod_security ##
$ vi /etc/httpd/conf/httpd.conf

We'll need to load our Apache config file to include our dependencies (BEFORE the mod_security module) and the mod_security file module itself:

## Load dependencies ##
LoadFile /usr/lib/libxml2.so
LoadFile /usr/lib/liblua5.1.so
## Load mod_security ##
LoadModule security2_module modules/mod_security2.so

We'll save our configuration changes and restart Apache:

## Restart Apache! ##
$ sudo /etc/init.d/httpd restart

As I mentioned at the top of this post, our installation of mod_security is good, but we want to enhance our ruleset with the help of OWASP. If you've made it this far, you won't have a problem following a similar process to install OWASP:

## OWASP ##
$ cd /etc/httpd/
$ git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git
$ mv owasp-modsecurity-crs modsecurity-crs

Just like with mod_security, we'll set up our configuration file:

## OWASP configuration file ##
$ cd modsecurity-crs
$ cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_config.conf

Now we have mod_security and the OWASP core ruleset ready to go! The last step we need to take is to update the Apache config file to set up our basic ruleset:

## Apache configuration ##
$ vi /etc/httpd/conf/httpd.conf

We'll add an IfModule and point it to our new OWASP rule set at the end of the file:

<IfModule security2_module>
    Include modsecurity-crs/modsecurity_crs_10_config.conf
    Include modsecurity-crs/base_rules/*.conf
</IfModule>

And to complete the installation, we save the config file and restart Apache:

## Restart Apache! ##
$ sudo /etc/init.d/httpd restart

And we've got mod_security installed with the OWASP core ruleset! With this default installation, we're leveraging the rules the OWASP open source community has come up with, and we have the flexibility to tweak and enhance those rules as our needs dictate. If you have any questions about this installation or you have any other technical blog topics you'd like to hear from us about, please let us know!

-Cassandra

March 22, 2013

Social Media for Brands: Monitor Twitter Search via Email

If you're responsible for monitoring Twitter for conversations about your brand, you're faced with a challenge: You need to know what people are saying about your brand at all times AND you don't want to live your entire life in front of Twitter Search.

Over the years, a number of social media applications have been released specifically for brand managers and social media teams, but most of those applications (especially the free/inexpensive ones) differentiate themselves only by the quality of their analytics and how real-time their data is reported. If that's what you need, you have plenty of fantastic options. Those differentiators don't really help you if you want to take a more passive role in monitoring Twitter search ... You still have to log into the application to see your fancy dashboards with all of the information. Why can't the data come to you?

About three weeks ago, Hazzy stopped by my desk and asked if I'd help build a tool that uses the Twitter Search API to collect brand keywords mentions and send an email alert with those mentions in digest form every 30 minutes. The social media team had been using Twilert for these types of alerts since February 2012, but over the last few months, messages have been delayed due to issues connecting to Twitter search ... It seems that the service is so popular that it hits Twitter's limits on API calls. An email digest scheduled to be sent every thirty minutes ends up going out ten hours late, and ten hours is an eternity in social media time. We needed something a little more timely and reliable, so I got to work on a simple "Twitter Monitor" script to find all mentions of our keyword(s) on Twitter, email those results in a simple digest format, and repeat the process every 30 minutes when new mentions are found.

With Bear's Python-Twitter library on GitHub, connecting to the Twitter API is a breeze. Why did we use Bear's library in particular? Just look at his profile picture. Yeah ... 'nuff said. So with that Python wrapper to the Twitter API in place, I just had to figure out how to use the tools Twitter provided to get the job done. For the most part, the process was very clear, and Twitter actually made querying the search service much easier than we expected. The Search API finds all mentions of whatever string of characters you designate, so instead of creating an elaborate Boolean search for "SoftLayer OR #SoftLayer OR @SoftLayer ..." or any number of combinations of arbitrary strings, we could simply search for "SoftLayer" and have all of those results included. If you want to see only @ replies or hashtags, you can limit your search to those alone, but because "SoftLayer" isn't a word that gets thrown around much without referencing us, we wanted to see every instance. This is the code we ended up working with for the search functionality:

def status_by_search(search):
    statuses = api.GetSearch(term=search)
    results = filter(lambda x: x.id > get_log_value(), statuses)
    returns = []
    if len(results) > 0:
        for result in results:
            returns.append(format_status(result))
 
        new_tweets(results)
        return returns, len(returns)
    else:
        exit()

If you walk through the script, you'll notice that we want to return only unseen Tweets to our email recipients. Shortly after got the Twitter Monitor up and running, we noticed how easy it would be to get spammed with the same messages every time the script ran, so we had to filter our results accordingly. Twitter's API allows you to request tweets with a Tweet ID greater than one that you specify, however when I tried designating that "oldest" Tweet ID, we had mixed results ... Whether due to my ignorance or a fault in the implementation, we were getting fewer results than we should. Tweet IDs are unique and numerically sequential, so they can be relied upon as much as datetime (and far easier to boot), so I decided to use the highest Tweet ID from each batch of processed messages to filter the next set of results. The script stores that Tweet ID and uses a little bit of logic to determine which Tweets are newer than the last Tweet reported.

def new_tweets(results):
    if get_log_value() < max(result.id for result in results):
        set_log_value(max(result.id for result in results))
        return True
 
 
def get_log_value():
    with open('tweet.id', 'r') as f:
        return int(f.read())
 
 
def set_log_value(messageId):
    with open('tweet.id', 'w+') as f:
        f.write(str(messageId))

Once we culled out our new Tweets, we needed our script to email those results to our social media team. Luckily, we didn't have to reinvent the wheel here, and we added a few lines that enabled us to send an HTML-formatted email over any SMTP server. One of the downsides of the script is that login credentials for your SMTP server are stored in plaintext, so if you can come up with another alternative that adds a layer of security to those credentials (or lets you send with different kinds of credentials) we'd love for you to share it.

From that point, we could run the script manually from the server (or a laptop for that matter), and an email digest would be sent with new Tweets. Because we wanted to automate that process, I added a cron job that would run the script at the desired interval. As a bonus, if the script doesn't find any new Tweets since the last time it was run, it doesn't send an email, so you won't get spammed by "0 Results" messages overnight.

The script has been in action for a couple of weeks now, and it has gotten our social media team's seal of approval. We've added a few features here and there (like adding the number of Tweets in an email to the email's subject line), and I've enlisted the help of Kevin Landreth to clean up the code a little. Now, we're ready to share the SoftLayer Twitter Monitor script with the world via GitHub!

SoftLayer Twitter Monitor on GitHub

The script should work well right out of the box in any Python environment with the required libraries after a few simple configuration changes:

  • Get your Twitter Customer Secret, Access Token and Access Secret from https://dev.twitter.com/
  • Copy/paste that information where noted in the script.
  • Update your search term(s).
  • Enter your mailserver address and port.
  • Enter your email account credentials if you aren't working with an open relay.
  • Set the self.from_ and self.to values to your preference.
  • Ensure all of the Python requirements are met.
  • Configure a cron job to run the script your desired interval. For example, if you want to send emails every 10 minutes: */10 * * * * <path to python> <path to script> 2>&1 /dev/null

As soon as you add your information, you should be in business. You'll have an in-house Twitter Monitor that delivers a simple email digest of your new Twitter mentions at whatever interval you specify!

Like any good open source project, we want the community's feedback on how it can be improved or other features we could incorporate. This script uses the Search API, but we're also starting to play around with the Stream API and SoftLayer Message Queue to make some even cooler tools to automate brand monitoring on Twitter.

If you end up using the script and liking it, send SoftLayer a shout-out via Twitter and share it with your friends!

-@SoftLayerDevs

August 1, 2012

SoftLayer + Open Source + OSCON

While a handful of SoftLayer employees made their way to Boston for HostingCon, another ragtag group of SLayers journeyed to Portland to attend OSCON &mdash: the Open Source CONvention. OSCON attracts 2,500+ passionate members of the open source community, so the conference sessions and expo hall are filled with the most creative and innovative people on the Web. That's where we want to be.

Over the past few years, we've built a great reputation at OSCON as not only a great hosting provider, but also as the operator of one of the best booths on the expo hall floor. As usual, the switchballs were crowd pleasers, and we sponsored the show's Massage Booth, so we had great traffic through our booth all conference. When attendees left our booth, they were considerably more relaxed, they had the coolest swag at the show, and they had a better understanding of where SoftLayer fits in the open source space.

In addition to the conversations on the expo hall floor, we got to share a little expertise in a conference session. Senior Software Architect Harold Hannon presented an engaging educational session about how we implemented elasticsearch, Apache-based code that allows for scalable search for all kinds of documents in near real-time. At the moment, SoftLayer uses elasticsearch internally for hardware and ticketing, and we hope to extend this feature-rich scalable searching to our customers in an upcoming release of the customer portal. Because SoftLayer has built a great reputation for executing scalability well, Harold ended up presenting to a packed house (which you can see in the last few pictures of the slide show above).

SoftLayer's significant investment in open source platforms like OpenStack Swift Object Storage and CloudStack-based Private Clouds wound up being a big topic of discussion throughout the conference. Harold's elasticsearch presentation was a great conversation bridge to talk about the incredible search-and-retrieve functionality we implemented in our Object Storage service, and we were able to share and demonstrate how that functionality helps our customers manage large quantities of static data in cloud environments in an automated way.

The open source community has matured significantly over the past few years, and it's exciting to see that evolution. We aren't just talking about the incredibly popular open source operating systems like CentOS, Debian, Fedora, FreeBSD and Ubuntu that customers can get on a dedicated or cloud server ... We're talking about game-changing, innovative platforms that are redefining how the Internet works.

We want to thank the OSCON team for another phenomenal show, and if you attended the show but didn't get a switchball from us, I'm sure you'll have another chance at OSCON 2013. If you don't think you can wait that long, come find us at one of our other upcoming events!

-Summer

Categories: 
February 15, 2012

SoftLayer + OpenStack Swift = SoftLayer Object Storage

Since our inception in 2005, SoftLayer's goal has been to provide an array of on-demand data center and hosting services that combine exceptional access, control, scalability and security with unparalleled network robustness and ease of use ... That's why we're so excited to unveil SoftLayer Object Storage to our customers.

Based on OpenStack Object Storage (codenamed Swift) — open-source software that allows the creation of redundant, scalable object storage on clusters of standardized servers — SoftLayer Object Storage provides customers with new opportunities to leverage cost-effective cloud-based storage and to simultaneously realize significant capex-related cost savings.

OpenStack has been phenomenally successful thanks to a global software community comprised of developers and other technologists that has built and tweaked a standards-based, massively scalable open-source platform for public and private cloud computing. The simple goal of the OpenStack project is to deliver code that enables any organization to create and offer feature-rich cloud computing services from industry-standard hardware. The overarching OpenStack technology consists of several interrelated project components: One for compute, one for an image service, one for object storage, and a few more projects in development.

SoftLayer Object Storage
Like the OpenStack Swift system on which it is based, SoftLayer Object Storage is not a file system or real-time data-storage system, rather it's a long-term storage system for a more permanent type of static data that can be retrieved, leveraged and updated when necessary. Typical applications for this type of storage can involve virtual machine images, photo storage, email storage and backup archiving.

One of the primary benefits of Object Storage is the role that it can play in automating and streamlining data storage in cloud computing environments. SoftLayer Object Storage offers rich metadata features and search capability that can be leveraged to automate the way unstructured data gets accessed. In this way, SoftLayer Object Storage will provide organizations with new capabilities for improving overall data management and storage efficiency.

File Storage v. Object Storage
To better understand the difference between file storage and object storage, let's look at how file storage and object storage differ when it comes to metadata and search for a simple photo image. When a digital camera or camera-enabled phone snaps a photo, it embeds a series of metadata values in the image. If you save the image in a standard image file format, you can search for it by standard file properties like name, date and size. If you save the same image with additional metadata as an object, you can set object metadata values for the image (after reading them from the image file). This detail provides granular search capability based on the metadata keys and values, in addition to the standard object properties. Here is a sample comparison of an image's metadata value in both systems:

File Metadata Object Metadata
Name:img01.jpg Name:img01.jpg
Date: 2012-02-13 Date:2012-02-13
Size:1.2MB Size:1.2MB
Manufacturer:CASIO
Model:QV-4000
x-Resolution:72.00
y-Resolution:72.00
PixelXDimension:2240
PixelYDimension:1680
FNumber:f/4.0
Exposure Time:1/659 sec.

Using the rich metadata and search capability enabled by object storage, you would be able to search for all images with a dimension of 2240x1680 or a resolution of 72x72 in a quick/automated fashion. The object storage system "understands" more about what is being stored because it is able to differentiate files based on characteristics that you define.

What Makes SoftLayer Object Storage Different?
SoftLayer Object Storage features several unique features and ways for SoftLayer customers to upload, access and manage data:

  • Search — Quickly access information through user-defined metadata key-value pairs, file name or unique identifier
  • CDN — Serve your content globally over our high-performance content delivery network
  • Private Network — Free, secure private network traffic between all data centers and storage cluster nodes
  • API — Access to a full-feature OpenStack-compatible API with additional support for CDN and search integration
  • Portal — Web application integrated into the SoftLayer portal
  • Mobile — iPhone and Android mobile apps, with Windows Phone app coming soon
  • Language Bindings — Feature-complete bindings for Java, PHP, Python and Ruby*

*Language bindings, documentation, and guides are available on SLDN.

We think SoftLayer Object Storage will be attractive to a broad range of current and prospective customers, from web-centric businesses dependent on file sharing and content distribution to legal/medical/financial-services companies which possess large volumes of data that must be stored securely while remaining readily accessible.

SoftLayer Object Storage significantly extends our cloud-services portfolio while substantially enriching the storage capabilities that we bring to our customers. What are you waiting for? Go order yourself some object storage @ $0.12/GB!

-Marc

February 14, 2012

Open Source, OpenStack and SoftLayer

The open-source model has significantly revolutionized not only the IT industry but the business world as well. In fact, it was one of the key "flatteners" Thomas Friedman covered in his tour de force on globalization — The World is Flat. The trend toward collaborating on online projects — including open-source software, blogs, and Wikipedia — remains one of "the most disruptive forces of all."

The success of open-source projects like Linux, Ruby on Rails, and Android reveals the strength and diversity of having developers around the world contributing and providing feedback on code. The community becomes more than the sum of its parts, driving innovation and constant improvement. The case has been made for open source in and of itself, but a debate still rages over the developing case for businesses contributing to open source. Why would a business dedicate resources to the development of something it can't sell?

The answer is simple and straightforward: Contributing to open source fosters a community that can inspire, create and fuel the innovation a business needs to keep providing its customers with even better products. It makes sense ... Having hundreds of developers with different skills and perspectives working on a project can push that project further faster. The end result is a product that benefits the open-source community and the business world. The destiny of the community or the product cannot be defined by a single vendor or business; it's the democratization of technology.

Open-Source Cloud Platforms
Today, there are several open-source cloud platforms vying for industry dominance. SoftLayer has always been a big proponent and supporter of open source, and we've been involved with the OpenStack project from the beginning. In fact, we just announced SoftLayer Object Storage, an offering based on OpenStack Object Storage (code-named Swift). We'll provide code and support for Swift in hopes that it continues to grow and improve. The basic idea behind Swift Object Storage is to create redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data. I could go on and on about object storage, but I know Marc Jones has a blog specifically about SoftLayer Object Storage being published tomorrow, and I don't want to steal too much of his thunder.

We have to acknowledge and embrace the heterogeneous nature of IT industry. Just as you might use multiple operating systems and hypervisors, we're plan on working with a variety of open-source cloud platforms. Right now, we're looking into supporting initiatives like Eucalyptus, and we have our ear to the street to listen to what our customers are asking for. Our overarching goal is to provide our customers with much-needed technologies that are advancing the hosting industry, and one of the best ways to get to that end is to serve the needs of the open-source community.

As I write this blog post, I can't help but think of it in terms of a the Lord of Rings reference: "One ring to rule them all." The idea that "one ring" is all we need to focus on as a hosting provider just doesn't work when it comes to the open-source community ... It all comes down to enabling choice and flexibility. We'll keep investing in innovation wherever we can, and we'll let the market decide which ring will rule where.

What open-source projects are you working on now? How can SoftLayer get involved?

-Matt

February 14, 2011

The Black Cat

"Dogma." "Religion." What comes to mind when you hear these words? In the real world, you might think of Christianity, Islam, or Judaism. In the political world, you might think of Communism vs. Freedom. Closer to home, you might think of "red state" and "blue state."

Computers are deterministic, logical machines, yet they too have all the trappings of the world's major religions and dogmas. The desktop world is dominated by Microsoft and Apple to use the religion metaphor. All computing worlds could be broken down into "proprietary" and "open source" if we are talking about dogma.

Relevant to this discussion, the web development world has three major religions in those two dogmas: Microsoft's ASP.NET, the PHP world, and the Java world. My platform of choice has always been ASP.NET.

I am pretty solid in my reasons for preferring it over all others, and also pretty clear about the accidental reasons I found myself in this camp. Much how someone born into a particular religion is likely to freely adopt it at some point, I too ended up adopting ASP.NET for reasons that were nothing more than accidental.

I consider myself a 'citizen of the world' in more ways than one, and the opportunity to work at SoftLayer was an opportunity I couldn't turn down. I had to check my biases at the door, open my mind, and see how this side of the aisle does business. (And as if to remind me that a dogmatic shift has occurred in my professional life, Fox News continues to greet me every morning at the top of the stairs.)

To admit just some of my biases: How on earth did you build an enterprise-grade portal with a weakly typed language that doesn't require something as basic as a compiler? More importantly: Why? How does one work with such a thing? Some of you still use VI?? Seriously?

Fast forward about six months — just enough to say I am "proficient" in PHP and to have an exposure to the database side of things. The journey and rants are long and technical, but it should come as no surprise that I still prefer the Microsoft ecosystem over one based on PHP. I find it easier to work with, faster, and less error-prone than the alternative. The language is more structured, the tooling is better, and the framework better established and developer-oriented.

Humor me with this for a moment.

Assume for the sake of argument that my belief is correct — that Microsoft's offerings are indeed better than PHP's on every metric a developer can measure. If this is true, one might reasonably conclude that SoftLayer erred in its choice of development platform. Even though I will be the first to evangelize the virtues of the Microsoft ecosystem, I'll also be the first to say that this conclusion is wrong.

The conclusion is wrong because in asking the "Why?" in "Why SoftLayer chose the platform it did," I approached the question from the perspective of what's best for the developer. The question should have instead been phrased as: "What does SoftLayer's choice of platform say about our core values?"

It isn't exactly open source. Place the source code on any laptop, and you'll get the modern-day equivalent of summary execution: You will be fired.

It isn't developer convenience. It isn't needed. From what I've seen here, the developers have used their tools in a more extensive and architecturally correct way than I have in my time in the ASP.NET ecosystem.

The elusive answer can be summed up in one word: Independence. Fierce independence if you're into using superlatives.

While the Microsoft ecosystem may be the easiest on developers, it comes at price. Microsoft's ultimate responsibility is to the thousands of people that use its tools, so it has to steer its platforms in a way that fit the disparate needs of the many developers who rely on them. In relying on its own software, built on open-source offerings, SoftLayer can steer its platform in a way that benefits SoftLayer ... It has only its own needs to consider.

The soundness of this reality — and indeed, the necessity of being fully independent when one's core offering is the basic infrastructure that runs people's businesses should be obvious.

Very often we become overprotective of our dogmas, and fear that which we do not fully understand. To that end, I try to remember the words of an unlikely capitalist: "It doesn't matter if a cat is black or white, so long as it catches mice."

-George

Subscribe to open-source