Development Posts

June 5, 2014

Sysadmin Tips and Tricks - Understanding the 'Default Deny' Server Security Principle

In the desktop world, people tend to feel good about their system’s security when they have the latest anti-virus and anti-spyware installed and keep their applications up-to-date. Those of us who compute for a living know that this is nothing close to resembling a “secure” state. But it’s the best option for non-technical people at this time.

Servers, on the other hand, exist in a more hostile environment than desktop machines, which is why keeping them secure requires skilled professionals. This means not only doing things like keeping applications patched and up-to-date, but also grasping the underlying principles of system security. Doing that allows us to make informed and skillful decisions for our unique systems—because no one knows our servers as well as we do.

One very powerful concept is “Default Deny” (as in Deny by Default), by which means that "Everything, not explicitly permitted, is forbidden." What does this mean, and why is it important?

Let’s look at a simple example using file permissions. Let’s say you installed a CGI (Common Gateway Interface) application, such as some blog software, and you’re having trouble getting it to work. You’ve decided the problem is the permissions on the configuration file. In this case, user “rasto” is the owner of the file. You try chmodding it 755 and it works like this:

-rwxr-xr-x 1 rasto rasto 216 May 27 16:11 configuration.ini

Now that it works, you’re ready to move to your next project. But there’s a possible security problem here. As you can see, you have left the configuration file Readable and Executable by Other. There is almost certainly no reason for that because CGI scripts are typically run as the owner of the file. There is potentially no reason for users of the same group (or other random users of the system) to be able to Read this configuration file. After all, some configuration files contain database passwords. If I have access to another user on this system, I could simply “cat” the configuration file and get trivial access to your data!

So the trick is to find the least permissions required to run this script. With a little work, you may discover that it runs just fine with 700:

-rwx------ 1 rasto rasto 216 May 27 16:11 configuration.ini

By taking a little extra time, you have made your system much more secure. “Default Deny” means deny everything that is not explicitly required. The beautiful thing about this policy is that you can remove vulnerabilities that you neither comprehend nor even know about. Instead of making a list of “bad” things you essentially make a list of “good” things, and allow only those things to happen. You don’t even have to realize that someone could read the file because you’ve made it a policy to always allow the least amount of access possible to all things.

Another example might be to prune your php.ini to get rid of any expanded capabilities not required by PHP scripts running on your system. If a zero-day vulnerability arises in PHP that affects one of the things you’ve disallowed, it simply won’t affect you because you’ve disabled it by default.

Another scenario might be to remove packages from your system that are not being used. If someone tries to inject some Ruby code into your system, it won’t run without Ruby present. If you’re not using it, get rid of it, and it can’t be used against you.

Note: It’s very easy to be wrong about what is not being used on your system—you can definitely break things this way—I suggest a go-slow approach, particularly in regards to built-in packages.

The important thing is the concept. I hope you can see now why a Default Deny policy is a powerful tool to help keep your system more secure.

-Lee

June 3, 2014

My 5 Favorite Sublime Text 2 Plugins

I can’t believe that is was only a mere year ago since I learned of Sublime Text 2. I know, I know … where have I been? What kind of developer was I that I didn’t even know of Sublime Text? I’ll take the criticism, as I can honestly say it has been the best text editor I have ever used.

It’s extremely fast. I rarely wait for saves, uploads, or syntax highlighting, it keeps up with everything I do and allows me to develop directly from the keyboard. I hardly ever reach for my mouse!

It looks awesome. It has kind of retro-look for those developers who remember coding purely from the terminal or DOS. It really brings back memories.

It can be extended. Need some extra functionality that doesn’t come out-of-the-box? Sublime Text 2 has a range of available plugins that you can install to enhance your capabilities with this awesome text editor. In this blog, I’ll cover my top five favorite plugins of all time, what they do, and why they’re great!

1. BracketHighlighter

Many people believe that bracket highlighting should be a ready-made helper for developers of all languages. I agree on this point, however, at least Sublime Text 2 provides a plugin for this. It’s a very simple addition; it allows you to see if your opening brackets have an accompanying closing bracket. Many developers will tell you stories of these large and complex programs that consumed much of their time as they searched for one simple error … only to find that it was just a missing closing bracket.

In addition, it highlights opening and closing tags and quotes, for those of you who do a lot of HTML/XHTML, both bracket and tag settings are customizable.

For more details on the plugin check out the BracketHighlighter GitHub page.

2. DocBlockr

This is a neat plugin that speeds up and simplifies documentation. It supports PHP, Javascript, Java, Action Script, Objective C, C, C++, and Coffee Script.

By typing this:

/** (Press Enter)

The plugin automatically returns this:

/**
*
*/

Boom, the quickest way to document that I’ve ever seen.

In order to document your functions, just put it in your comment:

/** (Press Enter)
function myFunction(var1, var2) { }

And, it'll become:

/**
*[myFunction description]
* @param {[type]} var1 [description]
* @param {[type]} var2 [description]
* @param {[type]}
*/

function myFunction (var1, var2) { }

When you want to do variable documentation, the structure is similar:

/** (Press Enter)
myVar = 10

The plugin will fill out the documentation block like this:

/**
*[myVar description]
* @type {Number}
*/

Tell me that this isn’t nifty! If you want to try it out or just get a closer look at this plugin, head here.

3. Emmet (previously known as Zen Coding)

Unfortunately, I encountered some oddities when I tried to install Emmet with SublimeLinter, so I decided to disable the Linter in favor of Emmet to give it a spin. I absolutely love Emmet.

It provides a much more efficient way to code by providing what they call “abbreviations.” For example, if I want to create a div with an unordered list and one bullet point in it, Emmet lets me save myself a lot of time ... I can type this into Sublime:

div>ul>li

And press Control+E, and my code automatically turns into this:

<div>
    <ul>
        <li></li>
    </ul>
</div>

If I need to add multiple <li> tags, I can easily replicate them with a small addition:

div>ul>li*3

When I hit Control+E, voila! The unordered list structure is quickly generated:

<div>
    <ul>
        <li></li> 
        <li></li>
        <li></li>
    </ul>
</div>

That's just the tip of the iceberg when it comes to Emmet's functionality, and if you’re as impressed as I am, you should check out their site: http://docs.emmet.io/

4. SFTP

I think the title of the plugin says it all. It allows you to directly connect to your server and sync projects and files just by saving. You will never have to edit a file in a text editor, open your FTP client and upload the file again. Now, you can do it directly from Sublime Text 2.

When used in conjunction with Projects, you’ll find that you can easily save hours of time spent on remote uploading. By far, SFTP for Sublime Projects is one of the most essential plugins you’ll need for any project!

5. SideBarEnhancements

This is a small plugin that makes minor adjustments to the Files and Folders sidebar, providing a more intuitive interface. Though this doesn’t add much functionality, it can definitely speed things up. Take a look at the plugin on the SideBarEnhancements GitHub page

I hope this list of Sublime Text 2 plugins will enhance your capabilities and ease up your processes, as it has done for me. Give them a try and let me know what you think. Also, if you have a different favorite plugin, I’d love to hear about it.

-Cassandra

May 1, 2014

New App Release: SoftLayer Mobile for Windows 8.1

Today, the SoftLayer development team is launching a new platform accessibility tool for SoftLayer customers who want to easily manage their infrastructure from Windows. We've gotten a great response from the users of SoftLayer Mobile app for Windows Phone, so we turned our attention to creating an app for customers on Windows 8.1: SoftLayer Mobile for Windows 8.1.

With a growing number of users adopting and embracing Windows 8.1 on their PCs, and the Windows Store is becoming a vibrant community of useful apps for those customers. There are more than 145,000 apps on the Windows Store, and that number is expected to increase exponentially following Microsoft’s recent introduction of "Universal Apps for Windows Phone 8.1 and Windows 8.1.” With all that goodness and an expanding market, it was imperative for our mobile development team to build an app for customers using Windows 8.1 as their default OS or carrying Windows RT tablets.

Why Windows 8.1?

Our team wants to provide simple, efficient ways for customers to connect to SoftLayer infrastructure and perform any necessary management tasks while on-the-go. Our team is inspired by the power of connected devices in Windows ecosystem. By developing an app for Windows 8.1, we will slowly bring the phone, tablet and PC onto one streamlined platform — a concept many smart devices are adopting quickly.

What’s Fresh?

New Dashboard

The SoftLayer Mobile app for Windows 8.1 is a fresh new approach to its Windows Phone sibling. The app provides a dashboard view after authentication that provides a snapshot of some of the most commonly used information and controls in the portal.

Currently, the dashboard supports four different panels: Tickets, devices, accounting and bandwidth. All display an overview of relevant information for you and your environment. The dashboard also allows you to quickly add a ticket or make a one-time payment on your account.

SoftLayer Mobile for Windows 8.1

In-line Ticket Updates

In the new tickets module, you can update tickets without ever leaving the page. This functionality is similar to what you see on many social websites, and it's integrated to be seamless.

SoftLayer Mobile for Windows 8.1

Search Everywhere!

One of the coolest additions to the new app is the introduction of search functionality in each module. Now, you can search a ticket, a device, or an invoice by just typing into the search box! The search capability lets you spend less time scrolling and more time working.

SoftLayer Mobile for Windows 8.1

Bandwidth Display

Smart phones have apps that measure and report how much data you are using, and your infrastructure should be similarly transparent Bandwidth usage is an important aspect of server management, so we built the bandwidth module to show your infrastructure's public and private traffic for current and previous billing cycles. This view also helps you see when a server is about to reach its limits so that you can plan accordingly.

SoftLayer Mobile for Windows 8.1

The module provides two ways to look at the data:

  • In a tabular form by clicking the “Show/Hide Traffic Details” button.
  • In a graphic representation by clicking the “View Graph” button.

SoftLayer Mobile for Windows 8.1

Same Functionality. Better Experience.

Sometimes change is not always needed for a nicely crafted feature. The new app keeps the same feature richness of the Windows Phone app and arranges it in a user-friendly way. For example, in the devices module, you can navigate to between different tabs to get the information you need, from password lists and attached tickets to a specific device or monitoring alarms.

The “Remote Control” section on the module allows you to perform actions such as rebooting, power cycles, restarts and pinging servers. In addition, you can view hardware and software installed on the device along with the hardware and network components attached. In the current phone version, you can only see the root password for the device, but in the Windows 8.1 app, you see all passwords for the server.

SoftLayer Mobile for Windows 8.1

What's Next?

During the development of this app, the team's goal was to test to adopt a framework that would be ideal for scaling. More and more developers are adopting a Model-View-Model (MVVM) approach to mobile and web app development, so our goal was to use that approach for this project. The significant challenge we faced when adopting this approach was finding a well-supported framework that met our application's needs. We weren't able to find suitable frameworks that committed regular updates in SDKs or in APIs, so we ended up using the same MVVM principles without any underlying framework. In the end, the project allowed us to create our own framework for future projects!

There are many exciting features that are lined up for the Windows 8.1 app. Download it now: SoftLayer Mobile for Windows 8.1

After you try it out, please submit your feedback ... We want to keep improving the app by providing the features and functionality that matter most to you.

-Imran

April 23, 2014

Sysadmin Tips and Tricks - Stop Using Root!

A common mistake newer Linux system administrators make is the overuse of root. It seems so easy! Everything is so much simpler! But in the end, it’s not—and it’s only a matter of time before you wish you had not been so free and easy with your super-user, use. Let me try to convince you.

Let’s start with a little history. The antecedents of Linux go all the way back to the early 1970s, when computers cost tens of thousands of dollars (at least). With that kind of expense, you as a user would hardly have a computer sitting on your desk (not to mention they were at least refrigerator-sized), and you would also not have the use of it dedicated to your needs. What was obviously needed was an operating system that would allow multiple users to use the machine at once, via terminals, in order to make the most use of the computing resources available.

If you think about it, it’s clear that the operating system had to be very good at keeping users from being able to stomp on each other’s files and processes. So the early UNIX™ variants were multi-user systems from the get-go. In the ensuing forty years, these systems have only gotten better at keeping the various users and processes from harming each other. And this is the technology that you’re paying for when you use Linux or other modern variants.

Now, you may think, “That doesn’t apply to me—I’m the only user on my server!” But are you, really?

You probably run Apache, which is generally run as the user httpd or apache. Why not root? Because if you run Apache as root, then anyone on the outside who manages to get Apache to execute arbitrary code, would then have that code running as root! Next thing you know, they can execute "rm –rf /," or worse, invade your system altogether and steal proprietary information. By running as a non-root user, even if the attacker gets total access to that user, they are limited to what that user can touch. Thus, user httpd is compromised, but not the entire server.

The same thing is true for mail servers, FTP servers, and so on. They all rely on the Linux permissions system in order to give the programs access to as little as possible—ideally, only exactly what they need to do their jobs.

So, think of yourself as another process on the system. When you log in as your regular user, you are limited in what you can do. But this is not intended to harm you or irritate you—indeed; the system is designed to keep you from accidentally doing damage to your server.

For example, consider if you wanted to completely remove a directory called ‘home’ within your home directory. Note the ever so slight difference between the first command:

rm –R home

And the second command:

rm –R /home

The first command removes a directory called ‘home’ from wherever you happen to be sitting on the file system. The second removes all users’ home directories from the system. One little slash makes all the difference in the world. This is probably why it has been said that Linux gives you enough rope to hang yourself with. Executing the second command as root looks like this:

server:# rm –R /home 
server.com#

And it’s just gone! Whereas if you accidentally put that slash in there while logged in as your user, you would get:

server:# rm –R /home 
server:# rm: cannot remove `home’: Permission denied

This will annoy you, until you realize that if you’d done it as root you would have wiped out all your customers home directories.

In short, just like the processes that run on your machine, you would be well served to use only the permissions you need. This is why many Linux distributions today encourage the use of sudo—you don’t even become root, but just execute things as root when needed. It’s a good policy, and makes the best use of four decades of expertise that have gone into the system you are using.

- Lee

P.S. This is also why you pretty much never want to chmod 777 anything!

January 10, 2014

Platform Improvements: VLAN Management

As director of product development, I'm tasked with providing SoftLayer customers greater usability and self-service tools on our platform. Often, that challenge involves finding, testing, and introducing new products, but a significant amount of my attention focuses on internal projects to tweak and improve our existing products and services. To give you an idea of what that kind of "behind the scenes" project looks like, I'll fill you in on a few of the updates we recently rolled out to improve the way customers interact with and manage their Virtual LANs (VLANs).

VLANs play a significant role in SoftLayer's platform. In the most basic sense, VLANs fool servers into thinking they're behind the same network switch. If you have multiple servers in the same data center and behind the same router, you could have them all on the same VLAN, and all traffic between the servers would be handled at the layer-2 network level. For customers with multi-tier applications, zones can be created to isolate specific servers into separate VLANs — database servers, app servers, and Web servers can all be isolated in their own security partitions to meet specific security and/or compliance requirements.

In the past, VLANs were all issued distinct numbers so that we could logically and consistently differentiate them from each other. That utilitarian approach has proven to be functional, but we noticed an opportunity to make the naming and management of VLANs more customer-friendly without losing that functionality. Because many of our customers operate large environments with multiple VLANs, they've had the challenge of remembering which servers live behind which VLAN number, and the process of organizing all of that information was pretty daunting. Imagine an old telephone switchboard with criss-crossing wires connecting several numbered jacks (and not connecting others). This is where our new improvements come in.

Customers now have the ability to name their VLANs, and we've made updates that increase visibility into the resources (servers, firewalls, gateways, and subnets) that reside inside specific VLANs. In practice, that means you can name your VLAN that houses database servers "DB" or label it to pinpoint a specific department inside your organization. When you need to find one of those VLANs, you can easily search for it by name and make changes to it easily.

VLAN List View

VLAN Naming

VLAN Detail Page

VLAN Naming

While these little improvements may seem simple, they make life much easier for IT departments and sysadmins with large, complex environments. If you don't need this kind of functionality, we don't throw it in your face, but if you do need it, we make it clear and easily accessible.

If you ever come across quirks in the portal that you'd like us to address, please let us know. We love making big waves by announcing new products and services, but we get as much (or more) joy from finding subtle ways to streamline and improve the way our customers interact with our platform.

-Bryce

November 14, 2013

Enhancing Usability by Building User Confidence

Consider your experiences with web applications, and see if this scenario seems familiar: Your electricity bill has some incorrect charges on it. Fearing that you will have to spend 40 minutes on hold if you call in, you find that the electric company website has a support center where you can submit billing issues and questions; you are saved! You carefully fill out the form with your sixteen-digit account number and detailed description of the incorrect charges. You read it over and click the submit button. Your page goes blank for a couple of seconds, the form comes back with a note saying you typed in your phone number incorrectly, and the detailed description you spent eleven minutes meticulously writing is gone.

Web applications have gotten much better at preventing these kinds of user experiences over the past few years, and I'm sure that none of your applications have this problem (if they do, fix it right now!), but "usability" is more than just handling errors gracefully. Having a seamless process is only half the battle when it comes to giving your users a great experience with your application. The other half of the battle is a much more subjective: Your users need to feel confident in their success every step of the way. By keeping a few general guidelines in mind, you can instill confidence in your users so that they feel positive about your application from start to finish with whatever they are trying to accomplish.

1. Keep the user in a familiar context.

As the user in our electric company support application example, let's assume the process works and does not lose any of my information. I have to have faith that the application is going to do what I expect it to do when the page refreshes. Faith and unfamiliar technology do not exactly go hand in hand. Instead of having the form submit with a page refresh, the site's developers could introduce a progress wheel or other another kind of indicator that shows the data is being submitted while the content is still visible. If detailed content never goes away during the submission process, I'm confident that I still have access to my information.

Another example of the same principle is the use of modal windows. Modal windows are presented on top of a previous page, so users have a clear way of going back if they get confused or decide they navigated to the wrong place. By providing this new content on top of a familiar page, users are much less likely to feel disoriented if they get stuck or lost, and they will feel more confident when they're using the application.

2. Reassure the user with immediate feedback.

By communicating frequently and clearly, users are reassured, and they are much less likely to become anxious. Users want to see their actions get a response from your application. In our electric company support application example, imagine how much better the experience would be if a small blurb was displayed in red next to the phone number text box when I typed in my phone number in the wrong format. The immediate feedback would pinpoint the problem when it is easy to correct, and it would make me confident that when the phone number is updated, the application will continue to work as expected.

3. Provide warnings or extra information for dangerous or complicated operations.

When users are new to an application, they are not always sure which actions will have negative consequences. This is another great opportunity for communication. Providing notices or alerts for important or risky operations can offer a good dose of hesitation for new users who aren't prepared. Effective warnings or notices will tell the user when they will want to perform this action or what the negative consequences might be, so the user can make an informed decision. Users are confident with informed decisions because a lack information causes anxiety.

I learned how to implement this tip when I designed a wizard system for a previous employer that standardized how the company's application would walked users through any step-by-step process. My team decided early on to standardize a review step at the end of any implemented wizard. This was an extra step that every user had to go through for every wizard in the application, but it made all of the related processes much more usable and communicative. This extra information gave the users a chance to see the totality of the operation they were performing, and it gave them a chance to correct any mistakes. Implementing this tip resulted in users who were fully informed and confident throught the process of very complicated operations.

4. Do not assume your users know your terminology, and don't expect them to learn it.

Every organization has its own language. I have never encountered an exception to this rule. It cannot be helped! Inside your organization, you come up with a defined vocabulary for referencing the topics you have to work with every day, but your users won't necessarily understand the terminology you use internally. Some of your ardent users pick up on your language through osmosis, but the vast majority of users just get confused when they encounter terms they are not familiar with.

When interacting with users, refrain from using any of your internal language, and strictly adhere to a universally-accepted vocabulary. In many cases, you need shorthand to describe complex concepts that users will already understand. In this situation, always use universal or industry-wide vocabulary if it is available.

This practice can be challenging and will often require extra work. Let's say you have a page in your application dealing with "display devices," which could either be TVs or monitors. All of your employees talk about display devices because to your organization, they are essentially the same thing. The technology of your application handles all display devices in exactly the same way, so as good software designers you have this abstracted (or condensed for non-technical people) so that you have the least amount of code possible. The easiest route is to just have a page that talks about display devices. The challenge with that approach is that your users understand what monitors and TVs are, but they don't necessarily think of those as display devices.

If that's the case, you should use the words "monitors" and "TVs" when you're talking about display devices externally. This can be difficult, and it requires a lot of discipline, but when you provide familiar terminology, users won't be disoriented by basic terms. To make users more comfortable, speak to them in their language. Don't expect them to learn yours, because most of them won't.

When you look at usability through the subjective lens of user confidence, you'll find opportunities to enhance your user experience ... even when you aren't necessarily fixing anything that's broken. While it's difficult to quantify, confidence is at the heart of what makes people like or dislike any product or tool. Pay careful attention to the level of confidence your users have throughout your application, and your application can reach new heights.

-Tony

November 11, 2013

Sysadmin Tips and Tricks - Using the ‘for’ Loop in Bash

Ever have a bunch of files to rename or a large set of files to move to different directories? Ever find yourself copy/pasting nearly identical commands a few hundred times to get a job done? A system administrator's life is full of tedious tasks that can be eliminated or simplified with the proper tools. That's right ... Those tedious tasks don't have to be executed manually! I'd like to introduce you to one of the simplest tools to automate time-consuming repetitive processes in Bash — the for loop.

Whether you have been programming for a few weeks or a few decades, you should be able to quickly pick up on how the for loop works and what it can do for you. To get started, let's take a look at a few simple examples of what the for loop looks like. For these exercises, it's always best to use a temporary directory while you're learning and practicing for loops. The command is very powerful, and we wouldn't want you to damage your system while you're still learning.

Here is our temporary directory:

rasto@lmlatham:~/temp$ ls -la
total 8
drwxr-xr-x 2 rasto rasto 4096 Oct 23 15:54 .
drwxr-xr-x 34 rasto rasto 4096 Oct 23 16:00 ..
rasto@lmlatham:~/temp$

We want to fill the directory with files, so let's use the for loop:

rasto@lmlatham:~/temp$ for cats_are_cool in {a..z}; do touch $cats_are_cool; done;
rasto@lmlatham:~/temp$

Note: This should be typed all in one line.

Here's the result:

rasto@lmlatham:~/temp$ ls -l
total 0
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 a
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 b
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 c
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 d
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 e
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 f
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 g
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 h
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 i
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 j
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 k
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 l
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 m
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 n
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 o
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 p
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 q
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 r
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 s
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 t
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 u
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 v
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 w
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 x
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 y
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 z
rasto@lmlatham:~/temp$

How did that simple command populate the directory with all of the letters in the alphabet? Let's break it down.

for cats_are_cool in {a..z}

The for is the command we are running, which is built into the Bash shell. cats_are_cool is a variable we are declaring. The specific name of the variable can be whatever you want it to be. Traditionally people often use f, but the variable we're using is a little more fun. Hereafter, our variable will be referred to as $cats_are_cool (or $f if you used the more boring "f" variable). Aside: You may be familiar with declaring a variable without the $ sign, and then using the $sign to invoke it when declaring environment variables.

When our command is executed, the variable we declared in {a..z}, will assume each of the values of a to z. Next, we use the semicolon to indicate we are done with the first phase of our for loop. The next part starts with do, which say for each of a–z, do <some thing>. In this case, we are creating files by touching them via touch $cats_are_cool. The first time through the loop, the command creates a, the second time through b and so forth. We complete that command with a semicolon, then we declare we are finished with the loop with "done".

This might be a great time to experiment with the command above, making small changes, if you wish. Let's do a little more. I just realized that I made a mistake. I meant to give the files a .txt extension. This is how we'd make that happen:

for dogs_are_ok_too in {a..z}; do mv $dogs_are_ok_too $dogs_are_ok_too.txt; done;
Note: It would be perfectly okay to re-use $cats_are_cool here. The variables are not persistent between executions.

As you can see, I updated the command so that a would be renamed a.txt, b would be renamed b.txt and so forth. Why would I want to do that manually, 26 times? If we check our directory, we see that everything was completed in that single command:

rasto@lmlatham:~/temp$ ls -l
total 0
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 a.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 b.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 c.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 d.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 e.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 f.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 g.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 h.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 i.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 j.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 k.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 l.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 m.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 n.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 o.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 p.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 q.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 r.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 s.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 t.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 u.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 v.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 w.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 x.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 y.txt
-rw-rw-r-- 1 rasto rasto 0 Oct 23 16:13 z.txt
rasto@lmlatham:~/temp$

Now we have files, but we don't want them to be empty. Let's put some text in them:

for f in `ls`; do cat /etc/passwd > $f; done

Note the backticks around ls. In Bash, backticks mean, "execute this and return the results," so it's like you executed ls and fed the results to the for loop! Next, cat /etc/passwd is redirecting the results to $f, in filenames a.txt, b.txt, etc. Still with me?

So now I've got a bunch of files with copies of /etc/passwd in them. What if I never wanted files for a, g, or h? First, I'd get a list of just the files I want to get rid of:

rasto@lmlatham:~/temp$ ls | egrep 'a|g|h'
a.txt
g.txt
h.txt

Then I could plug that command into the for loop (using backticks again) and do the removal of those files:

for f in `ls | egrep 'a|g|h'`; do rm $f; done

I know these examples don't seem very complex, but they give you a great first-look at the kind of functionality made possible by the for loop in Bash. Give it a whirl. Once you start smartly incorporating it in your day-to-day operations, you'll save yourself massive amounts of time ... Especially when you come across thousands or tens of thousands of very similar tasks.

Don't do work a computer should do!

-Lee

September 24, 2013

Four Rules for Better Code Documentation

Last month, Jeremy shared some valuable information regarding technical debt on SLDN. In his post, he discussed how omitting pertinent information when you're developing for a project can cause more work to build up in the future. One of the most common areas developers overlook when it comes to technical debt is documentation. This oversight comes in two forms: A complete omission of any documentation and inadequate information when documentation does exist. Simply documenting the functionality of your code is a great start, but the best way to close the information gap and avoid technical debt that stems from documentation (or lack thereof) is to follow four simple rules.

1. Know Your Audience

When we're talking about code, it's safe to say you'll have a fairly technical audience; however, it is important to note the level of understanding your audience has on the code itself. While they should be able to grasp common terms and development concepts, they may be unfamiliar with the functionality you are programming. Because of this, it's a good idea to provide a link to an internal, technical knowledgebase or wiki that will provide in-depth details on the functionality of the technology they'll be working with. We try to use a combination of internal and external references that we think will provide the most knowledge to developers who may be looking at our code. Here's an example of that from our Dns_Domain class:

 * @SLDNDocumentation Service Overview <<< EOT
 * SoftLayer customers have the option of hosting DNS domains on the SoftLayer
 * name servers. Individual domains hosted on the SoftLayer name servers are
 * handled through the SoftLayer_Dns_Domain service.
 *
 * Domain changes are applied automatically by our nameservers, but changes may
 * not be received by the other name servers on the Internet for 72 hours after
 * your change. The SoftLayer_Dns_Domain service does not apply to customers who
 * run their own nameservers on servers purchased from SoftLayer.
 *
 * SoftLayer provides secondary DNS hosting services if you wish to maintain DNS
 * records on your name server, but have records replicated on SoftLayer's name
 * servers. Use the [[SoftLayer_Dns_Secondary]] service to manage secondary DNS
 * zones and transfers.
 * EOT
 *
 * @SLDNDocumentation Service External Link http://en.wikipedia.org/wiki/Domain_name_system Domain Name System at Wikipedia
 * @SLDNDocumentation Service External Link http://tools.ietf.org/html/rfc1035 RFC1035: Domain Names - Implementation and Specification at ietf.org
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Domain_ResourceRecord
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Domain_Reverse
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Secondary
 *

Enabling the user to learn more about a topic, product, or even a specific call alleviates the need for users to ask multiple questions regarding the "what" or "why" and will also minimize the need for you to explain more basic concepts regarding the technology supported by your code.

2. Be Consistent - Terminology

There are two main areas developers should focus on when it comes to consistency: Formatting and terminology.

Luckily, formatting is pretty simple. Most languages have a set of standards attached to them that extend to the Docblock, which is where the documentation portion of the code normally takes place. Docblocks can be used to provide an overview of the class, identify authors or product owners and provide additional reference to those using the code. The example below uses PHP's standards for documentation tagging and allows users to quickly identify the parameters and return value for the createObject method in the Dns_Domain class:

*
     * @param string $objectType
     * @param object $templateObject
     *
     * @return SoftLayer_Dns_Domain
     */
   public static function createObject($objectType = __CLASS__, $templateObject)

Keeping consistent when it comes to terminology is a bit more difficult; especially if there have been no standards in place before. As an example, we can look to one of the most common elements of hosting: the server. Some people call this a "box," a "physical instance" or simply "hardware." The server may be a name server, a mail server, a database server or a web server.

If your company has adopted a term, use that term. If they haven't, decide on a term with your coworkers and stick to it. It's important to be as specific as possible in your documentation to avoid any confusion, and when you adopt specific terms in your documentation, you'll also find that this consistency will carry over into conversations and meetings. As a result, training new team members on your code will go more smoothly, and it will be easier for other people to assist in maintaining your code's documentation.

Bonus: It's much easier to search and replace when you only have to search for one term.

3. Forget What You Know About Your Code ... But Only Temporarily

Regardless of the industry, people who write their own documentation tend to omit pertinent information about the topic. When I train technical writers, I use the peanut butter and jelly example: How would you explain the process of making a peanut butter and jelly sandwich? Many would-be instructors omit things that would result in a very poorly made sandwich ... if one could be made at all. If you don't tell the reader to get the jelly from the cupboard, how can they put jelly on the sandwich? It's important to ask yourself when writing, "Is there anything that I take for granted about this piece of code that other users might need or want to know?"

Think about a coding example where a method calls one or more methods automatically in order to do its job or a method acts like another method. In our API, the createObjects method uses the logic of the createObject method that we just discussed. While some developers may pick up on the connection based on the method's name, it is still important to reference the similarities so they can better understand exactly how the code works. We do this in two ways: First, we state that createObjects follows the logic of createObject in the overview. Second, we note that createObject is a related method. The code below shows exactly how we've implemented this:

     * @SLDNDocumentation Service Description Create multiple domains at once.
     *
     * @SLDNDocumentation Method Overview <<< EOT
     * Create multiple domains on the SoftLayer name servers. Each domain record
     * passed to ''createObjects'' follows the logic in the SoftLayer_Dns_Domain
     * ''createObject'' method.
     * EOT
     *
     * @SLDNDocumentation Method Associated Method SoftLayer_Dns_Domain::createObject

4. Peer Review

The last rule, and one that should not be skipped, is to always have a peer look over your documentation. There really isn't a lot of depth behind this one. In Development, we try to peer review documentation during the code review process. If new content is written during code changes or additions, developers can add content reviewers, who have the ability to add notes with revisions, suggestions and questions. Once all parties are satisfied with the outcome, we close out the review in the system and the content is updated in the next code release. With peer review of documentation, you'll catch typos, inconsistencies and gaps. It always helps to have a second set of eyes before your content hits its users.

Writing better documentation really is that easy: Know your audience, be consistent, don't take your knowledge for granted, and use the peer review process. I put these four rules into practice every day as a technical writer at SoftLayer, and they make my life so much easier. By following these rules, you'll have better documentation for your users and will hopefully eliminate some of that pesky technical debt.

Go, and create better documentation!

-Sarah

September 20, 2013

Building a Mobile App with jQuery Mobile: The Foundation

Based on conversations I've had in the past, at least half of web developers I've met have admitted to cracking open an Objective-C book at some point in their careers with high hopes of learning mobile development ... After all, who wouldn't want to create "the next big thing" for a market growing so phenomenally every year? I count myself among that majority: I've been steadily learning Objective-C over the past year, dedicating a bit of time every day, and I feel like I still lack skill-set required to create an original, complex application. Wouldn't it be great if we web developers could finally get our shot in the App Store without having to unlearn and relearn the particulars of coding a mobile application?

Luckily for us: There is!

The rock stars over at jQuery have created a framework called jQuery Mobile that allows developers to create cross-platform, responsive applications on a HTML5-based jQuery foundation. The framework allows for touch and mouse event support, so you're able to publish across multiple platforms, including iOS, Android, Blackberry, Kindle, Nook and on and on and on. If you're able to create web applications with jQuery, you can now create an awesome cross-platform app. All you have to do is create an app as if it was a dynamic HTML5 web page, and jQuery takes care of the rest.

Let's go through a real-world example to show this functionality in action. The first thing we need to do is fill in the <head> content with all of our necessary jQuery libraries:

<!DOCTYPE html>
<html>
<head>
    <title>SoftLayer Hello World!</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="http://code.jquery.com/mobile/1.3.2/jquery.mobile-1.3.2.min.css" />
    <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script>
    <script src="http://code.jquery.com/mobile/1.3.2/jquery.mobile-1.3.2.min.js"></script>
</head>

Now let's create a framework for our simplistic app in the <body> section of our page:

<body>
    <div data-role="page">
        <div data-role="header">
            <h1>My App!</h1>
        </div>
 
        <div data-role="content">
            <p>This is my application! Pretty cool, huh?</p>
        </div>
 
        <div data-role="footer">
            <h1>Bottom Footer</h1>
        </div>
 
    </div>
</body>
</html>

Even novice web developers should recognize the structure above. You have a header, content and a footer just as you would in a regular web page, but we're letting jQuery apply some "native-like" styling to those sections with the data-role attributes. This is what our simple app looks like so far: jQuery Mobile App Screenshot #1

While it's not very fancy (yet), you see that the style is well suited to the iPhone I'm using to show it off. Let's spice it up a bit and add a navigation bar. Since we want the navigation to be a part of the header section of our app, let's add an unordered list there:

<div data-role="header">
    <h1>My App!</h1>
        <div data-role="navbar">
            <ul>
                <li><a href="#home" class="ui-btn-active" data-icon="home" data-theme="b">Home</a></li>
                <li><a href="#softlayer_cool_news" data-icon="grid" data-theme="b">SL Cool News!</a></li>
                <li><a href="#softlayer_cool_stuff" data-icon="star" data-theme="b">SL Cool Stuff!</a></li>
            </ul>
        </div>
    </div>

You'll notice again that it's not much different from regular HTML. We've created a navbar div with an unordered list of menu items we'd like to add to the header: Home, SL Cool News and SL Cool Stuff. Notice in the anchor tag of each that there's an attribute called data-icon which defines which graphical icon we want to represent the navigation item. Let's have a peek at what it looks like now: jQuery Mobile App Screenshot #2

Our app isn't doing a whole lot yet, but you can see from our screenshot that the pieces are starting to come together nicely. Because we're developing our mobile app as an HTML5 app first, we're able to make quick changes and see those changes in real time from our phone's browser. Once we get the functionality we want to into our app, we can use a tool such as PhoneGap or Cordova to package our app into a ready-to-use standalone iPhone app (provided you're enrolled in the Apple Development Program, of course), or we can leave the app as-is for a very nifty mobile browser application.

In my next few blogs, I plan to expand on this topic by showing you some of the amazingly easy (and impressive) functionality available in jQuery Mobile. In the meantime, go grab a copy of jQuery Mobile and start playing around with it!

-Cassandra

September 16, 2013

Sysadmin Tips and Tricks - Using strace to Monitor System Calls

Linux admins often encounter rogue processes that die without explanation, go haywire without any meaningful log data or fail in other interesting ways without providing useful information that can help troubleshoot the problem. Have you ever wished you could just see what the program is trying to do behind the scenes? Well, you can — strace (system trace) is very often the answer. It is included with most distros' package managers, and the syntax should be pretty much identical on any Linux platform.

First, let's get rid of a misconception: strace is not a "debugger," and it isn't a programmer's tool. It's a system administrator's tool for monitoring system calls and signals. It doesn't involve any sophisticated configurations, and you don't have to learn any new commands ... In fact, the most common uses of strace involve the bash commands you learned the early on:

  • read
  • write
  • open
  • close
  • stat
  • fork
  • execute (execve)
  • chmod
  • chown

 

You simply "attach" strace to the process, and it will display all the system calls and signals resulting from that process. Instead of executing the command's built-in logic, strace just makes the process's normal calls to the system and returns the results of the command with any errors it encountered. And that's where the magic lies.

Let's look an example to show that behavior in action. First, become root — you'll need to be root for strace to function properly. Second, make a simple text file called 'test.txt' with these two lines in it:

# cat test.txt
Hi I'm a text file
there are only these two lines in me.

Now, let's execute the cat again via strace:

$ strace cat test.txt 
execve("/bin/cat", ["cat", "test.txt"], [/* 22 vars */]) = 0
brk(0)  = 0x9b7b000
uname({sys="Linux", node="ip-208-109-127-49.ip.secureserver.net", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=30671, ...}) = 0
mmap2(NULL, 30671, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f35000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000_\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1594552, ...}) = 0
mmap2(NULL, 1320356, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7df2000
mmap2(0xb7f2f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13c) = 0xb7f2f000
mmap2(0xb7f32000, 9636, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f32000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7df1000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7df0000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7df1b80, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7f2f000, 8192, PROT_READ) = 0
mprotect(0xb7f57000, 4096, PROT_READ) = 0
munmap(0xb7f35000, 30671) = 0
brk(0)  = 0x9b7b000
brk(0x9b9c000) = 0x9b9c000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
open("test.txt", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0
read(3, "Hi I'm a text file\nthere are onl"..., 4096) = 57
write(1, "Hi I'm a text file\nthere are onl"..., 57Hi I’m a text file
there are only these two lines in me.
) = 57
read(3, "", 4096) = 0
close(3) = 0
close(1) = 0
exit_group(0) = ?

Now that return may look really arcane, but if you study it a little bit, you'll see that it includes lots of information that even an ordinary admin can easily understand. The first line returned includes the execve system call where we'd execute /bin/cat with the parameter of test.txt. After that, you'll see the cat binary attempt to open some system libraries, and the brk and mmap2 calls to allocate memory. That stuff isn't usually particularly useful in the context we're working in here, but it's important to understand what's going on. What we're most interested in are often open calls:

open("test.txt", O_RDONLY|O_LARGEFILE) = 3

It looks like when we run cat test.txt, it will be opening "test.txt", doesn't it? In this situation, that information is not very surprising, but imagine if you are in a situation were you don't know what files a given file is trying to open ... strace immediately makes life easier. In this particular example, you'll see that "= 3" at the end, which is a temporary sort of "handle" for this particular file within the strace output. If you see a "read" call with '3' as the first parameter after this, you know it's reading from that file:

read(3, "Hi I'm a text file\nthere are onl"..., 4096) = 57

Pretty interesting, huh? strace defaults to just showing the first 32 or so characters in a read, but it also lets us know that there are 57 characters (including special characters) in the file! After the text is read into memory, we see it writing it to the screen, and delivering the actual output of the text file. Now that's a relatively simplified example, but it helps us understand what's going on behind the scenes.

Real World Example: Finding Log Files

Let's look at a real world example where we'll use strace for a specific purpose: You can't figure out where your Apache logs are being written, and you're too lazy to read the config file (or perhaps you can't find it). Wouldn't it be nice to follow everything Apache is doing when it starts up, including opening all its log files? Well you can:

strace -Ff -o output.txt -e open /etc/init.d/httpd restart

We are executing strace and telling it to follow all forks (-Ff), but this time we'll output to a file (-o output.txt) and only look for 'open' system calls to keep some of the chaff out of the output (-e open), and execute '/etc/init.d/httpd restart'. This will create a file called "output.txt" which we can use to find references to our log files:

#cat output.txt | grep log
[pid 13595] open("/etc/httpd/modules/mod_log_config.so", O_RDONLY) = 4
[pid 13595] open("/etc/httpd/modules/mod_logio.so", O_RDONLY) = 4
[pid 13595] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 10
[pid 13595] open("/etc/httpd/logs/ssl_error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 11
[pid 13595] open("/etc/httpd/logs/access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 12
[pid 13595] open("/etc/httpd/logs/cm4msaa7.com", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 13
[pid 13595] open("/etc/httpd/logs/ssl_access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 14
[pid 13595] open("/etc/httpd/logs/ssl_request_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 15
[pid 13595] open("/etc/httpd/modules/mod_log_config.so", O_RDONLY) = 9
[pid 13595] open("/etc/httpd/modules/mod_logio.so", O_RDONLY) = 9
[pid 13596] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 10
[pid 13596] open("/etc/httpd/logs/ssl_error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 11
open("/etc/httpd/logs/access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 12
open("/etc/httpd/logs/cm4msaa7.com", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 13
open("/etc/httpd/logs/ssl_access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 14
open("/etc/httpd/logs/ssl_request_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 15

The log files jump out at you don't they? Because we know that Apache will want to open its log files when it starts, all we have to do is we follow all the system calls it makes when it starts, and we'll find all of those files. Easy, right?

Real World Example: Locating Errors and Failures

Another valuable use of strace involves looking for errors. If a program fails when it makes a system call, you'll want to be able pinpoint any errors that might have caused that failure as you troubleshoot. In all cases where a system call fails, strace will return a line with "= -1" in the output, followed by an explanation. Note: The space before -1 is very important, and you'll see why in a moment.

For this example, let's say Apache isn't starting for some reason, and the logs aren't telling ua anything about why. Let's run strace:

strace -Ff -o output.txt -e open /etc/init.d/httpd start

Apache will attempt to restart, and when it fails, we can grep our output.txt for '= -1' to see any system calls that failed:

$ cat output.txt | grep '= -1'
[pid 13748] open("/etc/selinux/config", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/i686/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/i686/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/i686/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/i686/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libutil.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/etc/gai.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = -1 EACCES (Permission denied)

With experience, you'll come to understand which errors matter and which ones don't. Most often, the last error is the most significant. The first few lines show the program trying different libraries to see if they are available, so they don't really matter to us in our pursuit of what's going wrong with our Apache restart, so we scan down and find that the last line:

[pid 13748] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = -1 EACCES (Permission denied)

After a little investigation on that file, I see that some maniac as set the immutable attribute:

lsattr /etc/httpd/logs/error_log
----i-------- /etc/httpd/logs/error_log

Our error couldn't be found in the log file because Apache couldn't open it! You can imagine how long it might take to figure out this particular problem without strace, but with this useful tool, the cause can be found in minutes.

Go and Try It!

All major Linux distros have strace available — just type strace at the command line for the basic usage. If the command is not found, install it via your distribution's package manager. Get in there and try it yourself!

For a fun first exercise, bring up a text editor in one terminal, then strace the editor process in another with the -p flag (strace -p <process_id>) since we want to look at an already-running process. When you go back and type in the text editor, the system calls will be shown in strace as you type ... You see what's happening in real time!

-Lee

Subscribe to development