Posts Tagged 'Developers'

September 24, 2013

Four Rules for Better Code Documentation

Last month, Jeremy shared some valuable information regarding technical debt on SLDN. In his post, he discussed how omitting pertinent information when you're developing for a project can cause more work to build up in the future. One of the most common areas developers overlook when it comes to technical debt is documentation. This oversight comes in two forms: A complete omission of any documentation and inadequate information when documentation does exist. Simply documenting the functionality of your code is a great start, but the best way to close the information gap and avoid technical debt that stems from documentation (or lack thereof) is to follow four simple rules.

1. Know Your Audience

When we're talking about code, it's safe to say you'll have a fairly technical audience; however, it is important to note the level of understanding your audience has on the code itself. While they should be able to grasp common terms and development concepts, they may be unfamiliar with the functionality you are programming. Because of this, it's a good idea to provide a link to an internal, technical knowledgebase or wiki that will provide in-depth details on the functionality of the technology they'll be working with. We try to use a combination of internal and external references that we think will provide the most knowledge to developers who may be looking at our code. Here's an example of that from our Dns_Domain class:

 * @SLDNDocumentation Service Overview <<< EOT
 * SoftLayer customers have the option of hosting DNS domains on the SoftLayer
 * name servers. Individual domains hosted on the SoftLayer name servers are
 * handled through the SoftLayer_Dns_Domain service.
 *
 * Domain changes are applied automatically by our nameservers, but changes may
 * not be received by the other name servers on the Internet for 72 hours after
 * your change. The SoftLayer_Dns_Domain service does not apply to customers who
 * run their own nameservers on servers purchased from SoftLayer.
 *
 * SoftLayer provides secondary DNS hosting services if you wish to maintain DNS
 * records on your name server, but have records replicated on SoftLayer's name
 * servers. Use the [[SoftLayer_Dns_Secondary]] service to manage secondary DNS
 * zones and transfers.
 * EOT
 *
 * @SLDNDocumentation Service External Link http://en.wikipedia.org/wiki/Domain_name_system Domain Name System at Wikipedia
 * @SLDNDocumentation Service External Link http://tools.ietf.org/html/rfc1035 RFC1035: Domain Names - Implementation and Specification at ietf.org
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Domain_ResourceRecord
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Domain_Reverse
 * @SLDNDocumentation Service See Also SoftLayer_Dns_Secondary
 *

Enabling the user to learn more about a topic, product, or even a specific call alleviates the need for users to ask multiple questions regarding the "what" or "why" and will also minimize the need for you to explain more basic concepts regarding the technology supported by your code.

2. Be Consistent - Terminology

There are two main areas developers should focus on when it comes to consistency: Formatting and terminology.

Luckily, formatting is pretty simple. Most languages have a set of standards attached to them that extend to the Docblock, which is where the documentation portion of the code normally takes place. Docblocks can be used to provide an overview of the class, identify authors or product owners and provide additional reference to those using the code. The example below uses PHP's standards for documentation tagging and allows users to quickly identify the parameters and return value for the createObject method in the Dns_Domain class:

*
     * @param string $objectType
     * @param object $templateObject
     *
     * @return SoftLayer_Dns_Domain
     */
   public static function createObject($objectType = __CLASS__, $templateObject)

Keeping consistent when it comes to terminology is a bit more difficult; especially if there have been no standards in place before. As an example, we can look to one of the most common elements of hosting: the server. Some people call this a "box," a "physical instance" or simply "hardware." The server may be a name server, a mail server, a database server or a web server.

If your company has adopted a term, use that term. If they haven't, decide on a term with your coworkers and stick to it. It's important to be as specific as possible in your documentation to avoid any confusion, and when you adopt specific terms in your documentation, you'll also find that this consistency will carry over into conversations and meetings. As a result, training new team members on your code will go more smoothly, and it will be easier for other people to assist in maintaining your code's documentation.

Bonus: It's much easier to search and replace when you only have to search for one term.

3. Forget What You Know About Your Code ... But Only Temporarily

Regardless of the industry, people who write their own documentation tend to omit pertinent information about the topic. When I train technical writers, I use the peanut butter and jelly example: How would you explain the process of making a peanut butter and jelly sandwich? Many would-be instructors omit things that would result in a very poorly made sandwich ... if one could be made at all. If you don't tell the reader to get the jelly from the cupboard, how can they put jelly on the sandwich? It's important to ask yourself when writing, "Is there anything that I take for granted about this piece of code that other users might need or want to know?"

Think about a coding example where a method calls one or more methods automatically in order to do its job or a method acts like another method. In our API, the createObjects method uses the logic of the createObject method that we just discussed. While some developers may pick up on the connection based on the method's name, it is still important to reference the similarities so they can better understand exactly how the code works. We do this in two ways: First, we state that createObjects follows the logic of createObject in the overview. Second, we note that createObject is a related method. The code below shows exactly how we've implemented this:

     * @SLDNDocumentation Service Description Create multiple domains at once.
     *
     * @SLDNDocumentation Method Overview <<< EOT
     * Create multiple domains on the SoftLayer name servers. Each domain record
     * passed to ''createObjects'' follows the logic in the SoftLayer_Dns_Domain
     * ''createObject'' method.
     * EOT
     *
     * @SLDNDocumentation Method Associated Method SoftLayer_Dns_Domain::createObject

4. Peer Review

The last rule, and one that should not be skipped, is to always have a peer look over your documentation. There really isn't a lot of depth behind this one. In Development, we try to peer review documentation during the code review process. If new content is written during code changes or additions, developers can add content reviewers, who have the ability to add notes with revisions, suggestions and questions. Once all parties are satisfied with the outcome, we close out the review in the system and the content is updated in the next code release. With peer review of documentation, you'll catch typos, inconsistencies and gaps. It always helps to have a second set of eyes before your content hits its users.

Writing better documentation really is that easy: Know your audience, be consistent, don't take your knowledge for granted, and use the peer review process. I put these four rules into practice every day as a technical writer at SoftLayer, and they make my life so much easier. By following these rules, you'll have better documentation for your users and will hopefully eliminate some of that pesky technical debt.

Go, and create better documentation!

-Sarah

September 20, 2013

Building a Mobile App with jQuery Mobile: The Foundation

Based on conversations I've had in the past, at least half of web developers I've met have admitted to cracking open an Objective-C book at some point in their careers with high hopes of learning mobile development ... After all, who wouldn't want to create "the next big thing" for a market growing so phenomenally every year? I count myself among that majority: I've been steadily learning Objective-C over the past year, dedicating a bit of time every day, and I feel like I still lack skill-set required to create an original, complex application. Wouldn't it be great if we web developers could finally get our shot in the App Store without having to unlearn and relearn the particulars of coding a mobile application?

Luckily for us: There is!

The rock stars over at jQuery have created a framework called jQuery Mobile that allows developers to create cross-platform, responsive applications on a HTML5-based jQuery foundation. The framework allows for touch and mouse event support, so you're able to publish across multiple platforms, including iOS, Android, Blackberry, Kindle, Nook and on and on and on. If you're able to create web applications with jQuery, you can now create an awesome cross-platform app. All you have to do is create an app as if it was a dynamic HTML5 web page, and jQuery takes care of the rest.

Let's go through a real-world example to show this functionality in action. The first thing we need to do is fill in the <head> content with all of our necessary jQuery libraries:

<!DOCTYPE html>
<html>
<head>
    <title>SoftLayer Hello World!</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="http://code.jquery.com/mobile/1.3.2/jquery.mobile-1.3.2.min.css" />
    <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script>
    <script src="http://code.jquery.com/mobile/1.3.2/jquery.mobile-1.3.2.min.js"></script>
</head>

Now let's create a framework for our simplistic app in the <body> section of our page:

<body>
    <div data-role="page">
        <div data-role="header">
            <h1>My App!</h1>
        </div>
 
        <div data-role="content">
            <p>This is my application! Pretty cool, huh?</p>
        </div>
 
        <div data-role="footer">
            <h1>Bottom Footer</h1>
        </div>
 
    </div>
</body>
</html>

Even novice web developers should recognize the structure above. You have a header, content and a footer just as you would in a regular web page, but we're letting jQuery apply some "native-like" styling to those sections with the data-role attributes. This is what our simple app looks like so far: jQuery Mobile App Screenshot #1

While it's not very fancy (yet), you see that the style is well suited to the iPhone I'm using to show it off. Let's spice it up a bit and add a navigation bar. Since we want the navigation to be a part of the header section of our app, let's add an unordered list there:

<div data-role="header">
    <h1>My App!</h1>
        <div data-role="navbar">
            <ul>
                <li><a href="#home" class="ui-btn-active" data-icon="home" data-theme="b">Home</a></li>
                <li><a href="#softlayer_cool_news" data-icon="grid" data-theme="b">SL Cool News!</a></li>
                <li><a href="#softlayer_cool_stuff" data-icon="star" data-theme="b">SL Cool Stuff!</a></li>
            </ul>
        </div>
    </div>

You'll notice again that it's not much different from regular HTML. We've created a navbar div with an unordered list of menu items we'd like to add to the header: Home, SL Cool News and SL Cool Stuff. Notice in the anchor tag of each that there's an attribute called data-icon which defines which graphical icon we want to represent the navigation item. Let's have a peek at what it looks like now: jQuery Mobile App Screenshot #2

Our app isn't doing a whole lot yet, but you can see from our screenshot that the pieces are starting to come together nicely. Because we're developing our mobile app as an HTML5 app first, we're able to make quick changes and see those changes in real time from our phone's browser. Once we get the functionality we want to into our app, we can use a tool such as PhoneGap or Cordova to package our app into a ready-to-use standalone iPhone app (provided you're enrolled in the Apple Development Program, of course), or we can leave the app as-is for a very nifty mobile browser application.

In my next few blogs, I plan to expand on this topic by showing you some of the amazingly easy (and impressive) functionality available in jQuery Mobile. In the meantime, go grab a copy of jQuery Mobile and start playing around with it!

-Cassandra

September 16, 2013

Sysadmin Tips and Tricks - Using strace to Monitor System Calls

Linux admins often encounter rogue processes that die without explanation, go haywire without any meaningful log data or fail in other interesting ways without providing useful information that can help troubleshoot the problem. Have you ever wished you could just see what the program is trying to do behind the scenes? Well, you can — strace (system trace) is very often the answer. It is included with most distros' package managers, and the syntax should be pretty much identical on any Linux platform.

First, let's get rid of a misconception: strace is not a "debugger," and it isn't a programmer's tool. It's a system administrator's tool for monitoring system calls and signals. It doesn't involve any sophisticated configurations, and you don't have to learn any new commands ... In fact, the most common uses of strace involve the bash commands you learned the early on:

  • read
  • write
  • open
  • close
  • stat
  • fork
  • execute (execve)
  • chmod
  • chown

 

You simply "attach" strace to the process, and it will display all the system calls and signals resulting from that process. Instead of executing the command's built-in logic, strace just makes the process's normal calls to the system and returns the results of the command with any errors it encountered. And that's where the magic lies.

Let's look an example to show that behavior in action. First, become root — you'll need to be root for strace to function properly. Second, make a simple text file called 'test.txt' with these two lines in it:

# cat test.txt
Hi I'm a text file
there are only these two lines in me.

Now, let's execute the cat again via strace:

$ strace cat test.txt 
execve("/bin/cat", ["cat", "test.txt"], [/* 22 vars */]) = 0
brk(0)  = 0x9b7b000
uname({sys="Linux", node="ip-208-109-127-49.ip.secureserver.net", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=30671, ...}) = 0
mmap2(NULL, 30671, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f35000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000_\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1594552, ...}) = 0
mmap2(NULL, 1320356, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7df2000
mmap2(0xb7f2f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13c) = 0xb7f2f000
mmap2(0xb7f32000, 9636, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f32000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7df1000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7df0000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7df1b80, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7f2f000, 8192, PROT_READ) = 0
mprotect(0xb7f57000, 4096, PROT_READ) = 0
munmap(0xb7f35000, 30671) = 0
brk(0)  = 0x9b7b000
brk(0x9b9c000) = 0x9b9c000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
open("test.txt", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0
read(3, "Hi I'm a text file\nthere are onl"..., 4096) = 57
write(1, "Hi I'm a text file\nthere are onl"..., 57Hi I’m a text file
there are only these two lines in me.
) = 57
read(3, "", 4096) = 0
close(3) = 0
close(1) = 0
exit_group(0) = ?

Now that return may look really arcane, but if you study it a little bit, you'll see that it includes lots of information that even an ordinary admin can easily understand. The first line returned includes the execve system call where we'd execute /bin/cat with the parameter of test.txt. After that, you'll see the cat binary attempt to open some system libraries, and the brk and mmap2 calls to allocate memory. That stuff isn't usually particularly useful in the context we're working in here, but it's important to understand what's going on. What we're most interested in are often open calls:

open("test.txt", O_RDONLY|O_LARGEFILE) = 3

It looks like when we run cat test.txt, it will be opening "test.txt", doesn't it? In this situation, that information is not very surprising, but imagine if you are in a situation were you don't know what files a given file is trying to open ... strace immediately makes life easier. In this particular example, you'll see that "= 3" at the end, which is a temporary sort of "handle" for this particular file within the strace output. If you see a "read" call with '3' as the first parameter after this, you know it's reading from that file:

read(3, "Hi I'm a text file\nthere are onl"..., 4096) = 57

Pretty interesting, huh? strace defaults to just showing the first 32 or so characters in a read, but it also lets us know that there are 57 characters (including special characters) in the file! After the text is read into memory, we see it writing it to the screen, and delivering the actual output of the text file. Now that's a relatively simplified example, but it helps us understand what's going on behind the scenes.

Real World Example: Finding Log Files

Let's look at a real world example where we'll use strace for a specific purpose: You can't figure out where your Apache logs are being written, and you're too lazy to read the config file (or perhaps you can't find it). Wouldn't it be nice to follow everything Apache is doing when it starts up, including opening all its log files? Well you can:

strace -Ff -o output.txt -e open /etc/init.d/httpd restart

We are executing strace and telling it to follow all forks (-Ff), but this time we'll output to a file (-o output.txt) and only look for 'open' system calls to keep some of the chaff out of the output (-e open), and execute '/etc/init.d/httpd restart'. This will create a file called "output.txt" which we can use to find references to our log files:

#cat output.txt | grep log
[pid 13595] open("/etc/httpd/modules/mod_log_config.so", O_RDONLY) = 4
[pid 13595] open("/etc/httpd/modules/mod_logio.so", O_RDONLY) = 4
[pid 13595] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 10
[pid 13595] open("/etc/httpd/logs/ssl_error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 11
[pid 13595] open("/etc/httpd/logs/access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 12
[pid 13595] open("/etc/httpd/logs/cm4msaa7.com", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 13
[pid 13595] open("/etc/httpd/logs/ssl_access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 14
[pid 13595] open("/etc/httpd/logs/ssl_request_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 15
[pid 13595] open("/etc/httpd/modules/mod_log_config.so", O_RDONLY) = 9
[pid 13595] open("/etc/httpd/modules/mod_logio.so", O_RDONLY) = 9
[pid 13596] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 10
[pid 13596] open("/etc/httpd/logs/ssl_error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 11
open("/etc/httpd/logs/access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 12
open("/etc/httpd/logs/cm4msaa7.com", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 13
open("/etc/httpd/logs/ssl_access_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 14
open("/etc/httpd/logs/ssl_request_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 15

The log files jump out at you don't they? Because we know that Apache will want to open its log files when it starts, all we have to do is we follow all the system calls it makes when it starts, and we'll find all of those files. Easy, right?

Real World Example: Locating Errors and Failures

Another valuable use of strace involves looking for errors. If a program fails when it makes a system call, you'll want to be able pinpoint any errors that might have caused that failure as you troubleshoot. In all cases where a system call fails, strace will return a line with "= -1" in the output, followed by an explanation. Note: The space before -1 is very important, and you'll see why in a moment.

For this example, let's say Apache isn't starting for some reason, and the logs aren't telling ua anything about why. Let's run strace:

strace -Ff -o output.txt -e open /etc/init.d/httpd start

Apache will attempt to restart, and when it fails, we can grep our output.txt for '= -1' to see any system calls that failed:

$ cat output.txt | grep '= -1'
[pid 13748] open("/etc/selinux/config", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/i686/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/i686/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/tls/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/i686/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/i686/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/sse2/libperl.so", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libutil.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/etc/gai.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 13748] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = -1 EACCES (Permission denied)

With experience, you'll come to understand which errors matter and which ones don't. Most often, the last error is the most significant. The first few lines show the program trying different libraries to see if they are available, so they don't really matter to us in our pursuit of what's going wrong with our Apache restart, so we scan down and find that the last line:

[pid 13748] open("/etc/httpd/logs/error_log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = -1 EACCES (Permission denied)

After a little investigation on that file, I see that some maniac as set the immutable attribute:

lsattr /etc/httpd/logs/error_log
----i-------- /etc/httpd/logs/error_log

Our error couldn't be found in the log file because Apache couldn't open it! You can imagine how long it might take to figure out this particular problem without strace, but with this useful tool, the cause can be found in minutes.

Go and Try It!

All major Linux distros have strace available — just type strace at the command line for the basic usage. If the command is not found, install it via your distribution's package manager. Get in there and try it yourself!

For a fun first exercise, bring up a text editor in one terminal, then strace the editor process in another with the -p flag (strace -p <process_id>) since we want to look at an already-running process. When you go back and type in the text editor, the system calls will be shown in strace as you type ... You see what's happening in real time!

-Lee

December 19, 2012

SoftLayer API: Streamline. Simplify.

Building an API is a bit of a balancing act. You want your API to be simple and easy to use, and you want it to be feature-rich and completely customizable. Because those two desires happen to live on opposite ends of the spectrum, every API finds a different stasis in terms of how complex and customizable they are. The SoftLayer API was designed to provide customers with granular control of every action associated with any product or service on our platform; anything you can do in our customer portal can be done via our API. That depth of functionality might be intimidating to developers looking to dive in quickly and incorporate the SoftLayer platform into their applications, so our development team has been working to streamline and simplify some of the most common API services to make them even more accessible.

SoftLayer API

To get an idea of what their efforts look like in practice, Phil posted an SLDN blog with a perfect example of how they simplified cloud computing instance (CCI) creation via the API. The traditional CCI ordering process required developers to define nineteen data points:

Hostname
Domain name
complexType
Package Id
Location Id
Quantity to order
Number of cores
Amount of RAM
Remote management options
Port speeds
Public bandwidth allotment
Primary subnet size
Disk size
Operating system
Monitoring
Notification
Response
VPN Management - Private Network
Vulnerability Assessments & Management

While each of those data points is straightforward, you still have to define nineteen of them. You have all of those options when you check out through our shopping cart, so it makes sense that you'd have them in the API, but when it comes to ordering through the API, you don't necessarily need all of those options. Our development team observed our customers' API usage patterns, and they created the slimmed-down and efficient SoftLayer_Virtual_Guest::createObject — a method that only requires seven data points:

Hostname
Domain name
Number of cores
Amount of RAM
Hourly/monthly billing
Local vs SAN disk
Operating System

Without showing you a single line of code, you see the improvement. Default values were established for options like Port speeds and Monitoring based on customer usage patterns, and as a result, developers only have to provide half the data to place a new CCI order. Because each data point might require multiple lines of code, the volume of API code required to place an order is slimmed down even more. The best part is that if you find yourself needing to modify one of the now-default options like Port speeds or Monitoring, you still can!

As the development team finds other API services and methods that can be streamlined and simplified like this one, they'll ninja new solutions to make the API even more accessible. Have you tried coding to the SoftLayer API yet? If not, what's the biggest roadblock for you? If you're already a SLAPI coder, what other methods do you use often that could be streamlined?

-@khazard

December 5, 2012

Breaking Down 'Big Data' - Database Models

Forester defines big data as "techniques and technologies that make capturing value from data at an extreme scale economical." Gartner says, "Big data is the term adopted by the market to describe extreme information management and processing issues which exceed the capability of traditional information technology along one or multiple dimensions to support the use of the information assets." Big data demands extreme horizontal scale that traditional IT management can't handle, and it's not a challenge exclusive to the Facebooks, Twitters and Tumblrs of the world ... Just look at the Google search volume for "big data" over the past eight years:

Big Data Search Interest

Developers are collectively facing information overload. As storage has become more and more affordable, it's easier to justify collecting and saving more data. Users are more comfortable with creating and sharing content, and we're able to track, log and index metrics and activity that previously would have been deleted in consideration of space restraints or cost. As the information age progresses, we are collecting more and more data at an ever-accelerating pace, and we're sharing that data at an incredible rate.

To understand the different facets of this increased usage and demand, Gartner came up with the three V's of big data that vary significantly from traditional data requirements: Volume, Velocity and Variety. Larger, more abundant pieces of data ("Volume") are coming at a much faster speed ("Velocity") in formats like media and walls of text that don't easily fit into a column-and-row database structure ("Variety"). Given those equally important factors, many of the biggest players in the IT world have been hard at work to create solutions that provide the scale and speed developers need when they build social, analytics, gaming, financial or medical apps with large data sets.

When we talk about scaling databases here, we're talking about scaling horizontally across multiple servers rather than scaling vertically by upgrading a single server — adding more RAM, increasing HDD capacity, etc. It's important to make that distinction because it leads to a unique challenge shared by all distributed computer systems: The CAP Theorem. According to the CAP theorem, a distributed storage system must choose to sacrifice either consistency (that everyone sees the same data) or availability (that you can always read/write) while having partition tolerance (where the system continues to operate despite arbitrary message loss or failure of part of the system occurs).

Let's take a look at a few of the most common database models, what their strengths are, and how they handle the CAP theorem compromise of consistency v. availability:

Relational Databases

What They Do: Stores data in rows/columns. Parent-child records can be joined remotely on the server. Provides speed over scale. Some capacity for vertical scaling, poor capacity for horizontal scaling. This type of database is where most people start.
Horizontal Scaling: In a relational database system, horizontal scaling is possible via replication — dharing data between redundant nodes to ensure consistency — and some people have success sharding — horizontal partitioning of data — but those techniques add a lot of complexity.
CAP Balance: Prefer consistency over availability.
When to use: When you have highly structured data, and you know what you'll be storing. Great when production queries will be predictable.
Example Products: Oracle, SQLite, PostgreSQL, MySQL

Document-Oriented Databases

What They Do: Stores data in documents. Parent-child records can be stored in the same document and returned in a single fetch operation with no join. The server is aware of the fields stored within a document, can query on them, and return their properties selectively.
Horizontal Scaling: Horizontal scaling is provided via replication, or replication + sharding. Document-oriented databases also usually support relatively low-performance MapReduce for ad-hoc querying.
CAP Balance: Generally prefer consistency over availability
When to Use: When your concept of a "record" has relatively bounded growth, and can store all of its related properties in a single doc.
Example Products: MongoDB, CouchDB, BigCouch, Cloudant

Key-Value Stores

What They Do: Stores an arbitrary value at a key. Most can perform simple operations on a single value. Typically, each property of a record must be fetched in multiple trips, with Redis being an exception. Very simple, and very fast.
Horizontal Scaling: Horizontal scale is provided via sharding.
CAP Balance: Generally prefer consistency over availability.
When to Use: Very simple schemas, caching of upstream query results, or extreme speed scenarios (like real-time counters)
Example Products: CouchBase, Redis, PostgreSQL HStore, LevelDB

BigTable-Inspired Databases

What They Do: Data put into column-oriented stores inspired by Google's BigTable paper. It has tunable CAP parameters, and can be adjusted to prefer either consistency or availability. Both are sort of operationally intensive.
Horizontal Scaling: Good speed and very wide horizontal scale capabilities.
CAP Balance: Prefer consistency over availability
When to Use: When you need consistency and write performance that scales past the capabilities of a single machine. Hbase in particular has been used with around 1,000 nodes in production.
Example Products: Hbase, Cassandra (inspired by both BigTable and Dynamo)

Dynamo-Inspired Databases

What They Do: Distributed key/value stores inspired by Amazon's Dynamo paper. A key written to a dynamo ring is persisted in several nodes at once before a successful write is reported. Riak also provides a native MapReduce implementation.
Horizontal Scaling: Dynamo-inspired databases usually provide for the best scale and extremely strong data durability.
CAP Balance: Prefer availability over consistency,
When to Use: When the system must always be available for writes and effectively cannot lose data.
Example Products: Cassandra, Riak, BigCouch

Each of the database models has strengths and weaknesses, and there are huge communities that support each of the open source examples I gave in each model. If your database is a bottleneck or you're not getting the flexibility and scalability you need to handle your application's volume, velocity and variety of data, start looking at some of these "big data" solutions.

Tried any of the above models and have feedback that differs from ours? Leave a comment below and tell us about it!

-@marcalanjones

October 17, 2012

Tips and Tricks - jQuery Select2 Plugin

Web developers have the unique challenge of marrying coding logic and visual presentation to create an amazing user experience. Trying to find a balance between those two is pretty difficult, and it's easy to follow one or the other down the rabbit hole. What's a web developer to do?

I've always tried to go the "work smarter, not harder" route, and when it comes to balancing functionality and aesthetics, that usually means that I look around for plugins and open source projects that meet my needs. In the process of sprucing up an form, I came across jQuery Select2, and it quickly became one of my favorite plugins for form formatting. With minimal scripting and little modification, you get some pretty phenomenal results.

We've all encountered drop-down selection menus on web forms, and they usually look like this:

Option Select

Those basic drop-downs meet a developer's need for functionality, but they aren't winning any beauty pageants. Beyond the pure aesthetic concerns, when a menu contains dozens (or hundreds) of selectable options, it becomes a little unwieldy. That's why I was so excited to find Select2.

With Select2, you can turn the old, plain, boring-looking select boxes into beautiful, graceful and more-than-functional select widgets:

Pretty Option Select

Not only is the overall presentation of the data improved, Select2 also includes an auto-complete box. A user can narrow down the results quickly ad easily, and if you've got some of those endlessly scrolling select boxes of country names or currencies, your users will absolutely notice the change (and love you for it).

What's even sexier than the form facelift is that you can add the plugin to your form in a matter of minutes.

After we download Select2 and upload it to our box, we add our the jQuery library and scripts to the <head> of our document:

<script src="jquery.js" type="text/javascript"></script> 
<script src="select2.js" type="text/javascript"></script>

For the gorgeous styling, we'll also add Select2's included style sheet:

<link href="select2.css" rel="stylesheet"/>

Before we close our <head> tag, we invoke the Select2 function:

<script>
$(document).ready(function() { $("#selectPretty").select2(); });
</script>

At this point, Select2 is locked and load, and we just have to add the #selectPretty ID to the select element we want to improve:

<select id="selectPretty">
<option value="Option1">Option 1</option>
<option value="Option2">Option 2</option>
<option value="Option3">Option 3</option>
<option value="Option4">Option 4</option>
</select>

Notice: the selectPretty ID is what we defined when we invoked the Select2 function in our <head> tag.

With miniscule coding effort, we've made huge improvements to the presentation of our usually-boring select menu. It's so easy to implement that even the most black-and-white coding-minded web developers can add some pizzazz to their next form without having to get wrapped up in styling!

-Cassandra

September 6, 2012

Tips and Tricks - jQuery equalHeights Plugin

Last month, I posted a blog about dynamically resizing divs with jQuery, and we received a lot of positive feedback about it. My quest to avoid iframes proved to be helpful, so I thought I'd share a few more esoteric jQuery tips and tricks that may be of use to the developers and designers in the audience. As I thought back about other challenges I've faced as a coder, a great example came to mind: Making divs equal height, regardless of the amount of content inside.

I haven't seen many elegant div-based solutions for that relatively simple (and common) task, so I've noticed that many people struggle with it. Often, developers will turn back to the "Dark Side" of using tables to format the content since all columns would have the same height as the tallest column by default:

JQuery Tutorial

It was easy theme table columns and to achieve the coveted 100% height that many designers seek, but emulating that functionality with divs proves to be much more difficult. A div is like the Superman of HTML elements (faster-loading, more flexible, more dynamic, etc.), and while it has super powers, it also has its own Kryptonite-like weaknesses ... The one relevant to this blog post being that floating three div elements next to each other isn't going to give you the look of a table:

JQuery Tutorial

Each of the three divs has its own height, so if you're doing something as simple as applying background colors, you're going to wind up with an aesthetically unpleasing result: It's going to look funky.

You could get into some nifty HTML/CSS workarounds, but many frustrated theme creators and designers will tell you that if your parent elements don't have a height of a 100%, you're just wasting coding lines. Some complex solutions create the illusion of all three divs being the same height (which is arguably better than setting fixed heights), but that complexity can be difficult to scale and repeat if you need to perform similar tasks throughout your site or your application. The easiest way to get the functionality you want and the simplicity you need: The jQuery equalHeights plugin!

With a few class declarations in your existing HTML, you get the results you want, and with equalHeights, you can also specify the minimum and maximum parameters so it will create scrollable divs if the tallest element happens to be higher than your specified maximum.

How to Use jQuery equalHeights

First and foremost, include your JQuery lirbraries in the <HEAD> of your document:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js"></script>
<script language="javascript" type="text/javascript" src="jquery.equalheights.js"></script>

The equalHeights plugin is not a hosted library, so you have to host the file on your server (here's the link again).

With the required libraries called in our document, it's time to make the magic happen in your HTML.

Create Your Divs

<div class="divHeight">This DIV is medium sized, not too big and not too small, but just right.</div>
<div class="divHeight">This DIV has a lot of useful content and media that the user can interact with, thus it's very tall.</div>
<div class="divHeight">This DIV is tiny. Period.</div>

To have them line up next to each other, you'd have them float:left; in your CSS, and now you need to apply the equalHeights function.

Call the equalHeights Plugin
In order for the script to recognize the height of the tallest element, you'd need to call $(document).ready just before the </body> tag on your page. This will ensure that the page loads before the function runs.

The call looks like this:

<script type="text/javascript">$(document).ready(function() {
$(".divHeight").equalHeights();
});</script>

If you want to specify a minimum and maximum (i.e. The div should be at least this tall and should be no taller than [adds scrollbar if the div size exceeds] the maximum), just add the parameters:

<script type="text/javascript">$(document).ready(function() {
$(".divHeight").equalHeights(300, 600);
});</script>

The initial call does not change the appearance of the divs, but the time it takes to do the resizing is so miniscule that users will never notice. After that call is made and the height is returned, each div with the class of divHeight will inherit the the same height, and your divs will be nice and pretty:

JQuery Tutorial

This trick saved me a lot of headache and frustration, so hopefully it will do the same for you too!

-Cassandra

August 8, 2012

No iFrames! Dynamically Resize Divs with jQuery.

It's no secret that iframes are one of the most hated methods of web page layouts in the web development world — they are horrible for SEO, user experience and (usually) design. I was recently charged with creating a page that needed functionality similar to what iframes would normally provide, and I thought I'd share the non-iframe way I went about completing that project.

Before I get into the nitty-gritty of the project, I should probably unpack a few of the reasons why iframes are shunned. When a search engine indexes a page with iframes, each iframe is accurately recorded as a separate page — iframes embed the content of one we page inside of another, so it makes sense. Because each of those "pages" is represented in a single layout, if a user wanted to bookmark your site, they'd probably have a frustrating experience when they try to return to your site, only to find that they are sent directly to the content in one of the frames instead of seeing the entire layout. Most often, I see when when someone has a navigation bar in one frame and the main content in the other ... The user will bookmark the content frame, and when they return to the site, they have no way to navigate the pages. So what's a developer to do?

The project I was tasked with required the ability to resize only certain sections of a page, while asynchronously shrinking another section so that the entire page would always stay the same size, with only the two sections inside changing size.

Let's look at an example with two divs, side by side on a web page:

iFrame Tutorial

One div will contain a navigation menu to jump to different pages of the website (#sidebar), and the second div will contain all the content for that page (#content). If some of the elements in #sidebar are too long to read with the default width of the div, we want to let the user freely resize the two divs without changing the width of the page.

Our task is straightforward: When #sidebar expands in width, also expand the navigation and shrink #content along with the main content inside #content. If #sidebar shrinks, the navigation, #content and main content would respond accordingly as well:

iFrame Tutorial

It's a relatively easy concept to do with iFrames ... But then you remember that iframes are no longer cool (yes, there was a time long ago when iframes were cool). I decided to turn to my favorite alternative — jQuery — and the fix was actually a lot easier than I expected, and it worked beautifully. Let's run through a step-by-step tutorial.

1. HTML

Lay out your two divs:

<div id="sidebar"> 
<div class="sidebar-menu">
<!-- all your sidebar/navigational items go here -->
</div>
</div>
<div id="content">
<!-- all your main content goes here -->
</div>

2. CSS

Style your divs:

#sidebar {
       width: 49%;
}
#content {
width: 49%;
        float: left;
}

3. jQuery

Now that we have our two divs side by side, let's apply some jQuery magic. To do that, Let's include our jQuery files in the <HEAD> of our document:

<link href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/base/jquery-ui.css" rel="stylesheet" type="text/css"/>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/jquery-ui.min.js"></script>

Now that we have the necessary scripts, we can write our function:

<script type="text/javascript">
  $(document).ready(function() {
    $( "#sidebar" ).resizable({      
    });
    $("#sidebar ").bind("resize", function (event, ui) {
            var setWidth = $("#sidebar").width();
            $('#content).width(1224-setWidth);
            $('.menu).width(setWidth-6);
        });
  });
</script>

I know that might seem like an intimidating amount of information, so let's break it down:

   $( "#sidebar" ).resizable({      
   });

This portion simply makes the div with the ID of "sidebar" resizable (which accomplishes 33% of what we want it to do).

   $("#sidebar ").bind("resize", function (event, ui) {

By using the .bind, we are able to trigger other events when #sidebar is called.

            var setWidth = $("#sidebar").width();
            $('#content).width(1224-setWidth);

This is where the magic happens. We're grabbing the current width of #sidebar and subtracting it from the width you want your site to be. This code is what keeps your page stays the same width with only the divs changing sizes.

            $('.menu).width(setWidth-6);

This part of the code that expands the contents in the navigation along with #sidebar.

You can see a working example of iframe-like functionality with jQuery here: http://jqueryui.com/demos/resizable/

The only part you won't find there is the trick to adjust a corresponding div's size to make it grow/shrink with the first ... I had a heck of a time searching that on the web, so hopefully this quick tutorial will help other developers who might be searching for this kind of functionality!

- Cassandra

July 11, 2012

Mandrill: Tech Partner Spotlight

This is a guest blog with Chad Morris from our partner Mandrill. Mandrill is an email delivery platform built on and managed by MailChimp. Created for developers to set up and manage with minimal coding effort, Mandrill offers advanced tracking, easy-to-understand reports and hundreds of template options. In this video interview, Chad goes into detail about the history of the company as well as the major differences between Mandrill and MailChimp. In the near future, you'll see a separate guest blog from the Mandrill team with best practices for managing your email systems.

This guest blog series highlights companies in SoftLayer's Technology Partners Marketplace.
These Partners have built their businesses on the SoftLayer Platform, and we're excited for them to tell their stories. New Partners will be added to the Marketplace each month, so stay tuned for many more come.
May 10, 2012

The SoftLayer API and its 'Star Wars' Sibling

When I present about the SoftLayer API at conferences and meetups, I often use an image that shows how many of the different services in the API are interrelated and connected. As I started building the visual piece of my presentation, I noticed a curious "coincidence" about the layout of the visualization:

SoftLayer API Visualization

What does that look like to you?

You might need to squint your eyes and tilt your head or "look beyond the image" like it's one of those "Magic Eye" pictures, but if you're a geek like me, you can't help but notice a striking resemblance to one of the most iconic images from Star Wars:

SoftLayer API == Death Star?

The SoftLayer API looks like the Death Star.

The similarity is undeniable ... The question is whether that resemblance is coincidental or whether it tells us we can extrapolate some kind of fuller meaning as in light of the visible similarities. I can hear KHazzy now ... "Phil, While that's worth a chuckle and all, there is no way you can actually draw a relevant parallel between the SoftLayer API and The Death Star." While Alderaan may be far too remote for an effective demonstration, this task is no match for the power of the Phil-side.

Challenge Accepted.

The Death Star: A large space station constructed by the Galactic Empire equipped with a super-laser capable of destroying an entire planet.

The SoftLayer API: A robust set of services and methods which provide programmatic access to all portions of the SoftLayer Platform capable of automating any task: administrative, configuration or otherwise.

Each is the incredible result of innovation and design. The construction of the Death Star and creation of the SoftLayer API took years of hard work and a significant investment. Both are massive in scale, and they're both effective and ruthless when completing their objectives.

The most important distinction: The Death Star was made to destroy while the SoftLayer API was made to create ... The Death Star was designed to subjugate a resistance force and destroy anything in the empire's way. The SoftLayer API was designed to help customers create a unified, automated way of managing infrastructure; though in the process, admittedly that "creation" often involves subjugating redundant, compulsory tasks.

The Death Star and the SoftLayer API can both seem pretty daunting. It can be hard to find exactly what you need to solve all of your problems ... Whether that be an exhaust port or your first API call. Fear not, for I will be with you during your journey, and unlike Obi-Wan Kenobi, I'm not your only hope. There is no need for rebel spies to acquire the schematics for the API ... We publish them openly at sldn.softlayer.com, and we encourage our customers to break the API down into the pieces of functionality they need.

-Phil (@SoftLayerDevs)

Subscribe to developers