Cloudant: Data Layer for the Big Data Era

The recipe for big data app success: Start small. Iterate fast. Grow to epic proportions.

Unfortunately, most developers’ databases come up short when they try to simultaneously “iterate fast” and “grow to epic proportions” — those two steps are most often at odds. I know … I’ve been there. In a recent past life, I attacked petabyte-per-second data problems as a particle physicist at the Large Hadron Collider together with my colleagues and Cloudant co-founders, Alan Hoffman and Adam Kocoloski. Here are some lessons we learned the hard way:

1. Scaling a database yourself is brutally hard (both application level sharding and the master-slave model). It is harder with SQL than it is with NoSQL databases, but either way, the “scale it yourself” approach is loaded with unknowns, complications and operational expense.
2. Horizontal scaling on commodity hardware is a must. We got very good at this and ended up embedding Apache CouchDB behind a horizontal scaling framework to scale arbitrarily and stay running 24×7 with a minimal operational load.
3. The data layer must scale. It should be something that applications grow into, not out of.

That last point inspired Alan, Adam and me to co-found Cloudant.

What is Cloudant?
Cloudant is a scalable data layer (as a service) for Big Data apps. Built on CouchDB, JSON, and MapReduce, it lets developers focus on new features instead of the drudgery of growing or migrating databases. The Cloudant Data Layer is already big: It collects, stores, analyzes and distributes application data across a global network of secure, high-performance data centers, delivering low-latency and non-stop data access to users no matter where they’re located. You get to focus on your code; we’ve got data scalability and availability covered for you.

Scaling Your App on Cloudant
Cloudant is designed to support fast app iteration by developers. It’s based on the CouchDB NoSQL database where data is encapsulated and transferred as JSON documents. You don’t need to design and redesign SQL data models or migrate databases in order to create new app features. You don’t need to write object-relational mapping code either. The database resides behind an HTTP layer and provides a rich permission model, so you can access, secure and share your data via a RESTful API.

Your app is a tenant within a multi-tenant data layer that is already big and scalable. You get a URL end point for your data layer, get data in and out of it via HTTP, and we scale and secure it around the globe. Global data distribution and intelligent routing minimizes latency between your users and the data, which can add 100s of milliseconds per request (we’ve measured!). Additionally, Cloudant has an advanced system for prioritizing requests so that apps aren’t affected by ‘noisy neighbors’ in a multi-tenant system. We also offer a single-tenant data layer to companies who want it — your very own white-labeled data cloud. As your data volume and IO requests rise (or fall), Cloudant scales automatically, and because your data is replicated to multiple locations, it’s always available. Start small and grow to epic proportions? Check.

Other Data Management Gymnastics
The Cloudant Data Layer also makes it easy to add advanced functionality to your apps:

• Replicate data (all of it or sub-sets) to data centers, computers or even mobile devices for local processing (great for analytics) or off-line access (great for mobile users). Re-synching is automatic.
• Perform advanced analytics with built-in MapReduce and full-text indexing and search.
• Distribute your code with data — Cloudant can distribute and serve any kind of document, even HTML5 and other browser-based code, which makes it easy to scale your app and move processing from your back-end to the browser.

Why We Run on SoftLayer
Given the nature of our service, people always ask us where we have our infrastructure, and we’re quick to tell them we chose SoftLayer because we’re fanatical about performance. We measured latencies for different data centers run by other cloud providers, and it’s no contest: SoftLayer provides the lowest and most predictable latencies. Data centers that are thousands of miles apart perform almost as if they are on the same local area network. SoftLayer’s rapidly expanding global presence allows Cloudant to replicate data globally throughout North America, Europe and Asia (with plans to continue that expansion as quickly as SoftLayer can build new facilities).

The other major draw to SoftLayer was the transparency they provide about our infrastructure. If you run a data layer, IO matters! SoftLayer provisions dedicated hardware for us (rather than just virtual machines), and they actually tell us exactly what hardware we are running on, so we can tweak our systems to get the most bang for our buck.

Get Started with Cloudant for Free
If you’re interested to see what the Cloudant Data Layer could do for your app, sign up at cloudant.com to get your FREE global data presence created in an instant.

-Michael Miller, Cloudant

Separation of Concerns

It is important when sitting down for the first time to build a website to come up with an outline. Start by making a list of the parts of your website and the functions of those parts. I always start at the base of my web structure and work from there. HTML is always the foundation of a website; it defines the structure and outlines how you will display your data – plain and simple. It doesn’t have to include data or styles, nor does it need to be dynamic … At its essence, it’s a static file that browsers can cache.

Client-side scripting languages like JavaScript will take care of client-side animations and data dispersal, while cascading style sheets (CSS) take care of style and presentation, and server-side scripting languages like PHP or Perl can take care of data retrieval and formatting.

Data Transfer

Where is your data going to come from, and what format it will be in when the client receives it? Try to use a data format that is the most compatible with your scripting languages. I use JavaScript as my primary client side scripting program, so I try to use JSON as my data format, but that’s not always possible when dealing with APIs and transferring data from remote computers. JSON is quickly becoming a standard data format, but XML* is the most widely accepted format.

I prefer to use REST APIs as much as possible, because they sends the information directly on the client, rather than using the server as a proxy. However, if a REST API is not available or if there is a security risk involved, you get the advantage of being able to format the data on the server before pushing it to the client. Try to parse and format data as little as possible on the client side of things, the client should be concerned with placing data.

Changes in Data

In the past, websites were made from multiple HTML documents, each one containing different data. The structure of the pages were the same though, so the data changed, but the code was nearly identical. Later, using server side scripting programs, websites became more dynamic, displaying different data based on variables passed in the URL. Now, using AJAX or script injection, we can load new data into a static webpage without reloading. This means less redundant code, less load on the client, and better overall performance.

Page Structure

It is important when displaying data to understand when to change the structure of the page. I start by creating a structure for my home page – it needs to be very open and unrestricting so I can add pictures and text to build the site. Once the overall loose structure is established, I create a structure for displaying products (this will be more restrictive, containing tables and ordering tools). The idea is to have as few HTML structures as possible, but if you find that your data doesn’t fit or if you spend a lot of time positioning your data, then it might be time to create a new structure.

The Impact of a Modern Layout

Following these steps will lead to quicker, more efficient websites. This is (of course) not a new subject, and further understanding of web layout can be found in Model-View-Controller frameworks. If you find that you spend too much time writing code to interface with databases or place data, then frameworks are for you.

-Kevin

*If you have to deal with XML, make sure to include JavaScript libraries that make it easier to parse, like JQuery.

REST API
Although the REST software archetype is a difficult concept to explain, its practice has become natural to those of us who use the internet daily. Imagine that the information that you want to know is saved as a web page somewhere and all you have to do is type in the URL, it will prompt you for a username and password, and you will see the information that you requested.

Authentication
Before making a request you will need to find your API authentication token. To do this, log into your customer account and click API under the Support tab. Click the “Manage API Access” link. At the bottom of the next page you will see a drop-down menu that says “Select a User” and above it a tag that says “Generate a new API access key.” Select a user and click the “Generate API Key” button. You will see your username and the generated API key for that user. Copy this API key, as you’ll need it to send commands SoftLayer’s API.

“Hello World”
Unfortunately, there is no specific “Hello World” command in SoftLayer’s API, but there are some commands that are very simple and don’t require any variables, like the getObject() method. APIs are like component libraries, split into web services and methods of that service. The SLDN has a full list of SoftLayer’s web services to choose from. I am going to use the getObject() method from the SoftLayer_Account service in this example:

https://api.softlayer.com/rest/v3/SoftLayer_Account.xml

• You will be prompted for your username and API access key
• XML data type output

https://USERNAME:PASSWORD@api.softlayer.com/rest/v3/SoftLayer_Account.json

• Automatic authentication
• JSON data type output

The Request
Here is the basic REST request structure:

https://username:API key@api.service.softlayer.com/rest/v3/serviceName/InitializationParameter.returnDatatype

• All requests are sent via secure transfer (https://)
• Listing your username and API key before the URL allows for automatic HTTP authentication
• Service and serviceName both refer to the web service you are trying to access
• InitializationParameter is only used if the method you are calling requires an initialization Parameter
• SoftLayer’s REST API can respond with either JSON or XML data types; replace returnDatatype with the type you would like to receive.

The Data
Looking at the first link above, your browser should be able to output the response data in XML format, showing information about your account. More information about the format of the data can be found on the SLDN wiki.

REST Basics
When you start integrating this into a website you will want to get/make a function or library to handle advanced requests and to properly receive and disperse the response; I recommend using JQuery. This is the most basic example of a function call for SoftLayer’s API, I hope that it will help you get a feel for the information that you will need to pass to our server and the kind of response that you will receive.

-Kevin