Author Archive: Chris Gallo

May 11, 2016

Adventures in Bluemix: Migrating to MQ Light

One of my pet projects at SoftLayer is looking at a small collection of fancy scripts that scan through all registered Internet domain names to see how many of them are hosted on SoftLayer’s infrastructure. There are a lot of fun little challenges involved, but one of the biggest challenges is managing the distribution of work so that this scan doesn’t take all year. Queuing services are great for task distribution, and for my initial implementation I decided to give running a RabbitMQ instance a try, since at the time it was the only queuing service I was familiar with. Overall, it took me about a week and one beefy server to go from “I need a queue,” to “I have a queue that is actually doing what I need it to.”

While what I had set up worked, looking back, there is a lot about RabbitMQ that I didn’t really have the time to figure out properly. Around the time I finished the first run of this project, Bluemix announced that its MQLight service would allow connections from non-Bluemix resources. So when I got some free time, I decided to move the project to a Bluemix-hosted MQ Light queue, and take some notes on how the migration went.

Project overview

To better understand how much work was involved, let me quickly explain how the whole “scanning through every registered domain for SoftLayer hosted domains” thing works.

There are three main moving parts in the project:

  1. The Parser, which is responsible for reading through zone files (which are obtained from the various registrars), filtering out duplicates, and putting nicely formatted domains into a queue.
  2. The Resolver, which is responsible from taking the nicely formatted domains from queue #1, looking up the domain’s IP address, and putting the result into queue #2.
  3. The Checker, which takes the domains from queue #2, checks to see if the domains’ IPs belong to SoftLayer or not, and saves the result in a database.

Each queue entry is a package of about 500 domains, which is roughly 200Kb of text data consisting of the domain and some meta-data that I used to see how well everything was performing. There are around 160 million domains I need to review, and resolving a single domain can take anywhere from .001 seconds to four seconds, so being able to push domains quickly through queues is very important.

Things to be aware of

Going into this migration, I made a lot of assumptions about how things worked that caused me grief. So if you are in a similar situation, here is what I wish someone had told me.

AMQP 1.0: MQLight implements the AMQP 1.0 protocol, which is great, because it is the newest and greatest. As everyone knows, newer is usually better. The problem is that my application was using the python-pika library to connect to RabbitMQ, both of which implement AMQP 0.9, which isn’t fully compatible with AMQP 1.0. The Python library I was using gave me a version error when trying to connect to MQ Light. This required a bit of refactoring of my code in order to get everything working properly. The core ideas are the same, but some of the specific API calls are slightly different.

Persistence: Messages sent to a MQ Light queue without active subscribers will be lost, which took me a while to figure out. The UI indicates when this happens, so this is likely just a problem of me not reading the documentation properly and assuming MQ Light worked like RabbitMQ.



Messages sent to a MQLight queue without active subscribers will be lost.

Threads: The python-mqlight library uses threads fairly heavily, which is great for performance, but it makes programming a little more thought intensive. Make sure you wait for the connection to initialize before sending any messages, and make sure all your messages have been sent in before exiting.

Apache Proton: MQ Light is built on the Apache Qpid Proton project, and the Python library python-mqlight also uses this.

Setting up MQ Light

Aside from those small issues I mentioned, MQ Light was really easy to set up and start using, especially when compared to running my own RabbitMQ instance.



MQLight was really easy to set up and start using, especially when compared to running my own RabbitMQ instance.

  1. Set up the MQ Light Service in Bluemix.
  2. Install the python-mqlight library (or whatever library supports your language of choice). There are a variety of MQ Light Libraries.
  3. Try the send/receive examples.
  4. Write some code.
  5. Watch the messages come in, and profit.

That’s all there is to it. As a developer, the ease with which I can set up services to try is one of the best things about Bluemix, with MQ Light making a great addition to its portfolio of services.

Some real numbers

After I re-factored my code to be able to use either the pika or python-mqlight libraries interchangeably, I ran a sample set of data through each library to see what impact they had on overall performance, and I was pleasantly surprised to see the results.

Doing a full run-through of all domains would take about seven hours, so I ran this test with only 10,364 domains. Below are the running times for each section, in seconds.

Local RabbitMQ

This server was running on a 4 core, 49G Ram VSI.

Parser: 0.054s

Resolver: 90.485s

Checker: 0.0027s

Bluemix MQLight

Parser: 1.593s

Resolver: 86.756s

Checker: 6.766s

Since I am using the free, shared tier of MQ Light, I was honestly expecting much worse results. Having only a few seconds increase in runtime was a really big win for MQ Light.

Overall, I was very pleased working with MQ Light, and I highly suggest it as a starting place for anyone wanting to check out queuing services. It was easy to set up, free to try out, and pretty simple once I started to understand the basics.

-Chris

Categories: 
March 4, 2016

Adventures with Bluemix

Keeping up with the rapid evolution of web programming is frighteningly difficult—especially when you have a day job. To ensure I don’t get left behind, I like to build a small project every year or so with a collection of the most buzzworthy technologies I can find. Nothing particularly impressive, of course, but just a collection of buttons that do things. This year I am trying to get a good grasp on “as a Service,” which seems to be everywhere these days. Hopefully this adventure will prove educational.

Why use services when I can do it myself?

The main idea behind “as a Service” is that somewhere out there in the cloud, someone has figured out how to do a particular task really well. This someone is willing to provide you access to that for a small service fee—thereby letting you, the developer, focus as much time as possible on your code and not so much time worrying about optimal configurations of things that you need to work efficiently.

SoftLayer is an Infrastructure as a Service (IaaS) provider, which is what will be the home for my little application—due in large part because I already have a ton of experience running servers myself.

I’m a big fan of Python, so I’m going to start programing with the Pyramids framework as the base for my new application. Like the “as a Service” offerings, programming frameworks and libraries exist to help the developer focus on their code and leverage the expertise of others for the auxiliary components.

To make everything pretty, I am going to use Bootstrap.js, which is apparently the de facto front-end library these days.

For everything else I want to use, there will be an attached Bluemix service. For the uninitiated, Bluemix is a pretty awesome collection of tools for developing and deploying code. At its core, Bluemix uses Cloud Foundry to provision cloud resources and deploy code. For now, I’m going to deploy my own code, but what I’m really interested in are the add-on services that I can just drop into my application and get going. The first service I want to try out is going to be Cloudant nosql, which is a managed couchDB instance with a few added features like a pretty neat dashboard.

Welcome to Bluemix

Combining Bluemix services with SoftLayer servers

One of the great things about services in Bluemix is that they can be provisioned in a standalone deployment—meaning Bluemix services can be used by any computer with an Internet connection and therefore, so can my SoftLayer servers. Since Bluemix services are deployed on SoftLayer hardware (in general, but there are some exceptions), the latency between SoftLayer servers and Bluemix services should be minimal, which is nice.

Creating a Cloudant service in Bluemix is as easy as hitting the Create button in the console. Creating a simple web application in Pyramid took a bit longer, but the quick tutorial helped me learn about all the cool things the Pyramid project can do. I also got to skip all the mess with SQLAlchemy, since I’m storing all the data in Cloudant. All that’s required is a sane ID system (I am using uuid) and some json. No need to get bogged down with a rigid table structure since Cloudant is a document store. If I want to change the data format, I just need to upload a new copy of the data, and a new revision of that document will be automatically created.

After cobbling together a basic application that can publish and edit content, all I had to do to make everything look like it was designed intentionally was to add a few bootstrap classes to my templates. And then I had a ready to use website!

Conclusion

Although making a web application is still as intensive as it’s always been, at least using technology in an “as a Service” fashion helps cut down on all the tertiary technologies you need to become an expert on to get anything to work. Even though the application I created here was pretty simple, I hope to expand it to include some of the more interesting Bluemix services to see what kind of Frankenstein application I can manage to produce. There are currently 100 Bluemix services, so I think the hardest part is going to be figuring out which one to use next.

-Chris

December 21, 2015

Introducing API release notes and examples library

The website to find out what new and exciting changes are happening on the SoftLayer platform is now softlayer.github.io. Specifically, this website highlights any changes to the customer portal, the API, and any supporting systems. Please continue to rely on tickets created on your account for information regarding any upcoming maintenances and other service impacting events.

At SoftLayer, we follow agile development principles and release code in small but frequent iterations—usually about two every week. The changes featured in release notes on softlayer.github.io only cover what is publicly accessible. So while they may seem small, there are usually a greater number of behind-the-scenes changes happening.

Along with the release notes are a growing collection of useful example scripts on how to actually use the API in a variety of popular languages. While the number of examples is currently small, we are constantly adding examples as they come up, so keep checking back. We are generally inspired to add examples by the questions posted on Stack Overflow that have the SoftLayer tag, so keep posting your questions there, too.

-Chris

March 27, 2015

Building “A Thing” at Hackster.io’s Hardware Weekend

Introduction to Hackster.io

Over the weekend in San Francisco, I attended a very cool hackathon put together by the good folks at Hackster.io. Hackster.io’s Hardware Weekend is a series of hackathons all over the country designed to bring together people with a passion for building things, give them access to industry mentors, and see what fun and exciting things they come up with in two days. The registration desk was filled with all kinds of hardware modules to be used for whatever project you could dream up—from Intel Edison boards, the Grove Starter Kit, a few other things that I have no idea what they did, and of course, plenty of stickers.

After a delicious breakfast, we heard a variety of potential product pitches by the attendees, then everyone split off into groups to support their favorite ideas and turn them into a reality.

When not hard at work coding, soldering, or wiring up devices, the attendees heard talks from a variety of industry leaders, who shared their struggles and what worked for their products. The founder of spark.io gave a great talk on how his company began and where it is today.

Building a thing!
After lunch, Phil Jackson, SoftLayer’s lead technology evangelist, gave an eloquent crash course in SoftLayer and how to get your new thing onto the Internet of Things. Phil and I have a long history in Web development, so we provided answers to many questions on that subject. But when it comes to hardware, we are fairly green. So when we weren't helping teams get into the cloud, we tried our hand at building something ourselves.

We started off with some of the hardware handouts: an Edison board and the Grove Starter Kit. We wanted to complete a project that worked in the same time the rest of the teams had—and showed off some of the power of SoftLayer, too. Our idea was to use the Grove Kit’s heat sensor, display it on the LCD, and post the result to a IBM Cloudant database, which would then be displayed on a SoftLayer server as a live updating graph.

The first day consisted mostly of Googling variations on “Edison getting started,” “read Grove heat sensor,” “write to LCD”, etc. We started off simply, by trying to make an LED blink, which was pretty easy. Making the LED STOP blinking, however, was a bit more challenging. But we eventually figured out how to stop a program from running. We had a lot of trouble getting our project to work in Python, so we eventually admitted defeat and switched to writing node.js code, which was significantly easier (mostly because everything we needed was on stackoverflow).

After we got the general idea of how these little boards worked, our project came together very quickly at the end of Day 2—and not a moment too soon. The second I shouted, “IT WORKS!” it was time for presentations—and for us to give out the lot of Raspberry Pi we brought to some lucky winners.

And, without further ado, we present to you … the winners!

BiffShocker

This team wanted to mod out the Hackster’s DeLorean time machine to prevent Biff (or anyone else) from taking it out for a spin. They used a variety of sensors to monitor the DeLorean for any unusual or unauthorized activity, and if all else failed, were prepared to administer a deadly voltage through the steering wheel (represented by harmless LEDs in the demo) to stop the interloper from stealing their time machine. The team has a wonderful write up of the sensors they used, along with the products used to bring everything together.

This was a very energetic team who we hope will use their new Raspberry Pis to keep the space-time continuum clear.

KegTime

The KegTime project aimed to make us all more responsible drinkers by using an RFID reader to measure alcohol consumption and call Uber for you when you have had enough. They used a SoftLayer server to host all the drinking data, and used it to interact with Uber’s API to call a ride at the appropriate moment. Their demo included a working (and filled) keg with a pretty fancy LED-laden tap, which was very impressive. In recognition of their efforts to make us all more responsible drinkers, we awarded them five Raspberry Pis so they can continue to build cool projects to make the world a better place.

The Future of Hackster.io
Although this is the end of the event in San Francisco, there are many more Hackster.io events coming up in the near future. I will be going to Phoenix next on March 28 and look forward to all the new projects inventors come up with.

Be happy and keep hacking!

-Chris

Categories: 
January 27, 2015

Hello, IBM Bluemix!

Developers, if you'd prefer to focus on building new applications instead of customizing your own unique cloud infrastructure, IBM Bluemix provides building blocks to rapidly develop and deploy applications on the Platform as a Service (PaaS) level to make life easier for you. It’s an ecosystem of services based on Cloud Foundry, an open source project designed to make deploying and scaling an application as simple as possible. Leveraging an existing project like this is a large part of what makes Bluemix so easy to use.

Bluemix integrates with Jazz, IBM’s DevOps service, to help manage code, plan versions and release, and actually push code to production. You can still use it with your github projects, so no worries there.

And as a SoftLayer customer (or potential customer), you can rest assured that Bluemix projects can run on SoftLayer’s hardware and network.

Core Ideas

The Application
This is your code. Bluemix comes with a number of predefined buildpacks to get your language of choice up and running quickly, but you will still need to actually develop your application. Bluemix hasn’t solved that problem yet.
Buildpacks
A buildpack is a collection of scripts designed to set up your container and all of the application dependencies. If Bluemix doesn’t have a buildpack that suits your needs, you can always create your own. Extending a buildpack is pretty easy. Simply clone an existing one to use as a base, make your changes, commit it to your github repo, and then tell Bluemix about it so it can build your application properly.
Services
Bluemix has a long list of services you can bind to your application. Instead of making a MySQL server yourself, you can just bind the MySQL service to your application and start coding. Along with many of the standard services expected from a CloudFoundry project, there are also some IBM specific ones, like Watson as a service. While I haven’t had the time to learn about Watson personally, everyone I talk to says it’s a rather neat thing to have on your application.

Getting Started

I recommend reading this tutorial which will get you to a nice “hello world” application. Overall I found that going from “I have no idea what Bluemix is” to “I’ve created my own Bluemix application!” to be a rather pleasant experience.

Creating your first Bluemix project is only a few clicks away. A Bluemix 30 day free trial should give you plenty of time to get an idea if Bluemix is the right fit for you.

Bluemix is absolutely worth checking out. So, what are you waiting for? Give it a go!

- Chris

Subscribe to Author Archive: %