introductions

July 17, 2009

Combinations

As you may have noticed from a previous blog a while back, we here at SoftLayer really like Whataburger quite a bit. For those of you not lucky enough to live in the southern part of the US, Whataburger has a chain of hamburger restaurants that started in Texas but now stretches across the south from Florida to Arizona. For the record, their breakfast taquitos and hamburgers are second to none.

Every so often Whataburger runs an ad that states there are over 34,000 different ways that you can order your Whataburger. Every time I see this ad it makes me wonder, how many possible ways can you order a server with SoftLayer? I’ve always said to myself “one day I will calculate this and write a blog about it”, but I never seemed to get around to it until now. My ultimate motivation to complete this calculation came while discussing our “perceived” limited configuration options with a customer due to having a set of “standard” solutions.

As I started to research the math involved in this calculation, I quickly realized that the total number of combinations for all servers and services we offer would be too much to comprehend or calculate. Therefore I limited the scope to just options for a Nehalem processor server. I also enlisted the help of Greg Kinman, one of our interns, to help me with the math to make sure we got it right. Those Kinmans are multiplying around here (almost as fast as the configuration options)…

Greg calculated that there are 96,631,664,476,185,600,000 different ways to order a Nehalem server. That’s 97 quintillion possibilities for just one server when you take into account the various hardware, software, and service options! Once you start to consider ALL the server options and cloud instances offered, it seems to me that unlimited would be a better description for the configurations SoftLayer offers.

Greg, I owe you lunch for helping me with the calculations today. How about Whataburger?

July 15, 2009

Subjecting Subjectivity To Math

I recently read an article about an endeavor that is currently being undertaken to develop a “Speech Analysis Algorithm Crafted to Detect and Help Dissatisfied Customers”. In short, a team of engineers are hoping to create software that will recognize when a caller is becoming stressed and immediately phone a manager to alert them of a developing situation. Wow! It is rare that you would see math and science applied to something that is so subjective. After all, math is used to quantify and measure things all based on a known or a baseline. In this particular effort, I would surmise that the team of engineer’s most difficult task will be to determine how to establish a unique baseline for each unique call and caller. Once upon a time as a student of Electrical Engineering, I took on my share of convolution integrals and that’s a path that I do not care to venture down again. I’ve also taken on my share of convoluted customer calls in a past life and witnessed our frontline assisting customers in complex situations here at SoftLayer.

Until there is such an application that can detect and address a conversation that may be heading in the wrong direction, we have to rely on good ole’ training and experience. With each call and query, the baseline is reset. I’d even go further to say that with each exchange; the baseline is reset as our Customer Service Agents seek information to get to the root of the issue. It’s not hard to imagine the frustration that can build in a back-and-forth conversation as two people look to come to a solution or an amiable conclusion just as it is understandable that sometimes, a customer may simply need to vent. How do you calculate and anticipate those scenarios?

I wish much success to the team involved in the customer service speech analysis program. And programmatically speaking, I see many CASE, SWITCH, FOR, WHILE, BREAK, CONTINUE, IF, ELSE, ELSE IF, NEXT statements in your future. Good Luck!

July 13, 2009

What a View!

I can easily define myself as the crazy one up in the Seattle Datacenter. I like to ride dirt bikes, street bikes, go fast on the water, ride in small airplanes, I could go on and on how my co-workers (and friends/family) may think I am crazy when it comes to Adventures.

What can I say, I like a challenging experience.

One of those challenging experiences is working at SoftLayer, always preparing to be ahead of the rest in this industry, we're constantly learning new technologies and taking leaps and bounds. That is the reason why I love my job so much, we're always working with the latest and greatest, learning new stuff. Speaking of leaps and bounds, I finally did something the other day I have always wanted too. I signed up for an Advance Free Fall Skydive class and jumped out of a Cessna at 13,000 feet. Free falling at terminal speeds towards the earth, At 12,000 feet I mock pulled my parachute 3 times, so the instructors who where both holding onto me by their hands only could see I learned what to do. 6000 ft came, I locked onto my altimeter, 5500 I waived hands off to the instructors and they deployed below me, and I pulled my rip cord. That all happened in about 50 seconds after leaping out of the plane. For the next 8 minutes I saw the best view in Western Washington I have ever seen. One of the thoughts that came to my mind is the only thing to relate to how I have ever thought something was this nice, was the first time I walked into a SoftLayer Datacenter and admired how well thought out and nice it was.

I'm glad to say 19 months into this job and being part of the Operations team in Seattle, I walk in each day to the datacenter and can say the same thing day in and day out. Let's hope I can say the same thing about my second jump in a few weeks.

July 10, 2009

The Kinmans are Taking Over

Yeah, it’s official…we have another one: me, even though I’m only a summer intern. The day after my 16th birthday couldn’t have been a better opportunity for me to begin my quest for company domination. Well, I think that will come later.

Anyway, it is my first job at my first company, ever. Softlayer’s really nice. They paid me one hour’s wages for sitting in a conference room and filling out a frightening mountain of paperwork (OK, well, maybe it was more like 5 or 6 forms), and again another day for visiting a few datacenter rooms in the Infomart (I think it was compensation for hearing damage from the noise in the rooms). Michael Scott would have scoffed and turned his empty pockets inside out. Anyway, my point is to tell you that the Kinman clan is taking over Softlayer.

Yep, you got that right.

Steve has 4 kids who will probably grow up and get promoted to manage some section of the company each. Gary’s kid (me) already has a spot, and I’m up to my ears in details about servers, processors, RAM, disk drives, routers, switches, fiber, etc. Heck, maybe Mama Carol and Papa Willie will come back out of retirement to work here too. Oh and Steve’s wife and my mom and…well, you get the idea. If your name is Kinman, you’re gonna work at Softlayer, whether you like it or not.

Someday, you’re gonna wake up in some kind of cubicle in a building on International Parkway and you’re gonna wonder how the heck you ended up there. And HR is just gonna tell you, “Well…your last name IS Kinman…” They may have to stop paying us to fill out forms to keep from going bankrupt because there’s gonna be so many of us. With that in mind, I’d say that the average current non-Kinman Softlayer employee has about three years, plus-or-minus, before the co-worker next to them is, you guessed it, a Kinman. Wow. Congrats Carol and Willie.

…just kidding. But speculation is fun, right? Until it actually happens. Muahahahahaha.

July 8, 2009

Encrypted Hot Chocolate

Imagine this scene: you’re sitting at a local coffee shop, having a drink and browsing the web. While checking out your favorite news site you see an e-mail come in where someone is commenting on your blog post from that morning. This is odd because while you remember checking blogs, you don’t remember posting one. On investigating you find a blog post that you definitely did not create. As you look around wondering what is going on you should probably take a peek at the guy sitting in the comfy chair with his headphones on running a wireless sniffer to grab passwords out of the air.

How does this happen? The coffee shops I’ve seen all run open wireless access points. This is great for flexibility and serving the most people but if the Access Point is unencrypted it is quite possible to run an application which will listen to the wireless network and record packets. These packets can then be examined by a tool like the Wireshark application. Since the packets are not encrypted then things like username/password combos can be read in clear text from the packet.

Knowing what the problem is, how do we work around it? Since we cannot encrypt the wireless session, we’ll encrypt the data itself. One option would be to do a VPN from your laptop in the coffee shop to a location out in the world. This would channel all of your traffic out through this other system. It’s a good solution but it can be a bit technically complicated if you don’t have one already set up. If you’re really only concerned with encrypting your HTTP traffic you could use the PuTTy application to tunnel traffic via an encrypted session to a Unix server here at SoftLayer by using the OpenSSH ability to act as a SOCKS5 proxy.

When you define your connection in PuTTy you can go down to Connection > SSH > Tunnels and then place a port number, such as 8080, in the Source field. Select “Dynamic” and “Auto” then click the Add button. Connect to your Unix server here at SoftLayer. Next stop is the browser. The way I configure my browser for this is to go into Firefox > Tools > Options > Advanced > Network > Settings (under Connection). Select “Manual proxy configuration” and then in the field labeled SOCKS Host: put “127.0.0.1” and for the port use the port you specified above. Leave the type as “SOCKS v5”. Select OK and then in the URL bar type “about:config” which will let you do advanced configuration. In the filter field type in “network.proxy.socks_remote_dns”. Right click on it and select Toggle. This will mark it true.

Now if you pull up a website which will tell you the IP that you are coming from (such as http://whatismyip.com) you should see it report back the IP address of your server here at SoftLayer. This happens because Firefox has been told to use 127.0.0.1:8080 as its SOCKSv5 proxy and this traffic gets tunneled via the encrypted SSH session to your server at SoftLayer. The server here will do the DNS lookup (due to network.proxy.socks_remote_dns) and then send out the request. The response will be tunneled back to your browser.

You do have to remember to fire up the PuTTY session but this isn’t so hard since if you try and browse without it the browser tries to hit the SOCKSv5 proxy port specified and fails. Beyond that I’ve not run into any troubles using this trick.

And now I think I will head off for some hot chocolate myself.

Categories: 
July 6, 2009

The Cure for Irrelevance

I’ve been feeling rather irrelevant lately……yeah, yeah, I know . Watch out, because when lawyers feel irrelevant we sue!

Anyway, just thinking about the things going on in the rest of the world, like the wars in Iraq and Afghanistan. We have men and women over there who are still dying and losing their sanity on a daily basis, but for those of us who don’t have anyone close to us involved in the wars, it’s become a low hum…car bomb, soldier killed, hum, hum. But then I think of the daily terror that those who have loved ones go through – did she have to go out on patrol today? Did she get hurt? Did she get killed? Every day, every hour spent wondering if that person is safe and will come home again. That soldier is not irrelevant – he is making the greatest sacrifice so we can go on with our safe, secure lives over here.

My thoughts also turn to the people of Iran, and I find myself thinking: “If I lived in a repressive regime, would I be out on the streets in defiance of the government and particularly with the threat of being beaten, jailed or disappearing from the face of the earth?” I like to think I would, but I don’t know the answer, and that feeds my irrelevance. And come on Iran; give your people some credit. Make it at least look like the election wasn’t predetermined. You declare a landslide victory for Ahmadinejad (cut and paste, baby, cut and paste!) mere minutes after the polls close, yet the ballots are supposed to be hand counted. How can that work? I mean, wait an hour or so – pretend you counted. Please! Are the people of Iran irrelevant? No, they are making the greatest sacrifice in a battle for freedom, and in an uprising that may very well change the course of their history. The world is watching.

So how do I become relevant? (Assuming, that is, that a lawyer can ever be relevant). How do SLayers and SLackers become relevant? We go that extra mile. We’ve been dealing with cranky clients all day – keep the smile on the face and in the voice and treat them like they’ve just put in an order for 300+ servers a month. We don’t remain satisfied with the status quo – figure out how to make our system and services better, stronger, faster. We don’t rest on our laurels because we just had a major release, i.e., Cloudlayer instances. We look ahead and figure out or invent the next new thing our customers need or want. We scan the forums to keep a pulse on our clients (and it’s usually good for an eye-roll or two). We keep Lance and Mike out of trouble.

So am I like a U.S. soldier or the Iranian people? Not so much, but I can do things to stay relevant in my own little world.

Categories: 
July 4, 2009

Fourth of July

Fourth of July – Independence Day is more than just a day for us to hang out with friends and family across the United States and gather around the BBQ and watching fireworks and bombs blow up. It is a day that we celebrate our founding fathers courage and bravery in the pursuit of liberty and freedom.

If it wasn’t for these men and their dreams, I would not be sitting here at SoftLayer writing this blog for a company that loves us to share our words and views with others. I have been amazed how over the last few weeks how Twitter and other sites have helped the country of Iran speak their voice and let the world know what is going on over there. We would never know what is going on as their government would not allow it to be voiced on the state ran television.

So, as I am camping this Fourth of July in the San Juan Islands, fishing on the lake and watching the skies over Friday Harbor light up, I will be thankful for what our founding fathers accomplished on that day in 1776.

Categories: 
July 1, 2009

Pre-configuration and Upgrades

I recently bought a new computer for my wife. Being a developer, and a former hardware engineering student, I opted the buy the parts and assemble the machine ourselves. Actually assembling a computer these days doesn't take too long, it's the software that really gets you. Windows security updates, driver packs, incompatibilities, inconsistencies, broken websites, and just plain bad code plagued me for most of the night. The video card, in particular, has a “known issue” where it just “uh-oh” turns off the monitor when Windows starts. The issue was first reported in March of 2006, and has yet to be fixed.

This is why SoftLayer always tests and verifies the configurations we offer. We don't make the end user discover on their own that Debian doesn't work on Nehalems, we install it first to be sure. This is also why our order forms prevent customers from ordering pre-installed software that are incompatible with any of the rest of the order. We want to make sure that customers avoid the frustration of ordering things only to find out later that they don't work together.

The problem with desktop computers, especially for people who are particular about their configurations, is that you cannot buy a pre-configured machine where all the parts are exactly what you want. We attempted to get a computer from Dell, and HP, but neither company would even display all the specifications we were interested in, nevermind actually having the parts we desired. Usually pre-built systems skimp on important things like the motherboard or the power supply, giving you very little room to upgrade.

At SoftLayer, we don't cut corners on our systems, and we ensure that each customer can upgrade as high as they possibly can. Each machine type can support more RAM and hard drives than the default level, and we normally have spare machines handy at all levels so that once you outgrow the expansion capabilities of your current box, you can move to a new system type. If you're thinking of getting a dedicated server, but you're worried about the cost, visit the SoftLayer Outlet Store and start small. We have single-core Pentium Ds in the outlet store, and you can upgrade from there until you're running a 24-core Xeon system.

June 29, 2009

Leaving Normal

What is normal for a server? In support we get that question from time to time. The problem is that normal varies from server to server. A load average of 200 is probably not normal but a load of 5 to 10 very well could be normal, depending on the server's application. What to do?

Baselining to the rescue. The idea behind baselining is to get performance numbers on your application when things are "normal" so that you have solid math to indicate when things are not "normal".

What makes a good baseline? Things like RAM use (overall, per process, rate of change), number and types of processes running, processor usage, disk usage (total, per app), disk speed and network utilization are all good OS metrics. You can also get metrics from your application. E-mails per hour, web page generation time, and number of users logged in are good to know.

You can capture OS metrics using tools like top, free, ps and iostat on Linux. Actually if you have iostat you probably have 'sar' which is great for performance history. Sar has a process that runs every few minutes and records various OS counters including processor info, RAM use, disk I/O and the like.

For the Windows people you have Task Manager and Performance Monitor. Task Manager is pretty simple and gives mostly an overview. Perfmon is really where its at on Windows. Using PerfMon you can track dozens of performance counters on disk, proc, memory, the network and even application specific metrics if you are running apps like MS Exchange that support them.

As with most tasks related to being the lord and master of a server, performance monitoring isn't a one time thing. As you make changes to the system you have to run new baselines. Between changes you should run your performance routines periodically to see how things are changing. It is much easier to look into an issue if you spot it earlier rather than later.

Go forth and make sure all your baselines are belong to you!

*bonus cool points for those who knew the title of this blog was also the title of a "Roswell" episode.

Categories: 
June 24, 2009

Clouds and Elephants

So there I was after work today, sitting in my favorite watering hole drinking my Jagerbomb, when Caira, my bartender asked what was on my mind. I told her that I had been working with clouds and elephants all day at work and neither of those things are little. She laughed and asked if I had stopped anywhere to get a drink prior to her bar. I replied no, I'm serious I had to make some large clouds and a stampede of elephants work together. I then explained to her what Hadoop was. Hadoop is a popular open source implementation of Google's MapReduce. It allows transformation and extensive analysis of large data sets using thousands of nodes while processing peta-bytes of data. It is used by websites such as Yahoo!, Facebook, Google, and China's best search engine Baidu. I explained to her what cloud computing was (multiple computing nodes working together) hence my reference to the clouds, and how Hadoop was named after the stuffed elephant that belonged to one of the founders - Doug Cutting - child. Now she doesn't think I am as crazy.

Categories: 

Pages

Subscribe to introductions