Posts Tagged 'Interview'

May 18, 2011

Panopta: Tech Partner Spotlight

This is a guest blog from Jason Abate of Panopta, a SoftLayer Tech Marketplace Partner specializing in monitoring your servers and managing outages with tools and resources designed to help minimize the impact of outages to your online business.

5 Server Monitoring Best Practices

Prior to starting Panopta, I was responsible for the technology and operations side of a major international hosting company and worked with a number of large online businesses. During this time, I saw my share of major disasters and near catastrophes and had a chance to study what works and what doesn't when Murphy's Law inevitably hits.

Monitoring is a key component of any serious online infrastructure, and there are a wide range of options when it comes to monitoring tools — from commercial and open-source software that you install and manage locally to monitoring services like Panopta. The best solution depends on a number of criteria, but there are five major factors to consider when making this decision.

1. Get the Most Accurate View of Your Infrastructure
Accuracy is a dual-edged sword when it comes to monitoring that can hurt you in two different ways. Check too infrequently and you'll miss outages entirely, making you think that things are rosy when your customers or visitors are actually encountering problems. There are tools that check every 30 minutes or more, but these are useless to real production sites. You should make sure that you can perform a complete check of your systems every 60 seconds so that small problems aren't overlooked.

I've seen many people setup this high-resolution monitoring only to be hit with a barrage of alerts for frequent short-lived problems which were previously never detected. It may hurt to find this, but at least with information about the problem you can fix it once and for all.

The flip side to accuracy is that your monitoring system needs to verify outages to ensure they are real in order to avoid sending out false alerts. There's no faster way to train an operations team to ignore the monitoring system than with false alerts. You want your team to jump at alerts when they come in.

High-frequency checks that are confirmed from multiple physical locations will ensure you get the most accurate view of your infrastructure possible.

2. Monitor Every Component of Your Infrastructure
There are lots of components that make up a modern website or application, and any of them could break at any time. You need to make sure that you're watching all of these pieces, whether they're inside your firewall or outside. Lots of monitoring providers focus purely on remotely accessible network services, which are important but only one half of the picture. You also want an inside view of how your server's resources are being consumed, and how internal-only network devices (such as backend database servers) are performing.

Completeness also means that it's economically feasible to watch everything. If the pricing structure of your monitoring tool is setup in a way that makes it cost prohibitive to watch everything then the value of your monitoring setup is greatly diminished. The last thing you want to run into when troubleshooting a complex problem is to find that you don't have data about one crucial server because you weren't monitoring it.

Make sure your monitoring system is able to handle all of your server and network components and gives you a complete view of your infrastructure.

3.Notify the Right People at the Right Time
You know when the pager beeps or the phone rings about an outage, your heart beats a little faster. Of course, it's usually in the middle of the night and you're sleeping right?! As painful as it may be, you want your monitoring system to get you up when things are really hitting the fan - it's still better than hearing from angry customers (and bosses!) the next morning.

However, not all outages are created equally and you may not want to be woken up when one of your clustered webservers briefly goes down and then corrects itself a few minutes later. The key to a successful monitoring solution is to have plenty of flexibility in your notification setup including being able to setup different notification types based on the criticality of the service.

You also want to be able to escalate a problem, bringing in additional resources for long-running problems. This way outages don't go unnoticed for hours while the on-call admin who perpetually sleeps through pages gets more shut-eye.

Make sure that when it comes to notification, your monitoring system is able to work with your team's preferred setup, not the other way around.

4. Don't Just Detect Problems, Streamline Fixing Them
Sending out alerts about a problem is important, but it's just the first step in getting things back to normal. Ideally after being alerted an admin can jump in and solve whatever the problem is and life goes on. All too often though, things don't go this smoothly.

You've probably run into situations where an on-call admin is up most of the night with a problem. That's great, but when the rest of the team comes in the next morning they have no idea what was done. What if the problem comes up again? Are there important updates that need to be deployed to other servers?

Or maybe you have a big problem that attracts interest from your call center and support staff (your monitoring system did alert you before they walked up, right?) Or management from other departments interrupt to get updates on the problem so they can head off a possible PR disaster.

These are important to the operation of your business, but they pull administrators away from actually solving the problem, which just makes things worse. There should be a better way to handle these situations. Given it's central role in your infrastructure management, your monitoring system is in a great position to help streamline the problem solving process.

Make sure your monitoring system gives you tools to keep everyone on the same page by letting everyone easily communicate and log what was ultimately done to resolve the problem.

5. Demonstrate how Your Infrastructure is Performing
Your role as an administrator is to keep your infrastructure up and running. It's unfortunately a tough spot to be in - do your job really well and no one notices. But mess up, and it's clearly visible to everyone.

Solid reporting capabilities from your monitoring system give you a tool to help balance this situation. Be sure to get summary reports that can demonstrate how well things are running or make the argument for making changes and then following up to show progress. Availability reports also let you see a "big picture" view of how your infrastructure is performing that often gets lost in the chaos of day-to-day operations.

Detailed reporting gives you the data you need to accurately assess and promote the health of your infrastructure.

The Panopta Difference
There are quite a few options available for monitoring your servers, each of which come with trade offs. We've designed Panopta to focus on these five criteria, and having built on top of SoftLayer's infrastructure from the very beginning are excited to be a part of the SoftLayer Technology Marketplace.

I would encourage you to try out Panopta and other solutions and see which is the best fit to the specific requirements for your infrastructure and your team - you'll appreciate what a good night's sleep feels like when you don't have to worry about whether your infrastructure is up and running.

-Jason Abate, Panopta

This guest blog series highlights companies in SoftLayer's Technology Partners Marketplace.
These Partners have built their businesses on the SoftLayer Platform, and we're excited for them to tell their stories. New Partners will be added to the Marketplace each month, so stay tuned for many more come.
May 4, 2011

3 Bars | 3 Questions: Community Development

I've been on the hook for a 3 Bars | 3 Questions interview for a few weeks now, and I finally found a few minutes to chat with Kevin about what's going on in the world of SoftLayer Community Development. In the past two months, we've cranked everything up to 11 with the unveiling of our Technology Incubator Program and the Technology Partners Marketplace. Needless to say, we had a lot to talk about:

Over the past few weeks, we've posted video interviews and guest blogs from a few of our featured Technology Partner Marketplace participants, and you can expect to see more where that came from as we sign on new partners with killer applications and services that we can share with our customers. If you want to be one of those new partners, fill out our quick application, and we'll get the ball rolling!

I'm looking forward to the next installment of "3 Bars | 3 Questions" because "The Mitch" - the man, the myth, the legend - will be in the hot seat.

The Mitch

-@PaulFord

April 27, 2011

AppFirst: Tech Partner Spotlight

This is a guest blog from AppFirst, a SoftLayer Tech Marketplace Partner specializing in managing servers and applications with a SaaS-based monitoring solution.

How You Should Approach Monitoring in the Cloud

Monitoring in the cloud may sound like it's easy, but there's one important thing you need to know before you get started: traditional monitoring techniques simply don't work when you're in the cloud.

"But why?" you may ask. "Why can't I use Polling and Byte Code Injection in my cloud infrastructure?"

With Polling, you miss incidents between intervals, you only get the data that you requested, and you can only monitor parts of the application but not the whole thing. If you choose to use Polling for your cloud monitoring, you'll have to deal with missing important data you need.

And with Byte Code Injection, you only get data from within the language run-time, meaning you don't have the real data of what is happening across your application stack. It is inferred.

Using our own product on our production systems, we have learned three lessons about running in the cloud.

Lesson #1: Visibility = Control
By definition, running in the cloud means you are running in a shared environment. You don't have the CPU cycles your operating system reports you have, and sometimes, the hypervisor will throttle you. In our experience, some cloud vendors are much better at managing this than others. When running in some clouds, we've had huge variations in performance throughout the day, significantly impacting our end-users experience. One of the reasons we chose SoftLayer was because we didn't see those kinds of variances.

The reality is until you have visibility into what your application truly needs in terms of resources, you don't have control of your application and your user's experience. According to an Aberdeen study, 68% of the time IT finds out about application issues from end users. Don't let this be you!

Lesson #2: It's Okay to Use Local Storage
The laws of physics reign, so the disk is always the slowest piece. No getting around the fact there are physical elements involved like spindles and disks spinning. And then when you share it, as you do in the cloud, there can be other issues ... It all depends on the characteristics of your application. If it's serving up lots of static data, then cloud-based storage can most likely work for you. However, if you have lots of dynamic, small chunks of data, you are probably best served by using local storage. This is the architecture we had to go with given the nature of our application.

With servers around the world streaming application behavior data to our production system all the time and needing to process it to make it available in a browser, we had to use local storage. In case you are interested in reading more on this and RAM based designs here are some posts:

Lesson #3: Know the Profile of Your Subsystems
Knowing the profile of your subsystems and what they need in terms of resources is imperative to have the best performing application. A cloud-only deployment may not be right for you; hybrid (cloud and dedicated physical servers) might work better.

As we discussed in Lesson #2 you might need to have local, persistent storage. Again, some vendors do this better than others. SoftLayer, in our experience, has a very good, high bandwidth connection between their cloud and physical infrastructure. But you can't make these decisions in a vacuum. You need the data to tell you what parts of your application are network heavy, CPU intensive, and require a lot of memory in certain circumstances. We have learned a lot from using our own application on our production system. It's very quick and easy for you to start learning about the profile of your application too.

We are constantly learning more about deploying in the cloud, NoSQL databases, scalable architectures, and more. Check out the AppFirst blog regularly for the latest.

We'd like to give a special shout out thanks to SoftLayer! We're honored to be one of your launch partners in the new Technology Partners Marketplace.

-AppFirst

This guest blog series highlights companies in SoftLayer's Technology Partners Marketplace.
These Partners have built their businesses on the SoftLayer Platform, and we're excited for them to tell their stories. New Partners will be added to the Marketplace each month, so stay tuned for many more come.
Categories: 
March 11, 2011

3 Bars | 3 Questions: SoftLayer Sales

Will Charnock put me in the hot seat at the end of his 3 Bars | 3 Questions interview last week, so I welcomed Kevin into my office virtually to have a quick chat. He came equipped with three good questions about my experience with SoftLayer SLales, and I did my best to give three good answers. Here were the questions:

  1. What makes SoftLayer different from any of the other companies you've worked for?
  2. When you're hiring a new sales representative, what do you look for in that person?
  3. What are you most excited about when it comes to the next few months at SoftLayer from a sales perspective?

This week, we used a High Definition video chat, and the quality is pretty impressive. We're still working to improve and tweak the format and quality of these videos, so you might notice a few blips in the audio recording, but we'll get those ironed out soon.

All in all, the video chat was a lot of fun, and I'm looking forward to watching Drew Jenkins enjoy it in the next episode!

-Tom

March 4, 2011

3 Bars | 3 Questions: IPv6

Thanks to Marc's vote, I had the distinct honor of being the third guest on our "3 Bars | 3 Questions" series. The topic of conversation: IPv6.

Are we in a "The sky is falling!" situation yet? How can customers put pressure on their ISPs and software providers to add IPv6 support? How long with ARIN and SoftLayer have IPv4 addresses to give out now that IANA has released their entire free pool? Here's my take:

This video was recorded while Kevin was standing outside the Moscone Center in San Francisco at GDC 2011 over a mobile hotspot connection, so the video quality suffered a little. To silence the street noise, Kevin muted his side of the conversation as I spoke.

-Will

February 3, 2010

Custom Server Solutions

The other day I was at a well known coffee shop (take a wild guess) and was steamrolled by all of the marketing hype. Try this! Take one of these home! Only for a limited time! Often the presentation of too many options makes the decision a lot more difficult. I know it’s just coffee or tea but now I have the sudden urge to collect them all! And despite years of caffeine conditioning I don’t think my heart, stomach, or my bank account could take a collecting and consuming of all. I’m looking for something different this time, but again, too many options. The next logical step is asking the Barista what their favorite is. I figure they spend their entire workday around the stuff; they MUST have a good recommendation. All I was really looking for here was a “get this” and call it good. Only after I asked, did I remember that most people who frequent this caliber of coffee joint are really particular about their coffee. I, however, am not one of the ¼ soy milk, ¼ cream, no froth, low-fat, exactly 1723 crystals of sugar type of people, so I’m not really prepared for what comes next. Instead of a one-size-fits-all answer, I’m getting a barrage of questions about my preferences. While this was not really what I was after, it hit me that this Barista is building me a solution. I did, in fact, leave with a tasty seasonal coffee, custom tailored to my needs. Servers are a lot like coffee, they rely on gratuitous amounts of caffeine to be any good; that and, there is usually never a generic solution that is going to suit your needs. The sales team at SoftLayer is not there solely to assist you in placing an order for you; they are there to ask you questions about your intentions with the server so they can recommend the best possible solution. You can have your low-fat CentOS with a double-shot of 5570’s with “venti” gigs of ram. Just ask our sales team to brew you up a solution.

June 20, 2007

An Interview With an Elevator

SL: Good morning, thank you for taking the time to meet with me.
Elevator: Ding.

SL: Excellent. How would you describe the costs maintaining efficiencies in a hosting environment?
Elevator: Going up.

SL: Well, I think that’s obvious, depending on where you start. Perhaps a better way to phrase this would be, “How would you recommend leveraging existing technologies to implement an efficient execution of a hosting environment?”
Elevator: Ground floor

SL: Well said. I agree that it becomes difficult to put solutions into place after-the-fact, and that in order to run smoothly one must start with a solid plan and avoid retrofitting later. That ends up being far too costly and stifles resources a company should be using to grow their product. How would you describe the attitude of most large hosts with regards to “going green”?
Elevator: Please step away from the door.

SL: I too think that many datacenters out there are concerned with “stepping through” as it were to move operations in that direction. But, since the datacenters can hugely benefit from cost-savings due to reduced expenditures for cooling and power, it is very much worth the shift. What factors outside of the DC could play into making this shift easier?
Elevator: Lobby

SL: Well, I’m not sure that lobbying is the answer, though it may help. Really I was asking about computer manufacturers making the shift to properly-matched and high efficiency power supplies and processors. New technologies are making it easier for younger companies to go green, and older hosts are left trying to figure out how they can turn thousands of antiquated servers into efficient appliances. This goes back to your earlier comment regarding starting out with a solid plan making it easier to
Elevator: Ding

SL: Don’t interrupt me. Easier to maintain a plan than adjust and retrofit to a new one.
Elevator: Second Floor

SL: I’m not sure why you said that, it doesn’t make any sense. Having a host that doesn’t play catch-up constantly benefits the customer in several
Elevator: Ding

SL: Stop it.
Elevator: Third Floor

SL: You’re an idiot. I’m going to go interview the printer.

Categories: 
Subscribe to interview