Posts Tagged 'Maintenance'

October 1, 2014

Virtual Server Update

Good morning, afternoon, evening, or night, SoftLayer nation.

We want to give you an update and some more information on maintenance taking place right now with SoftLayer public and private node virtual servers.

As the world is becoming aware today, over the past week a security risk associated with Xen was identified by the Xen community and published as Xen Security Advisory 108 (XSA-108).

And as many are aware, Xen plays a role in our delivery of SoftLayer virtual servers.

Eliminating the vulnerability requires updating software on host nodes, and that requires downtime for the virtual servers running on those nodes.

Yeah, that’s not something anyone likes to hear. But customer security is of the utmost importance to us, so not doing it was not an option.

As soon as the risk was identified, our systems engineers and technology partners have been working nonstop to prepare the update.

On Sunday we notified every customer account that would be affected that we would have emergency maintenance in the middle of this week, and updated that notice each day.

And then yesterday we published that the maintenance would begin today at 3pm UTC, with a preliminary order of how the maintenance would roll out across all of our data centers.

We are updating host nodes data center by data center to complete the emergency maintenance as quickly as possible. This approach will minimize disruption for customers with failover infrastructure in multiple data centers.

The maintenance is under way and SoftLayer customers can follow it, live, on our forum at http://sftlyr.com/xs101.

-@SoftLayer

August 22, 2013

Network Cabling Controversy: Zip Ties v. Hook & Loop Ties

More than 210,000 users have watched a YouTube video of our data center operations team cabling a row of server racks in San Jose. More than 95 percent of the ratings left on the video are positive, and more than 160 comments have been posted in response. To some, those numbers probably seem unbelievable, but to anyone who has ever cabled a data center rack or dealt with a poorly cabled data center rack, the time-lapse video is enthralling, and it seems to have catalyzed a healthy debate: At least a dozen comments on the video question/criticize how we organize and secure the cables on each of our server racks. It's high time we addressed this "zip ties v. hook & loop (Velcro®)" cable bundling controversy.

The most widely recognized standards for network cabling have been published by the Telecommunications Industry Association and Electronics Industries Alliance (TIA/EIA). Unfortunately, those standards don't specify the physical method to secure cables, but it's generally understood that if you tie cables too tight, the cable's geometry will be affected, possibly deforming the copper, modifying the twisted pairs or otherwise physically causing performance degradation. This understanding begs the question of whether zip ties are inherently inferior to hook & loop ties for network cabling applications.

As you might have observed in the "Cabling a Data Center Rack" video, SoftLayer uses nylon zip ties when we bundle and secure the network cables on our data center server racks. The decision to use zip ties rather than hook & loop ties was made during SoftLayer's infancy. Our team had a vision for an automated data center that wouldn't require much server/cable movement after a rack is installed, and zip ties were much stronger and more "permanent" than hook & loop ties. Zip ties allow us to tighten our cable bundles easily so those bundles are more structurally solid (and prettier). In short, zip ties were better for SoftLayer data centers than hook & loop ties.

That conclusion is contrary to the prevailing opinion in the world of networking that zip ties are evil and that hook & loop ties are among only a few acceptable materials for "good" network cabling. We hear audible gasps from some network engineers when they see those little strips of nylon bundling our Ethernet cables. We know exactly what they're thinking: Zip ties negatively impact network performance because they're easily over-tightened, and cables in zip-tied bundles are more difficult to replace. After they pick their jaws up off the floor, we debunk those myths.

The first myth (that zip ties can negatively impact network performance) is entirely valid, but its significance is much greater in theory than it is in practice. While I couldn't track down any scientific experiments that demonstrate the maximum tension a cable tie can exert on a bundle of cables before the traffic through those cables is affected, I have a good amount of empirical evidence to fall back on from SoftLayer data centers. Since 2006, SoftLayer has installed more than 400,000 patch cables in data centers around the world (using zip ties), and we've *never* encountered a fault in a network cable that was the result of a zip tie being over-tightened ... And we're not shy about tightening those ties.

The fact that nylon zip ties are cheaper than most (all?) of the other more "acceptable" options is a fringe benefit. By securing our cable bundles tightly, we keep our server racks clean and uniform:

SoftLayer Cabling

The second myth (that cables in zip-tied bundles are more difficult to replace) is also somewhat flawed when it comes to SoftLayer's use case. Every rack is pre-wired to deliver five Ethernet cables — two public, two private and one out-of-band management — to each "rack U," which provides enough connections to support a full rack of 1U servers. If larger servers are installed in a rack, we won't need all of the network cables wired to the rack, but if those servers are ever replaced with smaller servers, we don't have to re-run network cabling. Network cables aren't exposed to the tension, pressure or environmental changes of being moved around (even when servers are moved), so external forces don't cause much wear. The most common physical "failures" of network cables are typically associated with RJ45 jack crimp issues, and those RJ45 ends are easily replaced.

Let's say a cable does need to be replaced, though. Servers in SoftLayer data centers have redundant public and private network connections, but in this theoretical example, we'll assume network traffic can only travel over one network connection and a data center technician has to physically replace the cable connecting the server to the network switch. With all of those zip ties around those cable bundles, how long do you think it would take to bring that connection back online? (Hint: That's kind of a trick question.) See for yourself:

The answer in practice is "less than one minute" ... The "trick" in that trick question is that the zip ties around the cable bundles are irrelevant when it comes to physically replacing a network connection. Data center technicians use temporary cables to make a direct server-to-switch connection, and they schedule an appropriate time to perform a permanent replacement (which actually involves removing and replacing zip ties). In the video above, we show a temporary cable being installed in about 45 seconds, and we also demonstrate the process of creating, installing and bundling a permanent network cable replacement. Even with all of those villainous zip ties, everything is done in less than 18 minutes.

Many of the comments on YouTube bemoan the idea of having to replace a single cable in one of these zip-tied bundles, but as you can see, the process isn't very laborious, and it doesn't vary significantly from the amount of time it would take to perform the same maintenance with a Velcro®-secured cable bundle.

Zip ties are inferior to hook & loop ties for network cabling? Myth(s): Busted.

-@khazard

P.S. Shout-out to Elijah Fleites at DAL05 for expertly replacing the network cable on an internal server for the purposes of this video!

August 15, 2012

Managing Support Tickets: Email Subscriptions

This week, the development team rolled out some behind-the-scenes support functionality that I think a lot of our customers will want to take advantage of, so I put together this quick blog post to spread the word about it. With the new release, the support department is able to create "Ticket Email Subscriptions" for different ticket groups on every customer account. As a customer, you might not be jumping up and down with joy after reading that one-sentence description, but after you hear a little more about the functionality, if you're not clapping, I hope you'll at least give us a thumbs-up.

To understand the utility of the new ticket email subscription functionality, let's look at how normal tickets work in the SoftLayer portal without email subscriptions:

User Creates Ticket

  1. User A creates a ticket.
  2. User A becomes the owner of that ticket.
  3. When SoftLayer responds to the ticket, an email notification is sent to User A to let him/her know that the ticket has been updated.

SoftLayer Creates Ticket

  1. SoftLayer team creates a ticket on a customer's account.
  2. The primary customer contact on the account is notified of the new ticket.
  3. Customer logs into the portal and responds to ticket.
  4. Customer gets notifications of updates (as described above).

There's nothing wrong with the existing support notification process, but that doesn't mean there aren't ways to make the process better. What if User A creates an urgent ticket on his/her way out the door to go on vacation? User B and User C aren't notified when an update is posted on User A's ticket, so the other users aren't able to get to the ticket and respond as quickly as they would have if they received the notification. What if the primary customer contact on the account isn't the best person to receive a monitoring alert? The administrator who will investigate the monitoring alert has to see the new ticket on the account or hear about it from the primary contact (who got the notification).

Ticket email subscriptions allow for customers to set contact addresses to be notified when a ticket is created, edited or moved in a particular ticket group. Here are the ticket groups differentiated in our initial release:

  • Billing - Any ticket in our Billing department
  • Maintenance - Scheduled maintenance notifications for specific servers
  • Network Protection - DDoS mitigation and Null Routes
  • Monitoring - Host Down Alerts
  • CST, SysAdmin and Hardware - Any ticket in our support and data center departments
  • Managed Services - Tickets that relate to any managed services
  • Network Maintenance - Scheduled network maintenance

You'll notice that Abuse isn't included in this list, and the only reason it's omitted is because you've always been able to designate a contact on your account for abuse-related tickets ... Ticket subscriptions extend that functionality to other ticket groups.

Because only one email address can be "subscribed" to notifications in each ticket group, we recommend that customers use their own distribution lists as the email contacts. With a DL as the contact, you can enable multiple users in your organization to receive notifications, and you can add and remove users from each distribution list on your end quickly and easily.

When User A creates a ticket with the data center and goes on vacation, as soon as SoftLayer responds to the ticket, User A will be notified (as usual), and the supportsubscription@yourdomain.com distribution will get notified as well. When a network maintenance is ticket is created by SoftLayer, the netmaintsubscription@yourdomain.com distribution will be notified.

Ticket email subscriptions are additive to the current update notification structure, and they are optional. If you want to set up ticket email subscriptions on your account, create a ticket for the support department and provide us with the email addresses you'd like to subscribe to each of the ticket groups.

We hope this tool helps provide an even better customer experience for you ... If you don't mind, I'm going to head back to the lab to work with the dev team to cook up more ways to add flexibility and improvements into the customer experience.

-Chris

March 7, 2011

March Madness - Customer Experience Style

If you are a SoftLayer customer you probably noticed a maintenance window early Sunday morning. If you aren't a SoftLayer customer, (you should be, and) you may have even noticed on quite a few social media outlets that we were trying to provide real-time updates about the maintenance progress, and our customers were doing so as well.

SoftLayer customers were given two internal tickets notifying them if they were to be affected, and when those tickets were created, the ticket system would have then sent an email to the admin user on that account. Additionally, our portal notification system was updated to show details about the window, and we created new threads in our customer forums to provide regular, centralized updates. We went as far as taking a few calls and meetings with customers to talk about their concerns with the maintenance timing and length because we know that any downtime is bad downtime in the world of hosting.

Saturday night, we had extra support on staff online, and our social media ninja was awake and letting the world know step by step what we were doing with real time status alerts. We wanted to be extremely transparent during the entire process. This was not a maintenance we could avoid, and we tried to roll as many different things that needed work into this maintenance without making a roll back impossible.

The maintenance itself went well, and as planned, most items that were taken down were back online well before the window ended. We ran into a few snags in bringing all of the CloudLayer CCIs back online, but even with those delays for a few customers, the work was completed by the time we committed to.

Now for the customer experience aspect. From reading various tweets from our customers, it seems like we should/could have done a few things even better: Been more proactive, sent standard email, attempted phone calls, etc.

While some of these options may be considered, not all are feasible. If you are one of the customers that tweeted, has blogged, is planning on tweeting, is planning on blogging or believes we're being anything less than genuine and transparent on our social media platforms, I want to hear from you.

Please comment on this blog, tweet me @skinman454, email me skinman@softlayer.com, call me at 214.442.0592, come by our office and visit.

Whatever it takes, just contact me. I can't put myself in your shoes and feel your pain on things like this unless we have a chance to talk about it. I look forward to our conversation.

-Skinman

June 1, 2009

Proper Care

Over the years I have had many motorized toys, including boats, cars, trucks, dirt bikes, quads, riding lawn mowers and others. I got my first mini bike when I was about 6 years old. That thing was powerful - it had a 4HP Engine on it. One day I was riding it on our 100 acre homestead and the chain broke. Well I just popped the kick stand up and left it there waiting for Dad to get home. Upon my father arriving I let him know the chain broke, he explained to me the proper maintenance one must do in order to keep a chain working, proper oiling techniques, making sure it has the right tension and more. A few years later I got my first two stroke dirt bike. I loved that thing! I rode it all weekend long and then I mixed the gas too lean and blew the top head of it. That’s when I learned how to maintain a 2 cycle engine. My uncle helped me rebuild the bike engine (or shall I say I handed him the tools, and he rebuilt it) With all motorized engines they need proper care and maintenance. I now take my car for an oil change every 4000 miles (even though they say it can go 5,000) and get everything checked out.

The same thing can be said for internet servers. Quite often I talk to people and they think they can just install their operating system, upload the applications they want to run and/or data they want to serve, and walk away from that machine for the next 12 months. That is wrong! Computer software is always updating and you need to stay on top of updating your software. Security threats are found hourly, Viruses are written daily to attack the threats found yesterday. Proper maintenance is the only way to make sure your data is safe and secure. That is why SoftLayer has partnered with companies that offer extended server management. We call them SoftLayer Certified Management Companies. You can find them in our forums. These companies like rackaid.com, seeksadmin.com, Bitpusher, and many more have all been certified by SoftLayer to know our infrastructure and work closely with us and many of our clients. They provide the same great level of customer service that is standard at SoftLayer and do a lot of the advanced administration tasks for our customers. We have teamed up with these managed services partners in order to provide our customers with the proper maintenance of their infrastructure. So if you haven’t done a security audit on one of your machines in a few months, I would suggest taking it to the service center and contacting one of these companies, so you can insure your machine is safe and secure!

April 22, 2008

Buying a House vs. Buying a Server

Thinking of buying a house – don’t. I have been through the 7 layers of candy cane to get into the one I just bought. Not only did I have to search to find the right Real Estate Agent that fit my needs and my busy schedule, I had to search around for the right house, in the right location for the right price – sound familiar?

After all of that I finally found the perfect place, got my funds in order, and then I had to sign the rights away to my first and third born sons. Never in my life (or whole life for that matter) have I had to sign and initial my name to so many documents at once.

Yay, I’ve got the house, now I get to furnish it so I can actually live there. More. Fun. Now I’m out there shopping around for furniture, kitchen appliances, bedroom furniture, bathroom things and all that other good stuff.

I’m completely exhausted from all of the above and work on top of that. I now see where y'all (I’m from Texas, cut me some slack) are coming from when shopping around for a server. I miss my lease. The whole process was simple, efficient and for all encompassing purposes much much easier. Sure, a house traditionally appreciates in value with time and a server in all likely purposes depreciates with time and there is a difference between the two, however I did want to compare these two markets in my cool little blog here.

Leasing a server is a lot like leasing a house or apartment, of which after this whole ordeal I do very much miss. In a server and house lease you are not liable for any maintenance on either. We at Softlayer will take care of all of that for you, and the landlord takes care of the house/apartment if anything breaks. If something comes up then you can get out of the lease quite easily as well as the cost is 3-5 times less than actually buying outright. Leasing also takes less time. You can get into a lease in just a couple of days on a house/apartment or a server for that matter in just a couple of hours. Buying a house takes months of preparation, searching, getting funds in order, signing and moving in. On the other hand, buying a server can take days, weeks, and months even depending on your setup, co-location facility and how fast you can get it into a live production setting. There are literally a million different variables that go into both of these equations for time, but this is just generally speaking. I’m sure there are many of you out there that can buy both in a matter of minutes and have everything setup and running just as fast.

In the long run my house will in all probability appreciate in value so it will be worth it – aside from the hassles and giving up my first and third born sons. However buying a server and co-locating it somewhere for years will likely depreciate with value as technology changes so rapidly in this day in age. Likely your old servers will not work as efficiently as new technology comes out and will eventually become obsolete. In the housing market my house will be around and habitable so to speak (depending on how rough I treat her) for decades to come. However as I pointed out earlier they are very different markets but the lease/buy theory is still the same.

To sum it all up – don’t buy, lease, you’ll thank me later.

-Michael

Categories: 
Subscribe to maintenance