Posts Tagged 'System'

February 15, 2013

Cedexis: SoftLayer "Master Model Builder"

Think of the many components of our cloud infrastrucutre as analogous to LEGO bricks. If our overarching vision is to help customers "Build the Future," then our products are "building blocks" that can be purposed and repurposed to create scalable, high-performance architecture. Like LEGO bricks, each of our components is compatible with every other component in our catalog, so our customers are essentially showing off their Master Model Builder skills as they incorporate unique combinations of infrastructure and API functionality into their own product offerings. Cedexis has proven to be one of those SoftLayer "Master Model Builders."

As you might remember from their Technology Partner Marketplace feature, Cedexis offers a content and application delivery system that helps users balance traffic based on availability, performance and cost. They've recently posted a blog about how they integrated the SoftLayer API into their system to detect an unresponsive server (disabled network interface), divert traffic at the DNS routing level and return it as soon as the server became available again (re-enabled the network interface) ... all through the automation of their Openmix service:

They've taken the building blocks of SoftLayer infrastructure and API connectivity to create a feature-rich platform that improves the uptime and performance for sites and applications using Openmix. Beyond the traffic shaping around unreachable servers, Cedexis also incorporated the ability to move traffic between servers based on the amount of bandwidth you have remaining in a given month or based on the response times it sees between servers in different data centers. You can even make load balancing decisions based on SoftLayer's server management data with Fusion — one of their newest products.

The tools and access Cedexis uses to power these Openmix features are available to all of our customers via the SoftLayer API, and if you've ever wondered how to combine our blocks into your environment in unique, dynamic and useful ways, Cedexis gives a perfect example. In the Product Development group, we love to see these kinds of implementations, so if you're using SoftLayer in an innovative way, don't keep it a secret!

-Bryce

June 25, 2012

Tips from the Abuse Department: Part 2 - Responding to Abuse Reports

If you're a SoftLayer customer, you don't want to hear from the Abuse department. We know that. The unfortunate reality when it comes to hosting a server is that compromises can happen, mistakes can be made, and even the most scrupulous reseller can fall victim to a fraudulent sign-up or sly spammer. If someone reports abusive behavior originating from one of your servers on our network, it's important to be able to communicate effectively with the Abuse department and build a healthy working relationship.

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We'll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we'll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the "Tips from the Abuse Department" series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You'd be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren't in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don't see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you're taking and the time you need to resolve the matter, we're able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you're performing an investigation of how that activity was started. We'll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise ... But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, "Fixed," is not sufficient to for the Abuse department to consider the matter resolved, but we also don't need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don't provide those details in your first response, you can bet we'll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

  1. How/why the issue occurred
  2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you're digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I'm ending on a similar note to my last blog post because it's just that important! We understand getting an abuse ticket is a hassle, but please remember that we're doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

June 18, 2012

Tips from the Abuse Department: Part 1 - Reporting Abuse

SoftLayer has a dedicated team working around the clock to address complaints of abuse on our network. We receive these complaints via feedback loops from other providers, spam blacklisting services such as Spamcop and Spamhaus, various industry contacts and mailing lists. Some of the most valuable complaints we receive are from our users, though. We appreciate people taking the time to let us know about problems on our network, and we find these complaints particularly valuable as they are non-automated and direct from the source.

It stands to reason that the more efficient people are at reporting abuse, the more efficient we can be at shutting down the activity, so I've compiled some tips and resources to make this process easier. Enjoy!

Review our Legal Page

Not only does this page contain our contact details, there's a wealth of information on our policies including what we consider abuse and how we handle reported issues. For starters, you may want to review our AUP (Acceptable Use Policy) to get a feel for our stance on abuse and how we mitigate it.

Follow Proper Guidelines

In addition to our own policies, there are legal aspects we must consider. For example, a claim of copyright infringement must be submitted in the form of a properly formatted DMCA, pursuant to the Digital Millennium Copyright Act. Our legal page contains crucial information on what is required to make a copyright claim, as well as information on how to submit a subpoena or court order. We take abuse very seriously, but we must adhere to the law as well as our privacy policy in order to protect our customers' businesses and our company from liability.

Include Evidence

Evidence can take the form of any number of things. A few common examples:

  • A copy of the alleged spam message with full headers intact.
  • A snippet from your log file showing malicious activity.
  • The full URL of a phishing page.

Without evidence that clearly ties abusive activity to a server on our network, we are unable to relay a complaint to our customer. Keep in mind that the complaint must be in a format that allows us to verify it and pass it along, which typically means an email or hard copy. While our website does have contact numbers and addresses, email is your best bet for most types of complaints.

Use Keywords

We use a mail client specifically developed for abuse desks, and it is configured with a host of rules used for filtering and prioritization. Descriptive subject lines with keywords indicating the issue type are very useful. Including the words "Spam," "Phishing" or "Copyright" in your subject line helps make sure your email is sent to the correct queue and, if applicable, receives expedited processing. Including the domain name and IP address in the body of the email is also helpful.

Follow Up

We work hard to investigate and resolve all complaints received however, due to volume, we typically do not respond to complaining parties. That said, we often rely on user complaints to determine if an issue has resumed or is ongoing so feel free to send a new complaint if activity persists.

Be Respectful

The only portion of your complaint we are likely to relay to our customer is the evidence itself along with any useful notes, which means that paragraph of profanity is read only by hardworking SoftLayer employees. We understand the frustration of being on the receiving end of spam or a DDOS, but please be professional and try to understand our position. We are on your side!

Hopefully you've found some of this information useful. When in doubt, submit your complaint to abuse@softlayer.com and we can offer further guidance. Stay tuned for Part 2, where I'll offer suggestions for SoftLayer customers about how to facilitate better communication with our Abuse department to avoid service interruption if an abuse complaint is filed against you.

-Jennifer

August 29, 2011

UNIX Sysadmin Boot Camp: Your Logs and You

We're a few exercises into UNIX Sysadmin Boot Camp, and if you're keeping up, you've learned about SSH and bash. In those sessions, our focus was to tell the server what we wanted it to do. In this session, we're going to look at the logs of what the server has done.

Logs are like an overbearing mother who sneakily follows her teenage son around and writes down the addresses of each house he visits. When he realizes he lost a really important piece of baseball history at one of those houses, he'll be glad he has that list so he can go desperately search for the soon-to-be-noticed missing bat. Ahem.

MAKE BEST FRIENDS WITH THIS DIRECTORY: /var/log/

When something goes wrong – when there's hitch in the flux capacitor or too many gigawatts in the main reactor – your logs will be there to let you know what's going on, and you can pinpoint the error with educated vengeance. So treat your logs with respect.

One of the best places to start harnessing this logged goodness is /var/log/messages. This log file reports all general errors with network and media, among other things. As you add to and learn your server's command line environment, you'll see specific logs for applications as well, so it's a very good idea to keep a keen eye on these. They just might save your life ... or server.

Some of the most commonly used logs (may vary with different Linux distributions):

  • /var/log/message – General message- and system-related info
  • /var/log/cron.log – Cron job logs
  • /var/log/maillog – Mail server logs
  • /var/log/kern.log – Kernel logs
  • /var/log/httpd/ – Apache access and error logs
  • /var/log/boot.log – System boot logs
  • /var/log/mysqld.log – MySQL database server logs
  • /var/log/secure – SSH authentication logs
  • /var/log/auth.log – Authentication logs
  • /var/log/qmail/ – Qmail log directory (more files inside this directory)
  • /var/log/utmp or /var/log/wtmp – Login records file
  • /var/log/yum.log – Yum log files

There are plenty more in-depth logs – particularly involving raw system components – and others that act similarly to logs but are a bit more active like tcpdumps. Those are a little more advanced to interpret, so I'll save them for another guide and another day.

At this point in our UNIX workout series, you're familiar with the command line, you know the basics of how to tell your server what to do and you just learned how to let the server tell you what it's done. There's still a bit of work to be done before you can call yourself a UNIX ninja, but you're well on your way. In our next installment, we're going to take a step back and talk about p455w0rd5.

Keep learning.

-Ryan

February 21, 2011

Building a Data Center | Part 1: Follow the Flow

The electrical distribution system in a data center is an important concept that many IT professionals overlook. Understanding the basics of your electrical distribution system can save downtime and aid in troubleshooting power problems in your cabinets. It's easy to understand if you follow the flow.

As with many introductory lessons in electricity, I will use the analogy of a flowing river to help describe the flow of electricity in a data center. The river is akin to wires, the amount of water is the voltage and the speed the water moves is the current flow also known as amps. So, when looking at an electrical system, think about a flowing river and the paths that it must take to get to and from its source to the ocean.

External Power Sources
The preferred source of electrical power is delivered to a data center by the local utility company. Once that utility power enters the building, its first stop is usually going to be the ATS or Automatic Transfer Switch. This electro-mechanical device is fed power from two or more sources – a Primary and an Emergency. While the primary source is available, it sits happily and flows power to a series of distribution breakers, often called "switch gear." These large breakers are designed to carry hundreds or thousands of amps and pass that power to your uninterruptible power supply (UPS) units and other facility infrastructure: lighting, HVAC, fire life safety systems, etc.

If the Primary source becomes unavailable, the ATS triggers the emergency source. In our data center example, that means our on-site generators start up. It typically takes between 9 to 12 seconds for the generators to come up to speed to allow for full power generation. Once the ATS sees that the generators have started and are ready to supply power, it will switch the load from the primary source to the emergency source. This is called an open transition because the load is removed from the primary source during the switch to the emergency source.

UPS Units
Once the power leaves the ATS and switch gear, it is no longer important to know whether you are connected to the primary or emergency sources. The next step in the power flow is the UPS. Like a dam, the UPS system takes an untamed river and transforms it into something safe and usable: An uninterruptable source of power to your server cabinet.

This is achieved by a bank of batteries sized to support the IT load. The batteries are connected in-line with the supply and load, so while the ATS senses a utility outage and starts the emergency generators, the IT load is still supplied power. A typical UPS battery system is designed to support the IT load for a maximum of 10 minutes.

Another benefit of the UPS system is the ability to clean the incoming utility power. Normal utility power voltages vary wildly depending on what other loads the service is supplying. These voltage fluctuations are detrimental to power supplies in servers and can shorten their life spans or worse: destroy them. This is why most home computers have a surge suppressor to prevent power spikes from damaging your equipment. UPS units clean electrical power by converting utility power from AC to DC and back to AC again:

UPS

Power Distribution Units
After protecting and cleaning the power, the UPS power will flow to a group of power distribution units (PDUs). At this point, the voltage will normally be 480vac which is too high for most IT equipment. The PDU or a separate transformer has to to convert the 480 volts to a more usable voltage like 120vac or 277vac. Once the voltage is converted, the power is then distributed to electrical outlets via a common electrical breaker.

PDU technology has advanced, like all data center equipment, from simple breaker panels to complex devices capable of measuring IT loads, load balancing, alarm and fault monitoring and even automatic switching between two power sources instantly during an outage.

Power Strip
The final piece of equipment in the data center electrical system before your server is a power strip. Power strips are often mistakenly referred to as PDUs. The power strip is mounted in a cabinet and contains multiple electrical outlets, not electrical breakers. You plug the server power cord into the power strip, not the PDU. And from here, the flow of electricity finally reaches the sea of servers.

Here's a basic for a data center electrical distribution system:

Simplified Data Center Power Architecture

Our data centers are complex, and the entire building infrastructure is critical to its continuous operation. The electrical distribution system is at the heart of any critical facility, and it's vital that everyone working in and around critical sites knows at least the basics of your electrical distribution system.

In Part 2 of our "Building a Data Center" series, we'll cover how we keep the facility cool.

-John

November 13, 2008

Size Isn't Everything

A couple days ago, I took my daughter to her favorite store. We picked up a fair amount and on the way to the car she asked a simple question, or so I thought. “Why did they only fill these bags half way”. Confused I looked at the bags and realized she was holding a bag which had a large stuffed bear in it and was looking at a bag less than half full of canned food.

Being the person I am, rather than attempt to explain this to her I wanted to let her try and figure it out for herself so she would understand it better. When we got home, I filled the rest of the bag with cans and had her try and pick it up, as I expected the bag broke in her hands. I explained to her that the cans were much heavier then the bag. She still doesn’t quite understand the concept that the bag has 2 limits, size and weight but she is starting to understand this concept.

I thought about this story this morning when I started working on a project of determining how many containers a Virtuozzo server could handle based on its system requirements. Just like the bag, a Virtuozzo system has multiple limitations that need to be observed, the size of the containers as well as their “weight”. In this situation “weight” would be the drain on overall system resources. When attempting to determine how many containers a system can handle, you need to take into account not only how many will fit size wise, but also how much of the overall system resources each container will require.

It turns out this question is much easier to ask then to answer. You can take a small server such as a dual core with 4GB of RAM and put 20 or even 30 containers onto the server and have it run flawlessly when those containers are small and do not require much in the way of system recourses. At the same time however I can take a quad proc quad core with 64GB of RAM and grind it to a halt with 1 or 2 containers.

At the end of the day, I have found that you can make just about anything work, but before you attempt to determine what hardware you will need to run a Virtuozzo server, it’s a good idea to have an estimate of what you expect the containers to be doing. What could be worse than spending hours configuring a server and getting it online only to watch it grind to a halt because there are just too many containers completely saturating your system resources?

-Mathew

Categories: 
Subscribe to system