Posts Tagged ‘technical’

In our last “iptables Tips and Tricks” installment, we talked about Advanced Policy Firewall (APF) configuration, so it should come as no surprise that in this installment, we’re turning our attention to ConfigServer Security & Firewall (CSF). Before we get started, you should probably run through the list of warnings I include at the top of the APF blog post and make sure you have your Band-Aid ready in case you need it.

To get the ball rolling, we need to download CSF and install it on our server. In this post, we’re working with a CentOS 6.0 32-bit server, so our (root) terminal commands would look like this to download and install CSF:

$ wget http://www.configserver.com/free/csf.tgz #Download CSF using wget.
$ tar zxvf csf.tgz #Unpack it.
$ yum install perl-libwww-perl #Make sure perl modules are installed ...
$ yum install perl-Time-HiRes  #Otherwise it will generate an error.
$ cd csf
$ ./install.sh #Install CSF.
 
#MAKE SURE YOU HAVE YOUR BAND-AID READY
 
$ /etc/init.d/csf start #Start CSF. (Note: You can also use '$ service csf start')

Once you start CSF, you can see a list of the default rules that load at startup. CSF defaults to a DROP policy:

$ iptables -nL | grep policy
Chain INPUT (policy DROP)
Chain FORWARD (policy DROP)
Chain OUTPUT (policy DROP)

Don’t ever run “iptables -F” unless you want to lock yourself out. In fact, you might want to add “This server is running CSF – do not run ‘iptables -F’” to your /etc/motd, just as a reminder/warning to others.

CSF loads on startup by default. This means that if you get locked out, a simple reboot probably won’t fix the problem. Runlevels 2, 3, 4, and 5 are all on:

$ chkconfig --list | grep csf
csf             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Some features of CSF will not work unless you have certain iptables modules installed. I believe they are installed by default in CentOS, but if you custom-built your iptables, they might not all be installed. Run this script to see if all modules are installed:

$ /etc/csf/csftest.pl
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
 
RESULT: csf should function on this server

As I mentioned, this is the default iptables installation on a minimal CentOS 6.0 image, so chances are good that these modules are already installed on your system. It never hurts to check, though.

The CSF Configuration File

The primary CSF configuration is stored in the well-documented /etc/csf/csf.conf file. CSF is extremely configurable, so there are a lot of options to read over. Let’s take a look over some of the more important features:

Learn about CSF configuration options, Allow and Deny Lists, and the CSF Command Line Tool »

Too often, new customers can get overwhelmed by a small administrative task on a Linux server. One of the more common questions I see in technical support is when a drive partition runs out of space. The website appears offline, and on of my coworkers advises you to just free-up some space. “Just?! Where can I find files that are deletable without affecting my website?”

Don’t worry … it’s really quit simple. If you can use FTP (File Transfer Protocol), you can handle this bit of server management. Depending on the exact problem, we might instruct you to free up space by removing files in one of the following directories:

• /var/log
• /usr/local/cpanel
• /usr/local/apache/logs
• /usr/local/apache/domlogs

The reason these directories are usually overlooked is because they are not accessible by normal FTP users — users who only upload website content. When you upload website content to the server via FTP, the FTP user is limited to the directory structure for that website. Directories starting with “/var” and “/usr” cannot be accessed by these non-root users (The “root” user can access anything). And while root is a powerful user, for the sake of security, it is not normally allowed to log in over FTP because FTP is not secure … That’s where SFTP (Secure File Transfer Protocol) comes in.

Most FTP clients support SFTP, so you don’t have to learn a new environment to securely access any file on the server. Every FTP client is different, but I’ll illustrate with FileZilla because it’s free and available on Mac, Windows and Linux. If you don’t already have an FTP client, I highly recommend FileZilla. Because there are a few ways to use FileZilla to get an SFTP connection, I can share different options for you to try:

Quick Connect

The Quick Connect bar is the quickest way to connect to your server. Start FileZilla and look immediately under the toolbar for the Quick Connect bar:

Enter the hostname (IP address or domain name), “root” in the Username field, the root password in the Password field, and “22″ in the port field. Remember, port 22 is for SFTP, the same as SSH. Click the Quickconnect button to connect.

Using the Site Manager

The Site Manager lets you save your login details. Start FileZilla and you’ll see the following:

To open the Site Manager, click the left-most icon in tool bar or go to File >> Site Manager in the menu.

Enter an IP address or domain name for your server in the Host field, and select “SFTP” as your protocol. You’ll enter the root user’s login information, and you’re ready to connect by clicking the “Connect” button or you can click the “OK” button to save and close the dialog box.

If you just saved your settings and the Site Manager is not open, click the Site Manager icon again. From there, you can select the site under the “Select Entry” box, and you just have to click “Connect” to initiate the SFTP connection with your saved settings.

If you see a pop-up that warns of an “Unknown host key,” clicking the “Always trust this host, add this key to the cache” option will prevent this interruption from showing in the future. Once you click “OK” to complete the connection, your FileZilla screen should look like this:

Notice the “Remote site” section on the middle right of the FileZilla screen:

This area in FileZilla is the directory and file listing of the server. Navigate the server’s file structure here, and click “/” to access the top of the folder structure. You should see the “/usr” and “/var” directories, and you can explore the filesystem to delete the files technical support recommended to create space!

Message Log

If you have a problem connecting to your server by FTP or SFTP, the open area below the Quickconnect bar is the Message Log. If you can copy and paste this text into a ticket, you’ll help technical support troubleshoot your connection problems. Below is an example log of a successful FTP session:

Status: Connecting to server.example.com...
Response:   fzSftp started
Command:    open "root@server.example.com" 22
Command:    Trust new Hostkey: Once
Command:    Pass: **********
Status: Connected to server.example.com
Status: Retrieving directory listing...
Command:    pwd
Response:   Current directory is: "/root"
Command:    ls
Status: Listing directory /root
Status: Calculating timezone offset of server...
Command:    mtime ".lesshst"
Response:   1326387703
Status: Timezone offsets: Server: -21600 seconds. Local: -21600 seconds. Difference: 0 seconds.
Status: Directory listing successful

And here’s an example of a failed connection:

Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection timed out
Error:  Could not connect to server
Status: Waiting to retry...
Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection attempt interrupted by user

If you have any questions, leave them in a comment below. Enjoy your new-found SFTP powers!

-Lyndell

As a bit of an introduction, I began my career as a GSP and hosted LAMP sites with WHM for SMBs … NBD. If you’re not fluent in “Tech Geek Acronym,” that sentence may as well be written in Greek. If I were to de-acronym it, I’d say, “I began my career as a Game Service Provider” and hosted Linux, Apache, MySQL and PHP sites with Web Host Managed for Small- and Medium-sized Businesses … no big deal.” For many, the humble acronym is a cornerstone of what it means to be a true techie. Stringing together dozens of three-letter abbreviations (TLAs) to compose semi-coherent sentences would seem to demonstrate your mastery of technology … The problem is that if the reader of that sentence doesn’t have the context you have, it’s not very easy to easily get up to speed.

Every profession has their collection of acronyms. The little expressions serve as a verbal and written short hand for people who toil daily with the topics of their trade. I’m proud to confess that I’ve been using these minute medleys of letters for over twelve years. Given that I work on the Internet, I’ve been exposed to hundreds of acronyms in the fields of technology, business and management, and in my experience, I’ve had to break through several acronym “barriers” to get in the know. Because I happen to interact with customers every day as the manager of SoftLayer’s technical support department, I’ve encountered a few “Can you tell means?” responses, so I thought I’d write a quick blog post to clarify some of the common acronyms you may see in the SoftLayer vernacular.

Within support we have our CSTs (customer support technicians) and CSAs (customer support admins) who, with the help of SBTs (server build technicians), manage our massive fleet of servers. SBTs are the hands and eyes of our data centers, working closely with the hardware to ensure your server is online and operating in peak condition. The CSTs and CSAs are focused on the software and services that power your websites and applications.

Beyond employee title acronyms, you’ll probably see a collection of terms that describe the products and services that we manage. In support, we receive questions about accessing servers or CCIs (cloud computing instances) using KVM (Keyboard, Video and Mouse) or IPMI (Intelligent Platform Management Interface) through our VPN (Virtual Private Network). Once connected to our back-end network through a SSL (Secure Socket Layer), PPTP (Point-to-Point Tunnel Protocol) or IPSEC (Internet Protocol Security) VPN, you have access to services such as DNS (Domain Name Service), NAS (Network Attached Storage) or iSCSI (Internet Small Computer System Interface). Finally, while discussing our network, I often refer to http://www.softlayer.com/diagrams/pod-network-diagram/dal05 to show the difference between a VER (VPN Edge Router) and a BCS (Back-end Customer Switch).

If you run across an acronym you don’t understand in a ticket, please let us know so we can share its full meaning … By using these shortened terms, our team can provider faster service (and you can read their responses quicker). I know that seeing all the bold TLAs above may seem a little off-putting initially, but as you have a chance to read them in the context of some of the other acronyms you already know, I hope you have an “Aha!” moment … Like finding the Rosetta Stone or the Code of Hammurabi. Given the quick glance at the terms above, if you want to learn more about one of the TLAs in particular, leave a comment below, and we’ll respond in another comment with details.

CBNO

-Chris

Over the past 10 years, I’ve run or helped run all sizes of web sites and internet applications. I’ve seen everything from single-page brochure web sites to horizontally scaled interactive portals. And what I’ve learned is that it is all about the end-user experience.

I’m not a graphics specialist or a GUI designer. I just don’t have that in my DNA. I focus more on the technical side of things working on better ways to deliver content to the user. And in the purely technical area, the best thing to do to improve the user experience is to improve the delivery speed to the user.

There are a lot of tools out there that can be used to speed up delivery. CDN, for example, is an awesome way to get static content to an end user and is very scalable. But what about scaling out the application itself?

Traditionally, a simple Layer-4 Load Balancer has been a staple component of scalable applications. This type of Load Balancing can provide capacity during traffic peaks as well as increase availability. The application runs on several servers and the load balancer uses some simple methods (least connections, round robin, etc) to distribute the load. For a lot of applications this is sufficient to get content reliably and quickly to the end user. SoftLayer offers a relatively inexpensive load-balancing service for our customers that can provide this functionality.

There is another, more sophisticated, tool that can be used to manage internet application traffic. That is the “Application Delivery Controller” (obligatory Wikipedia link: http://en.wikipedia.org/wiki/Application_Delivery_Controller) or “Load Balancer on Steroids”. This class of traffic manager can act in Layer-7, the data layer. These devices can make decisions based on the actual content of the data packets, not just the source and destination.

And an ADC can do more than load balance. It can act as a Web Application Firewall to protect your data. It can speed up your application using SSL Offloading, Content Caching, TCP Optimization, and more. This type of device is very smart and very configurable and will help in the delivering the application to the end user.

At SoftLayer we have seen our customers achieve a lot of success with our Layer-4 Load Balancer product. But we are always looking for other tools to help our customers. We always have admired the advanced functionality in the appliance-based Application Delivery Controllers on the market. Finding a way to get this enterprise-grade technology to our customers in an affordable manner was problematic. When Citrix announced that they were going to create a version of their NetScaler product that didn’t require an appliance we were thrilled. With the announcement of the NetScaler VPX we finally thought we had found the right product that we could use to affordably provision this advanced technology on-demand to our customers.

SoftLayer is VERY excited to partner with Citrix to provide the NetScaler VPX Application Delivery Controller to our customers. Our customers can order a NetScaler VPX, and in a matter of minutes be managing the delivery of their online applications using one of the most sophisticated tools on the market. Citrix does a better job of promoting the product than I do, so here is the link to their site: http://citrix.com/English/ps2/products/product.asp?contentID=21679&ntref=hp_nav_US.

Remember, it’s all about the experience of the user at the other end of the wire. Find the right tools to manage that experience and you are most of the way there. Oh yeah, and find a good graphics designer too. That helps. So does good content.

-@nday91

SoftLayer Technical Support technicians train continuously for the challenges that are inherent in supporting the vast array of products that SoftLayer offers. Besides training individually in their time away from the NOC, technicians are always talking about issues they have seen, and the resolutions they implemented.

Knowledge gained by one tech in tackling and conquering a specific issue is shared with all for the betterment of the team. Like a gladiator in the bowels of the Roman Coliseum of old preparing for his fight, the SoftLayer Support technician must be ready to do battle. Disciplined cross-training is the order of the day; mental and physical preparation is key. A technician must enter the halls of a SoftLayer datacenter ready to conquer whatever comes through the gates! It is truly a battle worthy of the Coliseum.

You might ask how a day in the SoftLayer NOC resolving technical issues compares to a battle fought in the Roman Coliseum. Well, if you measure a “battle” by the excitement and tension in the air … the blood, the sweat, and yes, at times, the tears, the pain of defeat, and the celebration of victory, then SoftLayer Technical Support technicians are definitely involved in a true battle worthy of the Roman Coliseum on a daily basis.

Picture, if you will, a well-trained, focused individual walking into the Dallas Infomart with his security badge in hand. He is not there to pass the time or participate in some mind-numbing repetitive task. He is there to do battle with a beast named Technology. With a strategic plan in mind, he enters the elevator preparing himself mentally for what surely awaits upon entering his cubicle. As he opens the door to the NOC, he is greeted by his fellow “warriors”. Some are weary from battle, yet have a sense of satisfaction about them as a Roman Soldier of old looking across the battlefield at his conquered foe.

The stories of a multitude of battles won, and maybe even a few lost, are recounted. The technical warrior packs some sustenance from the chow line (the loaded NOC break room refrigerator), and settles into his chariot he likes to call a cubicle, pulling out his weapons, a keyboard and mouse, and bringing up the battlefield onscreen. He begins with the speed of a cheetah typing more and more furiously as each ticket darts to and fro trying to elude him. The warrior is undaunted. He will not be defeated today. Yes, he may need to look to his comrades in arms for assistance in flanking the enemy, but in the end, as a team of highly trained warriors, they will prevail.

This day will not be without its casualties, but the warrior must always repeat to himself, “I will not let our customer’s down. The enemy (technical issues) will not prevail…not on my watch.”

As did the citizens of Rome, I take great pride in our warriors and the superior way in which they continue to win battles for the glory of our customers. The inspiration for this writing came from a recent victory in which a warrior named Stefanus (Steve) stood in victory after wrestling with a beast of an issue, which he finally destroyed while the customer rejoiced and his wealth increased. All the warriors: Krishenus, Jamesus, and Samuel gathered around Stefanus to congratulate him on his victory. Of course, they all knew that the ultimate victory was enjoyed by the SoftLayer customer.

-David

Recently on one of our technical forums I contributed to a discussion about the Windows operating system. One of our director’s saw the post and thought it might be of interest to readers of the InnerLayer as well. The post focused on the pros and cons of Windows 2008 from the viewpoint of a systems / driver engineer (aka me). If you have no technical background, or interest in Microsoft operating system offerings, what follows probably will not be of interest to you—just the same, here is my two cents.

Microsoft is no different than any other developer when it comes to writing software–they get better with each iteration. There is not a person out there who would argue that the world of home computers would have been better off if none of us ever progressed beyond MS-DOS 1.0. Not to say there is anything wrong with MS-DOS. I love it. And still use it occasionally doing embedded work. But my point is that while there have certainly been some false starts along the way (can you say BOB), Microsoft’s operating systems generally get better with each release.

So why not go out and update everything the day the latest and greatest OS hits the shelves? Because as most of you know, there are bugs that have to get worked out. To add to that, the more complex the OS gets, the more bugs there are and the more time it takes to shake those bugs out. Windows Server 2008 is no different. In my experience there are still a number of troublesome issues with W2K8 that need to be addressed. Just to name a few:

• UAC (user access control) – these are the security features that give us so much headache. I’m not saying we don’t need the added security. I’m just saying this is a new arena for MS and they still have a lot to learn. After clicking YES, I REALLY REALLY REALLY WANT TO INSTALL SAID APPLICATION for the 40th time in a day, most administrators will opt to disable UAC, thereby thwarting the added security benefits entirely. If I were running this team at MS I’d require all my developers to take a good hard look at LINUX.
• UMD (user mode drivers) – the idea of running a device driver, or a portion of a device driver, in the restricted and therefore safe user memory of the kernel is a great idea in terms of improving OS reliability. I’ve seen numbers suggesting that as many as 90% of hard OS failures are caused by faulty third-party drivers mucking around in kernel mode. However implementing user mode drivers adds some new complexities if hardware manufacturers don’t want to take a performance hit and from my experience not all hardware vendors are up to speed yet.
• Driver Verification – this to me is the most troublesome and annoying issue right now with the 64-bit only version of W2K8. Only kernel mode software that has been certified in the MS lab is allowed to execute on a production boot of the OS. Period. Since I am writing this on the SoftLayer blog, I am assuming most of you are not selecting hardware and drivers to run on your boxes. We are handling that for you. But let me tell you it’s a pain in the butt to only run third party drivers that have been through the MS quality lab. Besides not being able to run drivers we have developed in house it is impossible for us to apply a patch from even the largest of hardware vendors without waiting on that patch to get submitted to MS and then cleared for the OS. A good example was a problem we ran into with an Intel Enet driver. Here at SoftLayer we found a bug in the driver and after a lot of back and forth with Intel’s Engineers we had a fix in hand. But that fix could not be applied to the W2K8 64-bit boxes until weeks later when the fix finally made it from Intel to MS and back to Intel and us again. Very frustrating.

Okay, so now that you see some of the reasons NOT to use MS Windows Server 2008 what are some of the reasons it’s at least worth taking a look at? Well here are just a few that I know of from some of the work I have done keeping up to speed with the latest driver model.

• Improved Memory Management – W2K8 issues fewer and larger disk I/O’s than its 2003 predecessor. This applies to standard disk fetching, but also paging and even read-aheads. On Windows 2003 it is not uncommon for disk writes to happen in blocks < 64KB and certainly never more than 64KB (this was a limitation that had been in place since Windows NT). On W2K8 the memory manager frequently handles writes as large as a full MB. Furthermore, writing related data in large blocks (especially during paging) significantly reduces fragmentation vastly improving read back times as well.
• Improved Data Reliability – Everyone knows how painful disk corruption can be. And everyone knows taking a server offline on a regular basis to run chkdsk and repair disk corruption is slow. One of the ideal improvements in terms of administering a websever is that W2K8 employs a technology called NTFS self-healing. This new feature built into the file system detects disk corruption on the fly and quarantines that sector, allowing system worker-threads to execute chkdsk like repairs on the corrupted area without taking the rest of the volume offline.
• Scalability – The W2K8 kernel introduces a number of streamlining factors that greatly enhance system wide performance. A minor but significant change to the operating system’s low level timer code, combined with new I/O completion handling, and more efficient thread pool, offer marked improvement on load-heavy server applications. I have read documentation supporting claims that the minimization in CPU synchronization alone results directly in a 30% gain on the number of concurrent Windows 2008 users over 2003. That’s not to say once you throw in all the added security and take the user mode driver hit you won’t be looking at 2003 speeds. I’m just pointing out hard kernel-level improvements that can be directly quantified by multiplying your resources against the number of saved CPU cycles.

Alright, no need to beat a dead horse. My hope was if nothing else to muddy the waters a bit. The majority of posts I read on our internal forums seemed to recommend avoiding W2K8 like the plague. I’m only suggesting while it is certainly not perfect, there are some benefits to at least taking it for a test drive. Besides, with SoftLayer’s handy dandy portal driven OS deployment, in the amount of time it took you to read all my rambling you might have already installed Windows Server 2008 and tried it out for yourself. Okay, maybe that’s a bit of an exaggeration. But still…you get the idea!

-William

I am sure everyone can remember the overwhelming feeling of getting their first server. The SoftLayer family recognizes that this can be a very discouraging time, thus we created a group of technical wizards who have the very specific goal of assisting newer clients who are in the process of learning how to use the tools provided by SoftLayer.

Have you ever wished you could copy data to your server without bandwidth concerns, or wondered how to reboot a server that is no longer responding? Ever wonder what the RescueLayer is and who it rescues? What is NAS, iSCSI, a firewall, how do I load balance? What is the CDNLayer and how can it help me? These are just a few of things we can help you better understand. This will allow you the leisure to ponder the ways to make your server more profitable. No more wasting time creating support tickets when you can do it yourself fast and easy. We can show you how. Here is a little more about the team:

What do we do for fun – Our hobbies include Aviation, Camping, Music, and Automotive Engineering. Some like the outdoors, golf, Karate, poker, etc and some spend their off hours with the family and kids. Smarts – The technical abilities in the group range from Automation to Xen and everything in between including disaster recovery, portal and backend database design, server administration, load balancing, hosting in general (the what to do’s and what not to do’s), ASP and PHP development, developing online collaborations sites, MMO gaming, and LAMP stacks to Windows. We try to be as well rounded as we can. The group has years and years of experience in the hosting, internet, ISP, and system administration arena and we are here and ready to help.

We are STAT! The SoftLayer Technical Assurance Team, pleasure to meet you. How can we help?

-John