Posts Tagged 'Tips And Tricks'

June 5, 2014

Sysadmin Tips and Tricks - Understanding the 'Default Deny' Server Security Principle

In the desktop world, people tend to feel good about their system’s security when they have the latest anti-virus and anti-spyware installed and keep their applications up-to-date. Those of us who compute for a living know that this is nothing close to resembling a “secure” state. But it’s the best option for non-technical people at this time.

Servers, on the other hand, exist in a more hostile environment than desktop machines, which is why keeping them secure requires skilled professionals. This means not only doing things like keeping applications patched and up-to-date, but also grasping the underlying principles of system security. Doing that allows us to make informed and skillful decisions for our unique systems—because no one knows our servers as well as we do.

One very powerful concept is “Default Deny” (as in Deny by Default), by which means that "Everything, not explicitly permitted, is forbidden." What does this mean, and why is it important?

Let’s look at a simple example using file permissions. Let’s say you installed a CGI (Common Gateway Interface) application, such as some blog software, and you’re having trouble getting it to work. You’ve decided the problem is the permissions on the configuration file. In this case, user “rasto” is the owner of the file. You try chmodding it 755 and it works like this:

-rwxr-xr-x 1 rasto rasto 216 May 27 16:11 configuration.ini

Now that it works, you’re ready to move to your next project. But there’s a possible security problem here. As you can see, you have left the configuration file Readable and Executable by Other. There is almost certainly no reason for that because CGI scripts are typically run as the owner of the file. There is potentially no reason for users of the same group (or other random users of the system) to be able to Read this configuration file. After all, some configuration files contain database passwords. If I have access to another user on this system, I could simply “cat” the configuration file and get trivial access to your data!

So the trick is to find the least permissions required to run this script. With a little work, you may discover that it runs just fine with 700:

-rwx------ 1 rasto rasto 216 May 27 16:11 configuration.ini

By taking a little extra time, you have made your system much more secure. “Default Deny” means deny everything that is not explicitly required. The beautiful thing about this policy is that you can remove vulnerabilities that you neither comprehend nor even know about. Instead of making a list of “bad” things you essentially make a list of “good” things, and allow only those things to happen. You don’t even have to realize that someone could read the file because you’ve made it a policy to always allow the least amount of access possible to all things.

Another example might be to prune your php.ini to get rid of any expanded capabilities not required by PHP scripts running on your system. If a zero-day vulnerability arises in PHP that affects one of the things you’ve disallowed, it simply won’t affect you because you’ve disabled it by default.

Another scenario might be to remove packages from your system that are not being used. If someone tries to inject some Ruby code into your system, it won’t run without Ruby present. If you’re not using it, get rid of it, and it can’t be used against you.

Note: It’s very easy to be wrong about what is not being used on your system—you can definitely break things this way—I suggest a go-slow approach, particularly in regards to built-in packages.

The important thing is the concept. I hope you can see now why a Default Deny policy is a powerful tool to help keep your system more secure.

-Lee

May 8, 2014

SoftLayer Security: Questions and Answers

When I talk to IBM Business Partners about SoftLayer, one of the most important topics of discussion is security. We ask businesses to trust SoftLayer with their business-critical data, so it’s important that SoftLayer’s physical and network security is as transparent and understandable as possible.

After going through the notes I’ve taken in many of these client meetings, I pulled out the ten most frequently asked questions about security, and I’ve compiled answers.

Q1: How is SoftLayer secured? What security measures does SoftLayer have in place to ensure my workloads are safe?

A: This “big picture” question is the most common security-related question I’ve heard. SoftLayer’s approach to security involves several distinct layers, so it’s tough to generalize every aspect in a single response. Here are some of the highlights:

  • SoftLayer’s security management is aligned with U.S. government standards based on NIST 800-53 framework, a catalog of security and privacy controls defined for U.S. federal government information systems. SoftLayer maintains SOC 2 Type II reporting compliance for every data center. SOC 2 reports are audits against controls covering security, availability, and process integrity. SoftLayer’s data centers are also monitored 24x7 for both network and on-site security.
  • Security is maintained through automation (less likely for human error) and audit controls. Server room access is limited to authorized employees only, and every location is protected against physical intrusion.
  • Customers can create a multi-layer security architecture to suit their needs. SoftLayer offers several on-demand server and network security devices, such as firewalls and gateway appliances.
  • SoftLayer integrates three distinct network topologies for each physical or virtual server and offers security solutions for systems, applications, and data as well. Each customer has one or many VLANs in each data center facility, and only users and servers the customer authorizes can access servers in those VLANs.
  • SoftLayer offers single-tenant resources, so customers have complete control and transparency into their servers.

Q2: Does SoftLayer destroy my data when I’ve de-provisioned a compute resource?

A: Yes. When a customer cancels any physical or virtual server, all data is erased using Department of Defense (DoD) 5220.22-m standards.

Q3: How does SoftLayer protect my servers against distributed denial of service (DDoS) attacks?

A: A SoftLayer Network Operations Center (NOC) team monitors network performance and security 24x7. Automated DDoS mitigation controls are in place should a DDoS attack occur.

It’s important to clarify here that the primary objective of this DDoS mitigation is to maintain performance integrity of the overall cloud infrastructure. With that in mind, SoftLayer can’t stop a customer from being attacked, but it can shield the customer (and any other customers in the same network) from the effects of the attack. If necessary, SoftLayer will remove the target from the public network for periods of time and null-routes incoming connections. Because of SoftLayer’s three-tiered network architecture, a customer would still have access to the targeted system via the private network.

Q4: How is communication segmented from other tenants using SoftLayer?

A: SoftLayer utilizes industry standard VLANs and switch access control lists (ACLs) to segment customer environments. Customers have the ability to add and manage their own VLANs, providing additional security even inside their own accounts. ACLs are configured to permit or deny any specified network packet (data) to be directed along a switch.

Q5: How is my data kept private? How can I confirm that SoftLayer can’t read my confidential data?

A: This question is common customers who deal with sensitive workloads such as HIPAA-protected documentation, employee records, case files, and so on.

SoftLayer customers are encouraged to deploy a gateway device (e.g. Vyatta appliance) on which they can configure encryption protocols. Because the gateway device is the first hop into SoftLayer’s network, it provides an encrypted tunnel to traverse the VLANs that reside on SoftLayer. When securing compute and storage resources, customers can deploy single tenant dedicated storage devices to establish isolated workloads, and they can even encrypt their hard drives from the OS level to protect data at rest. Encrypting the hard drive helps safeguard data even if SoftLayer were to replace a drive or something similar.

Q6: Does SoftLayer track and log customer environments?

A: Yes. SoftLayer audits and tracks all user activity in our customer portal. Some examples of what is tracked include:

  • User access, both failed and authenticated attempts (destination IP is shown on a report)
  • Compute resources users deploy or cancel
  • APIs for each call (who called the API, the API call and function, etc.)
  • Intrusion Protection and Detection services that observe traffic to customer hosts
  • Additionally, customers have root access to operating systems on their servers, so they can implement additional logging of their own.

Q7: Can I disable access to some of my users through the customer portal?

A: Yes. SoftLayer has very granular ACLs. User entitlements are segmented into different categories, including Support, Security, and Hardware. SoftLayer also gives customers the ability to limit access to public and private networks. Customers can even limit user access to specific bare metal or virtual server.

Q8: Does SoftLayer patch my operating system?

A: For unmanaged cloud servers, no. Once the updated operating system is deployed on a customer’s server, SoftLayer doesn’t touch it.

If you want help with that hands-on server administration, SoftLayer offers managed hosting. In a managed hosting environment, Technical Account Managers (TAMs) are assigned as focal points for customer requests and issues. TAMs help with reports and trending data that provide recommendations to mitigate potential issues (including OS patching).

Q9: Is SoftLayer suited to run HIPAA workloads?

A: Yes. SoftLayer has a number of customers running HIPAA workloads on both bare metal and single-tenant virtual servers. A Business Associate Agreement (BAA), signed by SoftLayer and the customers, clearly define the shared responsibilities for data security: SoftLayer is solely responsible for the security of the physical data center, along with the SoftLayer-provided infrastructure.

Q10: Can SoftLayer run government workloads? Does SoftLayer use the FISMA standards?

A: The Federal Information Security Management Act (FISMA) defines a framework for managing information security that must be followed for all federal information systems. Some state institutions don’t require FISMA, but look to cloud hosting companies to be aligned to the FIMSA guidelines.

Today, two SoftLayer data centers are audited to the FISMA standards – Dallas (DAL05) and Washington, D.C. (WDC01). Customers looking for the FISMA standard can deploy their workloads in those data centers. Future plans include having data centers that comply with more stringent FedRAMP requests.

For additional information, I highly recommend the on-demand SoftLayer Fundamentals session, “Keep safe – securing your SoftLayer virtual instance.” Also, check out Allan Tate’s Thoughts on Cloud blog, “HIPAA and cloud computing: What you need to know” for more on how SoftLayer handles HIPPA-related workloads.

-Darrel Haswell

Darrel Haswell is a Worldwide Channel Solutions Architect for SoftLayer, an IBM Company.

May 2, 2014

Keyboard Shortcuts in the SoftLayer Customer Portal

I’m excited to introduce a new feature in the SoftLayer customer portal: Keyboard shortcuts!

Keyboard shortcuts give you quick access to the most commonly used features by simply typing a few characters. For those who prefer never having to reach for the mouse to navigate an application, you should find these handy additions quite helpful.

After you log into the Customer Portal, type “?” (shift + forward slash) on any page, and you'll see a full list of available keyboard shortcuts:

Keyboard Shortcuts

On the Keyboard Shortcuts help page, you have the option to enable or disable the functionality based on your preference. Keyboard shortcuts are enabled by default. Disabling this feature will turn off all keyboard shortcuts except the “?” shortcut so that you can access the enable/disable feature preference in the future if you change your mind. This preference is stored in a cookie in your browser, so changing computers or deleting your cookie will re-enable the feature.

The shortcuts are grouped into three sets: Global, Tabs, and Grids.

Keyboard Shortcuts

Global Navigation

You have the ability to navigate to any page in our application by typing in the respective position number in the menu combined with dashes (-). For example, typing 1-5-2 will open Support (1) > Help (5) > Portal Tour (2).

Use the “go to” key combinations to jump to a new location from anywhere in the portal. For example, type (g) and (d) to visit the Device List. Typing (g) and (u) allows you to access the list of portal users, and (g) and (t) takes you to view tickets. If you want to add a new ticket from anywhere in the portal, type (+) and (t). It’s that simple.

Keyboard Shortcuts

Tabs

Many of the pages within the portal have tabs that appear just above the main content of the page. These tabs often allow content to be filtered, or provide access to additional features related to the page topic. Each tab can be accessed by using a simple two-keystroke combination, such as (t) then (f) to reveal the Filter tab on the page.

Keyboard Shortcuts

Grids

Whenever a page contains a grid — a tabular listing — you can now perform common operations from the keyboard. Jump quickly from page to page (first/last or next/previous) or refresh the grid contents with a single keystroke.

Keyboard Shortcuts

Please give this new feature a try for yourself! We welcome your feedback. Please let us know if you would like to have us implement any other keyboard shortcuts in the future.

-Daniel

April 23, 2014

Security: 10 Tips for Hardening a Linux Server

In light of all the complex and specialized attacks on Internet-facing servers, it’s very important to protect your cloud assets from malicious assailants whose sole purpose is to leach, alter, expose, siphon sensitive data, or even to shut you down. From someone who does a lot of Linux deployments, I like to have handy a Linux template with some extra security policies configured.

Securing your environment starts during the ordering process when you are deploying server resources. Sometimes you want to deploy a quick server without putting it behind an extra hardware firewall layer or deploying it with an APF (Advance Policy Firewall). Here are a couple of security hardening tips I have set on my Linux template to have a solid base level of security when I deploy a Linux system.

Note: The following instructions assume that you are using CentOS or Red Hat Enterprise Linux.

1. Change the Root Password
Log in to your server and change the root password if you didn’t use a SSH key to gain access to your Linux system.

  • passwd - Make sure it’s strong.
  • Don't intend on using root.

2. Create a New User
The root user is the only user created on a new Linux install. You should add a new user for your own access and use of the server.

  • useradd <username>
  • passwd <username> (Make sure this is a strong password that’s different from your root password.)

3. Change the Password Age Requirements
Change the password age so you’ll be forced to change your password in a given period of time:

  • chage –M 60 –m 7 –w 7 <username>
    • M: Minimum of days required between password changes
    • m: Maximum days the password is valid
    • w: The number of days before password will warn of expiration

4. Disable Root Login
As Lee suggested in the last blog, you should Stop Using Root!

  • When you need super-user permissions, use sudo instead of su. Sudo is more secure than using su: When a user uses sudo to execute root-level commands, all commands are tracked by default in /var/log/secure. Furthermore, users will have to authenticate themselves to run sudo commands for a short period of time.

5. Use Secure Shell (SSH)
rlogin and telnet protocols don’t use an encrypted format, just plain text. I recommend using SSH protocol for remote log in and file transfers. SSH allows you to use encryption technology while communicating with your sever. SSH is still open to many different types of attacks, though. I suggest using the following to lock SSH down a little bit more:

  • Remove the ability to SSH as root:
    1. vi /etc/ssh/sshd_config.
    2. Find #PermitRootLogin yes and change to PermitRootLogin no.
    3. Run service sshd restart.
  • Change the default SSH 22 port. You can even utilize RSA keys instead of passwords for extra protection.

6. Update Kernel and Software
Ensure your kernel and software patches are up to date. I like to make sure my Linux kernel and software are always up to date because patches are constantly being released with corrected security flaws and exploits. Remember you have access to SoftLayer’s private network for updates and patches, so you don’t have to expose your server to the public network to get updates. Run this with sudo to get updates in RedHat or CentOS: yum update.

7. Strip Your System
Clean your system of unwanted packages. I strip my system to avoid installing unnecessary software to avoid vulnerabilities. This is called “reducing the attack surface.” Packages like NFS, Samba, even the X Windows desktops (i.e., Gnome or KDE) contain vulnerabilities. Here’s how reduce the attack surface:

  • List what is installed: yum list installed
  • List the package name: yum list <package-name>
  • Remove the package: yum remove <package-name>

8. Use Security Extensions
Use a security extension such as SELinux on RHEL or CentOS when you’re able. SELinux provides a flexible Mandatory Access Control (MAC); running a MAC kernel protects the system from malicious or flawed applications that can damage or destroy the system. You’ll have to explore the official Red Hat documentation, which explains SELinux configuration. To check if SELinux is running, run sestatus.

9. Add a Welcome/Warning
Add a welcome or warning display for when users remote into your system. The message can be created using MOTD (message of the day). MOTD’s sole purpose is to display messages on console or SSH session logins. I like for my MOTDs to read “Welcome to <hostname>. All connections are being monitored and recorded.”

  • I recommend vi /etc/motd

10. Monitor Your Logs
Monitor logs whenever you can. Some example logs that you can audit:

  • System boot log: /var/log/boot.log
  • Authentication log: /var/log/secure
  • Log in records file: /var/log/utmp or /var/log/wtmp:
  • Where whole system logs or current activity are available: /var/log/message
  • Authentication logs: /var/log/auth.log
  • Kernel logs: /var/log/kern.log
  • Crond logs (cron job): /var/log/cron.log
  • Mail server logs: /var/log/maillog

You can even move these logs to a bare metal server to prevent intruders from easily modifying them.

This is just the tip of the iceberg when securing your Linux server. While not the most secure system, it gives you breathing room if you have to deploy quick servers for short duration tests, and so on. You can build more security into your server later for longer, more permanent-type servers.

- Darrel Haswell

Darrel Haswell is an advisory SoftLayer Business Partner Solution Architect.

Categories: 
April 23, 2014

Sysadmin Tips and Tricks - Stop Using Root!

A common mistake newer Linux system administrators make is the overuse of root. It seems so easy! Everything is so much simpler! But in the end, it’s not—and it’s only a matter of time before you wish you had not been so free and easy with your super-user, use. Let me try to convince you.

Let’s start with a little history. The antecedents of Linux go all the way back to the early 1970s, when computers cost tens of thousands of dollars (at least). With that kind of expense, you as a user would hardly have a computer sitting on your desk (not to mention they were at least refrigerator-sized), and you would also not have the use of it dedicated to your needs. What was obviously needed was an operating system that would allow multiple users to use the machine at once, via terminals, in order to make the most use of the computing resources available.

If you think about it, it’s clear that the operating system had to be very good at keeping users from being able to stomp on each other’s files and processes. So the early UNIX™ variants were multi-user systems from the get-go. In the ensuing forty years, these systems have only gotten better at keeping the various users and processes from harming each other. And this is the technology that you’re paying for when you use Linux or other modern variants.

Now, you may think, “That doesn’t apply to me—I’m the only user on my server!” But are you, really?

You probably run Apache, which is generally run as the user httpd or apache. Why not root? Because if you run Apache as root, then anyone on the outside who manages to get Apache to execute arbitrary code, would then have that code running as root! Next thing you know, they can execute "rm –rf /," or worse, invade your system altogether and steal proprietary information. By running as a non-root user, even if the attacker gets total access to that user, they are limited to what that user can touch. Thus, user httpd is compromised, but not the entire server.

The same thing is true for mail servers, FTP servers, and so on. They all rely on the Linux permissions system in order to give the programs access to as little as possible—ideally, only exactly what they need to do their jobs.

So, think of yourself as another process on the system. When you log in as your regular user, you are limited in what you can do. But this is not intended to harm you or irritate you—indeed; the system is designed to keep you from accidentally doing damage to your server.

For example, consider if you wanted to completely remove a directory called ‘home’ within your home directory. Note the ever so slight difference between the first command:

rm –R home

And the second command:

rm –R /home

The first command removes a directory called ‘home’ from wherever you happen to be sitting on the file system. The second removes all users’ home directories from the system. One little slash makes all the difference in the world. This is probably why it has been said that Linux gives you enough rope to hang yourself with. Executing the second command as root looks like this:

server:# rm –R /home 
server.com#

And it’s just gone! Whereas if you accidentally put that slash in there while logged in as your user, you would get:

server:# rm –R /home 
server:# rm: cannot remove `home’: Permission denied

This will annoy you, until you realize that if you’d done it as root you would have wiped out all your customers home directories.

In short, just like the processes that run on your machine, you would be well served to use only the permissions you need. This is why many Linux distributions today encourage the use of sudo—you don’t even become root, but just execute things as root when needed. It’s a good policy, and makes the best use of four decades of expertise that have gone into the system you are using.

- Lee

P.S. This is also why you pretty much never want to chmod 777 anything!

Subscribe to tips-and-tricks