March 5, 2008

Outsource IT: Part III

Third in a series of three! In other words you won't have to read this stuff anymore after this one. I will get back to the fun ones. I might try to make this one fun along the way. So I left off on the last one discussing some of the financial reasons and technical reasons to outsource your servers. This blog will be geared towards some ideas floating around in my head on what would be some good examples of outsourcing.

You have to step back and look at it from a different angle. If you aren't ready to outsource the whole farm just yet, then you can go about it in a couple of different ways. One, you can outsource your sandbox, development, and/or test environment. We all know that with SAS 70 and SOX you have to have all of these (or most of them anyway). And outsourcing might be a good way of getting them in place. The cool thing about outsourcing any or all of those are you have a pristine environment and if it does get polluted somehow you can just reload the OS quickly and painlessly and try to tear it up again. Outsourced servers are great for this type of scenario. You can even get a few servers and carve them up virtually and have even more toys to play with. Now, you can just go buy new servers and have this in house but when they break or they are obsolete then you get to buy more. With an outsource model you can buy 1 or 100 and have them for 1 month or 2 years, it's up to you, your needs, and your budget. You can add hardware, memory, change the OS daily, and only buy the License for a month instead of having to buy it outright when you buy your own servers. I personally believe this is a really good way to get acclimated to outsourcing and test the waters both with yourself and your boss. You always have to make sure they are ok with the way you are doing things. Well, sometimes anyway.

Another option with outsourcing is outsourcing production. Some bosses out in the world aren't ready for this yet, but they will be. They like keeping their data close by and having multiple copies and instances and USB keys with copies on it, etc. That's just the nature of data. Now we all know that you can have the same if not more redundancy in the outsourced model too, it is just hard to explain to them sometimes. I have to give them credit. Think about all the data in the world and how much of it we need to use every day. If folks like them didn't demand that we techies keep it safe the world might have a bad day, I know I would. I use tons of data everyday (might be a fun blog).

If you decide to outsource dev/test or production you have the ability to scale quickly and accordingly when dealing with technology. Not having to be bogged down by worrying about hardware lead times, dealing with accounts payable, the receiving dock, and all the other worries you have when buying hardware is a liberating feeling. I know what you are thinking; I have been over this side of it a few times so I will just leave it at that but the numbers and today's technology make it all come together and make good business sense.

Outsource IT!


March 4, 2008

What’s it like to work at Seattle Softlayer?

I am one of the new guys in the Seattle datacenter. Since I started, people often ask me that question or some variation of it. The short answer: It's wonderful, I love it.

The long answer is...well, long.

First some background! I'm a recent addition to the IT industry. Since I joined the workforce, I've worked in fast food, security, retail, and even a call center. After my call center job, I decided to take my computer hobby and turn it into a career. I received my associates degree and found that I really needed employment. Luckily, my school helped me search.

For almost three months, I received calls from IT recruiters. I interviewed with 4 companies. I finally had a position lined up that was very promising.

Then my Career Services recruiter called, asking if I wanted to come in for an on-site interview, as SoftLayer was interested in recruiting students. I was hesitant, as I already had a position lined up. She convinced me to interview anyway. The day before the interview, she sent me tons of information about SoftLayer. After poking around, I found the InnerLayer. (Has this come full circle or what?)

After reading a few blogs, I realized that SoftLayer had the perfect corporate culture for me.

The next day I interviewed. Joshua Rushe, VP of Operations conducted the interview. He was warm, friendly, and very down to earth. In some ways it was the strangest interview I've ever had. Josh appeared to be more interested in me as a person, than my qualifications. (Of course, he had already seen my résumé.) I was more than a little shocked when he ignored my prepared portfolio. We spent most of the interview time talking about the work, what SoftLayer expected out of an employee, and SoftLayer's corporate culture.

Here I am, 2.5 months later, and I love my position.

So enough of background! What's it like to work at SoftLayer?

We work hard, and we play hard. In the few moments between working hard, when we have the time, we joke around and have fun. SoftLayer recently flew myself and a co-worker down to Dallas to work alongside the great people down there. I learned a great deal. For instance, I learned at the cable labeling party in Dallas that we can work hard and have fun at the same time!

The people are helpful and friendly. The work is challenging and rewarding. It's nice to know at the end of the day that I've done my part to help the internet grow.

March 3, 2008

I'm NOC Gonna Get Sick!

**Cough, cough, sniffle, sniffle, hack hack**

These are the famous noises that come from the NOC every so often. I swore and swore that I wouldn't get sick. To be honest, there was something going around about four months ago, and I was just about the only one that didn't get sick, and I was King-of-the-NOC!

Not this time.

Emails were sent out -- "Clean your workstations -- wash your hands -- don't throw your used Kleenex tissues at other NOC personnel -- and for the love of God, don't get sick". Oops. So, one by one, each NOC technician started getting sick. One down, two down, three down…

Then it hit me.

You know how it starts, don't act dumb. It all starts with that sore throat, that isn't that sore, but makes you wonder if you're getting sick, and everything ends up becoming a psychological battle of "do's and don'ts" to get better, before you get any worse. It never works. You start feeling that sore throat, which gets worse as every hour goes by, as you start overdosing on Vitamin C drops/pills. Then you think, "I don't just need Vitamin C, right!?" So, then you did around the infamous Softlayer NOC Pharmacy, and start overdosing on off-brand multivitamin, Centrum wanna-be's**.

Things get worse.

So, by the end of the shift, your throat feels like it's on fire. You have to make a Wal-Mart run at 12am in the morning (depending on your shift), and you buy every little piece of medicine you think you might need to make life better while you are…Sick.

So, for a few days, you end up chugging cough syrup, feeding on Centrum wanna-be's, Vitamin C pills/drops, Halls Mentho-lyptus "Mountain Menthol" cough drops, Airborne Formula (more on this later)...and VITAMIN GUMBALLS!!

That's right, folks. We have vitamin gumballs, and they are GROSS! The pink one is okay, probably the best out of them all, but it still taste like rubber. Ugh. Now, as for the Airborne Formula, I just don't trust it. I mean, people say it's GREAT, however, I need proof. I mean, come on, it was created by a second grade teacher. Was this teacher a doctor before he/she decided to actually teach kids multiplication? Think people. Think.

Overall, most of us in the NOC got sick, including me. I'm just now getting over this, while I still fight off a tickle cough, but I’m sure this will never be the end. So folks, keep taking that Vitamin C and that Centrum wanna-be, and don’t get sick. I’m sure next time I’m NOC gonna get sick!

** Centrum Wanna-be is what I call Off-Brand (Equate) Multivitamin Tablets.


February 28, 2008

Companion Cube

For the gaming/programmer community, Portal is THE GAME. Seemingly from nowhere, this game burst upon the scene and took the nerd world by storm. Excellent storyline, snappy dialog, challenging puzzles, and an awesome space-warping gun combine to create an incredibly memorable game.

Nearly overnight, the hacker lexicon got some new words and phrases. This guide will help you make sense of most hacker conversations you may hear that refer to Portal.

The Cake is a Lie

The artificially intelligent computer that runs the series of puzzles (known as the Aperture Science Computer Aided Enrichment Center) goads your character along by stating that "cake" will be served to you if you survive all the tests. However, graffiti on the walls of the center proclaim otherwise... stating that the cake being promised is a great, big lie.

Hackers generally use this phrase either as an icebreaker or as a description of a situation where somebody is motivated to do a difficult task for a promised but unverified reward.

I'm making a note here: Huge Success!

At the end of the game, the computer gives you your final review. At the beginning of the review, it says "This was a triumph. I'm making a note here: Huge Success!"

Like Trekkies shouting "Qua'pla!" (Klingon for "Success!"), programmers are now known to say that they are "making a note here: Huge Success!"

The Companion Cube

The Companion Cube is an inert storage cube imbued with a personality by the game programmers to trick game players into carrying this cube throughout a puzzle, but then requiring them to destroy it at the end. They did such a good job, however, that game players have become attached to this "Companion Cube," going so far as to build little paper models or buy plushies of this "character." Generally, you'll hear a hacker talk about how they would never let go of their companion cube, or something along those lines.

Aperture Science Thing We Don't Know What It Does

Programmers find long multi-word names for products to be humorous. The game developers played on this concept:

  • It's not the lab. It's the Aperture Science Computer Aided Enrichment Center.
  • It's not a storage box. It's an Aperture Science Weighted Storage Cube.
  • It's not a button. It's an Aperture Science 1500 Megawatt Superconducing Super Button.

The joke is to take, say, a mouse, and turn it into an "Aperture Science Rotational Axis 2 Dimensional Vector Detecting Peripheral".

Now You're Thinking in Portals

With this guide handy, you can start to understand the conversations of your Portal crazed coworkers. You will no longer be confused when a coworker bursts out laughing when holding a slice of cake. You won't wonder why he has a background with little hearts all over it, displaying a strange box. You can now safely laugh at any name longer than 4 words, knowing that it's most likely a joke. This won't help with any of the other strange things developers say, but at least their conversations should be a little more transparent now.

For more information, check Wikipedia, or better yet, Watch the trailer and play the game.


February 27, 2008


It’s a fact -- all software ends up relying on a piece of hardware at some point. And hardware can fail. But the secret is to create redundancy to minimize the impact if hardware does fail.
RAIDS, load balancers, redundant power supplies, cloud computing - the list goes on. And we support them all. Many of these options are not mandatory, but I wish they were! That’s where the customer comes in – it is critical to understand the value of the application and data sitting on the hardware and set a redundancy and recovery plan that fits.

Keep your DATA safe:

  • RAID - For starters *everyone* should have a RAID 1, 5, or 10. This keeps your server online in the event of a drive failure.

The best approach – RAID 10 all the way. You get the benefits of a RAID 0 (striping across 2 drives so you get the data almost twice as fast) and the security of RAID 1 (mirroring data on 2 separate drives) all rolled into one. I think every server should have this as a default.

  • Separate Backups – EVault Backup, ISCSI Storage, FTP/NAS Storage, your own NAS server or just a different server. Lose data just once (or have the ability to recover it painlessly) and these will pay for themselves. Remember, hardware is not the only way in which you can lose data -– hackers, software failures, and human error will always be a risk.

StorageLayer. Use it or lose it.

Going further:

  • Redundant servers in different locations – spread your servers out across different datacenters and use a load balancer. Nothing is safer than a duplicate server 1000’s of miles away. That’s why we have invested in a second data center – to keep your data and business safe.

Check 'em out in our Services > Network Services section.

The future:

  • Solid state drives – aww yeah baby. They are coming.

Solid state drives are just that – a drive with no moving parts. No more platters or read/write heads. I mean come on, hard drives are essentially using the same basics that old record players use. CD’s use this technology too. And you see where those went (can you say iPod? I prefer my iPod touch. I have never had an iPod until now so I skipped right to the new fancy pants model. Can you tell I just got it?).

Check out these comparison tests of solid state drives vs. conventional ones:

  • Faster, faster, faster! –- Processors, memory, drives, network -- everything is getting much faster. And in part by redundancy (dual and quad core processors, dual and quad processor motherboards). See? Redundancy is the way of the future!

We have 4 Intel Xeon Quadcore Tigertown processors on one motherboard. That’s 16 processors on one server! Shazam!

  • Robot DC patrol sharks – yep. Got the plans on my desk right now. But I can’t take all the credit, Josh R. suggested this one, I just make things happen.

I work to keep all of our hardware running in tip top condition. But I look at the bigger picture when it comes to hardware – how to completely eliminate the impact of any hardware issue. That’s why I suggest all the redundancies listed above. While I can reduce the probability of hardware issues with testing, monitoring of firmware updates, proper handling procedures, choosing quality components, etc., redundancy is the ultimate solution to invisible hardware.

Hardwhere?, if you will.


February 21, 2008

What It's Like to be a Data Center Technician

As you may have guessed SoftLayer isn't just sales team members, data center managers and development team members. There is also a pretty important group of people who hideaway in their cubicles and can be seen running around our state of the art server rooms from time to time. I am of course talking about us DC Techs; you might know us from our ticket signature "SoftLayer CSA."

I had a question brought up for the first time while on a phone call with a customer, his question was,

"What is it like to be a data center technician?"

I could only laugh just a little bit as I looked around the office and saw several of my co-workers engaging in the organized chaos we call Datacenter Operations. You see, with datacenter operations there is no "daily routine" to follow, there isn't a "what to expect" sheet posted somewhere to prepare us for the day. We have to rely on experience and each other to keep our beloved customers happy. So would you like to know my answer to this customer?

"It depends on the ticket I'm working!"

I say that because this particular customer was calling about a networking issue. In this instance I was his "network engineer", helping him resolve an issue with secondary IP addresses. As I said before, not every issue is the same from one minute to the next so it keeps us on top of our game. One second I am a networking engineer, the next a hardware technician, the next a Systems Administrator. On some occasions we DC techs can be all three at once! It's because of this fact that I enjoy coming to work each and every day. I never know what problem will arise or what I will learn in the coming hours.

I decided to write this after a very long shift, because I think a lot of our customers and people who read this blog would like to know what exactly it's like. Of course there are good days and bad days, sometimes we make mistakes or take a little longer to reply to a ticket than we should. But for the vast majority of the time, our phone calls are ending with "Thanks so much!", and our tickets are ending with "Great Job, You guys are awesome!", and our customers are going to sleep knowing their server is in good hands.

Now what question do all of us DC Techs have? That's simple:

What is it like being a SoftLayer customer?

Judging by everything I have seen recently, with our company expanding to Seattle, building new datacenters, and shattering several of our own sales records, I think we're doing a pretty good job of putting everything you want from a dedicated hosting provider at your fingertips. There is always work to be done, and I speak for everyone here in the office when I say the most important thing to a DC Tech and the company as a whole are our bosses, and we currently have around 4,500 of you around the world and growing!

I’ll see you in the tickets soon!


February 15, 2008

Ordering Lunch - How Hard Can it Be?

Every so often on a slammed sale days, I offer to pay for lunch for the sales team to keep everyone at their desks focused on sales rather than worrying about food. Other times, a very nice customer might offer to pay for lunch one day for the sales team. Regardless of the situation, I usually task someone with ordering and picking up the food so the rest of the team can focus on sales. Seems pretty simple right? Somehow it never seems to go as planned.

Here are two examples:

How to spend $200 on lunch for 6:

Daniel one of our Senior Account Managers calls me on his way into work (he comes in at 11:00PM), here is the conversation:

Daniel: "Hey Steven, I see its really slammed at work want me to pick up lunch on the way in?"

Steven: "Sure, go ahead no one has had time to get up from their desk much less get lunch. Pick something up an Ill buy lunch today for the team".

Daniel: "What should I get"

Steven: "Whatever is fine, gotta go the phone is ringing"

Daniel shows up a bit later with a ton of food, enough to feed half the office not just sales. A really nice Fajitas feast with all the fixings, hot sauce, cheese, beans, guacamole, rice, pretty much everything. I thought to myself, wow Daniel did a really good job here this is excellent. Then I get the bill... It was over $200 for takeout lunch for 6 people. I promptly tell Daniel he is no longer on lunch delivery team, and that $200 for lunch is a bit much. Two months later I am still trying to work up the courage to put that one on an expense report.

How to spend $25 on lunch for 10:

A particularly grateful customer contacted us saying that he wanted to buy lunch for the sales and a couple of networking team members that helped him out with a recent issue. Mary another one of our Senior Account Managers was tasked with the order this time and after much discussion back and forth between Pizza and Mexican food, we settle on Mexican food. I am thinking to myself, thank goodness Daniel isn't in charge of this order, Vik (the customer) probably doesn't want to pay $200 for lunch. When the food arrives, I step out into the sales area to examine the feast. Much to my surprise there is only two very small bags of food half full.

I announce out loud:

"Where is the rest of the food? This isn't close to enough to feed 10 people."
I'm told "that's it, that's all we got".

No cheese, no hot sauce, no guacamole... this is a far cry from the spread Daniel got last time and there was no chance of it feeding 8 people. Ultimately I send someone back for more food.

So what is the lesson learned here? The sale team is excellent at selling SoftLayer services, and managing customer relationships. They can tell you the difference between and why you want a Single processor 5000 series server vs. a Single processor 3000 series server, they can tell you why your video streaming site needs to run on a server with SAS drives and not SATAII drives, and they can tell you all about StorageLayer and how it can help you. What cant they do for you? They can't get the Mexican food order for lunch correct.

Next time we will stick with Pizza.


February 13, 2008

The Usage Of Complex Algorithms For Password Generation

Passwords are difficult. On the first hand, you want to create a password that uncrackable by anyone, lest they be teenage hackers or CSI experts with magical hacking tools. On the other hand, the password has be rememberable by you yourself, lest only teenage hakcers and CSI experts with magical hacking tools are able to access your data.

So, how do you make passwords?

One of the more secure ways are to use a random letter generator, like random.org, to build random strings, pick one, and memorize it. It's pretty secure (random.org uses real random noise to produce it's random numbers)and with seven random alphanumeric characters, the password search space is about 2.2 trillion combinations! But are you really going to remember "QRSr0Fu" or "W96TUON" two weeks from now? (My generated set had "myELlRK" which I might be able to remember...) If you type your password every hour or so, you might remember this by muscle memory pretty quick. Just in time to have to change it, I bet.

Another way is to take a word or phrase, turn some letters into |33+sp34k, and you get something more random, but much more rememberable. So, for example, "minivan" becomes "m1n1v4n!" and "washington" becomes "w4sh1ngt0n!?!" These are actually quite rememberable; the use of non-standard characters disallows the use of rainbow tables and dictionary attacks, so they're much less suseptable to cracking. However, what happens when you forget the "!", or that "Washington" gets "?!?" or that you did NOT turn "t" into "+"? You could end up going through a few cycles trying to "guess" your own password. Again, if you use it all the time, you'll learn by muscle memory. And this lets you come up with some cool passwords, like "c4p+41nK1rk". How can you beat that?

My favorite way, however, lets you write your password down in plain sight. I tend to cycle through passwords, and if you're anything like me you have two online banking passwords, four credit card or loan company passwords, a work domain password, 6 email passwords, a home log in password, etc, etc, etc. If you take the easy way out and use the same password everywhere, you end up making kittens and security experts cry. If, however, you have a completely separate randomized combination for each account, your brain will get stuck in an infinite loop. Using this method, you get to write down your passwords and tack them to the wall. Or put 'em on a sticky note. In plain sight. Email them to yourself without a care. It uses a special type of encryption to keep your password safe. Not AES or DES or TEA or other TLAs. I call this "Hippocampy Encryption" (named in honor of the part of the brain that does memory type activities).

The key is to write down a set of clues that will tell you (but only you) what your password is. You can add symbols to help you remember what kind of encoding to use for your password. Here's a password I just made up right now as an example:

Shawn's rival ^
shout your home team

Because everything on this note is simply a hint for your specific brain to recall a password, it's specific to you. Hints don't even have to have anything to do with the subject. The hint "Red October" could tell you the word "fortworth", whereas for me, I'd be trying "R4M1US", "M1SSL3S", "jackryan", "TomClancy", etc. You can string three or four hints together for a password. Note, these create long passwords, and your coworkers may start to believe that you have a superhuman capacity for memorizing long strings of randomized data. Do not do anything to dissuade them from this belief. And, because the hints point to common words and numbers already lodged in your grey matter, you may be suprised just how fast you type in that 20 character long password. Compared to my speed on 7 character random strings, it's blazing.

And due to the pattern matching ability of your brain, remembering the passwords are easy. Lets say you've written your clue on the back of one of your business cards, so you have it handy if you need it. After a few days, just SEEING a business card will bring your new password to the front of your mind. After a while, you'll stop needing your hint sheet, as you'll just remember the password. And when it comes time to change your password, shred your card and your postit, post a new one (in a different color if you can, helps the brain), and give yourself a few days. Unlike scrawling your random digits on a paper or card, even if somebody stole your "Hippocampically Encoded" card, they would have to REALLY know you (or be a really good guesser) to get the password. Even with your card, you've reduced them to brute searching. And if your card/note turns up missing, it takes about 30 seconds to whip up a new hint sheet. Not only is your attacker brute forcing your hint sheet, but it's the wrong hint sheet anyway!

So... have you guessed my password above? It's GARYkemp!1071Max. 'Course, you'd only know that if you knew that I played Pokemon and left my rival's name at default, that I decided that "^" meant "Make it all uppercase", that my home team is the Kemp High School (and that I was talking high school football), that by "Shout" I meant "give it an exclamation point", but that the whole word should be lower case (because the hint is), that Esirpretne is "Enterprise" backwards, and that I meant to make the serial numbers backwards (but not the NCC part), and that by Sam (a very common name) I meant "Give me the name of Sam's partner in that incredibly funny cartoon by Steve Purcell, Sam and Max: Freelance Police." The period is just decoration. If you did guess it, contact the NSA. I hear they're hiring people like you.


February 11, 2008

Spares at the Ready

In Steve's last post he talked about the logic of outsourcing. The rationale included the cost of redundant internet connections, the cost of the server, UPS, small AC, etc. He covers a lot of good reasons to get the server out of the broom closet and into a real datacenter. However, I would like to add one more often over looked component to that argument: the Spares Kit.

Let's say that you do purchase your own server and you set it up in the broom closet (or a real datacenter for that matter) and you get the necessary power, cooling and internet connectivity for it. What about spare parts?

If you lose a hard drive on that server, do you have a spare one available for replacement? Maybe so - that's a common part with mechanical features that is liable to fail - so you might have that covered. Not only do you have a spare drive, the server is configured with some level of RAID so you're probably well covered there.

What if that RAID card fails? It happens - and it happens with all different brands of cards.

What about RAM? Do you keep a spare RAM DIMM handy or if you see failures on one stick, do you just plan to remove it and run with less RAM until you can get more on site? The application might run slower because it's memory starved or because now your memory is not interleaved - but that might be a risk you are willing to take.

How about a power supply? Do you keep an extra one of those handy? Maybe you keep a spare. Or, you have dual power supplies. Are those power supplies plugged into separate power strips on separate circuits backed up by separate UPSs?

What if the NIC on the motherboard gets flaky or goes out completely? Do you keep a spare motherboard handy?

If you rely on out of band management of your server via an IPMI, Lights Out or DRAC card - what happens if that card goes bad while you're on vacation?

Even if you have all necessary spare parts for your server or you have multiple servers in a load balanced configuration inside the broom closet; what happens if you lose your switch or your load balancer or your router or your... What happens if that little AC you purchased shuts down on Friday night and the broom closet heats up all weekend until the server overheats? Do you have temperature sensors in the closet that are configured to send you an alert - so that now you have to drive back to the office to empty the water pail of the spot cooler?

You might think that some of these scenarios are a bit far fetched but I can certainly assure you that they're not. At SoftLayer, we have spares of everything. We maintain hundreds of servers in inventory at all times, we maintain a completely stocked inventory room full of critical components, and we staff it all 24/7 and back it all up with a 4 hour SLA.

Some people do have all of their bases covered. Some people are willing to take a chance, and even if you convince your employer that it's ok to take those chances, how do you think the boss will respond when something actually happens and critical services are offline?


February 8, 2008

Outsource It: Part II

Wow, I like all of this feedback guys! Really! I had been chewing on that blog for a while. I was basically trying to decide how to write it and apparently the format worked and got some juices flowing on our forums. I was going to post this on the Forums but I think it is a bit too long and isn't using the forums standards. So here is my follow up to TheRabbit in Blog format.

A bit about me; I am an old guy (shh don't tell the guys I play Racquetball with) and I have been in LOTS of different companies of various sizes and types of business. Back when the internet was young and dial-up was the name of the game, I played in that field. In fact, I see a lot of familiar faces here every day. They all stayed in that field and honed their skills and are the guts behind SoftLayer today.

I went out into the world to see what it was all about. I decided I wanted to be technical and since I was a Windows guy it would have to be Microsoft. So I took the tests and got my MCSE and then worked for Alliance Data Systems, a Cargo Airline, A college in Dallas, Cement Company, and a small Outsourced IT company, then I met back up with these guys and here I sit.

So I used some of my experiences with all of those places to write the last blog. Here are a few of those experiences so you can see where it came from.

Alliance Data Systems had great DC's and lots of cash, they didn't need to outsource because they spent the money to do things correctly and had their own raised floor DC's and connectivity, etc. It was a cool place to work and I learned quite a bit. They did things right.

Cargo Airline - Well they tried. We built out a new office building at the airport and we had an office with no carpet, and extra cooling for our server "room". We had some old boat anchor HP equipment and a single IBM server for the JD Edwards accounting box and boy was it slow. We were using Windows 2000 with AD and DHCP to hand out IP's. Funny story, we merged with a really "smart" software company and part of the merger was that the powers from that company got the reigns and could run our IS department. Maybe they are reading this... (evil grin) - So the first thing they did was pulled DHCP out of the mix and went all static IP's because they were easier to track. "You can just enter them in a spreadsheet!" I was told. "Then you know that a 10.x.1.x is accounting, and a 10.x.2.x is sales, etc, etc." I still laugh about that decision today. Ok, back to the real subject. This company didn't spend the kind of money needed to have a good core of systems, and network and therefore the applications suffered. Most of the apps they wrote or used were Web apps and could have been housed in an outsourced facility.

College in Dallas - Believe it or not, the college had some pretty cool DC's on the Campus. They were secure and if I forgot my jacket I froze my butt off. They used Compaq 1u's like sliced bread. Server after server for student access, student records and it was all Citrix apps that students and faculty could connect to. To me it SCREAMED outsource. Think of the electric bills they paid to freeze my butt off, think of the purchasing department that had to buy all those machines. Think of how much they paid me to un-box those servers and rack them, and cable them, and install the OS from CD, and install Citrix and the apps. Then the accounting department had to track them and make sure they were paid for and depreciate them. Granted, even if they outsourced them the purchasing group still has to order them online and the accounting department has to give us a Visa but that is the extent of it. We have Truck days of joy and do all the manual labor for you and we automate the OS install. Then it is just down to the Tech installing Citrix and the apps from the comfort of his desk remotely.

Cement Company, one of my favorite places to work. I was in charge of the Citrix farm, Exchange and RightFax. Oh what fun. They had over 40 home built apps that ran on Citrix. We had 3 DC's, Dallas, Midlothian, and Virginia. They were Top of the line! If you were a rat and liked chewing through cables and you are into Liebert cooling systems from the early 60's! Ok, it might not have been the 60's but they were old. The DC in Midlothian was the best. We finally boarded up the windows facing west because we figured a lot of the extra heat was due to the Texas sun baking them. Ok, funny story #2. While un-boxing and racking a few Dell 1U servers (again they paid me a pretty good salary for my Citrix and Exchange skills, and here I am un-boxing and racking again) my helper decided that it was time to drop test a Dell. I was behind the rack and there was really nothing I could do except watch this brand new Dell server go crashing to the floor from above his head. After reseeding all the cards, CPU, and memory, we crossed our fingers and it fired up. It was a bit warped and bent but we strategically jammed it in between 2 straight servers and it took some of the flex out of the bent box and it worked great, might even still be working today. As you can tell some outsourcing by them would be good as well; Even if it is just the Development and test systems. We lined up like ants at a sugar sack begging for servers for Dev and Test but they were NEVER in the budget. Another great point I think, Capital Expense vs Monthly Expense. For a huge company it is MUCH easier to get them to sign off on a monthly expense.

Outsourced IT - Here is the one that wins it all. My job was to go around Dallas to small and medium sized businesses and be their IT guy. My main focus of course was Citrix and Exchange but you just never knew what you were going to walk in on. One plumbing company had their servers in a barn. An auto parts supplier had theirs in the back of a storage building behind the restroom. Use your imagination. But the ones that got me the most were Doctors offices. Broom Closets, Office Managers offices, just in the hall out in the open, you name it and I saw servers there. Most of the offices already had a T1 in place so connectivity wasn't the real issue. An interesting point is that I always had to sign a Hipaa form to be legal to work on the systems. It amazed me that these systems were so accessible to anyone that might have had access to the office. I wonder if the maid service had to sign Hipaa forms since the servers were right in the open. Sometimes right behind the trash cans. 90% of the medical software I came in contact with was WEB software which is easily outsourceable. And the number 1 complaint I heard from office managers and Doctors was, "I want to connect from home. Can you help me?" So of course we would setup remote access. But it never failed. During Storms they would lose power or connectivity. Or the building power would drop for construction, or a car would hit a pole. There were always issues. I swayed a few high tech Docs to finally consider and try outsourcing and they loved it. A few even use thin clients in the office now and everything happens in a DC. They love it.

I still say that no matter what size business you have OUTSOURCE IT! Maybe not all of it, but for DEV and Test, a hot site AD controller, Web App Servers, Giant DB Servers that live behind those web app servers, Web Farms...etc be the ball and give it a try. We won't argue!



Subscribe to executive-blog