Is every IT department eternally busy?

I don't understand it. Well, actually, I do, but I'll say "I don't understand" just to make it sound like I'm in more disbelief than I really am. The underlying theme of every conversation I've had with a customer, vendor, or otherwise is that IT is busy and doesn't have the resources. It seems as if any project requiring IT resources automatically gets held up in the "IT vortex" of priorities. IT departments are really not this busy.

The IT prioritization issue

Something is fundamentally wrong when IT resources are scheduled 9-12 months out. My gut feel is that what the business is being held responsible to do compared to what IT is being scheduled to do is at times, misaligned. This is not necessarily the fault of IT but rather the business leaders (in which IT should have a seat at the table).

From an operations and manufacturing perspective, we automatically know that it's standard procedure to have a lengthy-but-calculated process for launching new products to the market. Yet, there's perhaps somewhat less frustration with operations & manufacturing on these timelines compared to the frustration I commonly witness with IT scheduling. How can this be?

The business doesn't understand IT

Business users simply don't understand what they can't see. Operations and manufacturing have the luxury of producing physical products and business users can see the progress of this development and can visually comprehend the effort that's put into a product launch.

The challenge for IT leaders is to visualize the various IT processes, illustrate how they plug into the rest of the organization, and be viewed as an ally to projects and not a barrier to entry. It pains me to think how many projects are killed in the course of 12 months because IT resources are required.

Business users need a scorecard for IT projects

Business users need help -- they can't be faulted for not understanding IT, their lack of understanding is on the shoulders of IT leaders. Business users need a scorecard to appropriately rate their projects. There will be some things that internal IT cannot do in the amount of time required. But this doesn't mean "kill the project!" It may mean outsourcing the project to another firm under the guidance of IT.

Unfortunately, many IT shops are one of the following:

• Run as if all IT-related ideas should be their own. This translates to extended timelines and lost time-to-market as IT puts the stops on a project because it wasn't "their idea."
• Operated under the watchful eye of a CFO. This translates to IT operating under the perception that "spending money = bad".

What's an IT leader to do?

1. Openly share your project pipeline (disguising projects that are sensitive, if necessary). The business as a whole needs to understand what projects are in the pipeline. If they don't, they will approach you without any context to why your team is so busy -- which is a problem.
2. Openly share the methodology for prioritizing projects with the business users who are submitting the requests. This will help them understand the process so that when they come to you with future projects, they will be armed the right data...and not with a preconceived notion that the project will "take a while" because "IT is always busy." Knowing the "why" behind project prioritization goes a long way.
3. Accept the fact that you are not all-knowing. It's OK for business users to come to IT with a new idea. You're responsibility is leveraging technology to solve business problems and bringing their idea to life. Without you, it's just an unrealized idea.
4. Spending money is not a bad thing. If you report to the CFO, the CFO is not against spending money. The CFO views business differently and is looking at the return on investment (ROI). If you cannot put in requests for money which outline the ROI, then be prepared to have many of your projects denied.
5. Don't forget to position yourself as a "go to" person.

Quick question: Since when it is a bad thing to save money?

Now, I understand it can present challenges from governing the use of open source as it pertains to compliancy issues. However, this is no different than governing the proper licensing of Microsoft products, too. The fact that it's open source doesn't make open source a "problem." If you have problem governing open source utilization, then you have a larger software/infrastructure governance problem.

I recently came across an article that really hit home for me. It's titled "How to determine if you delegate enough." To quote the article:

How do I know when I am delegating enough? I think that the answer is very simple: You are not delegating enough if the questions that you are getting are easy for you to answer.

If your subordinate comes to you with an easy question ... the possibility is that the answer was indeed simple but you didn't share the necessary information, requiring the subordinate to ask the question. This may mean you retain some information in order to feel that you have not lost control, but it causes your people to be frustrated and to feel that you don't trust them. It's important for you to disclose to your subordinates all of the information that they need to do their jobs.

Shifting from "doer" to manager & avoiding "doing nothing" as a manager

In my recent role change, I'm now managing a larger range of projects and a larger group of people. I've always admired managers and executives who could get their hands dirty. I think this garners respect and establishes credibility with any manager. In this new role, my biggest challenge is identifying the tasks to take on myself and to delegate to my staff. While there are many simple projects and tasks I am perfectly capable of handling without the help of my staff, I need to be able to identify which tasks make the most sense for me to be focusing my time on.

Enter RescueTime - How I measure where I'm spending my time

I recently came across a new free service called RescueTime. It's a web-based service with Mac and PC "listeners" that install quickly and effortlessly so I can monitor my application usage and website visits. The benefit here is that my "doer" tasks vs. my "management" tasks are clearly divided between the websites and applications I use:

• For example: firing up the Mac OS Terminal app and hopping on a server to move some files around is clearly a "doer" role. I file the "Terminal" app as a "development" task/app.
• On the other hand, reviewing Omniture web analytics data is most definitely a web marketing role, so I flag all Omniture sites as "web marketing" tasks.
• Email, Excel, Visio, Microsoft Projects are all tagged as "eBusiness" applications.

The RescueTime Dashboard

Since I have a dedicated laptop specifically for work, the only time I really use it is for work-based functions. Since the RescueTime "listeners" time out after 2 minutes of inactivity, the service provides a very accurate portrayal of the true time spent working (for me, at least). Of course, when I'm in meetings and the laptop is idle, this "work time" is not captured. Since the majority of my job involves a computer, it's pretty safe to say that it captures most of my work-related activities on any given day:

Below we see (for the week of March 30th) I spent 55 hrs and 50 minutes working.

Below we see how my 55 hrs and 50 minutes that week broke down into the "tagged" sites and applications I used:

RescueTime then tells me how efficient I've been based on the above data collection and my account settings (I rate "ebusiness" as more efficient than "development"):

I'm still spending too much time "doing" and not delegating

When logged into the RescueTime dashboard, you can hover over the above red bars to see the total time spent on the tags. "development" was comprised of more than 11 hours of work. That's 20% of time dedicated to "doing" which is much too high. RescueTime has also calculated my efficiency. Efficiency is described as:

A score based on the ratio of productive activity to distracting activity. To improve this, spend a higher percentage of time on productive tasks.

Without the RescueTime software and web service, I'd have no way of quantifying this information other than going by gut feel.  Now that I know I'm spending too much time, I am not worrying as much about being perceived as a manager who doesn't like getting their hands dirty. RescueTime is enabling me to focus on more accurately managing and delegating the tasks and projects for my staff.

This would be completely unacceptable for a consumer website. I guess since the state requires registration and you have no other choice but to visit their site around their maintenance schedule or (gasp) go into a local DOT office, they really don't have to cater to the schedules of citizens.

Website maintenance, regardless of industry, should be done during off-peak hours. Granted, Sunday mornings are probably not peak hours, they are still more prone to visits than they would be between the hours of 2AM and 7AM.

Here are some examples of how you can position yourself more effectively with other departments in your organization so they look at you as the "go to" person not by just the title on your business card, but by the value you bring to their business functions.

The title on your business card defines what you do, not how you do it
The title on your business card does not always mean you are viewed as the "go to" person for your functional area -- I have experienced plenty of people in business who are avoided at all costs due to lack of strategic and/or big picture thinking within an organization or on a project. Your title defines what you do, but how you go about doing it is another game completely.

In IT, understanding the needs of another department is extremely important when they come to you with a question or request. Nobody likes feeling stupid, and this is one area where IT typically falls short -- fail to understand the problem, provide short, non-descriptive answers to questions, and allow the uneducated business person to craft the design requirements for a (web) project that makes little sense. This then results in an application or solution that underdelivers due to poor planning and creates a customer (the employee in the other department) who is unhappy.

Picture yourself bringing your car to a mechanic for service...how do you want to be treated?
(This is often times an easy analogy to make, so if you already understand cars, then pick another area where you are not as knowledgeable in and put yourself in the position of that customer.) You drive your car into the shop -- it's vibrating whenever you "drive it" and you obviously want it fixed because A.) it's annoying and B.) it seems very unsafe!

Now, there are two ways to approach this: probe deeper, ask questions to help you navigate the troubleshooting process with the customer face-to-face, or take the car and run a series of tests that run the risk of looking at an area of the car that is not broken (and in the end not be able to find anything wrong -- we've all experienced this, and it's frustrating!). Vibration in a car can be a number of things -- bad brakes, unbalanced tires, unbalanced driveshaft -- the list goes on, and can be varying degrees of technical explanation depending on the customer's expertise on the matter. A "go to" person asks questions because they genuinely want to help.

Sometimes the problem described is not the source of the problem at all
Odds are the customer doesn't know exactly what the problem is, but they may suggest a fix because they don't want to appear stupid in front of you. This happens countless times in business! They could say "the tires feel out of balance" but in reality, it could be that the vibration only occurs during braking, which generally points to warped brake rotors (among other things). Being the responsible businessperson you are, you would always start this process by going back a step or two to understand the customer's needs and a description of the problem.

This will ultimately lead to a more accurate and timely resolution to a problem and a solution that hits the nail on the head. Part of being the "go to" person is providing that guidance that other departments lack -- knowing they can come to you with an idea and you can help them make the most of it without making them look incompetent is critical in business. You will turn them into repeat customers.

Your customers (fellow employees) don't really care about your deadlines
Another problem area in IT is the ability to turn on a dime or the tendency to paint a dreary picture from a resource standpoint. Just like bringing your car to the shop, you don't really care to hear about all of the other work the shop has queued up, so spare your own internal customers (fellow employees) the details. Explain that you want to help them, understand their timeline, and fit their project into the mix where you can. If you make other departments feel like you're doing them a favor for every request they come to you with and that it feels like pulling teeth just to get some time, you will lose your position as the "go to" person. Likely, they will look elsewhere or even outsource -- at which point you're cut out of the process completely.

A "go to" person follows up.
Ever brought your car to a mechanic, the expected due date comes and goes, and you never hear from the service manager? Avoid this situation at all costs. Manage your customer's expectations, and provide reasons for why you're going to be later than promised. Things happen, deadlines change, but how you manage the situation will also improve the satisfaction and perceived value you bring to a project and will ultimately paint you as a "go to" person. The "go to" person gets things done and follows up when they have or haven't been accomplished. It's just that simple.

Work through the process and/or problem, don't work around it or point fingers
Many times IT is looked at for solutions to sharing data or information with other internal departments or external customers. This often times means creating a new process for the application or implementation you're building on behalf of a department. If the success of your project implementation depends on the actions of another person or department, then work with them until their job is finished. Unless explicitly told to do so, don't "pass the buck" and assume your work is done when another department has to get involved. Part of being a "go to" person is finding the answers to problems that are outside of your current knowledge or your functional area's expertise.

Your internal customers may not realize the amount of collaboration involved, so don't hesitate to give them a high-level (notice the phrase "high level" -- avoid the technical details!) overview of what's being done throughout the project. This is what will make them look to you in the future for other projects and view you as a "go to" person.

Web-based support that can be trusted for a speedy response
Rackspace has both phone and web-based tech support, but they are so speedy and courteous at responding to web-based support messages that I routinely go there first for a question, request, or resolution to a problem. How many companies can you say this about?

You know the drill: the cryptic support hierarchy laid out on a company site, insisting you search their knowledgebase first (which is always sub-par and rarely has the answer to your question), presenting you with a customer service contact form, and if you're lucky, a toll-free support number.

I can't tell you how many times my only point of contact is an e-mail address at a vendor either for their "technical support" or for an account manager. I always am concerned that the e-mail will go into a black hole and never be returned (e-mail is such a terrible way to manage tasks). I've been so accustomed to this, that I always end up BCCing myself so that I know to flag the e-mail for later follow-up. Otherwise, I'll forget about it and I may never hear back unless I resend the e-mail.

A customer knoweledgebase that just works
The Rackspace support model is equally as fantastic. You have an account manager and essentially a "dedicated" team of support techs of varying skill sets that you'll basically always deal with throughout the week. Each message is signed by the tech, an entire log of your conversation is available in their support ticket system, and it all just works -- it's a complete CRM solution customized for their business and the customer benefits from their collaboration. No matter who you talk to, they can access the same information anyone else in the company can access (aside from sensitive information like server passwords). What a concept!

Better yet, the account managers can access what the support people are doing with your account. Not the case at some vendors who have departments operating in silos.

Managing customer expectations
There's nothing I love more than to report a problem or submit a request, get a response within a 2-hour window with a list of "next steps" and when it will be resolved by. Sometimes a customer problem is not a simple, 30-minute fix, either. Even the problems that keep Rackspace scratching their heads over the course of the week are kept up-to-date on a daily basis each week and always have a senior technician checking in on the status each day.

With these other vendors, I am the one checking in with them. Where's the client service in that?

Anyway, thank you, Rackspace. Always a pleasure to deal with you.

Big-picture thinking
The CIO is a strategic position because it requires big-picture thinking, and the ability to quickly and effectively identify broken processes (or lack thereof) where technology can be integrated to improve efficiency and the bottom line -- or better yet, drive new business and capture market share. Driving new business and capturing market share is really a critical area where a CIO can contribute -- because it requires a heavy dose of business intelligence and market awareness.

But I thought the CIO was the head of the IT department?
The IT department is an operational entity, responsible for executing the support of the daily technical needs of employees, implementing new technology as a result of a business need, or implementing/building new services as a result of a business need. IT still needs the "business need" delivered to them as well as an operational manager who manages the department. Depending on the size of the organization, there may be multiple operational managers within the various divisions of IT. The CIO will guide the functional managers in IT to implement solutions that satisfy current and future business needs/problems.

Why don't other departments just talk to the operational manager(s) of IT?
Typically, departments (marketing, HR, Finance, engineering, etc.), lack the technical expertise to be able to identify exactly what they need to solve their problem. IT can normally listen to these problems and provide a very specific solution. The problem is when these solutions are implemented in silos. Over time, you have many "one-off" IT projects, built as temporary fixes or workarounds that gradually grow into a substantial maintenance burden and waste of IT resources.

This is where a CIO-role plays a strategic role. The CIO has visibility to multiple departments and layers of the organization. The CIO thinks about all of the needs of the various departments, takes future needs into consideration, and plans for scalability. The "silo effect" is neither fun or fair to anybody in IT and as a company grows, managing all of these silos becomes cumbersome and uninspiring -- and costly. To undo years and years of silo'd development is usually a major undertaking.

Alright, so the CIO doesn't manage the IT employees. What is the CIO involved in, then?
To quote an excellent editor's note from InformationWeek, "CIOs are less involved in day-to-day operations and technology implementation and more involved in business strategy, revenue generation, business-process management, and customer relations." This article was written in July 2004 and still holds true today. The only addition I would make is that the CIO is also heavily involved in online strategy because it is such a critical customer touch-point and is a major source of revenue generation (directly via e-commerce or indirectly via offline sales), customer relations, and requires strategic business-process management.

4th Place: INetU

I really liked their website, the fact that they promoted their customer satisfaction rating (which is really high, at 96%), and they offered FreeBSD (my OS of choice, and where my comfort level is). Unfortunately, their pricing structure is not setup for high-bandwidth environments, as their bid came in much, much higher than the other 3 ISPs who were bidding on our business. Kudos to them for not straying from their core competency, but probably not a viable consideration for us in the future unless they make significant changes to their billing model.

3rd Place: Rackspace

I have real-world experience with Rackspace and their managed hosting, so I knew what to expect in terms of service and support. They did an excellent job in coming down on the costs of their hardware to match PEER 1's quote, but just were not able to match the bandwidth (they could not go higher than 1000 GB/mo -- as this is their standard bandwidth package). They could have gone higher, but at $1/GB, which would have gotten very expensive for us. Had bandwidth not been the primary driver behind our decision, Rackspace might have been our final decision, just based on my past experience with them as a managed hosting provider.

2nd Place: PEER 1

Excellent sales follow-up, paid attention to my blog, and had the best overall price between INetU, Rackspace, and PEER 1. However, in the end, they did not offer FreeBSD for an operating system and required a 12/31/06 commitment date, which would have meant 2 months of overlapped billing with our current ipHouse colocation contract and the new PEER 1 managed contract. Also, switching ISPs would have meant dealing with DNS switches for our customer's domains, dealing with e-mail account downtime for customers, transferring data over the Internet to the new servers...the list of "cons" went on and on. Granted, this would have been the situation for INetU, Rackspace, or PEER 1...which is why we went with ipHouse:

1st Place: ipHouse

Our current ISP where we've colocated for many years now. We're comfortable with their support, their sales staff, and their network reliability. This, combined with the fact that they will run a managed FreeBSD server for us, matched the PEER 1 bandwidth quote, beat the PEER 1 hardware quote (PEER 1 uses commodity hardware to keep costs down where as ipHouse is getting us new Dell servers). Plus, not having to deal with the IP change, DNS changes, e-mail downtime, etc...everything just made sense for us to stay with the current provider, and for less than we're paying today. We also bumped up our colocation contract expiration date by 30 days to get started with managed hosting even sooner. Bottom line: no overlap in hosting contracts, faster time-to-production with managed hosting, more bandwidth, better/faster/more reliable servers, less cost.

Moral of the story

• If you're near the end of your hosting contract, always shop around to see what else is out there. The grass may be greener on the other side.
• Put a competitor's bid in the hands of your current ISP and see what they can do about meeting or beating the bid.
• Prices are usually negotiable. Never take the first offer and always talk to the rep via phone or e-mail about the quote so they can understand your level of commitment for hosting.
• By renegotiating, you may be able to enter into a new/better contract a month or two sooner than expected -- and thus not have to wait for your existing contract to expire, as long as it means your current ISP keeps you as a customer for a longer period of time.
• It's more expensive for an ISP to obtain a new customer than it is to retain an existing customer for slightly less money.

• Calendars
• Chat
• Customized Start Page
• Domain Web Pages

I realized that calendars and chat were basically available, what I didn't realize was that you could toggle them on/off. I also realized that there wasn't a waiting list for signing up new domain names any more (though the service is still in "beta"), too. I proceeded to setup google apps for 7 other domain names, making a total of 8 domains that I use Google apps for. After trying to add a 9th domain, I got this message:

The Google Team

Bummer! But, I can't complain. I'll take free e-mail hosting for 8 domains. This also means I can clear off all of the e-mail accounts on our mail server which makes one less thing to setup when we switch ISPs. What I also like is the ability to host web pages on a domain name with Google, meaning I can eliminate all of the one-page domains that we host, as well. Can't wait to see what's next with the service -- hopefully full integration with Picasa, Gmail mobile, and other Google applications/utilities.

Take Peer 1, a managed hosting company who we are strongly considering to have handle SuperMotors.net hosting. Everything checks out on their website - professionally designed, up-to-date information, current press releases. Not much in terms of comments on their managed hosting service online, but that may be because this is still a relatively new market for managed hosting with not a lot of competitors and customers. The price looks good, the company looks good, but when I ask for 2-3 existing customer references (of equal or higher contract terms as we're considering) from the sales rep I've been working with, I get nothing. Last communcation from the rep was on November 13th. It's now December 5th. I've sent follow-up e-mails re-requesting 2-3 customer references on November 14th, November 26th, and December 1st.

I guess e-mails are not good enough and I'll need to call. It shouldn't be this hard for us to spend money. Not only is this annoying, but it speaks volumes about their commitment to customer service. Do they even have 2-3 customers that are going to give them top marks? Strike 1 for Peer 1!

So, when implemented over the course of the entire month, we saw an overall 27% reduction in bandwidth. Surprisingly, our absolute unique visitor count was down 14% for November (holidays do this to our site), but page views actually increased by 1%. Page views are ultimately the driver in bandwidth utilization. I interpret this to mean that fewer users were able to do more on our site because they were downloading content in less amount of time.

So, why does the test tool from above show an 81% reduction in bandwidth yet we only saw an overall reduction of 27%? This is because images (and other files) are not compressed when delivered to web clients. Since we primarily serve images and videos, the rest of our bandwidth still remains largely untouched by mod_deflate -- which is by design.

This bandwidth reduction is very good news for us. As previously posted, we are leaving the world of colocation and moving to managed hosting -- with a new billing model. The "total bandwidth used" billing model favors us even more now that we've reduced bandwidth usage by 27%. This translates into squeezing more data out of the pipe than previously anticipated. We'll be able to maintain our fixed costs and increase revenue, thanks to this little module. The more data we push out, the more page views we serve, and the more revenue we make from our CPM-based advertising model with Tribal Fusion and our in-house ad inventory (read the challenges we face when we introduce Ajax functionality and how it'll affect our CPM-based ad model).

The mod_deflate module does increase CPU usage due to the need to compress each page sent out. However, this had very little impact on us as we had plenty of processing power to spare. Your results may vary, so keep an eye on CPU usage when you implement this module. The slight increase in CPU usage was worth the risk, because the 27% reduction in bandwidth was a much bigger gain for us.

12/4/06 Edit: Ajaxian has an article on gzip compression with some user responses. Interesting insights (in the comments section) on the law of diminishing returns with gzip compression used on ajax-enabled pages. 

In our never-ending quest to drive down operational costs of running a hobby-site-turned-online-business, it was critical for us to find an affordable colocation provider in terms of rack rental fees, bandwidth fees, reliability, and 24/7 data center access.

These reasons for not entering into multi-year hosting contracts are not to do any disservice to our current colocation provider. We have been nothing but happy with the experience. Changes in online business, technology, personal, and professional lives of small business owners are sometimes very unexpected, and it's these changes that we've experienced that I wish to share with you in our 5 reasons to avoid multi-year hosting contracts:

1.) The price of bandwidth continues to drop

This may seem obvious, but it is completely true. Since we began colocating, the competitive landscape has increased many times over. Prices, levels of service, and popularity of colocation and managed hosting have changed significantly. We are nearing the end of a 2-year contract and let me tell you, in March of 2005 when we renewed our contract, it was the best deal at the time. But guess what? It's very expensive compared to what can be had now. We find ourselves today locked into 2005 pricing.
Lesson learned: It may look like a cost-savings at the time to drive down your immediate monthly costs by a few dollars. The money wasted down the road is considerable.

2.) The price of server hardware continues to drop

What seemed like an "investment in the future" in mid-2004 when we built our server, turned out to be anything but. The simple fact is that a server investment is a considerable expense for a small business, especially for a community-driven site that can grow at exponential rates. At the time, 300 GB SATA drives were the biggest, baddest drive you could buy. Today, just over 1 year later, 750 GB drives are coming into the realm of affordability for the common man. We quickly outgrew our server in the course of just 1 year and found ourselves in need to make another significant investment. While still commodity hardware, it does get more expensive to "do it right."

Bound and determined to again "invest in the future," we made a significant investment which would later turn out to be a complete disaster. While not really linked to colocation in any way, this type of situation is a potential risk in any business building their own servers and working with the lowest cost vendor. Hindsight is always 20/20; if I were to do it all over again, we would have purchased a Dell PowerEdge server for considerably more money and just called it a day. However, that would not have saved us from #3.

3.) You WILL outgrow your servers

Unless you have a SAN, which is highly unlikely for a small startup, you will outgrow your storage capacity more and more often. For our business, file storage is a critical component...more so than database performance. We can get away with a fairly light-weight database server due to our database structure and server optimization with FreeBSD and MySQL, but there is no magic to file storage. We just need more. And we need more space ALL THE TIME. This creates a very difficult financial model because it truly is exponential growth. The more users that hop on broadband internet connections, the easier it is for them to post more data. The more digital cameras become affordable, the more pictures they post. The more megapixels digital cameras offer, the larger the file size is that gets posted. Today, with the popularity of video and how easily the average consumer can record video and digitize it on their home computer, this creates an even larger demand for storage.

4.) You WILL outgrow your original bandwidth requirements

When we locked into a 2-year contract in 2005, we were locked into specific Mbps 95th-percentile billing. 95th percentile billing is a great way to control your monthly costs, but is an even better way to choke your growth potential. Community-based sites offering photo, audio, and video hosting like SuperMotors are much better suited for total bandwidth transferred. Furthermore, what we found is that in order to increase our bandwidth, this meant committing to the bandwidth through the end of our contract or signing a new contract (and thus extending the original contract).

Due to the nature of 95th percentile billing, we did cap it. An open 100 mbps connection to the internet on a 95th percentile billing model is a recipe for disaster for a site like ours, financially speaking. A user that posts a link to a video on a popular forum could single-handedly increase monthly hosting costs many times over because of the spike in traffic that their shared video could cause. Under total data transferred, this is a much more manageable billing model for our site.

But back to the bandwidth requirements -- your requirements will surely increase over time. It's better to start low and gradually increase your monthly costs than to commit to where you think you'll be in 2 years just to get the better monthly deal now. You'll spend more money in the long-run by committing early. While there is no guarantee, the trend definitely says bandwidth will continue to drop -- which is another reason to ramp up your bandwidth needs.

5.) These days, managed hosting is completely affordable

Thanks to increased competition, more affordable server hardware, and bandwidth prices dropping, viable companies like Peer1, Rackspace, and INetU have made a business out of providing managed hosting. Through Peer1 managed hosting, we will get 40% more storage space, 4 times the bandwidth (based on our monthly average data transferred), 24x7 managed hosting, a 1-hour hardware replacement Service Level Agreement (SLA), and a 100% network uptime guarantee on the core Peer1 network all for 5% LESS than what we pay for colocation today. This is an incredible change in landscape compared to where we were nearly 2 years ago when we were "planning for the future" with buying and managing our own server hardware and locking into the "affordable" 2-year colocation contract.

Best of all, we no longer have to manage or own any hardware. There are no more sleepless nights wondering if the servers will crash or repairing the servers when they do crash. We can focus 100% on building our online community and sleep peacefully at night knowing we have a solid service level agreement and 24x7 tech support.

We will be committing to a 1-year contract (minimum commitment). At any point in time we can upgrade our servers to support additional storage or processing needs as well as add additional bandwidth on an as-needed basis. This model keeps our operational costs at an all-time low while giving us the flexibility to scale without signficant financial risk (like our CCSI vendor fiasco) and without significant financial/time investment (like purchasing, building, and setting up a server). At the end of 1 year, we can always re-evaluate the competitive landscape to see if it makes sense to move our business elsewhere. Moving an online business from ISP to ISP is certainly no joy-ride, but knowing we have the option puts us in a position of power come time to renegotiate a contract.

We have the wrong billing model
Additionally, we are realizing that the 95th-percentile billing model is very, very wrong for our type of traffic and our type of online business. Total data transferred is by far the better of the two from a cost and performance standpoint. We are billed on 95th percentile based on ipHouse's billing model, not because they are evil in any way. We do not fault them at all for their billing methodology -- you will find a mix of billing models with many hosting companies.

Searching for managed hosting
But, all is not lost. We are nearing the end of our contract (3/1/07) at the time of this writing, so I am doing my due-diligence in finding a reliable managed hosting provider for us. While we have some decent hardware that we will really have no use for after we switch from colocation to managed hosting, in the long run, and managed solution is better for us anyway. With me living on the east coast, Kirk and Elliot back in the midwest, it makes it hard to manage our own servers. Plus, Greensboro, NC isn't really a hotbed of technology, so colocation options are few and far between...and a little on the pricey side. Neighboring cities such as Raleigh and Charlotte have viable options, but again, the convenience aspect is not there as those are both 60-minute and 90-minute drives, respectively. Not the type of response time to a server problem that we want to deal with. Nor is it convenient to drive that far if there's a hardware problem.

So, managed hosting enters the picture. Wow, what a decision to make. We are essentially entrusting our entire business into the reliablity of another company. Previously, we at least owned our own equipment and could at any time take said equipment and put it somewhere else. Granted, we were in a 2-year contract with ipHouse, but it was always a nice safeguard to be able to just pull your equipment and essentially your data away from one place and put it at another place.

The Players
In my quest, I have found 3 managed hosting providers that I would consider trustworthy and reliable for the size of our online business. In no particular order, they are:

1. Rackspace
2. INetU
3. PEER 1

My delimma is a little more complex in that I'm used to managing our own servers and hardware, so it's difficult giving up this level of control. Fortunately all 3 providers above provide full root access to the server. However, being the FreeBSD guy that I am, only INetU offers FreeBSD hosting among the 3. Rackspace and Peer 1 both offer Windows Server (heh) and Red Hat Linux Enterprise.

FreeBSD guy scared of Red Hat Linux Enterprise (yes, I'm not afraid to admit it)
Knowing very little about Red Hat Linux Enterprise is a concern to me. I've had experience working with it at Levolor as we have a load balanced and fully managed environment at Rackspace for Levolor.com. The support has been phenomenal, to say the least. INetU is significantly more expensive, but offers FreeBSD. Peer 1 I have no experience with, but has the best pricing, by leaps and bounds.

Decision to be made by 12/31/06
I will continue to post my findings as I work with sales reps from all three companies and compare pricing, services, and good ol' fashioned "gut feel" as I work with each company. In the end, we will be comitting to a 1 or 2-year contract with whatever company we work with. As of this writing, Peer 1 is the most attractive deal, Rackspace a close second, and INetU a distant third. My decisions could easily be swayed in the next month as we aim to lock into a contract between the 12/31/06 and 1/31/07 timeframe.

example1.com:smtp.anotherserver.com:2525
This tells qmail to send all e-mails destined for example1.com through smtp.anotherserver.com on port 2525 thus bypassing the port 25 blocking by your ISP.

In /etc/sysctl.conf (things I added):
vfs.vmiodirenable=1
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=65535
kern.maxprocperuid=32768

vfs.vmiodirenable -- recommended by others to be set like above. Do this if you have a high-traffic server.
kern.maxfiles -- max files allowed to be open at one time by the kernel.
kern.maxfilesperproc -- max files allowed to be open by a process at a time.
maxprocperuid -- max processes per user id to be running at a time.

These are the settings that I needed to add to custom kernal to enable dummynet:
options DUMMYNET # (enables dummynet for bandwidth throttling)
options HZ=1000 # (dummynet option highly recommended by man but I'm not really sure what its purpose is)

Dummynet then gets activated by setting up entries in ipfw. Basically, you create a pipe for data to go through, and then a bandwidth rule for that pipe. I added this /etc/rc.firewall so that these were implemented at startup. Like ipfw, you can add/modify/delete these optiosn on-the-fly:
# DummyNET config: limits outgoing bandwidth, doesn't
# limit incoming bandwidth.
# 2/16/2004
#
# Throttle traffic from: 192.168.1.10 (main site & e-mail)
${fwcmd} add pipe 1 ip from 192.168.1.10 to any
${fwcmd} pipe 1 config bw 1250Kbit/s

# Throttle traffic from: 192.168.1.11 (users & hosted sites)
${fwcmd} add pipe 2 ip from 192.168.1.11 to any
${fwcmd} pipe 2 config bw 512Kbit/s

Basically, what the above does is setup all outgoing bandwidth from IP address 192.168.1.10 to use up to 1,250 kbps or 1.25 Mbps of bandwidth. Incoming bandwidth is not throttled. I setup a much lower bandwidth ceiling for pipe #2 as you can see. These sites are less important that they receive bandwidth and are of much lower usage anyway, so nobody will really even know that they are being throttled. The key was to throttle bandwidth down a lot for secondary sites while keeping the ceiling significantly higher for the main site, but still keep it within reason and not allow it to go way beyond what we could afford.