B2B/B2C online strategy and enterprise eBusiness blog

It’s been a particularly bad week for customer and client service from two very important vendors to us (*ahem*…Omniture & scene7). Rather than focus on the negatives, I will instead post my 100% satisfaction with the service and support of Rackspace Managed Hosting.

Web-based support that can be trusted for a speedy response
Rackspace has both phone and web-based tech support, but they are so speedy and courteous at responding to web-based support messages that I routinely go there first for a question, request, or resolution to a problem. How many companies can you say this about?

You know the drill: the cryptic support hierarchy laid out on a company site, insisting you search their knowledgebase first (which is always sub-par and rarely has the answer to your question), presenting you with a customer service contact form, and if you’re lucky, a toll-free support number.

I can’t tell you how many times my only point of contact is an e-mail address at a vendor either for their “technical support” or for an account manager. I always am concerned that the e-mail will go into a black hole and never be returned (e-mail is such a terrible way to manage tasks). I’ve been so accustomed to this, that I always end up BCCing myself so that I know to flag the e-mail for later follow-up. Otherwise, I’ll forget about it and I may never hear back unless I resend the e-mail.

A customer knoweledgebase that just works
The Rackspace support model is equally as fantastic. You have an account manager and essentially a “dedicated” team of support techs of varying skill sets that you’ll basically always deal with throughout the week. Each message is signed by the tech, an entire log of your conversation is available in their support ticket system, and it all just works — it’s a complete CRM solution customized for their business and the customer benefits from their collaboration. No matter who you talk to, they can access the same information anyone else in the company can access (aside from sensitive information like server passwords). What a concept!

Better yet, the account managers can access what the support people are doing with your account. Not the case at some vendors who have departments operating in silos.

Managing customer expectations
There’s nothing I love more than to report a problem or submit a request, get a response within a 2-hour window with a list of “next steps” and when it will be resolved by. Sometimes a customer problem is not a simple, 30-minute fix, either. Even the problems that keep Rackspace scratching their heads over the course of the week are kept up-to-date on a daily basis each week and always have a senior technician checking in on the status each day.

With these other vendors, I am the one checking in with them. Where’s the client service in that?

Anyway, thank you, Rackspace. Always a pleasure to deal with you.

In a previous post, I outlined how we were looking at managed hosting for SuperMotors, to replace the current colocation contract we’ve had for nearly 2 years. After much negotiation with Rackspace, PEER 1, and INetU, none of them were able to come up with a proposal that was attractive enough as ipHouse’s offer. Here’s how things went down:

4th Place: INetU

I really liked their website, the fact that they promoted their customer satisfaction rating (which is really high, at 96%), and they offered FreeBSD (my OS of choice, and where my comfort level is). Unfortunately, their pricing structure is not setup for high-bandwidth environments, as their bid came in much, much higher than the other 3 ISPs who were bidding on our business. Kudos to them for not straying from their core competency, but probably not a viable consideration for us in the future unless they make significant changes to their billing model.

3rd Place: Rackspace

I have real-world experience with Rackspace and their managed hosting, so I knew what to expect in terms of service and support. They did an excellent job in coming down on the costs of their hardware to match PEER 1’s quote, but just were not able to match the bandwidth (they could not go higher than 1000 GB/mo — as this is their standard bandwidth package). They could have gone higher, but at $1/GB, which would have gotten very expensive for us. Had bandwidth not been the primary driver behind our decision, Rackspace might have been our final decision, just based on my past experience with them as a managed hosting provider.

2nd Place: PEER 1

Excellent sales follow-up, paid attention to my blog, and had the best overall price between INetU, Rackspace, and PEER 1. However, in the end, they did not offer FreeBSD for an operating system and required a 12/31/06 commitment date, which would have meant 2 months of overlapped billing with our current ipHouse colocation contract and the new PEER 1 managed contract. Also, switching ISPs would have meant dealing with DNS switches for our customer’s domains, dealing with e-mail account downtime for customers, transferring data over the Internet to the new servers…the list of “cons” went on and on. Granted, this would have been the situation for INetU, Rackspace, or PEER 1…which is why we went with ipHouse:

1st Place: ipHouse

Our current ISP where we’ve colocated for many years now. We’re comfortable with their support, their sales staff, and their network reliability. This, combined with the fact that they will run a managed FreeBSD server for us, matched the PEER 1 bandwidth quote, beat the PEER 1 hardware quote (PEER 1 uses commodity hardware to keep costs down where as ipHouse is getting us new Dell servers). Plus, not having to deal with the IP change, DNS changes, e-mail downtime, etc…everything just made sense for us to stay with the current provider, and for less than we’re paying today. We also bumped up our colocation contract expiration date by 30 days to get started with managed hosting even sooner. Bottom line: no overlap in hosting contracts, faster time-to-production with managed hosting, more bandwidth, better/faster/more reliable servers, less cost.

Moral of the story

• If you’re near the end of your hosting contract, always shop around to see what else is out there. The grass may be greener on the other side.
• Put a competitor’s bid in the hands of your current ISP and see what they can do about meeting or beating the bid.
• Prices are usually negotiable. Never take the first offer and always talk to the rep via phone or e-mail about the quote so they can understand your level of commitment for hosting.
• By renegotiating, you may be able to enter into a new/better contract a month or two sooner than expected — and thus not have to wait for your existing contract to expire, as long as it means your current ISP keeps you as a customer for a longer period of time.
• It’s more expensive for an ISP to obtain a new customer than it is to retain an existing customer for slightly less money.

I deal with vendors on a daily basis, most of which are a pleasant experience. Then there are the others that are either, too pushy and only interested in the sale, and then there are others who cannot close the sale because of a simple question.

Take Peer 1, a managed hosting company who we are strongly considering to have handle SuperMotors.net hosting. Everything checks out on their website - professionally designed, up-to-date information, current press releases. Not much in terms of comments on their managed hosting service online, but that may be because this is still a relatively new market for managed hosting with not a lot of competitors and customers. The price looks good, the company looks good, but when I ask for 2-3 existing customer references (of equal or higher contract terms as we’re considering) from the sales rep I’ve been working with, I get nothing. Last communcation from the rep was on November 13th. It’s now December 5th. I’ve sent follow-up e-mails re-requesting 2-3 customer references on November 14th, November 26th, and December 1st.

I guess e-mails are not good enough and I’ll need to call. It shouldn’t be this hard for us to spend money. Not only is this annoying, but it speaks volumes about their commitment to customer service. Do they even have 2-3 customers that are going to give them top marks? Strike 1 for Peer 1!

In the ongoing battle of optimizing, tweaking, and testing, in November we enabled apache’s mod_deflate module on SuperMotors.net. Literally adding about 10 lines of code to httpd.conf (the apache config file), all text-based content on our pages now use gzip compression when delivered to web browsers that support gzip compression. By using this tool, you can see if your site currently utilizes gzip compression. When testing SuperMotors.net with this tool, our homepage size was originally 49K, but with gzip compression thanks to mod_deflate, it is reduced to 9K. This is an 81% savings in bandwidth usage.

So, when implemented over the course of the entire month, we saw an overall 27% reduction in bandwidth. Surprisingly, our absolute unique visitor count was down 14% for November (holidays do this to our site), but page views actually increased by 1%. Page views are ultimately the driver in bandwidth utilization. I interpret this to mean that fewer users were able to do more on our site because they were downloading content in less amount of time.

So, why does the test tool from above show an 81% reduction in bandwidth yet we only saw an overall reduction of 27%? This is because images (and other files) are not compressed when delivered to web clients. Since we primarily serve images and videos, the rest of our bandwidth still remains largely untouched by mod_deflate — which is by design.

This bandwidth reduction is very good news for us. As previously posted, we are leaving the world of colocation and moving to managed hosting — with a new billing model. The “total bandwidth used” billing model favors us even more now that we’ve reduced bandwidth usage by 27%. This translates into squeezing more data out of the pipe than previously anticipated. We’ll be able to maintain our fixed costs and increase revenue, thanks to this little module. The more data we push out, the more page views we serve, and the more revenue we make from our CPM-based advertising model with Tribal Fusion and our in-house ad inventory (read the challenges we face when we introduce Ajax functionality and how it’ll affect our CPM-based ad model).

The mod_deflate module does increase CPU usage due to the need to compress each page sent out. However, this had very little impact on us as we had plenty of processing power to spare. Your results may vary, so keep an eye on CPU usage when you implement this module. The slight increase in CPU usage was worth the risk, because the 27% reduction in bandwidth was a much bigger gain for us.

12/4/06 Edit: Ajaxian has an article on gzip compression with some user responses. Interesting insights (in the comments section) on the law of diminishing returns with gzip compression used on ajax-enabled pages. 

Since its inception in 1998 as SuperFord.org (originally as a place for friends to post information about their vehicles) to the transition to a database-driven online community (in May 2001 re-launched as SuperMotors), SuperMotors has been running on commodity hardware built and funded by us three owners of the business. We were fortunate enough to have the expertise of web development, server building, and FreeBSD administration shared between the three of us. When we eventually outgrew our commercial-grade cable internet service in the early 2000’s, we entered the wonderful world of colocation.

In our never-ending quest to drive down operational costs of running a hobby-site-turned-online-business, it was critical for us to find an affordable colocation provider in terms of rack rental fees, bandwidth fees, reliability, and 24/7 data center access.

These reasons for not entering into multi-year hosting contracts are not to do any disservice to our current colocation provider. We have been nothing but happy with the experience. Changes in online business, technology, personal, and professional lives of small business owners are sometimes very unexpected, and it’s these changes that we’ve experienced that I wish to share with you in our 5 reasons to avoid multi-year hosting contracts:

1.) The price of bandwidth continues to drop

This may seem obvious, but it is completely true. Since we began colocating, the competitive landscape has increased many times over. Prices, levels of service, and popularity of colocation and managed hosting have changed significantly. We are nearing the end of a 2-year contract and let me tell you, in March of 2005 when we renewed our contract, it was the best deal at the time. But guess what? It’s very expensive compared to what can be had now. We find ourselves today locked into 2005 pricing.
Lesson learned: It may look like a cost-savings at the time to drive down your immediate monthly costs by a few dollars. The money wasted down the road is considerable.

2.) The price of server hardware continues to drop

What seemed like an “investment in the future” in mid-2004 when we built our server, turned out to be anything but. The simple fact is that a server investment is a considerable expense for a small business, especially for a community-driven site that can grow at exponential rates. At the time, 300 GB SATA drives were the biggest, baddest drive you could buy. Today, just over 1 year later, 750 GB drives are coming into the realm of affordability for the common man. We quickly outgrew our server in the course of just 1 year and found ourselves in need to make another significant investment. While still commodity hardware, it does get more expensive to “do it right.”

Bound and determined to again “invest in the future,” we made a significant investment which would later turn out to be a complete disaster. While not really linked to colocation in any way, this type of situation is a potential risk in any business building their own servers and working with the lowest cost vendor. Hindsight is always 20/20; if I were to do it all over again, we would have purchased a Dell PowerEdge server for considerably more money and just called it a day. However, that would not have saved us from #3.

3.) You WILL outgrow your servers

Unless you have a SAN, which is highly unlikely for a small startup, you will outgrow your storage capacity more and more often. For our business, file storage is a critical component…more so than database performance. We can get away with a fairly light-weight database server due to our database structure and server optimization with FreeBSD and MySQL, but there is no magic to file storage. We just need more. And we need more space ALL THE TIME. This creates a very difficult financial model because it truly is exponential growth. The more users that hop on broadband internet connections, the easier it is for them to post more data. The more digital cameras become affordable, the more pictures they post. The more megapixels digital cameras offer, the larger the file size is that gets posted. Today, with the popularity of video and how easily the average consumer can record video and digitize it on their home computer, this creates an even larger demand for storage.

4.) You WILL outgrow your original bandwidth requirements

When we locked into a 2-year contract in 2005, we were locked into specific Mbps 95th-percentile billing. 95th percentile billing is a great way to control your monthly costs, but is an even better way to choke your growth potential. Community-based sites offering photo, audio, and video hosting like SuperMotors are much better suited for total bandwidth transferred. Furthermore, what we found is that in order to increase our bandwidth, this meant committing to the bandwidth through the end of our contract or signing a new contract (and thus extending the original contract).

Due to the nature of 95th percentile billing, we did cap it. An open 100 mbps connection to the internet on a 95th percentile billing model is a recipe for disaster for a site like ours, financially speaking. A user that posts a link to a video on a popular forum could single-handedly increase monthly hosting costs many times over because of the spike in traffic that their shared video could cause. Under total data transferred, this is a much more manageable billing model for our site.

But back to the bandwidth requirements — your requirements will surely increase over time. It’s better to start low and gradually increase your monthly costs than to commit to where you think you’ll be in 2 years just to get the better monthly deal now. You’ll spend more money in the long-run by committing early. While there is no guarantee, the trend definitely says bandwidth will continue to drop — which is another reason to ramp up your bandwidth needs.

5.) These days, managed hosting is completely affordable

Thanks to increased competition, more affordable server hardware, and bandwidth prices dropping, viable companies like Peer1, Rackspace, and INetU have made a business out of providing managed hosting. Through Peer1 managed hosting, we will get 40% more storage space, 4 times the bandwidth (based on our monthly average data transferred), 24×7 managed hosting, a 1-hour hardware replacement Service Level Agreement (SLA), and a 100% network uptime guarantee on the core Peer1 network all for 5% LESS than what we pay for colocation today. This is an incredible change in landscape compared to where we were nearly 2 years ago when we were “planning for the future” with buying and managing our own server hardware and locking into the “affordable” 2-year colocation contract.

Best of all, we no longer have to manage or own any hardware. There are no more sleepless nights wondering if the servers will crash or repairing the servers when they do crash. We can focus 100% on building our online community and sleep peacefully at night knowing we have a solid service level agreement and 24×7 tech support.

We will be committing to a 1-year contract (minimum commitment). At any point in time we can upgrade our servers to support additional storage or processing needs as well as add additional bandwidth on an as-needed basis. This model keeps our operational costs at an all-time low while giving us the flexibility to scale without signficant financial risk (like our CCSI vendor fiasco) and without significant financial/time investment (like purchasing, building, and setting up a server). At the end of 1 year, we can always re-evaluate the competitive landscape to see if it makes sense to move our business elsewhere. Moving an online business from ISP to ISP is certainly no joy-ride, but knowing we have the option puts us in a position of power come time to renegotiate a contract.

In Q4 of 2005, we (SuperMotors) were looking to expand our servers as we were experiencing growing pains and needed a solution that would last us for several years. Previously, we had built really cheap tower-based servers using server-grade components supplied by General Nanosystems of St. Paul, MN. General Nano was great in that is was local to us, we did not need to keep spare computer parts on hand (which meant no need to buy extra hardware we would not immediately use) because they were within a few miles drive of our colocation facility and were open all the time. If a drive failed, a motherboard died, a stick of memory went south, we had replacement parts basically “down the street.” It was a great scenario during our startup phase.

As we began to outgrow the basic tower offerings and were looking for more serious “server grade” components, I began looking online for solutions. Dell was very well-respected in this regard, the hardware replacement policies were outstanding, but they also had the pricetag to go along with them. For 3 guys trying to juggle day jobs and a startup company on the side, we had to be as efficient as possible. So, after much searching online, we felt we had found a reputable source for a custom server at a reasonable price: CCSI (aka Opertonics aka RackMountEquipment.com).

In hindsight, what a horrible, horrible decision it was.

The sales support process was very disappointing. We made the mistake of paying in FULL before ever receiving a thing. Granted, this is how most companies work, but considering they were unknown, we should have negotiated some other terms. With that said, all I can really do is post the timeline of events that would transpire over 8 months until we demanded a refund for a server that never worked (these are my notes to CCSI in an e-mail to help explain the position they had put us in):

1. 11/2/05 check mailed for original purchase of machine for $XXXX.XX (cost hidden to protect us ). 3 drives purchased separately by SuperMotors.
2. 11/9/05 (check received), told after Thanksgiving, possibly first week of December to ship
3. 12/6/05, system built, but issues with motherboard
4. 12/14/05, changing out motherboard, but no bench space to do it. Expect to ship on 12/16, NLT 12/19
5. 1/9/06, Server ships, tracking number provided 60 days after check received.
6. Due to my moving across country on 1/3/06, I could not setup machine. This would be delayed and we would incur setup charges by our ISP to configure the server due to me no longer being in the same city/state as our colocation facility. We incur 3 hours of setup time @ $120/hr only to find that there is a hardware issue. Purchased a new 3WARE 9550SX card. Still will not run reliably.
7. Shipped to CCSI for delivery on Wed, 6/21. You said you would get right on it upon receiving.
8. 6/21, server arrives. Repairs not started as promised via e-mail.
9. 6/28, no update yet.
10. 7/5, Bad memory. Replaced.
11. 7/13, hard drive drops offline. Don’t know why. More troubleshooting necessary.
12. 7/20/06, lost a drive. Looks like the controller is making contact with the heat sink on the southbridge chip on the motherboard. We’re making adjustments (trimming the heat sink) and maybe an insulating shield and will try again. Hopefully more results tomorrow.

We did end up getting a refund for about 17% of what our initial investment was. After much hemming and hawing, they gave into our demands after repeated e-mails from me. It was a morbidly hilarious chain of events, I must say. I can’t imagine treating a customer like we were treated. Incompetence and unwillingness to work with us on all sides of the sale, but thankfully we got some cash back, albeit 17%.

So, here we sit with a pile of hardware that doesn’t amount to a complete server and 17% of our investment back in our hands. They claimed to have never issued a refund before, so I’m happy to say we were the first. I am posting this because I did not find any other reviews on the company, nor do they have a Better Business Bureau record, and I want others who are researching CCSI to be aware of their tactics. At the time, we were a small firm looking to maximize our limited budget, and we ended up losing out rather significantly on the deal. If you would like any other information on this company, do not hesitate to contact me. I have a complete chain of e-mails that documents the entire ordeal.

Goodbye colocation
We (SuperMotors) are nearing the end of our 2-year co-location agreement with ipHouse. Their service has been great, and it was nice to have our servers for SuperMotors located within walking distance of my old job in downtown Minneapolis. However, I’ve sinced relocated to North Carolina, so dealing with broken hardware or a misbehaving server is a little hard when the server is now an 18-hour drive (or 3-hour flight) away.

We have the wrong billing model
Additionally, we are realizing that the 95th-percentile billing model is very, very wrong for our type of traffic and our type of online business. Total data transferred is by far the better of the two from a cost and performance standpoint. We are billed on 95th percentile based on ipHouse’s billing model, not because they are evil in any way. We do not fault them at all for their billing methodology — you will find a mix of billing models with many hosting companies.

Searching for managed hosting
But, all is not lost. We are nearing the end of our contract (3/1/07) at the time of this writing, so I am doing my due-diligence in finding a reliable managed hosting provider for us. While we have some decent hardware that we will really have no use for after we switch from colocation to managed hosting, in the long run, and managed solution is better for us anyway. With me living on the east coast, Kirk and Elliot back in the midwest, it makes it hard to manage our own servers. Plus, Greensboro, NC isn’t really a hotbed of technology, so colocation options are few and far between…and a little on the pricey side. Neighboring cities such as Raleigh and Charlotte have viable options, but again, the convenience aspect is not there as those are both 60-minute and 90-minute drives, respectively. Not the type of response time to a server problem that we want to deal with. Nor is it convenient to drive that far if there’s a hardware problem.

So, managed hosting enters the picture. Wow, what a decision to make. We are essentially entrusting our entire business into the reliablity of another company. Previously, we at least owned our own equipment and could at any time take said equipment and put it somewhere else. Granted, we were in a 2-year contract with ipHouse, but it was always a nice safeguard to be able to just pull your equipment and essentially your data away from one place and put it at another place.

The Players
In my quest, I have found 3 managed hosting providers that I would consider trustworthy and reliable for the size of our online business. In no particular order, they are:

1. Rackspace
2. INetU
3. PEER 1

My delimma is a little more complex in that I’m used to managing our own servers and hardware, so it’s difficult giving up this level of control. Fortunately all 3 providers above provide full root access to the server. However, being the FreeBSD guy that I am, only INetU offers FreeBSD hosting among the 3. Rackspace and Peer 1 both offer Windows Server (heh) and Red Hat Linux Enterprise.

FreeBSD guy scared of Red Hat Linux Enterprise (yes, I’m not afraid to admit it)
Knowing very little about Red Hat Linux Enterprise is a concern to me. I’ve had experience working with it at Levolor as we have a load balanced and fully managed environment at Rackspace for Levolor.com. The support has been phenomenal, to say the least. INetU is significantly more expensive, but offers FreeBSD. Peer 1 I have no experience with, but has the best pricing, by leaps and bounds.

Decision to be made by 12/31/06
I will continue to post my findings as I work with sales reps from all three companies and compare pricing, services, and good ol’ fashioned “gut feel” as I work with each company. In the end, we will be comitting to a 1 or 2-year contract with whatever company we work with. As of this writing, Peer 1 is the most attractive deal, Rackspace a close second, and INetU a distant third. My decisions could easily be swayed in the next month as we aim to lock into a contract between the 12/31/06 and 1/31/07 timeframe.