B2B/B2C online strategy and enterprise eBusiness blog

In a previous post, I outlined how we were looking at managed hosting for SuperMotors, to replace the current colocation contract we’ve had for nearly 2 years. After much negotiation with Rackspace, PEER 1, and INetU, none of them were able to come up with a proposal that was attractive enough as ipHouse’s offer. Here’s how things went down:

4th Place: INetU

I really liked their website, the fact that they promoted their customer satisfaction rating (which is really high, at 96%), and they offered FreeBSD (my OS of choice, and where my comfort level is). Unfortunately, their pricing structure is not setup for high-bandwidth environments, as their bid came in much, much higher than the other 3 ISPs who were bidding on our business. Kudos to them for not straying from their core competency, but probably not a viable consideration for us in the future unless they make significant changes to their billing model.

3rd Place: Rackspace

I have real-world experience with Rackspace and their managed hosting, so I knew what to expect in terms of service and support. They did an excellent job in coming down on the costs of their hardware to match PEER 1’s quote, but just were not able to match the bandwidth (they could not go higher than 1000 GB/mo — as this is their standard bandwidth package). They could have gone higher, but at $1/GB, which would have gotten very expensive for us. Had bandwidth not been the primary driver behind our decision, Rackspace might have been our final decision, just based on my past experience with them as a managed hosting provider.

2nd Place: PEER 1

Excellent sales follow-up, paid attention to my blog, and had the best overall price between INetU, Rackspace, and PEER 1. However, in the end, they did not offer FreeBSD for an operating system and required a 12/31/06 commitment date, which would have meant 2 months of overlapped billing with our current ipHouse colocation contract and the new PEER 1 managed contract. Also, switching ISPs would have meant dealing with DNS switches for our customer’s domains, dealing with e-mail account downtime for customers, transferring data over the Internet to the new servers…the list of “cons” went on and on. Granted, this would have been the situation for INetU, Rackspace, or PEER 1…which is why we went with ipHouse:

1st Place: ipHouse

Our current ISP where we’ve colocated for many years now. We’re comfortable with their support, their sales staff, and their network reliability. This, combined with the fact that they will run a managed FreeBSD server for us, matched the PEER 1 bandwidth quote, beat the PEER 1 hardware quote (PEER 1 uses commodity hardware to keep costs down where as ipHouse is getting us new Dell servers). Plus, not having to deal with the IP change, DNS changes, e-mail downtime, etc…everything just made sense for us to stay with the current provider, and for less than we’re paying today. We also bumped up our colocation contract expiration date by 30 days to get started with managed hosting even sooner. Bottom line: no overlap in hosting contracts, faster time-to-production with managed hosting, more bandwidth, better/faster/more reliable servers, less cost.

Moral of the story

• If you’re near the end of your hosting contract, always shop around to see what else is out there. The grass may be greener on the other side.
• Put a competitor’s bid in the hands of your current ISP and see what they can do about meeting or beating the bid.
• Prices are usually negotiable. Never take the first offer and always talk to the rep via phone or e-mail about the quote so they can understand your level of commitment for hosting.
• By renegotiating, you may be able to enter into a new/better contract a month or two sooner than expected — and thus not have to wait for your existing contract to expire, as long as it means your current ISP keeps you as a customer for a longer period of time.
• It’s more expensive for an ISP to obtain a new customer than it is to retain an existing customer for slightly less money.

Since its inception in 1998 as SuperFord.org (originally as a place for friends to post information about their vehicles) to the transition to a database-driven online community (in May 2001 re-launched as SuperMotors), SuperMotors has been running on commodity hardware built and funded by us three owners of the business. We were fortunate enough to have the expertise of web development, server building, and FreeBSD administration shared between the three of us. When we eventually outgrew our commercial-grade cable internet service in the early 2000’s, we entered the wonderful world of colocation.

In our never-ending quest to drive down operational costs of running a hobby-site-turned-online-business, it was critical for us to find an affordable colocation provider in terms of rack rental fees, bandwidth fees, reliability, and 24/7 data center access.

These reasons for not entering into multi-year hosting contracts are not to do any disservice to our current colocation provider. We have been nothing but happy with the experience. Changes in online business, technology, personal, and professional lives of small business owners are sometimes very unexpected, and it’s these changes that we’ve experienced that I wish to share with you in our 5 reasons to avoid multi-year hosting contracts:

1.) The price of bandwidth continues to drop

This may seem obvious, but it is completely true. Since we began colocating, the competitive landscape has increased many times over. Prices, levels of service, and popularity of colocation and managed hosting have changed significantly. We are nearing the end of a 2-year contract and let me tell you, in March of 2005 when we renewed our contract, it was the best deal at the time. But guess what? It’s very expensive compared to what can be had now. We find ourselves today locked into 2005 pricing.
Lesson learned: It may look like a cost-savings at the time to drive down your immediate monthly costs by a few dollars. The money wasted down the road is considerable.

2.) The price of server hardware continues to drop

What seemed like an “investment in the future” in mid-2004 when we built our server, turned out to be anything but. The simple fact is that a server investment is a considerable expense for a small business, especially for a community-driven site that can grow at exponential rates. At the time, 300 GB SATA drives were the biggest, baddest drive you could buy. Today, just over 1 year later, 750 GB drives are coming into the realm of affordability for the common man. We quickly outgrew our server in the course of just 1 year and found ourselves in need to make another significant investment. While still commodity hardware, it does get more expensive to “do it right.”

Bound and determined to again “invest in the future,” we made a significant investment which would later turn out to be a complete disaster. While not really linked to colocation in any way, this type of situation is a potential risk in any business building their own servers and working with the lowest cost vendor. Hindsight is always 20/20; if I were to do it all over again, we would have purchased a Dell PowerEdge server for considerably more money and just called it a day. However, that would not have saved us from #3.

3.) You WILL outgrow your servers

Unless you have a SAN, which is highly unlikely for a small startup, you will outgrow your storage capacity more and more often. For our business, file storage is a critical component…more so than database performance. We can get away with a fairly light-weight database server due to our database structure and server optimization with FreeBSD and MySQL, but there is no magic to file storage. We just need more. And we need more space ALL THE TIME. This creates a very difficult financial model because it truly is exponential growth. The more users that hop on broadband internet connections, the easier it is for them to post more data. The more digital cameras become affordable, the more pictures they post. The more megapixels digital cameras offer, the larger the file size is that gets posted. Today, with the popularity of video and how easily the average consumer can record video and digitize it on their home computer, this creates an even larger demand for storage.

4.) You WILL outgrow your original bandwidth requirements

When we locked into a 2-year contract in 2005, we were locked into specific Mbps 95th-percentile billing. 95th percentile billing is a great way to control your monthly costs, but is an even better way to choke your growth potential. Community-based sites offering photo, audio, and video hosting like SuperMotors are much better suited for total bandwidth transferred. Furthermore, what we found is that in order to increase our bandwidth, this meant committing to the bandwidth through the end of our contract or signing a new contract (and thus extending the original contract).

Due to the nature of 95th percentile billing, we did cap it. An open 100 mbps connection to the internet on a 95th percentile billing model is a recipe for disaster for a site like ours, financially speaking. A user that posts a link to a video on a popular forum could single-handedly increase monthly hosting costs many times over because of the spike in traffic that their shared video could cause. Under total data transferred, this is a much more manageable billing model for our site.

But back to the bandwidth requirements — your requirements will surely increase over time. It’s better to start low and gradually increase your monthly costs than to commit to where you think you’ll be in 2 years just to get the better monthly deal now. You’ll spend more money in the long-run by committing early. While there is no guarantee, the trend definitely says bandwidth will continue to drop — which is another reason to ramp up your bandwidth needs.

5.) These days, managed hosting is completely affordable

Thanks to increased competition, more affordable server hardware, and bandwidth prices dropping, viable companies like Peer1, Rackspace, and INetU have made a business out of providing managed hosting. Through Peer1 managed hosting, we will get 40% more storage space, 4 times the bandwidth (based on our monthly average data transferred), 24×7 managed hosting, a 1-hour hardware replacement Service Level Agreement (SLA), and a 100% network uptime guarantee on the core Peer1 network all for 5% LESS than what we pay for colocation today. This is an incredible change in landscape compared to where we were nearly 2 years ago when we were “planning for the future” with buying and managing our own server hardware and locking into the “affordable” 2-year colocation contract.

Best of all, we no longer have to manage or own any hardware. There are no more sleepless nights wondering if the servers will crash or repairing the servers when they do crash. We can focus 100% on building our online community and sleep peacefully at night knowing we have a solid service level agreement and 24×7 tech support.

We will be committing to a 1-year contract (minimum commitment). At any point in time we can upgrade our servers to support additional storage or processing needs as well as add additional bandwidth on an as-needed basis. This model keeps our operational costs at an all-time low while giving us the flexibility to scale without signficant financial risk (like our CCSI vendor fiasco) and without significant financial/time investment (like purchasing, building, and setting up a server). At the end of 1 year, we can always re-evaluate the competitive landscape to see if it makes sense to move our business elsewhere. Moving an online business from ISP to ISP is certainly no joy-ride, but knowing we have the option puts us in a position of power come time to renegotiate a contract.

Goodbye colocation
We (SuperMotors) are nearing the end of our 2-year co-location agreement with ipHouse. Their service has been great, and it was nice to have our servers for SuperMotors located within walking distance of my old job in downtown Minneapolis. However, I’ve sinced relocated to North Carolina, so dealing with broken hardware or a misbehaving server is a little hard when the server is now an 18-hour drive (or 3-hour flight) away.

We have the wrong billing model
Additionally, we are realizing that the 95th-percentile billing model is very, very wrong for our type of traffic and our type of online business. Total data transferred is by far the better of the two from a cost and performance standpoint. We are billed on 95th percentile based on ipHouse’s billing model, not because they are evil in any way. We do not fault them at all for their billing methodology — you will find a mix of billing models with many hosting companies.

Searching for managed hosting
But, all is not lost. We are nearing the end of our contract (3/1/07) at the time of this writing, so I am doing my due-diligence in finding a reliable managed hosting provider for us. While we have some decent hardware that we will really have no use for after we switch from colocation to managed hosting, in the long run, and managed solution is better for us anyway. With me living on the east coast, Kirk and Elliot back in the midwest, it makes it hard to manage our own servers. Plus, Greensboro, NC isn’t really a hotbed of technology, so colocation options are few and far between…and a little on the pricey side. Neighboring cities such as Raleigh and Charlotte have viable options, but again, the convenience aspect is not there as those are both 60-minute and 90-minute drives, respectively. Not the type of response time to a server problem that we want to deal with. Nor is it convenient to drive that far if there’s a hardware problem.

So, managed hosting enters the picture. Wow, what a decision to make. We are essentially entrusting our entire business into the reliablity of another company. Previously, we at least owned our own equipment and could at any time take said equipment and put it somewhere else. Granted, we were in a 2-year contract with ipHouse, but it was always a nice safeguard to be able to just pull your equipment and essentially your data away from one place and put it at another place.

The Players
In my quest, I have found 3 managed hosting providers that I would consider trustworthy and reliable for the size of our online business. In no particular order, they are:

1. Rackspace
2. INetU
3. PEER 1

My delimma is a little more complex in that I’m used to managing our own servers and hardware, so it’s difficult giving up this level of control. Fortunately all 3 providers above provide full root access to the server. However, being the FreeBSD guy that I am, only INetU offers FreeBSD hosting among the 3. Rackspace and Peer 1 both offer Windows Server (heh) and Red Hat Linux Enterprise.

FreeBSD guy scared of Red Hat Linux Enterprise (yes, I’m not afraid to admit it)
Knowing very little about Red Hat Linux Enterprise is a concern to me. I’ve had experience working with it at Levolor as we have a load balanced and fully managed environment at Rackspace for Levolor.com. The support has been phenomenal, to say the least. INetU is significantly more expensive, but offers FreeBSD. Peer 1 I have no experience with, but has the best pricing, by leaps and bounds.

Decision to be made by 12/31/06
I will continue to post my findings as I work with sales reps from all three companies and compare pricing, services, and good ol’ fashioned “gut feel” as I work with each company. In the end, we will be comitting to a 1 or 2-year contract with whatever company we work with. As of this writing, Peer 1 is the most attractive deal, Rackspace a close second, and INetU a distant third. My decisions could easily be swayed in the next month as we aim to lock into a contract between the 12/31/06 and 1/31/07 timeframe.

Charter communications blocks all outgoing traffic on port 25 that is not destined for its SMTP server. This means that if you run an SMTP server on your local network and are trying to send out e-mails to other servers on port 25, it will not work. To bypass this (using qmail), setup a file called /var/qmail/control/smtproutes and in it do the following:

example1.com:smtp.anotherserver.com:2525
This tells qmail to send all e-mails destined for example1.com through smtp.anotherserver.com on port 2525 thus bypassing the port 25 blocking by your ISP.

kern.ipc.somaxconn kern.maxfiles kern.maxfilesperproc kern.maxprocperuid kern.ipc.nmbclusters
These is probably one of the most crucial areas to tune, otherwise your services are brought to their knees by hitting maxproc or maxfile ceilings. For some reason the base install of FreeBSD is not designed for a heavily trafficked server which always makes initital configuration a PITA because it’s never easy or quick. Anyway, here’s what I did:
In /boot/loader.conf (things I added):
kern.ipc.nmbclusters=32768
nmbclusters has to do with tcp traffic. When this is set to low, you may see things like unable to open unix socket, httpd requests could get denied…basically if this limit is reached, services stop working because there’s no more “room” for them to operate within.

In /etc/sysctl.conf (things I added):
vfs.vmiodirenable=1
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=65535
kern.maxprocperuid=32768

vfs.vmiodirenable — recommended by others to be set like above. Do this if you have a high-traffic server.
kern.maxfiles — max files allowed to be open at one time by the kernel.
kern.maxfilesperproc — max files allowed to be open by a process at a time.
maxprocperuid — max processes per user id to be running at a time.

Bandwidth usage became a very sensitive issue as I ventured into running a site that used a lot of bandwidth. Being colocated and basically hooked up to the ISP’s backbone meant that I could potentially get hit with enough traffic to make me go broke in one month alone. This was very scary, so in an effort to enable us to traffic, throttle, or manage bandwidth usage, I found that Dummynet was the best solution. Dummynet was originally setup to allow admins to simulate slower connections by allowing them to set bandwidth limitations on particular services, ports, IPs, network cards, etc. I use it for throttling outgoing bandwidth on a specific IP address. Perfect for my application of hosting a high-traffic webserver with a lot of outbound traffic and very little inbound traffic.

These are the settings that I needed to add to custom kernal to enable dummynet:
options DUMMYNET # (enables dummynet for bandwidth throttling)
options HZ=1000 # (dummynet option highly recommended by man but I’m not really sure what its purpose is)

Dummynet then gets activated by setting up entries in ipfw. Basically, you create a pipe for data to go through, and then a bandwidth rule for that pipe. I added this /etc/rc.firewall so that these were implemented at startup. Like ipfw, you can add/modify/delete these optiosn on-the-fly:
# DummyNET config: limits outgoing bandwidth, doesn’t
# limit incoming bandwidth.
# 2/16/2004
#
# Throttle traffic from: 192.168.1.10 (main site & e-mail)
${fwcmd} add pipe 1 ip from 192.168.1.10 to any
${fwcmd} pipe 1 config bw 1250Kbit/s

# Throttle traffic from: 192.168.1.11 (users & hosted sites)
${fwcmd} add pipe 2 ip from 192.168.1.11 to any
${fwcmd} pipe 2 config bw 512Kbit/s

Basically, what the above does is setup all outgoing bandwidth from IP address 192.168.1.10 to use up to 1,250 kbps or 1.25 Mbps of bandwidth. Incoming bandwidth is not throttled. I setup a much lower bandwidth ceiling for pipe #2 as you can see. These sites are less important that they receive bandwidth and are of much lower usage anyway, so nobody will really even know that they are being throttled. The key was to throttle bandwidth down a lot for secondary sites while keeping the ceiling significantly higher for the main site, but still keep it within reason and not allow it to go way beyond what we could afford.