B2B/B2C online strategy and enterprise eBusiness blog

Today, while spending the day in Chicago for an instore marketing tradeshow, I was taking care of business via my BlackBerry. On the cab ride to the airport, an urgent issue came up with the staging server and a bug the systems people discovered. The bug was preventing them from doing any more work. I received the e-mail from them on my BlackBerry and proceeded to chat further with them via Yahoo! IM. I then hopped onto AOL IM to chat with the developer. He was able to fix the bug in the development environment. I then ssh’d into the staging server, pushed the development changes forward to the staging server, and sync’d our multiple staging servers with the latest build of the code. All from the backseat of a cab while on my BlackBerry.

I love this thing. This never would have been possible without it.

For the longest time I’ve wanted a PDA-style cell phone. I still have a relatively standad Nokia cell phone, the 8620, which has the fold-open QWERTY keyboard. This is great for simple text messaging. It can do e-mail, but not very well — it’s a tad on the slow side. What I liked about it was that I could access multiple e-mail accounts. Multiple account access is important to me because I maintain my day job’s e-mail, SuperMotors e-mail, and personal e-mail…and several others related to both jobs. I had held off on getting a PDA-style phone because they’re still a bit pricey, and I wasn’t sure how they would effectively handle all of my accounts.

Yesterday, I finally got setup with a BlackBerry 8700 through my day job. It’s awesome. I spent the evening playing around with it, downloading software and just learning the OS. Surfing the Internet is awesome on this little thing. I just need to figure out some of the firewall issues with our corporate security policy so I can do more things on the Internet with it. Being able to SSH, FTP, read e-mail, view my calendar, and open Word/Excel attachments in the palm of my hand is quite impressive. It’s fast too and has great screen resolution. There’s an option to setup other Internet mail accounts which doesn’t seem to be working yet, but at least I’ve got my day job e-mail sync’d up with our Exchange server.

Charter communications blocks all outgoing traffic on port 25 that is not destined for its SMTP server. This means that if you run an SMTP server on your local network and are trying to send out e-mails to other servers on port 25, it will not work. To bypass this (using qmail), setup a file called /var/qmail/control/smtproutes and in it do the following:

example1.com:smtp.anotherserver.com:2525
This tells qmail to send all e-mails destined for example1.com through smtp.anotherserver.com on port 2525 thus bypassing the port 25 blocking by your ISP.

kern.ipc.somaxconn kern.maxfiles kern.maxfilesperproc kern.maxprocperuid kern.ipc.nmbclusters
These is probably one of the most crucial areas to tune, otherwise your services are brought to their knees by hitting maxproc or maxfile ceilings. For some reason the base install of FreeBSD is not designed for a heavily trafficked server which always makes initital configuration a PITA because it’s never easy or quick. Anyway, here’s what I did:
In /boot/loader.conf (things I added):
kern.ipc.nmbclusters=32768
nmbclusters has to do with tcp traffic. When this is set to low, you may see things like unable to open unix socket, httpd requests could get denied…basically if this limit is reached, services stop working because there’s no more “room” for them to operate within.

In /etc/sysctl.conf (things I added):
vfs.vmiodirenable=1
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=65535
kern.maxprocperuid=32768

vfs.vmiodirenable — recommended by others to be set like above. Do this if you have a high-traffic server.
kern.maxfiles — max files allowed to be open at one time by the kernel.
kern.maxfilesperproc — max files allowed to be open by a process at a time.
maxprocperuid — max processes per user id to be running at a time.

Bandwidth usage became a very sensitive issue as I ventured into running a site that used a lot of bandwidth. Being colocated and basically hooked up to the ISP’s backbone meant that I could potentially get hit with enough traffic to make me go broke in one month alone. This was very scary, so in an effort to enable us to traffic, throttle, or manage bandwidth usage, I found that Dummynet was the best solution. Dummynet was originally setup to allow admins to simulate slower connections by allowing them to set bandwidth limitations on particular services, ports, IPs, network cards, etc. I use it for throttling outgoing bandwidth on a specific IP address. Perfect for my application of hosting a high-traffic webserver with a lot of outbound traffic and very little inbound traffic.

These are the settings that I needed to add to custom kernal to enable dummynet:
options DUMMYNET # (enables dummynet for bandwidth throttling)
options HZ=1000 # (dummynet option highly recommended by man but I’m not really sure what its purpose is)

Dummynet then gets activated by setting up entries in ipfw. Basically, you create a pipe for data to go through, and then a bandwidth rule for that pipe. I added this /etc/rc.firewall so that these were implemented at startup. Like ipfw, you can add/modify/delete these optiosn on-the-fly:
# DummyNET config: limits outgoing bandwidth, doesn’t
# limit incoming bandwidth.
# 2/16/2004
#
# Throttle traffic from: 192.168.1.10 (main site & e-mail)
${fwcmd} add pipe 1 ip from 192.168.1.10 to any
${fwcmd} pipe 1 config bw 1250Kbit/s

# Throttle traffic from: 192.168.1.11 (users & hosted sites)
${fwcmd} add pipe 2 ip from 192.168.1.11 to any
${fwcmd} pipe 2 config bw 512Kbit/s

Basically, what the above does is setup all outgoing bandwidth from IP address 192.168.1.10 to use up to 1,250 kbps or 1.25 Mbps of bandwidth. Incoming bandwidth is not throttled. I setup a much lower bandwidth ceiling for pipe #2 as you can see. These sites are less important that they receive bandwidth and are of much lower usage anyway, so nobody will really even know that they are being throttled. The key was to throttle bandwidth down a lot for secondary sites while keeping the ceiling significantly higher for the main site, but still keep it within reason and not allow it to go way beyond what we could afford.

I love the collective power of online communities. However, sometimes it really works against your best intentions. A recent upgrade to facebook.com, a large community-based site, brought about an internet riot of massive proportions. Scores threatened to leave the service after new features were added that simply were making it easier to do things that were already available on the site.

http://blog.facebook.com/blog.php?post=2208197130

We run into this all the time on SuperMotors. Granted, our userbase is not in the millions like Facebook’s is, but it still hits home when we work months on new improvements and enhanced features that are greeted by luke-warm reactions or feelings of complete hostility towards us. It keeps us honest, though. We never do anything to purposely make our site operate worse, which is what Facebook did, but sometimes the best intentions can be your worst nightmare.

The net/net, I think, is that new features need to be introduced before they are…well, introduced. It is the responsibility of business owners and site owners who are frequented by hundreds, thousands, or millions to manage the expectations of users. Nobody likes to visit their favorite site one day only to have it completely changed the next day without warning. This of course, is easier said than done!

Top marks for iTunes 7. Downloaded it yesterday and am playing with it for the first time this evening. The interface facelift is very nice. I really like the fluid display of album covers as you play different songs from your playlist.

I’m currently downloading “Gone in 60 Seconds” to test out the video capabilities. 1.3 GB to download. I really hope this is burnable to DVD. If so, that is huge. Not sure on the quality, but it will probably be “good enough” for now. The convenience factor is awesome and to be able to watch it anytime in the background on the computer will be very cool. Now I just need the new video ipod and the a new iTV that Apple is slated to roll out in early ‘07.