Today, while spending the day in Chicago for an instore marketing tradeshow, I was taking care of business via my BlackBerry. On the cab ride to the airport, an urgent issue came up with the staging server and a bug the systems people discovered. The bug was preventing them from doing any more work. I received the e-mail from them on my BlackBerry and proceeded to chat further with them via Yahoo! IM. I then hopped onto AOL IM to chat with the developer. He was able to fix the bug in the development environment. I then ssh’d into the staging server, pushed the development changes forward to the staging server, and sync’d our multiple staging servers with the latest build of the code. All from the backseat of a cab while on my BlackBerry.

I love this thing. This never would have been possible without it.

For the longest time I’ve wanted a PDA-style cell phone. I still have a relatively standad Nokia cell phone, the 8620, which has the fold-open QWERTY keyboard. This is great for simple text messaging. It can do e-mail, but not very well — it’s a tad on the slow side. What I liked about it was that I could access multiple e-mail accounts. Multiple account access is important to me because I maintain my day job’s e-mail, SuperMotors e-mail, and personal e-mail…and several others related to both jobs. I had held off on getting a PDA-style phone because they’re still a bit pricey, and I wasn’t sure how they would effectively handle all of my accounts.

Yesterday, I finally got setup with a BlackBerry 8700 through my day job. It’s awesome. I spent the evening playing around with it, downloading software and just learning the OS. Surfing the Internet is awesome on this little thing. I just need to figure out some of the firewall issues with our corporate security policy so I can do more things on the Internet with it. Being able to SSH, FTP, read e-mail, view my calendar, and open Word/Excel attachments in the palm of my hand is quite impressive. It’s fast too and has great screen resolution. There’s an option to setup other Internet mail accounts which doesn’t seem to be working yet, but at least I’ve got my day job e-mail sync’d up with our Exchange server.

Charter communications blocks all outgoing traffic on port 25 that is not destined for its SMTP server. This means that if you run an SMTP server on your local network and are trying to send out e-mails to other servers on port 25, it will not work. To bypass this (using qmail), setup a file called /var/qmail/control/smtproutes and in it do the following:

example1.com:smtp.anotherserver.com:2525
This tells qmail to send all e-mails destined for example1.com through smtp.anotherserver.com on port 2525 thus bypassing the port 25 blocking by your ISP.

kern.ipc.somaxconn kern.maxfiles kern.maxfilesperproc kern.maxprocperuid kern.ipc.nmbclusters
These is probably one of the most crucial areas to tune, otherwise your services are brought to their knees by hitting maxproc or maxfile ceilings. For some reason the base install of FreeBSD is not designed for a heavily trafficked server which always makes initital configuration a PITA because it’s never easy or quick. Anyway, here’s what I did:
In /boot/loader.conf (things I added):
kern.ipc.nmbclusters=32768
nmbclusters has to do with tcp traffic. When this is set to low, you may see things like unable to open unix socket, httpd requests could get denied…basically if this limit is reached, services stop working because there’s no more “room” for them to operate within.

In /etc/sysctl.conf (things I added):
vfs.vmiodirenable=1
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=65535
kern.maxprocperuid=32768

vfs.vmiodirenable — recommended by others to be set like above. Do this if you have a high-traffic server.
kern.maxfiles — max files allowed to be open at one time by the kernel.
kern.maxfilesperproc — max files allowed to be open by a process at a time.
maxprocperuid — max processes per user id to be running at a time.

Bandwidth usage became a very sensitive issue as I ventured into running a site that used a lot of bandwidth. Being colocated and basically hooked up to the ISP’s backbone meant that I could potentially get hit with enough traffic to make me go broke in one month alone. This was very scary, so in an effort to enable us to traffic, throttle, or manage bandwidth usage, I found that Dummynet was the best solution. Dummynet was originally setup to allow admins to simulate slower connections by allowing them to set bandwidth limitations on particular services, ports, IPs, network cards, etc. I use it for throttling outgoing bandwidth on a specific IP address. Perfect for my application of hosting a high-traffic webserver with a lot of outbound traffic and very little inbound traffic.

These are the settings that I needed to add to custom kernal to enable dummynet:
options DUMMYNET # (enables dummynet for bandwidth throttling)
options HZ=1000 # (dummynet option highly recommended by man but I’m not really sure what its purpose is)

Dummynet then gets activated by setting up entries in ipfw. Basically, you create a pipe for data to go through, and then a bandwidth rule for that pipe. I added this /etc/rc.firewall so that these were implemented at startup. Like ipfw, you can add/modify/delete these optiosn on-the-fly:
# DummyNET config: limits outgoing bandwidth, doesn’t
# limit incoming bandwidth.
# 2/16/2004
#
# Throttle traffic from: 192.168.1.10 (main site & e-mail)
${fwcmd} add pipe 1 ip from 192.168.1.10 to any
${fwcmd} pipe 1 config bw 1250Kbit/s

# Throttle traffic from: 192.168.1.11 (users & hosted sites)
${fwcmd} add pipe 2 ip from 192.168.1.11 to any
${fwcmd} pipe 2 config bw 512Kbit/s

Basically, what the above does is setup all outgoing bandwidth from IP address 192.168.1.10 to use up to 1,250 kbps or 1.25 Mbps of bandwidth. Incoming bandwidth is not throttled. I setup a much lower bandwidth ceiling for pipe #2 as you can see. These sites are less important that they receive bandwidth and are of much lower usage anyway, so nobody will really even know that they are being throttled. The key was to throttle bandwidth down a lot for secondary sites while keeping the ceiling significantly higher for the main site, but still keep it within reason and not allow it to go way beyond what we could afford.