Update 11/9/06: I just realized that pen handles servers in the order they’re listed on the command line. This means my code can cause all the Mongrel processes on a single server to be used while the other server is idle. I’ve updated the code to interweave the various hosts together, so that the load is even distributed, even under less than near-maximum loads. So, download again!
(Holy crap! I haven’t posted since August?!? Time to throw down on some technical issues, lest I be consigned to the Dustbin Of History! Lemme get this mad knowledge out of the way, and I’ll let you know what I’ve been up to. Meet you at the bottom!)
A Mongrel cluster walks into a bar and orders a bowl of water…
So. You’ve looking to set up Mongrel cluster, but the idea of compiling Apache 2.2 for each of your machines or moving over to the experimental branch doesn’t sit well with the part of you that doesn’t like hanging out in pickup truck beds on the freeway. Perfectly reasonable. You’ve decided to stick with Apache 2.0 because the nice folks at Debian manage the packages. (Don’t have Apache configured yet? Check this out!)
Now you’ve got your Apache config all lined up (knock on wood), and it’s time to make the decision: what the hell is Apache going to hand requests off to? A single Mongrel process? No. Hah. No. This is a business, not a blog in a bottle. Lighty and FastCGI? Oh hells no. Zed isn’t answering the same questions over and over again just so’s we can punk out and not use his amazingly cool HTTP server. mod_rewrite and some crazy-pantsed text file rotation? Not so much. No, the choice seems to come down to either Pound or Pen. Which to use? Oh, which to use?
Pound: Where stray Mongrels go to die
Pound is nice. Pound is great. Pound smells like fresh popcorn. Pound is not what you should use, and I’ll try to be brief about it. Pound replaces the X_FORWARDED_FOR header on every request, which for its usual use (an HTTP front-end) is great–it means the server cluster can still tell where the requests are coming from. But if it’s behind an Apache configuration, it will change the headers on each request to show that each request is coming from Apache–127.0.0.1. And Rails, even in production mode, will show a debug screen to local requests. This combination means that Rails will treat all requests as local, even if they originated with Evils McHackerson von Hideypants III. Awwwwkward.
Also? This means that your exception notification code will never fire, so not only are you handing out debug information to folks who can cause unhandled exceptions, but unless you’re camping out on your production log, you’ll have no idea that your app is bombing.
Worst of all, there’s no way to disable this functionality short of hacking the source and recompiling. Pound, regardless of what it needs to do, will always change the X_FORWARDED_FOR, which makes it pretty crap at what we need it to do.
Bottom line: Apache + Pound + Mongrel = NO BAD BAD NO NAUGHTY BAD
So what’s left?
Pen: Mongrels don’t have thumbs
Well, there’s Pen. It’s also a load-balancing proxy, but handily it lets us choose whether or not the X_FORWARDED_FOR header gets changed. It’s about as fast as Pound, and about as stable. Its SSL support is still experimental, but that’s OK, since we’re proxying between local services, not serving as an external front-end. (If we were, we’d probably use Pound.)
The only downside is that it doesn’t come handily packaged to run as a service. That’s ok, folks, because we’re programmers. Right?
pen-service: Life’s too short to start services manually
Here you go: pen-service_0.2.tar.gz
Installation
server:~$ wget http://blog.codahale.com/wp-content/pen-service_0.2.tar.gz
server:~$ tar -xzvf pen-service_0.2.tar.gz
server:~$ cd pen-service
server:~$ sudo mkdir /etc/pen
server:~$ sudo mv pen.conf /etc/pen
server:~$ sudo mv pen-service /etc/init.d/pen
server:~$ sudo chmod +x /etc/init.d/pen
server:~$ sudo update-rc.d pen defaults
server:~$ sudo vi /etc/pen/pen.conf
Configuration
A simple three-Mongrel cluster, starting at port 8000 and ending on port 8002. It listens on port 8080, doesn’t keep track of which clients go with which server, and has a PID file in /var/run. Each Mongrel process can take a maximum of one connection at a time, since Rails is not multi-threaded. You can bump this up, but Mongrel will just queue the requests, defeating the purpose of load-balancing.
/etc/pen/pen.conf
hosts:
- 127.0.0.1:
ports: 8000..8002
max_connections: 1
port: 8080
disable_tracking: true
pid_file: /var/run/pen.pid
Full explanation:
# This is YAML.
#
# The backend hosts to be proxied to. Ports can be a single port (8000) or a
# range (8000..8002).
hosts:
- 127.0.0.1:
ports: 8000..8002
max_connections: 1
# Used to send Pen commands.
# Default: none
control_port: 8081
# Attach X_FORWARDED_FOR headers. (Don't.)
# Default: false
proxy_headers: true
# Use poll() instead of select().
# Default: false
use_poll: tue
# Dump any debug data in ASCII format.
# Default: false
debug_ascii: true
# Put servers that don't respond on a blacklist for 5 seconds.
# Default: 0
blacklist_time: 5
# Maximum number of HTTP clients.
# Default: 2048
max_clients: 1024
# Log debug information.
# Default: false
debug: false
# If all servers are down, direct to this host.
emergency_host: backup.railsapp.com:80
# Hash the IP before assigning it to a server.
# Default: false
use_ip_hash: true
# chroot Pen to this directory.
# Default: none
chroot: /var/chroot/pen/
# Log Pen info to this file.
# Default: none
log_file: /var/log/pen.log
# Use blocking sockets. (Don't.)
# Default: false
disable_asynchronous_sockets: false
# Use simple round-robin assignments instead of sticky sessions.
# Default: false
disable_tracking: true
# Don't load balance. (Don't?)
# Default: false
disable_failover: false
# Connection timeout, in seconds.
# Default: 5
timeout: 10
# Run Pen as this user.
# Default: none (whoever runs pen-service, usually root.)
user: www-data
# Log Pen's pid to this file.
# Default: /var/run/pen.pid
pid_file: /var/run/railsapp/cluster1.pid
# Maximum number of simultaneous connections.
# Default: 256
max_connections: 500
# Log HTML-formatted usage statistics to this file.
# Default: none
stats_file: /var/www/stats/pen-stats.html
Ok, more about me
I’ve been working like crazy with an amazing team on an amazing application: Wesabe. Everything you hate about your bank website? We fix. Everything you need to know about your money, past, present, and future? You get it. We’re going live reeeeeaally soon, so check us out!
Also, I’m going to be a panel member for this month’s Ruby Tuesday, the San Francisco Bay Area’s Ruby user group. The topic is “Putting Rails To Work,” and I’ll get to squabble with folks like Josh Susser, with whom I’ve only squabbled bloggo-a-bloggo. I’m pretty sure we’ll have launched by then, so I’ll probably be all strung out on developer post-partum depression, or covered in sweat and soot from putting out fires. Expect to see folding chairs fly, folks. (I can’t help but feel a little out of place, seeing my name next to Mr. Rails Associations, Rabble, Florian frickin’ Weber, and Chris “Err The Blog” Wanstrath. Who wants to bet that someone’ll just point to me and shout: “Hey! Who the hell is that guy!” If that happens, I’ll spin around and point at Josh and yell “Yeah, who the hell are you?!”) I’m sure it’ll be a blast–Josh and I owe each other reconciliatory beers from way back when, and I really want to talk with Rabble about his social activism (little known fact–my degree was in Peace & Conflict Studies, with an emphasis in Nonviolent Social Movements).
(This month’s Ruby Tuesday is at Odeo, which means it’s all full. So, uh, crash it. And bring some beer–Daddy’s gotta speak in public.)
26 comments »