codahale.com٭blog

server:~$ wget http://blog.codahale.com/wp-content/pen-service_0.2.tar.gz
server:~$ tar -xzvf pen-service_0.2.tar.gz
server:~$ cd pen-service
server:~$ sudo mkdir /etc/pen
server:~$ sudo mv pen.conf /etc/pen
server:~$ sudo mv pen-service /etc/init.d/pen
server:~$ sudo chmod +x /etc/init.d/pen
server:~$ sudo update-rc.d pen defaults
server:~$ sudo vi /etc/pen/pen.conf

hosts:
- 127.0.0.1:
    ports: 8000..8002
    max_connections: 1
port: 8080
disable_tracking: true
pid_file: /var/run/pen.pid

# This is YAML.
#
# The backend hosts to be proxied to. Ports can be a single port (8000) or a
# range (8000..8002).
hosts:
- 127.0.0.1:
    ports: 8000..8002
    max_connections: 1

# Used to send Pen commands.
# Default: none
control_port: 8081

# Attach X_FORWARDED_FOR headers. (Don't.)
# Default: false
proxy_headers: true

# Use poll() instead of select().
# Default: false
use_poll: tue

# Dump any debug data in ASCII format.
# Default: false
debug_ascii: true

# Put servers that don't respond on a blacklist for 5 seconds.
# Default: 0
blacklist_time: 5

# Maximum number of HTTP clients.
# Default: 2048
max_clients: 1024

# Log debug information.
# Default: false
debug: false

# If all servers are down, direct to this host.
emergency_host: backup.railsapp.com:80

# Hash the IP before assigning it to a server.
# Default: false
use_ip_hash: true

# chroot Pen to this directory.
# Default: none
chroot: /var/chroot/pen/

# Log Pen info to this file.
# Default: none
log_file: /var/log/pen.log

# Use blocking sockets. (Don't.)
# Default: false
disable_asynchronous_sockets: false

# Use simple round-robin assignments instead of sticky sessions.
# Default: false
disable_tracking: true

# Don't load balance. (Don't?)
# Default: false
disable_failover: false

# Connection timeout, in seconds.
# Default: 5
timeout: 10

# Run Pen as this user.
# Default: none (whoever runs pen-service, usually root.)
user: www-data

# Log Pen's pid to this file.
# Default: /var/run/pen.pid
pid_file: /var/run/railsapp/cluster1.pid

# Maximum number of simultaneous connections.
# Default: 256
max_connections: 500

# Log HTML-formatted usage statistics to this file.
# Default: none
stats_file: /var/www/stats/pen-stats.html

26 Responses to “Pound vs. Pen: Because You Need A Load-Balancing Proxy”

1. Daniel Bristot Says:
   November 7th, 2006 at 5:48am

   Pacaket Filter + Free/Open|BSD + CARP

2. Coda Says:
   November 7th, 2006 at 9:55am

   Daniel, I like pf as much as the next guy, but I’m not switching to FreeBSD/OpenBSD just so’s I can proxy requests from Apache to localhost and have pf round-robin that to the app servers. The point of having Apache is to have it serve up static files quickly and efficiently while passing on dynamic requests to the Mongrel cluster. CARP doesn’t come into the picture at all–if I wanted to spend money on more hardware, I’d buy a hardware load-balancer. If I wanted a hot failover option, I’d buy two and use VRRP.

3. Ezra Says:
   November 7th, 2006 at 11:35pm

   Man It’s all about nginx. No need for anything besides nginx+mongrel. nginx is running on many many nodes balancing to many many more mongrels on our servers right now. Nginx is probably the most solid and easiest to manage part of our stack. It just works with simple setup and does everything you need from a rails front end. Really can’t say enough good things about nginx.

4. Coda Says:
   November 8th, 2006 at 12:09am

   Ezra–If it were my own box, I’d totally be down for trying out some Russian Craziness. I hear good things from people I trust, you included. As it is, the system I developed this for is protecting bank data, among other things, and I’m totally not interesting in using something I have to hand-compile. When nginx makes it into Debian Stable, I’m all over it. Until then, I’ll deal with Pen.

5. court3nay Says:
   November 10th, 2006 at 11:26am

   Why is mongrel behind apache? You have some complex rewrite rules in there? If possible, you should have pound going to apache OR your mongrels, not to apache -> mongrels.

   http://blog.tupleshop.com/2006/7/8/deploying-rails-with-pound-in-front-of-mongrel-lighttpd-and-apache

   With a simple pound->mongrel and pound->lighttpd setup (where the filename determines whether mongrel or lighty get the request) the IP comes in to rails just fine (the requests don’t go through lighttpd at all if it’s a dynamic request).

   http://mongrel.rubyforge.org/docs/pound.html

6. Coda Says:
   November 10th, 2006 at 12:08pm

   court3nay: Apache is up front for SSL, compression, logs, security, some minor PHP, etc. We serve a lot of static files, and having to go through Pound would be that much more load. As it is, Apache can use sendfile(2) and throw static files as quickly as our machine can handle. In a few months we’ll probably have a hardware load-balancer, so this goes out the window and we’ll probably have a Pound+Mongrel/Apache split.

7. Matt Says:
   December 4th, 2006 at 4:16pm

   Way cool… I’ve been putting pen behind Apache just like this for a while, for the exact same reasons. I’m curious, though… Would you consider it useful to have your script process a directory full of config files instead of just one, like mongrel_cluster does? And then be able to manage them each specifically by specifying a config file?

   I’m thinking that it would be useful because then you could deal with a “pack of pens”, as it were… I run multiple apps off of one machine, so I need to be able to run multiple pens.

   I might take a whack at it and see how it goes…

8. Zak Mandhro Says:
   December 6th, 2006 at 1:13pm

   Has anyone noticed problems with lighttpd and mod_proxy since version 1.4.13? I was using lighttpd and mongrel on Ubuntu Edgy Eft (6.10). I didn’t run into any problems for a week. I even tried httperf with 100% success.

   After reading Zed’s note on lighttpd + mod_proxy, I got concerned that there might be problems that my httperf’ed URL is not covering. So I jumped over to pound and now the whole Rails application seems noticeably slower.

   Has anyone else noticed mod_proxy issues with 1.4.13? Or noticed slower response with Pound?

9. Coda Says:
   December 6th, 2006 at 1:55pm

   Matt–Feel free to hack away. The functionality you’re talking about makes a lot of sense, it’s just not something we’ve needed. I don’t think I said it specifically, but pen-service is under the MIT license, so have at it.

   Zak–Did you use the –rate flag with httperf? That’s the key to overloading your server and seeing where things break.

10. armando Says:
    December 14th, 2006 at 11:32am

    I love your blog! :-)

    Seriously, it’s really comforting to go back to apache-2.0. Down the road, apache-2.2 and the proxy balancer may be the way to go, but keeping with the generally stable apache packaged for various distros keep things quite simple.

    Thanks!

11. Zak Mandhro » Edgy on Rails? Get compiling! Says:
    January 11th, 2007 at 8:59pm

    [...] Pound is a decent standalone load balancer. You can learn more about using Pound with Mongrel here. It is quite configurable and comes standard with Ubuntu. Well, sort of. The Ubuntu Edgy package doesn’t work. In my case, it kept crashing without any warnings. Coda Hale doesn’t like Pound because Pound makes Mongrel think the request came from the local machine. You may want to consider Pen instead. One advantage here is that you can use lighttpd as a web server and proxy and let Pound/Pen do the balancing. If you do want Pound, you will have to compile from source. Rob Orsini’s explains how to setup lighty, Pound and Mongrel here. Long story short, download Pound and untar (tar xfz) it. PLAIN TEXT CODE: [...]

12. Jason Says:
    January 25th, 2007 at 4:35pm

    OK, took me several hours but I found out how to stop rails thinking requests from pound behind apache are from localhost!

    In your environment.rb:

    module ActionController::Rescue
    def local_request?
    false
    end
    end

13. infotage.net » Blog Archive » The poor state of mongrel_cluster frontend web servers Says:
    January 25th, 2007 at 10:24pm

    [...] the proxy requests and that is the proxy solution pen (0.15.0, plus pen-service taken from “Pound vs. Pen“). Pen can correctly be configured to only send 1 request to each mongrel instance and queue [...]

14. al Says:
    January 29th, 2007 at 2:28pm

    Hi,

    have you looked into haproxy (http://haproxy.1wt.eu/) as a 3 lb?

    BR

    Aleks

15. grant Says:
    January 31st, 2007 at 8:27am

    hmm.

    - doesn’t mongrel use sendfile as well, now, for statics?

    - bank data? yeah, i’d wait for a vendor’s binary release too. liability issues?

    - nginx has a smaller footprint and consumes less on the fly {than pound} due to not using a thread-per-connection model.

16. Angry, Manchester Says:
    February 1st, 2007 at 5:32am

    mod_rewrite and some crazy-pantsed text file rotation?

    What exactly is wrong with this method?

17. Pete Says:
    March 18th, 2007 at 3:05pm

    I’ve been working to get a soon-to-be production server up and running, and this has been very helpful, thanks! Apache 1.3 -> pen -> mongrels looks like it’s going to work quite well for us.

    I did run into one issue. I was doing some load testing to see how things held up, and when handing lots of requests I was getting proxy errors from apache. The culprit was pen, it was dropping connections when trying to handle too many because it had been told that each mongrel only wanted 1 connection at a time. So, max_connections: 1 looks like a bad idea (unless there’s some way to configure pen to queue the connections?).

    After bumping it up, the problem went away. One could also leave it out, but the pen-service script above would need to be tweaked to leave it out as well (it defaults to 1 if empty).

    Thanks again,
    -pete

18. Wishlisting Blog » Blog Archive » Deploying Ruby on Rails, Mongrel, and Pen on a MediaTemple (dv) 3.0 Server Says:
    March 19th, 2007 at 12:47pm

    [...] Because this server comes with Apache pre-installed, and it’s not 2.2 (which supports mod_proxy_balancer), I decided to use the Pen load balancer. (Coda Hale sold me on this) [...]

19. Pete Says:
    March 26th, 2007 at 8:28am

    Answering my own question here, in case anyone else runs into the same issues I did. I asked above

    …unless there’s some way to configure pen to queue the connections?

    There is! If pen is configured to only accept the same number of connections as there are mongrel servers, the pen will do the right thing: queue extra connections instead of dropping them, and hand them to the first free mongrel.

    This is better than bumping up the number of connections allowed to each mongrel because that approach can cause pen to add connections to a mongrel that is currently running a slow task, rather then waiting for the first mongrel to free up.

    If invoking pen directly, add -x to the command line. If using Coda’s startup script, set max_connections: (this is the root level max_connections option, not the one under the hosts config).

20. Pete Says:
    March 26th, 2007 at 8:31am

    Oops, bad markup. Last paragraph again:

    If invoking pen directly, add -x [number-of-mongrels] to the command line. If using Coda’s startup script, set max_connections: [number-of-mongrels] (this is the root level max_connections option, not the one under the hosts config).

21. Jason Says:
    September 21st, 2007 at 7:48am

    Great article!

    Anyone interested in the “pack of pens” concept (to use Coda’s script with multiple config files/pen instances) and doesnt feel like coding it themselves check out http://code.google.com/p/packofpens/

22. Jason Frankovitz Says:
    November 25th, 2007 at 7:12pm

    I’m interested in using the pen startup script, but when I run it I get this error:

    
me@foo (2499) $ sudo /etc/init.d/pen start
Password:
/etc/init.d/pen:52:in `to_s': undefined method `each' for -2:Fixnum (NoMethodError)
from /etc/init.d/pen:48:in `to_s'
from /etc/init.d/pen:90:in `start'
from /etc/init.d/pen:113


    The lines it’s complaining about are these:

    
me@foo (2503) $ cat -n /etc/init.d/pen | egrep "48|52|90|113"
48 @hosts = @options['hosts'].map do |host_options|
52 for port in eval(host_options.values.first['ports'])
90 system “pen #{@options}”
113 Pen.send(ARGV.map{ |x| x.to_s.downcase.intern }.first || :usage)


    I’m using the pen.conf from above. Any idea what’s going wrong here? Thanks for any tips!

23. Tomasz N Says:
    January 23rd, 2008 at 9:33am

    Just from brief looking it looks you dropped `pound` because of 1 reason: it changes X-Forwarded-For?
    To lazy to hack source and recompile? or some other-not-mentioned reasons?

24. Ivo Taseff Says:
    April 15th, 2008 at 12:23am

    Hi, I`m using pound and I can see the real source of the requests coming to the apache server. All you need is that module compiled and loaded in apache.

    LoadModule rpaf_module modules/mod_rpaf-2.0.so

25. Gavin Conway Says:
    May 12th, 2008 at 10:47am

    You write articles like you’re talking to a friend? Be more concise!

26. johny b bad Says:
    May 30th, 2008 at 3:10pm

    like talking to a retarded friend.

    Thank you though, as painful as it is to read, insightful, useful info.

Copyright © 2005-2006 Coda Hale. All content on this website (including text, photographs, audio files, and any other original works),