codahale.com٭blog

Pound vs. Pen: Because You Need A Load-Balancing Proxy

Update 11/9/06: I just realized that pen handles servers in the order they’re listed on the command line. This means my code can cause all the Mongrel processes on a single server to be used while the other server is idle. I’ve updated the code to interweave the various hosts together, so that the load is even distributed, even under less than near-maximum loads. So, download again!

(Holy crap! I haven’t posted since August?!? Time to throw down on some technical issues, lest I be consigned to the Dustbin Of History! Lemme get this mad knowledge out of the way, and I’ll let you know what I’ve been up to. Meet you at the bottom!)

A Mongrel cluster walks into a bar and orders a bowl of water…

So. You’ve looking to set up Mongrel cluster, but the idea of compiling Apache 2.2 for each of your machines or moving over to the experimental branch doesn’t sit well with the part of you that doesn’t like hanging out in pickup truck beds on the freeway. Perfectly reasonable. You’ve decided to stick with Apache 2.0 because the nice folks at Debian manage the packages. (Don’t have Apache configured yet? Check this out!)

Now you’ve got your Apache config all lined up (knock on wood), and it’s time to make the decision: what the hell is Apache going to hand requests off to? A single Mongrel process? No. Hah. No. This is a business, not a blog in a bottle. Lighty and FastCGI? Oh hells no. Zed isn’t answering the same questions over and over again just so’s we can punk out and not use his amazingly cool HTTP server. mod_rewrite and some crazy-pantsed text file rotation? Not so much. No, the choice seems to come down to either Pound or Pen. Which to use? Oh, which to use?

Pound: Where stray Mongrels go to die

Pound is nice. Pound is great. Pound smells like fresh popcorn. Pound is not what you should use, and I’ll try to be brief about it. Pound replaces the X_FORWARDED_FOR header on every request, which for its usual use (an HTTP front-end) is great–it means the server cluster can still tell where the requests are coming from. But if it’s behind an Apache configuration, it will change the headers on each request to show that each request is coming from Apache–127.0.0.1. And Rails, even in production mode, will show a debug screen to local requests. This combination means that Rails will treat all requests as local, even if they originated with Evils McHackerson von Hideypants III. Awwwwkward.

Also? This means that your exception notification code will never fire, so not only are you handing out debug information to folks who can cause unhandled exceptions, but unless you’re camping out on your production log, you’ll have no idea that your app is bombing.

Worst of all, there’s no way to disable this functionality short of hacking the source and recompiling. Pound, regardless of what it needs to do, will always change the X_FORWARDED_FOR, which makes it pretty crap at what we need it to do.

Bottom line: Apache + Pound + Mongrel = NO BAD BAD NO NAUGHTY BAD

So what’s left?

Pen: Mongrels don’t have thumbs

Well, there’s Pen. It’s also a load-balancing proxy, but handily it lets us choose whether or not the X_FORWARDED_FOR header gets changed. It’s about as fast as Pound, and about as stable. Its SSL support is still experimental, but that’s OK, since we’re proxying between local services, not serving as an external front-end. (If we were, we’d probably use Pound.)

The only downside is that it doesn’t come handily packaged to run as a service. That’s ok, folks, because we’re programmers. Right?

pen-service: Life’s too short to start services manually

Here you go: pen-service_0.2.tar.gz

Installation

server:~$ wget http://blog.codahale.com/wp-content/pen-service_0.2.tar.gz
server:~$ tar -xzvf pen-service_0.2.tar.gz
server:~$ cd pen-service
server:~$ sudo mkdir /etc/pen
server:~$ sudo mv pen.conf /etc/pen
server:~$ sudo mv pen-service /etc/init.d/pen
server:~$ sudo chmod +x /etc/init.d/pen
server:~$ sudo update-rc.d pen defaults
server:~$ sudo vi /etc/pen/pen.conf

Configuration

A simple three-Mongrel cluster, starting at port 8000 and ending on port 8002. It listens on port 8080, doesn’t keep track of which clients go with which server, and has a PID file in /var/run. Each Mongrel process can take a maximum of one connection at a time, since Rails is not multi-threaded. You can bump this up, but Mongrel will just queue the requests, defeating the purpose of load-balancing.

/etc/pen/pen.conf

hosts:
- 127.0.0.1:
    ports: 8000..8002
    max_connections: 1
port: 8080
disable_tracking: true
pid_file: /var/run/pen.pid

Full explanation:

# This is YAML.
#
# The backend hosts to be proxied to. Ports can be a single port (8000) or a
# range (8000..8002).
hosts:
- 127.0.0.1:
    ports: 8000..8002
    max_connections: 1

# Used to send Pen commands.
# Default: none
control_port: 8081

# Attach X_FORWARDED_FOR headers. (Don't.)
# Default: false
proxy_headers: true

# Use poll() instead of select().
# Default: false
use_poll: tue

# Dump any debug data in ASCII format.
# Default: false
debug_ascii: true

# Put servers that don't respond on a blacklist for 5 seconds.
# Default: 0
blacklist_time: 5

# Maximum number of HTTP clients.
# Default: 2048
max_clients: 1024

# Log debug information.
# Default: false
debug: false

# If all servers are down, direct to this host.
emergency_host: backup.railsapp.com:80

# Hash the IP before assigning it to a server.
# Default: false
use_ip_hash: true

# chroot Pen to this directory.
# Default: none
chroot: /var/chroot/pen/

# Log Pen info to this file.
# Default: none
log_file: /var/log/pen.log

# Use blocking sockets. (Don't.)
# Default: false
disable_asynchronous_sockets: false

# Use simple round-robin assignments instead of sticky sessions.
# Default: false
disable_tracking: true

# Don't load balance. (Don't?)
# Default: false
disable_failover: false

# Connection timeout, in seconds.
# Default: 5
timeout: 10

# Run Pen as this user.
# Default: none (whoever runs pen-service, usually root.)
user: www-data

# Log Pen's pid to this file.
# Default: /var/run/pen.pid
pid_file: /var/run/railsapp/cluster1.pid

# Maximum number of simultaneous connections.
# Default: 256
max_connections: 500

# Log HTML-formatted usage statistics to this file.
# Default: none
stats_file: /var/www/stats/pen-stats.html

Ok, more about me

I’ve been working like crazy with an amazing team on an amazing application: Wesabe. Everything you hate about your bank website? We fix. Everything you need to know about your money, past, present, and future? You get it. We’re going live reeeeeaally soon, so check us out!

Also, I’m going to be a panel member for this month’s Ruby Tuesday, the San Francisco Bay Area’s Ruby user group. The topic is “Putting Rails To Work,” and I’ll get to squabble with folks like Josh Susser, with whom I’ve only squabbled bloggo-a-bloggo. I’m pretty sure we’ll have launched by then, so I’ll probably be all strung out on developer post-partum depression, or covered in sweat and soot from putting out fires. Expect to see folding chairs fly, folks. (I can’t help but feel a little out of place, seeing my name next to Mr. Rails Associations, Rabble, Florian frickin’ Weber, and Chris “Err The Blog” Wanstrath. Who wants to bet that someone’ll just point to me and shout: “Hey! Who the hell is that guy!” If that happens, I’ll spin around and point at Josh and yell “Yeah, who the hell are you?!”) I’m sure it’ll be a blast–Josh and I owe each other reconciliatory beers from way back when, and I really want to talk with Rabble about his social activism (little known fact–my degree was in Peace & Conflict Studies, with an emphasis in Nonviolent Social Movements).

(This month’s Ruby Tuesday is at Odeo, which means it’s all full. So, uh, crash it. And bring some beer–Daddy’s gotta speak in public.)

Stupid Ruby Tricks: String#to_proc

I was thinking about one of the cool bits of ActiveSupport, Symbol#to_proc, which allows you to do this:

[1, 2, 3].map(&:to_s) #=> ["1", "2", "3"]

That’s useful for sorting, enumerating, mapping, etc. Quite handy, and it saves you the trouble of writing a block to return a single value. I was recently someplace with a fair bit of time to kill and no reading material, and I began to wonder… how does this work?

Magic via one of the few typographic characters I can’t even write

Oh, the ampersand. Curly figure of import. That little pretzely freak is the sin qua non for this entire operation. You can tell, because Array#map acts all angry when you try to just pass it a Symbol:

irb(main):001:0> [1, 2, 3].map(:to_s)
ArgumentError: wrong number of arguments (1 for 0)
        from (irb):1:in `map'
        from (irb):1

The ampersand is how Ruby denotes a block variable.

Let’s try to pass our block in a different way:

my_block = lamdba{ |x| x.to_s }
[1, 2, 3].map(my_block)

Ruby will raise an ArgumentError because it’s trying to pass your block as a regular method parameter, rather than as a block parameter. Let’s try passing it as a block parameter:

[1, 2, 3].map(&my_block) #=> ["1", "2", "3"]

Sweet! It worked!

Now we’ve got a better of idea of what the hell we’re actually doing. When we write map(&:to_s), we’re saying “call the map function and pass it this block.” But a Symbol isn’t a block!

Very true, and what does Ruby try to do when presented with a type mismatch? Coerce! How? Conversion methods! Anything with a to_proc method can be converted to a proc. Our first example can be rewritten as follows:

[1, 2, 3].map(&:to_s.to_proc) #=> ["1", "2", "3"]

This means that anything with a to_proc method can partake of this lovely syntactic sugar.

Pushing the limits for no apparent reason

And now for the inevitable section of any article on programming that I write: taking things too far.

class String
  def to_proc
    eval "Proc.new { |*args| args.first#{self} }"
  end
end

robots = []
robots << { :name => 'Jeevesotron',  :type => 'robo-butler' }
robots << { :name => 'Crushmonster', :type => 'robo-baby-sitter' }
robots << { :name => 'Fluffy',       :type => 'doombot' }

robots.map(&'[:name].upcase') #=> ["JEEVESOTRON", "CRUSHMONSTER", "FLUFFY"]

It may not be useful, but it sure as hell is fun.

CRUD Explained

As it’s used on the Rails community, CRUD is when you limit your controller actions to index, new, edit, show, and destroy.

I eagerly await counter-examples.

Singletons: Only The Lonely

I’ve never really spent much time on design patterns, but recently I’ve been reading Head First Design Patterns and I can’t help but notice how much easier Ruby makes certain patterns. Composition over inheritance becomes as easy as sneezing with mixins, and the thoughtful Ruby folks have already implemented quite a few.

I don’t use Singleton as much as I should (preferring instead to just stuff all the methods in the class), but for objects which have an expensive or complicated instantiation method, it’s extremely useful. (I mean, that’s why it’s a pattern, right?) Also, the Singleton mixin takes care of all the threading complications to which Singletons often fall prey.

But effective and useful it may be, Singleton is ugly, and for one big reason: instance. I hate typing instance all over the place, and it’s been a big stumbling block in my acceptance of the Singleton mixin. I understand why it’s there–the pattern doesn’t mention moving methods from instance to class scope–but on a syntactic level it irks me. If I’m using a singleton, I really shouldn’t need to know that it is a singleton–it could be a magic happy fairy princess castle for thread-safe methods, for all I care.

So yeah, I went and did something about it. I combined the power of Singleton and Proxy. Check it out:

module Singleton
  class << self
    def included_with_proxy(base)
      included_without_proxy(base)
      base.class_eval do
        class << self
          def method_missing_with_proxy(m, *args)
            if instance.respond_to?(m)
              instance.send(m, *args)
            else
              method_missing_without_proxy(m, *args)
            end
          end
          alias_method_chain :method_missing, :proxy

          def methods_with_proxy
            return (methods_without_proxy + instance.methods).uniq
          end
          alias_method_chain :methods, :proxy

          def respond_to_with_proxy?(m)
            return respond_to_without_proxy?(m) || instance.respond_to?(m)
          end
          alias_method_chain :respond_to?, :proxy
        end
      end
    end
    alias_method_chain :included, :proxy
  end
end

You’ll need to require singleton and activesupport for this one, and since it uses alias_method_chain, you’ll need to either be running Rails edge or grab it from the source.

What it does is pretty simple: if it’s not already a class method, Singleton passes it on to its instance:

class Magician
  include Singleton

  def dove
    "*poof* A dove!"
  end

  def segway
    "*rides Segway around*"
  end
end

Magician.dove   #=> "*poof* A dove!"
Magician.segway #=> "*rides Segway around*"

No more instance, though it’s there if you need it.

Rails programmer needed!

So I’m going to be leaving Gilsson shortly for a new startup in Berkeley, which means that my old job is now open. If you’re within commuting distance from Hayward, CA and are looking for a Rails/IT position, check this out:

Responsibilities:

• Maintain local network of 5-10 workstations and 1-2 file and database servers, with a mix of Windows XP and Linux.
• Improve and extend our Rails-powered e-commerce codebase, eventually rolling out new versions of two of our online stores.
• Improve and maintain our Rails-powered product management database.
• Maintain our Fedora Core 5 production Rails web server (Apache 2.2 + Mongrel, natch).
• Develop custom applications for in-house use.

Requirements:

• Strong experience with Ruby on Rails, web development and associated technologies, including Javascript, XHTML/CSS, XML, AJAX/AHAH, and RSS.
• Experience with SQL in general, and using and configuring MySQL specifically.
• Experience with e-commerce.
• Commitment to web standards and accessibility.
• Experience with test-driven development.
• Familiarity with Subversion and Trac a plus.
• Ability to be self-directed, complete tasks with minimal management oversight.
• Flexibility and good on-the-fly problem-solving skills.

It’s a great place to work, and Ming (the CEO) is an awesome boss. There’s a lot of flexibility in work hours, and a lot of autonomy in the position–you get problems, you make solutions. (Plus, if you apply, you’ll probably get to meet me.) All of the code is well-documented, and all the important bits have full test coverage–no Daily WTF material here.

If this sounds like your kind of gig, email your resume to ming@gilsson.com with the subject line “IT Manager”. If you have any questions about the job, email me and I’ll do my best to answer them.

Regarding the new job, I’m going to be doing some Rails programming with a small team of people who I really admire, including the author of one of the books on my desk right now. I guess that’s one book I can leave for the new person… ;-)