My first patch to Rails was accepted today. I was bored yesterday, and talking with Hampton, who was working on a really cool patch himself (which I’m sure he will explain in good time). We both noticed that four or five of the unit tests for ActiveRecord::Calculations were bombing out, and naturally did what Rails hackers do:
Hampton: Weird…. a fresh checkout is causing test errors…
Coda: On calculations?
Hampton: Something broke between 4485 and 4491
Hampton: However, the last to change to AR happened at 4484
Coda: Yeah, it’s broken. I’m looking into why.
Hampton: RACE!
81 seconds later…
Hampton: Found it!
Hampton totally schooled me. I had just gotten my debug code up and running when, just 81 seconds after the green light, Hampton found the problem. That’s why he gets mentioned in keynote addresses, I guess.
Yes, the newly added Enumerable#sum was mucking up ActiveRecord::AssociationProxy’s ability to pass off the sum method to ActiveRecord::Calculations#sum. Fixing it, however, was a hell of a lot harder, since neither of us had played around with ActiveRecord::AssociationProxy. Thankfully, I figured out that sum needed to be added to ActiveRecord::AssociationCollection, which is what’s actually used when ActiveRecord::Base#find returns an array of records instead of Array. I made the fix, verified that it passed all the unit tests, and submitted it. Nicholas Seckar commited the patch today.
So I finally got to give back to Rails, even if it was just a four or five lines of code. That makes me really happy. Rails is an open-source project, and it’s important that the community give back to it as much as possible. The greatest gift you can give a project like Rails is a patch for an outstanding ticket which conforms to the style guidelines, passes existing unit tests, and has new unit tests for any new code.
I hope they get around to my patch for eagerly-loaded associations not respecting the :order option. That’d be awesome.
I’ve started getting interested in Ruby Quiz as a way to keep my Ruby-fu sharp, and I think I’m gonna make it a regular feature here.
Today’s quiz question revolves around Pascal’s Triangle.
Here’s my answer, after the fold.
2 comments »More and more Rails developers are finding out that deploying a Rails application isn’t as simple as upload and rename; Rails apps work best when running all the time, and many Rails programmers are moving from traditional, shared hosts, like Dreamhost, to virtual private servers, like Rimuhosting, which allow them full control (and responsibility) of production servers.
Given this freedom, there are now a huge number of options available for deploying a Rails application, which is rare for such a new technology. So what to use? Apache + FastCGI? Lighttpd + FastCGI? SCGI? Apache 2? Litespeed? Mongrel? In what configuration? So many questions, so few answers. Here’s what I’ve been deleriously happy with, and how to do it yourself: Mongrel, Apache 2.2, and mod_proxy_balancer, using Capistrano to keep everything at your fingertips.
Mongrel is the hot new kid on the block. It’s a hybrid Ruby/C HTTP server designed to be small, fast, and brutally secure. It’s written by Zed Shaw, who is easily one of the most entertaining and delightfully cranky programmers I’ve ever read. (Zed, if you’re ever in the Bay Area, I owe you a pint.) While it’s a general purpose library, Mongrel is shaping up to be an amazing application server for Rails.
Well, horses for courses. Mongrel is awesome at being a tight little application server for Rails, but when it comes to configurability, speed serving static files, and stability, Apache 2.2 beats the pants off Mongrel. But the main reason is that Rails does not play well with concurrency, and unless your code somehow magically gets around this, it will eat flaming death if it tries to do two things at the same time. So just Mongrel by itself would be slow at serving up static content like images, CSS files, etc., and it would necessarily be limited to a single request at a time, unless the main point of your Rails application was to publicly eat flaming death. (”It’s a feature!”)
So we’re going to be using Apache 2.2 as the front-end for Mongrel, which allows us to both serve up static content like it’s going out of style, but also to use a huge wealth of modules, like mod_deflate, which will improve your site’s responsiveness and download time. We’ll run a cluster of Mongrel servers locally, and route requests through Apache’s mod_proxy_balancer, which uses a sophisticated algorithm to make sure all the Mongrel servers feel equally loved.
Apache is the heavy-weight in the web server world, and for good reason. It’s stable, fast, extensible, free as in Beery Speech, and all sorts of enterprisey. It’s gotten a bad rap in the Rails world, though, mainly due to its patchy FastCGI performance. It’s true, Apache + FastCGI is a horrible, horrible solution, unless your problem is “how can I waste my time on a dodgy server config?” in which case you shouldn’t be using the Intarweb while drunk. Finally, Apache currently has a great proxy balancer, which is why we’re using it.
This is not to speak ill of other front-end web servers, and you should feel free to use whatever you’re comfortable with. This article, though, is about how to set up what I’ve got set up, because I can vouch for it being awesome. Chip in on the comments if you’ve got a Mongrel band with a different lead singer and let us know how it’s going for you.
I’m going to assume you’ve got Apache 2.2, Ruby 1.8.4, Subversion, and whatever database backend you need installed. That all depends on your operating system. Personally, I’ve had a great experience with Fedora Core 5, but I won’t make fun of you if you use Gentoo, Ubuntu, RHEL, Debian, FreeBSD, or whatever. Just make sure you can install and easily maintain your various pieces of software. Also, make sure you’ve got all the various low-level development tools, like a compiler or two, and the development packages for Ruby and such.
Your application has to be using either ActiveRecordStore, SQLSessionStore, or MemCached to keep track of session data. Files are the worst possible way to deal with sessions, and when you have more than one server reading and writing from the same file-based session store it’s gonna blow up big.
You should have a Capistrano script which will at least run the setup tasks correctly. If you need help with that, there are many blog articles and even a full-blown, if slightly outdated, manual on the damn thing.
Finally, none of this is going to work without root access (or an impossibly permissive server), and you’ll just piss off your admins if you try otherwise.
At this point you should have an Apache 2.2 “Welcome to Apache” style installation, and your Rails application should work just fine in WEBrick mode. If that doesn’t work, you’ve got problems which are best dealt with by tech support, IRC vets, or Friends Who Owe You Favors. If you aren’t at that stage yet, bookmark this page and get to work. I’ll wait until you’re ready.
Ready? Go!
Because otherwise, the rest of this how-to is just going to be super awkward:
[server]$ sudo gem install daemons gem_plugin mongrel mongrel_cluster --include-dependenciesThis will download, compile, and install Mongrel and its dependencies, daemons, a library which allows Mongrel to curl up at your server’s feet like a good little poochy poozle, and gem_plugin, which allows folks like yourself to write gem-based, module add-ons to Mongrel. It will also install mongrel_cluster, which contains a few tools which make managing a cluster of Mongrel servers easier.
Finally, it installs sendfile. I’ve already said that Mongrel is slow for static content, and it’s true. However, it’s a smart pup (and Zed’s a smart, smart man). In a proxying configuration, Mongrel will use sendfile, if it’s installed, to send Apache a reference to a file instead of reading it from disk and farting it out to Apache over a local HTTP configuration. This allows Mongrel to do the heavy lifting required of man’s best friend–running your Rails application–without getting bogged down in the minutae of CSS files, images, and Javascript files. Long story short: your site is faster. Many people skip installing sendfile, or don’t know about it, and because of this, they usally write Apache+Mongrel off as slow. Sendfile is the secret ingredient which allows Apache+Mongrel to be blazing fast.
I used to advise installing sendfile, but it doesn’t help–Apache serves the static content, so the 20%-or-so boost on static files is never seen–and it can even cause some pretty severe stability issues. So. No sendfile, and if you’ve got it installed, uninstall it, pronto. (This comes straight from Zed, who would know such things.)
Be sure to install at least mongrel_cluster (and thus, Mongrel) on your workstation–deployment via Capistrano will be complicated, otherwise, since you’ll have to reimplement all the code they’ve thoughtfully laid out for you already.
At this point you’re ready to get your Mongrel configuration on.
First, you need to figure out how much memory you’ve got to spare, how much traffic you’re expecting, and how much money you’re willing to expend. An average Mongrel process can use between 15MiB and 40MiB of memory, depending on what your application does, and you need to figure out how many you can stuff into your server without popping it at the seams. Right now I’m running a decently-sized application with Apache, MySQL, and a cluster of three Mongrel servers in a 128MiB virtual private server with a bit of padding on top to keep the lid on. My advice is to start with two or three and build up to it. As you’ll see later, adding more clusters is stupid-easy. For now, we’ll assume that a cluster of three will do nicely.
Pop open a terminal window, amble on over to your Rails project directory on your workstation, and crank out a cluster config file:
[workstation]$ mongrel_rails cluster::configure -e production \\
-p 8000 \\
-a 127.0.0.1 \\
-N 3 \\
-c /path/to/your/capistrano/setup's/currentThis will add a file, mongrel_cluster.yml, to your config directory, which, when run, will launch a cluster of three Mongrel servers, starting on port 8000, listening on the local interface (and not to evil meanies on the internet).
Before we get to using it, however, we’ll need to make one small change for testing purposes. Open up mongrel_cluster.yml, and comment out the line with address: 127.0.0.1 on it (it’s YAML, so just add a hash at the beginning of the line). We’ll add this back in later, but for now we’ll want to make sure our cluster is behaving properly. Add mongrel_cluster.yml to Subversion, and check it in.
Mongrel_cluster comes with some drop-in replacements for Capistrano’s restart and spinner tasks. Open your deploy.rb and add the following to the top:
require 'mongrel_cluster/recipes'Now add the following to your deploy.rb settings:
set :mongrel_conf, "#{current_path}/config/mongrel_cluster.yml"Make sure you don’t have any old code in your deploy.rb obscuring the dark, shining beauty of mongrel_cluster. Also be sure you add the :mongrel_conf statement after setting :deploy_to. Otherwise, as Jesse Clark notes in the comments below, you’re in for some weirdness.
Finally, we need to register your Mongrel cluster as a system-level service, which will allow us to make sure it pops back up when your server reboots.
First, open a terminal window in your server’s services script directory. For most of you, this will be /etc/init.d/. Link the mongrel_cluster script from the gem directory to the services directory:
[server]$ sudo ln -s /usr/lib/ruby/gems/1.8/gems/mongrel_cluster-0.1.3/resources/mongrel_cluster mongrel_cluster
[server]$ sudo chmod +x mongrel_clusterThose you on Red Hat-based systems should add mongrel_cluster as a recognized service:
[server]$ sudo /sbin/chkconfig --level 345 mongrel_cluster onIf you’re running a Debian flavor, Ben Curtis assures me this works:
[server]$ sudo /usr/sbin/update-rc.d mongrel_cluster defaultsThose of you using otherwise will have to fend for yourselves. (Let me know how!)
Finally, link your mongrel_cluster.yml from your app’s config directory to /etc/mongrel_cluster:
[server]$ sudo mkdir -p /etc/mongrel_cluster
[server]$ sudo ln -s /your/rails/app/config/mongrel_cluster.yml /etc/mongrel_cluster/yourapp.ymlAt this point, your server is rarin’ and ready to go.
Well, time to deploy, eh? Almost.
Before you begin, make sure your server can access your Subversion repository. Capistrano unfortunately doesn’t allow you to enter a password when Subversion prompts you for one, so go to a temporary directory and check out the entirety of your application. Subversion will remember your passwords once you’ve entered them.
Got your server playing well with Subversion? Then you’re ready to go!
In a loud voice, say something profound and memorable (”That’s one small step for Rails…”), and deploy that bastard:
[workstation]$ cap cold_deployThis should checkout the latest version of your code, set up all the symlinks and other fun stuff, and finally start the Mongrel cluster. If it doesn’t, try running it with --trace and see what’s going on. Nine times out of ten, it’s some fiddly bit you forgot about in your haste to get this thing up and running.
If all went well (or was fixed after you let the magic smoke out), you should have three web servers running on your server:
Visit each one in your web browser, and make sure everything works. Fix all the problems you see until each works fine.
Once everything is to your satisfaction, open config/mongrel_cluster.yml back up again, and uncomment the address: 127.0.0.1 line. Save and check in your changes, and re-deploy:
[workstation]$ cap deployEverything should go smoothly. Make sure the three 800x ports on your web server are no longer accessible–the only public entrance to your web site should be through Apache!
Now your cluster backend is done! Congrats. Go get a beer, or whatever refreshing beverage the grown-ups let you have. Time to get to work on the front-end.
First, find out where your Apache config files are. If your distro isn’t a total crazy-pants, it’ll be something like /etc/httpd/ or /etc/apache2. Within that directory should be conf.d, which is where all of your server-specific config files go. We’re going to take a DRY approach to all this, and try to reuse as many bits of our config files as possible, to avoid the all-too-common problems associated with overlapping anything. Now, all files in the conf.d directory which end in .conf will be automatically included. Keep that in mind.
Your Apache installation no doubt includes a certain amount of cruft. Some of that you will need to remove immediately, like config files which block what we’re trying to do, and some of it you’ll want to remove later, like modules you don’t need.
In my installation, at least, the ‘Welcome to Fedora Core 5′ page that every URL produced was created by a default file, welcome.conf. It’s in the way, so out it goes:
[server]$ sudo mv welcome.conf welcome.conf.disabledLet’s lay out the file structure I’ve been using to keep my config files sane:
Feel free to replace ‘app’ with the actual name of your application.
And now, for the meat.
The common bits of configuration. This links a Directory or VirtualHost with the proxy-balancer, includes the proper mod_rewrite configuration for Rails, adds support for Capistrano’s enable_web and disable_web commands, and enables mod_deflate for most compressible formats.
ServerName myapp.com
DocumentRoot /var/rails/myapp.com/current/public
<Directory "/var/rails/myapp.com/current/public">
Options FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
RewriteEngine On
# Make sure people go to www.myapp.com, not myapp.com
RewriteCond %{HTTP_HOST} ^myapp\.com$ [NC]
RewriteRule ^(.*)$ http://www.myapp.com$1 [R=301,L]
# Yes, I've read no-www.com, but my site already has much Google-Fu on
# www.blah.com. Feel free to comment this out.
# Uncomment for rewrite debugging
#RewriteLog logs/myapp_rewrite_log
#RewriteLogLevel 9
# Check for maintenance file and redirect all requests
RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
RewriteCond %{SCRIPT_FILENAME} !maintenance.html
RewriteRule ^.*$ /system/maintenance.html [L]
# Rewrite index to check for static
RewriteRule ^/$ /index.html [QSA]
# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ $1.html [QSA]
# Redirect all non-static requests to cluster
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]
# Deflate
AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html
# Uncomment for deflate debugging
#DeflateFilterNote Input input_info
#DeflateFilterNote Output output_info
#DeflateFilterNote Ratio ratio_info
#LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
#CustomLog logs/myapp_deflate_log deflateThis is pretty simple, it just ties the settings in myapp.common to a VirtualHost on port 80.
<VirtualHost *:80>
Include /etc/httpd/conf.d/myapp.common
ErrorLog logs/myapp_errors_log
CustomLog logs/myapp_log combined
</VirtualHost>This is loaded by Apache to configure a mod_proxy_balancer cluster mapped to the internal Mongrel servers.
<Proxy balancer://mongrel_cluster>
BalancerMember http://127.0.0.1:8000
BalancerMember http://127.0.0.1:8001
BalancerMember http://127.0.0.1:8002
</Proxy>This provides a front-end for the proxy load balancing, which you can access from inside your server at http://localhost:8080.
Listen 8080
<VirtualHost *:8080>
<Location />
SetHandler balancer-manager
Deny from all
Allow from localhost
</Location>
</VirtualHost>
Time to kick over httpd and see if everything worked.
[server]$ sudo /sbin/service httpd restartIf all goes well, your web server is now configured as a Mongrel cluster backend to an Apache 2.2/mod_proxy_balancer frontend. You should feel happy.
This one will depend greatly on your distro, but it does contain one super-important tip for getting a TLS/SSL version of your Rails app working in this setup.
[server]$ sudo apt-get install mod_sslMy install added a file, ssl.conf to the conf.d directory. We’ll use this as a starting point, but let’s do it on our own terms:
[server]$ sudo mv ssl.conf ssl.conf.original
[server]$ sudo cp ssl.conf.original myapp.secure.conf
Now open myapp.secure.conf in your favorite editor (vi, right?), and scroll down to the VirtualHost it adds on port 443 (with Fedora Core 5, this was line 81). Inside the VirtualHost, add this:
Include /etc/httpd/conf.d/myapp.common
# This is required to convince Rails (via mod_proxy_balancer) that we're
# actually using HTTPS.
RequestHeader set X_FORWARDED_PROTO 'https'That last bit is super important, especially if you’re using the ssl_requirements plugin from 37signals (and why wouldn’t you?). Otherwise, the Mongrel servers won’t know if an incoming request came in secure or not, which means you’ll have no way of knowing if you really should be processing that credit card or not.
Now, either look up the SSLCertificateFile and SSLCertificateKeyFile entries in myapp.secure.conf and replace those files with your cert and key, respectively, or edit the entries to point to where you have your cert and key stored.
[server]$ sudo /sbin/server httpd restartNow see if that puppy works!
Personally, it’s worked incredibly well. My “banging tin pots together” benchmarking with httperf indicated that where the Apache1.3+FastCGI setup on Dreamhost could handle ~30 reqs/sec before it would start dropping connections, the Apache2.2+Mongrel setup on a 128MiB Rimuhosting VPS can handle ~90 reqs/sec before things get hinky. In addition, the amount of downtime due to lost worker threads or fudged FastCGI connections was nil. The only times the site went down could be directly traced to me cutting first and measuring second.
Apache+Mongrel is shaping up to be the preferred deployment platform for Rails applications. Recently, the official Ruby On Rails site moved to Apache+Mongrel for their blog, wiki, and manuals site, all of which are much more responsive and have much lower downtimes. Rails Machines, an exciting new “rapid application deployment” hosting service for Rails applications, is using Apache+Mongrel for all their services. Rick Olsen, Rails Core member, has been running both his blog and Rails Weenie on ApacheLitespeed+Mongrel recently, and some people like Jonathon Weiss have moved large sites over to this architecture.
The great thing about this deployment architecture is that it leverages one of Rails’ greatest strengths: share-nothing scaling. All you have to do is add more boxes, and Apache+Mongrel makes that as easy as changing your deployment recipe and your myapp.proxy_cluster.conf file, then restarting the services. Your Mongrel clusters can span boxes and networks, making scaling a simple matter of plugging in hardware.
Long story short: it’s golden.
I cribbed much of the initial Apache configuration and Mongrel ideas from Bradley Taylor’s Fluxura blog, as well as Jonathan Weiss’s article. Any errors I’m willing to acknowledge as my own, but leave a comment if you find something that doesn’t work for you! Finally, thanks to all the commenters with questions, corrections, and thanks. This article wouldn’t be nearly as accurate without you all. Thanks for trying to follow my instructions. ;-)
Enjoy!
261 comments »Moo hoo ha ha!
The web site I’ve been working on for the past three and a half months is done. Dig it:
Only the finest in GPS antennas, cables, cases, and other accessories.
Send me an email if you see anything wrong, eh?
WOO!
1 comment »