I hate running in place, and that’s often what low-level stack changes feel like. My users don’t care if the website is served by Mongrel, WEBrick, Thin, Passenger, or whatever as long as it does what they want. What they do care about is downtime, and as the Rails boot time for Obsidian Portal has gotten slower and slower, downtime in production was becoming a major issue. Every deployment froze the site for up to 3 minutes while Passenger was restarting, making me very hesitant to do any deploys. I wanted to get back to the old days when I would deploy four or five times a day just to tweak wording or move an element around on a page. Enter the Unicorn…
Unicorn has been covered extensively elsewhere, so I won’t rehash everything, but the main selling point for me was the zero-downtime deployments. Unfortunately, while that’s Unicorn’s number one feature, it requires a fair bit of work to setup.
preload_app
The key to the zero-downtime deploys is the preload_app setting. If you can set this to true, Unicorn can do its magic. If set to false, as far as I can tell, the promise of zero-downtime deploys is out the window.
Setting it to true mainly involves making sure that socket connections are correctly established and maintained in the worker threads. All the examples show how to do this with ActiveRecord, but in a larger project, you may have many other socket connections to deal with. If you’re like me, you may even have some connections that you’re really not sure about. I decided to make a list of things to check:
My Potential Sockets
- MySQL / ActiveRecord
- MongoDB (raw driver, no ODM)
- Redis
- Sphinx / ThinkingSphinx
To make things more confusing, if I understand correctly, you really only need to be worried about connections that are established at Rails boot. If the connection is established in your application code, like the first time a service is used, then I think it’s fine to ignore it. In my case, this is what I do with my raw Redis and MongoDB connections. I establish them in the application code, not at bootup. Finally, I’m really not sure how or when ThinkingSphinx connects to Sphinx. I assume it’s a socket, but I dug around in the code for a bit and couldn’t find anything.
Ultimately, I went with the let’s-see-what-happens approach and tried a few deploys where I handle the ActiveRecord connection and ignore everything else. The app ran fine and all my services could talk to each other, so I feel pretty good about the setup.
Besides handling the socket connections, it’s just as important to tell the old Unicorn instance to shut down after a “restart”. That shutdown cycle is part of the recipe for zero-downtime deployments. In the recommended configuration, Unicorn does not shut down on its own. Instead it clones itself and you are responsible for killing the old master process once your new workers are ready to serve requests. Unicorn makes it fairly easy, but it’s a different mentality than other systems in the past.
Lots of setup
Unicorn involved significantly more setup for me than Passenger, especially with all the reading necessary to understand preload_app and its consequences. Plus, if you add more socket connections in the future, you’ll have to remember to update your Unicorn after_fork, meaning even more maintenance cost in the future. That kind of dependency is a big downside. However, the zero-downtime deploys actually do work, and that’s a big plus.
Takeaway 1: Just set preload_app to true and do what it takes to get that working right. If you’re using Unicorn for zero-downtime deployments, this is how you want it.
Takeaway 2: Weighed against my desire to get back to deploying-at-will, I’d say the switch to Unicorn was a success, and I recommend it to other Passenger users. However, expect to spend a day or two getting your preload_app and after_fork settings right.
Also: Nginx
Since I’m a glutton for punishment, I decided that as long as I’m switching from Passenger to Unicorn, I might as well switch from Apache to Nginx. I’ve been a loyal Apache follower for a long time, but I wanted to get some Nginx experience just for my own edification. Overall, I’d say that Nginx is simpler and easier to learn. I was able to install it, set up SSL, proxy to Unicorn, setup the Capistrano maintenance page, and even add some funky rewrite rules, and it was easier than the last time I messed with Apache. I think everything I needed was available without installing extra modules, although perhaps Chef and/or apt-get did that for me without asking.
Takeaway: If you’re starting from scratch and have to choose between Apache and Nginx, go with Nginx, even if you already have a strong familiarity with Apache. You’ll be pleasantly surprised.
Parting Words: Another Win for Chef
Once again, Chef was a huge help with all of this. If you can’t go with something as simple as Heroku, then you need Chef (or whatever config tool you prefer). Thanks to Chef (and the Opscode platform), I was able to build and tweak my entire Nginx/Unicorn setup in my staging environment over the course of two days and get my recipes rock solid. Then, deployment to production was a matter of running chef-client and cap deploy. Production downtime was about 10 minutes, and everything worked perfectly once nginx/unicorn came up. No missed steps or bungled configs.
Takeaway: Scripting up your server stack config is awesome, regardless of how small or simple your setup is. Use Heroku if you can, and Chef if you can’t.
