Mongrel, currently the mainstream Rails server, has a nasty habit of not deleting its PID files when it crashes and burns. This then occasionally stops automatic restarts because mongrel sees its old PID files, assumes it’s running (without a check) and refuses to start.
All that needs to be done is a startup check that the process mentioned in the PID file is actually running. This has been implemented in a patch here which I’ve integrated and successfully tested. If you’d like to use the file I generated, I’ve attached it below. Simply replace /path/to/gems/mongrel-1.1.3/bin/mongrel_rails with the attached file and run as normal. You’ll want to make a backup copy first, of course!
Also on the “things to do while hacking mongrel” list is to change the fucking version so it’s actually up to date as echoed in your console log. The file is lib/mongrel/const.rb line 68,
MONGREL_VERSION="1.1.3".freeze
which the maintainers obviously missed.
Mongrel is an aptly named bit of software, poorly maintained and with many long-standing patches which should have been included long ago. This particular patch fixes an annoying bug which shouldn’t exist any more – and harks back to March 2007, 10 months ago. That’s a pretty unforgivably long time not to include something as basic as this. And as far as I can tell the mongrel_rails program hasn’t changed significantly since last time – there’s no excuse for this, and other, patches to have not made it in by now. This is the third time I’ve manually patched it like this, and was only reminded to do so in the latest version (1.1.3) when I noticed – surprise! – it was again refusing to start with a stale PID.
I’d be more worried if there weren’t several promising alternative servers on the rise. Hopefully the Rails community’s dalliance with the very doglike mongrel won’t last too much longer…
mongrel_rails – full file
mongrel_stale_pid_file.patch – same thing in patch form. Run
$ cd /path/to/gems/mongrel-1.1.3/ $ patch -p0 < /path/to/mongrel_stale_pid_file.patch
Both from http://textsnippets.com/posts/show/931.
Testing (on MacOSX):
$ cd ~/Desktop $ /usr/bin/ruby /usr/bin/mongrel_rails start -d -e development -p 3000 -a 127.0.0.1 -P /Users/sho/Desktop/mongrel.pid -c /rails/myapp # mongrel starts ... $ cp mongrel.pid another.pid $ /usr/bin/ruby /usr/bin/mongrel_rails stop -P /Users/sho/Desktop/mongrel.pid Sending TERM to Mongrel at PID 1605...Done. $ cp another.pid mongrel.pid $ /usr/bin/ruby /usr/bin/mongrel_rails start -d -e development -p 3000 -a 127.0.0.1 -P /Users/sho/Desktop/mongrel.pid -c /rails/myapp ** !!! PID file /Users/sho/Desktop/mongrel.pid exists, but is stale, and will be deleted so that this mongrel can run. # mongrel starts as hoped .. success!
Tags: mongrel
February 28th, 2008 at 6:51 am
[...] much to beat The CrashMaster™ mongrel. In fact, I have no love for mongrel at all (please see my recent posts on hacking it so it doesn’t refuse to launch upon encountering its own PID files …) and so I’m switching to thin, effective now. I’ll let you know how it [...]
March 1st, 2008 at 12:33 am
Just looked into God. Looks like quite a nice monitoring package. It explicitly handles stale PID files like those that Mongrel chokes on. Wonder if Thin chokes too when it finds stale PID files.
March 1st, 2008 at 1:37 am
I should hope not! Maybe I’ll test that, actually ..
Yeah, I’ve taken a look at God as well but haven’t tried to run it yet. I have a nicely working monit setup and after customising everything including capistrano recipes, etc, I would need a good reason to migrate. Monit has been extremely reliable for me, can monitor a wide variety of applications – and to be honest its setup files are if anything more readable that God’s. The sole advantage I can see to using God right now is that you can have a range of ports under one setting:
%w{8200 8201 8202}.each do |port|. I have them listed individually, maybe I should look into that more…As for the matter of dealing with stale PID files, I feel that is the fault of the application and so place the blame solely at mongrel’s feet. The solution isn’t to build mongrel-cleaning-up features into monitoring apps, the solution is to fix or get rid of fucking mongrel. I vote for the latter!
As for starting and stopping these apps .. none of these solutions are ideal from my perspective. They are reinventing the wheel, duplicating what should be (IMO) OS functions, and introducing too many new factors and things to go wrong. Launching apps should be done by inetd/rc.conf/launchd/whatever, not some 3rd party monitoring app running on top of everything. Writing a custom monitoring script for each machine seems fragile and duplicative. I don’t have a perfect answer yet – though I do have some ideas – but I like to think of these solutions as stopgaps until we can find something more in line with what I consider best practise for controlling a server.
September 7th, 2008 at 4:01 am
unless `ps axo pid= | grep #{pid} | grep -v grep`.length > 0 is safer.
ps ax alone will include the name of all processes which could have arguments that include the pid number.