So, we’ve got a promising new web server in the ruby world – thin. Fantastic news, and it’s using some excellent libraries – the brilliant eventmachine, and the ragel HTTP parser from mongrel (ie, the only good thing about mongrel) – both of which I am using in other projects. Very promising, looks well designed, maintainable and clean code. Unlike mongrel.
So, there’s only one thing we care about with web servers. What’s the performance?
Here’s some stats on an actual serious rails app doing actual work with an actual database and everything. Production mode and I gave each server a couple of “warm-up” runs before the run copied below. I’m just going to include the whole output.
$ ab -n 1000 -c 50 http://0.0.0.0:3000/ Server Software: Mongrel Server Hostname: 0.0.0.0 Server Port: 3000 Document Path: / Document Length: 1418 bytes Concurrency Level: 50 Time taken for tests: 18.633091 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 1779000 bytes HTML transferred: 1418000 bytes Requests per second: 53.67 [#/sec] (mean) Time per request: 931.655 [ms] (mean) Time per request: 18.633 [ms] (mean, across all concurrent requests) Transfer rate: 93.22 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.9 0 10 Processing: 23 909 130.8 927 1763 Waiting: 19 908 131.1 927 1763 Total: 23 909 130.1 927 1763 Percentage of the requests served within a certain time (ms) 50% 927 66% 936 75% 944 80% 949 90% 973 95% 1061 98% 1067 99% 1069 100% 1763 (longest request) Server Software: thin Server Hostname: 0.0.0.0 Server Port: 3000 Document Path: / Document Length: 1418 bytes Concurrency Level: 50 Time taken for tests: 18.120868 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 1746000 bytes HTML transferred: 1418000 bytes Requests per second: 55.18 [#/sec] (mean) Time per request: 906.043 [ms] (mean) Time per request: 18.121 [ms] (mean, across all concurrent requests) Transfer rate: 94.09 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 2.7 1 31 Processing: 683 893 123.6 881 1710 Waiting: 605 721 120.3 718 1473 Total: 683 894 123.7 882 1711 Percentage of the requests served within a certain time (ms) 50% 882 66% 888 75% 902 80% 908 90% 961 95% 1002 98% 1474 99% 1710 100% 1711 (longest request)
Hm, nothing much in that – an insignificant improvement. Basically, that’s Rails slowness we’re measuring here. Let’s cut most of that out of the picture and go straight for a “pulse” controller – returns absolutely nothing but the two-byte string “OK” meaning that it’s actually running. I implemented that for monitoring without hitting the front page every few seconds with monit. Let’s take a look:
Server Software: Mongrel Server Hostname: 0.0.0.0 Server Port: 3000 Document Path: /api/pulse Document Length: 2 bytes Concurrency Level: 50 Time taken for tests: 8.405170 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 269000 bytes HTML transferred: 2000 bytes Requests per second: 118.97 [#/sec] (mean) Time per request: 420.259 [ms] (mean) Time per request: 8.405 [ms] (mean, across all concurrent requests) Transfer rate: 31.17 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.6 0 4 Processing: 14 412 72.4 399 810 Waiting: 13 411 72.3 398 808 Total: 14 412 72.0 399 810 Percentage of the requests served within a certain time (ms) 50% 399 66% 470 75% 477 80% 480 90% 486 95% 489 98% 491 99% 493 100% 810 (longest request) Server Software: thin Server Hostname: 0.0.0.0 Server Port: 3000 Document Path: /api/pulse Document Length: 2 bytes Concurrency Level: 50 Time taken for tests: 6.65994 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 236000 bytes HTML transferred: 2000 bytes Requests per second: 164.85 [#/sec] (mean) Time per request: 303.300 [ms] (mean) Time per request: 6.066 [ms] (mean, across all concurrent requests) Transfer rate: 37.92 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.2 1 10 Processing: 200 298 49.2 278 460 Waiting: 54 239 47.5 224 404 Total: 202 299 49.0 278 460 Percentage of the requests served within a certain time (ms) 50% 278 66% 294 75% 354 80% 356 90% 362 95% 372 98% 457 99% 460 100% 460 (longest request)
That’s much more of an improvement.
I’ve tested it at some length now and have encountered no stability problems – not that it takes much to beat The CrashMaster™ mongrel. In fact, I have no love for mongrel at all (please see my recent posts on hacking it so it doesn’t refuse to launch upon encountering its own PID files from previous crashes) and so I’m switching to thin, effective now. I’ll let you know how it goes!
UPDATE: Yet another potential competitor has emerged – ebb, which appears to be even faster than Thin in (artificial) benchmarks. However, it’s definitely not ready for prime time – it ran an order of magnitude slower than Thin in my initial testing, and I eventually force quit it after becoming unresponsive. Still, those benchmarks paint a pretty promising picture of what a C implementation can do. Here’s my bad results anyway.
$ ebb_rails start -e production $ ab -n 100 -c 5 http://0.0.0.0:3000/api/pulse # i reduced the numbers Server Software: Server Hostname: 0.0.0.0 Server Port: 3000 Document Path: /api/pulse Document Length: 2 bytes Concurrency Level: 5 Time taken for tests: 49.116267 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 18800 bytes HTML transferred: 200 bytes Requests per second: 2.04 [#/sec] (mean) Time per request: 2455.813 [ms] (mean) Time per request: 491.163 [ms] (mean, across all concurrent requests) Transfer rate: 0.37 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.8 0 8 Processing: 442 2394 552.0 2379 3699 Waiting: 442 2394 552.1 2378 3699 Total: 442 2394 552.1 2379 3699 Percentage of the requests served within a certain time (ms) 50% 2379 66% 2661 75% 2795 80% 2835 90% 2997 95% 3481 98% 3665 99% 3699 100% 3699 (longest request)
Obviously something is horribly wrong with that result.
February 29th, 2008 at 9:48 pm
Thanks very much for posting that.
I’d love to see Mongrel’s virtual hegemony in the Rails world knocked own by a leaner, cleaner contender like Thin. Of course, I know precisely nothing about Thin, other than what you’ve written here, so I will be very interested in hearing more impressions from you over time.
Perhaps this is the beginning of Thin/nginx displacing the dominant Apache/Mongrel duo.
March 1st, 2008 at 2:01 am
We can but hope! I’ve played with nginx, but not in earnest – I really do not want to run *two* HTTP servers, and I just haven’t had enough time to look into how to do SSL etc. It does seem pretty good though. Not like I use any of Apache’s capabilities anyway – well, none that nginx can’t also do. Seriously, who uses any of Apache’s modules besides PHP and possibly mod_svn? I’ve always hated that module system, actually, it’s caused me nothing but trouble. Compiling Apache is more difficult than compiling fucking linux.
In terms of raw performance gains for Rails apps I think there’s only so much scope for improvement at the app server anyway. For all my complaints about mongrel, and I have many, it’s pretty fast and so replacing it hasn’t had all that much of an impact – the 25% or so in the second test above is a pretty unrealistic best case scenario. In real world usage I doubt it will make more than a percent or two difference.
My main hope for thin is that it can be usefully expanded and modified. Mongrel’s maintenance is shockingly bad and many apps resort to horrendous monkey patches to try to improve things – see evented_mongrel or the hacks necessary to even run ThingFish for evidence. Hopefully the thin server will be far more flexible and maintainable, so the community can begin to standardise around something because it’s good, not just because there’s no other choice.
I think the real speed benefits are going to come from elsewhere, however. I don’t know if you’ve been tracking edge rails lately but there’s been a ton of performance enhancements included recently and developers seem to be incresingly shifting towards optimisation patches – a very welcome development. Furthermore, most of Rails is now ready for ruby 1.9, as is Thin (but not mongrel!) – I think we’ll see a considerable increase in performance because of that. Combining all these new developments I believe we’ll see performance up by 50%+ in many cases as Rails 2.1 (the target for ruby 1.9 and all this new optimisation) is finalised and people start switching to 1.9 – excellent news.
March 1st, 2008 at 3:02 am
I actually don’t mind running a proxy server like nginx alongside Apache. Basically, Apache on one IP for all the “normal” virtual hosts, and nginx on another IP for my one-and-only Rails app. I actually quite like the compartmentalization aspect. Rails really is such a different beast that I think I prefer to keep it separate.
I don’t “track” Edge Rails in the sense of actually running it, but I do keep an eye on what’s being committed, more or less. It is excellent to see effort going into optimization. There can’t be too much of it as far as I am concerned.
March 1st, 2008 at 3:29 am
Wincent, here’s a good link for busy people who nonetheless want to keep somewhat up to date with edge rails: http://blog.codefront.net/category/ruby-on-rails/
Well, I guess I don’t mind running another server – especially if it segregates and simplifies things. I think the real reason I haven’t put nginx into place “in production” is mainly because my current apache configuration is a beast, and I’d have to redo it, and set up something else. Since what is in place now works, I’ve been avoiding it : ) But I’ve already taken the step of redoing my nameserver, separating out IPs by application types, so I’m on the way to it. Unfortunately I host a couple of “production” rails apps for actual paying clients so I have to be at least moderately careful. Running other people’s shit is such a hassle, actually, that I’m thinking of buying them a couple year’s account at EngineYard or somewhere, out of my own pocket, just to be rid of it .. sigh.
BTW just looked at your twitter (lol – who would have thought I would ever say that) and you’ve already submitted a fucking compat patch for Thin. Jesus Christ that was quick! But why the hell do you want to stay with 1.8.5? I understand the appeal of staying with a stock system but I’d sure as hell make an exception for ruby.
For myself I have no choice but to run 1.9. I’ve been writing a server which requires unicode ability in several important ways – you may recall the extensive Rails work to paper over the cracks, an awful lot of pointless work I was quite unwilling to duplicate. The speed improvements, threading improvements, and GC enhancements seal the deal. I wouldn’t switch to it right now for production – hell, a crashing bug in ruby 1.9 was only fixed a couple weeks ago, and only in trunk – but digging in for the long haul with 1.8.5 seems strange to me.
Hell, I’m not even particularly fond of RHEL. I have never seen any evidence whatsoever that it is any more secure than free linux versions, and I have no intention of ever calling RH support – the very idea seems kind of strange and anachronistic! The extended support means nothing to me, as I update far more regularly anyway. RHEL5 has adopted yum in favour of up2date – a wise move as yum is far better. I have installed and use yum in RHEL4, keeping up2date only for system updates. The VM capability would be nice if it was cross platform; it’s not. I have difficulty naming any particular aspect of RHEL that I like all that much, and if I’d had to pay extra for it on my hosting plan I would have gone with something else. My policy is to be platform agnostic to the greatest possible extent.
Did you read anything with the announcement of FreeBSD 7 the other day? Some pretty compelling things in there! ZFS support for one – an older version for sure, and still somewhat experimental, but at least it’s in. We ain’t gonna see that in linux anytime soon. And the dynamic TCP socket buffer resizing – and SCTP support – are very exciting:
Here’s a good rundown of FreeBSD 7.
March 1st, 2008 at 4:08 am
Ah, good old FreeBSD. It still has a place in my heart. I always liked it more than Linux (emotionally speaking that is, not talking about technical merits, I think they’re both pretty good), in the same way that I like Ruby, affectively speaking, more than PHP or Perl or Python.
As for the benefits of RHEL, it’s all about the certification and integration for me. Basically they stand behind the entire package and you now it’s guaranteed to work, be rock solid stable with no compatibility-breaking changes, always kept up to date with security fixes, right through to 2014. It’s about piece of mind and low-maintenance for me as an administrator.
And that’s exactly the reason why I prefer to stick with the Red Hat-supplied Ruby, even though it’s only 1.8.5. Keep things in perspective: there is not that much difference between 1.8.5 and 1.8.6, and all the important bug fixes and security fixes are backported by Red Hat. And the 1.9 series is not recommended for production use, it’s for playing with only. I know you could take a gamble on it, deploy it, and watch it carefully for breakage, but I simply don’t enjoy playing those kind of games. I want my server to be like a fridge: plug it into the wall and forget about it.
And it’s this fridge-like quality that makes me like RHEL. I don’t care about their tech support and I’ve never even thought about it. But the fact that you can write your app and deploy it in 2008 and know that it is guaranteed to work up until 2014 is a huge plus for me. Some see the fact that RHEL 5.1’s Ruby is forever locked at 1.8.5 as a disadvantage, but I see it as a win.
It’s all about horses for courses. I fully recognize that others have different needs and wants and for them tracking 1.8 and even 1.9 is what suits them.
I’m just hopping that the Rails core team can show at least some discipline and not run off release new versions of Rails which require ever-newer versions of Ruby. If they want to be taken seriously in the Enterprise market then they’ll have to accept that enterprise doesn’t want to track the “Edge” nor upgrade to the latest and greatest Ruby preview every six months.
March 1st, 2008 at 4:45 am
Enterprise, you say? Fuck the enterprise and their stupid, misguided conservatism. It’s the “enterprise” attitude which is why we all have to test our web pages with IE6. If it’s a choice between progress and pleasing the suits, I choose progress every time and fuck the suits.
I like your fridge analogy, but web servers are more complex than that. The internet evolves at breakneck speed, and you simply can’t expect today’s technology to be acceptable – or even usable – in 5 years’ time. So many emerging technologies will require huge upgrades – I am thinking HTML5 with its persistent connections, which will require changes to Apache/whatever unless you want to keep a full-fledged server instance sitting there in memory waiting for a serverside event to be sent, which may never even happen. Soon you’ll have to implement IPV6. DBs are changing for the better. It’s a time of great change, actually – I can’t remember such an atmosphere of progress since the late 90s.
I understand all the points you make, and moreso, I know why you make them – I’m a server admin too! But I just think expecting to be able to plug a server in and forget about it for 5 years is kind of strange. The internet is very far from “done”, and 5 years is an awfully long time.
And as for FreeBSD, I know exactly what you mean! I also “like” it, from an emotional standpoint, more than linux. I think it’s more intuitive, better laid out, and just all-round nicer. OSX’s BSD flavour only enhances that feeling! But for all that linux is way more common, and I’d be pretty cautious adopting anything else.
Bah. Don’t take my comments the wrong way. I know what you mean and I agree with much of it. I guess I’m just annoyed at the general lack of innovation on the net in the last few years – and it’s very much to do with the don’t rock the boat, compatibility-at-all-costs “enterprise” attitude. I have a pretty big chip on my shoulder about that, as you can probably tell – it reminds me all too much of MS and the web’s stagnation due to IE6. Bring on the change, I say, and if it breaks someone’s five-year-old software then so fucken’ be it.
To me the ideal RHEL usage case would be a DNS server. DNS isn’t going anywhere, it changes extremely slowly. It’s a very good idea to have a RHEL nameserver – set it to auto-update and forget about it for years at a time. But for the web – with its emerging new paradigms, HTML5 on the way, revolutionary database concepts on the verge of mainstream, and ever-greater convergence of communications technologies – the Red Hat model is wrong, IMO.
Horses for courses, like you said
March 1st, 2008 at 5:07 am
It sounds like we don’t share the same definition of the word “enterprise” either. Strip off all the negative associations you have with the word until all that’s left is the idea of “rock-solid, robust, non-changing platform upon which to build applications”, and that’s the essence of “enterprise” that I was getting at. It’s basically the only part of it that I care about, and the reason why Red Hat “Enterprise” Linux is attractive to me.
And I think we have very different goals, from what you’re saying. All the new stuff you’re talking about on the horizon isn’t really important to me at all.
I’m not a “startup”, I’m not looking to jump on the “Web 2.0″ bandwagon or the “Web 3.0″ one when it comes, I’m not looking to get millions in venture capital, not looking to launch the next Flickr, Twitter, Facebook or any application named as “number-noun”, “color-noun” or ending in a consonant followed by the letter R.
I sell desktop software and I want to make a useful, easy-to-navigate, clean and helpful site for my customers. So for me there is absolutely no interest in or need to worry about HTML 5 nor emerging DB trends, nor scalability into the millions-of-users range. Given my usage case (the horses for courses thing), the ability to write an app and deploy it in 2008, and still be using it in 2014 is actually a very good thing. Of couse, the application won’t sit in stone for six years, it will evolve incrementally, but I don’t want to have to worry about the platform moving underneath it.
March 1st, 2008 at 5:26 am
Yeah, I understand. We have different goals and needs and I think your position is entirely reasonable – though I wouldn’t bet on Rails being 1.8.5-compatible in 2014!
Sorry if my previous comment came across as offensive – that wasn’t my intention. But I think you’ll find that my own highly negative definition of “enterprise” is very common. It’s nothing but a meaningless buzzword to me – except of course for the negative connotation. To me any software declaring itself as “enterprise”-ready is the equivalent of a sysadmin proclaiming they have an MCSE – a worthless keyword, more often a contraindicator of quality than not, and designed solely for clueless middle managers who fancy themselves captains of industry but in reality don’t know what the hell they’re talking about. “Nobody ever got fired for buying IBM” – you’ve probably heard that before. “Nobody ever got fired for buying “enterprise” software” could be a modern equivalent.
Hehe. Sorry for the bile.
If you’d worked in corporate IT, you’d be spitting it too : )
March 1st, 2008 at 6:35 am
Yeah, I know what you mean. Normally the word “enterprise” has a negative connotation for me; similar to the word “consultant” which conjures up images of conceited men in suits and ties charging more than they deserve. But it basically was the best word I had to denote what I was talking about.
And yes, you’re right: it’s highly unlikely that Rails will be 1.8.5 compatible in 2014. I do hope, however, that they settle down a bit as time goes on. Probably wishful thinking I know, but their track record in terms of API stability up until now has been pretty appalling. Staying even remotely current with Rails is akin to building a house on top of an unanchored raft. And worst of all, the Rails hype machine has kicked in to make following the trunk hip and cool (living on “the edge”, as they say). I don’t think it’s hip and cool. I think it’s irresponsible.
March 1st, 2008 at 7:05 am
You’re right about Rails there, but I think it’s a good sign that they’re concentrating, as I said, more on optimisation these days. That seems to indicate a shift away from radical change and more towards perfecting what’s there already. I think many people are also sick of the rate of change – even for such a young project, it’s beginning to get real critical mass in terms of installed base which will slow that down. Anyway, I think most of the deprecation has been done and I’d expect 2.0 to be pretty stable, tweaking excepted.
And, well, edge rails would be crazy for production. And if you’re adopting a change-often, publish-often kind of approach, using edge to develop could easily lead to you pushing out something that breaks server side. I don’t use edge rails either, but I admit I’m definitely looking forward to 2.1.
Hell, forget the shifting sands of the API. I’ll be happy when the currently implemented features work properly. I have never been able to get
< %= javascript_include_tag :all, :cache => true %>to work reliably, norconfig.action_controller.asset_host. I admit it’s possibly my fault, but can’t for the life of me get it working and stay working. I’ve turned both of those off until further notice.Anyway, as we’ve discussed before, the ideal solution to the problem of updating “base software” versions separately from the core OS is virtualisation, and RHEL5 will serve you well in that department. How much RAM has your new server got, out of interest?
We should be doing this in a podcast, man : )
March 1st, 2008 at 7:38 am
Yes, I’m looking forward to 2.1 as well. There are a couple of features which will enable me to remove some fairly ugly code, and as you say, the optimizations are very welcome.
I get the impression though that most of the work is being done by one guy.
The server’s got 2 GB, which I imagine should be enough for now. Can always add more if needed, but I’m only going to deploy one app. I’m thinking of doing it with 4 nginx worker processes and about 4 Thin application servers. Will adjust those numbers according to how it pans out.
Speaking of podcasts, it’s a definite option now. Got my new mic the other day. A Blue Snowball. Not the highest quality one on the market, but probably the best combination of convenience (USB), looks, performance, and price. I bought it principally to do screencasts.
I see you’re on Twitter now. I wonder if you’ll be impressed as I am with it. In the two days since I signed up it’s been down or malfunctioning about 5% of the time.
March 1st, 2008 at 7:56 am
I signed up for Twitter a long time ago! Mainly to reserve my name in case it becomes good. As you can see I haven’t used it much, even had to look up my password to log in. Haven’t noticed any downtime, but haven’t really used it enough to notice. Twitter is an example of such a bad name that I am prejudiced against the site from the very beginning. I know, there’s worse, but still.
Damn. Now you mention microphones, my good ones are both in JP! Well, I have a good one here but nothing to plug it into. I’d forgotten about that. Might have to wait a little longer.
June 6th, 2008 at 10:31 pm
[...] use my /api/pulse controller action from previous testing. This is going to be Webrick-only, thin installs and seems to work but I haven’t yet figured [...]