Ah, Maglev. It seems that only yesterday I was ripped to shreds by a furious mob for daring to question your public claims of drastic speed gains over every other Ruby interpreter.
And so imagine my delight today, when the first alpha release is finally available! Sure, it’s only an alpha release, so it’s not going to be quite as good as the final product – but surely it will live up to the claims in the RailsConf 2008 demonstration? I mean, it was that good even then! Surely it must be even better now!
Anyway, I obviously had to install it and put these claims to the test.
I followed the instructions on – with a couple of changes, such as that I use .profile instead of .bashrc.
I decided to test with as close to a real world library as I could. That post mentions that Maglev supports Sinatra, so I thought that would be an ideal test. Then again, it also says it supports RubyGems but I couldn’t get that to work at all. C extensions are out too, so we’ll be using Webrick.
To get around the RubyGems problem, I manually downloaded Rack and Sinatra and placed the contents of their lib directories into a single folder. I then created a trivial sinatra app as shown:
require File.expand_path(::File.dirname(__FILE__)) + '/sinatra.rb' require File.expand_path(::File.dirname(__FILE__)) + '/rack.rb' get '/' do "Maglev Rocks!" end
I decided to run good old ab against it. My exact command was:
ab -kc 10 -t 10 http://127.0.0.1:4567/
Because I have other servers installed in Ruby 1.8.7 and 1.9.1, I forced sinatra to serve using webrick as follows:
ruby1.9 testmaglev.rb -s webrick
Considering the past hype of order-of-magnitude speed hikes, I was ready to be blown away. Are you ready to be blown away? Oh yeah baby, 50 times faster comin’ right up!
RESULTS
Ruby 1.8.7: Requests per second: 131.38 [#/sec] (mean)
Ruby 1.9.1: Requests per second: 144.77 [#/sec] (mean)
Maglev: Requests per second: 97.78 [#/sec] (mean)
Well golly gosh. Look at that. It’s not faster after all – in fact it’s climbing the same steep performance hill ever other interpreter has had to climb, Smalltalk magic notwithstanding. How about that.
I said the benchmarks shown at RailsConf were bullshit and would not even remotely reflect the real world performance of the final product. I said that until the interpreter implemented all of ruby, it was not ruby, and it was worthless to measure its performance. I poured skepticism upon the notion that a new interpreter, just by dint of some doubtfully superior Smalltalk heritage, could leapfrog all previous contenders with a two-order-of-magnitude performance boost.
Looks like I was right. But don’t take my word for it, run the tests yourself. I will be accepting contrite apologies in comments … 
November 24th, 2009 at 2:44 am
Surprised nobody has commented on this one.
Let me start he ball rolling: you were right then, and you are right now too. Maybe someone will learn a lesson from all this (or is that hoping for too much?).