Sho Fukamachi Online

One of the things I miss most about Rails when working in gems is the reload! function, which rebuilds the environment to update anything that’s changed since the last save. Well, I wanted to recreate that functionality, but hopefully in a really simple way.

The good news is, it’s actually pretty easy. I have two ways of doing it, pick the one you like best.

module AutoReload
 
  @@required_mod_times = {}
 
  def reload!
    diffs = AutoReload.differences # we can only call it once per reload, obviously
    if diffs.size > 0
      diffs.each {|f| Kernel.load(f)}
      puts "reloaded #{diffs.size} file(s): #{diffs.join(', ')}"
    else
      puts "nothing to reload"
    end
  end
 
  def self.update_modtimes
    $".each do |f|
      @@required_mod_times[f] = File.mtime(f) if File.exists?(f)
    end
  end
 
  def self.differences
    oldlist = @@required_mod_times.clone
    AutoReload.update_modtimes
    newlist = @@required_mod_times.clone
    oldlist.delete_if {|key, value| newlist[key] == value }
    oldlist.keys.uniq
  end
 
end
 
include AutoReload

You will then need to initialise it somewhere after all your requires. AutoReload.update_modtimes will do the trick. If you can’t manage that, it will only work properly after the first time you use reload!.

This is the way to do it if you have a lot of files, I think, since it maintains a list of what was changed when, and then only reloads changed files.

Note that it’s not perfect. It will only be able to find files which are in local path, ie won’t be able to reload gems. However, that’s all I need for now.

The next way is even simpler, since it doesn’t bother to maintain a list - it just blindly reloads everything it can:

def reload!
  diffs = []
  $".each {|f| diffs << f if File.exists?(f)}
  if diffs.size > 0
    diffs.each {|f| Kernel.load(f)}
    puts "reloaded #{diffs.size} file(s): #{diffs.join(', ')}"
  else
    puts "nothing to reload"
  end
end

As you can see, this just blindly reloads everything it can find. Probably not the best if you have a lot of constants etc, but for a simple project could be just the ticket. The good news is you don’t need to initialise it. if you have a lot of files you probably should return "OK" or something, else you’ll have pages of reloads scrolling past.

Let’s bear in mind that this kind of trick is always a bit of a hack. Kernel.load() has no ability to unload anything, even if it doesn’t appear in the file anymore. All it can do is overwrite. If you break your code by deleting something important, then reloading using this kind of trick won’t show it up - the object is still there until you reload ruby itself. It’s a convenience thing only, so don’t rely on it too much, do a full reload once in a while.

However, for my use case - making a lot of small changes and working in a very interactive manner with irb - this is a real time-saver, hope you find it useful too.

If you’d prefer it to happen automatically, rspec-style, there is a gem available which will do this for you here which basically does the same thing, just every 1 second instead of manually.

Any Ruby programmers who are reading this, I’m experiencing a strange issue regarding Digest::MD5. Let me show you:

>> require 'digest/md5'
=> true
>> Digest::MD5.digest "Les Rhythmes Digitales"
=> "\213U\3601\260%<j\267-\343(\213I\030\347"

What the fuck is that huge escaped thing? A unicode issue?

Check out the same from bash:

$ md5 -s "Les Rhythmes Digitales"
MD5 ("Les Rhythmes Digitales") = 8b55f031b0253c6ab72de3288b4918e7

Now that looks more like what I expect from an MD5 hash. Is Digest::MD5 mangling the text into some kind of weird invalid unicode?

Let’s try with $KCODE set:

>> $KCODE = "UTF8"
=> "UTF8"
>> require 'digest/md5'
=> true
>> Digest::MD5.digest "Les Rhythmes Digitales"
=> "\213U?1?%<j\267-?(?I\030\347"

Great. Any different in 1.9?

irb(main):003:0> Digest::MD5.digest "Les Rhythmes Digitales"
=> "\x8BU\xF01\xB0%<j\xB7-\xE3(\x8BI\x18\xE7"

Different again. At least I can see the characters in there, though. This is causing some pain.

Am I doing something hopelessly wrong? Somewhere in all these, some character encoding crap is going down. I can’t believe I’m the only one having these problems, and they render it difficult to use hashed passwords. I am working around the issue by shelling out to bash for now, but would like to get it fixed.

UPDATE: About 2 minutes after writing that, I realised I need to use Digest::MD5.hexdigest, not plain digest. I have no idea what the difference is supposed to be, but oh well, lesson learned. Apparently writing complaints on this blog helps me solve problems, so expect it to continue.

>> Digest::MD5.hexdigest "Les Rhythmes Digitales"
=> "8b55f031b0253c6ab72de3288b4918e7"

That’s it! I have had enough. I have had enough of DateTime, time strings, datetime strings, Time.parse(), MySQL time, JSON time, CouchDB time, Ruby time, system time and all the rest of it.

I have come to realise that there is one, and only one, appropriate way to store time so everything can understand it without endless string conversion problems, and that is in a numeric format of milliseconds since Epoch UTC.

The only appropriate time to convert from milliseconds into a human-readable string is upon presentation to an actual human - ie, in the View. Or, if you like, store a second time field in any records you save or pass around - just make sure your program doesn’t care about those.

This revelation comes from YET ANOTHER journey into ISO document land as I realised that not only do I have no idea how to store milliseconds in a JSON date/time string, but neither does anyone else. RIGHT!! THAT IS IT! From now on, dates are an integer.

Maybe.

Anyway, here’s some notes on getting millisecond-precision time references in and out of Ruby and JS, both of whose time classes I am no fan of, more for me than anything else..

Ruby example:

 
# Getting milliseconds since Epoch out of Ruby:
 
time_float = Time.now.to_f
time_ms = (1000* time_float).to_i
 
#writing
 
>> t = Time.now.utc
=> Thu Nov 20 23:01:32 UTC 2008
>> time_float = t.to_f
=> 1227222092.50133
>> time_ms = (1000* time_float).to_i
=> 1227222092501
 
#reading
 
>> n = Time.at(time_ms / 1000.0).utc
=> Thu Nov 20 23:01:32 UTC 2008
>> t
=> Thu Nov 20 23:01:32 UTC 2008
 
# check we retained usec through the process
 
>> n.usec # note we lost microsecond precision, this is intended
=> 501000
>> t.usec
=> 501333

Reducing Ruby Time.now to millisecond precision:

time_float = Time.now.utc.to_f
time_msp = ("%0.3f" % time_float).to_f

That doesn’t have much to do with time, I just thought it was cool. Note that apparently doing precision reduction with strings is faster.

Javascript

// Reading an ms-since-epoch time:
js> date_from_above = new Date(1227222092501)
Fri Nov 21 2008 10:01:32 GMT+1100 (EST)
js> date_from_above.toUTCString()
Thu, 20 Nov 2008 23:01:32 GMT // note same as above 
 
// Make a new current date
js> t = new Date
Fri Nov 21 2008 10:28:12 GMT+1100 (EST)
 
// Output date object in milliseconds-since-epoch format
js> t.valueOf()
1227223692041

So, there’s been some Internet Drama over the for-profit video release of a presentation from RubyConf 2008 by Gregg Pollack of Envycasts. Here are some reactions and comments collected from various sources over the last ~24 hours.

My own (highly negative) review:

Having watched the offending video yesterday, it is pretty easy to see why the guy can’t make money as a developer and so is trying to jump on the “screencast money train” a la Peepcode. I would question if it’s even worth downloading for free.

It’s nothing but a superficial high level tour of threads, messaging, and profiling, with some mildly interesting speed tips at the end which should really have been a single blog post. The “research” the guy did is evident all right - as in, it’s obvious he just looked it all up for the presentation and has never actually used any of this stuff. The “tips” he gives - when he gets around to giving any - are unremarkable at best and downright wrong at worst - he actually seems to recommend using RSS as an interprocess message queue, which is a really stupid idea.

He also includes a video overlay of himself giving the whole speech down the bottom of the screen, for that little “distracting touch of narcissm”. He pronounces “memoize” to rhyme with “turquoise”, and spends the first 3 minutes of a 40 minute paid presentation making an unfunny joke.

On the whole I think the guy might actually have done everyone a favour by making the video pay-only. The presentation is not even worth the 40 minutes it takes to watch, let alone $9, and the less people who think that RSS is a good way to implement distributed processing the better.

None of this excuses the presenter’s actions re. RubyConf but in this case, I think it will be a self-correcting problem. There is some expectation that someone demanding money for their training videos might at least have some experience working with the subject on which they present. I expect this video to destroy the clown’s professional reputation just as surely as his money-grubbing actions have destroyed his personal credibility.

I am hardly alone in this assessment. Let me quote several other people from a number of sources, anonymous because I have used them without permission. That said, if you have a problem with being quoted anonymously, let me know and I will remove your comment immediately.

Couldn’t even be bothered watching it for free:

Ugh…

Couldn’t get through it. The music was just too annoying. So I skimmed. Probably spent 4 minutes watching it.

Yes, all pretty superficial stuff and nothing really useful in there.

I don’t really like those tutorials which are just a tour of add-ons/plug-ins/gems etc written by other people (and as you know, Ruby people are all too ready to embrace other people’s code). I’m more interested in seeing interesting, original and innovative code.

Another prominent developer is unimpressed:

there is nothing in this talk which cannot be discovered in a couple of minutes using google, or by reading a couple of howtos

An IP sleuth points the finger …

The “speed tips” at the end are stolen directly from igvita.com, without attribution of course, and the ruby threading graphs look suspiciously similar too. This video is basically nothing but a visual presentation of the content from someone else’s blog - unpaid, of course. To pay for it would be to encourage this kind of blatant theft.

Tsk, tsk.

UPDATE: More:

I had a strange feeling of deja vu when I saw this talk. I felt like Id seen it before, somehow, and recently. later that day I logged on and was trying to find out where I’d read it before. Turns out the whole thing was lifted from Igvita.com, with minor changes. No credit given at all. If I was the guy from igvita.com, I would be pissed.

Storage space conscious:

Not only is this video not worth 40 minutes of my time or $9 of my money, it’s not even worth 200M of my hard disk space. Deleted.

A message of support:

I fully support Mr Polack’s actions in this matter. Anyone dumb enough to 1. buy this POS and 2. implement its suggestions (RSS? Are you fucking KIDDING me?) deserves to have their money stolen and their app grind to a messy halt. Polack is doing us all a favour, why so harsh?

A compelling argument there.

UPDATE 2:

Surprise!

The following gems are shamefully incompatible with Ruby 1.9, which has now been available for testing for almost a year. These are just the ones which have personally affected me; there are bound to be MANY more.

Mongrel
FasterCSV
LibIDN
EventMachine - pure ruby works but is slow

Authors: please fix your gems!

More to come as I think of them. Suggestions welcome.

Since I’m having no joy on the CouchRest support forum or in mail to the author, here’s a monkeypatch to implement simple field validations in CouchRest::Model.

module CouchRest
 
  class Model
 
  class_inheritable_accessor :validation_fields
 
  def self.validate *fields
    self.validation_fields = fields.map {|f| f.to_s}
  end
 
  def valid?
    self.invalid_fields.blank?
  end
 
  def invalid_fields
    return [] unless self.validation_fields
    check = self.validation_fields.clone
    self.validation_fields.each do |c|
      check.delete_if {|x| x == c } if !self[c].blank?
    end
    return check
  end
 
  def save
    if new_record?
      create if self.valid?
    else
      update
    end
  end
 
  end # class Model
 
end # module CouchRest

Just throw a line like validate :field1, :field2, :field3 anywhere in the Model and it’ll refuse to save without those fields being present.

Stay tuned for the upcoming $9 screencast on use of this incredible new feature, development of which has taken minutes of my precious, precious time.

I have a directory containing gems, and I often change them. To save myself the task of manually re-installing each one whenever anything changes, I wrote a simple script which keeps track of the files and runs rake tasks whenever any is modified.

The script is almost self-explanatory, but here’s the gist:

- first time you run it, it writes a YAML file with a big hash of a relevant files and their hashes
- after that, it performs the same md5 hashes and if the hash has changed it runs rake gem:uninstall rake gem:repackage rake gem:install in the directory where the change was noticed, then rewrites the hash.

It’s pretty scrappy, and needs to be placed and run from the base dir of the gems. And it doesn’t check manifests, yet.

require 'pathname'
require 'find'
require 'digest/md5'
require 'yaml'
 
@allfiles = []
@new = {}
@to_update = []
 
@root_dir = ENV['PWD']
 
def get_new_file_list
  Dir.chdir @root_dir
  Find.find('./') do |f| 
    @allfiles << f 
  end
  @allfiles.delete_if {|e| e.split('/').size == 2 } # ie in root dir
  @allfiles.delete_if {|e| e.include?('pkg') } # ignore build dirs
end
 
def recurse_files
  @new = {}
  @allfiles.each do |f|
    if !FileTest.directory?(f) && File.exists?(f)
      @new[f] = Digest::MD5.hexdigest(File.read(f))
    end
  end
end
 
def write_new
  Dir.chdir @root_dir
  File.open("md5.yaml", "w+") do |file|
    file.write @new.to_yaml
  end
end
 
def differences
  newclone = @new.clone
  oldclone = @old.clone
  newclone.delete_if {|key, value| @old[key] == value }
  oldclone.delete_if {|key, value| @new[key] == value }
  diff = newclone.merge(oldclone)
  diff.keys.each do |k|
    @to_update << k.split('/')[1]
  end
  @to_update.uniq!
end
 
def cycle_gem gem_dir
  puts "updating #{gem_dir}..."
  dir = @root_dir + '/' +  gem_dir
  Dir.chdir(dir)
  system('rake gem:uninstall')
  system('rake gem:repackage')
  system('rake gem:install')
end
 
get_new_file_list
recurse_files
 
if File::exists?('md5.yaml')
  @old = YAML.load_file('md5.yaml')
  differences
else
  write_new
end
 
if @to_update.size > 0
  @to_update.each do |g| cycle_gem g end
  get_new_file_list
  recurse_files
  write_new
  puts "updated #{@to_update.join(' ')}"
else
  puts "nothing to update"
end

The videos are up for the recent Lone Star Ruby Conference 2008, and a lot of them look great. Rails Envy have compiled a short (21m) summary video of the talks which you can view or download here.

Just want to get downloading? Here’s a handy wget file so you can grab them all.

Here’s my initial stab at a Rails Session model for CouchDB. The marshalling stuff is taken from the example SQLBypass class in the ActiveRecord code.

You’ll need a recent CouchRest and trunk CouchDB, probably.

class CouchSession < Hash
  @@session_db = CouchRest.database!('http://localhost:5984/sessions')
 
  attr_writer :data
 
  def self.find_by_session_id(session_id)
    self.new(@@session_db.get(session_id))
    rescue
    self.new(:id => session_id)
  end
 
  def self.marshal(data)   ActiveSupport::Base64.encode64(Marshal.dump(data)) if data end
  def self.unmarshal(data) Marshal.load(ActiveSupport::Base64.decode64(data)) if data end
 
  def initialize(attributes = {})
    self['_id'] = attributes['_id'] ||= attributes[:id]
    self['marshaled_data'] = attributes['marshaled_data'] ||= attributes[:marshalled_data]
    self['_rev'] = attributes['_rev'] if attributes['_rev']
  end
 
  def data
    unless @data
      if self['marshaled_data']
        @data, @marshaled_data = self.class.unmarshal(self['marshaled_data']) || {}, nil
      else
        @data = {}
      end
    end
    @data
  end
 
  def loaded?
    !! @data
  end
 
  def session_id
    self['_id']
  end
 
  def save
    self['marshaled_data'] = self.class.marshal(data)
    self['data'] = data
    self['updated_at'] = Time.now
    save_record = @@session_db.save(self)
    self['_rev'] = save_record['rev']
  end
 
  def destroy
    @@session_db.delete(self['_id'])
  end
end

Nice and short - possibly the shortest Rails session class I have seen. The beauty of CouchRest/CouchDB! And we descend from hash so we can just save the object straight - after marshalling, of course. Cool, huh?

Note that I am actually writing the raw data as well as the marshalled data into the saved doc, for troubleshooting/interest purposes. Feel free to remove that.

Not pretty, but it works. Just save it like a normal model. You’ll need to put these into environment.rb:

config.action_controller.session_store = :active_record_store
CGI::Session::ActiveRecordStore.session_class = CouchSession

Note also that I have ignored any differentiation between the record ID and the session ID, negating the need for any special overrides in ApplicationController. However, the session IDs Rails generates are large and you might find them unattractive in CouchDB - it would be fairly simple to separate them, but then you’d need a new map view and an override. I feel it’s simpler to just use the Session ID as the doc ID and damn the torpedoes. YMMV.

Improvements? See something wrong with it? Let me know!

It was harder than I thought to write a useful method_missing class for Hash object, so I thought I’d share the fruit of my labours here.

class Hash
  def method_missing(m, *args, &block)
    method = m.to_s
    is_assignment = method[-1,1] == '='
    key = is_assignment ? method[0..-2] : method[0..-1]
    self[key] = args.first if args && is_assignment
    raise NoMethodError if !self[key]  
    return self[key]
  end
end

The difficulty lies in detecting and acting correctly in situations where we want to assign a value to the hash. Annoyingly, we have to go and actually check for an = which inexplicably gets attached to the end of the method if present.

Anyway, let’s see if it works:

>> hsh = {'ping' => 'pong'}
=> {"ping"=>"pong"}
>> hsh.ping
=> "pong"
>> hsh.ping2
NoMethodError: NoMethodError
	from /path/initializers/hacks.rb:31:in 'method_missing'
	from (irb):3
>> hsh.ping2 = "pong2"
=> "pong2"
>> hsh.ping2
=> "pong2"
>> hsh.keys
=> ["ping2", "ping"]

Yes, pretty basic, but I tried and failed to find anything that handled assignation that wasn’t a overblown nightmare or didn’t actually work so thought I’d share. As far as I can see this is the Simplest Thing That Can Possibly Work™ for this case, any improvements welcome.

Also note that we are standardising on hash keys being strings here. If you want to use that code with Symbol keys (popular in Rails), you’d have to extend it a bit. I had actually been under the impression that symbols and strings in hashes were going to be considered equal in Ruby 1.9 but apparently just in comparison, not access.

Another useful addition might be a hash of “acceptable” keys and their types as a class variable, which you’d feed with a DataMapper-style list of properties for the class. Check any inserts against that and you’ve got basic validations, sans the 20,000 lines of impenetrable black box magic of the popular mega-libraries.

One more thing: obviously I would not recommend actually putting this on Hash proper. Monkeypatching such a basic class is asking for trouble. Instead, create a new class descending from Hash, create your objects in that, and use it from there.

UPDATE: Mash looks to be a decent library for doing something similar if you don’t trust my hacks or if you prefer a gem. Personally a gem has to do something pretty fucking special for me to require it these days but you might prefer otherwise. It’s too big for me and tries to do too much, most of which I don’t want. But, it’s a *lot* smaller than going for a full AR object or something.

On another note, something to bear in mind is that method_missing is slow. Using it to look inside a hash is about 5 times slower than accessing the key directly. That probably doesn’t matter, but it might pay to think twice before putting a method_missing call inside a loop that’s going to run a lot.

1,000,00 lookups (in seconds):

string hash: 1.041939
symbol hash: 0.699153
method missing hash: 4.645116

UPDATE 2: I’m a sucker for writing these little mini-benchmarks. The above was this code:

  def self.test
    count = 0
    hsh = {'ping' => "PONG"}
    time = Time.now
    1_000_000.times do
      count += 1 if hsh['ping'] == "PONG"
    end
    puts "string hash: " + (Time.now - time).to_s
    count = 0
    hsh = {:ping => "PONG"}
    time = Time.now
    1_000_000.times do
      count += 1 if hsh[:ping] == "PONG"
    end
    puts "symbol hash: " + (Time.now - time).to_s
    count = 0
    chsh = CouchHash.new # monkeypatched hash ; )
    chsh['ping'] = "PONG"
    time = Time.now
    1_000_000.times do
      count += 1 if chsh.ping == "PONG"
    end
    puts "method missing hash: " + (Time.now - time).to_s
 
  end

Note the speed. Not *that* bad. I thought about posting the code, as I always do, but thought “guh, I’m not posting that ugly crap. I can clean that up” .. so did. But the only way I could think of to trigger the different lookup methods was via an eval(), and what that did to the results is instructive.

  def self.test
    hsh = CouchHash.new
    hsh['ping'] = hsh[:ping] = "PONG"
    tests = ["['ping']", "[:ping]", ".ping"]
    tests.each do |call|
      time = Time.now
      1_000_000.times do
        raise if !(eval("hsh" + call) == "PONG")
      end
      puts call + ': ' + (Time.now - time).to_s
    end
  end

Wow, doesn’t that look better! Well, one thing doesn’t look better .. the results:

['ping']: 6.494104
[:ping]: 5.557214
.ping: 9.39655

Woah. The eval slowed us down by a factor of 10 in the case of the symbol lookup. Lesson: don’t use eval. Especially in combination with method_missing!!

There’s a bit of a gotcha I encountered recently with OpenStruct. If you create a new one from a Hash with the key, uh, “hash” - then try to read the value back out by doing mystruct.hash - you’ll get the object ID instead. No idea why, I’d certainly never realised .hash was a synonym for .object_id before, but it fails in the worst possible way - looks like it works, accepts the information, and then casually hands you back the wrong value when you least expect it:

>> require 'ostruct'
=> true
>> hsh = {:hash => 'hash!'}
=> {:hash=>"hash!"}
>> os = OpenStruct.new(hsh)
=> #<OpenStruct hash="hash!">
>> os.hash
=> 18782890

Can’t have that. I had already extended Hash to ease OpenStruct creation, so I modified that to check for key/keyword collisions first.

class Hash
  def to_struct
    if self.keys.map {|k| OpenStruct.respond_to?(k)}.include?(true)
      raise 'reserved word as hash key'
    end
    OpenStruct.new(self)
  end
end

Gotta love those Ruby one-liners : )

So why do this kind of thing at all? Well, Rails has spoiled me with all its method_missing sugar. So if I’m doing something like this:

SITE_CONFIG = {:default_language => 'en'}

I don’t want to access it like this:

SITE_CONFIG[:default_language]

OK, pretty silly reason, but it makes a difference to me! Hey, that’s why I use Ruby in the first place.

>> SITE_CONFIG = {:default_language => 'en'}.to_struct
=> #<OpenStruct default_language="en">
>> SITE_CONFIG.default_language
=> "en"

It’s the little things.

Of course the real lesson here is not to use common keywords/class methods as hash keys or anywhere else.

I think this is the only way to do it.

>> Time.now.strftime("%Y/%m/%d %H:%M:%S.#{Time.now.usec} %z")
=> "2008/08/20 01:22:28.367899 +0000"

At the microsecond level, it’s possible for Time.now to change mid-evaluation, so if you *really* care about timing you could freeze the time object first and read from that:

>> now = Time.now
=> Wed Aug 20 01:40:26 +0000 2008
>> now.strftime("%Y/%m/%d %H:%M:%S.#{now.usec} %z %Z")
=> "2008/08/20 01:40:26.597940 +0000 UTC"

Happily, Time.parse will read that straight out of the box:

>> n = now.strftime("%Y/%m/%d %H:%M:%S.#{now.usec} %z %Z")
=> "2008/08/20 01:40:26.597940 +0000 UTC"
>> Time.parse(n)
=> Wed Aug 20 01:40:26 +0000 2008
>> Time.parse(n).usec
=> 597940

I have no idea what the Ruby guys are thinking with these version numbers.

They want to make a lot of changes, leading in to 2.0. Well, that’s fine. Great! Major version numbers are just the place to make sweeping changes, cleaning up a lot of stuff, progressing wherever they want. That is exactly the right thing to do.

Hang on .. they want to, uh, declare 1.9 their kind of “work branch” for 2.0. Uh .. right. What was wrong with using, say, a 2.0 beta branch? But .. hm, ok, well, it’s still OK, I guess, to make reasonable changes in a minor version .. kind of. Kind of negates the whole point of having the major version number in the first place, unless it’s just counting up in .1 increments, but whatever, they’re in charge. Personally I would have thought that 1.9 should be 1.8 on YARV, but whatever, it’s their ball game.

But 1.8.7? Incompatibilities from backported features from 1.9 - which is itself a kind of “future backport” of 2.0? What the fuck? Why the fucking hell are they backporting shit from 1.9 into 1.8 when it is not completely compatible?! Do they even understand the point of these version numbers?!

How Version Numbers Should Be Used

A Ruby version number: [major].[minor].[tiny]-p[patchlevel]

Major version number: do whatever the hell you want! Got breaking stuff and design changes? Here’s the place to do it!
Minor version number: For big new features and major changes, eg YARV. Generally backwards compatible, but .. well .. shouldn’t break stuff if you can help it but if you REALLY need to, OK .. kind of.
Tiny version number: for tweaks, fixes and optimisations only; ok to add (small) new features, should NOT break anything backwards
Patchlevel: absolutely should fucking not break anything backwards and to be perfectly honest should not even exist, and the fact that it does exist is a sign of bad release practises

They should not have made major (read: incompatible) changes in 1.8.7. They absolutely should NOT under any fucking circumstances have made incompatible changes in 1.8.6-p230!! What the hell is going on?

The Ruby team need to basically shift all their versioning one decimal point to the left. They are putting shit in patchlevel which should be in tiny - and there are WAY too many patchlevels. They should wait a bit and then roll up changes from trunk every few months, only levelling up when they’ve got something important or urgent. As it stands they’ve been going through one every few days, it’s ridiculous. May as well just use the checkin number. They are making changes in tiny which should be in minor. And for 1.9 they are using minor when they should be in major. Why the hell are they acting like this? It’s not like new numbers cost anything?!

As much as it pains me to say it, Ruby could learn from Rails here. Ditch the fucking patchlevel bullshit. 1.8.6 should equal 1.8.6, no questions asked, no possibility for difference, no fucking patchlevels or anything else. If you need a new version, 1.8.7, which is always backwards compatible. *Never* break backwards compatibility in a minor branch. And if you need more than 10 tiny versions in a minor branch, then just keep going - if 10.4.11 is good enough for Apple, then 1.8.11 is good enough for Ruby.

This security update should be 1.8.7. And what about the current 1.8.7? It should not even exist! What is the purpose of 1.8.7? To get us ready for 1.9? NO! 1.8.7 should be a straight progression, bugfixes and security updates for 1.8.6! The current 1.8.7 should be 1.9, which is getting us ready for 2.0!

The other day, I needed to install 1.8.6 on a new server. I installed the latest tarball of 1.8.6 from Ruby-lang, which was patchlevel 114. They have skipped over a hundred patchlevels in the last couple of days and now insist 230 is the minimum. Hint: if over a hundred of these patchlevels were not important enough to be bundled into the downloadable tarball, maybe they should not be patchlevels at all. Go look at ftp.ruby-lang.org. Look at their tarballs in 1.8 branch. There is *no* tarball between 114 and 230. So, uh, why isn’t p114 called 1.8.7 and p230 called 1.8.8?? Or 1.8.9, or whatever? What is the point of a patchlevel you can’t even download a tarball of?

Ruby versioning is completely fucked up.

Some people just can’t stop achieving! Not content with introducing breaking, backwards-incompatible changes into the Ruby 1.8 branch with 1.8.7, the Ruby Core team have shot for the moon, gone for gold, and doubled down all their bets - and made Ruby 1.8.6-p230 incompatible with Ruby 1.8.6-p(any other version). Of course, there is an URGENT!!!111ONE zero days exploit or something in any version of 1.8.6 that’s not the one released today - the incompatible one.

So let’s sum this up:

Ruby 1.9.0 = incompatible with Ruby 1.8.x
Ruby 1.8.7 = incompatible with Ruby 1.8.6
Ruby 1.8.6-p230 = incompatible with Ruby 1.8.6-p(<230)
Ruby 1.8.6-p(<230) = UR SERVER IZ OWNED LOL

Well that is just fucking great. Well, I guess if your app won't start, no one can hack it! Can't get more secure than that! Mission accomplished!

It seems like this Ruby Spec project is arriving not a fucking moment too soon. When the core team can’t even keep Ruby 1.8.6 compatible with, uh, Ruby 1.8.6, we got problems.

I hacked together this script which dumps the structure of your database into a DataMapper-compatible class header file. You’ll need these for your Models if you’re planning on playing around with datamapper - should save you a bit of typing.

There’s a bit of CouchDB boilerplate in there, feel free to strip that out if you’re not using that. The large sql_type case statement just covered what I had, which was mostly postgres and will need modification for MySQL - but it will tell you what you need to change, should be obvious enough. Note that I am coercing pretty much everything to basic Integer/String - this is for CouchDB but you can change it to match the types you want. And yes, I know it’s ugly!

# datamapper_dump.rake
# Sho Fukamachi 2008
# place in lib/tasks and run with: rake db_to_dm
 
def tables_we_want
  skip_tables = ["schema_info", "sessions"] # this is slow enough as is without sessions
  ActiveRecord::Base.establish_connection
  #ActiveRecord::Base.connection.schema_search_path = "path1, path2" - uncomment for PGSQL
  ActiveRecord::Base.connection.tables - skip_tables
end
 
task :db_to_dm => :environment do
 
  sql  = "SELECT * FROM %s"
  dir = RAILS_ROOT + '/db/dm_temp/'
  FileUtils.mkdir_p(dir)
  FileUtils.chdir(dir)
 
  ActiveRecord::Base.establish_connection
  #ActiveRecord::Base.connection.schema_search_path = "path1, path2" - uncomment for PGSQL
 
  puts "Dumping Schema into DM format..."
 
  File.open("models.rb", "w+") do |file|
    file.write "# Models dump File for DataMapper/CouchDB\n"
    file.write "# Sho Fukamachi 2008\n"
    file.write "\n"
    tables_we_want.each do |table_name|
    class_header = <<-EOF
class #{table_name.singularize.capitalize}
 
  include DataMapper::Resource
 
  def self.default_repository_name
    :couchdb
  end
 
  # required for CouchDB
  property :id, String, :key => true, :field => :_id
  property :rev, String, :field => :_rev
 
  # regular properties
EOF
    file.write class_header
    ActiveRecord::Base.connection.columns(table_name).each do |c|
      if !c.sql_type.scan('integer').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('character').empty?
        file.write "  property :#{c.name}, String\n"
      elsif !c.sql_type.scan('text').empty?
        file.write "  property :#{c.name}, Text\n"
      elsif !c.sql_type.scan('datetime').empty?
        file.write "  property :#{c.name}, DateTime\n"
      elsif !c.sql_type.scan('uuid').empty?
        file.write "  property :#{c.name}, String\n"
      elsif !c.sql_type.scan('boolean').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('double precision').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('bigint').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('smallint').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('date').empty?
        file.write "  property :#{c.name}, Date\n"
      elsif !c.sql_type.scan('timestamp without time zone').empty?
        file.write "  property :#{c.name}, DateTime\n"
      elsif !c.sql_type.scan('time without time zone').empty?
        file.write "  property :#{c.name}, Time\n"
      elsif !c.sql_type.scan('bytea').empty?
        puts "INVALID: column #{c.name} in #{table_name.capitalize}: binary data is not allowed here."
        puts 'You will need to rethink your schema for use in CouchDB. Hint: Attachments.'
        puts 'SKIPPING!'
      else
        file.write "COLUMN: #{c.name}, UNKNOWN DATA TYPE #{c.sql_type} - CHANGE ME!\n"
      end
    end
    file.write "end\n"
    file.write "\n"
    end
  end
  puts "dump succeeded. Look in db/dm_temp/models.rb"
end

Installed the new “stable, recommended” version of Ruby as linked from ruby-lang.org, 1.8.7, and finding your Rails sessions breaking?

You’ll get an error like this:

/!\ FAILSAFE /!\  Tue Jun 10 15:09:34 -0500 2008
  Status: 500 Internal Server Error
  wrong number of arguments (2 for 1)
    /usr/local/lib/ruby/1.8/cgi/session.rb:267:in `respond_to?'

This will be fixed in Ruby 1.8.7-2 but for now install this patch and restart the server.

I also had some enumerator errors, post a comment if you want a fix for that too.

After reading this article about geohashing, I thought I’d share an alternative implementation I have which uses a matrix to translate coordinates to and from a string.

People are generally scared of geolocation, but that’s because they’re overthinking it. Forget all that stuff about mapping onto a sphere - a set of coordinates are just a point on a 2D plane with dimensions 360*360. The fact that that plane is then mapped onto a sphere (well, a spheroid) is something that 99.9% of developers don’t have to worry about. All you care about most of the time is whether location X is near location Y.

So, since we have these vectors on a plane of known bounds, if we want to represent some coordinates as a single string, all we have to do is map them onto a very simple matrix.

Happily, our plane is of dimensions 360*360 - and we have 36 unique characters to play with! So what we’ll do is divide the plane into squares, then sucessively divide out the coordinates as we increase resolution on the plane. That way we reduce the vector into a string, keep the resolution, and retain the ability to compare to other nearby locations. We can also, of course, extract them out again later.

Encoding

Let’s start with the point 39.286534 -76.613558, like the article linked above.

@point = [39.286534, -76.613558]

First, we need to normalise the numbers. Add 180 to both

@workpoint = @point
@workpoint.map! {|e| e =  e + 180}
=> [219.286534, 103.386442]

Now, our matrix. We have 36 characters in base36, so that is 36 squares we can have, 6 on each side. So we will start cutting the number at 60 degrees intervals.

First we need a matrix. We are just going to reuse this again and again, so we only need one. Dimensions 6*6 for a total of 36 squares.

require 'matrix'
@charmap = Matrix.rows([
["0", "1", "2", "3", "4", "5"], 
["6", "7", "8", "9", "A", "B"], 
["C", "D", "E", "F", "G", "H"], 
["I", "J", "K", "L", "M", "N"], 
["O", "P", "Q", "R", "S", "T"], 
["U", "V", "W", "X", "Y", "Z"]
])

I had some code that generated that, but I can’t find it .. so just declare it manually. You’ll need to require ‘matrix’. If we were really serious we would do things a little different but we’re focussing on simplicity here and don’t want to start messing around with origins, etc.

>> @charmap[0,0]
=> "0"
>> @charmap[0,1]
=> "1"
>> @charmap[1,1]
=> "7"
>> @charmap[2,1]
=> "D"
>> @charmap[5,5]
=> "Z"

looking good.

First slice

So! The first cut.

slice = 60.0
first = @workpoint.map {|e| (e / slice).to_i}
=> [3, 1]

Note we will cast to integer to get rid of the remainder, which we don’t care about. All we want is to find the first digit.

which is …

code = @charmap[first[0]<