In April, 2008 I posted about how TinyURL is a very bad idea. One of my reasons for it being a bad idea:
If you don’t give the proper URL, I can’t tell:
- Whether it’s a link to some likely spam/other hostile site
The phishing scam seen on Twitter is exactly this - sending people obfuscated URLs then hoping they don’t notice the actual domain when they open them. Working, too.
From The Skeptic’s Guide:
McCarthy decided that her son Evan’s autism was caused by vaccines - because she had heard the rumors and her “mommy instinct” told her it was right. She then confirmed her beliefs by attending Google University’s confirmation bias program.
Ah, that’s a very popular course …
Finally realised which song Sabali, by Amadou & Mariam off their album Welcome to Mali reminded me of. It’s Solex (Close to the Edge) by Michael Woods, off Ministry of Sound 2004.
Guess it’s not all that similar .. but it was driving me nuts until I finally remembered the other song…
I present my review of MetaCritic’s Best Albums of 2008.
Read the rest of this entry »
A kookaburra decided to sit on my balcony, so I was trying to feed it. Unfortunately, it wasn’t very interested in cheese and bread, and I didn’t have any non-frozen meat, so coming to me for food must have been a disappointment. Ah well ..
From a house near my apartment.
Unfortunately, being the bastard that I am, if I was the postman I’d make certain this house received a special extra helping of junk mail.
A recent post on Coding Horror about the nature of probability paradoxes struck a chord with me; I’ve always been fascinated by probability and its counterintuitive nature. Even though I rationally understand such concepts, I really do find it hard to “internalise” the thing. I thought I’d write about it to try and explain how to understand the problem.
The paradox in question is the Boy/Girl problem, which wikipedia has an explanation to here. However, I don’t like the Wikipedia explanation either. I will try to do better.
The question is:
“A family has two children. You know that (at least) one of the children is a boy. What is the probability that the other child is a girl?”.
The kneejerk response to this is “50%”. This is completely understandable and comes from a lifetime of learning that “chance has no memory”. The chance of a child being a boy or a girl is 50%, right? So how does knowing the sex of one of the children possibly affect the sex of the other?
The answer is that you have to consider the whole range of outcomes, you have to consider the results of the two childbirth events as a whole. The fact is, there are two children, with two possibilities each, so the range of possible outcomes looks like this:
BB BG GB GG
Where B and G mean boy and girl, obviously. Each of these four outcomes has equal probability.
However, since you have been told that one of the children is a boy, you are forced to remove one of the outcomes, leaving you with:
BB BG GB
See it? Since you know for sure that there are not two girls, there are only three possibilities left. In two of those three possibilities, the other child is a girl. So the chance of the other child being a girl is actually two thirds - 66.67%.
That is counterintuitive enough, but is fairly understandable when explained. Where it really gets freaky is when you introduce order. This is where my brain absolutely chokes and I have real difficulty accepting at an intuitive level the implications of having this extra information.
Let’s ask the question again, with information about the order:
“A family has two children. You know that the eldest child is a boy. What is the probability that the other child is a girl?”.
Sounds the same, right? My intuitive soul practically screams that the outcome should be the same. But it’s not.
Let’s look at the total range of possible outcomes again.
BB BG GB GG
We know the eldest child is a boy. Now we have to remove not one, but TWO, possible outcomes:
BB GB
In other words, the probability of the other child being a girl is back to 50%.
This seems utterly insane. How the fucking hell could knowing the order of birth influence probability in this way? After all, they have to be either younger or older, didn’t they? One or the other - it’s assumed! How can this change anything?
Well, the thing to understand here is that we’re not talking about two events any more. We’ve actually removed one of the events, so don’t need to consider four possible outcomes, of which one has been removed. We’ve considering one event with two possible outcomes - 50%.
In order to understand it more, let’s actually switch analogies to tossing a coin. We’re not used to thinking about order in relation to children, but in coin-tossing it is natural. When it comes down to it, though, it’s all the same thing. Let’s re-ask the questions - in terms of coin tosses.
Question: A coin was tossed twice. At least once, it came up heads. What’s the chance it came up tails the other time?
Outcomes: HH HT TH TT
Eliminate: TT
Remainder: HH HT TH
Answer: Two out of three times, ie. 66.67%
And now, let’s introduce our knowledge of the order:
Question: A coin was tossed twice, and came up heads the second time. What’s the chance it came up tails the first time?
Outcomes: HH HT TH TT
Eliminate: HT TT
Remainder: HH TH
Answer: 50%
Suddenly our knowledge of the order seems valuable. We’ve been given more information, and as a result the probability question is far more specific and back in line with the “independent event” intuitive expectations we had in the first place. By knowing everything about one of the events, we remove it from the equation.
Understanding how knowing the order influences probability has powerful ramifications. For example, we can now understand why the “statistical” result of 66.67% for the first “other child” example above doesn’t square with our intuitive expectation of 50%. When we first consider the problem, we can’t separate things out and are thinking in terms of “if the family has one boy, and another child is born, then that child has 50% chance of being either sex”. But see? That’s because we removed the event! If we don’t know the order, it’s back to 66.67%.
To me, rephrasing the question in terms of coins produces an “a-ha” moment in which I can intuitively grasp why the probability has suddenly “changed”. I hope the explanation works equally well for you.
One of the things I miss most about Rails when working in gems is the reload! function, which rebuilds the environment to update anything that’s changed since the last save. Well, I wanted to recreate that functionality, but hopefully in a really simple way.
The good news is, it’s actually pretty easy. I have two ways of doing it, pick the one you like best.
module AutoReload @@required_mod_times = {} def reload! diffs = AutoReload.differences # we can only call it once per reload, obviously if diffs.size > 0 diffs.each {|f| Kernel.load(f)} puts "reloaded #{diffs.size} file(s): #{diffs.join(', ')}" else puts "nothing to reload" end end def self.update_modtimes $".each do |f| @@required_mod_times[f] = File.mtime(f) if File.exists?(f) end end def self.differences oldlist = @@required_mod_times.clone AutoReload.update_modtimes newlist = @@required_mod_times.clone oldlist.delete_if {|key, value| newlist[key] == value } oldlist.keys.uniq end end include AutoReload
You will then need to initialise it somewhere after all your requires. AutoReload.update_modtimes will do the trick. If you can’t manage that, it will only work properly after the first time you use reload!.
This is the way to do it if you have a lot of files, I think, since it maintains a list of what was changed when, and then only reloads changed files.
Note that it’s not perfect. It will only be able to find files which are in local path, ie won’t be able to reload gems. However, that’s all I need for now.
The next way is even simpler, since it doesn’t bother to maintain a list - it just blindly reloads everything it can:
def reload! diffs = [] $".each {|f| diffs << f if File.exists?(f)} if diffs.size > 0 diffs.each {|f| Kernel.load(f)} puts "reloaded #{diffs.size} file(s): #{diffs.join(', ')}" else puts "nothing to reload" end end
As you can see, this just blindly reloads everything it can find. Probably not the best if you have a lot of constants etc, but for a simple project could be just the ticket. The good news is you don’t need to initialise it. if you have a lot of files you probably should return "OK" or something, else you’ll have pages of reloads scrolling past.
Let’s bear in mind that this kind of trick is always a bit of a hack. Kernel.load() has no ability to unload anything, even if it doesn’t appear in the file anymore. All it can do is overwrite. If you break your code by deleting something important, then reloading using this kind of trick won’t show it up - the object is still there until you reload ruby itself. It’s a convenience thing only, so don’t rely on it too much, do a full reload once in a while.
However, for my use case - making a lot of small changes and working in a very interactive manner with irb - this is a real time-saver, hope you find it useful too.
If you’d prefer it to happen automatically, rspec-style, there is a gem available which will do this for you here which basically does the same thing, just every 1 second instead of manually.
I’d love to know what people thought were the best albums for 2008.
Here’s my shortlist:
1. Josh Fix (US) - Free at Last (pretty stable #1)
2. Girl Talk (US) - Feed the Animals (if you don’t like this album, you don’t like music)
3. Roger Joseph Manning Jr. (US) - Catnip Dynamite (as featured here - good strong #3)
4. Nightwish (FI) - Dark Passion Play (Far from their best but OK)
5. open
I just can’t think of any other standouts this year. I’m interested in suggestions/reminders. Anyone?
Wow, top 3 are all American. That’s a turnaround. And can’t help but notice there’s no Asian representation. Maybe that’s because I didn’t like anything this year, or maybe I’m just sick of all things Japanese and didn’t give anything a fair chance. Could be either. I was down on US, up on JP for a long time; seems to be somewhat reversed now. Ever changing world!
Christmas! I love Christmas. The best part of Christmas is being able to play all the christmas songs I love, which by culture I can’t really play the rest of the year.
What is Christmas Music? Christmas Music equals:
1. Trans Siberian Orchestra
2. Home Alone theme
3. The Red Army Choir, which is a little too “in your face” normally so I substitute the “Hunt For Red October” soundtrack, but I’ll include a bonus track here for purists
I need a playlist, I hear you ask? Your wish is my command, dearest reader. After you hear this playlist, you won’t care about any other presents. You’ve already reached the top. You may as well kill yourself after listening to this, actually, ‘cos it’s all downhill from here. Don’t tell anyone I said that.
All right. Let’s go! This christmas is gonna ROCK!!!!
Part 1 - Trans Siberian Orchestra
Track 1: What Child is This? (MP3, 13.6M)
A great starting track which will let the relatives know what they’re in for. Sets the scene with pensive, edgy piano, then kicks ass (in the service of the Lord) when the chorus finally arrives. Perfect beginning to your Christmas lunch!!
Track 2: O Come All Ye Faithful / O Holy Night (MP3, 9.9M)
Another great track which will get feet tapping around the table. Hey, is this Christmas, or ROCKmas? Hint: Both.
Track 3: A Star To Follow (MP3, 8.8M)
Controversial choice, this one. You see, I hate the beginning. Goes for way too long, and in the past I’ve gone so far as to edit that whole beginning out. However, maybe we can get away with it .. the crowd has been lulled into a trance by the first two all-winner-no-filler anthem classics, so maybe they can tolerate 1:08 of annoying chants. Gets good after that. I might post the edited version if anyone wants it, I got it right here.
That’s the thing with Trans Siberian Orchestra. I love their sound, and their concept, but sometimes I can’t stand their arrangements. Ah well, can’t have it all.
Track 4: Faith Noel (MP3, 10.4M)
God I’m sick of hearing stupid little kids. Let’s get back to what we’re all here for: AXE SOLO!!!!! Oh come let us adore him…
Track 5a: Christmas Canon Rock (MP3, 11.6M)
Track 5b: Queen Of The Winter Night (MP3, 7.3M)
Kind of filler but everyone loves these songs.
Track 6: Boughs Of Holly (MP3, 7.1M)
Oh yeah, back into it. Deck the Halls with Boughs of AWESOME! By now everyone should be nice and drunk and not paying much attention anyway. All they know is that this is the best christmas EVAR. This is the time to make a move on your cousin, if you haven’t already! (of course you have!)
Track 7: The March of the Kings/Hark! The Herald Angels Sing (MP3, 6.2M)
More of the same, really. Feel free to skip it if you think it’s a bit noisy.
Track 8: Joy/Angels We Have Heard on High (MP3, 6.3M)
Jesu, joy of man’s desiring. Nice slow wrap-up to the TSO.
Part 2: Japanese section
Free free to ignore this if you hate Japan. Godless heathens, who do they think they are making Christmas music, anyway?
Track 9: 山下達郎: CHRISTMAS EVE (MP3, 5.9M)
Track 9: 山下達郎: CHRISTMAS EVE (english version) (MP3, 5.9M)
English and Japanese versions of this classic. The Japanese is better, but if it gives you jungle flashbacks you might want to stick with the Queen’s Tongue.
Part 3: Soundtracks
Track 10: Home Alone Main Title (”Somewhere In My Memory”) (MP3, 6.7M)
What could evoke more Christmas cheer than the thought of young McCauley Culkin evading bumbling home invaders? That’s right, nothing.
Track 11: O Holy Night (Home Alone) (MP3, 3.8M)
A nice arrangement of this classic.
Track 12: Hymn to the Red October (MP3, 7.1M)
In 1984, the USSR’s best submarine captain in their newest sub violates orders and heads for the USA. Is he trying to defect, or to start a war?
Another Christmas classic.
Part 4: Bonus
Track 13: The Red Army Choir - National Anthem Of The USSR (MP3, 8.6M)
Self explanatory. The very essence of Christmas, distilled.
——————–
That’s all, folks. After that lot’s played through - you won’t have any choice but to hit START OVER. This never gets old. Trust me.
Any requests, tell santa to put your comment in my little comment stocking below! Ho, ho, ho!
I have always loved Kate Bush’s 1978 song Wuthering Heights. I’d love to be able to sing like that! Maybe after my operation.
Wuthering Heights (MP3, 11.4M)
Lyrics after the break. Most of the lyrics floating around are wrong, I’ve corrected them.
One more Nightwish song, since my moods change like the weather. A mere 10:22 in duration.
Beauty of the Beast (MP3 23.8M)
Another in my tradition of having nothing to say, so posting music. The lyrics in this are a bit, uh, over the top, but still kind of work.
Song of the Week is “I thought about it first” by Josh Fix, cementing the end of my multi-month Nightwish binge. I particularly love the occasional “advance downbeat” (I’m sure there’s a proper name for that), which is really effective and leaves you waiting for the next one.
The album, Free At Last, is in first place for my Album of the Year 2008.
Download (MP3, 10.8M)
Lyrics after the break (transcribed by me - I guess someone has to do it first).
Any Ruby programmers who are reading this, I’m experiencing a strange issue regarding Digest::MD5. Let me show you:
>> require 'digest/md5' => true >> Digest::MD5.digest "Les Rhythmes Digitales" => "\213U\3601\260%<j\267-\343(\213I\030\347"
What the fuck is that huge escaped thing? A unicode issue?
Check out the same from bash:
$ md5 -s "Les Rhythmes Digitales" MD5 ("Les Rhythmes Digitales") = 8b55f031b0253c6ab72de3288b4918e7
Now that looks more like what I expect from an MD5 hash. Is Digest::MD5 mangling the text into some kind of weird invalid unicode?
Let’s try with $KCODE set:
>> $KCODE = "UTF8" => "UTF8" >> require 'digest/md5' => true >> Digest::MD5.digest "Les Rhythmes Digitales" => "\213U?1?%<j\267-?(?I\030\347"
Great. Any different in 1.9?
irb(main):003:0> Digest::MD5.digest "Les Rhythmes Digitales" => "\x8BU\xF01\xB0%<j\xB7-\xE3(\x8BI\x18\xE7"
Different again. At least I can see the characters in there, though. This is causing some pain.
Am I doing something hopelessly wrong? Somewhere in all these, some character encoding crap is going down. I can’t believe I’m the only one having these problems, and they render it difficult to use hashed passwords. I am working around the issue by shelling out to bash for now, but would like to get it fixed.
UPDATE: About 2 minutes after writing that, I realised I need to use Digest::MD5.hexdigest, not plain digest. I have no idea what the difference is supposed to be, but oh well, lesson learned. Apparently writing complaints on this blog helps me solve problems, so expect it to continue.
>> Digest::MD5.hexdigest "Les Rhythmes Digitales" => "8b55f031b0253c6ab72de3288b4918e7"
That’s it! I have had enough. I have had enough of DateTime, time strings, datetime strings, Time.parse(), MySQL time, JSON time, CouchDB time, Ruby time, system time and all the rest of it.
I have come to realise that there is one, and only one, appropriate way to store time so everything can understand it without endless string conversion problems, and that is in a numeric format of milliseconds since Epoch UTC.
The only appropriate time to convert from milliseconds into a human-readable string is upon presentation to an actual human - ie, in the View. Or, if you like, store a second time field in any records you save or pass around - just make sure your program doesn’t care about those.
This revelation comes from YET ANOTHER journey into ISO document land as I realised that not only do I have no idea how to store milliseconds in a JSON date/time string, but neither does anyone else. RIGHT!! THAT IS IT! From now on, dates are an integer.
Maybe.
Anyway, here’s some notes on getting millisecond-precision time references in and out of Ruby and JS, both of whose time classes I am no fan of, more for me than anything else..
Ruby example:
# Getting milliseconds since Epoch out of Ruby: time_float = Time.now.to_f time_ms = (1000* time_float).to_i #writing >> t = Time.now.utc => Thu Nov 20 23:01:32 UTC 2008 >> time_float = t.to_f => 1227222092.50133 >> time_ms = (1000* time_float).to_i => 1227222092501 #reading >> n = Time.at(time_ms / 1000.0).utc => Thu Nov 20 23:01:32 UTC 2008 >> t => Thu Nov 20 23:01:32 UTC 2008 # check we retained usec through the process >> n.usec # note we lost microsecond precision, this is intended => 501000 >> t.usec => 501333
Reducing Ruby Time.now to millisecond precision:
time_float = Time.now.utc.to_f time_msp = ("%0.3f" % time_float).to_f
That doesn’t have much to do with time, I just thought it was cool. Note that apparently doing precision reduction with strings is faster.
Javascript
// Reading an ms-since-epoch time: js> date_from_above = new Date(1227222092501) Fri Nov 21 2008 10:01:32 GMT+1100 (EST) js> date_from_above.toUTCString() Thu, 20 Nov 2008 23:01:32 GMT // note same as above // Make a new current date js> t = new Date Fri Nov 21 2008 10:28:12 GMT+1100 (EST) // Output date object in milliseconds-since-epoch format js> t.valueOf() 1227223692041
So, there’s been some Internet Drama over the for-profit video release of a presentation from RubyConf 2008 by Gregg Pollack of Envycasts. Here are some reactions and comments collected from various sources over the last ~24 hours.
My own (highly negative) review:
Having watched the offending video yesterday, it is pretty easy to see why the guy can’t make money as a developer and so is trying to jump on the “screencast money train” a la Peepcode. I would question if it’s even worth downloading for free.
It’s nothing but a superficial high level tour of threads, messaging, and profiling, with some mildly interesting speed tips at the end which should really have been a single blog post. The “research” the guy did is evident all right - as in, it’s obvious he just looked it all up for the presentation and has never actually used any of this stuff. The “tips” he gives - when he gets around to giving any - are unremarkable at best and downright wrong at worst - he actually seems to recommend using RSS as an interprocess message queue, which is a really stupid idea.
He also includes a video overlay of himself giving the whole speech down the bottom of the screen, for that little “distracting touch of narcissm”. He pronounces “memoize” to rhyme with “turquoise”, and spends the first 3 minutes of a 40 minute paid presentation making an unfunny joke.
On the whole I think the guy might actually have done everyone a favour by making the video pay-only. The presentation is not even worth the 40 minutes it takes to watch, let alone $9, and the less people who think that RSS is a good way to implement distributed processing the better.
None of this excuses the presenter’s actions re. RubyConf but in this case, I think it will be a self-correcting problem. There is some expectation that someone demanding money for their training videos might at least have some experience working with the subject on which they present. I expect this video to destroy the clown’s professional reputation just as surely as his money-grubbing actions have destroyed his personal credibility.
I am hardly alone in this assessment. Let me quote several other people from a number of sources, anonymous because I have used them without permission. That said, if you have a problem with being quoted anonymously, let me know and I will remove your comment immediately.
Couldn’t even be bothered watching it for free:
Ugh…
Couldn’t get through it. The music was just too annoying. So I skimmed. Probably spent 4 minutes watching it.
Yes, all pretty superficial stuff and nothing really useful in there.
I don’t really like those tutorials which are just a tour of add-ons/plug-ins/gems etc written by other people (and as you know, Ruby people are all too ready to embrace other people’s code). I’m more interested in seeing interesting, original and innovative code.
Another prominent developer is unimpressed:
there is nothing in this talk which cannot be discovered in a couple of minutes using google, or by reading a couple of howtos
An IP sleuth points the finger …
The “speed tips” at the end are stolen directly from igvita.com, without attribution of course, and the ruby threading graphs look suspiciously similar too. This video is basically nothing but a visual presentation of the content from someone else’s blog - unpaid, of course. To pay for it would be to encourage this kind of blatant theft.
Tsk, tsk.
UPDATE: More:
I had a strange feeling of deja vu when I saw this talk. I felt like Id seen it before, somehow, and recently. later that day I logged on and was trying to find out where I’d read it before. Turns out the whole thing was lifted from Igvita.com, with minor changes. No credit given at all. If I was the guy from igvita.com, I would be pissed.
Storage space conscious:
Not only is this video not worth 40 minutes of my time or $9 of my money, it’s not even worth 200M of my hard disk space. Deleted.
A message of support:
I fully support Mr Polack’s actions in this matter. Anyone dumb enough to 1. buy this POS and 2. implement its suggestions (RSS? Are you fucking KIDDING me?) deserves to have their money stolen and their app grind to a messy halt. Polack is doing us all a favour, why so harsh?
A compelling argument there.
UPDATE 2:
Surprise!
The following gems are shamefully incompatible with Ruby 1.9, which has now been available for testing for almost a year. These are just the ones which have personally affected me; there are bound to be MANY more.
Mongrel
FasterCSV
LibIDN
EventMachine - pure ruby works but is slow
Authors: please fix your gems!
More to come as I think of them. Suggestions welcome.
Since I’m having no joy on the CouchRest support forum or in mail to the author, here’s a monkeypatch to implement simple field validations in CouchRest::Model.
module CouchRest class Model class_inheritable_accessor :validation_fields def self.validate *fields self.validation_fields = fields.map {|f| f.to_s} end def valid? self.invalid_fields.blank? end def invalid_fields return [] unless self.validation_fields check = self.validation_fields.clone self.validation_fields.each do |c| check.delete_if {|x| x == c } if !self[c].blank? end return check end def save if new_record? create if self.valid? else update end end end # class Model end # module CouchRest
Just throw a line like validate :field1, :field2, :field3 anywhere in the Model and it’ll refuse to save without those fields being present.
Stay tuned for the upcoming $9 screencast on use of this incredible new feature, development of which has taken minutes of my precious, precious time.
Since I’m obviously just posting every song from Nightwish’s 2004 album Once one by one, why not continue with another favourite from that brilliant album.
Today’s is Nightwish - The Siren (M4A, 34.4M)
This is Nightwish at their theatrical best - some might find the lengthy instrumentals offputting, but to me the strong imagery carries the song, and the story, forward perfectly. I personally find the chorus of this song, especially the second time around, to be incredibly aurally pleasing - a complex, evocative, multi-layered progression of sound.
Lyrics after the jump, although knowing them is unnecessary and, if anything, detracts from this kind of song.
I have a directory containing gems, and I often change them. To save myself the task of manually re-installing each one whenever anything changes, I wrote a simple script which keeps track of the files and runs rake tasks whenever any is modified.
The script is almost self-explanatory, but here’s the gist:
- first time you run it, it writes a YAML file with a big hash of a relevant files and their hashes
- after that, it performs the same md5 hashes and if the hash has changed it runs rake gem:uninstall rake gem:repackage rake gem:install in the directory where the change was noticed, then rewrites the hash.
It’s pretty scrappy, and needs to be placed and run from the base dir of the gems. And it doesn’t check manifests, yet.
require 'pathname' require 'find' require 'digest/md5' require 'yaml' @allfiles = [] @new = {} @to_update = [] @root_dir = ENV['PWD'] def get_new_file_list Dir.chdir @root_dir Find.find('./') do |f| @allfiles << f end @allfiles.delete_if {|e| e.split('/').size == 2 } # ie in root dir @allfiles.delete_if {|e| e.include?('pkg') } # ignore build dirs end def recurse_files @new = {} @allfiles.each do |f| if !FileTest.directory?(f) && File.exists?(f) @new[f] = Digest::MD5.hexdigest(File.read(f)) end end end def write_new Dir.chdir @root_dir File.open("md5.yaml", "w+") do |file| file.write @new.to_yaml end end def differences newclone = @new.clone oldclone = @old.clone newclone.delete_if {|key, value| @old[key] == value } oldclone.delete_if {|key, value| @new[key] == value } diff = newclone.merge(oldclone) diff.keys.each do |k| @to_update << k.split('/')[1] end @to_update.uniq! end def cycle_gem gem_dir puts "updating #{gem_dir}..." dir = @root_dir + '/' + gem_dir Dir.chdir(dir) system('rake gem:uninstall') system('rake gem:repackage') system('rake gem:install') end get_new_file_list recurse_files if File::exists?('md5.yaml') @old = YAML.load_file('md5.yaml') differences else write_new end if @to_update.size > 0 @to_update.each do |g| cycle_gem g end get_new_file_list recurse_files write_new puts "updated #{@to_update.join(' ')}" else puts "nothing to update" end