Sho Fukamachi Online

Output from my trusty svn_up script, which also pulls git:

git detected in arkx
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in arkx, retry manually?
 
git detected in couchrest
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in couchrest, retry manually?
 
git detected in globalite
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in globalite, retry manually?
 
git detected in masquerade
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in masquerade, retry manually?
 
git detected in rack
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in rack, retry manually?
 
git detected in relaxdb
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in relaxdb, retry manually?
 
git detected in rest-client
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in rest-client, retry manually?
 
git detected in thin
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in thin, retry manually?

Does anyone else think that it is not really acceptable for a large number of important source repositories to just go down at the drop of a hat?

UPDATE: from github.wordpress.com:

github 7:46 pm on August 23, 2008 | 0 | # |
We’ve taken the site down for approximately 3 hours while we’re upgrading our servers.

THREE HOURS??!!!

There’s a bit of a gotcha I encountered recently with OpenStruct. If you create a new one from a Hash with the key, uh, “hash” - then try to read the value back out by doing mystruct.hash - you’ll get the object ID instead. No idea why, I’d certainly never realised .hash was a synonym for .object_id before, but it fails in the worst possible way - looks like it works, accepts the information, and then casually hands you back the wrong value when you least expect it:

>> require 'ostruct'
=> true
>> hsh = {:hash => 'hash!'}
=> {:hash=>"hash!"}
>> os = OpenStruct.new(hsh)
=> #<OpenStruct hash="hash!">
>> os.hash
=> 18782890

Can’t have that. I had already extended Hash to ease OpenStruct creation, so I modified that to check for key/keyword collisions first.

class Hash
  def to_struct
    if self.keys.map {|k| OpenStruct.respond_to?(k)}.include?(true)
      raise 'reserved word as hash key'
    end
    OpenStruct.new(self)
  end
end

Gotta love those Ruby one-liners : )

So why do this kind of thing at all? Well, Rails has spoiled me with all its method_missing sugar. So if I’m doing something like this:

SITE_CONFIG = {:default_language => 'en'}

I don’t want to access it like this:

SITE_CONFIG[:default_language]

OK, pretty silly reason, but it makes a difference to me! Hey, that’s why I use Ruby in the first place.

>> SITE_CONFIG = {:default_language => 'en'}.to_struct
=> #<OpenStruct default_language="en">
>> SITE_CONFIG.default_language
=> "en"

It’s the little things.

Of course the real lesson here is not to use common keywords/class methods as hash keys or anywhere else.

I think this is the only way to do it.

>> Time.now.strftime("%Y/%m/%d %H:%M:%S.#{Time.now.usec} %z")
=> "2008/08/20 01:22:28.367899 +0000"

At the microsecond level, it’s possible for Time.now to change mid-evaluation, so if you *really* care about timing you could freeze the time object first and read from that:

>> now = Time.now
=> Wed Aug 20 01:40:26 +0000 2008
>> now.strftime("%Y/%m/%d %H:%M:%S.#{now.usec} %z %Z")
=> "2008/08/20 01:40:26.597940 +0000 UTC"

Happily, Time.parse will read that straight out of the box:

>> n = now.strftime("%Y/%m/%d %H:%M:%S.#{now.usec} %z %Z")
=> "2008/08/20 01:40:26.597940 +0000 UTC"
>> Time.parse(n)
=> Wed Aug 20 01:40:26 +0000 2008
>> Time.parse(n).usec
=> 597940

To celebrate Nightwish’s promotion to a probation A-list band, I’ll celebrate by uploading their 14-minute track The Poet and the Pendulum, the bombastic, overblown, teen-poetry-esque but very enjoyable first track from their most recent album.

The Poet and the Pendulum - mp3, 32M

If 14-minute Finnish symphonic power metal rock operas aren’t your thing, I recommend this previous post of a 15-minute Japanese symphonic power metal rock opera, still available.

Lyrics after the break for those who love to sing along!

Read the rest of this entry »

I promoted Nightwish to A-list status, and demoted a few to B-list - let’s see how long that lasts. I also deleted any C-list bands, since there’s so many of them a list becomes kind of endless and futile.

Interesting to maintain a “list of favourite bands” over multiple years. Revisiting it every year or so certainly weeds out the spurious entries, and re-affirms the deserving.

Song of the week is from the “new” Bryan Scary album, Flight of the Knife.

Imitation of the Sky

I’ve been on a bit of a new music spree recently, having gotten sick of everything I had, so expect a mini-flood of “songs of the week/day/hour”.

Gee, I guess centralising source control around a single website might have some downsides after all.

Australian ABC’s new iView internet service, in which they offer good-quality streaming flash video of most of their interesting shows, is fantastic. ABC is Australia’s tax-funded government broadcasting system; it’s generally pretty good.

This is exactly what I want ABC to be doing with my tax dollars. We pay for all this content to be made, it should be available free to everyone on the net, in good quality, at any time. This first version is an excellent start. It’s still a bit slow and the quality could be better, but I’m sure that will come in time. Much better than the BBC’s “iPlayer” application. Now hopefully they will drastically increase not only the quality but the range and back catalogue, and add more download options.

Now if only SBS would offer something similar - that is the only other channel worth watching on Australian television and as soon as they do so I won’t be able to see much point owning a television at all.

Check it out if you’re interested in watching Australian taxpayer-funded TV programming. ABC News, 4 Corners and Foreign Correspondent are best for news. If you’re after comedy, the Chaser is currently on break but The Gruen Transfer is high quality.

Oh, and it’s unmetered for iiNet : ) Internode to unmeter it soon too, apparently.

What the hell is going on with Starbucks?

They are closing 61 out of their 84 locations in Australia, but I can’t see why. One of the locations due to be closed I know very well, it’s Starbucks in Chatswood - a rich suburb and it’s the only Starbucks there. I walk past it a lot and it’s always full.

How can you lose money selling $6 coffee to rich Chinese 16 hours a day? I had assumed they were raking in the cash. The Taiwanese tea shop across the road has premises twice as large, charges less, has half the patronage, and they’re doing fine. No other shops around are closing.

Sure, American businesses sometimes misjudge the Australian market. Krispy Kreme has struggled here, discovering belatedly that oil-soaked donuts are not exactly part of the staple diet here - but even so the KK store in Chatswood is still doing fine after downsizing. And it’s not like coffee is unpopular; quite the opposite. In fact I noticed a new café opened just last week, very close to the SB. Something doesn’t add up!

Anyway, my favourite branches (Darling Harbour and George St) will be staying open so it’s not like I care all that much - just amazed that anyone could possibly lose money with such a seemingly foolproof brand. Starbucks Inc has obviously screwed something up in a big way Stateside and are cutting back international operations as a result. Wonder if the Japan stores are next.

UPDATE:

I’ve been thinking about this some more, and factoring in my own purchasing habits to understanding SB’s problems in AU.

Let’s think about my purchasing patterns:

Number of times I have bought prepared beverages, hot or cold, outside the house in the last year: maybe 100+

Number of purchases from Boost Juice: 30+
Number of purchases of pearl milk tea or other chinese drink: 20+
Number of purchases of coffee from a non-SB café: 50+
Number of purchases from Starbucks: uh, 1 or 2, I forget, both green tea frappucinos in summer

I think that, upon further reflection, I am beginning to see the problem.

Maybe Starbuck’s competition in Australia is much stronger than in the USA, or indeed Japan. Here, if I want coffee, I very rarely buy it from SB. There are plenty of other cafés around, and generally their coffee is better than SB.

If I want a cold drink, I will usually buy either a fruit drink from Boost or some kind of chinese pearl milk tea. Both are delicious, Boost is healthy, and both are cheaper than SB. Also more convenient, open later, and I know some very nice bubble tea places with nice gardens to sit and read.

Basically Boost wins over SB 9 times out of 10. If I’m going to pay $5 or more for a drink, I want to actually get something for my money - my favourite drink from Boost has a banana and yoghurt, sweetened with honey and plenty of milk. SB is little more than water, flavour, corn syrup and crushed ice. I like it sometimes, sure, but when I think about how often I actually buy that - not very.

I drank SB much more in Japan and I think that’s where I started thinking I liked it more than now. But upon reflection I only drank it so much there because there simply wasn’t anything else. Plus, it’s cheaper there, and the stores are open later. I remember wishing they had Boost there, and some better coffee places too - now I’m in AU I simply go to them instead.

Maybe SB isn’t such a sure fire success in Australia after all.

UPDATE 2:

Some good analysis here and here.

Wakatta means “saturday”. Understand?

Using the excellent Modo, I mapped a few of Nasa’s Blue Marble earth image series onto a sphere. Here’s one rendering.

That’s a 5400*2700px texture map, which was enough to bring this 2GB machine to its knees. God knows how much memory you’d need to use the 21600*10800 map. I foolishly tried it, and force quit the program about 10 minutes later : /

I also did the “earth city lights” map:

As you can see, I am a true master of realistic lighting. Who needs a stupid “sun”.

I was pleased to see that the girls in the Production I.G. animation Real Drive are drifting more in line with reality in terms of the female japanese physique:

That’s much more in line with the chubby sailor-suited figures you’ll see on the yamanote line than the usual super-skinny animation girls, which stretched credulity even for westerners.

Unfortunately, I have no idea what Real Drive is even about, despite watching 3 episodes so far. Some kind of point about how “diving” (VR access) in the “meta-real” (some kind of cyberspace environment) is analagous to diving in the actual ocean - which, by the way, seems to have come to life? It is incomprehensible. Looks nice, though, and it’s always interesting to see Masamune Shirow and Production I.G.’s vision of the near-ish future.

A more interesting reading might be that ubiquitous networking, now and in the future, is becoming such a fixture in the modern world, and such an unknown technology for the majority, that it is now being treated in fiction as a kind of naturally-occurring environment, with all the emergent behaviour that entails and even being accessed in a similar physical way. This isn’t new, of course - the mapping of physical gestures onto actions executed in a cyberspace context dates back to Neuromancer and maybe beyond. But it’s certainly becoming more common - this is the second animation series I’ve encountered in a year which revolves completely around the concept of unexpected emergent behaviour in an omnipresent networked environment.

On another note, in my opinion, sometime in the last year average fansub animation downloaded using bittorrent has exceeded DVD quality in both picture and subtitles/extras. A 350M mkv file, which typically includes excellent peer-reviewed subtitles and is compressed using H264, is higher resolution and generally just looks, sounds and is excellent. It is available sooner, is typically replaced if problems are found, and has fantastic high-speed distribution. Bluray might be better but is completely unavailable, and would be overpriced and months or years late if it did appear.

Nice going, media companies - another reason not to buy your tacky plastic disks, delivered years late, with shoddy production values and at eye-watering prices.

Another in my series of “lessons I’ve learnt as I progress in my journey from rank amateur to serious developer”.

Some time ago I posted a series of articles discussing the use of “advanced” database techniques with Rails, mostly utilising Postgres. See here:

Switching to PostgreSQL
Strict databases are great for discipline
hacking native UUID support into Schema:Dump
UUIDs in Rails redux

Anyone visiting those old articles these days will notice a new update: don’t do this. Why, you might ask? Since I thought it was such a good idea at the time.

Turns out that being “clever” with your DB setup is almost certainly more trouble than it’s worth.

Sure, using PGSQL schemas to share tables between your applications seems like a nice, elegant solution compared to throwing everything in one DB and using table namespacing to distinsguish between them. Sure, using the schemas saves you from having to define table names in every model where you access a “foreign” table. Unfortunately, it’s a giant pain in the ass to maintain and constricts your use of migrations.

Sure, hacking Rails to read and write PGSQL’s native UUID format is elegant and looks kewl in your migrations, etc. However, maintaining it is a giant pain in the ass and kills cross-DB compatibility.

These are examples of me being “too clever”. Not trying to brag or anything here, I’m using the term in a self-deprecating manner. What happened was that I, a relatively inexperienced developer, thought I’d found some “cool” solution to a problem I was having - or imagined I was having. I then “solved” the problem in a manner which was worse than anything the “problem” itself presented.

DO NOT use postgresql schemas unless you have a damn good reason for it. “Making your tables look nice” is not a good enough reason.

DO NOT use database-specific data types unless you have a damn good reason for it. Again, “looking nice” is not nearly enough.

UUIDs are strings, pure and simple. Using a UUID data type in PostgreSQL gains you one thing - checking that anything you try and save in there looks like a UUID. That should be in your own tests. What you lose is cross-platform compatibility, ability to use standard Rails, reliance on hacks to schema dumper, and more. And they’re strings, right? So store them in a fricking String field. Anything else is just not worth the hassle.

Schemas seem cool and elegant, and they are. Unfortunately, what you want isn’t “cool and elegant”, it’s “easy to work with and convenient to maintain and update and migrate and change around when you need to”.

It took me hours, days, to implement the changes to use the above features of PGSQL. It took a couple of hours to undo all of that and go back to standard data types, standard layouts, minimum complexity. Wasted work and time? Sure, but I learnt the lesson I’m trying to convey here - keep it simple, don’t be clever, maintain cross-compatibility and stick to the lowest common denominator unless you have a really, really good reason not to - and mine didn’t count in the end.

I’m still using PGSQL, of course - I like it, and even MySQL 6 inexplicably still doesn’t offer sub-second precision in DATETIME or TIMESTAMP, which I want. And I’m certainly not moving away from the UUID approach, which I firmly believe is best - all of that still stands. But I’ve gotten rid of all the special data structures and non-standard migrations and database-specific data types, because the miniscule aesthetic benefit just was not worth the loss of freedom and convenience everywhere else.

Live and learn, eh.

Thank you TextMate for rendering my computer unusable for 5 minutes, taking every last byte of RAM available and swapping every other process out while you apparently loaded every file in my project into memory at once (including multi-hundred-MB log files) and then leisurely (but disk- and CPU-intensively) sifted through it after I had the temerity to actually use your “search in project” function.

Next time I’ll remember to use git grep. Oh, and now I love how TM is still using 708M of memory (RSIZE) even after I’ve closed everything.

Isn’t TM about due for an update? How about better memory management and a way to exclude folders from search for starters?

By the Black Mages, from their newest album, The Black Mages 3, and is an adaptation of The Opera of Maria and Draco from Final Fantasy 6.

If you are a fan of 15-minute Japanese Rock Opera, click here for heaven. (35M) Also you can find some translated lyrics here.

The rest of the album, unfortunately, is not up the standard of The Skies Above to my ears. Maybe it’s just because I don’t personally like the songs chosen but I feel the ones that were are patchier and less suited to straight rock playthrough than those on TSA. Strange, because there’s a huge wealth of material to choose from - Uematsu can, of course, use whatever FF music he likes, and there’s a *lot* of that. 3 albums and no Chocobo theme? What’s going on!!

I just learnt a new trick in Safari I haven’t seen anywhere else. Double click in the blank area of the “tabs” bar (ie, to the right of any existing tabs) to spawn and select a new tab.

Short and sweet.

I unthinkingly executed an N+1 database operation on a table with in excess of 2 million records, doing a lookup on another table for every .. single .. one.

So let’s see .. I estimate it will need to do about 100k writes. So that’s 2*2,000,000 reads, then 100,00 writes .. a mere 4,100,000 accesses in total. Even better, it’s MySQL, and I don’t have the C adapter for that installed - so every access does a round trip through the treacle-slow pure Ruby adapter.

Oops. If it’s still going when I wake up tomorrow I might cancel it!

UPDATE: It was actually done in maybe 6 hours (wasn’t paying close attention), which I was pretty impressed by. I guess it’s not that much - MySQL probably had the whole thing in memory after a while and as I said, it was mostly reads. Still, 200 or so queries/second - not bad.

In the wake of the ISO-3166 split (CS to RS / ME) I was faced with the task of updating my “Countries” table when things randomly began failing. This brings back countless delightful memories of my original battle to construct a unified ISO-compliant countries/languages table from a while ago.

Well, that was easy enough - I didn’t bother with any re-imports, just added the new records by hand and all is well. But I thought I’d take the opportunity for some more fun by finally reforming my custom “Languages” table, in accordance with the upcoming updates to RFC4646 (I think - all the numbers are beginning to blend together). As long-time readers might remember, last time I tried to do this I gave up in frustration and wrote my own fricking language table.

Well, it’s good to see ISO hasn’t given up its core principles of crazy, archaic DB structures and weird, breaking proclamations of what they want to call languages this year. Take for example the language of Chinese Mandarin (Simplified). Now what do you think the ISO code for that is?

Is it zh_cn? zhs? zh_zhs? zh_man? zho_man? zh_Hans? All of them good guesses, all of them wrong.

The new standard name for Simplified Mandarin is zh_cmn. CMN? I have never even heard of that one, and certainly never seen it. A Google search gets a stunning three (3) hits. Are you even supposed to preface with the 639-1 alpha2 any more? Oh I get it - it’s supposed to be zh-cmn. That gets 2300 hits on Google, for a language with close to a billion speakers.

What the hell? This is the standard now? I wish they’d just make up their minds, and that IETF and ISO would actually cooperate.

Combine this with the ISO “macrolanguage” system and you have a recipe for ever more hours of fun. Needless to say, ISO delivers the macrolanguages mapping table in an unusable, world’s worst practise TSV file with no primary keys and incomprehensible, capital-letter-including column names. Even the recommended table names, which include hyphens, are invalid in all modern DBs.

Oh, and ISO provides a downloadable DB for ISO-3166 1 and 2. It’s in MS Access format. Explains a lot.

So let’s sum up. We have ISO 3166 1, 2 and 3 .. and 4 and 5 and 6 which defines country codes. We have ISO 639n which defines languages … and some more country codes. We have RFC4646 which defines how browsers should use country and language codes. We have the Unicode mappings. We have more ISO codes, this time numeric. We have Timezones, which are different again.

What a fucking mess. And by the way, I have NO idea what Australian English is even supposed to be in the new ISO639-3. I don’t see it anywhere here. I can tell you what Australian Aborigines Sign Language is, though - asw. That will come in handy for all those sign language websites I’m working on as we speak.

I have no idea what the Ruby guys are thinking with these version numbers.

They want to make a lot of changes, leading in to 2.0. Well, that’s fine. Great! Major version numbers are just the place to make sweeping changes, cleaning up a lot of stuff, progressing wherever they want. That is exactly the right thing to do.

Hang on .. they want to, uh, declare 1.9 their kind of “work branch” for 2.0. Uh .. right. What was wrong with using, say, a 2.0 beta branch? But .. hm, ok, well, it’s still OK, I guess, to make reasonable changes in a minor version .. kind of. Kind of negates the whole point of having the major version number in the first place, unless it’s just counting up in .1 increments, but whatever, they’re in charge. Personally I would have thought that 1.9 should be 1.8 on YARV, but whatever, it’s their ball game.

But 1.8.7? Incompatibilities from backported features from 1.9 - which is itself a kind of “future backport” of 2.0? What the fuck? Why the fucking hell are they backporting shit from 1.9 into 1.8 when it is not completely compatible?! Do they even understand the point of these version numbers?!

How Version Numbers Should Be Used

A Ruby version number: [major].[minor].[tiny]-p[patchlevel]

Major version number: do whatever the hell you want! Got breaking stuff and design changes? Here’s the place to do it!
Minor version number: For big new features and major changes, eg YARV. Generally backwards compatible, but .. well .. shouldn’t break stuff if you can help it but if you REALLY need to, OK .. kind of.
Tiny version number: for tweaks, fixes and optimisations only; ok to add (small) new features, should NOT break anything backwards
Patchlevel: absolutely should fucking not break anything backwards and to be perfectly honest should not even exist, and the fact that it does exist is a sign of bad release practises

They should not have made major (read: incompatible) changes in 1.8.7. They absolutely should NOT under any fucking circumstances have made incompatible changes in 1.8.6-p230!! What the hell is going on?

The Ruby team need to basically shift all their versioning one decimal point to the left. They are putting shit in patchlevel which should be in tiny - and there are WAY too many patchlevels. They should wait a bit and then roll up changes from trunk every few months, only levelling up when they’ve got something important or urgent. As it stands they’ve been going through one every few days, it’s ridiculous. May as well just use the checkin number. They are making changes in tiny which should be in minor. And for 1.9 they are using minor when they should be in major. Why the hell are they acting like this? It’s not like new numbers cost anything?!

As much as it pains me to say it, Ruby could learn from Rails here. Ditch the fucking patchlevel bullshit. 1.8.6 should equal 1.8.6, no questions asked, no possibility for difference, no fucking patchlevels or anything else. If you need a new version, 1.8.7, which is always backwards compatible. *Never* break backwards compatibility in a minor branch. And if you need more than 10 tiny versions in a minor branch, then just keep going - if 10.4.11 is good enough for Apple, then 1.8.11 is good enough for Ruby.

This security update should be 1.8.7. And what about the current 1.8.7? It should not even exist! What is the purpose of 1.8.7? To get us ready for 1.9? NO! 1.8.7 should be a straight progression, bugfixes and security updates for 1.8.6! The current 1.8.7 should be 1.9, which is getting us ready for 2.0!

The other day, I needed to install 1.8.6 on a new server. I installed the latest tarball of 1.8.6 from Ruby-lang, which was patchlevel 114. They have skipped over a hundred patchlevels in the last couple of days and now insist 230 is the minimum. Hint: if over a hundred of these patchlevels were not important enough to be bundled into the downloadable tarball, maybe they should not be patchlevels at all. Go look at ftp.ruby-lang.org. Look at their tarballs in 1.8 branch. There is *no* tarball between 114 and 230. So, uh, why isn’t p114 called 1.8.7 and p230 called 1.8.8?? Or 1.8.9, or whatever? What is the point of a patchlevel you can’t even download a tarball of?

Ruby versioning is completely fucked up.

Some people just can’t stop achieving! Not content with introducing breaking, backwards-incompatible changes into the Ruby 1.8 branch with 1.8.7, the Ruby Core team have shot for the moon, gone for gold, and doubled down all their bets - and made Ruby 1.8.6-p230 incompatible with Ruby 1.8.6-p(any other version). Of course, there is an URGENT!!!111ONE zero days exploit or something in any version of 1.8.6 that’s not the one released today - the incompatible one.

So let’s sum this up:

Ruby 1.9.0 = incompatible with Ruby 1.8.x
Ruby 1.8.7 = incompatible with Ruby 1.8.6
Ruby 1.8.6-p230 = incompatible with Ruby 1.8.6-p(<230)
Ruby 1.8.6-p(<230) = UR SERVER IZ OWNED LOL

Well that is just fucking great. Well, I guess if your app won't start, no one can hack it! Can't get more secure than that! Mission accomplished!

It seems like this Ruby Spec project is arriving not a fucking moment too soon. When the core team can’t even keep Ruby 1.8.6 compatible with, uh, Ruby 1.8.6, we got problems.