Archive for the ‘programming’ Category

Hosting Rails is a bitch

Friday, October 5th, 2007

No matter what anyone says, hosting Rails applications is – and is likely to remain for the foreseeable future – a cast-iron bitch. An expensive, slow one, too.

I’m no stranger to web hosting. I set up my first web server over 10 years ago – manually, of course, none of these control panels, and have done it many times since, as technology changed and languages shifted. I’ve been responsible for the production web presence of a number of companies, and along the way I’ve dabbled with a number of server-side application technologies, including ColdFusion and even Java (ugh!) but mostly PHP, since that’s what everyone uses.

And for good reason. PHP has many flaws as a language, mainly to do with security and the hideous mess most projects in PHP comprise, but it’s a joy to deploy. Here’s how it works.

PHP is actually two things. It’s a language, and a compiled executable which will interpret and act on commands issued in that language. It integrates nicely with the most popular, and largely considered best, front-end web server software, apache, so that what happens when a PHP web page is requested is as such:

  1. Apache reads the web page, typically page.php, and determines it requires action by the PHP interpretor, and creates an instance of that
  2. The commands in the page are relayed to the PHP interpretor instance
  3. The interpreter processes the commands and performs whatever action is required, typically HTML generation and/or database actions
  4. Apache receives back the data and serves it inline with the rest of the web page.

This all happens extremely quickly, and since PHP is integrated so tightly with Apache, you only have to worry about maintaining that one application. Sure, you configure PHP – via a single .ini file. Everything else is Apache all the way, and it’s a highly reliable and very fast way to serve web pages.

An especially welcome benefit from this arrangement is that you only require one Apache server running to serve any number of PHP applications. You can easily configure Apache to serve any number of domains via its VHosts system, any or all of which may contain active PHP pages. Apache simply instantiates (and destroys) PHP instances, which respond to the relevant pages, whenever required. The PHP executable doing the interpreting neither knows nor cares where the commands it’s interpreting come from. You’ve effectively created a “pool” of servers, any of which can respond to any PHP application, and Apache can balance its load amongst them as required.

The situation with a Rails application is very different, and not in a good way.

The current “best practise” for deploying a single Rails application is as follows:

1. Apache server front end, with a vhost interpreting the URL and relaying relevant requests to a cluster of mongrel servers, addressable by (local) IP
2. User then configures and maintains a seperate cluster of mongrel servers dedicated to that single Rails application and responding on configured IPs as set in Apache
3. Apache is further configured to go through mongrel for every non-static request.

This is then repeated for every Rails application on the box.

This has several disadvantages compared to the situation with PHP, to wit:

  1. Administrator becomes responsible for setting up, maintaining, and constantly monitoring a collection of independently-running applications, in addition to the core Apache server
  2. Mongrel servers not currently being used are not destroyed, but continue running, wasting memory (every instance of mongrel currently running on my server uses in excess of 100M each)
  3. Members in a cluster must be predefined and cannot be allocated on the fly according to need. For example, If you have 1GB of memory and 3 Rails apps, you are limited to distributing about 10 instances of mongrel between them. You must decide this in advance. If one of the applications suddenly receives a lot of hits, performance will suffer while the other mongrels, for the other apps, sit unused – the opposite of the “pool” you have access to in a PHP situation

And this is before even mentioning the underlying speed of mongrel, Ruby and Rails in general, which can only be described as horrible.

So, the sequence of events when a page request for your Rails app comes in is as follows:

  1. Apache receives request, determine the request is for your rails app
  2. Apache picks a server in your mongrel cluster, and sends the request (NOT commands, as in PHP) to it
  3. Mongrel, moving with the lightning speed of a slug through treacle, goes through the request page and sends the relevant parts to the ruby interpreter, itself not much of a speed demon, which, after a lengthy time out for self-reflection and “me time” reluctantly listens to, thinks about, thinks about some more, and then finally responds to the command. Repeat for every ruby command on the page
  4. The mongrel cluster receives and compiles the completed requests and, taking its sweet time, eventually manages to respond with the complete page
  5. Apache passes on the completed page to the client.

Not many more steps there, I admit. But it’s the speed of it which is the killer. PHP commands are detected, seperated from enclosing HTML and dispatched from within Apache, a lightning-fast and time-proven, highly reliable web server written in C. But in Rails, the job of going through the page to find Ruby requests and dispatch them is performed by Mongrel – itself written in Ruby. I don’t have any figures depicting how much slower it is, but it is – a lot.

Combine that with the inflexibility of distributing resources where needed, and having to independently spawn and monitor a seperate cluster of servers for every seperate application and you’ll understand that when people say hosting Rails is a nightmare, they’re right.

Rails has a lot of things going for it. It’s a joy to write in, mostly, and the development environment is great. It’s much, much more productive than the likes of Java while being much, much nicer than PHP.

But it’s slow as hell, and deployment sucks, and don’t let anyone tell you otherwise.

graffiti in my httpd.conf

Tuesday, September 18th, 2007

So, you want to break up your httpd.conf file, because it’s too damn long, but you don’t want to split it into a bunch of little includes because then it’s even more unweildy?

Why not do what I do, and put huge graffiti all through it so visually scrolling through it is much easier.

Easy to miss the start of the VirtualHost section when it’s just a tiny line in your editor. But much harder to miss this:

httpd-vhosts

You can use this site to make them! Don’t forget to comment it out!

Auto Migrations

Tuesday, September 18th, 2007

I have a love/hate relationship with Rails’ “Migrations” system. While it’s fantastically useful, and absolutely necessary if you’re ever planning on changing your schema after going live, I find it incredibly fiddly and annoying creating and modifying all those little files. They’re all numbered, you basically have to make a new file (and add to subversion) every single time you make a DB change – which is all the fucking time.

So I never end up using it. I *should* but I don’t. Isn’t that always the way. What I always end up doing is just making the changes directly in the DB, then exporting the DB schema to schema.rb, copying it into a big first migration, using that, and always having this plan to switch to migrations sometime after I’ve finished active development – I just can’t stand it getting in the way all the time before then.

So imagine my delight at Err The Blog’s latest plugin – “Auto Migrations”. Here’s the deal – you maintain one big schema file, which you already have in schema.rb, and whenever you change it you run the plugin – it looks at the DB, sees if anything needs to be added, removed, or indexed – and does it. One file. No more little migrations files. But you retain the ability to do live, in-place updates.

Fucking Brilliant. This is exactly what I wanted, and I didn’t even realise it. Take a look here. Tested and working for me. This kind of shit is exactly why I love Rails.

Upgrading Apache, MySQL and PHP on RHEL4 the FUN way

Sunday, September 16th, 2007

UPDATE: You know what? Fuck, double fuck and triple fuck doing that shit by hand. I ended up scrapping all my work, uninstalling practically everything, adding http://www.jasonlitka.com/media/EL4/x86_64/ to my yum repository, and just installing from there.

Worked first fucking time.

I wouldn’t mind spending whole days crawling through dependency hell if I was, say, in prison and had nothing else to do but understand the minutiae of lib-apr.so linking. But I’m not in prison, I’ve got a lot to do, and so being able to install, say, apache 2.2 (the whole reason I started on this mission) in 5 minutes instead of 15 hours is pretty fucking compelling.

I like compiling from source and doing it myself. But trying to manually coax 3 or 4 co-dependent software system to play nicely is just too fucking hard. In the end it was subversion that did it – I fucking could not get it to talk to Apache. Hours of searching, trying what I found – useless, and potentially damaging.

Using 3rd-party RPMs is inherently a matter of trust. But using 3rd party software is also a matter of trust. I trust Apache to not contain hidden rootkits, even though I haven’t personally audited every line of source. In the same way, I trust the RPMs of the above repository to not have been tampered with. In the absence of unlimited time and manpower, that’s the only smart option I see to upgrade anything complex on a RHEL system.

You can read the original outdated entry if you want by clicking below.

(more…)

switching to macports for all development

Thursday, June 21st, 2007

I’m tired of manually upgrading packages on osx. The recent subversion update was the last straw – I’m nuking the lot and switching to macports, using this excellent guide. Here’s the new method of upgrading subversion:

sudo port upgrade subversion +tools

But is it up to date, you ask?

Shadow:~ sho$ sudo port sync
Shadow:~ sho$ port list | grep subversion
subversion                     @1.4.4          devel/subversion

Yep.

An additional plus from going the macports route is its use of the /opt directory. Previously I’ve been in favour of /usr/local – just out of tradition more than anything else I guess. But by using a new, otherwise unused directory – and /opt still has unix cred – we gain the wonderful advantage of just being able to trash that directory if anything screws up and starting again. A far cry from the current needle-in-the haystack methods.

One more thing, if you’re thinking of going by that guide above: you can just chain the port installs as such:

sudo port install mysql5 +server ruby rb-rubygems rb-termios rb-mysql subversion +tools

Just run that, and go do something else for an hour. Same deal with rubygems. Beautiful.

UPDATE: You’ll also want to do this if you don’t want to see your shiny new /opt folder in finder:

Shadow:~ sho$ sudo /Developer/Tools/SetFile -a V /opt

Sets the “invisible” bit. You’ll need to relaunch finder to see the result. By the way, isn’t OSX supposed to have a /.hidden file? I see it referenced everywhere, but I don’t have one …

Choosing Colours in Illustrator

Wednesday, June 13th, 2007

Making graphics for your web app? These days, that means they have to be gradients. Everything’s a fucking gradient these days, and I admit it slowly forms an impression on you – any site you come across not using them for every fucking thing seems kind of old and not very web 2.0. Ridiculous I know, but you slowly come to subconciously think that.

So you’re choosing colours for your stupid obligatory gradients, and you’re having difficulty finding ones that look good on the page. My advice? Use one of the colour pickers built into Illustrator. I can never “make” good colours by picking them on a wheel or, worse, generating them with RGB sliders. But when you have a whole lot of “good” colours in swatch form next to each other, the right ones kind of spring out at you. Well, maybe graphic designers I know wouldn’t be so impressed with my choices .. but I certainly feel they’re better than the ones I’ve made historically.

You can open the “swatch library” by means of the arrow-in-a-circle pop-out menu above the “swatch” pane in Illustrator. Select “open swatch library” and take your pick. I’ve come to really like the TOYO library, the colours in it just suit me for some reason, but you can try them all out.

Here’s a nice graphic made by this means. The colours are just .. nice. Bear in mind it’s just the top 20px or so of this being used, I find the colours really pleasing – and much better than the 20 or so attempts I made myself.

gradient

Very close to the default wordpress banner graphic!

Trial and Error

Sunday, June 3rd, 2007

As I’ve progressed in not only programming skill but also – and, although it sounds strange, probably more importantly – program design skill, I keep finding awful, horrible, terrible mistakes I made just months ago and slap myself on the forehead, exclaiming: What the FUCK was I thinking??!!

Take the current example, which has just consumed most of a day’s worth of time. It’s for a project of mine, a web site which I hope to release very shortly. Like many websites, it has a number of different types of content – mostly static content like menu items and writing on pages, somewhat dynamic content like FAQ items, and fully dynamic content like a news blog. And, importantly, all of this content must be localisable.

Now I don’t know what I had been drinking the day I made these decisions but up until a few hours ago this site had not one, not two, not even three but FOUR distinct, parallel, mutually incommunicado content systems. All of them need maintenance, so that’s four sets of forms, four (actually more like 10) models, any number of weird helpers and controllers to extract data from the 10 or so database tables. It all worked, but it was just a nightmare to even try to juggle it all in your head, let alone build more stuff for and on top of it.

Oh sure, I remember why I made those decisions. I wanted an FAQ page, for example. An FAQ needs categories, and it needs questions and answers. I had a fairly clean interface-translation system from another project, but I couldn’t shoehorn an faq-categories tree relationship in there – so I built another. Same with News – it seemed like more effort to try to unify an attached blog with the current system than to simply build a new one, so that’s what I did. And same with the TWO site content systems – one was for short unformatted sentences and labels, one was for long texts with textile support, links, formatting. I didn’t know how to do them both with one system, so I made two.

A few months down the line and this shit is totally unworkable. Part of building any kind of significantly sized web site is building admin tools as well, or you just can’t cope with adding the text, let alone trying to get others to help you on it. Juggling 4 different content systems just made me want to give up! And yet I knew the work to replace it all with a better system would be painful.

Ah well. Finally the pain of having to use the different systems, and the pure shame of being the author of such a horrible mess, got the better of me and I’ve now unified 3 of the 4 systems. It’s still a hack job, as I’ve got some serious thinking to do about how to handle it all in a manner I can actually be proud of, but it’s miles better than what was there before. The fourth – the blog – will also have to wait. Oh hang on – did I mention there’s actually a fifth? But that only translates names of places, it’s fairly static. I’d like to put that into the new system as well, but it’ll require some thinking. Oh and did I mention I’d like to use this new cms for multiple sites ..

A proper content management system must be totally flexible. You must be able to assign arbitrary attributes to any object in it, at any level. You should be able to say, “see this? that’s an faq category. this, this and this objects descend from it, and of them, this and this need textile. And here are their localised children.” I still haven’t figured out the best way to do all of this, and my research into other popular CMS methodologies has not been encouraging. But I have the feeling it will involve a great big table of text, a table of possible attributes, and another great big table with a membership-style mapping of attributes to that text. Hard to see the best way to do it, but it’ll be an interesting design exercise.

So that’s how we learn .. by trial and error. Think you’re saving time with shoddy design? Think again.

UPDATE NOTE: D2E2D0 E4EDDE

Updating multiple child records in a single form

Monday, May 28th, 2007

I wanted to update multiple child items on a single form. The situation is – I have an order form, and I’d like to give the customer the opportunity to edit quantity of individual line items in a text box, then update them all at once with a single “update” button, or even better, RJS. Pretty easy, huh?

Apparently not. After literally hours of experimentation, here’s what I came up with, but warning: it’s ugly.

controller setup:

 
  def basket
     = find_cart # children => cart_items
  end

in the view:

<% form_for :cart_item, :url => {:action => 'update_cart'} do |form| %> 
 <% for  in .cart_items %>
   <%= form.text_field :quantity, {:size => 4, :index => .id}  %>
<% end %>
 
<%= number_to_currency(.total_price) %>
<%= submit_tag "Update Cart" %>
<% end %>

Note that I’ve added an “index” field to the text field, otherwise there’s no way to tell which row you’re operating on. This generates a params hash as follows:

{"commit"=>"Update Cart", "cart_item"=>{"2"=>{"quantity"=>"4"}, "3"=>{"quantity"=>"2"}}}

Which is a 3-layer nested hash!

So the update_cart action in the controller has to go through that somehow, and here’s where it gets ugly:

  def update_cart
     = find_cart
    for key in params[:cart_item].keys
        quantity = params[:cart_item][key][:quantity].to_i
        cart_item = CartItem.find(key.to_i)
        cart_item.change_quantity(quantity) #method in the model
    end
    redirect_to :action => :basket
  end

Ugly, huh. But as far as I know this is, if not the only, then at least the simplest way to do it. I’m not aware of any helper in Rails to go through that hash for you, even though it generated it with the “index” function. Hardly the worst thing I’ve ever seen, but for such a common task I kind of expected it to be easier ..

Next task: AJAX that bitch so a change in the text field updates the record dynamically. This is going to be “fun”…

Error Driven Development

Monday, May 28th, 2007

You know what, all this time I’ve been working and I hadn’t even realised.

I’ve invented a whole new style of development!

You’ve heard of all those old boring kinds, like “Test Driven Development” and “Agile Development”. Well, throw all those old books away – here’s the heads up on the funky new shit.

It’s called “Error Driven Development” and it’s awesome.

The basic idea is simple enough (it’s always the simplest ideas that are the best!). You’re building a website. You get heaps of errors. You solve them, exposing more errors. Repeat this process until there’s no more errors, not that you can see right now anyway – and that’s what counts.

Tools Required: “refresh” button on browser, patience

Stay tuned for the book, tentatively titled RuntimeError: Called id for nil, which would mistakenly be 4 — if you really wanted the id of nil, use object_id

UPDATE: I’ve thought of a more crowd-pleasing name: “Exceptional Development”. Yup, my projects certainly are “exceptional” .. lol.

Procrastination

Monday, May 28th, 2007

I’ve recommended Paul Graham’s essays so many times it’s boring even to me. But I couldn’t resist passing on this fantastic quote from his thesis on procrastination:

Errands are so effective at killing great projects that a lot of people use them for that purpose. Someone who has decided to write a novel, for example, will suddenly find that the house needs cleaning. People who fail to write novels don’t do it by sitting in front of a blank page for days without writing anything. They do it by feeding the cat, going out to buy something they need for their apartment, meeting a friend for coffee, checking email. “I don’t have time to work,” they say. And they don’t; they’ve made sure of that.

Feeding the cat, checking email .. updating your blog …

Localisation inside collection_select

Wednesday, May 23rd, 2007

After spending SIX FUCKING HOURS trying to make proper localisation work with AJAX-chained select menus (select by itself is buggy and wouldn’t accept my html_options – that was an hour or two right there), I finally had an epiphany.

The Objective:

Obtain a localised name from inside a collection_select.

The Problem:

collection_select does not provide direct access to child objects, only attributes of the child as accessible through the model. For example, if you want the id of the child object, you access it as :id, and if it ain’t in the model, you ain’t accessing it.

The Initial Solution:

Create a method for the model in question to go and translate itself when called

The Reason That Didn’t Work:

collection_select went off and rendered itself somewhere out of scope so the model had no idea of the current language as provided by the controller via both session and instance variables

The Solution:

All Rails developers are familiar with the trick of putting a cattr_accessor of current_user into the User model to make, surprise, the current user available outside the limited range of the controller’s instance variables and session objects.

The solution was to do this for language as well:

# language.rb
class Language < ActiveRecord::Base
  cattr_accessor :current_language
end
 
#application.rb
 
  before_filter :set_language
  after_filter  :clear_language
 
    def set_language
      if session[:language_id]
        Language.current_language = session[:language_id]
      else
        Language.current_language = 1
      end
    end
 
    def clear_language
      Language.current_language = nil
    end
 
#model_you_want_translated.rb
  def localname
    trans(self, Language.current_language)
  end
 
#the collection_select which now works
collection_select(:model_name, :name, , :id, :localname)

Seems pretty easy and obvious now I look at it, but believe me, the troubleshooting process was pretty opaque and unintuitive, and it took a long, long time to figure out what the hell was wrong. Wish I’d thought of this way of doing things earlier, would have saved me a lot of time in an earlier project … always like this!

Rails: dump database to fixtures preserving UTF8

Friday, May 18th, 2007

I needed to dump my database to fixtures for testing purposes. None of the commonly referenced scripts could do this while preserving UTF8 characters.

I modified this script to use the Ya2YAML utf8-compliant gem. You will need to install Ya2YAML:

sudo gem install ya2yaml

And save this script into lib/tasks, with the name extract_fixtures.rake:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
desc 'Create YAML test fixtures from data in an existing database.  
Defaults to development database.  Set RAILS_ENV to override.'
 
require 'Ya2YAML'
 
task :extract_fixtures => :environment do
  sql  = "SELECT * FROM %s"
  skip_tables = ["schema_info"]
  ActiveRecord::Base.establish_connection
  (ActiveRecord::Base.connection.tables - skip_tables).each do |table_name|
    i = "000"
    File.open("#{RAILS_ROOT}/test/fixtures/#{table_name}.yml", 'w') do |file|
      data = ActiveRecord::Base.connection.select_all(sql % table_name)
      file.write data.inject({}) { |hash, record|
        hash["#{table_name}_#{i.succ!}"] = record
        hash
      }.ya2yaml
    end
  end
end

You can then run it:

rake extract_fixtures

.. and behold as your fixtures directory is filled with proper unicode YAML files. It will overwrite anything else there, so be careful.

Testing times

Tuesday, May 15th, 2007

Well, I finally admitted I need testing in my Rails application. I must admit, I held out for a long time – I just couldn’t see the need! “I’ll test it by reloading the damn page!” said I. But these words could only come from inexperience. As a project grows, is refactored, splits into modules within modules, you just can’t test by loading the page any more. It’s just not practical. You can’t hold everything in your head all the time. It doesn’t scale.

I had been avoiding tests for a long time .. a year, in fact. But in the last few days I’ve been bitten by several horrible bugs which I have belatedly realised were not spotted earlier due to my lack of proper multi-user testing. So, after a day of work, I’m pleased to announce – my first achievements in automated testing:

23 tests, 32 assertions, 19 failures, 1 errors

heh. But at least it isn’t literally crashing the tester anymore.

Now to basically write the whole application again as functional tests!

“Type” a reserved word in Rails database

Monday, May 14th, 2007

I spent an hour trying to figure out why my SQL query crashed ActiveRecordso you don’t have to.

Don’t have a column name “type” in your DB, and if you do, don’t put an integer in it – or AR will crash out with this informative error:

/usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1358: syntax error, unexpected tINTEGER
Object::1
         ^
        from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1358:in `compute_type'

Just putting this up here so hopefully someone will find it in a search; I couldn’t.

UPDATE: it *is* apparently reserved. How about that:

self.inheritance_column = 'type'

I guess I’m just not used to thinking of column names as something with which you need to consider namespace clashes .. lesson learnt!

base36 conversion in ruby

Friday, April 27th, 2007

Another one of those “god I love ruby moments”:

convert base10 integer “@int” into a base36 string:

@int.to_s(36)

convert base36 string “@str” into a base10 integer:

@str.to_i(36)

I could not fucking believe it was so simple.

Electropaint for OSX Intel

Monday, April 23rd, 2007

My favourite screensaver ever is Electropaint, which ships with SGI machines. An OSX port (with source!) is available, but inexplicably not compiled for intel – so I compiled it, and . Intel only because I have no idea how to make a universal binary. Check it out, it’s great.

I also love IGS Twisted, but recent versions of that have gotten too dark!

>cd to…

Sunday, February 18th, 2007

A tiny app for the mac that goes in your finder toolbar, which does just one thing: launches terminal and automatically cds to the current window. One of those incredibly useful little utilities you will find yourself using 50 times a day. Open source, totally free .. wonderful!

Find it .

The hell of GeoCodes

Thursday, February 15th, 2007

You would think, wouldn’t you, that in these days of the internet and technology and a chicken on every table that there would be some reasonable, logical and standardised way of describing where you are. Well, my friend, you would be sadly mistaken!

Say, for example, that you were implementing a geotagging system – in other words, a simple series of hierarchical steps that, once you get to the bottom of the chain, hopefully describes roughly where you are. You’d think that there was some global accepted standard for doing this, perhaps from ISO or the IETF .. a simple, clean, well-documented and logically sound method for implementing this kind of thing.

Hell no. What there is is an absolute pig’s breakfast of incompatible, illogical, ever-changing and apparently totally arbitrary conflicting closed semi-standards. There’s no one obvious winner, not even a kind-of winner, and there isn’t even one that’s likeable. They’re all a complete joke.

One first begins to realise the abject state of things when one, embarking on this journey of enlightenment, realises that no-one can even agree on how many continents there are. You’d think it was pretty obvious – I’ve always accepted that it was 7, which seemed to be a fairly logical and geographically sound system of division. Well, apparently not! There are enough people arguing that it’s less than that, or more than that, or arguing where the lines are, that there isn’t even a number system for continents.

But don’t worry, it doesn’t end there! Next is countries! Actually, for this level of division there is a faily reasonable system – it’s called ISO 3166-1, and it basically assigns every country a number. It’s free, fairly consistent, and except for the odd idiocy (for example, in 1998 they decided to change the UK’s 2-letter code from UK to .. wait for it .. GB. Why? What was wrong with UK? Was everyone getting it mixed up with all those other UKs?) it’s OK. No idea how they derived the numbers, there seems to be no logic to it at all, but at least we end up with a list of country names and numbers.

However, if you thought this reasonableness would extend to the next product from ISO I examined, ISO-3166-2, think again! Firstly, 3166-2 isn’t even a free standard. ISO expect you to buy it from them – a $500 PDF detailing what they think things should be called! Forget that! Luckily, free (albeit somewhat outdated) versions are floating around the internet .. or you could just piece together the jigsaw from the million or so wikipedia articles about it. However, don’t waste your time.

The $500 might almost be justifiable if the standard made any sense. There is no sense to 3166-2 at all. Even basic naming is ridiculously inconsistent – a subcountry area might be referred to by a number, a number with a 0 in front of it (problematic when you remember this is a varchar), a one-letter code, a two-letter code, or hell, why not a 3-letter code! never mind that the maximum number of entries for any one country is 50 for America.

This is totally inappropriate for my needs – which I thought to be pretty simple. I just want a basic, hierarchical system of continents, countries, states and cities, tied together by some kind of rational system, in a form I can implement in a data structure. What do I get? a bunch of unsorted, unstructured, inconsistent plaintext with no rationality to it at all. And that’s the killer, really – if I liked the system, even if it was difficult to implement, I might still use it. But the ISO system is the opposite from likeable, it’s detestable – and don’t even get me started on the competing FIPS American system.

So what can I do? I need numbers. I need unique numbers for every place of interest to me, and I need a strong hierachy. So I invented my own system, based on a conjunction of the useful parts of the ISO efforts and Google’s AdWords subcountry system, itself an ISO derivation. So I’ve made a list of the top 535 subcountries in the world, given them numbers, and set it all up. It’s totally non-standard, totally un-inclusive, heavily biased towards those countries I actually give a fuck about, and that’s all great. OK, solved, kinda, down to the subcountry (ie state/province) level.

Now cities. Don’t remind me of cities, please…I’ve got 3558 unsorted cities here, waiting for some database transformations to categorise them, and that’s just with a population over 100,000 … : (

God I love Rails (and Ruby)

Sunday, February 11th, 2007

So there I was. I had to come up with something to do this: Get a list of groups that user 1 is a member of, but user 2 is not a member of. So I’m sitting there thinking of what kind of horrible-looking query I’m going to have to come up with to do that.

And then I thought .. man, wouldn’t it be awesome if I could just do .groups – .groups!

Just for a laugh, I tried it. It worked. Cue tears of joy. I’ll probably write the custom query anyway, since it’ll be less expensive .. but .. god, I love Rails. Doesn’t that shit just take the cake.

UPDATE: yes, I am aware this is ruby feature. Nonetheless, the ActiveRecord component, which brings the automatic table relationship, is all Rails. Good stuff.

mind-bending relational databases

Friday, February 9th, 2007

I don’t know how most people feel about relational database programming, but god damn I find it tiring. Maybe there’s some genetically altered super-breed of hybrid enhanced data modellers who usually work on this stuff, but I find conceptualising, implementing and working with this shit a total mindfuck after a while, especially if it’s something I made some time ago and I forgot all the weird custom has_many :throughs and two-way associations.

Once there’s more than about 3 models in play, I just can’t fucking hold it in my brain any more. Thank god for consoles because without console-based trial and error my productivity would be close to zero. I cannot even imagine what it would be like if I actually had to compile anything before running my 20 erroroneous “tries” before I finally manage to construct the query I want!

It is so tiring for my poor non-enhanced brain that I find I can only concentrate on “deep” database work for about 10 minutes at a time before I become totally overwhelmed and have to take a break, of which this post is an example.

The worst thing is that I am well aware that in terms of programming, what I am doing is not even remotely difficult, which does not improve my self-confidence.

artist's impression of my database