Posts Tagged ‘couchdb’

Simple Couch comet listener with EM

Monday, May 25th, 2009

So couch trunk has now got the long-awaited comet update-tracking functionality, obsoleting pretty much every other way of doing update notification at a stroke. I’ve been looking forward to this for a while – I want to throw an EM daemon or two on the comet URL; they’ll listen for changes and do cache invalidations/search index additions asynchronously. Yes, I could just expire the cache synchronously upon save but that gets very fiddly – I want to store the seq number in the cache so the expiration/update sequence is fully replayable. Doing that synchronously would involve another query to the DB to find the current seq, inviting race conditions – forget it. Also, I need to do message dispatch to diverse clients who might not be using the web server at all; I need all updates to flow through a router, and that can’t be in the web app.

Anyway, here’s a simple EM script which listens to the (charmingly undocumented) comet URL and does whatever you want with the updates. If you were doing anything complex you’d probably want to send the process off into a defer operation.

require 'rubygems'
require 'eventmachine'
require 'socket'
require 'json'
 
module CouchListener
  def initialize sock
     = sock
  end
 
  def receive_data data
    data.each_line do |d|
      next if !d.split("\"").include?("seq")
      puts "raw: #{d.inspect}"
      begin
        json_data = JSON.parse(d)
        puts "JSON: #{json_data.inspect}"
        puts "updated: id #{json_data["id"]}, seq #{json_data["seq"]}"
      rescue Exception => e # TODO definitely do not want to rescue in production
        puts "JSON parse failed with error #{e}" 
      end
    end
  end
 
  def unbind
    EM.next_tick do
      data = .read
    end
  end
end
 
CURRENT_SEQ = "0" # you'll want to replace this with whatever is current
DB_NAME = "test_comet"
 
EM.run{
  $sock = TCPSocket.new('localhost', 5984)
  $sock.write("GET /#{DB_NAME}/_changes?continuous=true&since=#{CURRENT_SEQ} HTTP/1.1\r\n\r\n")
  EM.attach $sock, CouchListener, $sock
}

Rivers of Data

Thursday, March 19th, 2009

This output appeared on my console while tuning some CouchDB reduces.

CouchDB Art

CouchDB session model for Rails

Friday, September 19th, 2008

Here’s my initial stab at a Rails Session model for CouchDB. The marshalling stuff is taken from the example SQLBypass class in the ActiveRecord code.

You’ll need a recent and trunk CouchDB, probably.

class CouchSession < Hash
  @ = CouchRest.database!('http://localhost:5984/sessions')
 
  attr_writer :data
 
  def self.find_by_session_id(session_id)
    self.new(@.get(session_id))
    rescue
    self.new(:id => session_id)
  end
 
  def self.marshal(data)   ActiveSupport::Base64.encode64(Marshal.dump(data)) if data end
  def self.unmarshal(data) Marshal.load(ActiveSupport::Base64.decode64(data)) if data end
 
  def initialize(attributes = {})
    self['_id'] = attributes['_id'] ||= attributes[:id]
    self['marshaled_data'] = attributes['marshaled_data'] ||= attributes[:marshalled_data]
    self['_rev'] = attributes['_rev'] if attributes['_rev']
  end
 
  def data
    unless 
      if self['marshaled_data']
        ,  = self.class.unmarshal(self['marshaled_data']) || {}, nil
      else
         = {}
      end
    end
    
  end
 
  def loaded?
    !! 
  end
 
  def session_id
    self['_id']
  end
 
  def save
    self['marshaled_data'] = self.class.marshal(data)
    self['data'] = data
    self['updated_at'] = Time.now
    save_record = @.save(self)
    self['_rev'] = save_record['rev']
  end
 
  def destroy
    @.delete(self['_id'])
  end
end

Nice and short – possibly the shortest Rails session class I have seen. The beauty of CouchRest/CouchDB! And we descend from hash so we can just save the object straight – after marshalling, of course. Cool, huh?

Note that I am actually writing the raw data as well as the marshalled data into the saved doc, for troubleshooting/interest purposes. Feel free to remove that.

Not pretty, but it works. Just save it like a normal model. You’ll need to put these into environment.rb:

config.action_controller.session_store = :active_record_store
CGI::Session::ActiveRecordStore.session_class = CouchSession

Note also that I have ignored any differentiation between the record ID and the session ID, negating the need for any special overrides in ApplicationController. However, the session IDs Rails generates are large and you might find them unattractive in CouchDB – it would be fairly simple to separate them, but then you’d need a new map view and an override. I feel it’s simpler to just use the Session ID as the doc ID and damn the torpedoes. YMMV.

Improvements? See something wrong with it? Let me know! ;-)

ActiveRecord to DataMapper/CouchDB class header script

Thursday, June 12th, 2008

I hacked together this script which dumps the structure of your database into a DataMapper-compatible class header file. You’ll need these for your Models if you’re planning on playing around with datamapper – should save you a bit of typing.

There’s a bit of CouchDB boilerplate in there, feel free to strip that out if you’re not using that. The large sql_type case statement just covered what I had, which was mostly postgres and will need modification for MySQL – but it will tell you what you need to change, should be obvious enough. Note that I am coercing pretty much everything to basic Integer/String – this is for CouchDB but you can change it to match the types you want. And yes, I know it’s ugly!

# datamapper_dump.rake
# Sho Fukamachi 2008
# place in lib/tasks and run with: rake db_to_dm
 
def tables_we_want
  skip_tables = ["schema_info", "sessions"] # this is slow enough as is without sessions
  ActiveRecord::Base.establish_connection
  #ActiveRecord::Base.connection.schema_search_path = "path1, path2" - uncomment for PGSQL
  ActiveRecord::Base.connection.tables - skip_tables
end
 
task :db_to_dm => :environment do
 
  sql  = "SELECT * FROM %s"
  dir = RAILS_ROOT + '/db/dm_temp/'
  FileUtils.mkdir_p(dir)
  FileUtils.chdir(dir)
 
  ActiveRecord::Base.establish_connection
  #ActiveRecord::Base.connection.schema_search_path = "path1, path2" - uncomment for PGSQL
 
  puts "Dumping Schema into DM format..."
 
  File.open("models.rb", "w+") do |file|
    file.write "# Models dump File for DataMapper/CouchDB\n"
    file.write "# Sho Fukamachi 2008\n"
    file.write "\n"
    tables_we_want.each do |table_name|
    class_header = <<-EOF
class #{table_name.singularize.capitalize}
 
  include DataMapper::Resource
 
  def self.default_repository_name
    :couchdb
  end
 
  # required for CouchDB
  property :id, String, :key => true, :field => :_id
  property :rev, String, :field => :_rev
 
  # regular properties
EOF
    file.write class_header
    ActiveRecord::Base.connection.columns(table_name).each do |c|
      if !c.sql_type.scan('integer').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('character').empty?
        file.write "  property :#{c.name}, String\n"
      elsif !c.sql_type.scan('text').empty?
        file.write "  property :#{c.name}, Text\n"
      elsif !c.sql_type.scan('datetime').empty?
        file.write "  property :#{c.name}, DateTime\n"
      elsif !c.sql_type.scan('uuid').empty?
        file.write "  property :#{c.name}, String\n"
      elsif !c.sql_type.scan('boolean').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('double precision').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('bigint').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('smallint').empty?
        file.write "  property :#{c.name}, Integer\n"
      elsif !c.sql_type.scan('date').empty?
        file.write "  property :#{c.name}, Date\n"
      elsif !c.sql_type.scan('timestamp without time zone').empty?
        file.write "  property :#{c.name}, DateTime\n"
      elsif !c.sql_type.scan('time without time zone').empty?
        file.write "  property :#{c.name}, Time\n"
      elsif !c.sql_type.scan('bytea').empty?
        puts "INVALID: column #{c.name} in #{table_name.capitalize}: binary data is not allowed here."
        puts 'You will need to rethink your schema for use in CouchDB. Hint: Attachments.'
        puts 'SKIPPING!'
      else
        file.write "COLUMN: #{c.name}, UNKNOWN DATA TYPE #{c.sql_type} - CHANGE ME!\n"
      end
    end
    file.write "end\n"
    file.write "\n"
    end
  end
  puts "dump succeeded. Look in db/dm_temp/models.rb"
end

StrokeDB

Tuesday, February 12th, 2008

Another competitor to the exciting CouchDB project has emerged – and this time it’s in pure Ruby, not god damn Erlang, so it’s very interesting to me. Check it out here.

By the way, another project I’ve talked about before, ThingFish, has been through countless revisions and code growth – there’s not a day that goes by when I’m not treated to 50 or so changed files in response to my svn up. But the focus of the project seems to have changed from being a document-centric database to being some kind of network file store. None of it works, as far as I can tell, and I have no idea what they are doing.. Thingfish developers: what on earth is your project for?

Anyway, exciting times for this class of database, which I strongly believe is the future of large-scale web apps.

Installing couchdb on RHEL4

Tuesday, October 23rd, 2007

1. Install erlang

Ignore the instructions here which are misleading and outdated.

cd ~/src
mkdir erlang
cd erlang
wget http://www.erlang.org/download/otp_src_R11B-5.tar.gz
tar xzvf otp_src_R11B-5.tar.gz
cd otp_src_R11B-5
./configure && make && sudo make install
## ignore warnings about jinterface and odbc

2. Solve dependencies. You’ll need EPEL

yum install icu libicu-devel

3. install couchdb

cd ~/src
svn checkout http://couchdb.googlecode.com/svn/trunk/ couchdb
cd couchdb
./bootstrap && ./configure && make && sudo make install

You may need to change one line in configure.ac: You don’t need to do this in latest builds (last tested 0.73a)

38
39
--FLAGS="-I${ERLANG_HEADERS_DIRECTORY} -I/opt/local/lib/erlang/usr/include/"
++FLAGS="-I${ERLANG_HEADERS_DIRECTORY} -I/usr/local/lib/erlang/usr/include/"

or if you don’t want to do that, this nasty hack will paper over the problem: (unnecessary in recent builds)

ln -s /usr/local /opt/local

4. start couchdb

sudo couchdb
couch 0.7.0a548 (LogLevel=info)
CouchDB is starting.
CouchDB has started. Time to relax.

UPDATED to cover recent builds.

couchdb: switching to experimental branch

Friday, October 12th, 2007

Current CouchDB development is progressing in an experimental branch named “interface-changes”, and it’s not in trunk yet. As the name implies, the interface is changing and it’s very useful.

To switch your svn checkout to this branch, do this:

svn sw http://couchdb.googlecode.com/svn/branches/interfacechanges/
./bootstrap && ./configure --prefix=/opt/local && make clean && make && sudo make install

You can then have a look at the “documentation” to see some of the new changes.

While still alpha, it’s still a very interesting project. The prospect of getting something like Map/Reduce capability natively in a database is almost too exciting for words, if you’re a data nerd like me. Well, figuratively speaking, Map’s there, but Reduce isn’t yet – still, it’s great to get your hands on what seems sure to become a Big Thing so early. Oh, and in case you were wondering, COUCH stands for “Cluster of Unreliable Commodity Hardware”.

Imagine GFS and Map/Reduce, baked into a single databse, with JSON in/out, pluggable query language, and native REST .. what’s not to love?

Laika’s ThingFish is a Ruby-based competitor. You might think I’d be more interested in that, since it’s in my favourite language! Not so. The very thought of a Database programmed in Ruby actually gives me an instant waking nightmare along the lines of running through treacle, gulliver in lilliputia, 80286SXs with 12MB of RAM, etc. And using mongrel server for a database!

UPDATE: it’s been merged to trunk, so just forget that interface-changes stuff.

Installing CouchDB on MacOSX

Monday, October 8th, 2007

If you, like me, have been keeping an eye on the extremely interesting CouchDB Project then maybe you’ve thought about playing around with it a bit on your Mac. This turns out to be very simple.

I assume you have macports w/ subversion (and XCode, of course) installed already. Most of this comes straight from the Readme but I hate reading those, I just want someone to write what I have to do – maybe you do too, so here you are!

UPDATE: 11th October

Reinstalling after noticing 60 commits in 3 days .. now that’s active development.

(more…)