The Barrettocracy

I learned something else at the Lone Star Ruby Conference that I was just able to put to use.

First, some background…

I am using the built-in object serialization that Rails provides to marshal a whole bunch of objects while the app waits for the user’s confirmation to proceed. The objects are then read back and reconstituted. It’s really slick how it works. The objects are serialized in YAML format in the database. The database that I am using in this app is Sybase, and the data row cannot exceed 32k, so I was looking to make sure the yamilfied objects didn’t get over that size threshold.

The problem is that on big sets the CPU would spike, and the process would run longer than the browser would wait for a response. There was some processing to find existing versions of the objects in the database, and if there was an existing version to flag the changes. I had already optimized this pushing all the existing objects into a RBTree – perhaps a subject of a future post. It was still too slow, though.

When you have something that runs too slow, you can use ruby-prof to see where the most expensive operations are. Doing this I learned that to_yaml is pretty expensive.

To illustrate this I baked up a little example:

#!/usr/local/bin/ruby

require 'yaml'

class FooTest
  attr_accessor :the_hash, :thing1, :thing2, :long_string

  def initialize()
    self.thing1 ||= 100
    self.thing2 ||= 200
    self.long_string ||= "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    self.the_hash ||= { :a => 100, :b => 200, :c => "hello, world "}
  end
end

the_array = Array.new
1.upto(1000) do |x|
  test = FooTest.new
  the_array << test
  if the_array.to_yaml.size > 18000
    the_array = Array.new
    # this would be the serialization record insert
  end
end

I didn’t actually use ruby-prof to test this example – I just used time (time script.rb) from the command line since it is a cheesy little script.

Doing so, the results are:

real	0m25.523s
user	0m25.280s
sys	0m0.119s

When you change from assessing that the size is less than a certain number of characters to a certain size (if thearray.size > 100…) then the stats come down to this:

real	0m0.035s
user	0m0.028s
sys	0m0.006s

25 seconds vs 3 seconds. That’s a big win! Now I just need to calculate the average object size, lower it to give some wiggle room, and I have the number of objects I can dump at once.