Starfish: not yet MapReduce

Lucas Carlson writes about using his Starfish library to do MapReduce. He's not really doing MapReduce, but I like his Starfish library.

I mention this because I wrote a basic MapReduce library in Ruby at Sharpcast. Perhaps I can use Starfish as a good distributed processing abstraction.

Here's the ubiquitous word-frequency program using my MapReduce:

worker :word_count do
       # [filename,line] => list([word,1])
       map :reads => :text, :writes => :yaml do |filename,string|
               string.split.each {|w| emit_intermediate(w, 1)}
       end

       # [word,counts] => list([word,freq])
       reduce :reads => :yaml, :writes => :yaml do |word,counts|
               sum = counts.inject(0) {|sum,v| sum + v}
               emit word, sum
       end
end

I'll see if I can open-source it.