Hadoop Meeting

I am blogging this from Santa Clara, attending (alongwith, my guess anouther 400 people) the first Hadoop meeting, organized by Yahoo.  What has brought all of us here?  It is clear that there are a class of applications (TBD, the full scale of what they are) that benefit from the infrastructure that the googles and the yahoos use.  Of course, large scale text analytics, of course graph analysis, but many many more. Log analysis, data mining, optimization to name a few more.  So Doug Cutting (the same guy who gave us Lucene, which I love) has now given us Hadoop (map-reduce) and other people have built PIGs and JAQLs and HBases above and now not only Yahoo, but also, by the show of hands here, many many other people are beginning to use it for their applications.

Technically, it is easy to get jaded and say that this is all old wine in new bottle.  That to some effect was what Mike Stonebraker and Dave DeWitt said in this blog.  It is also a full employment act for reimplementation of all database gems in this new model.  However, that is besides the point.  Innovation and invention should not be confused.  Hadoop is innovative, it might or might not be inventive. 

It is great to see the interest in this open community.  I am here because I feel this is very important for IBM clients in the future.  I am trying to understand the use cases, and therefore the application part of the day is much more useful for me than just the pure technology, which having done databases, I get easily.