Monthly Archives: June 2009

A new Cascading pipe – MultiGroupBy

By Nathan Marz | Published: June 23, 2009
Cascading is an awesome tool, but there’s a particular situation I have run into a few times where the abstractions have broken down. The situation occurs when you have multiple pipes that you need to group together on a common key, but other than the key the pipes have different fields. Let’s use the following [...]
Posted in Cascading, Hadoop, MapReduce | 1 Comment

Multiple ways of copying data out of HDFS

By cesar | Published: June 11, 2009
There are multiple ways of getting data out of HDFS on to a local machine that does not belong to the HDFS cluster. The method used really depends on the needs of the data-transfer. The simplest way of getting data out of HDFS on to a non-cluster machine is to use the functions built into the [...]
Posted in HDFS, Hadoop | 4 Comments

Backing up Hadoop’s HDFS

By cesar | Published: June 5, 2009
Very little information can be found on methods of backing up data on a Hadoop cluster. To be clear, by Hadoop cluster I mean HDFS. Any solution would have to take a few things into account. The first is how to keep your data in multiple places for security. There is [...]
Posted in HDFS, Hadoop | 1 Comment

Thrift Union Pattern

By bryan | Published: June 3, 2009
At Rapleaf, we use Thrift structs as the basic cornerstone of many of our processes. I won’t go into great detail talking about Thrift in general, but suffice it to say that Thrift is easy and flexible enough to be used as the primary means of storing data and communicating between our various components. One limitation [...]
Posted in Miscellaneous, Thrift | 1 Comment