Tag Archives: Hadoop

Command-line auto completion for Hadoop DFS commands

By chris | Published: November 17, 2009
We like to keep things simple here at Rapleaf. One small tweak we made right after we installed hadoop was to alias 'hadoop dfs' to 'hdfs'. It rolls off the fingers nicely. We are also constantly typing 'hdfs -ls this' or 'hdfs -du that'. If we are not sure what this/that [...]
Posted in HDFS, Hadoop, bash | Also tagged , , , , , | 4 Comments

Dead Simple MapReduce Workflow Configuration

By Nathan Marz | Published: November 9, 2009
If you use MapReduce for any real-world application, chances are your workflow consists of more than one MapReduce job. Rapleaf has workflows consisting of over one hundred jobs. A lot of times, you need to make configurations to the workflow that should apply to every job. For example, you may want each job to run [...]
Posted in Hadoop, MapReduce | Also tagged | Leave a comment

Getting the serial terminal to work over IPMI on a Dell R410

By cesar | Published: October 19, 2009
As avid readers of the blog know, we use Hadoop a lot and talk about it quite a bit. We are in the process of expanding our Hadoop cluster and decided to go with the new Dell R410 1U machines.  From talks with other Hadoop users the sweet-spot is one spindle (drive) for every [...]
Posted in Hadoop | Also tagged | Leave a comment

A Glance at the Hadoop Failure Model

By Nathan Marz | Published: July 17, 2009
Hadoop is designed to be a fault tolerant system. Jobs should be resilient to nodes going down and other random failures. Hadoop isn’t perfect however, as I still see jobs failing due to random causes every now and again. I decided to investigate the significance of the different factors that play into a job failing. A [...]
Posted in Hadoop, MapReduce | Also tagged | 2 Comments