This was easy to implement once I found the programmable completion tool by Ian Macdonald. I just added the following section to bash_completion:

[cc lang="bash" tab_size="2"]
# hdfs(1) completion
#
have hadoop &&
_hdfs()
{
local cur prev

COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}
prev=${COMP_WORDS[COMP_CWORD-1]}

if [[ "$prev" == hdfs ]]; then
COMPREPLY=( $( compgen -W ‘-ls -lsr -du -dus -count -mv -cp -rm
-rmr -expunge -put -copyFromLocal -moveToLocal -mkdir -setrep
-touchz -test -stat -tail -chmod -chown -chgrp -help’ — $cur ) )
fi

if [[ "$prev" == -ls ]] || [[ "$prev" == -lsr ]] ||
[[ "$prev" == -du ]] || [[ "$prev" == -dus ]] ||
[[ "$prev" == -cat ]] || [[ "$prev" == -mkdir ]] ||
[[ "$prev" == -put ]] || [[ "$prev" == -rm ]] ||
[[ "$prev" == -rmr ]] || [[ "$prev" == -tail ]] ||
[[ "$prev" == -cp ]]; then
if [[ -z "$cur" ]]; then
COMPREPLY=( $( compgen -W “$( hdfs -ls / 2>-|grep -v ^Found|awk ‘{print $8}’ )” — “$cur” ) )
elif [[ `echo $cur | grep /$` ]]; then
COMPREPLY=( $( compgen -W “$( hdfs -ls $cur 2>-|grep -v ^Found|awk ‘{print $8}’ )” — “$cur” ) )
else
COMPREPLY=( $( compgen -W “$( hdfs -ls $cur* 2>-|grep -v ^Found|awk ‘{print $8}’ )” — “$cur” ) )
fi
fi
} &&
complete -F _hdfs hdfs
[/cc]

I’m sure there are some ways to make the code more elegant, but it is called HackLeaf, after all. This bit of code builds on top of other functions in the script, but the basic idea is pretty simple. cur contains the current word you are typing, so this would be a partial command or partial path. prev contains the previous word. If the previous word is hdfs, then we present the user with valid arguments to hdfs. If the previous word is -ls (or any other command where you want a path/file), then present the user with the possibilities for that path or partial path. HDFS defaults to the user’s home directory if no path is provided, so we override that by presenting the user with the possibilities under “/”. Finally, COMPREPLY returns the possibilities to the user on the command-line.

Be sure to check out some of the other features of bash_completion, particularly ssh and chkconfig.

One way to do this would be to configure each job at the code level. Unfortunately, this can be tedious and error-prone, since if you add a new job to the workflow you need to remember to add the proper configurations. Fortunately, there’s a better way.

Hadoop has a static method called “Configuration.addDefaultResource” that allows you to specify a file to be loaded into the configuration by default. Like Cascading Style Sheets, Hadoop will load the configurations one file at a time, with configurations from later files overriding those from earlier ones. A static initializer in the JobConf class causes Hadoop to load in “mapred-default.xml” and “mapred-site.xml”.

To create an application level configuration, you will want to perform the following steps:

1. Create an “application-site.xml” file and put it in the same directory from which you run your job jar.
2. In the main method of your code, add the lines:

[cc lang="java"]
new JobConf(); //ensure mapred-default and mapred-site get loaded in first
Configuration.addDefaultResource(“application-site.xml”);
[/cc]

3. Ensure that “.” is in your classpath when you run the job jar so that Hadoop can find the “application-site.xml” resource.

A side benefit of this approach is that if you need to tweak some settings in the middle of a workflow, you just need to edit the application-site.xml file and each subsequent job will pick up the new settings.

The one problem we encountered when I received the first test machine was getting the Serial Console to work though the machine’s IPMI interface.  Rapleaf has some other Dell servers and never had any problems, but this machine is new and shiny and just didn’t work with our default configuration.

The first change we needed to make was change the way the BIOS redirects the serial console.  To enter the BIOS press “F2″ after the machine has finished checking its memory.  Once the computer gets to the BIOS go down to “Serial Communication,” you should be seeing the following screen.

The options should be as follows:

• Serial Communication ………. On with Console Redirection via COM2
• Serial Port Address …………… Serial Device1=COM1,Serial Device2=COM2
• External Serial Connector ….. Serial Device2
• Failsafe Baud Rate …………… 115200
• Remote Terminal Type ………. VT100/VT220
• Redirection After Boot ……….. Enabled

When this is done, exit the BIOS and enable IPMI on the machine by pressing CTRL-E when the prompt to modify the IPMI configuration appears and give it an IP address either static or DHCP.

Now there’s a couple of modifications that need to be done to Linux for this to work.

1) The first is modify your /boot/grub/grub.conf file and add “console=ttyS1,115200″ at the end of your kernel parameters.  Ours looks like this:

title Rapleaf Linux (2.6.22)
root (hd0,0)
kernel /vmlinuz-2.6.22 ro root=LABEL=/ console=ttyS1,115200
initrd /initrd-2.6.22.img

2) The last two lines in /etc/inittab should be:

# Run agetty COM2/ttyS1
s1:2345:respawn:/sbin/agetty -L -f /etc/issueserial ttyS1 115200 vt100-nav

3) The last line of /etc/securetty should be

ttyS1

Once all this is done the machine can be rebooted and you should be able to interact with the boot process through IPMI.  Good luck!

A Hadoop job fails if the same task fails some predetermined amount of times (by default, four). This is set through the properties “mapred.map.max.attempts” and “mapred.reduce.max.attempts”. For a job to fail randomly, an individual task will need to fail randomly this predetermined amount of times. A task can fail randomly for a variety of reasons – a few of the ones we’ve seen are disks getting full, a variety of bugs in Hadoop, and hardware failures.

The formula for the probability of a job failing randomly can be derived as follows:


Pr[individual task failing maximum #times] = Pr[task failing] ^ (max task failures)
Pr[task succeeding] = 1 - Pr[individual task failing maximum #times]
Pr[job succeeding] = Pr[task succeeding] ^ (num tasks)
Pr[job failing] = 1 - Pr[job succeeding]

Pr[job failing] = 1 - (1-Pr[task failing] ^ (max task failures))^(num tasks)


The maximum amount of task failures is set through the property “mapred.max.tracker.failures” and defaults to 4.

Let’s take a significant workload of 100,000 map tasks and see what the numbers look like:

As the probability of a task failing goes above 1%, the probability of the job failing rapidly increases. It is very important to keep the cluster stable and keep the failure rate relatively small, as these numbers show Hadoop’s failure model only goes so far. We can also see the importance of the “max task failures” parameter, as values under 4 cause the probability of job failures to rise to significant values even with a 0.5% probability of task failure.

Reducers run for a much longer period of time than mappers, which means a reducer has more time for a random event to cause it to fail. We can therefore say that the probability of a reducer failing is much higher than a mapper failing. This is balanced out by the fact that there are a much smaller amount of reducers. Let’s look at some numbers more representative of a job failing due to reducers failing:

The probabilities of a reducer failing need to go up to 10% to have a significant chance of failure.

Bad Nodes

One more variable to consider in the model is bad nodes. Oftentimes nodes go bad and every task run on them fails, whether because of a disk going bad, the disk filling up, or other causes. With a bad node, you typically see a handful of mappers and reducers fail before the node gets blacklisted and no more tasks are assigned to it. In order to simplify our analysis, let’s assume that each bad node causes a fixed number of tasks to fail. Additionally, let’s assume a task can only be affected by a bad node once, which is reasonable because nodes are blacklisted fairly quickly. Let’s call the tasks which fail once due to a bad node “b-tasks” and the other tasks “n-tasks”. A “b-task” starts with one failure, so it needs to fail randomly “max task failures – 1″ times to cause the job to fail. On our cluster, we typically see a bad node cause three tasks to automatically fail, so using that number the modified formula ends up looking like:


#b-tasks = #bad nodes * 3
Pr[all b-tasks succeeding] = (1-Pr[task failing] ^ (max task failures - 1))^(#b-tasks)
Pr[all n-tasks succeeding] = (1-Pr[task failing] ^ (max task failures))^(num tasks - #b-tasks)
Pr[job succeeding] = Pr[all b-tasks succeeding] * Pr[all n-tasks succeeding]
Pr[job succeeding] = (1-Pr[task failing] ^ (max task failures - 1))^(#b-tasks) * (1-Pr[task failing] ^ (max task failures))^(num tasks - #b-tasks)
Pr[job failing] = 1 - Pr[job succeeding]

Pr[job failing] = 1 - (1-Pr[task failing] ^ (max task failures - 1))^(#b-tasks) * (1-Pr[task failing] ^ (max task failures))^(num tasks - #b-tasks)


Since there are so many mappers, the results of the formula won’t change for a handful of bad nodes. Given that the number of reducers is relatively small though, the numbers do change somewhat:

Happily the numbers aren’t too drastic – five bad nodes causes the failure rate to increase by 1.5x to 2x.

In the end, Hadoop is fairly fault tolerant as long as the probability of a task failing is kept relatively low. Based on the numbers we’ve looked at, 4 is a good value to use for “max task failures”, and you should start worrying about cluster stability when the task failure rate approaches 1%. You could always increase the “max task failures” properties to increase robustness, but if you are having that many failures you will be suffering performance penalties and would be better off making your cluster more stable.