Dead Simple MapReduce Workflow Configuration

If you use MapReduce for any real-world application, chances are your workflow consists of more than one MapReduce job. Rapleaf has workflows consisting of over one hundred jobs. A lot of times, you need to make configurations to the workflow that should apply to every job. For example, you may want each job to run in the same fair scheduler pool or use a certain number of reducers.

One way to do this would be to configure each job at the code level. Unfortunately, this can be tedious and error-prone, since if you add a new job to the workflow you need to remember to add the proper configurations. Fortunately, there’s a better way.

Hadoop has a static method called “Configuration.addDefaultResource” that allows you to specify a file to be loaded into the configuration by default. Like Cascading Style Sheets, Hadoop will load the configurations one file at a time, with configurations from later files overriding those from earlier ones. A static initializer in the JobConf class causes Hadoop to load in “mapred-default.xml” and “mapred-site.xml”.

To create an application level configuration, you will want to perform the following steps:

1. Create an “application-site.xml” file and put it in the same directory from which you run your job jar.
2. In the main method of your code, add the lines:

3. Ensure that “.” is in your classpath when you run the job jar so that Hadoop can find the “application-site.xml” resource.

A side benefit of this approach is that if you need to tweak some settings in the middle of a workflow, you just need to edit the application-site.xml file and each subsequent job will pick up the new settings.