Author Archives: nathan marz

Cycles of Doom in Batch Processing Workflows

By nathan marz | Published: December 18, 2009

We integrate new data into our databases via a large batch processing workflow. The execution time of this workflow directly affects the time it takes to get new data to our customers, so keeping the runtime small is of paramount importance to us. There’s an interesting effect that can happen which we’ve dubbed the “cycle [...]

Posted in Hadoop, MapReduce | 10 Comments

Dead Simple MapReduce Workflow Configuration

By nathan marz | Published: November 9, 2009

If you use MapReduce for any real-world application, chances are your workflow consists of more than one MapReduce job. Rapleaf has workflows consisting of over one hundred jobs. A lot of times, you need to make configurations to the workflow that should apply to every job. For example, you may want each job to run [...]

Posted in Hadoop, MapReduce | Tagged , | Leave a comment

Batch Querying with Cascading

By nathan marz | Published: September 25, 2009

As promised at the meetup yesterday, I’ve released the source code for batch querying with Cascading at http://github.com/nathanmarz/cascading-batch-query/ Here is the presentation from the meetup: Batch Querying with Cascading

Posted in Miscellaneous | 3 Comments

Evaluating a Predictive Model

By nathan marz | Published: September 16, 2009

We often try to correlate our data with various consumer behaviors, such as paying back loans or completing a transaction. To do this we produce models that emit the probability of the person doing the behavior. In this post, I’ll describe a technique we use for evaluating the performance of our predictive models. There are [...]

Posted in Miscellaneous | 6 Comments

Consistent Sampling using Hash Function

By nathan marz | Published: August 18, 2009

I learned this technique in a lecture I saw by Ron Kohavi. The idea is that we want to create a random sample but we want to make sure that if a value appears multiple times, it is either always sampled or never sampled. Let’s look at a specific example – let’s say we have [...]

Posted in Miscellaneous | Leave a comment

Using random numbers in Hadoop MapReduce is dangerous

By nathan marz | Published: August 14, 2009

If you’re using random numbers in your MapReduce jobs, you could be suffering from data loss. The cause of the data loss is subtle and happens due to Hadoop’s behavior in dealing with TaskTrackers that are lost in the middle of a job. Let’s go through an example of how the data loss can occur. [...]

Posted in Hadoop, MapReduce | 11 Comments

A Glance at the Hadoop Failure Model

By nathan marz | Published: July 17, 2009

Hadoop is designed to be a fault tolerant system. Jobs should be resilient to nodes going down and other random failures. Hadoop isn’t perfect however, as I still see jobs failing due to random causes every now and again. I decided to investigate the significance of the different factors that play into a job failing. [...]

Posted in Hadoop, MapReduce | Tagged , | 3 Comments

A new Cascading pipe – MultiGroupBy

By nathan marz | Published: June 23, 2009

Cascading is an awesome tool, but there’s a particular situation I have run into a few times where the abstractions have broken down. The situation occurs when you have multiple pipes that you need to group together on a common key, but other than the key the pipes have different fields. Let’s use the following [...]

Posted in Cascading, Hadoop, MapReduce | 1 Comment

Rapleaf Challenge Problem

By nathan marz | Published: December 11, 2008

We’ve created a challenge problem based on one of the core problems we’ve had to solve in our MapReduce workflow. A word of warning – this isn’t one of those toy problems other companies put out on their careers page. This one is so hard it will make you cry. Rapleaf Challenge Problem

Posted in Hadoop, Miscellaneous | 12 Comments

Give me liberty or give me death, but don’t give me small files!

By nathan marz | Published: November 20, 2008

Small files are the bane of Hadoop MapReduce. 300GB of data kept in a few files versus thousands of files can cause a 100x performance difference in jobs run over that data. For this reason, it is of paramount importance to keep files on HDFS large. There are many reasons for this. With larger files, [...]

Posted in Hadoop, MapReduce | 4 Comments

  • Rapleaf Is Hiring!

    We are looking for engineers who want to solve challenging problems.

    We have great people, do great work, and have great perks.

    Know someone who might be interested? Refer a friend and get $5,000 for successful hires.

    See our current openings at
    www.rapleaf.com/careers