Comments on: The Wrath of DrWho, or Unpredictable Hadoop Memory Usage http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/ For engineers by engineers Mon, 30 Aug 2010 16:14:07 -0700 hourly 1 By: Todd Lipcon http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1475 Todd Lipcon Mon, 11 Jan 2010 04:47:17 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1475 Here's a bit of further explanation (actually mostly copy-pasted from our response to Rapleaf's support ticket): Since Hadoop uses fork and not vfork, it actually needs to make another copy of the memory space of your process. Fork uses copy-on-write, so it's still fairly efficient, but if the process that's forking has a large memory footprint, it can get ENOMEM if there isn't a ton of swap available. Basically, Linux will overcommit memory up to a certain configurable amount, but if the total vmem footprint of your processes exceeds a certain threshold, it will stop allowing it. One way to confirm that this is the issue is to temporarily enable complete overcommitting of memory by tweaking /proc/sys/vm/overcommit_memory to the value 1 ("always overcommit"). I cannot recommend running with this setting - the default heuristic (value 0) is much safer. Here are some short-term workarounds: The first is to simply configure this node with a ton of swap space. This will figure into the memory overcommit calculations so that it won't return this error. Alternatively, you could fiddle with the value of /proc/sys/vm/overcommit_ratio - perhaps setting it to 100 or 200 instead of the default 50 would solve the issue. The second workaround would be to force the UserGroupInformation to get cached earlier in your process, before you allocate large amounts of memory. You should be able to do this by calling UserGroupInformation.login(conf) at the top of your application. Long term, this is an acknowledged bug/issue with Hadoop. http://issues.apache.org/jira/browse/HADOOP-4998 is the JIRA that will hopefully solve it - it proposes to implement a JNI library that will allow access to these unixy things without having to fork separate processes. Eli from Cloudera has started working on this ticket and we hope to include it in CDH once it's finished and committed. -Todd from Cloudera Here’s a bit of further explanation (actually mostly copy-pasted from our response to Rapleaf’s support ticket):

Since Hadoop uses fork and not vfork, it actually needs to make another copy of the memory space of your process. Fork uses copy-on-write, so it’s still fairly efficient, but if the process that’s forking has a large memory footprint, it can get ENOMEM if there isn’t a ton of swap available. Basically, Linux will overcommit memory up to a certain configurable amount, but if the total vmem footprint of your processes exceeds a certain threshold, it will stop allowing it.

One way to confirm that this is the issue is to temporarily enable complete overcommitting of memory by tweaking /proc/sys/vm/overcommit_memory to the value 1 (“always overcommit”). I cannot recommend running with this setting – the default heuristic (value 0) is much safer.

Here are some short-term workarounds:

The first is to simply configure this node with a ton of swap space. This will figure into the memory overcommit calculations so that it won’t return this error. Alternatively, you could fiddle with the value of /proc/sys/vm/overcommit_ratio – perhaps setting it to 100 or 200 instead of the default 50 would solve the issue.

The second workaround would be to force the UserGroupInformation to get cached earlier in your process, before you allocate large amounts of memory. You should be able to do this by calling UserGroupInformation.login(conf) at the top of your application.

Long term, this is an acknowledged bug/issue with Hadoop. http://issues.apache.org/jira/browse/HADOOP-4998 is the JIRA that will hopefully solve it – it proposes to implement a JNI library that will allow access to these unixy things without having to fork separate processes. Eli from Cloudera has started working on this ticket and we hope to include it in CDH once it’s finished and committed.

-Todd from Cloudera

]]> By: Charles Glommen http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1469 Charles Glommen Wed, 06 Jan 2010 23:54:54 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1469 Ah, I see. I'm not explicitly using copyToLocal, but have noticed that call in the stack trace when this occurs. It seems to occur when I provide tens of thousands of input files for a single map/reduce. Ah, I see. I’m not explicitly using copyToLocal, but have noticed that call in the stack trace when this occurs. It seems to occur when I provide tens of thousands of input files for a single map/reduce.

]]> By: bryan http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1468 bryan Wed, 06 Jan 2010 23:40:19 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1468 @Charles: Sadly, it's fairly application-specific. In our instance, we just made sure that we did all of our copyToLocal calls before we started doing the memory-intensive side of the application. This way, the shell calls happened before the JVM had accumulated a large heap size. I'd recommend trying this, and if that won't work, you could consider not using copyToLocal (that is, streaming the files directly). @Charles: Sadly, it’s fairly application-specific. In our instance, we just made sure that we did all of our copyToLocal calls before we started doing the memory-intensive side of the application. This way, the shell calls happened before the JVM had accumulated a large heap size. I’d recommend trying this, and if that won’t work, you could consider not using copyToLocal (that is, streaming the files directly).

]]> By: Charles Glommen http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1467 Charles Glommen Wed, 06 Jan 2010 23:31:05 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1467 I am experiencing a similar, if not identical, issue with the OutOfMemoryError. Can you provide more details into how you fixed this? I am experiencing a similar, if not identical, issue with the OutOfMemoryError. Can you provide more details into how you fixed this?

]]> By: bryan http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1465 bryan Wed, 06 Jan 2010 16:00:03 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1465 @harish: it definitely should, and most of the time, it does. However, I think there are some oddities around what happens when you ask the OS for 2x the physical memory of the machine. There's this "overcommit" parameter that governs this behavior which I must admit to not fully understanding. Additionally, I have a suspicion that when there's a reasonable amount of concurrency, the garbage collector could somehow interfere with copy on write. @harish: it definitely should, and most of the time, it does. However, I think there are some oddities around what happens when you ask the OS for 2x the physical memory of the machine. There’s this “overcommit” parameter that governs this behavior which I must admit to not fully understanding. Additionally, I have a suspicion that when there’s a reasonable amount of concurrency, the garbage collector could somehow interfere with copy on write.

]]> By: Harish Mallipeddi http://blog.rapleaf.com/dev/2010/01/05/the-wrath-of-drwho-or-unpredictable-hadoop-memory-usage/comment-page-1/#comment-1464 Harish Mallipeddi Wed, 06 Jan 2010 13:09:49 +0000 http://blog.rapleaf.com/dev/?p=382#comment-1464 This is interesting. Shouldn't copy-on-write semantics of fork() prevent this problem? This is interesting. Shouldn’t copy-on-write semantics of fork() prevent this problem?

]]>