Comments on: Cycles of Doom in Batch Processing Workflows

By: James Brown

James Brown — Fri, 08 Jan 2010 19:46:59 +0000

@Jon: LOL at the concept that batch processing time is only CPU bound in any given system. In this context, adding disks does not mean I am only looking to increase storage.

I will leave your misinterpretation of my 1982 comment as a homework exercise.

By: Jon

Jon — Thu, 07 Jan 2010 20:45:17 +0000

LOL at hadoop being 1982 or the concept that more disk space reduces processing time. When you design/architect a system and find that your “go to solution” is throwing hardware at it, you know you have a poor design.

By: Trøll Åndersen

Trøll Åndersen — Wed, 06 Jan 2010 19:24:15 +0000

@James Brown In Norway’s summer we salt your kind and leave you out in the sun for a few weeks. We are not amused by your joke involving integers and lunar cycles.

@ Aaron Faen, why are you paying so much for disk? That sounds like the price we paid for back when I was managing the Iralalaru project in ‘02.

By: bryan

bryan — Mon, 04 Jan 2010 16:52:11 +0000

@James Brown: Disks and RAM may be cheap, but the machines to house them are not. Nor, more importantly, are the cabinets, cooling, or power. Blithely scaling up (or out) is not always the solution, no matter what all the cloud folks would like us to believe.

@Aaron Anderson: In our application, we’re not talking about a mainframe, so we don’t have to buy enterprise disks. We have a big cluster of “commodity” machines that use standard SATA disks, so adding 2TB is not nearly the $200k investment. However, having an extra 2TB of storage is pretty much never the problem we need to solve.

By: Aaron Anderson

Aaron Anderson — Mon, 04 Jan 2010 02:37:04 +0000

Enterprise disk is NOT cheap. I really hate that people think that. Even IT managers say that “disk is cheap, throw some in the mainframe… I’ve got a 1.5TB drive on my desk that cost me $150 dollars. ” When mainframe storage for an IBM series might cost upwards of 200,000 dollars for a disk pack to hold 2TB.

This article is so 2010.

By: James Brown

James Brown — Wed, 30 Dec 2009 17:56:18 +0000

Why should I waste my time with math when I can keep adding cheap disks and ram?

This article is so 1982.

By: The mathematics behind Hadoop-based systems — thoughts from the red planet

The mathematics behind Hadoop-based systems — thoughts from the red planet — Tue, 29 Dec 2009 05:35:13 +0000

[...] originally wrote about this effect here. I hope you find this writeup to be clearer and more extensive. [...]

By: markI

markI — Thu, 24 Dec 2009 04:55:04 +0000

huuu… If you increase the number of machines and diminish the size of the blocks, theoretically you can reduce a lot of overhead, no? Then again if real time is your goal a finite state machine approach might become more scalable, but that means redo your architecture without Hadoop! How much data are we talking about? 100Tb? 1Pb?

By: Nathan Marz

Nathan Marz — Fri, 18 Dec 2009 21:28:47 +0000

@MikeD: It depends on how scalable your workflow is. If your processors contend on shared resources, then you will have limited ability to reduce that “P” parameter by increasing the number of processors.

By: MikeD

MikeD — Fri, 18 Dec 2009 19:38:19 +0000

How is this model affected by elastic computing (spinning up more processing machines)? Is that not helpful since bottlenecks may be storage services?