Comments for Engineering Rapleaf http://blog.rapleaf.com/dev For engineers by engineers Mon, 30 Aug 2010 16:14:07 -0700 hourly 1 Comment on Analyzing some interesting networks for Map/Reduce clusters by bryan http://blog.rapleaf.com/dev/2010/08/26/analyzing-some-interesting-networks-for-mapreduce-clusters/comment-page-1/#comment-1673 bryan Mon, 30 Aug 2010 16:14:07 +0000 http://blog.rapleaf.com/dev/?p=775#comment-1673 Today, we're using a single core because we don't actually need redundancy. Our backend system is isolated from anything that touches the user, so if the core happened to fail, we'd be alright while we replaced it. I'm not all that sure what you mean about layer 3 vs layer 4 in this context. As far as I can tell, we don't really need layer 4 switching capabilities like load balancing and flow control - every node is doing the exact same thing all the time. We have certainly considered adding "shortcut" links between individual pairs of racks in order to take more traffic off the core switch, but so far it's not clear we need that extra throughput, and the benefit would come at a configuration cost. Today, we’re using a single core because we don’t actually need redundancy. Our backend system is isolated from anything that touches the user, so if the core happened to fail, we’d be alright while we replaced it.

I’m not all that sure what you mean about layer 3 vs layer 4 in this context. As far as I can tell, we don’t really need layer 4 switching capabilities like load balancing and flow control – every node is doing the exact same thing all the time. We have certainly considered adding “shortcut” links between individual pairs of racks in order to take more traffic off the core switch, but so far it’s not clear we need that extra throughput, and the benefit would come at a configuration cost.

]]> Comment on Analyzing some interesting networks for Map/Reduce clusters by leon http://blog.rapleaf.com/dev/2010/08/26/analyzing-some-interesting-networks-for-mapreduce-clusters/comment-page-1/#comment-1671 leon Sun, 29 Aug 2010 22:28:15 +0000 http://blog.rapleaf.com/dev/?p=775#comment-1671 Interesting and informative post, but a few things jump out at me; Any particular reason you seem to be doing alot traffic management and aggregation at layer 3 instead of traffic management at layer 4 with standard routing mechanisms? also, why's there a single core? is redundancy not a requirement for the topology at all? (performance over throughput? - but even then i'm confused as to why there's no inter-rack / layer 4 interconnects if high throughput is required? ) Interesting and informative post, but a few things jump out at me; Any particular reason you seem to be doing alot traffic management and aggregation at layer 3 instead of traffic management at layer 4 with standard routing mechanisms? also, why’s there a single core? is redundancy not a requirement for the topology at all? (performance over throughput? – but even then i’m confused as to why there’s no inter-rack / layer 4 interconnects if high throughput is required? )

]]> Comment on Very Fast Batch Superset Queries by links for 2010-08-15 « Blarney Fellow http://blog.rapleaf.com/dev/2010/08/11/very-fast-batch-superset-queries/comment-page-1/#comment-1649 links for 2010-08-15 « Blarney Fellow Mon, 16 Aug 2010 01:34:11 +0000 http://blog.rapleaf.com/dev/?p=674#comment-1649 [...] Very Fast Batch Superset Queries | Engineering Rapleaf (tags: trie clustering algorithm search) [...] [...] Very Fast Batch Superset Queries | Engineering Rapleaf (tags: trie clustering algorithm search) [...]

]]> Comment on BloomFilter by Bloom Filter – It’s Variants- and their Applications « Appolo85's Blog http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/comment-page-1/#comment-1632 Bloom Filter – It’s Variants- and their Applications « Appolo85's Blog Tue, 03 Aug 2010 11:01:17 +0000 http://blog.rapleaf.com/dev/?p=6#comment-1632 [...] This post from Rapleaf  was an interesting read. [...] [...] This post from Rapleaf  was an interesting read. [...]

]]> Comment on Anonymouse by Rattle » Week #1275 http://blog.rapleaf.com/dev/2010/07/20/anonymouse/comment-page-1/#comment-1627 Rattle » Week #1275 Fri, 30 Jul 2010 14:48:47 +0000 http://blog.rapleaf.com/dev/?p=579#comment-1627 [...] Anonymouse – great example of user privacy protection [...] [...] Anonymouse – great example of user privacy protection [...]

]]> Comment on Anonymouse by Microsoft Made $4.5 Billion Last Quarter; IBM Getting Analytical; New York Times Digital Revenues At 26% http://blog.rapleaf.com/dev/2010/07/20/anonymouse/comment-page-1/#comment-1620 Microsoft Made $4.5 Billion Last Quarter; IBM Getting Analytical; New York Times Digital Revenues At 26% Fri, 23 Jul 2010 04:05:44 +0000 http://blog.rapleaf.com/dev/?p=579#comment-1620 [...] The Rapleaf engineering team takes the microphone on the company's development blog to introduce what it calls "Anonymouse" and how it tackles privacy requirements. From the blog: "The goal of Anonymouse is to selectively exclude data from the cookies we drop so that our users are sufficiently indistinguishable." Take a deep dive here. [...] [...] The Rapleaf engineering team takes the microphone on the company's development blog to introduce what it calls "Anonymouse" and how it tackles privacy requirements. From the blog: "The goal of Anonymouse is to selectively exclude data from the cookies we drop so that our users are sufficiently indistinguishable." Take a deep dive here. [...]

]]> Comment on Anonymouse by Abram http://blog.rapleaf.com/dev/2010/07/20/anonymouse/comment-page-1/#comment-1618 Abram Thu, 22 Jul 2010 08:03:59 +0000 http://blog.rapleaf.com/dev/?p=579#comment-1618 This post is awesome. I've been kicking around ideas like this (though I don't actually have datasets yet). This post brought my thinking ahead many steps down the path. Thank you! Keep it up! You're solving a human problem, not just a business problem. Surely you've changed the future of privacy. This post is awesome. I’ve been kicking around ideas like this (though I don’t actually have datasets yet). This post brought my thinking ahead many steps down the path. Thank you! Keep it up! You’re solving a human problem, not just a business problem. Surely you’ve changed the future of privacy.

]]> Comment on Anonymouse by How Rapleaf Anonymizes Information | Rapleaf http://blog.rapleaf.com/dev/2010/07/20/anonymouse/comment-page-1/#comment-1617 How Rapleaf Anonymizes Information | Rapleaf Wed, 21 Jul 2010 19:02:35 +0000 http://blog.rapleaf.com/dev/?p=579#comment-1617 [...] Read the full post here Share this post: [...] [...] Read the full post here Share this post: [...]

]]> Comment on Fully async Thrift client in Java by bryan http://blog.rapleaf.com/dev/2010/06/23/fully-async-thrift-client-in-java/comment-page-1/#comment-1613 bryan Tue, 13 Jul 2010 16:17:10 +0000 http://blog.rapleaf.com/dev/?p=572#comment-1613 Wow, you are totally correct. I improved the tests, and it uncovered exactly the problem you highlighted. I opened https://issues.apache.org/jira/browse/THRIFT-818 to track the issue, and I have a patch up with a fix already. I'd love it if you could try it out and see if it makes everything work for you. Wow, you are totally correct. I improved the tests, and it uncovered exactly the problem you highlighted.

I opened https://issues.apache.org/jira/browse/THRIFT-818 to track the issue, and I have a patch up with a fix already. I’d love it if you could try it out and see if it makes everything work for you.

]]> Comment on Fully async Thrift client in Java by Michael Brindamour http://blog.rapleaf.com/dev/2010/06/23/fully-async-thrift-client-in-java/comment-page-1/#comment-1612 Michael Brindamour Tue, 13 Jul 2010 15:16:31 +0000 http://blog.rapleaf.com/dev/?p=572#comment-1612 Thanks! Yes, I discovered that shortly after, looking at the test. I'm very close - remaining issue is that it appears that the method call arguments are not being sent along with the call - looking at the TAsyncMethodCall class, it seems that prepareMethodCall, which calls write_args, is called during creation, before the arguments get set on the object. I tried a couple of work arounds, but haven't been successful yet. Have you tried this code, and verified success of the message receipt and deserialization on the server side? The unit test shipped with the thrift code is returning a hardcoded value, so the test wouldn't actually expose this problem. Thanks! Yes, I discovered that shortly after, looking at the test. I’m very close – remaining issue is that it appears that the method call arguments are not being sent along with the call – looking at the TAsyncMethodCall class, it seems that prepareMethodCall, which calls write_args, is called during creation, before the arguments get set on the object. I tried a couple of work arounds, but haven’t been successful yet. Have you tried this code, and verified success of the message receipt and deserialization on the server side? The unit test shipped with the thrift code is returning a hardcoded value, so the test wouldn’t actually expose this problem.

]]>