-
Recent Posts
Popular Posts
- Rent or Own: Amazon EC2 vs. Colocation Comparison for Hadoop Clusters 27 comment(s) | 10834 view(s)
- Mysql Replication Adapter 26 comment(s) | 6657 view(s)
- Making sure Ruby Daemons die 20 comment(s) | 7353 view(s)
- Matching Impedance: When to use HBase 19 comment(s) | 22273 view(s)
- Goodbye MapReduce, Hello Cascading 17 comment(s) | 9672 view(s)
- Rapleaf Challenge Problem 12 comment(s) | 3792 view(s)
- BloomFilter 11 comment(s) | 5441 view(s)
- Using random numbers in Hadoop MapReduce is dangerous 11 comment(s) | 4029 view(s)
- Ruby and HBase 10 comment(s) | 5264 view(s)
- Cycles of Doom in Batch Processing Workflows 10 comment(s) | 2660 view(s)
Categories
- Anonymouse (1)
- Apache (1)
- bash (1)
- Cascading (6)
- Daemons (1)
- encryption (1)
- Extensions (2)
- Google (1)
- Grub (1)
- Hadoop (22)
- HBase (6)
- HDFS (4)
- Kickstart (1)
- MapReduce (9)
- mcrypt (1)
- Miscellaneous (26)
- Mongrel (2)
- Mysql (2)
- OpenSocial (1)
- Operations (1)
- Ruby (7)
- Security (2)
- Thrift (6)
- Xen (1)
Archives
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- March 2009
- February 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
Author Archives: bryan
Analyzing network load in Map/Reduce
Hadoop Map/Reduce can put a heavy toll on your network. Just how heavy, though, isn’t obvious. This is an especially important consideration when you are expanding your cluster. Rapleaf recently encountered this situation, and in the process we devised a neat theoretical model for analyzing how network topology affects Map/Reduce.
When does Hadoop put the most [...]
Thrift 0.4.0 Released
During the time it took for us to get everything sorted out for the 0.3.0 release, we accumulated more than enough changes to justify 0.4.0.
There are a ton more changes in this release. (You can find the full summary here.) In addition to the usual bug fixes and performance improvements, there are two really [...]
Posted in Miscellaneous Leave a comment
Cassandra Summit
I had the pleasure of attending Riptano’s very-well-executed Cassandra Summit on Tuesday. Having always been interested in big databases, not to mention my stint as a committer on HBase, I’ve long been roughly aware that Cassandra exists, but never managed to learn about it very deeply. As such, the Summit provided some really interesting opportunities [...]
Posted in Miscellaneous Leave a comment
Reading files quickly in Java
I came across a really interesting, well-done blog post today about the quickest way to do high-performance file IO in Java. It does a really good job of breaking down the alternatives of how to get bytes into memory, covering both traditional and NIO options in a good amount of detail. It’s a must-read for [...]
Thrift 0.3.0 Released
After seven separate release candidates, Thrift 0.3.0 is finally released! This version includes many, many fixes over Thrift 0.2 in areas of stability, features, and performance. If you’ve been holding off on upgrading, then now is the perfect opportunity.
You can find the distribution here.
Fully async Thrift client in Java
Thrift has had an asynchronous server implementation for Java for quite some time, but users have been asking for a way to have an asynchronous client since the very beginning.
The motivation behind this style of client is usually performance. Imagine you take a bunch of time and make a highly optimized web application that makes [...]
Avoiding Java varargs snafus
Since Java 1.5, Java has allowed you to take advantage of “varargs“, a usability feature that many other languages support. It lets you write really clean code and support some pretty cool use cases.
However, there is at least one possible pitfall of using varargs. Consider the method below:
public boolean filter() {
(do some filtering)
}
filter();
Let’s say you [...]
Parallelized bloom filter creation with Map/Reduce
As we’ve mentioned in the past, bloom filters are an important part of our workflow. They allow us to quickly skip a large portion of the records that we’re not interested in, thinning out the amount of data that has to be CoGrouped in our Cascading flows.
Up until recently, we’ve just been creating our bloom [...]
Posted in Miscellaneous 2 Comments
Faster string to UTF-8 encoding in Java
Update: It turns out that after further investigation, the performance improvements didn’t hold up when some uncovered correctness bugs forced some code changes. The patch was rolled back, so we’re stuck with the same old encoding mechanism. Sigh.
I’ve spent a lot of time profiling Thrift serialization and deserialization, and one thing that has always stood [...]
Posted in Thrift 3 Comments
Analyzing some interesting networks for Map/Reduce clusters