Author Archives: Bryan Duxbury

Recently, while working on the Collector, I noticed that we had an issue with graceful shutdown of our servers. The Collector uses a JVM shutdown hook to catch the SIGTERM and take some cleanup actions before allowing the exit to go on. However, every time I would try to gracefully shut down a server, I’d [...]

For some time now, Rapleaf has been hard at work converting a critical portion of our infrastructure from a MySQL-based system to a Hadoop-based one. We see it as a much more obvious path to linear scalability of our processing pipeline. Since scalability is our goal, a technology that has obviously found its way into [...]

Here are links to the videos of all the presentations we had last week. The Collector – Multi-Writer Appends into HDFS by Bryan Duxbury, Software Engineer at Rapleaf http://www.vimeo.com/2084824 Katta – Distributed Lucene Index in Production by Stefan Groshupf, Founder/CTO at 101tec Inc. and Co-Founder at Scale Unlimited Inc. http://www.vimeo.com/2085140 Debugging and Tuning Map-Reduce Applications [...]

Last night at the Rapleaf-hosted Hadoop meetup, I talked about a project we’ve created here at Rapleaf called the Collector. Basically, Rapleaf is starting to Hadoopify our workflow, and like a lot of people out there, we’ve found the need to manage many processes writing to HDFS so that our data can be processed by [...]