For engineers, by engineers.

Last week was our quarterly Hackleaf, and this time around a handful of us set out to solve a slightly more physical problem than we usually tackle: conference room abuse. We have nine conference rooms in our office these days, and even though we regularly use Google Calendar to schedule meetings, we still struggle with rooms being hijacked.

It’s almost always an accident – two people start chatting, one inevitably says “Let’s grab a room,” and then they just walk around the office until they happen upon an unoccupied room.  Unfortunately, it’s often the case that while the room is currently unoccupied, another meeting is scheduled to start very soon. In a few minutes, the rightful occupants show up and discover that someone is already using their room! If the squatters are just using the whiteboard, it’s easy to boot them out, but if they’re taking a phone call, the people who actually reserved the room have to find a new one. The ability to do ad-hoc meetings is something we want to keep. We just want to help room-grabbers know when it is OK to grab and when it isn’t.

Enter the Roominator, an open-source system of hardware and software that helps facilitate informed ad-hoc reservations.

The hardware consists of two parts: a display unit that’s posted outside of each conference room, and a controller unit that’s stashed in our wiring closet. The display unit shows the current and upcoming reservations and an LED status indicator that can tell you from a distance whether a room is “good to grab”. It also has a pair of buttons – one to make an ad-hoc reservation and one to cancel the current reservation. The controller unit interfaces with all the displays to distribute power and data, both of which run over a single standard Cat5e cable. Both the controller and the displays are Arduino-based.

The software component is a Rails web site that allows for configuration and integrates with Google Calendar. Reservations made via Google Calendar are sync’d with the Roominator, and vice-versa. The controller unit polls the web site for the information it should pass to the displays.

This project was a lot of fun for all involved, and we definitely went outside of our comfort zone with this one. There’s a lot of polishing to do in order to get the UI just right – and we still have to manufacture another seven display units by hand! (No copy/paste in the real world, after all.)

Want your own Roominator setup? All the source code and schematics are on GitHub. If you decide to go down the road of building your own units, please get in touch with us so we can collaborate!

Ever since we got a ping-pong table in the office a few months ago, table tennis has taken off in a pretty big way at Rapleaf. Not too long ago we started up a simple ladder in order to keep track of the One True Pong Champion.

Rather than 'second-worst', I prefer to think of it as 'twenty-first-best'.

As you can see, Sean (“scar”) Carr is our reigning champion, with Piotr (“The Great”) in hot pursuit, and Abhishek (“The Jain of Pain”) having recently unseated Vlad (“The Impaler”) for third.

Though we certainly enjoy playing against one another, we’re also itching to show off our skills to the wider world. Because if there’s one thing better than being awesome, it’s being awesome all up in other people’s business.* Therefore, let it be known:

Rapleaf is hereby open to table tennis challenges from any and all Bay Area startups.

If your company is interested in getting demolished by Rapleaf’s elite ping-pong wrecking crew, send an inquiry to pong@rapleaf.com and we’ll work out the details from there.

*All statements expressed herein are wholly the opinion of the post author and do not reflect in any way the sentiments, values, or beliefs of Rapleaf or its management.

One of the most interesting things that we do at Rapleaf is use our existing data to deduce or infer new data. For example, a person’s name is often highly correlated with a specific gender. After doing lots and lots of regression, we usually end up with a simple HashMap loaded from a file that our code uses to actually make the deduction at runtime. The map is small enough to fit in memory, so access is very fast.

However, memory is a scarce resource, particularly in the Hadoop context. If we don’t manage it carefully, a misbehaving task can lead to our entire cluster swapping or crashing. Ideally, we’d like to get the most bang for our memory buck, even if it costs us a little CPU time, since we can bear a slowdown if it avoids a catastrophic failure.

Let’s back up for a second and think about the kind of data we want to store. For example, look at the “name to gender” mapping. In essence, you have a name string and a gender enum, like this:

The critical thing to notice is that a lot of the names in the map share common prefixes. It’d be great if we could avoid storing that duplicate data in-memory when we’re running.

It turns out that there’s a great data structure for this purpose called a radix tree, which is an optimized form of a trie. This structure compresses common prefixes together so that they don’t have to be stored, and has performance comparable to a hash. For things like English words or names, you can often get a huge amount of savings from using a radix tree.

Surprisingly, we had trouble locating a memory-efficient implementation of a String-keyed radix tree map out on the Internet, so we decided to make one ourselves. You can find the project on GitHub, complete with tests. The basics (get, put) work pretty well, but there aren’t a lot of frills right now, so we’d love to see contributions in that department. On our test data set, we found that the ImmutableStringRadixTreeMap shaved about 44% of the overhead of HashMap! That’s great savings.

Rapleaf started out using Ruby and Rails extensively to build out our systems. We loved the flexibility that it gave us to quickly put together a functional application. Tools like ActiveRecord are huge productivity boosters, saving us the trouble of hand-coding database interaction and letting us focus directly on our application.

However, we evolved to need something more than Ruby. While we continue to use Rails for our website and our internal administration site, we’ve moved to using Java (and Hadoop/Cascading) to power our processing pipeline. Oftentimes, our Java applications need to access the same databases as our Ruby sites.

This seems like a simple problem, but we’ve encountered a couple issues. Firstly, although Java has a number of database abstraction packages, we didn’t find them quite up to the level of ActiveRecord in terms of ease of use or flexibility. Packages like Hibernate are cumbersome and complex compared to ActiveRecord, and that’s led to our developers using them inconsistently or avoiding frameworks altogether.

Framework aside, there’s still an even more significant problem – namely, how do we scalably share our growing schema across two separate languages/frameworks? Between our two main production databases, we have over 220 tables. At that number, manually maintaining more than one version of our data definition for different frameworks is tedious and error-prone. In practice, they won’t be synchronized correctly, leading to confusion at deployment time when problems are finally uncovered.

Our initial solution was to limp along with carefully hand-coded Java models that mirrored their Ruby counterparts. This was a pretty poor solution, as it meant that only the minimum set of models required for the current application would get built, and features were lacking. We finally decided that we’d had enough. What if we could establish a parallel framework in Java that leveraged the work we already do in ActiveRecord?

It turns out that this is easier than it sounds. Over the course of the last month, we’ve build Jack, a set of Ruby scripts and a concordant library of Java classes that together allow you to convert your Ruby ActiveRecord migrations and models into automatically generated Java classes. The general idea is that we parse schema.rb for the table schema and each of the models for their associations and other configuration elements. Once we’ve got it all parsed up, it’s trivial to generate brand-new Java classes.

We’ve decided to open source this project in hopes that others can benefit from and contribute to the project. You can find the code and some documentation on GitHub. We look forward to hearing your thoughts!

Cloudera has posted a new blog post contributed by one of Rapleaf’s software engineers. It’s a high-level overview of all the ways we use Cloudera’s Distribution for Hadoop at Rapleaf. (Here’s a tip: we use it a lot). Go check it out!