While we are completely happy with the CompactProtocol for permanent data storage and RPC, there is one place where it’s not a perfect fit. Since all our objects are Thrift structs, we pass those same structs around during our Cascading flows. We’ve built a connector (extremely similar to this one) that allows these Thrift structs to be serialized in the CompactProtocol when Cascading serializes Tuples between mappers and reducers, and this all works great. However, as part of our never-ending performance efforts, one of the things we are always striving to do is to decrease the amount of data that needs to be shipped between mappers and reducers. You’re probably thinking, isn’t the CompactProtocol… compact? The answer is yes. It’s the smallest representation we could come up with for a general-purpose Thrift protocol.

But wait! Recall that Thrift as a framework is designed to allow smooth transitions for users using different versions of their schema (like an old client contacting an updated server). Thrift does this by making sure that the data that gets serialized is sufficient to fully describe itself – serialized messages contain markers about field IDs and field data types. Using these markers, it’s possible for the Thrift library to skip over unrecognized fields. But this ability comes at the cost of a few bytes of overhead here and there, and these bytes can really add up when you have 200 billion records to move around.

Look back at our imperfect use case: intermediate serialization during a Cascading flow. When would the Thrift schema ever change during a single job? The answer is that it wouldn’t. This means that the extra bytes that the CompactProtocol spends on the feature to support changing schemas is 100% waste. It would be great if we didn’t have to pay the cost of a feature we don’t use.

This is exactly what the TupleProtocol does. Instead of being a general-purpose protocol, TupleProtocol is a purpose-built protocol designed specifically for the case when you know beyond the shadow of a doubt that the schema for a record cannot change. With this precondition, it can avoid writing out the type markers and field IDs and just uses the metadata implicit in the code itself to guide deserialization. For instance, it knows that what’s coming up next is going to be a string field called “foo” because it can assume the writer of the data wrote it exactly to that spec. By dumping this extra overhead, users of the TupleProtocol can see a significant decrease in the size of serialized objects. One sample job we tested saw about a 5% size decrease. The actual savings you see will vary a lot depending on what kind of data you have in your structs, but in general, the more non-string fields you have, the more benefit you’ll see. For us, just based on the 5% figure, it amounts to tens of gigabytes less shuffle in our biggest jobs.

The TupleProtocol is currently committed to Thrift TRUNK and will be released as part of Thrift 0.8. If you’d like to try it out now, we’d love to hear your feedback!

Last week was our quarterly Hackleaf, and this time around a handful of us set out to solve a slightly more physical problem than we usually tackle: conference room abuse. We have nine conference rooms in our office these days, and even though we regularly use Google Calendar to schedule meetings, we still struggle with rooms being hijacked.

It’s almost always an accident – two people start chatting, one inevitably says “Let’s grab a room,” and then they just walk around the office until they happen upon an unoccupied room.  Unfortunately, it’s often the case that while the room is currently unoccupied, another meeting is scheduled to start very soon. In a few minutes, the rightful occupants show up and discover that someone is already using their room! If the squatters are just using the whiteboard, it’s easy to boot them out, but if they’re taking a phone call, the people who actually reserved the room have to find a new one. The ability to do ad-hoc meetings is something we want to keep. We just want to help room-grabbers know when it is OK to grab and when it isn’t.

Enter the Roominator, an open-source system of hardware and software that helps facilitate informed ad-hoc reservations.

The hardware consists of two parts: a display unit that’s posted outside of each conference room, and a controller unit that’s stashed in our wiring closet. The display unit shows the current and upcoming reservations and an LED status indicator that can tell you from a distance whether a room is “good to grab”. It also has a pair of buttons – one to make an ad-hoc reservation and one to cancel the current reservation. The controller unit interfaces with all the displays to distribute power and data, both of which run over a single standard Cat5e cable. Both the controller and the displays are Arduino-based.

The software component is a Rails web site that allows for configuration and integrates with Google Calendar. Reservations made via Google Calendar are sync’d with the Roominator, and vice-versa. The controller unit polls the web site for the information it should pass to the displays.

This project was a lot of fun for all involved, and we definitely went outside of our comfort zone with this one. There’s a lot of polishing to do in order to get the UI just right – and we still have to manufacture another seven display units by hand! (No copy/paste in the real world, after all.)

Want your own Roominator setup? All the source code and schematics are on GitHub. If you decide to go down the road of building your own units, please get in touch with us so we can collaborate!

My gut told me that there should be a performance difference between a built-in language construct and a library, but experience has taught me not to guess when it comes to performance. A quick Google search led to a lot of posts deflecting the main question with cries of “premature optimization!”, and thus did not help me at all.

I ended up writing a micro-benchmark program (code on GitHub) that exercises Lock and synchronized() equally to measure total throughput. I measured two different situations, single-threaded and two-threaded, because of previous reading that indicated the JVM was pretty good at optimizing the uncontended case.

The results were clear and fairly unsurprising: synchronized() is substantially faster. In the single-threaded test, synchronized() was about 7.5x faster on average than Lock.lock(). In the two-threaded test, synchronized() was still the clear winner, about 2x faster on average.

Bottom line: if you can use synchronized() instead of Lock, then you definitely should use synchronized().

However, memory is a scarce resource, particularly in the Hadoop context. If we don’t manage it carefully, a misbehaving task can lead to our entire cluster swapping or crashing. Ideally, we’d like to get the most bang for our memory buck, even if it costs us a little CPU time, since we can bear a slowdown if it avoids a catastrophic failure.

Let’s back up for a second and think about the kind of data we want to store. For example, look at the “name to gender” mapping. In essence, you have a name string and a gender enum, like this:

The critical thing to notice is that a lot of the names in the map share common prefixes. It’d be great if we could avoid storing that duplicate data in-memory when we’re running.

It turns out that there’s a great data structure for this purpose called a radix tree, which is an optimized form of a trie. This structure compresses common prefixes together so that they don’t have to be stored, and has performance comparable to a hash. For things like English words or names, you can often get a huge amount of savings from using a radix tree.

Surprisingly, we had trouble locating a memory-efficient implementation of a String-keyed radix tree map out on the Internet, so we decided to make one ourselves. You can find the project on GitHub, complete with tests. The basics (get, put) work pretty well, but there aren’t a lot of frills right now, so we’d love to see contributions in that department. On our test data set, we found that the ImmutableStringRadixTreeMap shaved about 44% of the overhead of HashMap! That’s great savings.

However, we evolved to need something more than Ruby. While we continue to use Rails for our website and our internal administration site, we’ve moved to using Java (and Hadoop/Cascading) to power our processing pipeline. Oftentimes, our Java applications need to access the same databases as our Ruby sites.

This seems like a simple problem, but we’ve encountered a couple issues. Firstly, although Java has a number of database abstraction packages, we didn’t find them quite up to the level of ActiveRecord in terms of ease of use or flexibility. Packages like Hibernate are cumbersome and complex compared to ActiveRecord, and that’s led to our developers using them inconsistently or avoiding frameworks altogether.

Framework aside, there’s still an even more significant problem – namely, how do we scalably share our growing schema across two separate languages/frameworks? Between our two main production databases, we have over 220 tables. At that number, manually maintaining more than one version of our data definition for different frameworks is tedious and error-prone. In practice, they won’t be synchronized correctly, leading to confusion at deployment time when problems are finally uncovered.

Our initial solution was to limp along with carefully hand-coded Java models that mirrored their Ruby counterparts. This was a pretty poor solution, as it meant that only the minimum set of models required for the current application would get built, and features were lacking. We finally decided that we’d had enough. What if we could establish a parallel framework in Java that leveraged the work we already do in ActiveRecord?

It turns out that this is easier than it sounds. Over the course of the last month, we’ve build Jack, a set of Ruby scripts and a concordant library of Java classes that together allow you to convert your Ruby ActiveRecord migrations and models into automatically generated Java classes. The general idea is that we parse schema.rb for the table schema and each of the models for their associations and other configuration elements. Once we’ve got it all parsed up, it’s trivial to generate brand-new Java classes.

We’ve decided to open source this project in hopes that others can benefit from and contribute to the project. You can find the code and some documentation on GitHub. We look forward to hearing your thoughts!

Our use case is very particular: we have tons of data that needs to get processed, producing a lot of data points for individual people, which then need to be made randomly accessible so they can be served through our API. You can think of it as the “process and publish” pattern.

For the processing component, Hadoop and Cascading were an obvious choice. However, making our results randomly accessible for the API was more challenging. We couldn’t find an existing solution that was fast, scalable, and perhaps most importantly, wouldn’t degrade performance during updates. Our API needs to have lightning-fast responses so that our customers can use it in realtime to personalize their users’ experiences, and it’s just not acceptable for us to have periods where reads contend with writes while we’re updating.

We boiled this all down to the following key requirements:

1. Random reads need to be fast – reliably on the order of a few milliseconds.
2. Datastores need to scale to terabytes, with keys and values on the order of kilobytes.
3. We need to be able to push out hundreds of millions of updates a day, but they don’t have to happen in realtime. Most will come from our Hadoop cluster.
4. Read performance should not suffer while updates are in progress.

Additionally, we identified a few non-requirements:

1. During the update process, it doesn’t matter if there is more than one version of our datastores available. Our application is tolerant of this inconsistency.
2. We have no need for random writes.

The system we came up with is tailored to meet these needs. It consists of a fast, read-only data server backed by a custom-designed batch-updatable file format, a set of tools for writing these files from Hadoop, and a special daemon process that manages the deploy of data from the Hadoop cluster to the actual server machines. Clients of Hank are aware of ongoing updates and avoid connecting to servers that are busy. When the time comes to push out a new version of our data, the data deployer allows only a fraction of the data servers to perform an update at a time, making sure that sufficient data serving capacity remains online.

There’s a more detailed look at the architecture and infrastructure of the project, and you can find the code on GitHub, which is shared under the Apache Software License. This codebase is still a work in progress – our older, internal version was in need of a serious refactor – but most of the necessary pieces are there, and we’re going to finish the development in the open. We’d love to hear your thoughts on the project and would doubly love to get your contributions, whatever form they might take.

If most or all of the bits were usually set, this would be an unavoidable issue, but in our case, the set bits were very rare – usually no more than 20 or 30. This leads to a large amount of wasted space. The traditional approach to sparse sets like these is to just store the position number of the set variables directly in a collection. To allow for the full 2500 position numbers, we needed a short int, meaning that with this approach, the memory size of the collection is 2 bytes times the number of elements. A sparse set of 30 elements will take a lot less memory than the equivalent bitset (60 bytes versus 313), but in our application, it’s still too much.

What if we could combine the tight-packing benefits of a bitset with the low impact of the sparse set? It turns out that we can. In our application, some of the variables are set pretty frequently (1/4 records have it set) and some are set very rarely (1/10000), with a gradient of various frequencies between. We can exploit this gradient to make really efficient storage decisions. Here’s the graph of frequencies over our set:

It comes down to this: in a bitset, each possible variable in your set costs you one bit, whether or not it’s set. In a sparse set, with our example of 2500 variables, each variable costs 16 bits, but only when it’s set. Comparing these two costs gives a clear tradeoff point. When a given variable is set on at least one of every 16 records, you should store it in a bitset; when it is set on one out of every 17 or more records, then you should store it in the sparse set.

To build this type of set in practice, you just need to get your variables into a list ordered by frequency of occurrence and then apply the cutoff. Everything above should be managed through a bitset, and everything below should be managed by a collection. We found it handy to wrap both of these sets up in a single class so that the user can be indifferent to where the data is stored.

Before Thrift 0.4, no matter how much you might want to, there was no way to avoid doing an extra byte[] copy for each binary field that you deserialized, despite the fact that virtually all deserialization happens directly from an in-memory byte[] buffer. Thrift 0.4 changed that by switching the underlying type of binary fields from byte[] to the Java NIO construct ByteBuffer; Thrift 0.5 elaborated on this theme by making it easier to get the byte[]s that everyone expected while still offering access to the ByteBuffer for more advanced operations.

So how do you actually use this feature to speed up your servers? Let’s take a look at a pair of examples. In both examples, we’ll use the following Thrift file as our base:


struct A {
1: required binary foo;
}

service SomeService {
A read();
void logFoo(1: A a);
}


The logFoo method

Let’s pretend that your objective is to log the contents of the foo field to some stream. Here’s how you might do it naively:


private DataOutputStream out;

public void logFoo(A a) throws TException {
byte[] value = a.getFoo();
out.writeInt(value.length);
out.write(value);
}


Seems simple enough. So what’s wrong here? The problem is that calling getFoo() causes a byte[] copy. It’s hidden from you by the method, but it’s happening nonetheless. The copy you create is only used for an instant, becoming garbage after you pass it to write(), and then the entire A object becomes garbage.

Here’s the right way to do it:


private DataOutputStream out;

public void logFoo(A a) throws TException {
ByteBuffer value = a.bufferForFoo();
out.writeInt(value.remaining());
out.write(value.array(), value.arrayOffset() + value.position(), value.remaining());
}


There’s a lot here, so let’s break it down. First, notice that we called “bufferForFoo” instead of “getFoo”. This returns a ByteBuffer instead of a byte[]. Then, we use the remaining() method to get the number of bytes in the buffer that belong to this value. Finally, we go to the write() call, but this time using the “array, offset, length” version of write. This allows us to reference a subarray directly from the array that backs value without any intermediate copying. There’s some trickiness that goes into understanding why the first element in the backing byte array is arrayOffset() + position(), but for right now, trust me that it’s the case.

It’s a small difference with a bit more code, but depending on the size of foo, you could see a substantial boost in performance.

The read method

Now let’s look at things from the other side of the equation. Let’s say that the objective of the read() method is to read the bytes of foo from an input stream and return them wrapped in an instance of A. Here’s what the naive approach might look like:


public A read() throws TException {
// assume that "in" is a DataInputStream
int fooLength = in.readInt();
byte[] value = new byte[fooLength];
in.readFully(value);
A result = new A();
result.setFoo(value);
return result;
}


The problem with this method is that every call leads to a new short-lived instance of A and a brand new perfect-sized byte[]. Both of these will become garbage very soon, and allocating the new byte[] every time is a drag on your CPU.

Let’s focus on how we can reuse the byte[] for now and think about the A instance some other time. There are many possible strategies for caching your buffers, but here’s a simple one:


private final ThreadLocal bufferCache;

public A read() throws TException {
int fooLength = in.readInt();
byte[] value = bufferCache.get();
if (value.length < fooLength) {
value = new byte[fooLength];
bufferCache.set(value);
}
in.readFully(value, 0, fooLength);
A result = new A();
result.setFoo(ByteBuffer.wrap(value, 0, fooLength);
return result;
}


There’s a good bit more to this version. First, note that we’re using Java’s ThreadLocal capability to support us keeping a single byte[] per active thread. This makes sure that each thread servicing a client won’t interfere with any other, and there’s no contention (synchronization) for the thread-local buffer. Next, after we figure out how much we need to read, we make a point of checking if we have enough buffer space to complete the read. If not, we replace our buffer with a new, bigger one. Then we complete the read into the buffer – this time specifying the length we want to read, rather than letting the length of the buffer imply the size of the read. This ensures that on subsequent reads, when fooLength is less than value.length, we don’t try to read more than we wanted to. Finally, instead of passing the entire buffer into the foo field, we pass in a ByteBuffer that wraps just the portion that contains the value we read this time.

By using this technique, we’ve avoided one copy per call plus an unknown number of byte[] allocations – if your record sizes vary a lot, then it will take some time before the buffer has to expand to accommodate the biggest one, but after that, you won’t need any more allocations. If your records are fixed size, then you should reach that point immediately.

Just as a brief refresher, T represents the max throughput that the shuffle operation will be able to demand during its peak, and we compute it by determining which component of the network will saturate first.

The “triangle”
When Rapleaf had only 120 machines in three racks, our network looked like this:

The Dell switches we’d purchased have a configuration that allows them to be “stacked” together via a proprietary 48Gbps link. The interesting thing about this configuration is that communications from any one rack switch to another rack switch only have to cross one link and two of the three total switches. If we compute the T numbers for this architecture, we get:

This architecture provides a whopping 331Gbps T! That’s actually more than all the machines connected to the network can actually even use – with all 144 ports in use, we could only produce 288Gbps of traffic. It’s really interesting that this network architecture is so speedy, particularly because it’s so cheap to put together – in the neighborhood of $6000 for the entire network. This is a great starter network. You can build it up organically from one to three switches and never have to worry about your network being a bottleneck.

It has some downsides, though. Using both the provided stacking modules prevents you from taking advantage of any of the SFP ports, so you don’t have any high-speed options for connecting to an external non-cluster network. This means that getting data in and out of the cluster can be a challenge, and that there really isn’t any way to expand this architecture with more computing power. In practice, you’ll have to sacrifice 4-8 of the ports on each switch to be bonded together into an uplink to another network. Also, the stacking cables only reach a max of 10 feet, so your racks will have to be physically close to each other.

Naive expansion of the “triangle”
When we were planning to expand from three to four racks, the first thing we considered was connecting a fourth switch to our existing triangle network. The best uplink we could have managed was an 8Gbps bonded Ethernet link like so:

You might already see where this is going. Take a look at the T numbers:

It turns out that the weakest link in the entire network, the new 8Gbps link, has to carry a really large proportion of T, and saturates at a measly 42G! It’s a massive reduction over the configuration described above. If you were to implement this network, you might find yourself in the unenviable position of having your jobs execute more slowly despite having added more machines.

I don’t think that anyone should ever build this network. Even if 42G is enough for your application, this network just won’t grow with you.

Daisy chains
A network topology you might find tempting when reading about various “stacking” options is a daisy chain. Per Dell’s product sheet, you can stack up to 12 of their switches together via the 48Gbps links:

With the T values computed:

It should come as no surprise that the network link in the dead center of the chain will be where saturation occurs first – about 1/4 of all traffic is going to have to cross this link in each direction, leading to a saturation point of 192Gbps. An interesting observation, though, is that no matter the number of racks in the chain, the middle link will always have the same saturation point, since it will always see 1/4 of the traffic.

Since it has such a high saturation point, this topology is somewhat attractive. However, it can be impractical to implement. Since the stacking cables are so short, you are required to locate all your cabinets right next to one another – or to keep all of the switches in one rack and run 480 Ethernet cables to the individual racks! Neither of these approaches are conducive to incremental growth and easy management in a shared facility. Also, once you expand to the max of 12 switches, you are left with no incremental growth path. Finally, you are still subject to all the other limitations of the stacking configuration as described above.

This style of network only really seems to be merited when you know you won’t grow beyond a certain size and you have great control over your racks’ physical positioning.

“Star” networks
The star network topology is one of the most common, and it benefits from being easy to assemble and well understood. Consider this sample from my original post on the analysis technique, pre-annotated with T values:

It consists of four rack switches and one central switch, all the aforementioned Dell 6248s, and connected by 8Gbps links. It saturates at a relatively low 42Gbps. Unlike the triangle network, all the traffic headed to another rack must pass through a single link to the backbone, and unlike the daisy chain network, the links aren’t strong enough to make up for that.

However, this topology still has room for more racks. Let’s see what happens if we plugged in another two racks’ worth of machines like so:

Note that while the uplinks still saturate first, the T that it takes to saturate them has gone up! This is because the relative proportion of the total traffic that goes over each link has actually reduced. The thing to take away is that some network topologies, like this one, keep scaling as you add new nodes. It might make sense to leave yourself room to grow, rather than buying the minimum that fits your needs.

Rapleaf’s current network has taken these lessons to heart. Here’s what we’re using today:

We switched from Dell to Juniper for our hardware, using this switch as our backbone and one of these at the top of each rack. The links between the rack switches and the backbone are made over a single 10G Ethernet connection. We also reduced the size of a rack from 40 machines to 20, since we’re power- and cooling-limited to 20 nodes per cabinet anyway. If you run the T numbers, you end up with:

The links still saturate first at a respectable 92Gbps. This performance seems to be suitable for our application, but the really great thing about this topology is its flexibility. We can expand from our current 160 machines all the way up to 480 machines without any of the switches ever becoming overwhelmed. Or, if we discover that our applications are especially bandwidth-hungry, we can bond an extra 10G Ethernet from each rack switch to the backbone, doubling our T for the price of new cables.

When does Hadoop put the most stress on the network?
The two phases of a Map/Reduce job that are candidates for high network stress are shuffling and reduce output. During the shuffle phase, each of your reducers will contact every other machine in the cluster to collect intermediate files. During the reduce output phase, the final results of the whole job will be written to HDFS, usually with three replicas.

Intuitively, since the data is written out three times, it might seem like the reduce output stage is the most intense period of network traffic; however, it turns out that the shuffle has the most potential to max out your network due to the fact that each node contacts each other node, rather than exactly two other nodes. Note, however, that the reduce output phase might take longer than the shuffle – it just won’t stress the network as much.

Just what do you mean, “stress the network”?
When I speak of “stressing the network”, I’m talking about throughput. More specifically, I am talking about the average aggregate peak throughput. There’s a lot in that term, so let’s break it down. “Throughput” refers to the rate at which we can transfer data, usually measured in Gbps. “Aggregate” refers to the fact that we’re summing up all the throughput across the whole cluster. “Average” refers to the fact that while some components may see higher or lower values, we want the average. Finally, “peak” refers to the fact that we’re interested in the throughput at the peak of stress, rather than at some other time.

Put another way, the average aggregate peak throughput is the aggregate throughput at which some component in the network saturates, that is when it is carrying its maximum throughput capacity. For a link like an Ethernet cable, the max capacity is determined by the rating of the cable and the ports it’s plugged into. For instance, Gigabtit Ethernet is rated for 1Gbps of throughput. For a switch, the rating of the backplane, also specified in Gbps, determines its capacity. Once one component in the network saturates, even if there are other unsaturated components, the job as a whole won’t be able to go any faster.

The model
The objective of our model is to figure out exactly what average aggregate peak throughput (henceforth abbreviated T) a given network topology can bear before some component saturates. To do this, we’ll figure out what proportion of T each link and switch in the network has to carry, then solve some simple equations to determine what value of T causes each component to saturate. The one with the lowest value of T is the one that we care about.

First, let’s quickly look at what happens during the shuffle phase. Let’s assume that we’re operating on a common star-topology network, with four individual top-of-rack switches (labeled A through D) connected via a central backbone switch:

Each of the racks in this cluster contains the same number of nodes, so when the map phase is over, each rack has exactly 1/4 of the total intermediate data, represented in the diagram by the small boxes behind each cloud of nodes. On an individual rack, say Rack A, 1/4 of the data will be transferred to another host in the same rack, with the remaining 3/4 to be split evenly amongst racks B, C, and D. Likewise, each of the other racks will be sending 1/4 of their data to Rack A.

Since this network is symmetrical, each rack behaves identically. And since each rack has 1/4 of the total data, you can multiply things out to determine the following: 1/16 of the total data is going to stay in place in each rack, 3/16 will be transferred in, and 3/16 will be transferred out.

If you sum up all the numbers, you can annotate the diagram like so:

There are a few things worth noting here. Firstly, the links between switches have two separate numbers, one for each direction. This is because virtually all network connections are full-duplex, meaning they can carry their rated capacity in both directions. If you happen to have a half-duplex connection, then you must sum both of the numbers together to get the true proportion. Second, the backbone switch carries more of the throughput than the top-of-rack switches. This is because the backbone has to carry all of the traffic that leaves each of the rack switches. Also, it’s important to avoid double-counting the traffic that passes through the backbone. Think of a switch as a toll booth that can take traffic in either direction – you only need to count traffic that comes in and out once. Thus, the 3/4 proportion in the backbone switch is computed by summing up only the traffic that is coming in.

Now that we have computed what proportion of T passes through each component, we can solve for the value of T that causes the component to saturate. To do that, we’ll need concrete numbers for the capacities of our links and switches. Let’s assume that the links between the rack switches and the backbone are 8Gbps, and that all the switches are inexpensive 48-port Dell switches that have a backplane capacity of 184Gbps. The connections to individual machines are made via 1Gbps Ethernet. If a switch can handle up to 184Gbps, and it must carry 7/16 of T, then for what T does the switch carry 184Gbps? You can produce an equation from this question: 7/16 T = 184Gbps, which you can solve easily: T = 184 * 16 / 7 = 420Gbps. Here’s what the diagram looks like with this step applied to all switches and links:

I’ve highlighted the inter-switch links in this version, because as you can see, they become saturated when T is only 42Gbps, as opposed to the switches, which saturate at between 250Gbps and 420Gbps. This means that at its most intense, the shuffle phase will average about 42Gbps of throughput across the entire cluster. This architecture could support up to 160 machines; if they are sharing fairly, this means that each machine would have about 42Gbps / 160 = 250Mbps of throughput available for their transfers.