We monitor our API request times very carefully, and while the average is ~5ms, there have always been a small number of outliers that take hundreds of milliseconds. Recently, though, the number of outliers has grown, and the timing info in our logs gave us reason to believe it had to do with garbage collection. The main sign was that calls which should take virtually no time (like memcached) were sometimes taking hundreds of milliseconds even when load was normal.

To confirm our theory, we turned on verbose GC logs with the following flags:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:gc-`date +%s`.log

We used date to generate a dynamic name for the log so the previous one wouldn’t get overwritten when restarting the server. And to be clear, we were already using -XX:+UseConcMarkSweepGC. If you are concerned about pauses and aren’t using this collector, you should start using it.

As soon as we looked at the resulting logs it was clear GC was a problem. The garbage collector was running pretty much all the time with minor collections happening at least once a second, if not more frequently, and major collections happening about every 3 minutes. This, all in an app that creates no long lived-objects after startup, so there shouldn’t really be the need for major collections (which are what pause the entire VM).

To make interpreting the GC logs easier we used GCViewer. It was great to be able to visualize our memory usage and compare tweaks we made with our baseline. When we viewed the data as a graph, we noticed a really interesting pattern: after minor collections (the little drops), total memory usage would go up, but after major collections (the big drops), total memory usage would return to the same low level as before. Even though we weren’t leaking memory overall, on a short time scale, it looked like we were.

Blue line is memory usage. Black lines are major collections.

So the key question is: since all of our objects are short lived, why were the minor collections leaving memory around? This type of situation is what a generational garbage collector is ideally suited for, right? It doesn’t take a deep understanding of Java GC to realize the problem was that our young generation was too small, which caused even some short lived objects to be promoted to the tenured generation.

To understand why this was happening requires a bit more understanding, though. The young generation is split up into three parts: Eden space, and two Survivor spaces. New objects are allocated in Eden, and when minor collections occur the objects that are still live are copied to one of the Survivor spaces. At any given time one of the Survivor spaces is empty and is the destination of the next minor collection. The other Survivor space holds the live objects from the previous minor collection, and gets collected during the next minor collection. Since it’s assumed most objects die young, the Survivor spaces are relatively small.

So what happens when the live objects from a minor collection can’t fit into a survivor space? The leftover objects are moved to the tenured generation which is only cleaned by major collections. In our case the problem was that the minor collections were so frequent that a decent percentage of objects were still live at each run—enough that they couldn’t all fit in the survivor space, causing even relatively short lived objects to be tenured. We realized we had to get minor collections to happen less often and have more space for surviving objects so they aren’t prematurely tenured.

We played around with some GC parameters to find the right balance for our application and arrived at this:

-XX:NewSize=768m -Xms4g

Which resulted in a much better GC graph:

Hooray! No more trending up; no more major collections! Problem solved, right?

Well mostly. But there are still special circumstances in which memory usage is not as friendly-looking as in the graph above. At the core of our problems with garbage collection is that we’re allocating a lot of memory in a very short amount of time. Addressing that issue will be the topic of a follow-up post. (Spoiler: We significantly decrease our memory usage.)

While we are completely happy with the CompactProtocol for permanent data storage and RPC, there is one place where it’s not a perfect fit. Since all our objects are Thrift structs, we pass those same structs around during our Cascading flows. We’ve built a connector (extremely similar to this one) that allows these Thrift structs to be serialized in the CompactProtocol when Cascading serializes Tuples between mappers and reducers, and this all works great. However, as part of our never-ending performance efforts, one of the things we are always striving to do is to decrease the amount of data that needs to be shipped between mappers and reducers. You’re probably thinking, isn’t the CompactProtocol… compact? The answer is yes. It’s the smallest representation we could come up with for a general-purpose Thrift protocol.

But wait! Recall that Thrift as a framework is designed to allow smooth transitions for users using different versions of their schema (like an old client contacting an updated server). Thrift does this by making sure that the data that gets serialized is sufficient to fully describe itself – serialized messages contain markers about field IDs and field data types. Using these markers, it’s possible for the Thrift library to skip over unrecognized fields. But this ability comes at the cost of a few bytes of overhead here and there, and these bytes can really add up when you have 200 billion records to move around.

Look back at our imperfect use case: intermediate serialization during a Cascading flow. When would the Thrift schema ever change during a single job? The answer is that it wouldn’t. This means that the extra bytes that the CompactProtocol spends on the feature to support changing schemas is 100% waste. It would be great if we didn’t have to pay the cost of a feature we don’t use.

This is exactly what the TupleProtocol does. Instead of being a general-purpose protocol, TupleProtocol is a purpose-built protocol designed specifically for the case when you know beyond the shadow of a doubt that the schema for a record cannot change. With this precondition, it can avoid writing out the type markers and field IDs and just uses the metadata implicit in the code itself to guide deserialization. For instance, it knows that what’s coming up next is going to be a string field called “foo” because it can assume the writer of the data wrote it exactly to that spec. By dumping this extra overhead, users of the TupleProtocol can see a significant decrease in the size of serialized objects. One sample job we tested saw about a 5% size decrease. The actual savings you see will vary a lot depending on what kind of data you have in your structs, but in general, the more non-string fields you have, the more benefit you’ll see. For us, just based on the 5% figure, it amounts to tens of gigabytes less shuffle in our biggest jobs.

The TupleProtocol is currently committed to Thrift TRUNK and will be released as part of Thrift 0.8. If you’d like to try it out now, we’d love to hear your feedback!

As my internship draws to a close, I’d like to share some of my experiences to give you a better sense of what life is like here, just as some former Rapleaf interns have done.

Near the end of my second week, I was shocked when my mentor, an experienced member of the engineering squad charged with helping me adapt and stay on track, asked me what I wanted to do here.  I held my tongue for a bit, because all I could think at first was: “You mean I get to choose?”  He explained several of the different projects he thought would be a good fit between Rapleaf’s vision and my skills and interests: name and postal address normalization, improving the quality of our income data, and improving the quality of our internal-use-only people-search engine.  (For a sense of what name and postal address normalization accomplishes, try out our name normalizer, and imagine adding the same capability for mailing addresses.)  Income data quality improvement seemed the most appealing, so I started on that.

Reflecting on this summer, I appreciate more fully that I am an extremely lucky person.  Rapleaf engineers consistently tackle sophisticated problems with an inspiring attitude.  We’re invested in what we do, holding strong opinions about how it should be done, but we respect each other enough to resolve conflicts peacefully and productively.  Such patience is uncommon, yet Rapleafers learn to cultivate and uphold solid relationships with one another.  The maturity and brilliance of our team have enabled me to learn at a tremendous rate.  I receive lots of helpful feedback about the code I write and broader team-wide decisions I propose.  Because Rapleaf’s management approach is so open to input, my propositions are always taken seriously, even if they’re bird-brained. They have given me the confidence to keep trying as well as insights into the tradeoffs our veterans continually have to evaluate.  I never cease to be impressed by how seriously my colleagues take any ethical considerations involved in such debates.

I used to say “they” when referring to Rapleaf, but from now on it will be “we.”  I am proud to be one of us.

This will be the last puzzle from me, because my internship here is drawing to a close. Thanks to all of you for working the puzzles and brightening my day. If you’re craving more problems, try out our newly updated challenge questions. Please send any solutions you come up with to challenge@rapleaf.com and, if you have one handy, include a copy of your résumé.

UPDATE: Try the problem with twelve coins instead of eight, although eight coins could still be a good warm-up.

UPDATE: Feel free to share solutions to the eight coin version too!

The fund is being launched in partnership with North Bridge Venture Partners, Rembrandt Ventures, Founders Fund, Highland Capital Partners, SoftTech VC, MIT Entrepreneurship Center, Harvard College Entrepreneurship Forum, Stanford BASES and many more.

Applications are accepted on a rolling basis on the Rapfund site. We’re looking for awesome start-ups based on Rapleaf data that will take personalization to the next level.

To learn more, visit www.rapleaf.com/rapfund

We ran into some minor difficulties at first. We had some old migrations that used the old naming scheme (they were numbered consecutively instead of using timestamps) and were mixed in with our newer migrations. Since we wanted our new tools to be able to assume that we were using the timestamp naming convention instead of the old consecutive numbering convention, we went back and renamed the old migrations. This seemed to work fine, and our new tools worked great when developing on our local machines.

About a week went by and everything was fine. Then, all of a sudden, our website went down. We started investigating, and we found that a bunch of important database tables were completely empty. It took a bit of digging around, but eventually we tracked the problem back to a migration that we had renamed a week earlier. Apparently this was the first time that we had tried to run rake db:migrate in production since Sweepleaf, and renaming the migration caused Rails to re-run it (since it now had a different number and looked like a brand new migration, even though it had already been run in the past). To complete the perfect storm, this particular migration happened to be copied from an old schema.rb, which meant that there were :force => true directives after each create_table. This meant that since this migration had already been run in the past, the tables were dropped and recreated.

Once we identified the problem, it was simply a matter of restoring the database from our backups. We ran into a few hiccups when trying to replay the database logs, but ultimately we got everything resolved.

While it seemed like a fiasco at the time, we learned a number of valuable lessons from this incident.

1. Editing old migrations — particularly renaming them — is bad. Just don’t do it.
2. Back up your databases. Seriously.
3. Want to be sure something like this never happens? Take “drop table” permissions away from the user that you use to run migrations. Dropping tables is a cleanup task that can be done (carefully) by administrators.

Ben P. decided to kick off an internal game of code golf , which we’re hoping will be the first of many. The objective is to solve a certain problem with the fewest number of characters.

SPOILER ALERT: solutions are at the bottom of this post.

——-
Problem 1: implement FizzBuzz:

A couple points for clarification: the output needs to be exactly as specified – no extraneous quotation marks.  Here’s an example output. You also can’t rely on the user having an internet connection or anything other than ruby 1.8.7 installed (including bash).
——

Ben gave the team a 4-day deadline. The next day, he announced updates: Alex P. was at 34 characters “although it might break the rules.” Gareth was at 63 characters with a normal ruby program.

The winning code was 57 characters long. Below is a 58-character solution we found that does some interesting things, and the internet record is 50 characters. Can you beat that?

Happy hacking!

—

58- character solution:

1.upto(?d){|x|puts (y=’Fizz’*1[x%3]+’Buzz’*1[x%5])[1]?y:x}

What might Greg’s integers be?