-
Recent Posts
Popular Posts
- Rent or Own: Amazon EC2 vs. Colocation Comparison for Hadoop Clusters 27 comment(s) | 10889 view(s)
- Mysql Replication Adapter 26 comment(s) | 6699 view(s)
- Making sure Ruby Daemons die 20 comment(s) | 7400 view(s)
- Matching Impedance: When to use HBase 19 comment(s) | 22418 view(s)
- Goodbye MapReduce, Hello Cascading 17 comment(s) | 9747 view(s)
- Rapleaf Challenge Problem 12 comment(s) | 3824 view(s)
- BloomFilter 11 comment(s) | 5483 view(s)
- Using random numbers in Hadoop MapReduce is dangerous 11 comment(s) | 4057 view(s)
- Ruby and HBase 10 comment(s) | 5292 view(s)
- Cycles of Doom in Batch Processing Workflows 10 comment(s) | 2678 view(s)
Categories
- Anonymouse (2)
- Apache (1)
- bash (1)
- Cascading (6)
- Daemons (1)
- encryption (1)
- Extensions (2)
- Google (1)
- Grub (1)
- Hadoop (22)
- HBase (6)
- HDFS (4)
- Kickstart (1)
- MapReduce (9)
- mcrypt (1)
- Miscellaneous (26)
- Mongrel (2)
- Mysql (2)
- OpenSocial (1)
- Operations (1)
- Ruby (7)
- Security (2)
- Thrift (6)
- Xen (1)
Archives
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- March 2009
- February 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007

Application Deployment at Rapleaf
One challenge we constantly face at Rapleaf is how to remain agile as we grow. In the early days, we would say “I’m going to deploy” to a handful of other engineers in the room. When we needed to move beyond that, we designed a deployment system with 3 key objectives:
Remain agile. Engineers should be able to deploy as frequently as they need to.
Communicate. Other engineers should know when a deploy is about to happen and when it is finished.
Must be robust. Have access controls, prevent multiple deploys and check for common problems.
We were able to create a simple deployment system using Screen, CruiseControl, Capistrano and Openfire (Jabber). Screen was an obvious choice when we considered all of our requirements. Before screen, we all deployed from our local machines. There were a couple of problems with this: many people could deploy at once and their environments could be different. Using a shared screen on a single machine solved both of these problems. Screen also has access controls, multiple windows and scrollback history. We have a window for each application and our screenrc file looks something like this:
# Set status line
caption always "%{B}%t deploy %{R}REMEMBER TO 'svn up' %{B}%? @%u%?%? %{d}[%h]%?%=%D %M %d %C %A"
# command usually invoked by C-a " would also be available as C-a space
bind ' ' windowlist -b
# Create a screen window per application
screen -ln -t "app1" -h 5000 1
screen -ln -t "app2" -h 5000 2
# Allow multiple users to connect
multiuser on
# Engineers that can deploy
acladd engineer1,engineer2
CruiseControl is a great continuous integration tool. We always specify a revision when we deploy, so we can easily check cruise to make sure that the revision has passed the test suite.
Capistrano is the basis for all of our application deploys, including java apps. We identified several functions we wanted to perform in every deploy and put those into a common library. Those common functions include: notifying a jabber conference room on start/finish, running a sanity check on revision being deployed and dealing with our load balancer. Capistrano has many other nice features that we use, but I won’t go into that here.
We’ve been running this system for a while and really like it. We were able to meet all of our objectives with a simple design and some great open-source tools.