While we’re on the timeout tip, I’d like to talk about a fail-safe timeout for mongrel rails. We’ve talked about making sure that net/http and memcache-client behave, but what about other slow actions? Since mongrel only processes one rails request at a time, other requests can start to pile up. In rare cases, those other requests would wait forever. It would be nice to make sure that each rails request takes no longer than a set amount of time.
Since there is no timeout around a rails request in mongrel, we decided to put one in. Mongrel rails uses a mutex around all rails requests (taken from the process method in mongrel-1.0.1/lib/mongrel/rails.rb line 77):
@active_request_path = request.params["PATH_INFO"]
Dispatcher.dispatch(cgi, ActionController::CgiRequest::DEFAULT...
@active_request_path = nil
}
Our solution was to redefine the @guard instance variable to make sure the rails request timed out after the lock was obtained. We first constructed a simple TimeoutMutex class:
def initialize(timeout = 30)
super()
@timeout = timeout
end
def synchronize
lock
begin
Timeout::timeout(@timeout) {yield}
ensure
unlock
end
end
end
We then popped open the RailsHandler class in mongrel and redefined @guard:
cattr_accessor :rails_timeout
alias :original_initialize :initialize
@@rails_timeout = 30
def initialize(dir, mime_map = {})
original_initialize(dir, mime_map = {})
@guard = TimeoutMutex.new(@@rails_timeout)
end
end
The @guard mutex is also used in the reload! method, so that will also get the same timeout. Given the fact that we don’t use this method and mongrel startup reads: “HUP => reload (without restart). It might not work well.”, we decided not to worry about it.
If you want to have a different timeout value, just set Mongrel::Rails::RailsHandler.rails_timeout. Since timeout throws an exception into the thread it times out, you could end up with an exception at any point in your rails code. Most of the time, this would tell you which methods need attention. You can also catch the Timeout::Error exception in you application controller’s rescue_action_in_public method.
2 Comments
Doesn’t mongrel have a default 60 second timeout for all actions?
I see the TimeoutError raised in the reap_dead_workers function (mongrel.rb:665) in our logs occasionally when there’s a backup.
No. You can test this by creating an action that sleeps forever and see what happens. I stopped watching after hanging the browser for 30 minutes.
Mongrel calls reap_dead_workers for 3 different reasons: too many open files (lines 651 & 746), max processors (734) and graceful shutdown (688). Graceful shutdown gives workers 60 seconds to finish, then kills them. Too many open files should only happen if you have a large number of requests or some other process on the machine is eating up file handles. Max processors can be set when calling HttpServer.new, but defaults to 1 Billion (2**30 – 1).
Keep in mind this is all on mongrel-1.0.1. I haven’t looked through the code for later versions.