Using Capistrano To Launch Long Running Background Tasks

Capistrano is my deployment tool of choice. It is powerful, allowing you to write deployment recipes in Ruby and unobtrusive. You don't have to install any extra software on the target boxes. I think this is a big advantage over other heavier deployment management systems like Puppet or Chef. Isn't having to install server infrastructure to manage your server infrastructure a chicken and egg problem?

Anyway, I often find myself writing tasks that will kick off a long running background job that does something useful on a deployment node. This is ridiculously simple in Capistrano, right?

task :bg_job do
  run "nohup ruby #{current_release}/lib/background_job.rb > #{current_release}/log/background_job.log &"
end

Then run it.

  cap production bg_job HOSTS=server01

Oops, that cap command just sits there and never returns. WTF? What is happening is Capistrano is dutifully waiting for output from the standard output and error streams which are still open on the server side. They will remain open until the background task completes our you hit CTRL-C out of boredom.

The solution? Just make sure that you don't leave any open streams hanging around and make sure your background job isn't waiting for you to type something either. So we rewrite the task like so ...

task :bg_job do
  run "nohup ruby #{current_release}/lib/background_job.rb > #{current_release}/log/background_job.log < /dev/null 2>&1 &"
end

And Voila! The cap invocation returns immediately and if you log into server01, your job will be running merrily along to completion. The keys are making sure you re-route standard error to standard output (the 2>&1 part) and you detach the background process from waiting for input from the calling shell (the < /dev/null) part.