Add support for sequential job definitions

[njakobsen]

Jobs now support a sequence option that will cause jobs with the same sequence value to run sequentially when scheduled for the same time. sequential: true can be passed to the every block to automatically set a common sequence value for each contained job.

every 3.hours, sequential: true do
  # Jobs in this block that don't set their own sequence will default to run in the same sequence (not in parallel)
  runner "MyModel.some_process", halt_sequence_on_failure: true # If this job fails, subsequent jobs will not run (defaults to false)
  rake "my:rake:task"
  command "/usr/bin/my_great_command", sequence: nil # This job runs in parallel with other jobs that are not part of a sequence
  runner "MyModel.task_to_run_at_four_thirty_in_the_morning", sequence: 'other' # This job runs sequentially with other jobs in the sequence called 'other'
end

Outstanding Questions

• Is there a need to configure the command delimiter? i.e. is && always what we want, or do we want to be able to use ; so the sequence continues even if one command fails. This feature has been added.

Note

• This PR also includes a previous PR of mine that allows the options set on an every block to be overridden by those set on the individual jobs in the block.