Quick and easy job queues, based on delayed_job
CakePackages is the type of site that would do well with background processes. Rather than try to build - and fail - a job/worker system, I set out to wrap an existing system.
I am aware of a few plugins for cakephp, notably the queue plugin by David Persson, and the cron-mailer plugin by Pierre-Emmanuel Fringant. However, I don't wish to run and maintain yet another service, and my cron tasks aren't email-based at all. The other thing was the lack of documentation for David's plugin, and I am hesitant to familiarize myself with so many tools at once.
At SeatGeek, we use a PHP port of delayed_job, which works very well in production. It's a neat little set of classes that abstracts away most of the pain of creating new jobs and workers. Unfortunately, the only system I've ever used it in was Symfony (you'll notice we include a sample Symfony task with the package).
So here this is, Cake DJJob. Use it and abuse it to create Jobs/workers whenever necessary :)
- CakePHP 1.3
- Patience
- PHP 5
- PDO (Ships with PHP >= 5.1)
- (Optional) PCNTL library
You are required to install both djjob and cake_djjob. Note that you'll need a jobs table just like the one included with djjob. A cake_schema file is forthcoming.
[Manual]
- Download this: http://github.com/seatgeek/djjob/zipball/master
- Unzip that download.
- Copy the resulting folder to app/plugins
- Rename the folder you just copied to
Djjob/Vendor
[GIT Submodule]
In your app directory type:
git submodule add git://github.com/seatgeek/djjob.git Plugin/Djjob/Vendor
git submodule init
git submodule update
[GIT Clone]
In your plugin directory type
git clone git://github.com/seatgeek/djjob.git Djjob/Vendor
h3. Cake DJJob Installation
[Manual]
- Download this: http://github.com/josegonzalez/cake_djjob/zipball/master
- Unzip that download.
- Copy the resulting folder to app/Plugin
- Rename the folder you just copied to
CakeDjjob
[GIT Submodule]
In your app directory type:
git submodule add git://github.com/josegonzalez/cake_djjob.git Plugin/CakeDjjob
git submodule init
git submodule update
[GIT Clone]
In your Plugin directory type
git clone git://github.com/josegonzalez/cake_djjob.git CakeDjjob
You can create new jobs in the Lib/Job folder of your application or of any plugin. When creating plugin jobs, the plugin name must be in the job class name, but not the job class file. For example, the CakeDjjob plugin contains a CakeDjjob_TestJob in Lib/Job/TestJob.php. All plugins must be separated from the job classname using an underscore, and there can only be one underscore in a classname. This is so you can use jobs from plugins as well as from the application.
Jobs respond to a perform() method, as in djjob. In CakeDjjob, all jobs should extend the CakeJob class. This is not required, but it allows for increased base functionality in your jobs. For example, an application's ForgotPasswordJob might be composed as follows:
<?php
class ForgotPasswordJob extends CakeJob {
var $email = null;
function __construct($email) {
$this->email = $email;
}
function perform() {
$this->out('Some debug output');
// Using models
$this->loadModel('User');
$user = $this->User->findByEmail($this->email);
if (empty($user)) {
return;
}
// Complex email stuff here...
$this->loadComponent('Email');
$this->Email->_set(array(
'to' => $this->email,
'from' => 'noreply@example.com',
'subject' => 'Forgot your password?',
'replyTo' => 'noreply@example.com',
'template' => null,
'sendAs' => 'both',
));
$this->Email->send("Some text to indicate how to reset password...");
}
}
?>Extending the @CakeJob@ also allows the usage of the following methods:
- @loadModel($modelClass, $id = null)@ ** Works exactly as Controller::loadModel() works
- @loadComponent($componentClass)@ ** Loads, initializes, and attaches a Component
- @out($message = null, $newlines = 1)@ ** Outputs messages to standard out
- @err($message = null, $newlines = 1)@ ** Outputs error messages to standard error
- @nl($multiplier = 1, $print = false)@ ** Outputs a new line to standard out
- @hr($newlines = 0)@ ** Outputs a horizontal rule to standard out
By extending @CakeJob@, you get an instant base off of which writing jobs will be quick and easy.
You can attach the @CakeDjjob@ Component to your controller like so:
<?php
App::uses('ForgotPasswordJob', 'Lib/Job');
class UsersController extends AppController {
public $components = array('CakeDjjob.CakeDjjob');
public function forgot_password() {
if (($email = $this->User->forgot($this->data)) != false) {
$this->CakeDjjob->enqueue(new ForgotPasswordJob($email));
$this->Session->setFlash('You will shortly be sent a password reset email :)');
} else {
$this->Session->setFlash('There is no such email in our system. Please try again.');
}
}
}
?>
It is also possible to queue jobs from a Model by attaching the included @CakeDjjob@ behavior:
<?php
App::uses('ForgotPasswordJob', 'Lib/Job');
class User extends AppModel {
public $actsAs = array('CakeDjjob.CakeDjjob');
public function forgot_password($email) {
return $this->enqueue(new ForgotPasswordJob($email));
}
public function bulkAction($emails = array(), $config = array()) {
$jobs = array();
foreach ($emails as $email) {
$jobs[] = new BulkPasswordResetJob($email, $config);
}
return $this->bulkEnqueue($jobs);
}
}
?>
- Requires (PHP 5 >= 5.1.3) and ReflectionClass when loading jobs with arguments
- This is an alternative to manually importing the Job using App::uses().
- The load function requires only that the first parameter be the name of the job to load.
- It assumes that the Job will be imported from /app/Lib/Job
- You can pass it as many parameters as needed as they are simply passed through to your Job
<?php
class User extends AppModel {
public $actsAs = array('CakeDjjob.CakeDjjob');
public function forgot_password($email) {
return $this->enqueue($this->CakeDjjob->load('ForgotPasswordJob', $email));
}
public function bulkAction($emails = array(), $config = array()) {
$jobs = array();
foreach ($emails as $email)
$jobs[] = $this->CakeDjjob->load('BulkPasswordResetJob', $email, $config);
}
return $this->bulkEnqueue($jobs);
}
}
?>
Included with this plugin is a task to run jobs.
cake worker run
The above statement will run a single worker on the default queue forever. It can be added to your cron or by some other process manager such as God. The following params are available for the worker:
- --connection ** set db config . default config: default
- --type ** pdo name for connection . default type: mysql
- --queue ** queue to pull jobs from. default queue: default
- --count ** run of jobs to run before exiting (0 for unlimited). default count: 0
- --sleep ** sleep seconds after finding no new jobs. default seconds: 5
- --max ** max of retries for a given job. default retries: 5
- --debug ** debug to set for worker. default level: 0
It's possible to clean a particular job queue, by either resetting locked jobs or deleting failed jobs:
cake worker cleanup
This task can take the connection, type, queue, and debug arguments, as well as a few custom ones:
- --action ** action to perform on cleanup task. default value: null
- --date ** date offset. default value: current evaluation of date('Y-m-d H:i:s')
- --save ** allow cleanup to modify database. default value: 1
If you need to get the status of a particular job queue, you can call the status method on the task:
cake worker status
This task can take the connection, type, queue, and debug arguments as well.
- Allow alternative methods of plugin prefixing
- Circumvent class-loading via a custom-serialization method
- Configuring location of job classes
- Screencast
- Unit Tests
- More documentation (DeferredEmail class)
- -Task to allow creation of jobs from shells-
Copyright (c) 2011 Jose Diaz-Gonzalez
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.