Scrumdo Extras Guide

This guide gives some information on how to create a ScrumDo project based Extra.

What is an Extra?

A ScrumDo Extra provides extra functionality in a plugin-like way. Most of the time, they contain functionality to syncronize ScrumDo stories with external sites.

Getting started

To create a ScrumDo extra, create a new Python module to represent your Extra. This module should contain a class that extends the apps.extras.plugins.interface.ScrumdoProjectExtra class.

See apps.extras.plugins.github_issues.plugin for an exampe.

Then, open up your settings.py and add an entry to SCRUMDO_EXTRAS that points to your plugin class.

Example:

SCRUMDO_EXTRAS = ("extras.plugins.github_issues.GitHubIssuesExtra",
                  "extras.plugins.example.ExampleExtra",)

Writing your Extra

In your Extra class you must implement the following methods:

  def getName(self):
    "Returns a user-friendly version of the name of this extra.  Generally should be just a couple words long."      
    
  def getLogo(self):
    "Returns a URL to a logo that can be used on the config page."
          
  def getSlug(self):
    " Returns a version of the name consisting of only letters, numbers, or dashes.  Max length, 25 chars    "

  def getDescription(self):
    " Returns a user-friendly description of this extra.  This text will be passed through a Markdown filter when displayed to the user. "     

  def getShortStatus( self,  project ):
    """ Should return a string representing the current status that this extra has for a given project.  
      Examples: 'Successfully syncronized on 1/1/2010' or 'Syncronization last failed' or 'Everything OK' """

  def doProjectConfiguration( self, request, project , stage=""):
    " Should return a django style response that handles any configuration that this extra may need. "

The following methods are optional, but you'll likely want to implement many of them:

  def associate( self, project):
    "called when an extra is first associated with a project."
    
  def unassociate( self, project):
    "called when an extra is removed from a project."
    
  def initialSync( self, project):
    """ Does whatever needs doing for an initial sync of the project. 
        An extra's configuration should add this event to the queue when
        it's ready.  """
      
  def pullProject( self, project ):
    """ Should cause a full pull syncronization of this extra from whatever external source 
        there is.  This will be called on a scheduled basis for all active projects.  The project 
        parameter be an apps.projects.models.Project object.    """    
      
  def storyUpdated( self, project, story ):
    "Called when a story is updated in a project that this extra is associated with."
    
  def storyDeleted( self, project, external_id):
    """Called when a story is deleted in a project that this extra is associated with.
       Note: the ScrumDo story has already been deleted by the time this method is called. """
    
  def storyCreated( self, project, story):
    "Called when a story is created in a project that this extra is associated with."
    
  def storyStatusChange( self, project, story):
    "Called when a story's status has changed in a project that this extra is associated with."

Syncronization Queue

In general, implementing the methods above should provide you with the neccessary touch-points to implement a story syncronization plugin. Behind the scenes, those methods are called by an external script (called extras_sync.py) outside of the lifecycle of the web request. That script process a syncronization queue to know what to do.

The actual flow goes like this:

A user creates a new ScrumDo Story
The projects app dispatches an apps.projects.signals.story_created signal
The extras app handles that signal in apps.extras.manager.ExtrasManager.onStoryUpdated
- It creates a new SyncronizationQueue entry representing this action
At some later time, the extras_sync.py script runs (generally invoked via cron every couple minutes)
- It grabs the list of SyncronizationQueue objects
  - For each one, invokes the requested method on the plugin class

That desribes how create, update, or delete events get propgated from ScrumDo to an external system.

To syncronize external systems with ScrumDo, the flow goes like this:

The extras_sync.py script is run with the --pull command line argument (generally invoked via cron every couple hours)
- It populates the SyncronizationQueue with a pull request for each project/extra combination.
At some later time, the extras_sync.py script runs with no arguments (generally invoked via cron every couple minutes)
- It grabs the list of SyncronizationQueue objects
  - For each one, invokes the requested method on the plugin class (in this case the pullProject)

You can manually create a pull request in the syncronization queue when it makes sense to do so. For instance, the GitHub-issues Extra creates one whenever it's configured to cause the system to pull down a list of issues.

Configuring your extra

Above, we mentioned a method called doProjectConfiguration.

  def doProjectConfiguration( self, request, project , stage=""):
    " Should return a django style response that handles any configuration that this extra may need. "

In this method, you get a chance to directly interact with the user. You should treat this method like a Django view method. The user will see it after they click on the "configure" button on the extras page.

This method should handle prompting the user for any configuration logic that it requires. For instance, this is what the GitHub-issues extra displays:

Depending on the needs of your Extra, you may need to do other initialization logic as well. Here's what the GitHub-issues method looks like:

  def doProjectConfiguration( self, request, project , stage=""):
    """Handles a request to do configuration for the github_issues extra.
       This displays a form asking for credentials / repository information,
       then saves that with the saveConfiguration() api in ScrumdoProjectExtra base
       class.  After a successful configuration, we redirect back to the extras page.
       (Should each extra be responsible for that?)"""
         
    # The super class has a helper getConfiguration method that will return a dict of options.
    configuration = self.getConfiguration( project.slug )    
    if request.method == "POST":
      form = forms.GitHubIssuesConfig( request.POST )
      if form.is_valid():
        configuration = form.cleaned_data
        configuration["status"] = "Configuration Saved"
      
        # The super class has a helper saveConfiguration method that will save a 
        # dict of options for later retrieval by getConfiguration
        self.saveConfiguration( project.slug, configuration )        
      
        if configuration["upload"]:
          # Need to queue an intial action to upload our project.
          self.manager.queueSyncAction(self.getSlug(), project, SyncronizationQueue.ACTION_INITIAL_SYNC)
        
        return HttpResponseRedirect(reverse("project_extras_url",kwargs={'project_slug':project.slug}))
    else:  
      form = forms.GitHubIssuesConfig(initial=configuration)
    return render_to_response("extras/github_issues/configure.html", {
        "project":project,
        "extra":self,
        "form":form
      }, context_instance=RequestContext(request))

You can have a multi-step configuration wizard by utilizing the optional stage parameter following a pattern like this:

 def doProjectConfiguration( self, request, project  ):                
    if stage == "":
        return doConfigurationSummary(extra, request, project, configuration)
    
    if stage == "filter":
        return doFilterConfiguration(extra, request, project, configuration)

    if stage == "statuses":
        return doStatusConfiguration(extra, request, project, configuration)

    if stage == "summary":
        return doConfigurationSummary(extra, request, project, configuration)

    return doCredentialsConfiguration(extra, request, project, configuration)



def doStatusConfiguration(extra, request, project, configuration):
    ...
    if request.method == "POST":
        form = forms.StatusForm( statuses, request.POST )
        if form.is_valid():
            ...
            return HttpResponseRedirect( reverse('configure_extra_with_stage', args=[extra.getSlug(), project.slug, "summary"]) )
    else:
        form = forms.StatusForm( statuses )
        
    return render_to_response("plugins/boom/config_status.html",
        {"project":project,
          "extra":extra,
          "form":form
        }, context_instance=RequestContext(request))

Debugging

All of the syncronization operations are queued in the background and are executed by an external script (called extras_sync.py). This means that you'll need to manually run this script in your development environment to test your code. To do this, start up your ScrumDo environment and then run the script on the command line.

Recommended syncronization logic

On a call to pullProject():

Load the list of external stories (or story-like objects)
For each external story:
- Check if there is an entry in StoryQueue for it.
  - If it exists, update that StoryQueue object if neccessary.
  - If not, check if there is an entry in Story for it
    - If it exists, update it if neccessary
    - If it doesn't exist, create a StoryQueue object for it.

On a call to pushProject():

Load the list of local stories
For each local story:
- Check if it has an ExternalStoryMapping already
  - If it exists, update the remote service object
  - If not
    - create a remote service object.
    - create a new ExternalStoryMapping

**On a call to storyUpdated **

Check if an ExternalStoryMapping exists
- If so, perform the operation (update) on that remote object.
- If not, create the remote object and create an ExternalStoryMapping

**On a call to storyDeleted **

Check if an ExternalStoryMapping exists
- If so, perform the operation (delete) on that remote object.
- If not, do nothing