Project Roadmap

[claymcleod]

The following are a (semi-)prioritized list of project goals that we've set out to achieve with Sprocket. Keep in mind that not all work will occur in this repository—some will occur in wdl, some will occur here, some will occur in sprocket-vscode, and some will occur in new purpose-built repositories.

Core facilities

• Build a full lexer and parser (wdl-grammar) that is targeted at the WDL v1.2 specification.
• Build a concrete syntax tree (wdl-grammar) implementation that will support incremental updates via red/green trees.
• Build an abstract syntax tree (wdl-ast) to wrap all of the semantically relevant bits.
• Build infrastructure for linting and validation errors (across wdl-lint and wdl-ast).
• Build an analysis engine (wdl-analysis) that resolves cross-file dependencies.
• Build an initial formatting engine (wdl-format).
• Build an LSP that makes these lints and validation errors available in editors (wdl-lsp).
• Write an execution engine that fully evaluates and executes WDL v1.2 workflows locally (wdl-engine).
• Extend the execution engine to work on multiple platforms (via wdl-engine and crankshaft).
• Create an off-the-shelf documentation tool for pleasantly releasing documentation for any WDL workflow (wdl-doc).

Configuration

• Provide configuration for the suite of tools above available at canonical system paths.
  • Introduce configuration for enabling and disabling individual lint rules.
  • Introduce configuration for enabling and disabling groups of lint rules.
  • Introduce configuration for format parameters.
  • Introduce configuration for linting parameters.
  • Introduce configuration for the execution engine.

Editor support

• Create a first-class extension for WDL in VSCode (vscode-sprocket).
  • Improve syntax-highlighting to support WDL v1.2 via the TextMate grammars.
  • Semantic code highlighting (integrated in with the LSP).
  • Integrate code formatting into the extension.
  • Create a solid set of templates.
  • Consider automated documentation block generation.

Running Workflows

• Consider dynamically generating help messages for WDL workflow/task inputs when sprocket run is used and adding those to the help message dynamically.

Developer Experience

• Ease the issue of Docker container image building being decoupled from pipeline testing. This probably means adding some configuration to Sprocket.toml that allows you to specify the location of certain Docker images to discrete Dockerfiles in your project and transparently building then before each run.
• Introduce a local mode that bypasses Docker containers and uses your underlying environment during testing. This allows you to focus on iterating on the functionality by tweaking your local environment and then coming back to build/productionize a Docker image later (once you've encountered and fixed all the stumbling blocks you run into in your local environment).
• Consider how we can make Sprocket the go-to tool for end-to-end testing of bioinformatics tool development. For example, if you write a WDL wrapper for your a bioinformatics tool you wrote that lives alongside that tool in the repo, you should be able to make a few end-to-end tests in the tests/ directory, run sprocket test, and ensure all the results still look good (or similar). This will also encourage folks to write WDL wrappers for their tools that live in the tool repos themselves.

Extensions to WDL

The following are extensions to WDL that are needed to really improve bioinformatics software development using WDL.

• Documentation should be introduced using comments. Deprecate the input and output parameter information that is part of the language.
• Support for symbolic importing should be introduced such that you can import libraries from a package manager (without needing a HTTP or local path), effectively allowing the decoupling dependency management from the language.