Draft Changelog

The following items should be added to the changelog for the next Toil release, at which point the list should be cleared. Add new items at the bottom:

Version 6.x Draft

Changelog

Highlighted Features Added

• WDL and CWL task standard output and standard error logs that are not captured by the workflow will now be logged at INFO level and stored in the --writeLogs/--writeLogsGzip directory. (#4657)
• Use a default log limit of 100MiB (#4788)
• toil debug-job now has --retrieveTaskDirectory <dir> which will set up a job's downloaded files under <dir> and try to stop the job after doing the downloads. Jobs can call self.files_downloaded_hook() to provide a stopping point for this mode. (#4815)
• toil debug-job can now reconstruct the inside-the-container environment for CWL and WDL tasks. (#4815)
• Added support for caching on Slurm and other HPC schedulers (#4775)
• Replace all instances of boto2 with boto3 for all Toil AWS code (#4718)

Breaking Changes

• Stats and logging system again uses job display name (#4755)
• --disableProgress is once again a flag that doesn't take an argument (#4758)

CWL

• Don't clear out user-provided values for the --default-container option (#4730)

WDL

• WDL job names now include numbers for scatters (#4755)
• Multi-line WDL placeholder substitutions no longer interfere with de-indenting WDL command blocks (https://github.com/chanzuckerberg/miniwdl/issues/665)
• Standard error for failed tasks is now always logged to the worker log somewhere (#4781)
• Update WDL conformance tests on CI (#4875)

Kubernetes

Dependencies

• Deps: removed the ruaml.yaml.string plugin dependency for a simpler solution (#4760)

Misc

• Toil will no longer warn about a missing XDG_RUNTIME_DIR (#4769)
• Read the Docs and CI docs builds should have Graphviz installed (pending CI image rebuild) (#4734)
• Add more Python3.12 compatibility by replacing the one function from distutils that we use, strtobool(). (#4765)
• Set default cache folders to be accessible between toil-wdl-runner workflows (Same as MiniWDL/Singularity defaults) (#4761)
• Set toil-wdl-runner cache folders on Toil managed clusters to be at /var/lib/toil (#4761)
• Fall back to assuming machine has 1 core when CPU count is unavailable. (#4545)
• FileJobStore now supports filenames that get modified when percent-encoded (#4779)
• Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (#4820) (#4826)
• Toil will now wait --jobStoreTimeout seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (#3814)
• Toil job descriptions no longer have a command field, and we track the link to the job body and the command to invoke the Toil worker separately. (#4811)
• Several typos in the docs were fixed (#4889)

Thank you to our contributors: