December 2021 Release 2

With ROBOT 1.8.3, which patches an issue with ROBOT report (custom queries) and also gets rid of Log4J.

No changes to ODK repo, so no need to run updates. To update your ODK container:

docker pull obolibrary/odkfull

December 2021 release

Attention: breaking changes below!

• Updated to ROBOT 1.8.2
• Updated to DOSDP tools 0.18
• Adding use_base_merging to config files, which enables the BASE file pipeline, a completely different way to handle imports. This feature is not ready for primetime, but if you are interested in testing this, get in touch. For more details read (pull).
• Adding back ssh/scp
• Migrated to Java 11 as the base Java in ODK (pull)
• Adding make_base feature that allows to autogenerate base files from ontologies where they do not exist (pull)
• Adding new command sh run.sh make no-mirror-refresh-imports which refreshes imports without refreshing mirrors. Can be used for individual ontologies as well.
• Making owltools where necessary configurable with a bespoke memory parameter (pull)
• Fixing the GitHub action to auto-deploy the documentation (pr).
• Fixed a bug where the DOSDP pages in ODK where generated in the wrong part of the mkdocs documentation.
• New command sh run.sh make explain_unsat which generates a nicely formatted set of explanations for your unsatisfiable classes (pull)
• Adding method to measure the memory consumption of your builds. For example, you can now run IMAGE=odklite ODK_DEBUG=yes ./run.sh make prepare_release to run your release on the (much lighter) odklite container of ODK, and get a nice benchmark summary at the end:

### DEBUG STATS ###
Elapsed time: 7:49.24
Peak memory: 6517356 kb

• Breaking changes:
  • Imported modules are no longer generated in OBO format automatically (only OWL). This reflects the fact that in most ontologies, even those where the editors file is in OBO format, imported ontologies are usually in OWL format. You can use the export_obo option to add them back (pull)
  • Equivalent class default setting changed from all to asserted only (pull). This means that from now on, if you dont change the setting deliberately, your pipeline will fail if their are equivalent classes that are not deliberately asserted.
  • OWL 2 DL profile checking is now true by default. You have to actively switch it off by setting ensure_owl2dl_profile to FALSE in your config file.

To update to the latest version of ODK:

docker pull obolibrary/odkfull

To mirgate your repository to the latest version of ODK:

cd myont/src/ontology
sh run.sh make update_repo
sh run.sh make update_repo #(needs to be run twice since the update process also updates the update process.)

October 2021 release

• Important: The way we install python packages has changed significantly: we are now using fixed version dependencies (issue). If there are problems with the versions of packages we are using, please let us know immediately.
• We are now using multi-stage builds in ODK for docker. The obolibrary/odklite container is considerably lighter (smaller) than the normal odkfull container you have been using so far, and it should be sufficient for most ODK pipelines.
• Mirrors are now downloaded with CURL instead of ROBOT, and configurable with mirror_retry_download and mirror_max_time_download in the import_group section of the ODK config file (PR). Note that the default mirror_max_time_download is 200 sec (a bit more than 3 minutes), which may be tight for some huge ontologies.
• odkfull now includes Soufflé.
• For those who are using DOSDP patterns we re-introduced the pattern schema check even if pattern generation is skipped (it is very fast).
• Created a simple way to cite ODK from within GitHub, using the CITATION.cff file. If you go to https://github.com/INCATools/ontology-development-kit, you will now see a "cite this repo" section under the About section.
• Changed the default README.md to include a better reference to ODK, the correct ontology description text and the correct edit file extension.
• Added a GitHub action to deploy the ODK-based mkdocs documentation (issue).
• Created a page for frequently used ODK commands.
• Bugfixes:
  • project repo name was not read correctly during make update_repo causing it to be named "False" (commit)
  • Fixed a bug where mirrors were not considered precious
  • Fixed a bug where where the report directories where not created when running a sparql report command (commit)
  • Fixed a bug where report command incorrectly did not include components in the check (issue)
  • Fixed a bug where myont-odk.yaml was not created when using command line mode during seeding (issue)
  • Added some tests for external tools (issue) to ODK built process
  • Konclude now works on the arm64 variant.

June 2020 release

• Switched to a more up-to-date base image (ubuntu-20.04, pull)
• Lots of technical changes on how to manage releases on dockerhub, especially multi-arch (same image should now work on M1, i.e. arm64 and amd64 machines, pull)
• Pip install operations now using python3 -m pip (pull)
• Introduced a new option (remove_owl_nothing) in repo config to remove mentions of owl:Nothing in releases
• New python packages: linkml, kgx and funowl
• Moved a comprehensive test suite to GitHub actions (pull)
• Updated the documentation on how to create a new repo with the ODK
• Updated Repo README with more docs, a list of significant external contributors (please just let me know if anyone else needs to be added), core team etc.
• Bugfixes:
  • Unnecessary use of .FORCE in custom components removed (pull)
  • Fix another bug with OWL DL profile validation (pull)
  • Changed the -g/--skipgit option be be a flag (pull)
  • Some fixes to how subsets are handled (pull)

May 6 2021 release

v1.2.28 (6 May 2021)

• New DOSDP tools version 0.17
• Major: we developed an Apple M1 compatible ODK image! Thanks to @gouttegd who did all the hard work. Super excited about this! The new image can be found here: https://hub.docker.com/r/obotools/odkfull_m1/
• Some major fixes to the automatic documentation generator, see for example https://pato-ontology.github.io/pato/. Please get in touch if you want help setting up your own awesome documentation site!
• A bug fix in the OWL2DL validation pipeline for testing
• A bug fix for the subset injection during Module creation, see here
• Minor fixes to GitHub instructions (switching wordings from master to main)

2021-04-04 release

v1.2.27 (4 April 2021)

• revived odklite image with a minimum install (smaller in size than odkfull, with just robot and owltools!)
• revived robot image with just robot installed
• Added Jena 3.12.0 to odkfull image
• Added module configs (see docs)
• added a complete new documentation system for ontologies using mkdocs (see docs)
• added OWL2 DL profile checking (see here for example)
• Added sqlite3, dos2unix, and aha to odkfull image
• Added a use_custom_import_module feature (see docs for example)
• Added ability to add completely customised release artefacts (see Uberon repo for example)
• Revised update mechanism. Please run the make update_repo multiple times if you encounter problems and to ensure that all changes are picked up.

v.1.2.26 release (February 2020)

• New versions:
  • fastobo validator to new version (pull
  • ROBOT 1.8.1 (lots of new changes, see: Changelog)
  • DOSDPTOOLS 0.16 (lots of speed-up for bulk pattern generation)
• New features:
  • new python dependencies (cogs, a tool to directly manage tsv files in your repo on Google sheets)
  • stable serialisation order for JSON files using jq's walk function. -> this decreases the size of the diff for git!
  • Some improvements to logging when seeding a new repo, to make it easier to find errors
  • A new method to validate the id-ranges file can be invoked using sh run.sh make validate_idranges (after update to latest ODK repo)
  • modules are now annotated with a dc:source annotation to their original ontology (version)
• New configuration options:
  • module_type (example slme, example minimal, example mireot). Direct support for MIREOT and a new module type, minimal.
  • To encourage stable versions and releases, ODK, by default, merges imports into the release files. Previously, we continued to release the imports as well - which we do not recommend anymore. If you still wish to release your imports as usual, you can set a flag release_imports in the import_group section of your makefile (see example).
  • the same as the above applies for reports (see example)
  • The custom sparql checks, and the custom sparql exports, are now directly configurable
    • custom_sparql_checks : Chose which additional sparql checks you want to run. The related sparql query must be named CHECKNAME-violation.sparql, and be placed in the src/sparql directory (see example)
    • custom_sparql_exports : Chose which additional sparql checks you want to run. The related sparql query must be named CHECKNAME-violation.sparql, and be placed in the src/sparql directory (see example)
    • custom_sparql_exports : Chose which custom reports to generate. The related sparql query must be named CHECKNAME.sparql, and be placed in the src/sparql directory (see example)
  • git_main_branch : The main branch for your repo, default main, or (now discouraged, previously) master.
  • ci: continuous integration defaults; currently available: travis, github_actions
  • create_obo_metadata: This is mainly for new OBO ontologies. If true, OBO Markdown and PURL configs are created.
  • export_project_yaml: Default False. If set to True, project.yaml is created in the top level of the repo.
• Removed a few files from the standard config. This is all part of an effort to slimming down the ODK to the least number of necessary files checked into version control: src/ontology/Dockerfile,src/ontology/patterns.sh, src/ontology/release.sh, src/ontology/test.sh, and some temporary files. The patterns directory and all its contents only appear now when use_dosdps=TRUE.
• Technical:
  • Refactored ODK Dockerfile (merged some layers)
  • added jq 1.6 which is not available via apt-get (yet).
  • added sssom python package, but its in so alpha-alpha state, it should be used with caution

February 2021 ODK 1.2.26 release (amended)

Some hotfixes and a features added to v.1.2.26 release (all pertaining to ODK repo template, not the tools)

• Hotfixes:
  • The new mireot module technique was buggy and is therefore removed again. Sorry; we will try again next time. You can still use the custom option to implement mireot yourself!
  • A change in the way imports were processed introduced a very high memory footprint for large ontologies and slowed stuff down. If you do not have a lot of memory (and time!) available, you should use the following new flags: is_large and use_gzipped. is_large: TRUE introduces a special handling for the ontology that is faster and consumes less memory when creating an import. Using use_gzipped will try to download the ontology from its gzipped location. Make sure its actually there (we know its the case for chebi and pr at least)!

import_group:
  products: 
    - id: pr
      use_gzipped: TRUE
      is_large: TRUE
    - id: chebi
      use_gzipped: TRUE
      is_large: TRUE

• An irrelevant file (keeprelations.txt) was still generated even if needed when seeding a new repo.
• Module type STAR was accidentally hard coded default for slme. Now changed to BOT as it was.
• CI configs where not correctly copied by update routine. Now it does. Note for the changes to be picked up, you need to run sh run.sh make update_repo twice (once for updating the update script itself)!
• Geeky (but necessary) all phony make goals are now correctly declared as .PHONY.
• Some last minute features:
  • In new repos, the README.md is now generated with the correct, appropriate banners.
  • We now have a new feature, custom_makefile_header, that allows injecting a custom header into the Makefile. Most mortals wont need this, but this is how it goes:

custom_makefile_header: |
  ### Workflow
  #
  # Tasks to edit and release OMRSE.
  #
  # #### Edit
  #
  # 1. [Prepare release](prepare_release)
  # 2. [Refresh imports](all_imports)
  # 3. [Update repo to latest ODK](update_repo)

• all features and fixes here: #397

2020-11-18 November release (Hotfix)

updated v1.2.25 (18 November 2020)

• Updated ROBOT to new version 1.7.2, which includes some hotfixes for ROBOT report and update to whelk 1.0.4
• Fixed a bug (#376) that prevented certain things (like imports and pattern generation processes) to be printed when running the Makefile.

2020-11-06 (1.2.24) ODK Release

• Updated ROBOT to new version 1.7.1
• Added the (highly experimental) ability to ODK to run OBO dashboard (see instructions and examples).
• Added more python packages to ODK, see requirements.txt.
• Added a new set of configurations for ROBOT report. WARNING:
  report_fail_on option is now deprecated in favour of a new block of options:

robot_report:
  use_labels: TRUE
  fail_on: None
  custom_profile: TRUE
  report_on:
    - .owl
    - .obo
    - edit

• use_labels: allows switching labels on and off in the ROBOT report, see here
• fail_on: is the old report_fail_on option, see here.
• custom_profile: allows switching on custom profiles, see here
• report_on allows specifying which files to run the report over (for example hp.owl, hp.obo, hp-edit.owl).