This file documents the development processes that have not (yet) been automated and the practices that developers should follow when working on NIFSTD.
All of these processes expect the full NIF-Ontology development environment to have been set up according to development-setup.md.
All relative filepaths listed in code sections start from the base
folder of this repository ../ from the location of this file.
The uri.neuinfo.org resolver points to the master branch on GitHub
(see the resolver config).
Therefore, please do not push to the master branch.
If you are making changes to the ontology that are more than cosmetic
please push to the staging branch. That way merging from staging to
master can be accompanied by a SciGraph build and pushed out to services
so that everything can stay 'mostly' in sync.
Please submit a pull request (even if it is from another branch in the SciCrunch org repo) so that we can run all the requisite checks to make sure that the ontology is consistent.
catalog-v001.xml.example should also always be updated on master
There is an example ontology catalog catalog-v001.xml.example
that should be installed by running cp catalog-v001.xml.example catalog-v001.xml. The file itself
is not tracked to prevent conflicts when switching branches.
Sometimes we don't want to have to retrieve remote copies of files every time we start protege, or we need to run tests for patches that are applied to remote imports when loading the ontology into SciGraph.
cd ttl/
cat <(head -n -1 catalog-v001.xml) ../catalog-extras <(tail -n 1 catalog-v001.xml) > catalog-new
mv catalog-new catalog-v001.xmlLocal copies can be obtained using the following.
cd ttl
curl --header 'Accept-Encoding: gzip' --compressed --location --create-dirs \
$(cat ../catalog-extras | cut -d'=' -f4 | cut -d'"' -f2 | sed 's/^/-o /') \
--remote-name-all $(cat ../catalog-extras | cut -d'=' -f3 | cut -d'"' -f2)Due to a bug in protege we need to patch our import chain in order to for the reasoner to run correctly.
patch -p1 -i dev-reasoner.patch
# OR
cd ttl/
patch -p2 -i ../dev-reasoner.patch
# Then
protege # can be launched as you see fit- In protege menu bar
File -> Openand choosettl/nif.ttl(after first loadFile -> Open recentis faster). - Make sure reasoner is set to ELK (menu bar
Reasoner -> ELK 0.4.3) - Ctrl-R
Protege does not have to be run from ttl/ in order to find ttl/catalog-v001.xml,
however the file opened must be in ttl/.
- Edit your file and save.
- If using protege
git diffto make sure that any new triples have landed in the correct file. - Execute prior to commit processes.
touch filename.ttl
add filename.ttl
`make_catalog` # NOTE this is broken at the moment- Add an entry in catalog-v001.xml by hand.
Note that you will need to make sure that
catalog-v001.xmlis copied from the example order to make use of the new entry.
Before you run make_catalog make sure that the import chain is local (unpatched)
otherwise it will take an extremely long time to fetch files.
ttlfmt $(git status -s | grep ttl | grep M | cut -d' ' -f3)
# OR if you are working on files that need to be normalized
qnamefix $(git status -s | grep ttl | grep M | cut -d' ' -f3)All remote imports should be imported into NIFSTD via a bridge file. This file will hold any local additions that we want to make.
- Create a new bridge file by copying an existing bridge file and modifying
as needed to import the remote file from its canonical iri, and to import
filename-dead.ttl. - Add an entry in catalog-extras for the new import.
- Create
ttl/generated/filename-dead.ttlby runningnecromancy http://myurl.org/filename.owl. - Whenever there is a new release repeat step 3.
If there is an existing ttl file that has been formatted with ttlfmt as described above then create a new branch, add the file, and submit a pull request for review.
If there is a chance that this file may need to be regenerated from an external source please add the external source and the generation code via the process described in pyontutils/nifstd/development/README.md. This does not need to be done if ttl file will become the source of truth once integrated into NIFSTD.
Any set of terms that requires more than direct transformation to ttl should also follow the process described in pyontutils/nifstd/development/README.md.
- Remove
/tmp/ttlcmp.patchif it exists ttlcmp.sh(lives inpyontutils/pyontutils)- Review
/tmp/ttlcmp.patchfor annotation and object properties.
Full documentation of defaults and command line arguments can be found by running
--help for any of the commands listed in this section.
All of these will soon be set up to run on cron jobs and automatically submit pull requests to the NIF-Ontology repository.
All of these scripts expect pyontutils and the NIF-Ontology repos to be
in the same directory ${HOME}/git/pyontutils ${HOME}/git/NIF-Ontology.
I am in the process of making this configurable (for some scripts is already is).
The following sections require you to have already run pipenv install --skip-lock
and pipenv shell. For registry-sync you may want to run pipenv install --skip-lock --dev
in which case see the
development installation notes
for potential pitfalls.
Start with
cd ~/git/pyontutils && # adjust accordingly
pipenv shellBRANCH="registry-$(date +%s)"
registry-sync &&
cd ${PATH_TO_ONTOLOGY} &&
git pull && # alternately git clone and set pushurl -> script-user or something
git checkout -B ${BRANCH} &&
git push --set-upstream origin ${BRANCH} &&
echo success
# TODO auto github api pull
TODO
TODO
The exact details of the release process are still being worked out so this section may be out of date. Check the underlying documentation and the python scripts/commits for the latest information.
Besides the checks listed as part of the development process we need to integrate automated tests, probably using ROBOT. See also #60.
Only bump the versionInfo string on nif.ttl for major releases (e.g. one where nif.ttl is pushed to bioportal). Our versionIRI implementation on top of the unix epoch provides the logical end of granular versioning so we will use that for all practical versioning needs. It seems unlikely that we will bump to a 4.0 release any time in the near future so simply increase the count on the version by one for each release.
Run ontutils version-iri ttl/nif.ttl from the root of the ontology git
directory. Then run ontutils iri-commit NIF-Ontology -l ${git-base} and
copy and paste the git commit command and execute it. This can now be run
on any branch and will resolve correctly when merged to master.
TODO: Should we add a process for adding a versionIRI to individual files whenever there is a substantial non-cosmetic change? It is already possible to use the epoch based versionIRIs to refer to files even if we do not include the versionIRI in the file explicitly.
If the changes are minor don't bother with a pull request, run
git checkout master && git merge staging and then push.
If there are more major changes, such as changes to modelling of a domain
or deprecation of ontology files, submit a pull request, even if you are the
only person who will participate in it. This can provide a nice summary for
someone who needs to review the history of changes but shouldn't have to dig
through the full commit log.
This process has been automated via run-load-graph-nifstd
or run-load-graph-sparc. See the nifstd
SciGraph README
for details.
This only needs to be done if the services config has changed.
This process has been automated via run-build-deploy-nistd or
run-build-deploy-sparc. See the nifstd
SciGraph README
for details.
See the RPM secion of the the pyontutils scigraph readme for the rest of the instructions for deployment.
- Restart the
ontreeprocess on aux-resolver. This is needed to flush the cache of the import chain.
Information about the implementation of ontology.neuinfo.org is mostly covered in
the resolver readme.
See pyontutils/nifstd/resolver/aux-resolver.conf
and pyontutils/nifstd/resolver/nif-ont-resolver.conf.
See pyontutils/README.md#requirements
for details on pandoc and orgstrap. For the best results use >=app-text/emacs-26.1 and >=app-text/pandoc-2.2.3.2.
For centos 7 (i.e. the current aux-resolver sever) this means
- download, extract, build, and install emacs
- download, extract, and install the latest version of pandoc.
git clone https://github.com/tgbugs/orgstrap.gitand runorgstrap/orgstrap
On aux-resolver SCICRUNCH_API_KEY=$(cat api-key) ont-docs
Use a crawler to check for broken links. For example using pylinkvalidator.py one can run the following.
pylinkvalidate.py \
--progress \
--depth 1 \
--workers 4 \
--parser=lxml \
--types=a \
--accepted-hosts=github.com,scicrunch.org,bioontology.org,fairsharing.org,neuinfo.org,interlex.org,wordpress.com,w3.org \
http://ontology.neuinfo.org/docs/On aux-resolver cp -a pyontutils/pyontutils/doc_build/docs /var/www/ontology-docs
There are a couple of places where there are external records of the ontology that a maintainer or curator needs to keep up to date.
- https://scicrunch.org/resolver/SCR_005414 (NIF Ontology)
- https://scicrunch.org/resolver/SCR_005402 (NeuroLex)
- https://scicrunch.org/resolver/SCR_016178 (InterLex)
- https://bioportal.bioontology.org/ontologies/NIFSTD
- https://bioportal.bioontology.org/ontologies/NIFSUBCELL
- https://bioportal.bioontology.org/ontologies/NPOKB
- https://bioportal.bioontology.org/ontologies/ATOM
- https://fairsharing.org/bsg-s002584 (NeuroLex)
- https://fairsharing.org/bsg-s002628 (NIF Ontology)
- https://fairsharing.org/bsg-s002835 (subcellular)
Pages that need to be kept up to date / sourced from material here.