Skip to content

Commit

Permalink
Update README (#77)
Browse files Browse the repository at this point in the history
Update README

Closes #64


---------

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>
  • Loading branch information
dweindl and dilpath authored Jul 2, 2024
1 parent 3d0bc1b commit a09a120
Showing 1 changed file with 42 additions and 22 deletions.
64 changes: 42 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,48 +5,68 @@ The PEtab test suite is a collection of test models for the
It is intended to be used to verify and quantify PEtab-support by developers
of tools for model simulation and parameter estimation.

## Download and install the test suite
## Downloading and installing the test suite

The PEtab test suite can be downloaded from GitHub via

git clone https://github.com/petab-dev/petab_test_suite

The test suite comes with all necessary files pregenerated.
In the [petabtests](petabtests) subdirectory, it contains a python module for
generating the tests and evaluating results. This can be installed via


### Python library

The PEtab test suite comes with a Python package named
[petabtests](petabtests) in the homonymous subdirectory.
It contains Python functions for generating the tests and evaluating results.
This can be installed via

cd petab_test_suite
pip3 install -e .

## Use the test suite
## Using the test suite

The [petabtests/cases](petabtests/cases) subdirectory contains a collection
of enumerated tests.
Each test contains a descriptive `wxyz.md` file, and a script file `wxyz.py`
file that can be used to generate all problem and solution files for the test.
The necessary files are in the same case-specific folder, starting with an
underscore.
In each case folder, there is a file `_wxyz.yaml` containing the parameter
estimation problem description, and a file `_wxyz_solution.yaml` containing
information on the expected results: chi2 value, log-likelihood, simulation
table reference, and tolerances.
### Files

After download, all necessary files are already pregenerated.
The [petabtests/cases](petabtests/cases) subdirectory contains different test suites
for different PEtab versions and model formats.
Each test suite is a collection of enumerated tests.
Each test consists of a single PEtab problem defined in `XXXX/_XXXX.yaml`
file and the expected result (chi2 value, log-likelihood, simulation
table reference, and tolerances) in `XXXX/_XXXX_solution.yaml`.
`XXXX/XXX.md` contains a short description of the respective test
problem that is not relevant for the execution of the test itself.

Further, installation of the petabtests module installs a routine to recreate
the problem and solution files for all test problems. It can be invoked via

cd petab_test_suite
petabtests_create

## Evaluate results
### Evaluate results

To evaluate how a tool performs on a given test problem, three metrics are
employed: Simulations, chi2 value and log-likelihood. A tool can be said to
cover a test problem if any of those values matches the ground truth values
up to some tolerance.

The python package provides convenience functions for evaluation in
The Python package provides convenience functions for evaluation in
[petabtests/evaluate.py](petabtests/evaluate.py).

-> [Overview of passed test cases for different tools supporting PEtab](https://github.com/PEtab-dev/PEtab#petab-features-supported-in-different-tools)

## Contributing

Contributions to the PEtab test suite are welcome.

### Adding a new test case

To add a new test case, create a new subdirectory in the respective test suite
directory under `petabtests/cases/`. The subdirectory name should be a
four-digit number, starting with the next available number.

Most files in the test suite are generated automatically. Those start with an
underscore, except for the `README.md` files. The only files that need to be
created manually are the `XXXX/XXXX.py` files that contain the script to
generate the test problem and solution files. Their content should be
self-explanatory For adding a new test case, copy `XXXX/XXXX.py` from an
existing test case and adjust it to the new test case.

All remaining files are generated by the `petabtests_create` script which
will be available on your `$PATH` after installing the provided Python library
(see above).

0 comments on commit a09a120

Please sign in to comment.