Pl/remove distribution table

README.md

@@ -5,42 +5,39 @@
 
 This is a draft for a PEtab NLME extension. It adds the following to the existing PEtab specification.
 
-## Individual table
+## YAML configuration file
 
-This table is an addition to the existing PEtab format. It introduces a classification of individuals (most fine grained grouping) into several (more coarse grained) groups. Random effects can be specific to each individual or to the groups they belong to.
+In addition to existing attributes, this extension uses `covariance_files` to define distributions of random effects. Each distribution has a `distributionId`, `distributionType` (e.g. normal, laplace) and `covariance` (defined in a TSV file). Distributions have zero mean. The `randomEffectFormula` in the random effect table can be used to shift the location of the distribution.
 
-### Detailed field description
-
-- `individualId` [STRING]
-- `${groupId}` [STRING, OPTIONAL]
-
-   Further columns may be used to group individuals into different categories.
+## Random effect table
 
-## Parameter table
-
-In addition to the existing descriptions of fixed effects, the NLME extension adds the following columns to describe random effects.
+In addition to the existing descriptions of fixed effects in the parameter table, the NLME extension adds a random effect table to describe random effects.
 
 ### Detailed field description
 
 - `randomVariableId` [STRING]
 
-   ID of the random variable. If left empty, there is no random effect defined on the parameter of this row.
+   ID of the random variable.
 
-- `randomEffectFormula` [STRING]
+- `distributionId` [STRING]
 
-   Random effect function as plain text formula expression. Must contain the `parameterId` and the `randomVariableId` and must not contain any other symbols (or should we allow that???).
+   Specifies the Id of the distribution from which this random effect is drawn. Any Id from the distributions listed in the YAML configuration file is allowed.
 
--  `distributionType` [STRING: 'normal' or 'laplace', OPTIONAL]
+- `estimate` [0|1]
 
-   Assumed random effect distribution. Only normal and laplace distributions are currently allowed (but we could allow any term in some ontology for distributions). Defaults to normal. Distributions are assumed to be zero-centered. Covariances are defined in the covariance table.
+   1 or 0, depending on, if the parameter is estimated (1) or set to zero (to shift a random effect away from zero, see `randomEffectFormula`).
 
-- `groupType` [STRING]
+- `parameterId` [STRING]
 
-   Defines how random effects are grouped together. Any column name from the individual table is allowed, except for covariate (i.e. numeric) columns.
+   ID of the associated fixed effect.
+
+- `randomEffectFormula` [STRING]
+
+   Random effect function as plain text formula expression. Must contain the `parameterId` and the `randomVariableId`. May contain other `randomVariableId`s from other distributions. Must not contain any other symbols (or should we allow that???).
 
 ## Covariance tables
 
-These tables are an addition to the existing PEtab format to describe the covariances of the random effects. Each `groupType` for which covariances are estimated has its own table. Four formats are allowed
+These tables are an addition to the existing PEtab format to describe the covariances of the distribtions from which the random effects are drawn. Each `distributionId` has its own table. Four formats are allowed
 
 - Matrix format
 
@@ -57,3 +54,7 @@ These tables are an addition to the existing PEtab format to describe the covari
 - No table
 
   If no table is provided, only the variances, but no covariances are estimated.
+
+## Measurement table
+
+In addition to existing collumns, the measurement table must contain every `distributionId` from the YAML configuration file as a column. Each data point (i.e. row) is assigned to one entity for each `distributionId`. This allows grouping of data points. As a simple example, the YAML configuration file defines just one `distributionId` that could be called `patient`. Rows that share the same entry in the `patient` column belong to the same patient. Therefore, they must share the same value for all random effects that are drawn from the `patient` distribution.

measurements.tsv

@@ -1,13 +1,13 @@
-observableId	simulationConditionId	individualId	measurement	time
-obs_B	high_dose	p1	9.4	1
-obs_B	high_dose	p1	1.5	10
-obs_B	low_dose	p1	1.8	1
-obs_B	low_dose	p1	1.06	10
-obs_B	high_dose	p2	9.6	1
-obs_B	high_dose	p2	1.52	10
-obs_B	low_dose	p2	2	1
-obs_B	low_dose	p2	1.08	10
-obs_B	high_dose	p3	9	1
-obs_B	high_dose	p3	1.4	10
-obs_B	low_dose	p3	1.6	1
-obs_B	low_dose	p3	1.04	10
+observableId	simulationConditionId	distributionA	anotherDistribution	yetAnotherDistribution	measurement	time
+obs_B	high_dose	patient1	stage1	blue	9.4	1
+obs_B	high_dose	patient1	stage1	blue	1.5	10
+obs_B	low_dose	patient1	stage1	blue	1.8	1
+obs_B	low_dose	patient1	stage1	blue	1.06	10
+obs_B	high_dose	patient2	stage1	brown	9.6	1
+obs_B	high_dose	patient2	stage1	brown	1.52	10
+obs_B	low_dose	patient2	stage1	brown	2	1
+obs_B	low_dose	patient2	stage1	brown	1.08	10
+obs_B	high_dose	patient3	stage2	brown	9	1
+obs_B	high_dose	patient3	stage2	brown	1.4	10
+obs_B	low_dose	patient3	stage2	brown	1.6	1
+obs_B	low_dose	patient3	stage2	brown	1.04	10

parameters.tsv

@@ -1,9 +1,9 @@
-parameterId	parameterScale	lowerBound	upperBound	nominalValue	estimate	objectivePriorType	objectivePriorParameters	randomVariableId	randomEffectFormula	distributionType	groupType
-k1	log10	0.1	10	1	1	laplace	0; 1	u1	k1*exp(u1)	normal	individualId
-k2	log10	0.1	10	1	1	laplace	0; 1	u2	k2 + u2	normal	individualId
-k3	log10	0.1	10	1	1	laplace	0; 1	u3	k3 * u3	normal	individualId
-k4	log10	0.1	10	1	1	laplace	0; 1	u4	k4 + u4	laplace	tumorStage
-k5	log10	0.1	10	1	1	laplace	0; 1	u5	k5 + u5	laplace	tumorStage
-k6	log10	0.1	10	1	1	laplace	0; 1	u6	k6 + u6	laplace	tumorStage
-k7	log10	0.1	10	1	1	laplace	0; 1	u7	k7 + u7	normal	sex
-k8	log10	0.1	10	1	1	laplace	0; 1				
+parameterId	parameterScale	lowerBound	upperBound	nominalValue	estimate	objectivePriorType	objectivePriorParameters
+k1	log10	0.1	10	1	1	laplace	0; 1
+k2	log10	0.1	10	1	1	laplace	0; 1
+k3	log10	0.1	10	1	1	laplace	0; 1
+k4	log10	0.1	10	1	1	laplace	0; 1
+k5	log10	0.1	10	1	1	laplace	0; 1
+k6	log10	0.1	10	1	1	laplace	0; 1
+k7	log10	0.1	10	1	1	laplace	0; 1
+k8	log10	0.1	10	1	1	laplace	0; 1

petab_nlme_example.yaml

@@ -1,5 +1,6 @@
 format_version: 2.0.0
 parameter_file: parameters.tsv
+random_effect_file: random_effects.tsv
 problems:
 - condition_files:
   - conditions.tsv
@@ -11,15 +12,16 @@ problems:
       location: model.xml
   observable_files:
   - observables.tsv
-  individual_files:
-  - individuals.tsv
   covariance_files:
-    covariance:
-      group: individual
-      locaion: covariance_individual.tsv
-    covariance:
-      group: tumorStage
-      location: covariance_tumorStage.tsv
-    covariance:
-      group: sex
-      location: covariance_sex.tsv
+    distribution:
+      distributionId: distributionA
+      distributionType: normal
+      covariance: distributionA.tsv
+    distribution:
+      distributionId: anotherDistribution
+      distributionType: lognormal
+      covariance: covariance_anotherDistribution.tsv
+    distribution:
+      distributionId: yetAnotherDistribution
+      distributionType: laplace
+      covariance: covariance_yetAnotherDistribution.tsv

random_effects.tsv

@@ -0,0 +1,8 @@
+randomVariableId	distributionId	estimate	parameterId	randomEffectFormula
+u1	distributionA	1	k1	k1*exp(u1+u4)
+u2	distributionA	1	k2	k2 + u2
+u3	distributionA	1	k3	k3 * u3
+u4	anotherDistribution	1	k1	k1*exp(u1+u4)
+u5	anotherDistribution	1	k5	k5 + u5
+u6	anotherDistribution	1	k6	k6 + u6
+u7	yetAnotherDistribution	1	k7	k7 + u7