| jupytext | kernelspec | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
(sec_tutorial_metadata)=
Metadata is information associated with entities that {program}tskit doesn't use or
interpret, but which is useful to pass on to downstream analysis such as sample ids,
dates etc. (see {ref}sec_metadata for a full discussion). Each
{ref}table<sec_tables_api_table> has a {class}MetadataSchema which details the
contents and encoding of the metadata for each row. A metadata schema is a JSON document
that conforms to JSON Schema
(The full schema for tskit is at {ref}sec_metadata_schema_schema). Here we use an
{ref}example tree sequence<sec_intro_downloading_datafiles>
which contains some demonstration metadata:
:tags: [remove-cell]
import msprime
import tskit
def metadata():
tables = msprime.sim_ancestry(4).dump_tables()
tables.individuals.metadata_schema = tskit.MetadataSchema(
{'additionalProperties': False,
'codec': 'json',
'properties': {'accession': {'description': 'ENA accession number',
'type': 'string'},
'pcr': {'description': 'Was PCR used on this sample',
'name': 'PCR Used',
'type': 'boolean'}},
'required': ['accession', 'pcr'],
'type': 'object'}
)
md = [
{'accession': 'ERS0001', 'pcr': True},
{'accession': 'ERS0002', 'pcr': True},
{'accession': 'ERS0003', 'pcr': True},
{'accession': 'ERS0004', 'pcr': False},
]
table = tables.individuals
copy = table.copy()
table.clear()
for m, row in zip(md, copy):
table.append(row.replace(metadata=m))
ts = tables.tree_sequence()
ts.dump("data/metadata.trees")
def create_notebook_data():
metadata()
# create_notebook_data() # uncomment to recreate the tree seqs used in this notebook
import tskit
import json
ts = tskit.load("data/metadata.trees")
(sec_tutorial_metadata_reading)=
Metadata is automatically decoded using the schema when accessed via a
{class}TreeSequence or {class}TableCollection Python API. For example:
print("Metadata for individual 0:", ts.individual(0).metadata) # Tree sequence access
print("Metadata for individual 0:", ts.tables.individuals[0].metadata) # Table access
Viewing the {class}MetadataSchema for a table can help with understanding
its metadata, as it can contain descriptions and constraints:
ts.table_metadata_schemas.individual
The same schema can be accessed via a {attr}~IndividualTable.metadata_schema attribute
on each table (printed prettily here using json.dumps)
schema = ts.tables.individuals.metadata_schema
print(json.dumps(schema.asdict(), indent=4)) # Print with indentations
The top-level metadata and schemas for the entire tree sequence are similarly
accessed with {attr}TreeSequence.metadata and {attr}TreeSequence.metadata_schema.
:::{note}
If there is no schema (i.e. it is equal to MetadataSchema(None)) for a table
or top-level metadata, then no decoding is performed and bytes will be returned.
:::
(sec_tutorial_metadata_modifying)=
If you are creating or modifying a tree sequence by changing the underlying tables,
you may want to record or add to the metadata. If the change fits into the same schema,
this is relatively simple, you can follow the
{ref}description of minor table edits<sec_tables_editing_minor> in the
{ref}sec_tables tutorial. However if it requires a change to the schema, this must be
done first, as it is then used to validate and encode the metadata.
Schemas in tskit are held in a {class}MetadataSchema.
A Python dict representation of the schema is passed to its constructor, which
will validate the schema. Here are a few examples: the first one allows arbitrary fields
to be added, the second one (which will construct the schema we printed above) does not:
basic_schema = tskit.MetadataSchema({'codec': 'json'})
complex_schema = tskit.MetadataSchema({
'codec': 'json',
'additionalProperties': False,
'properties': {'accession': {'description': 'ENA accession number',
'type': 'string'},
'pcr': {'description': 'Was PCR used on this sample',
'name': 'PCR Used',
'type': 'boolean'}},
'required': ['accession', 'pcr'],
'type': 'object',
})
This {class}MetadataSchema can then be assigned to a table or the top-level
tree sequence e.g. {attr}~IndividualTable.metadata_schema:
tables = tskit.TableCollection(sequence_length=1) # make a new, empty set of tables
tables.individuals.metadata_schema = complex_schema
This will overwrite any existing schema. Note that this will not validate any existing
metadata against the new schema. Now that the table has a schema, calls to
{meth}~IndividualTable.add_row will validate and encode the metadata:
row_id = tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True})
print(f"Row {row_id} added to the individuals table")
If we try to add metadata that doesn't fit the schema, such as accidentally using a string instead of a proper Python boolean, we'll get an error:
:tags: [raises-exception, output_scroll]
tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": "false"})
and because we set additionalProperties to False in the schema, an error is
also raised if we attempt to add new fields:
:tags: [raises-exception, output_scroll]
tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True, "newKey": 25})
To set the top-level metadata, just assign it. Validation and encoding happen as specified by the top-level metadata schema
tables.metadata_schema = basic_schema # Allows new fields to be added that are not validated
tables.metadata = {"mean_coverage": 200.5}
print(tables.metadata)
:::{note}
Provenance information, detailing the origin of the data, modification timestamps,
and (ideally) how the tree sequence can be reconstructed, should go in
{ref}sec_provenance, not metadata.
:::
To modify a schema --- for example to add a key --- first get the dict representation, modify, then write back:
schema_dict = tables.individuals.metadata_schema.schema
schema_dict["properties"]["newKey"] = {"type": "integer"}
tables.individuals.metadata_schema = tskit.MetadataSchema(schema_dict)
# Now this will work:
new_id = tables.individuals.add_row(metadata={'accession': 'abc123', 'pcr': False, 'newKey': 25})
print(tables.individuals[new_id].metadata)
To modify the metadata of rows in tables use the {ref}sec_tutorial_metadata_bulk.
(sec_tutorial_metadata_viewing_raw)=
If you need to see the raw (i.e. bytes) metadata, you just need to remove the schema, for instance:
individual_table = tables.individuals.copy() # don't change the original tables.individual
print("Metadata:\n", individual_table[0].metadata)
individual_table.metadata_schema = tskit.MetadataSchema(None)
print("\nRaw metadata:\n", individual_table[0].metadata)
(sec_tutorial_metadata_bulk)=
In the interests of efficiency each table's {meth}~NodeTable.packset_metadata method,
as well as the more general {meth}~NodeTable.set_columns and
{meth}~NodeTable.append_columns methods, do not attempt to validate or encode metadata.
You can call {meth}MetadataSchema.validate_and_encode_row directly to prepare metadata
for these methods:
metadata_column = [
{"accession": "etho1234", "pcr": True},
{"accession": "richard1235", "pcr": False},
{"accession": "albert1236", "pcr": True},
]
encoded_metadata_column = [
tables.individuals.metadata_schema.validate_and_encode_row(r) for r in metadata_column
]
md, md_offset = tskit.pack_bytes(encoded_metadata_column)
tables.individuals.set_columns(flags=[0, 0, 0], metadata=md, metadata_offset=md_offset)
tables.individuals
Or if all columns do not need to be set:
tables.individuals.packset_metadata(
[tables.individuals.metadata_schema.validate_and_encode_row(r) for r in metadata_column]
)
(sec_tutorial_metadata_binary)=
To disable the validation and encoding of metadata and store raw bytes pass None to
{class}MetadataSchema
tables.populations.metadata_schema = tskit.MetadataSchema(None)
tables.populations.add_row(metadata=b"SOME CUSTOM BYTES #!@")
print(tables.populations[0].metadata)