Data Inheritance: A cool concept on paper that we should not do in practice

[apirogov]

Metador supports schema inheritance. With the current approach, you can have multiple, possibly "conflicting" instances of related schemas and the only promise we do is that a child schema instance can be projected into a less informative parent schema - but there is no enforced relationship between two "sibling" objects.

This means that, while Metador organizes metadata objects or related schemas in a hierarchical way, it does not "deduplicate" redundancy of metadata sharing common parent schemas, it stores the objects "as is", they are not considered to be different "views" / projections of the same metadata object.

You can attach both a core.file and a core.imagefile to the same node, with completely different contents, and you cannot assume anything about how the stored objects relate to each other.

There would be another way to model metadata stored at a container node, which would interpret schema inheritance stronger - as data inheritance. In this interpretation, child schemas could "extend" the metadata stored at a node, in a sense then each node would have one metadata object and each schema is merely a projection of it.

To make "data inheritance" possible, i.e. be able to decompose schemas into common and special "parts", we need to enforce invariants mentioned in #7 for each node and make sure that:

all metadata objects attached to a node that share an attribute name with a common parent must agree on the value

This has possibly unintended consequences:

Implication 1: all related schemas used at the same node inheriting an attribute from a parent must have a non-empty intersection of allowed values for that attribute (and the stored data must have a value in that intersection)

Implication 2: adding one kind of metadata object can make it impossible to validly add another one (if they share inherited fields and augment the parent in incompatible ways)

If we would disallow "narrowing" types in children, we would still run into this problem when two schemas add the same field with different incompatible types - these could never be used at the same node.

Summary:

• The unintended consequences are way too complicated and incomprehensible to a normal user
• This is imposing very harsh constraints on stored metadata that immediately sound impractical

Conclusion:

Metador will not add data inheritance / automatic decomposition / deduplication to the stored metadata, so it will be the responsibility of the container creators to make sure that objects of related schemas are agreeing on common fields and attaching metadata in sensible ways.

[apirogov]

This was something I thought about after hearing about how metadata is handled in BIDS, where the project essentially has data inheritance (metadata on a higher directory level is inherited down to subdirectories to reduce redundancy).

The result translating this to metador was this outlined data inheritance concept, which naively could sound like a nice thing to have, but it actually isn't.