Evaluate indexing strategies in Elasticsearch for multiple classifications

[fmendezh]

Evaluate different approaches to index multiple taxonomies: multiples indices vs adding nested elements for multiple classifications.

[djtfmartin]

So far work has only focussed on adding nested elements to the elastic index for multiple classifications.

An important requirement is to be able search by taxonKey for a specified checklistKey. This requires a compound query and needs to be accurate to avoid a match on taxonKey and checklistKey, where the taxonKey is in one subobject (classification), and the checklistKey is in another subobject (classification) in the save index elastic document.

Currently considering 4 potential strategies:

• Nested objects for storing multiple classifications (current approach). The potential downside of this is nested objects are indexed as separate documents, hence querying them can be more expensive. This needs to be tested at volume.
• Dynamic objects for storing multiple classifications
• Encode checklistKey with taxonKeys - making a compound key and hence a unique taxonKey across checklists
• Make GUIDs mandatory for taxonKeys to avoid clashes (potentially provided by matching-ws services
  • CoL identifiers need a sensible prefix for indexing to avoid potential clashes
  • Legacy GBIF identifiers need a sensible prefix for indexing to avoid potential clashes
  • other sources such as WoRMS are ok