Enrich publication records - strategy

[nicolasfranck]

During indexation of publications in the biblio frontend, every publication record is enriched with data from JCR and ECOOM, by doing a lookup in the mongodb database.

In our effort to move these kind of enrichments from the frontoffice to the backoffice, we invented a new field
called external_fields where this data may be stored.

But when should this data be added?

• one can already import the data via the batch method external_fields.set. But that requires a whole setup to fetch records enrich them and send them back. And even then, it is not clear when to do this again. e.g. <id>,external_fields.set,"ecoom-bof-css",1
• create a "save hook", so that this data is added automatically. This means that we need to save every public publication record again to make this work for existing records. And here again: is this necessary every time you save the record?
• create a cronjob in nomad that adds these fields. Problem here is that the record can cause two updates in the frontend: one without the enrichments (before the job), and one with the enrichments (after the job run). Also this job should have some state to change publications incrementally (some state file to store the latest date_updated?) instead of changing all of them on every run.

Also, these enrichments only happen when publication record is updated, not when new enrichment data becomes available. With NATS? That would maybe complicate things, as there are two streams (from backoffice to frontend, and from nats) that may miss each other (a jcr record has an issn for something not yet available in backoffice, but the message needs to be skipped, and can never be fetched again).

I would vote for a save hook.