Platform-agnostic interoperability

[edan-bainglass]

As mentioned here and there during the pre-event meetings, the PREMISE project (partly funding this workshop) is focused on facilitating interoperability of open research data from experiments and simulations. Internally, we demonstrate our work via interoperability between the openBIS ELN and the AiiDA workflow management system (WFMS). However, we aim for a more generic, platform-agnostic data exchange.

Recent internal discussions resulted in the proposal to switch from platform-specific APIs to the use of platform-agnostic, semantically annotated transactional "crates" (hint hint) shipped from and received at platform-specific ports equipped with a set of tools to unpack and comprehend deliveries. The idea is to move from each platform implementing an API per interoperability partner to a single API per platform to handle deliveries, the requirements of which are to be discussed.

RDF seems a fitting standard for semantics. Though several RDF serializations exist, for the workshop, we propose its JSON-LD serialization for the following two reasons:

1. Several experts joining us at MADICES work with it extensively and may provide guidance
2. The RO-Crate specification (and ELN file format by extension) are based in JSON-LD

As such, I propose the following two working groups:

1. A main group focused on exploring and (ideally) implementing a minimal proof-of-concept demonstrator
2. A smaller group focused on expanding the above demonstrator to other serializations/tools

Suggestions for group members will follow shortly. For now, let's see if we can get a bit of discussion going on the topic.

[s4b7r]

Extend the "one interface-concept" beyond the ELN/LIMS <-> WFMS interface

I would like to extend the view on this proposal beyond ELN/LIMS <-> WFMS. I will borrow parts from @edan-bainglass and @ml-evs for this, to bring different parts into this discussion. Basically, I fully second...

The idea is to move from each platform implementing an API per interoperability partner to a single API per platform to handle deliveries

... and suggest extending that idea to connections between other parts as well.

I am currently working on multiple (sub-)projects, mostly PMD-related [LINK], which together can be sketched into Matthew's "Interoperability hierarchy" like this:

I am leaving out the blocks, that I currently don't have a clear link to, but don't think they are generally excluded from this idea.

ELN/LIMS <-> WFMS

The general idea brought up here from Edan / PREMISE starts with the connections (both ways) between ELN/LIMS and WFMS. For this part I just want to express agreement on the need for development from the "one interface per partner"-concept (which eventually equals implementing N^2 interfaces) to the "one interface to something standard"-concept (which eventually could approximate implementing N interfaces).

We are currently working on integrating some other workflow manager with different ELNs (or RDM tools in general). I currently think about merging the one interface-concept into that efforts as well and would like to discuss that in the workshop - I will collect some info from my colleagues to try to channel their developments into the workshop discussions, if that makes sense to you. (@edan-bainglass )

Meta-/ Data Extractors

The other interface I'd like to extend this perspective to is that between observations and WFMS+ELN/LIMS, which I mainly see in the form of Meta-/ Data Extractors. I would like to discuss merging ideas and developments from the MaRDA meta-data extractors working group and different projects in MaterialDigital. (@ml-evs )

(By the way: Taking the one interface-concept into consideration this boils down to: an interface "out of" Extractors. This is then no longer bound to mean the interface between observation and WFMS+ELN/LIMS but can interface to anything that wants to connect to Extractors.)

Some connection points here can be the data acquisition step in DiProMag's (one of my research project) "automated research workflows", some developments from Mat-o-Lab for transforming data [1][2], and the canonicalization step in PMD's Data Acquisition Pipeline.

To give some context, in DiProMag we build a pipeline to automate the steps from data source (in priciple either experimental or simulational), to structured data annotation, to (post-)processing, to knowledge graph. Mainly the first transformation step of this (acquisition & canonicalization) can play into the meta-/data extractors idea. I am currently working on having this in a state that is deployable for others in this workshop and will link to it, once it is published in that form. (I hope this will be ready for next week.)

Similar ideas regarding the acquisiton/extraction/canonicalization part can be identified in Mat-o-Lab, where also ckan is involved as a user interface one could say [3][4] (besides other uses ckan has there).

Particularly for this, I propose a working group that focuses on integrating the MaRDA meta-data extractor registry and API concepts into "pipelines" like the two roughly sketched concepts above (this can be one of them, I suggested, or one suggested by others, of course). This should demonstrate the benefits and feasability of having a "central" extractor registry and API specification to ease reusing already developed extractors.

This should play well together with the working groups proposed by @edan-bainglass .

[edan-bainglass]

@s4b7r we are very much on the same page. Your first sketch is precisely the use case of PREMISE (starting point and all). With respect to the MaRDA extractor system, I'm not sure I'm clear on how you want to use them for this purpose. Perhaps you can clarify this?

Update

Okay, I see where the extractors fit in the sketch. Missed it the first time around.