Hacking problems

[kjappelbaum]

Motivation: Make some place to already start write some code to start something that can last for after the workshop

1. Write a Python tool that automatically matches @petermr's digitized datasets to structures (and spectra) on materialsproject (and the uses optimade to find structures in other datasets). Big picture: make a dashboard that compiles all related information.
2. Publish a data dictionary similar to https://github.com/marda-dd/hello-world/tree/main/dwinston
3. How can we get enough information from @petermr's datasets to automatically start a simulation (e.g., using AiiDAlab - tagging @yakutovicha in case you have an idea of how we can bring in AiIDAlab or our ELN-AiiDAlab "integration" --- what would we need to do to make it more general?)

[kjappelbaum]

might also be a useful resource https://github.com/linkml/linkml-model-template

[ml-evs]

As discussed on Discord, potentially reaching out to developers of some web-based XAS tools like https://times-webxrs.stanford.edu/ and seeing if we can wrap their code for interop

[ml-evs]

Another vague idea:

• investigate existing tools for opening, plotting & exploring data "archives" that consist of a (collection of) JSON file(s), and a JSONSchema (as a example technology). Such files could also link to e.g. JCAMP-DX files containing spectra
  • investigate linking to semantics withing JSONSchema (see how LinkML does it, also DLITE and other semantic efforts). How can the viewers be made semantic-aware? e.g. rendering the data in particular way if it believes a given resource corresponds to a sample with a linked spectrum file.
  • Would a dataset extracted from the literature by @petermr (will email separately) be a good starting point?
  • This is in some sense a generalisation of https://github.com/IUPAC/IUPAC-FAIRSpec --- we could consider just extending it directly, but should wait for the FAIRSpec talk first :)
  • Finally, could make a simple web app that takes JCAMP files, allows you to enter metadata or labels to corresponding samples, then spits out a JSON file that follows FAIRSpec/otherwise

[ml-evs]

An x-ray oriented project could be a similar investigation of the XrayDB: https://github.com/xraypy/XrayDB

[kjappelbaum]

thats interesting, we just manually compiled a subset of this data for the ELN. Would be great to have a comprehensive, open, interoperable solution (unfortunately, the author of the library didn't have time to join the workshop - similar to the WebXAS folks)

[ml-evs]

They also have this file format https://github.com/XraySpectroscopy/XAS-Data-Interchange under a slightly different banner. It's a shame they couldn't join, they have lots of great resources online!

[ml-evs]

(unfortunately, the author of the library didn't have time to join the workshop - similar to the WebXAS folks)

Did they show any interest in us working on/with their code?

[kjappelbaum]

no response at all

[ml-evs]

FYI we will have a "hacking room" on Gather (see emails/Discord for links) that will run throughout the week.

[ml-evs]

Hacking idea from the last breakout (typing on my phone but can tidy up if there is interest). We discussed how parsing unstructured logs/output files is still very common. Everyone's writes their own parser that maps to their own data models. Usually these are bundled as part of larger packages that do the analysis (e.g. pymatgen, Ase, aiida in comp. mat. sci), and rewritten in multiple languages (e.g. the modular cheminfo parsers in JS).

Do we think a simple registry/framework for code objects that operate on files and return structured data would be a useful investment? e.g., a docker image per parser with a unified interface that also spits out a schema for the parsed data? Do such things already exist? Does this go any way to tackling the scalability of our current ecosystem, or is this just creating more laborious work? This could then motivate the development by the original raw file creators, like instrument manufacturers and code authors.

These could then be employed across multiple ELN/repository services and used for ETL in perhaps a more scalable way than is currently available. Given the wealth of existing parsers it would be easy to test this out quite quickly, and there is potential for nice integration with many existing services present at the workshop.

[ml-evs]

Following from my penultimate comment, also without bundling all the dependencies of the larger libraries, which may be an issue when deploying as part of already complex services.

Here is one such collection of parsers/drivers: https://github.com/CINF/PyExpLabSys

[kjappelbaum]

This is also quite similar to what Damien Jeannerat proposed all the time as "archive forge"

[kjappelbaum]

what do you spit out with this service? json with original key value pairs and one annotated with uris where they exist? (we do this a bit in cheminfo types where we always return some concepts that are in the gold book and put all the other manufacturer specific things into a misc)

[blaiszik]

Thanks I had missed this thread. We have an effort called MaterialsIO that is highly relevant here. Our concept is to have extractors that extract all of the metadata we can from a given format, and adapters that covert that metadata to specific formats. https://github.com/materials-data-facility/MaterialsIO

[ml-evs]

Thanks I had missed this thread. We have an effort called MaterialsIO that is highly relevant here. Our concept is to have extractors that extract all of the metadata we can from a given format, and adapters that covert that metadata to specific formats. https://github.com/materials-data-facility/MaterialsIO

This looks very much along the lines of what I had in mind for a Python API regarding the base classes, adapter, plugins, with the only additions being schemas defined for each parser (to aid adapter creation) and a more distributed registry of parsers. Is MaterialsIO still under active development and in use at MDF? David Elbert indicated that this could be a good MaRDA working group, but perhaps the work has already been done?

I think the containerisation/registry aspect of our original idea above would be quite useful for people running infrastructure, and perhaps MaterialsIO could be the first major library to wrap. I guess the main drawback of the idea is that it assumes all parsing is server-side, which is definitely not the case in JS projects like cheminfo, so containers would have to be actively deployed with web APIs for use in those projects.

[petermr]

10+ years ago I wrote JUMBO-Converters as a declarative approach to parsing. Was delighted to see Carlo had extended this. Declarative parsers can be implemented by a small amount of code and maintained by people with marginal programming skills. Be happy to chat about this.

[petermr]

WOW!!!
Incredible.

[ltalirz]

Very nice indeed!

Is there a way to use these logfile converters from python as well? So far I was just aware of the cclib, which can also output CML (albeit quite a bit more limited, see here for all parsed quantities).

I guess the focus of the two libraries is somewhat different, but there is a significant overlap in the codes supported (ADF, Gaussian, Molcas, MOPAC, Orca, Turbomole) which makes me wonder whether there would be an opportunity to share some of the burden of maintaining these parsers?

[petermr]

The code is declarative - different XML templates for each output with a generic Java parser. I think it would be a lot easier with modern Python. With a dictionary mapping lines to symbols we can probably leverage regexp parsers to do the hard stuff. It's probably easier to autogenerate a lot of the templates. Some outputs like Gaussian are written by so many different authors that it's almost formally impossible to write a parser .

I certainly think the work can be distributed.

[carlesbo]

Very nice indeed!

Thanks

Is there a way to use these logfile converters from python as well?

ioChem-BD is java code, and its API is python (also shell) friendly. We aim at having time and fingers soon to provide a Jupyther notebook environment as well, but probably this will be limited to local instances only, not on the open-public instance hosted by BSC since computing resources requirements.

So far I was just aware of the cclib, which [can also output CML
(https://cclib.github.io/how_to_parse.html#ccwrite) (albeit quite a bit more limited, see [here]
(https://cclib.github.io/data.html#details-of-current-implementation) for all parsed quantities).

Yes. Some new computational codes produce CML output directly (Gronor for instance)

I guess the focus of the two libraries is somewhat different, but there is a significant overlap in the codes supported (ADF, Gaussian, Molcas, MOPAC, Orca, Turbomole) which makes me wonder whether there would be an opportunity to share some of the burden of maintaining these parsers?

That would be great.
From our side, writing new parsers is not well documented yet, but the code is open-source.
We'd love to get volunteers involved, but this is a never-ending task, we know.

Some software companies are providing open-source libraries for processing their own binary format files, which is not a bad approach ffor us. That would allow extracting validated data and get rid of plane text output files. Also, MolSSI efforts in defining schemas are worthy, so we need to converge somewhere sooner or later.

[carlesbo]

The code is declarative - different XML templates for each output with a generic Java parser.

Yes

I think it would be a lot easier with modern Python. With a dictionary mapping lines to symbols we can probably leverage regexp parsers to do the hard stuff. It's probably easier to autogenerate a lot of the templates. Some outputs like Gaussian are written by so many different authors that it's almost formally impossible to write a parser .

At least, Gaussian is quite stable format, but adding new properties needs coding. ADF (now AMS ecosystem) is getting complex but their open-source libraries might allow parsing their binary files easily.

[petermr]

This is all wonderful.

One of the key features of JUMBO-Converters is that it (hopefully) chunks the output hierarchically into nested objects. (This can be hard with procedural code - you have to "read-to-next-section" . And J-C finds all the instances of subobjects (e.g. matrices) without you having to anticipate them.
The things that are hard are:

• no terminators to chunks
• random output , e.g. from system calls ("used 1234567 bytes of memory")
• numbers without words.

First, this makes me very happy! I will wear my JUMBO t-shirt today.
I'd also be happy to have a telcon to find out where this has developed to and what the remaining problems are.

J-C is a very good way of identifying the sub-objects that could/should be in data-dictionaries and ontologies.