Hackathon 2023-02 topic board

[christian-monch]

The 2023-02 metalad hackathon

The goal of this hackathon is three-fold:

• help the team to familiarize itself with metalad,
• identify shortcomings, feature requests, and bugs 🐞 (if there are any)
• work with the code-base: improve it, profile it, implement new features, etc.

Docs

Documentation can be found in a few different places. First, in the datalad handbook, and in the --help messages of the individual datalad-metalad commands. There is also an first steps gist here: MetaLad: first steps

Issues

The following issues are on the shortlist for the metalad 2023-02 hackathon

• issue Do we need a meta-search? #331
• Issue Create a reproducable performance measurement dataset #330
• Issue DOC: keep track of usage patterns, translate into useful instructions, add to docs #329
• Issue ENH: update metalad-shipped extractors to use new interface #328 (@jsheunis )
• Issue Extend meta-dump's path-parameter to include metadata format as a selection criteria #318
• Issue Should we ensure items are "clean" before adding metadata? #291 (@adswa is taking a look)
• Issue Add a meta-delete command #237 (@mslw is taking a look)
• Issue Export/import to/from file system-based metadata representation #216
• Issue Migration tool from metalad <= 0.2.1 metadata to metalad >= 0.3.0 metadata #118
• Issue extractors and their records should be identified by an extractor UUID and not name #67

Added 2023/02/25

• Issue DOC: Add the internal data model of metadata #315
• Issue Q: Should under-the-hood file retrieval get an INFO message? #327

Added 2023/02/28

• Issue Add a way to list all installed metalad extractors, filter, processors, pipelines, etc. #340
• Issue Add "metalad first steps" gist to the docs #347 (@mslw)

Feel free to suggest other issues or topics (create new issues) on which you would like to work.

[christian-monch]

As a result of our discussion in the morning, we would like to highlight that currently the file-level extractor paradigm leads to a large number of per-extraction subprocess invocations when running meta-conduct. There is an issue #268 and a related branch #298

[asmacdo]

ENH: Add "container extractor": datalad/datalad-container#198

[christian-monch]

Hackathon summary

The hackathon was held on 28th of February from 9:00 CET to 22:00 CET with four participants from Europe and three participants from the USA.

The desired results for this hackathon were:

• help the team to familiarize itself with metalad
• identify shortcomings, feature requests, and bugs
• work with the code-base: improve it, profile it, implement new features, etc.

The hackathon provided progress in all of those items.

The team was introduced to metalad in two interactive demo-sessions (9:00 and 14:00) which lead to a number of discussions and questions. That provided a good basic understanding of metalad. Following those introductions, the team worked with the code base and its documentation.

During the hackathon the team worked in the areas documentation, improvement identification and implementation, and extractor implementation. Generally it seems that the team has gained a good understanding of metalad and its basic usage, and is able to identify its utility for new and existing projects.

Detailed outcome

Documentation

• Issue DOC: More user-friendly README restructuring #338 and PR DOC: Make a history page from the README #339

• Issue Add "metalad first steps" gist to the docs #347 and PR Convert "Metalad first steps" gist to a docs chapter #352

• Issue update contributors information #344

• Issue Have a minimal CONTRIBUTING.md file #343

• Issue DOCS: Badly need meta-conduct docs and examples #341

Improvement identification and implementation:

• issue meta-extract: Catch non-existing paths earlier #354 and PR BF: Raise early and informative when extracting from a non-existing file #355

• issue meta-add allows adding file-level metadata with non-existent path #350

• issue logger setup is different from the core setup of the overall datalad. #348

• issue Better errors  #346

• issue MNT: Diverging software requirements in setup.cfg and requirements*.txt #345

• Issue Add a way to list all installed metalad extractors, filter, processors, pipelines, etc. #340

Extractor implementation:

• ENH: Add "container metadata extractor" datalad-container#198