Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] enable union backend #439

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

parkan
Copy link
Contributor

@parkan parkan commented Jul 10, 2024

this leverages built-in rclone features to allow backends that have one logical root (e.g. internetarchive) to seamlessly aggregate multiple paths into a single preparation

example usage:

RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''

this creates an implicit, ephemeral remote of the internetarchive type called ia and aggregates paths specified by --upstreams

this leverages built-in rclone features to allow backends that have one logical root (e.g. `internetarchive`) to seamlessly aggregate multiple paths into a single preparation

example usage:
```
RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''
```

this creates an implicit, ephemeral remote of the `internetarchive` type called `ia`  and aggregates paths specified by `--upstreams`
@parkan
Copy link
Contributor Author

parkan commented Jul 10, 2024

why this is a DRAFT: the functionality works as designed but is somewhat specific to the internetarchive remote type as it relies on the implicit creation of a "root" remote (RCLONE_CONFIG_IA_TYPE=internetarchive) on which the upstreams build

for other backend types this becomes potentially non-trivial as the entire remote configuration (auth etc) needs to be specified in env vars (RCLONE_CONFIG_*) and furthermore unions across heterogeneous remote types or other services are untested

all that being said we would really like to see this merged if possible as it is required for SPs to consume preparations created with union sources, to the best of my understanding

the main impediment in my mind is the docstring, which is currently automagically populated from the rclone library; amending it with clarifying comments regarding the above limitations may be sufficient to greenlight this, depending on the maintainer's views, but I'm not entirely sure how to do that

@ianconsolata
Copy link
Collaborator

I believe the commits needed to fix the tests can all be pulled from my fork: https://github.com/ianconsolata/singularity/commits/main/. Unfortunately, without maintainer rights I can't update this PR myself.

@Sankara-Jefferson Sankara-Jefferson merged commit a9d3438 into data-preservation-programs:main Jul 10, 2024
2 of 9 checks passed
Sankara-Jefferson pushed a commit that referenced this pull request Jul 10, 2024
These are some additional changes I made on a separate fork that cover
the errors introduced in #439, which should make the main build green
again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants