Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cross-back-end process graph processing #150

Closed
JeroenVerstraelen opened this issue Aug 20, 2024 · 5 comments
Closed

cross-back-end process graph processing #150

JeroenVerstraelen opened this issue Aug 20, 2024 · 5 comments
Assignees

Comments

@JeroenVerstraelen
Copy link
Contributor

Follow up on #95 and #53. Develop a cross-back end process graph splitting implementation that is useful in the current aggregator setup. It should be based on the algorithm developed during the feasibility study.

@soxofaan
Copy link
Member

It should be based on the algorithm developed during the feasibility study.

@JeroenVerstraelen can you point me to the details on this feasibility study?

soxofaan added a commit that referenced this issue Sep 13, 2024
soxofaan added a commit that referenced this issue Sep 16, 2024
decouple "load_collection" detection from subjob creation loop
soxofaan added a commit that referenced this issue Sep 16, 2024
… yielding

Introduce ProcessGraphSplitterInterface, with first LoadCollectionGraphSplitter implementation based on existing simple graph splitting logic
soxofaan added a commit that referenced this issue Sep 16, 2024
soxofaan added a commit that referenced this issue Sep 17, 2024
soxofaan added a commit that referenced this issue Sep 17, 2024
decouple "load_collection" detection from subjob creation loop
soxofaan added a commit that referenced this issue Sep 17, 2024
… yielding

Introduce ProcessGraphSplitterInterface, with first LoadCollectionGraphSplitter implementation based on existing simple graph splitting logic
soxofaan added a commit that referenced this issue Sep 18, 2024
also make graph walking more deterministic (e.g. to simplify test asserts)
soxofaan added a commit that referenced this issue Sep 18, 2024
Instead of dictionary

Also move type coercion logic to _FrozenNode __init__ to eliminate boilerplate code
soxofaan added a commit that referenced this issue Sep 18, 2024
also add consistency check of  _GraphViewer node map
soxofaan added a commit that referenced this issue Sep 18, 2024
soxofaan added a commit that referenced this issue Sep 19, 2024
…reate_crossbackend_job

and integrate it in test coverage of TestCrossBackendSplitting
soxofaan added a commit that referenced this issue Sep 19, 2024
soxofaan added a commit that referenced this issue Sep 19, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
decouple "load_collection" detection from subjob creation loop
soxofaan added a commit that referenced this issue Sep 20, 2024
… yielding

Introduce ProcessGraphSplitterInterface, with first LoadCollectionGraphSplitter implementation based on existing simple graph splitting logic
soxofaan added a commit that referenced this issue Sep 20, 2024
also make graph walking more deterministic (e.g. to simplify test asserts)
soxofaan added a commit that referenced this issue Sep 20, 2024
Instead of dictionary

Also move type coercion logic to _FrozenNode __init__ to eliminate boilerplate code
soxofaan added a commit that referenced this issue Sep 20, 2024
also add consistency check of  _GraphViewer node map
soxofaan added a commit that referenced this issue Sep 20, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
…reate_crossbackend_job

and integrate it in test coverage of TestCrossBackendSplitting
soxofaan added a commit that referenced this issue Sep 20, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
soxofaan added a commit that referenced this issue Sep 20, 2024
@soxofaan
Copy link
Member

There are still some todo's and plenty of idea's for future work, but I decided to merge the current PoC with 9b65520

@soxofaan
Copy link
Member

soxofaan commented Sep 20, 2024

As discussed in #156 the idea is currently to trigger this feature with a job option like

job_options={
    "split_strategy": {
        "crossbackend": {
            "method": "deep"
        },
    }
}

@soxofaan
Copy link
Member

soxofaan commented Sep 23, 2024

Follow up todo's and tasks (WIP)

@soxofaan
Copy link
Member

soxofaan commented Sep 27, 2024

made an initial demo notebook at
https://gist.github.com/soxofaan/c374928c32212031eb1518c1260eb954

unfortunately the job metadata visualization (including process graph viz) does not work in that online notebook viewer, so the explanation might not always make sense

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants