[Mothership PR] MVP of AIP-69 - A experimental usable version of Edge Executor #40900
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:dev-tools
area:Executors-core
jscheffl
force-pushed
the
feature/aip-69-remote-executor-mvp
branch
from
a10f33c
to
91dff32
Compare
jscheffl
force-pushed
the
feature/aip-69-remote-executor-mvp
branch
4 times, most recently
from
a260cf6
to
120710a
Compare
jscheffl
force-pushed
the
feature/aip-69-remote-executor-mvp
branch
12 times, most recently
from
3f97faa
to
c2f2700
Compare
jscheffl
force-pushed
the
feature/aip-69-remote-executor-mvp
branch
7 times, most recently
from
1959171
to
60d77e7
Compare
(cherry picked from commit c2672ed2ac356de6f925f1be9474d4fba678ae48)
jscheffl
force-pushed
the
feature/aip-69-remote-executor-mvp
branch
from
d5c7cf9
to
5495dad
Compare
|
All detailled split PRs have been merged, so this is not needed anymore. All features are now on main |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a DRAFT to further get to a MVP release of AIP-69. DO NOT MERGE (yet) - but please feedback on it if you like.
It is a continuation of PR #40224 which was a kind of PoC and step-by-step more stuff came onto this on top.
This is a kind of "Mothership" PR where all contributions are collected. Small parts will be separately opened as PR to get on main and finally I'd split this PR up into smaller pieces for easier review. In this PR all parts and work is collected.
Please treat this as experimental as noted in the docs, it is not expected to be rock-solid but a first usable version. It can be used to distribute the first task. Status of implementation is tracked in
docs/apache-airflow-providers-edge/edge_executor.rstbelow headline "Feature Backlog of MVP to Release Readiness" in this PR.If you want to test it you can:
breeze start-airflow --load-example-dags --backend postgres --executor EdgeExecutorexample_python_operatoror theIntegration Testwith a bit more complexity (this integration test has been moved toproviders/src/airflow/providers/edge/example_dags/integration_test.py). Wohoo!Based on progress and time I will sometimes re-base and push changes to this branch. But it will not be something that will be review-able.
I will not add any migration logic in case of updates for the additional tables in DB Scheme. If I update the draft PR and DB schema changes, DB might need to be reset. Of course migration support would need to be added before a "real" release.