Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have integration points for AIP-69 in Internal API #40901

Merged
merged 6 commits into from
Jul 21, 2024
Merged

Have integration points for AIP-69 in Internal API #40901

merged 6 commits into from
Jul 21, 2024

Conversation

jscheffl
Copy link
Contributor

@jscheffl jscheffl commented Jul 20, 2024 •
edited
Loading

In order to leverage the AIP-44 interface for AIP-69 / Remote Executor I need some small adjustments to "hook into" the RPC API. This PR extracts from PoC implementation of AIP-69 (#40900) the needed internal API adjustments:

  • The private cached method map _initialize_map() needs to be extended, not to replicate the function list I'd like to remove the private tag to extend it with a few methods in remote executor. Would not add it to be a "public" interface though.
  • Allowing to use a different API path from the standard internal API URL. For the remote case we would expose the URL to another (plugin hosted or separately operated) path, such a URL with path should be configurable as API endpoint
  • For the setup of the CLI for Remote worker a way to force usage of internal API as in the remote case we would configure other config.
  • Allow the (Remote Provider Package) to hook-in additional classes for serialization to Pydantic mapping for extended API communication via RPC

@jscheffl jscheffl added this to the Airflow 2.10.0 milestone Jul 20, 2024
@jscheffl jscheffl marked this pull request as ready for review July 20, 2024 12:57
@potiuk potiuk force-pushed the feature/have-integration-points-for-aip-69-in-internal-api branch from 39ff6f6 to e8291d3 Compare July 20, 2024 14:50
@potiuk
Copy link
Member

potiuk commented Jul 20, 2024

I rebased it after I fixed selective checks in #40904

potiuk
potiuk approved these changes Jul 20, 2024
View reviewed changes
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question.

@jscheffl jscheffl merged commit 6745cb8 into apache:main Jul 21, 2024
48 checks passed
@ephraimbuddy ephraimbuddy added AIP-44 Airflow Internal API changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) labels Jul 22, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@jscheffl @romsharon98
Have integration points for AIP-69 in Internal API (apache#40901)
d82d846 
* Have integration points for AIP-69 in Internal API

* Allow adding custom classes into serialization mapping

* Have integration points for AIP-69 in Internal API

* Fix pytest, reverse validation check for scheme

* Have integration points for AIP-69 in Internal API

* Review feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potiuk potiuk potiuk approved these changes

@hussein-awala hussein-awala hussein-awala approved these changes

@dstandish dstandish Awaiting requested review from dstandish

@vincbeck vincbeck Awaiting requested review from vincbeck

@kaxil kaxil Awaiting requested review from kaxil

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

Assignees
No one assigned
Labels
AIP-44 Airflow Internal API changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)
Projects
No open projects
AIP-44 - Internal API
Status: ✅ Done
Milestone
Airflow 2.10.0
Development

Successfully merging this pull request may close these issues.
4 participants
@jscheffl @potiuk @hussein-awala @ephraimbuddy