Skip to content

Latest commit

 

History

History
82 lines (51 loc) · 2.85 KB

executor.md

File metadata and controls

82 lines (51 loc) · 2.85 KB
Raw

Ingestion Executor

Certified

Overview

This Action executes ingestion recipes that are configured via the UI.

Capabilities

  • Executing datahub ingest command in a sub-process when an Execution Request command is received from DataHub. (Scheduled or manual ingestion run)
  • Resolving secrets within an ingestion recipe from DataHub
  • Reporting ingestion execution status to DataHub

Supported Events

  • MetadataChangeLogEvent_v1

Specifically, changes to the dataHubExecutionRequestInput and dataHubExecutionRequestSignal aspects of the dataHubExecutionRequest entity are required.

Action Quickstart

Prerequisites

DataHub Privileges

This action must be executed as a privileged DataHub user (e.g. using Personal Access Tokens). Specifically, the user must have the Manage Secrets Platform Privilege, which allows for retrieval of decrypted secrets for injection into an ingestion recipe.

An access token generated from a privileged account must be configured in the datahub configuration block of the YAML configuration, as shown in the example below.

Connecting to Ingestion Sources

In order for ingestion to run successfully, the process running the Actions must have network connectivity to any source systems that are required for ingestion.

For example, if the ingestion recipe is pulling from an internal DBMS, the actions container must be able to resolve & connect to that DBMS system for the ingestion command to run successfully.

Install the Plugin(s)

Run the following commands to install the relevant action plugin(s):

pip install 'acryl-datahub-actions[executor]'

Configure the Action Config

Use the following config(s) to get started with this Action.

name: "pipeline-name"
source:
  # source configs
action:
  type: "executor"
# Requires DataHub API configurations to report to DataHub
datahub:
  server: "${DATAHUB_GMS_PROTOCOL:-http}://${DATAHUB_GMS_HOST:-localhost}:${DATAHUB_GMS_PORT:-8080}"
  # token: <token> # Must have "Manage Secrets" privilege
View All Configuration Options
Field Required Default Description
executor_id ❌ default An executor ID assigned to the executor. This can be used to manage multiple distinct executors.

Troubleshooting

Quitting the Actions Framework

Currently, when you quit the Actions framework, any in-flight ingestion processing will continue to execute as a subprocess on your system. This means that there may be "orphaned" processes which are never marked as "Succeeded" or "Failed" in the UI, even though they may have completed.

To address this, simply "Cancel" the ingestion source on the UI once you've restarted the Ingestion Executor action.