S3 Remote Logging not working for KubernetesExecutor on AWS EKS

[seblum]

I am encountering an issue with remote logging to S3. I am running Airflow on an EKS cluster with KubernetesExecutor and KubernetesPodOperator. I digged through the documentation already but nothing worked so far.

The KubernetesOperator in my DAG looks like this:

    @task.kubernetes(
        image=<IMAGE>,
        name="preprocessing",
        namespace="airflow",
        in_cluster=True,
        get_logs=True,
        do_xcom_push=True,
        service_account_name="airflow-sa",
    )
    def preprocessing_op(mlflow_experiment_id):
    ...

When I run my DAG I get the following logs on the UI:

As soon as I let it fail ( in hope it stores the logs on S3 ), I get the following:

I am using the community helm chart version 8.7.1. as base. The following shows my settings.

airflow:
  users:
    - username: admin
      password: admin
      role: Admin
      email: admin@example.com
      firstName: admin
      lastName: admin
    - username: user
      password: user123
      role:
        - User
        - Viewer
      email: user@example.com
      firstName: user
      lastName: user
  usersUpdate: true
  legacyCommands: false
  image:
    repository: <my-own>
    tag: 2.6.2-python3.11
    pullPolicy: IfNotPresent
    pullSecret: ""
    uid: 50000
    gid: 0
  executor: KubernetesExecutor
  fernetKey: "<xyz>"

  config:
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
    AIRFLOW__LOGGING__LOGGING_LEVEL: "DEBUG" 
    AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
    AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://airflow-log-storage/"
    AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "AWSS3_Log_Storage"

  connections:
    - id: AWSS3_Log_Storage
      type: aws
      description: AWS connection to store logs on S3
      extra: |
        {
          "region_name": "eu-central-1"
        }
serviceAccount:
  create: true
  name: "airflow-sa"
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<123456>:role/airflow-s3-access-role"
workers:
  enabled: false
flower:
  enabled: false
postgresql:
  enabled: false
redis:
  enabled: false
externalDatabase:
  type: postgres
  host: ""
  port: ""
  database: airflow_db
  user: airflow_admin
  passwordSecret: ""
  passwordSecretKey: ""
  properties: ""
dags:
  gitSync:
    enabled: true
    repo: ""
    branch: ""
    revision: HEAD
    repoSubPath: "dags"
    httpSecret: ""
    httpSecretUsernameKey: username 
    httpSecretPasswordKey: password

The access via the connection "AWSS3_Log_Storage" and the serviceAccount works fine when I test the connection in the UI. My guess is that I also have access to the s3 bucket via the webserver, but no logs are written. I assume my pod does not have the rights or connection to be able to write, or doesn't get any information whatsoever to log remotely.

I am still completely stuck at this point and would appreciate any help.

EDIT: I was testing it with the normal @task operator so I assume it nothing related to the kubernetesPodOperator per se..

[thesuperzapper]

@seblum it's almost certainly a permission issue preventing the tasks from writing logs to the S3 bucket.

A few things you might want to look at:

1. Try and capture the logs of a task Pod, and see if it throws an error as it attempts to upload the logs to S3 (happens at the end of the task)
2. Try running a Pod manually using the same ServiceAccount, and ensure you're able to use the IRSA-provided auth to write files in the bucket (perhaps using the aws cli)
3. In that vein, just confirm what level of access is given to arn:aws:iam::<123456>:role/airflow-s3-access-role on the bucket you want to write to, it could just be an IAM mistake

[seblum]

Thanks for your reply @thesuperzapper. I was looking into your suggestions but none solved it:

1. I do not see any error messages regarding logging. In fact, logging doesn't seem to be mentioned anywhere.
2. Good suggestion. I ssh into a task pod and running an aws sts assume-role-with-web-identity and aws s3 cp test.txt s3://airflowdatabucket/airflow/logs worked. The provided IRSA-auth works smoothly as it seems. Two notes here I realized though. (a) I had to install the aws cli. I would assume airflow will not need it, but just to make sure I wanted to mention. (b) I realized none of the airflow.config parameters are written as environment variables. Shouldn't they? I was thinking at least AIRFLOW__LOGGING__REMOTE_LOGGING: "True" should be written to a task pod so it know it should write to a remote location.
   I checked further regarding the envs and I do see the config settings in the airflow-config-envs k8s secret. Yet when I check within a task pod, the .cfg does not set anyone of them.
3. Not an issue. I am running it with AmazonS3FullAccess atm for testing

[seblum]

Alright, the scheduler and the webserver state the following. Which explains why logging does not work / is not enabled, even though the connection of the helm chart is set.

I guess this explains the issue, though I haven't solved it yet.
I am running a custom airflow image and I assume its related to this. Updating it to the current version did not fix the issue.

ARG AIRFLOW_VERSION=2.6.3

FROM apache/airflow:2.6.3-python3.11
USER airflow

COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

Yet, I also figured I can specify localSettings in the helm chart. I thought this is optional though and the logging functionality is already provided by only setting the airflow.config variables within the helm chart.