Merge branch 'master' into bug/arraynode-terminal-eventing

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -33,6 +33,17 @@ If tests were added, say they were added here. Please make sure to add some test
 If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
 If tests were not added, please describe why they were not added and/or why it was difficult to add.
 -->
+### Labels
+
+Please add one or more of the following labels to categorize your PR:
+- **added**: For new features.
+- **changed**: For changes in existing functionality.
+- **deprecated**: For soon-to-be-removed features.
+- **removed**: For features being removed.
+- **fixed**: For any bug fixed.
+- **security**: In case of vulnerabilities
+
+This is important to improve the readability of release notes.
 
 ### Setup process
 

.github/workflows/checks.yml

@@ -9,6 +9,7 @@ on:
   push:
     branches:
       - master
+      - 'release-v**'
 env:
   GO_VERSION: "1.22"
   PRIORITIES: "P0"
@@ -32,8 +33,7 @@ jobs:
         component:
           - datacatalog
           - flyteadmin
-          # TODO(monorepo): Enable lint flytecopilot
-          # - flytecopilot
+          - flytecopilot
           - flytectl
           - flyteidl
           - flyteplugins

.github/workflows/dependency-review.yml

@@ -0,0 +1,18 @@
+name: 'Dependency Review'
+on: [pull_request]
+permissions:
+  contents: read
+  pull-requests: write
+jobs:
+  dependency-review:
+    runs-on: ubuntu-latest
+    steps:
+      - name: 'Checkout Repository'
+        uses: actions/checkout@v4
+      - name: Dependency Review
+        uses: actions/dependency-review-action@v4
+        with:
+          comment-summary-in-pr: on-failure
+          # Licenses need to come from https://spdx.org/licenses/
+          deny-licenses: GPL-1.0-only, GPL-1.0-or-later, GPL-2.0-only, GPL-2.0-or-later, GPL-3.0-only, GPL-3.0-or-later
+

.github/workflows/flytectl-install.yml

@@ -8,6 +8,7 @@ on:
   pull_request:
     paths:
       - flytectl/**
+      - 'release-v**'
   push:
     branches:
       - master

.github/workflows/flyteidl-buf-publish.yml

@@ -6,6 +6,7 @@ on:
       - artifacts-shell-2
       - artifacts
       - master
+      - 'release-v**'
     paths:
       - 'flyteidl/**'
 jobs:

.github/workflows/flyteidl-checks.yml

@@ -9,6 +9,7 @@ on:
   push:
     branches:
       - master
+      - 'release-v**'
 env:
   GO_VERSION: "1.22"
 jobs:

.github/workflows/helm-charts.yaml

@@ -10,6 +10,7 @@ on:
     branches:
       - master
       - rc/*
+      - 'release-v**'
   workflow_dispatch:
 
 jobs:

.github/workflows/publish.yml

@@ -59,7 +59,7 @@ jobs:
       - name: Before Build
         run: ${{ inputs.before-build }}
       - name: Build and Push Image
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ${{ inputs.dockerfile }}

.github/workflows/sandbox.yml

@@ -53,7 +53,7 @@ jobs:
           username: "${{ secrets.FLYTE_BOT_USERNAME }}"
           password: "${{ secrets.FLYTE_BOT_PAT }}"
       - name: Build and push DIND Image
-        uses: docker/build-push-action@v2
+        uses: docker/build-push-action@v6
         with:
           context: .
           platforms: linux/arm64, linux/amd64

.github/workflows/single-binary.yml

@@ -10,6 +10,7 @@ on:
     branches:
       - master
       - rc/*
+      - "release-v**"
   workflow_dispatch:
 
 jobs:
@@ -72,7 +73,7 @@ jobs:
         run: |
           mkdir -p docker/sandbox-bundled/images/tar/{arm64,amd64}
       - name: Export ARM64 Image
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: .
           platforms: linux/arm64
@@ -83,7 +84,7 @@ jobs:
           file: Dockerfile
           outputs: type=docker,dest=docker/sandbox-bundled/images/tar/arm64/flyte-binary.tar
       - name: Export AMD64 Image
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: .
           platforms: linux/amd64
@@ -94,7 +95,7 @@ jobs:
           file: Dockerfile
           outputs: type=docker,dest=docker/sandbox-bundled/images/tar/amd64/flyte-binary.tar
       - name: Upload single binary image
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v4
         with:
           name: single-binary-image
           path: docker/sandbox-bundled/images/tar
@@ -107,7 +108,7 @@ jobs:
           password: "${{ secrets.FLYTE_BOT_PAT }}"
       - name: Build and push Image
         if: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: .
           platforms: linux/arm64, linux/amd64
@@ -133,7 +134,7 @@ jobs:
           echo "FLYTESNACKS_VERSION=${FLYTESNACKS_VERSION}" >> ${GITHUB_ENV}
       - name: Checkout
         uses: actions/checkout@v4
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: single-binary-image
           path: docker/sandbox-bundled/images/tar
@@ -143,7 +144,7 @@ jobs:
           driver-opts: image=moby/buildkit:master
           buildkitd-flags: "--allow-insecure-entitlement security.insecure"
       - name: Build sandbox image for functional tests
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: docker/sandbox-bundled
           load: true
@@ -175,7 +176,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install uv
-          uv pip install --system flytekit flytekitplugins-deck-standard flytekitplugins-envd "numpy<2.0.0" pyarrow
+          uv pip install --system flytekit flytekitplugins-deck-standard "numpy<2.0.0" pyarrow pandas
           uv pip freeze
       - name: Checkout flytesnacks
         uses: actions/checkout@v4
@@ -188,14 +189,107 @@ jobs:
           flytekit_version=$(pip show flytekit | grep -i version | awk '{ print $2 }')
           while read -r line;
           do
-              pyflyte --config ./boilerplate/flyte/end2end/functional-test-config.yaml \
+              pyflyte -vv --config ./boilerplate/flyte/end2end/functional-test-config.yaml \
                   register \
                   --project flytesnacks \
                   --domain development \
                   --image cr.flyte.org/flyteorg/flytekit:py3.12-${flytekit_version} \
                   --version ${{ env.FLYTESNACKS_VERSION }} \
                   flytesnacks/$line;
           done < flytesnacks/flyte_tests.txt
+      - name: Install Pytest
+        run: |
+          pip install pytest
+      - name: End2End
+        run: |
+          make end2end_execute
+
+  sandbox-bundled-offloaded-functional-tests:
+    runs-on: ubuntu-latest
+    env:
+      FLYTESNACKS_PRIORITIES: "P0"
+      FLYTESNACKS_VERSION: ""
+    timeout-minutes: 60
+    needs: [build-and-push-single-binary-image]
+    steps:
+      - name: Set latest Flytesnacks release
+        if: ${{ env.FLYTESNACKS_VERSION == '' }}
+        run: |
+          FLYTESNACKS_VERSION="$(curl --silent https://api.github.com/repos/flyteorg/flytesnacks/releases/latest | jq -r .tag_name)"
+          echo "FLYTESNACKS_VERSION=${FLYTESNACKS_VERSION}" >> ${GITHUB_ENV}
+      - name: Checkout
+        uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
+        with:
+          name: single-binary-image
+          path: docker/sandbox-bundled/images/tar
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          driver-opts: image=moby/buildkit:master
+          buildkitd-flags: "--allow-insecure-entitlement security.insecure"
+      - name: Build sandbox image for functional tests
+        uses: docker/build-push-action@v6
+        with:
+          context: docker/sandbox-bundled
+          load: true
+          allow: "security.insecure"
+          tags: flyte-sandbox-bundled:local
+      # Without this, the GHA runner is under disk pressure and evicts all the pods.
+      # Buildx cache uses roughly 50% (7gb) of the GHA runner's disk (14gb).
+      - name: Prune Docker Buildx cache to reclaim storage
+        run: docker buildx prune --all --force
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.12"
+      - uses: unionai/flytectl-setup-action@v0.0.3
+      - name: Setup sandbox
+        run: |
+          mkdir -p ~/.flyte/sandbox
+          cat << EOF > ~/.flyte/sandbox/config.yaml
+          propeller:
+            literal-offloading-config:
+              enabled: true
+              min-size-in-mb-for-offloading: 0.000001
+          task_resources:
+            defaults:
+              cpu: "0"
+              memory: "0"
+            limits:
+              cpu: "0"
+              memory: "0"
+          EOF
+          flytectl demo start --image flyte-sandbox-bundled:local --imagePullPolicy Never
+        # By setting min-size-in-mb-for-offloading to 0.000001, we ensure that all tasks are offloaded.
+      - name: Install Python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install uv
+          uv pip install --system flytekit flytekitplugins-deck-standard "numpy<2.0.0" pyarrow pandas
+          uv pip freeze
+      - name: Checkout flytesnacks
+        uses: actions/checkout@v4
+        with:
+          repository: flyteorg/flytesnacks
+          path: flytesnacks
+          ref: ${{ env.FLYTESNACKS_VERSION }}
+      - name: Register specific tests
+        run: |
+          flytekit_version=$(pip show flytekit | grep -i version | awk '{ print $2 }')
+          while read -r line;
+          do
+              pyflyte -vv --config ./boilerplate/flyte/end2end/functional-test-config.yaml \
+                  register \
+                  --project flytesnacks \
+                  --domain development \
+                  --image cr.flyte.org/flyteorg/flytekit:py3.12-${flytekit_version} \
+                  --version ${{ env.FLYTESNACKS_VERSION }} \
+                  flytesnacks/$line;
+          done < flytesnacks/flyte_tests.txt
+      - name: Install Pytest
+        run: |
+          pip install pytest
       - name: End2End
         run: |
           make end2end_execute
@@ -207,7 +301,7 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: single-binary-image
           path: docker/sandbox-bundled/images/tar
@@ -239,7 +333,7 @@ jobs:
           username: "${{ secrets.FLYTE_BOT_USERNAME }}"
           password: "${{ secrets.FLYTE_BOT_PAT }}"
       - name: Build and push multi-arch image
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: docker/sandbox-bundled
           allow: "security.insecure"

.github/workflows/tests.yml

@@ -8,6 +8,7 @@ on:
   push:
     branches:
       - master
+      - 'release-v**'
   pull_request:
 jobs:
   compile:
@@ -69,6 +70,9 @@ jobs:
       - name: Build the documentation
         working-directory: ${{ github.workspace }}/flyte
         shell: bash -el {0}
+        env:
+          # this is a fake key for algolia docsearch to make docs build process pass
+          DOCSEARCH_API_KEY: fake_docsearch_api_key
         run: |
           conda activate monodocs-env
           make docs

.github/workflows/validate-helm-charts.yaml

@@ -4,6 +4,7 @@ on:
   pull_request:
     branches:
       - master
+      - 'release-v**'
     paths:
       - "charts/**"
       - "deployment/**"

.gitignore

@@ -30,12 +30,12 @@ vendor/
 /docker/sandbox-bundled/images/tar
 **/bin/
 docs/_tags/
-docs/flytectl
-docs/protos
-docs/flytekit
+docs/api/flytectl
+docs/api/flytekit
+docs/api/flyteidl
 docs/flytesnacks
 docs/examples
 docs/_src
 docs/_projects
-docs/api
 docs/tests
+empty-config.yaml

CHANGELOG/CHANGELOG-v1.13.1-rc1.md

@@ -0,0 +1,35 @@
+# Flyte v1.13.0-rc1 Release Notes
+
+## What's Changed
+* Add CustomHeaderMatcher to pass additional headers by @andrewwdye in https://github.com/flyteorg/flyte/pull/5563
+* Turn flyteidl and flytectl releases into manual gh workflows by @eapolinario in https://github.com/flyteorg/flyte/pull/5635
+* docs: fix typo by @cratiu222 in https://github.com/flyteorg/flyte/pull/5643
+* Use enable_deck=True in docs by @thomasjpfan in https://github.com/flyteorg/flyte/pull/5645
+* Fix flyteidl release  checkout all tags by @eapolinario in https://github.com/flyteorg/flyte/pull/5646
+* Install pyarrow in sandbox functional tests by @eapolinario in https://github.com/flyteorg/flyte/pull/5647
+* docs: add documentation for configuring notifications in GCP by @desihsu in https://github.com/flyteorg/flyte/pull/5545
+* Correct "sucessfile" to "successfile" by @shengyu7697 in https://github.com/flyteorg/flyte/pull/5652
+* Fix ordering for custom template values in cluster resource controller by @katrogan in https://github.com/flyteorg/flyte/pull/5648
+* Don't error when attempting to trigger schedules for inactive projects by @katrogan in https://github.com/flyteorg/flyte/pull/5649
+* Update Flyte components - v1.13.1-rc0 by @flyte-bot in https://github.com/flyteorg/flyte/pull/5656
+* Add offloaded path to literal by @katrogan in https://github.com/flyteorg/flyte/pull/5660
+* Improve error messaging for invalid arguments by @pingsutw in https://github.com/flyteorg/flyte/pull/5658
+* DOC-462 Update "Try Flyte in the browser" text by @neverett in https://github.com/flyteorg/flyte/pull/5654
+* DOC-533 Remove outdated duplicate notification config content by @neverett in https://github.com/flyteorg/flyte/pull/5672
+* Validate labels before creating flyte CRD by @pingsutw in https://github.com/flyteorg/flyte/pull/5671
+* Add FLYTE_INTERNAL_POD_NAME environment variable that holds the pod name by @bgedik in https://github.com/flyteorg/flyte/pull/5616
+* Upstream Using InMemory token cache for admin clientset in propeller by @pvditt in https://github.com/flyteorg/flyte/pull/5621
+* [Bug] Update resource failures w/ Finalizers set  (#423) by @pvditt in https://github.com/flyteorg/flyte/pull/5673
+* [BUG] array node eventing bump version by @pvditt in https://github.com/flyteorg/flyte/pull/5680
+* Add custominfo to agents by @ddl-rliu in https://github.com/flyteorg/flyte/pull/5604
+* [BUG] use deep copy of bit arrays when getting array node state by @pvditt in https://github.com/flyteorg/flyte/pull/5681
+* More concise definition of launchplan by @eapolinario in https://github.com/flyteorg/flyte/pull/5682
+* Auth/prevent lookup per call by @wild-endeavor in https://github.com/flyteorg/flyte/pull/5686
+
+## New Contributors
+* @cratiu222 made their first contribution in https://github.com/flyteorg/flyte/pull/5643
+* @desihsu made their first contribution in https://github.com/flyteorg/flyte/pull/5545
+* @shengyu7697 made their first contribution in https://github.com/flyteorg/flyte/pull/5652
+* @bgedik made their first contribution in https://github.com/flyteorg/flyte/pull/5616
+
+**Full Changelog**: https://github.com/flyteorg/flyte/compare/flytectl/v0.9.1...v1.13.1-rc1