Poetry fails to publish package to Jfrog's PyPi artifactory via trusted publishing

[jzizka91]

Hello,

I'm having a trouble with publishing packages to Jfrog's PyPi Artifactory by using a access token. Unfortunately only the http-basic.example auth method works for me. This method however requires to use both username and password(token). I'd rather use only the access-token without having to specify the username for the associated fetched access-token. I tried pypi-token.example auth method but it results in Wrong username was used.

  packages_publish:
    needs:
      - generate_packages
      - packages_test
    runs-on: ubuntu-20.04
    container: <PYTHONRUNTIME_IMAGE>
    env:
      OIDC_AUDIENCE: 'jfrog-github'
      OIDC_ITEGRATION_NAME: 'github-oidc-integration'
    #if: github.ref == 'refs/heads/main'
    permissions: write-all
    steps:
      - name: Install dependencies for auth
        run: apt update && apt install -y jq
      - name: Get ID Token
        id: idtoken
        run: |
          ID_TOKEN=$(curl -sLS -H "User-Agent: actions/oidc-client" -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
          "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=$OIDC_AUDIENCE" | jq .value | tr -d '"')
          echo "ID_TOKEN=${ID_TOKEN}" >> $GITHUB_OUTPUT
      - name: Fetch Access Token from Artifactory
        id: fetch_access_token
        env:
          ID_TOKEN: ${{ steps.idtoken.outputs.id_token }}
        run: |
          ACCESS_TOKEN=$(curl \
          -X POST \
          -H "Content-type: application/json" \
          https://example.jfrog.io/access/api/v1/oidc/token \
          -d \
          "{\"grant_type\": \"urn:ietf:params:oauth:grant-type:token-exchange\", \"subject_token_type\":\"urn:ietf:params:oauth:token-type:id_token\", \"subject_token\": \"$ID_TOKEN\", \"provider_name\": \"$OIDC_ITEGRATION_NAME\"}" | jq .access_token | tr -d '"')
          echo ACCESS_TOKEN=$ACCESS_TOKEN >> $GITHUB_OUTPUT
      - uses: actions/checkout@v3
      - name: Publish to artifactory
        env:
          POETRY_PYPI_TOKEN_EXAMPLE: ${{ steps.fetch_access_token.outputs.access_token }}
          POETRY_REPOSITORIES_EXAMPLE_URL: 'https://example.jfrog.io/artifactory/api/pypi/pypi-general-local'
        run: |
          cd packages/<package-example>
          sed -i "0,/\(version = \"[0-9]\+.[0-9]\+\)\"/s//\1.${{ github.run_number }}\"/" pyproject.toml
          poetry config pypi-token.example $POETRY_PYPI_TOKEN_EXAMPLE
          poetry publish --build -r example -vvv

Full error Publish output:

Publishing package-example (0.1.50) to example
 - Uploading package-example-0.1.50-py3-none-any.whl 0%
 - Uploading package-example-0.1.50-py3-none-any.whl 100%
  Stack trace:
  1  /opt/poetry/venv/lib/python3.10/site-packages/poetry/publishing/uploader.py:265 in _upload_file
      263│                     bar.display()
      264│                 else:
    → 265│                     resp.raise_for_status()
      266│             except (requests.ConnectionError, requests.HTTPError) as e:
      267│                 if self._io.output.is_decorated():
  HTTPError
  401 Client Error:  for url: https://example.jfrog.io/artifactory/api/pypi/pypi-general-local
  at /opt/poetry/venv/lib/python3.10/site-packages/requests/models.py:1021 in raise_for_status
      1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
      1018│             )
      1019│ 
      1020│         if http_error_msg:
    → 1021│             raise HTTPError(http_error_msg, response=self)
      1022│ 
      1023│     def close(self):
      1024│         """Releases the connection back to the pool. Once this method has been
      1025│         called the underlying ``raw`` object must not be accessed again.
The following error occurred when trying to handle this error:
  Stack trace:
  11  /opt/poetry/venv/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a
  10  /opt/poetry/venv/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 
   9  /opt/poetry/venv/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 
   8  /opt/poetry/venv/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code
   7  /opt/poetry/venv/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
   6  /opt/poetry/venv/lib/python3.10/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:
   5  /opt/poetry/venv/lib/python3.10/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1
   4  /opt/poetry/venv/lib/python3.10/site-packages/poetry/console/commands/publish.py:82 in handle
        80│         )
        81│ 
     →  82│         publisher.publish(
        83│             self.option("repository"),
        84│             self.option("username"),
   3  /opt/poetry/venv/lib/python3.10/site-packages/poetry/publishing/publisher.py:86 in publish
        84│         )
        85│ 
     →  86│         self._uploader.upload(
        87│             url,
        88│             cert=resolved_cert,
   2  /opt/poetry/venv/lib/python3.10/site-packages/poetry/publishing/uploader.py:107 in upload
       105│ 
       106│         try:
     → 107│             self._upload(session, url, dry_run, skip_existing)
       108│         finally:
       109│             session.close()
   1  /opt/poetry/venv/lib/python3.10/site-packages/poetry/publishing/uploader.py:191 in _upload
       189│     ) -> None:
       190│         for file in self.files:
     → 191│             self._upload_file(session, url, file, dry_run, skip_existing)
       192│ 
       193│     def _upload_file(
  UploadError
  HTTP Error 401:  | b'{\n  "errors" : [ {\n    "status" : 401,\n    "message" : "Wrong username was used"\n  } ]\n}'
  at /opt/poetry/venv/lib/python3.10/site-packages/poetry/publishing/uploader.py:271 in _upload_file
      267│                 if self._io.output.is_decorated():
      268│                     self._io.overwrite(
      269│                         f" - Uploading {file.name} FAILED"
      270│                     )
    → 271│                 raise UploadError(e)
      272│             finally:
      273│                 self._io.write_line("")
      274│ 
      275│     def _register(self, session: requests.Session, url: str) -> requests.Response:
Error: Process completed with exit code 1.

This works:

- name: Publish to artifactory
  env:
    POETRY_HTTP_BASIC_EXAMPLE_USERNAME: '${{ vars.jfrog_username }}`
    POETRY_HTTP_BASIC_EXAMPLE_PASSWORD: '${{ steps.fetch_access_token.outputs.access_token }}'
    POETRY_REPOSITORIES_EXAMPLE_URL: 'https://example.jfrog.io/artifactory/api/pypi/pypi-general-local'
  run: |
    cd packages/<package-example>
    sed -i "0,/\(version = \"[0-9]\+.[0-9]\+\)\"/s//\1.${{ github.run_number }}\"/" pyproject.toml
    poetry config http-basic.example $POETRY_HTTP_BASIC_EXAMPLE_USERNAME $POETRY_HTTP_BASIC_EXAMPLE_PASSWORD
    poetry publish --build -r example

POETRY_HTTP_BASIC_EXAMPLE_USERNAME - must be the username associated with the fetched access-token from Jfrog (user that exists on jfrog and was set in identity method within the Jfrog's OIDC Integration).

POETRY_HTTP_BASIC_EXAMPLE_PASSWORD - short-lived access token fetched from Jfrog.

I've also tried the http-basic.example auth method with using __token__ as the username and access-token as the password, but it keep returning the same error.

[abn]

JFrog access tokens do not work the same way as PyPI tokens.

https://jfrog.com/help/r/jfrog-platform-administration-documentation/basic-authentication

You either need to use basic authentication with the username set to the user generating the token (see above link), or make use of an auth header. The latter is not a feature that is supported, but potentially something we can add as a new feature if there is enough appetite for this.

With the above in mind, I think this is working as designed at the moment.

[jzizka91]

I figured out how to make it work using basic authentication and Jfrog's OpenID Connect Integration.

With this method, you don't need to store poetry username and password credentials in GitHub secrets.

Token (JWT) will be used as the password and the username that is associated with the token will be parsed from the token.

To make this work, you need to:

1. Configure an OIDC Integration on Jfrog
2. Add Identity Mappings for the already created OIDC Integration - Identity Mappings defines a set of conditions for matching an incoming OIDC claim to a specific authorization scope. If an incoming OIDC claim matches the defined conditions, Identity Mappings will generate a short-lived access token to grant access for publishing.

Example:

packages_publish:
    runs-on: ubuntu-latest
    container: ghcr.io/<poetry_image>
    env:
      OIDC_AUDIENCE: 'jfrog-github' # must match the exact name set in Jfrog's OIDC Integration
      OIDC_ITEGRATION_NAME: 'github-oidc-integration' # must match the exact name set in Jfrog's OIDC Integration
    permissions:
      id-token: write # this permission is mandatory for Jfrogs's OpenID Connect Integration/trusted publishing
      contents: read # required if job’s steps are manipulating with the repository, e.g. actions/checkout@v4.
      packages: read # required if job is using a container image
    steps:
      - name: Install dependencies for auth
        run: apt update && apt install -y jq
      - name: Get ID Token
        run: |
          ID_TOKEN=$(curl -sLS -H "User-Agent: actions/oidc-client" -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
          "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=$OIDC_AUDIENCE" | jq .value | tr -d '"')
          echo "ID_TOKEN=${ID_TOKEN}" >> $GITHUB_ENV
      - name: Fetch Access Token from Artifactory
        env:
          ID_TOKEN: ${{ env.ID_TOKEN }}
        run: |
          ACCESS_TOKEN=$(curl \
          -X POST \
          -H "Content-type: application/json" \
          https://example.jfrog.io/access/api/v1/oidc/token \
          -d \
          "{\"grant_type\": \"urn:ietf:params:oauth:grant-type:token-exchange\", \"subject_token_type\":\"urn:ietf:params:oauth:token-type:id_token\", \"subject_token\": \"$ID_TOKEN\", \"provider_name\": \"$OIDC_ITEGRATION_NAME\"}" | jq .access_token | tr -d '"')
          echo "::add-mask::$ACCESS_TOKEN"
          echo "ACCESS_TOKEN=${ACCESS_TOKEN}" >> $GITHUB_ENV
      - name: Parse username from access-token
        env:
          ACCESS_TOKEN: ${{ env.ACCESS_TOKEN }}
        shell: bash
        run: |
          USERNAME=$(jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$ACCESS_TOKEN" | jq -r .sub | sed 's/.*users\///')
          echo USERNAME=${USERNAME} >> $GITHUB_ENV
      - uses: actions/checkout@v4
      - name: Publish to artifactory
        env:
          JFROG_SERVICE_USERNAME: ${{ env.USERNAME }}
          JFROG_SERVICE_ACCESS_TOKEN: ${{ env.ACCESS_TOKEN }}
        run: |
          poetry config repositories.example"https://example.jfrog.io/artifactory/api/pypi/example"
          poetry config http-basic.example$JFROG_SERVICE_USERNAME $JFROG_SERVICE_ACCESS_TOKEN
          poetry publish --build -r example