add ability to customize kfp-launcher ConfigMap data contents

[gregsheremeta]

The issue resolved by this Pull Request:

Resolves https://issues.redhat.com/browse/RHOAIENG-4528

Description of your changes:

Allow the user to fully replace the data contents of the kfp-launcher ConfigMap. The kfp-launcher component requires a ConfigMap to exist in the namespace where it runs (i.e. the namespace where pipelines run). This ConfigMap contains object storage configuration, as well as pipeline root (object store root path where artifacts will be uploaded) configuration. Currently this ConfigMap must be named "kfp-launcher". We currently deploy a default copy of the kfp-launcher ConfigMap via DSPO, but a user may want to provide their own ConfigMap configuration, so that they can specify multiple object storage sources and paths.

Add a CustomKfpLauncherConfigMap parameter to DSPA.ApiServer. When specified, the data contents of the kfp-launcher ConfigMap that DSPO writes will be fully replaced with the data contents of the ConfigMap specified here.

Fixes: https://issues.redhat.com/browse/RHOAIENG-4528

Testing instructions

login to openshift
$ oc login --token=sha256~jMYSUPERSECRETTOKEN --server=https://api.greg-1234.bd9q.p3.openshiftapps.com:443

make a new project for dspo to be deployed
$ oc new-project dspo

check out the PR
$ gh pr checkout 681

deploy it using the pre-built image
$ make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-681" OPERATOR_NS=dspo

wait a bit for the deployment to finish, and make sure it's up

make a new project to deploy a DSPA into

first, create a ConfigMap to copy data from (click the plus button at the top).

Example:

kind: ConfigMap
apiVersion: v1
metadata:
  name: my-custom-kfp-launcher
  namespace: dspa1
data:
  defaultPipelineRoot: 's3://some-path'
  greg: isCool
  hey: |
    - what: isUp

Here's another more complex example (don't have screenshots for this one, but it should work the same)

kind: ConfigMap
apiVersion: v1
metadata:
  name: my-custom-kfp-launcher
  namespace: dspa1
data:
  defaultPipelineRoot: s3://mlpipeline/special/location
  providers: |-
    s3:
      default:
        endpoint: s3.amazonaws.com
        disableSSL: false
        region: us-east-2
        credentials:
          fromEnv: true
    gs:
      default:
        credentials:
          fromEnv: false
          secretRef:
            secretName: your-k8s-secret
            tokenKey: some-key-1

deploy the DSPA, also using the plus button. Don't forget to tell it about your new ConfigMap --

notice the customKfpLauncherConfigMap: my-custom-kfp-launcher line!

apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1
kind: DataSciencePipelinesApplication
metadata:
  name: dspa1
spec:
  dspVersion: v2
  apiServer:
    image: "quay.io/opendatahub/ds-pipelines-api-server:latest"
    argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:latest"
    argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:latest"
    customKfpLauncherConfigMap: my-custom-kfp-launcher
  persistenceAgent:
    image: "quay.io/opendatahub/ds-pipelines-persistenceagent:latest"
  scheduledWorkflow:
    image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:latest"
  mlmd:  
    deploy: true  # Optional component
    grpc:
      image: "quay.io/opendatahub/mlmd-grpc-server:latest"
    envoy:
      image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
  mlpipelineUI:
    deploy: true  # Optional component 
    image: "quay.io/opendatahub/ds-pipelines-frontend:latest"
  objectStorage:
    minio:
      deploy: true
      image: 'quay.io/opendatahub/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance'

wait a bit for DSPO to reconcile the new DSPA … once you start to see pods come up, should be good

then check the kfp-launcher ConfigMap. It's data section should look like what you put in the my-custom-kfp-launcher ConfigMap

Now delete the DSPA and redeploy it, but this time leave out the customKfpLauncherConfigMap: my-custom-kfp-launcher line from the DSPA. The kfp-launcher ConfigMap should now look how it usually does.

Checklist

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[dsp-developers]

A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681
An OCP cluster where you are logged in as cluster admin is required.

To use this image run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines-operator.git
cd data-science-pipelines-operator/
git fetch origin pull/681/head
git checkout -b pullrequest 167bb2433fe8fa74ab55b36197c1049174676a02
oc new-project opendatahub
make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-681"

More instructions here on how to deploy and test a Data Science Pipelines Application.

[VaniHaripriya]

/verified
Tested as per instructions, both the scenarios works as expected.

[gregsheremeta]

/hold

found a bug while writing the test :)

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681

[gregsheremeta]

@HumairAK here's the screenshots showing it works with a more complex yaml from your doc

source:

dest:

text diff:

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681

[hbelmiro]

Just left an optional nit.
I don't want to block the merge just because of that. So...
/lgtm

[dsp-developers]

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-681

[hbelmiro]

/lgtm

[HumairAK]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gregsheremeta]

/unhold