From bde843dabdeaeba0b1068874d423f552691747a6 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 3 Oct 2023 17:59:05 +0200 Subject: [PATCH 01/37] be clear about difference between workspace and artifact --- pages/adding-new-application/index.md | 40 ++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index c0f20623..8d9b4954 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -6,17 +6,49 @@ description: >- guide outlines the process. --- -The two main functionalities that make Plural work are dependency tracking between DevOps tools (Helm and Terraform) and templating. +The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. When a user sets up a new Plural workspace in a git repository a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. -Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the Plural API, which the CLI downloads. The CLI will then generate the `values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using the values saved in the `context.yaml` using its templating engine. +Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using the values saved in the `context.yaml` using its templating engine. + +E.g. the `tree` of the Plural Console application in your deployment repository after a `plural build console` might look like this: +```console +$ tree . +. +├── build.hcl +├── deploy.hcl +├── diff.hcl +├── helm +│ └── console +│ ├── Chart.lock +│ ├── Chart.yaml +│ ├── charts +│ │ └── console-0.7.79.tgz +│ ├── default-values.yaml +│ ├── templates +│ │ ├── NOTES.txt +│ │ ├── application.yaml +│ │ ├── license.yaml +│ │ └── secret.yaml +│ └── values.yaml +├── manifest.yaml +├── output.yaml +└── terraform + ├── aws + │ ├── deps.yaml + │ ├── main.tf + │ ├── node-group.tf + │ └── variables.tf + ├── main.tf + └── outputs.tf +``` ## Plural application artifacts -As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. Some extra files are necessary in Helm charts and Terraform modules for Plural to be able to understand their dependencies and run them through its templating engine. Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. +As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. Some extra configuration files are necessary in the applications artifact Helm charts and Terraform modules for Plural to be able to understand their dependencies and run them through its templating engine. Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. -The `values.yaml.tpl` and `terraform.tfvars` files are run through the Plural templating engine, which is similar to that of Helm, and are used to generate the `values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module. +During a `plural build` inside the deployment repository the `values.yaml.tpl` and `terraform.tfvars` files of the artifact are run through the Plural templating engine, which is similar to that of Helm, and are used to generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module. The next example is a snippet of the `values.yaml.tpl` file for Grafana: From f10320cdc72e6f23c3e064e18e51894c2a04124d Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 3 Oct 2023 18:02:47 +0200 Subject: [PATCH 02/37] mention deployment repository as it's used but not defined --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 8d9b4954..800564c9 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -8,7 +8,7 @@ description: >- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. -When a user sets up a new Plural workspace in a git repository a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. +When a user sets up a new Plural workspace in a git repository (we sometimes call that a **deployment repository**) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using the values saved in the `context.yaml` using its templating engine. From e70a0794523e2fdcc322e8680aa24ea1d0a32321 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 3 Oct 2023 18:37:57 +0200 Subject: [PATCH 03/37] explain plurals artifact structure and scaffolding --- pages/adding-new-application/index.md | 96 +++++++++++++++++++++++++-- 1 file changed, 91 insertions(+), 5 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 800564c9..3064790e 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -12,7 +12,8 @@ When a user sets up a new Plural workspace in a git repository (we sometimes cal Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using the values saved in the `context.yaml` using its templating engine. -E.g. the `tree` of the Plural Console application in your deployment repository after a `plural build console` might look like this: +For example, after the `plural build dagster` command the `tree` of the built Dagster application in your deployment repository might look like this: + ```console $ tree . . @@ -20,11 +21,11 @@ $ tree . ├── deploy.hcl ├── diff.hcl ├── helm -│ └── console +│ └── dagster │ ├── Chart.lock │ ├── Chart.yaml │ ├── charts -│ │ └── console-0.7.79.tgz +│ │ └── dagster-0.1.44.tgz │ ├── default-values.yaml │ ├── templates │ │ ├── NOTES.txt @@ -37,8 +38,10 @@ $ tree . └── terraform ├── aws │ ├── deps.yaml + │ ├── iam.tf │ ├── main.tf - │ ├── node-group.tf + │ ├── outputs.tf + │ ├── postgres.tf │ └── variables.tf ├── main.tf └── outputs.tf @@ -46,10 +49,93 @@ $ tree . ## Plural application artifacts -As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. Some extra configuration files are necessary in the applications artifact Helm charts and Terraform modules for Plural to be able to understand their dependencies and run them through its templating engine. Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. +As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. +Some extra configuration files are necessary in the applications artifact for Plural to be able to understand +- the Helm charts and Terraform modules dependencies to run them through its templating engine +- dependencies on other Plural artifacts +- platform specific components and infrastructure configurations +- as well as Plural's own package and version specs. + +As an example Dagster's artifact `tree` would look like this: + +```console +$ tree . +. +├── Pluralfile +├── helm +│ └── dagster +│ ├── Chart.lock +│ ├── Chart.yaml +│ ├── README.md +│ ├── charts +│ │ ├── config-overlays-0.1.1.tgz +│ │ ├── dagster-1.4.10.tgz +│ │ ├── postgres-0.1.16.tgz +│ │ └── test-base-0.1.10.tgz +│ ├── deps.yaml +│ ├── runbooks +│ │ └── scaling-manual.xml +│ ├── templates +│ │ ├── _helpers.tpl +│ │ ├── oidc.yaml +│ │ ├── runbooks.yaml +│ │ └── secret.yaml +│ ├── values.yaml +│ └── values.yaml.tpl +├── plural +│ ├── docs +│ │ ├── private-ingress.md +│ │ └── user-code.md +│ ├── icons +│ │ ├── dagster-primary-mark.png +│ │ └── dagster.png +│ ├── notes.tpl +│ ├── recipes +│ │ ├── dagster-aws.yaml +│ │ ├── dagster-azure.yaml +│ │ └── dagster-gcp.yaml +│ └── tags +│ ├── helm +│ │ └── dagster.yaml +│ └── terraform +│ ├── aws.yaml +│ ├── azure.yaml +│ └── gcp.yaml +├── repository.yaml +├── terraform +│ ├── aws +│ │ ├── deps.yaml +│ │ ├── iam.tf +│ │ ├── main.tf +│ │ ├── outputs.tf +│ │ ├── postgres.tf +│ │ ├── terraform.tfvars +│ │ └── variables.tf +│ ├── azure +│ │ ├── deps.yaml +│ │ ├── main.tf +│ │ ├── terraform.tfvars +│ │ └── variables.tf +│ └── gcp +│ ├── deps.yaml +│ ├── main.tf +│ ├── outputs.tf +│ ├── terraform.tfvars +│ └── variables.tf +└── vendor_images.yaml +``` + +Let's disect what's going on here. + + + +Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. During a `plural build` inside the deployment repository the `values.yaml.tpl` and `terraform.tfvars` files of the artifact are run through the Plural templating engine, which is similar to that of Helm, and are used to generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module. + +### Templating + The next example is a snippet of the `values.yaml.tpl` file for Grafana: ```yaml From e95c320d80be840100d14b8e93552c27f86b6865 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 12:08:17 +0200 Subject: [PATCH 04/37] draft --- pages/adding-new-application/index.md | 97 ++++++++++++++++++++++++++- 1 file changed, 94 insertions(+), 3 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 3064790e..81ab2b31 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -8,13 +8,97 @@ description: >- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. -When a user sets up a new Plural workspace in a git repository (we sometimes call that a **deployment repository**) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. +When a user sets up a new Plural workspace in a git repository (we sometimes call that a **deployment repository**) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. +Next, the user can install an application using the `plural bundle ` CLI command. +The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. +These inputs are saved in the `context.yaml` file. -Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using the values saved in the `context.yaml` using its templating engine. +For example, the `tree` of your deployment repository might look like this: +```console +❯ tree -L 1 . +. +├── bootstrap +├── cert-manager +├── console +├── dagster +├── postgres +├── terraform.tfstate +├── workspace.yaml +└── context.yaml +``` -For example, after the `plural build dagster` command the `tree` of the built Dagster application in your deployment repository might look like this: +The `workspace.yaml` might look like this: +``` +apiVersion: plural.sh/v1alpha1 +kind: ProjectManifest +metadata: + name: pluraldev +spec: + cluster: pluraldev + bucket: pluraldevsh-tf-state + project: "123456765432" + provider: aws + region: us-east-2 + owner: + email: plural-dev@pluraldev.sh + network: + subdomain: dev.plural.sh + pluraldns: false + bucketPrefix: pluraldev + context: {} +``` + +And the `context.yaml` like this. In the `configuration` section you can see the user input parametrization of the artifact. +``` +apiVersion: plural.sh/v1alpha1 +kind: Context +spec: + bundles: + - repository: dagster + name: dagster-aws + - repository: plural + name: plural-aws + - repository: console + name: console-aws + - repository: bootstrap + name: aws-efs + - repository: cert-manager + name: cert-manager-aws + - repository: ingress-nginx + name: ingress-nginx-aws + buckets: + - pluraldev-pluraldev-dagster + domains: + - console.dev.plural.sh + - dagster.dev.plural.sh + configuration: + bootstrap: + vpc_name: pluraldev + cert-manager: {} + console: + admin_name: admin + console_dns: console.dev.plural.sh + ... + repo_url: git@github.com:pluralsh/deployment-repo.git + dagster: + dagsterBucket: pluraldev-pluraldev-dagster + hostname: dagster.dev.plural.sh + ingress-nginx: {} + postgres: + wal_bucket: pluraldev-pluraldev-postgres-wal +``` + + +Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. +The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. +The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. +More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` injecting the values saved in the `context.yaml`. + +For example, after the `plural build dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: ```console +$ pwd +~/Repos/deployment-repository/dagster $ tree . . ├── build.hcl @@ -47,6 +131,9 @@ $ tree . └── outputs.tf ``` +Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. +Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. + ## Plural application artifacts As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. @@ -59,6 +146,8 @@ Some extra configuration files are necessary in the applications artifact for Pl As an example Dagster's artifact `tree` would look like this: ```console +$ pwd +~/Repos/plural-artifacts/dagster $ tree . . ├── Pluralfile @@ -127,6 +216,8 @@ $ tree . Let's disect what's going on here. +... + Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. From 1c74217b6efe1260763c1b35219c21fef92cf376 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 12:12:32 +0200 Subject: [PATCH 05/37] wording, improvements --- pages/adding-new-application/index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 81ab2b31..03f5014a 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -8,14 +8,14 @@ description: >- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. -When a user sets up a new Plural workspace in a git repository (we sometimes call that a **deployment repository**) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. +When a user sets up a new Plural workspace in a git repository (we shall call that a *deployment repository* from here on out) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. For example, the `tree` of your deployment repository might look like this: ```console -❯ tree -L 1 . +$ tree -L 1 . . ├── bootstrap ├── cert-manager @@ -28,7 +28,7 @@ For example, the `tree` of your deployment repository might look like this: ``` The `workspace.yaml` might look like this: -``` +```yaml apiVersion: plural.sh/v1alpha1 kind: ProjectManifest metadata: @@ -48,8 +48,8 @@ spec: context: {} ``` -And the `context.yaml` like this. In the `configuration` section you can see the user input parametrization of the artifact. -``` +And the `context.yaml` like this. In the `configuration` section you can see the user input parametrization of each artifact. +```yaml apiVersion: plural.sh/v1alpha1 kind: Context spec: @@ -89,10 +89,10 @@ spec: ``` -Next, the user runs `plural build` which will create a wrapper Helm chart and Terraform module. +Next, the user would run a `plural build` or `plural build ` which will create a wrapper Helm chart and Terraform module for that app in your deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. -The CLI will then generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` injecting the values saved in the `context.yaml`. +During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. +More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` (see below) injecting the values saved in the `context.yaml`. For example, after the `plural build dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: From 7a585fc55bc571af6109622c1a5e3b80cd36d17e Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 12:13:54 +0200 Subject: [PATCH 06/37] fix command --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 03f5014a..247bc40f 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -9,7 +9,7 @@ description: >- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. When a user sets up a new Plural workspace in a git repository (we shall call that a *deployment repository* from here on out) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. -Next, the user can install an application using the `plural bundle ` CLI command. +Next, the user can install an application using the `plural bundle install ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. From 2f8c862c0fd4446a8a39017f0f32821e945bbaf8 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 12:36:47 +0200 Subject: [PATCH 07/37] add input --- pages/adding-new-application/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 247bc40f..c853f158 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -8,7 +8,7 @@ description: >- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. -When a user sets up a new Plural workspace in a git repository (we shall call that a *deployment repository* from here on out) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. +When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle install ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. @@ -89,12 +89,12 @@ spec: ``` -Next, the user would run a `plural build` or `plural build ` which will create a wrapper Helm chart and Terraform module for that app in your deployment repository. +Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app in their deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` (see below) injecting the values saved in the `context.yaml`. +More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. -For example, after the `plural build dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: +For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: ```console $ pwd From a1895c1ef95407cd74c59c0a1eea2d19a9e62cd0 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 13:32:24 +0200 Subject: [PATCH 08/37] add some explanations --- pages/adding-new-application/index.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index c853f158..2d873fab 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -7,14 +7,23 @@ description: >- --- The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. +When you install an app from the marketplace into your cluster a number of things happen along the way across different components of Plural's architecture. +A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. + +## Install and Configure an Application + +In this section we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application using the `plural bundle install ` CLI command. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. -For example, the `tree` of your deployment repository might look like this: +For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: + ```console +$ pwd +~/Repos/deployment-repository $ tree -L 1 . . ├── bootstrap @@ -48,7 +57,7 @@ spec: context: {} ``` -And the `context.yaml` like this. In the `configuration` section you can see the user input parametrization of each artifact. +And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. ```yaml apiVersion: plural.sh/v1alpha1 kind: Context @@ -89,7 +98,7 @@ spec: ``` -Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app in their deployment repository. +Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. From a63abf304e9b56051a6652ca6c93ea210f1fac9f Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 15:09:53 +0200 Subject: [PATCH 09/37] wip --- pages/adding-new-application/index.md | 28 +++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 2d873fab..529d4aeb 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -101,7 +101,7 @@ spec: Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, the content of `default-values.yaml` will be a blend of both the artifact chart's default parametrization in its `values.yaml` and the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. +More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: @@ -143,7 +143,7 @@ $ tree . Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. -## Plural application artifacts +## Plural Application Artifact As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. Some extra configuration files are necessary in the applications artifact for Plural to be able to understand @@ -223,11 +223,31 @@ $ tree . └── vendor_images.yaml ``` -Let's disect what's going on here. +Let's disect this artifact's structure. -... +### Helm +The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. +Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions of them (if they're actively maintained, allow for required customization and fit Plural's quality standards) +as well as other helper charts, e.g. from Plural's module-library (#TODO: insert link). +If any additional resources are necessary, they can be added and templated in the same manner as in any other Helm chart. +Any default chart parametrization goes into your standard `values.yaml` file, e.g. resource requirements or limits, labels, annotation, entrypoint customizations, .... + +One thing that is unique about a Plural artifact's Helm chart is the ability to template values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. +It gives a means to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. +The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. +Another example would be configuration output from another Plural deployed application that lives in the same cluster, or configuration that you query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. +See [Templating](###Templating) for how powerful this additional templating layer can be. + +### Terraform +The `terraform` directory contains the app's provider-specific terraform modules any application component that does not live inside the cluster. +Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, if their deployment cannot be achieved by means of the artifact's Helm chart in a convenient manner. +> One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. + This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. + +### Plural +### Glue Files Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. From 9298ca88378f34a32166196b7357a068ff8a4178 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Wed, 4 Oct 2023 15:30:05 +0200 Subject: [PATCH 10/37] fix typos --- pages/adding-new-application/index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 529d4aeb..7a32c0fb 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -232,21 +232,24 @@ The `helm` directory contains the app's Helm chart as it will be available throu Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions of them (if they're actively maintained, allow for required customization and fit Plural's quality standards) as well as other helper charts, e.g. from Plural's module-library (#TODO: insert link). If any additional resources are necessary, they can be added and templated in the same manner as in any other Helm chart. -Any default chart parametrization goes into your standard `values.yaml` file, e.g. resource requirements or limits, labels, annotation, entrypoint customizations, .... +Any default chart parametrization goes into your standard `values.yaml` file, e.g. resource requirements or limits, labels, annotations, entrypoint customizations, and so on. One thing that is unique about a Plural artifact's Helm chart is the ability to template values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. It gives a means to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. -Another example would be configuration output from another Plural deployed application that lives in the same cluster, or configuration that you query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. +Another supported use case would be to pass configuration output from another Plural deployed applications that live in the same cluster, or configuration that you query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. See [Templating](###Templating) for how powerful this additional templating layer can be. ### Terraform The `terraform` directory contains the app's provider-specific terraform modules any application component that does not live inside the cluster. +One module for each cloud provider that the artifact will publish a bundle for. Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, if their deployment cannot be achieved by means of the artifact's Helm chart in a convenient manner. > One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. ### Plural +In the `plural` directory + ### Glue Files Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. From b7dc365802fb265dd52419f894d75243caa1504d Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Thu, 5 Oct 2023 15:49:01 +0200 Subject: [PATCH 11/37] wip --- pages/adding-new-application/index.md | 87 ++++++++++++++++++++++++--- 1 file changed, 78 insertions(+), 9 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 7a32c0fb..87634c7d 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -232,23 +232,92 @@ The `helm` directory contains the app's Helm chart as it will be available throu Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions of them (if they're actively maintained, allow for required customization and fit Plural's quality standards) as well as other helper charts, e.g. from Plural's module-library (#TODO: insert link). If any additional resources are necessary, they can be added and templated in the same manner as in any other Helm chart. -Any default chart parametrization goes into your standard `values.yaml` file, e.g. resource requirements or limits, labels, annotations, entrypoint customizations, and so on. +Any default chart parametrization goes into your standard `values.yaml` file, most prominently resource requirements or limits, labels, annotations, entrypoint customizations, and so on. -One thing that is unique about a Plural artifact's Helm chart is the ability to template values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. -It gives a means to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. +One thing that is unique about a Plural artifact's Helm chart is the ability to template in values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. +This enables us to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. -Another supported use case would be to pass configuration output from another Plural deployed applications that live in the same cluster, or configuration that you query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. +Another supported use case is to pass output from other Plural deployed applications that live in the same cluster, +or configuration that you can query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. See [Templating](###Templating) for how powerful this additional templating layer can be. ### Terraform -The `terraform` directory contains the app's provider-specific terraform modules any application component that does not live inside the cluster. -One module for each cloud provider that the artifact will publish a bundle for. -Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, if their deployment cannot be achieved by means of the artifact's Helm chart in a convenient manner. +The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. +For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. +Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, +if their deployment cannot be achieved by means of the artifact's Helm chart in a convenient manner. > One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. -### Plural -In the `plural` directory +### Plural Files +The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). +On the top-level directory of each artifact you'll also find a`repository.yaml`. + +The `repository.yaml` and recipe YAMLs are of particular importance and integral part of Plural's artifact packaging format. + +`repository.yaml`: +The metadata in this file informs the Plural API about the application this artifact encapsulates. +From its name, category and description, over where it can find the icon and docs to display in the marketplace, +over the notes template to prompt after installation, +to upstream repositories and homepage if applicable. + +`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. +Behind the scenes, every `plural bundle install` triggers the OIDC client creation, so you can use your Plural user to access an applications frontend. + +```yaml +name: dagster +description: A data orchestration platform for the development, production, and observation of data assets. +category: DATA +private: false +docs: plural/docs +icon: plural/icons/dagster-primary-mark.png +notes: plural/notes.tpl +gitUrl: https://github.com/dagster-io/dagster +homepage: https://dagster.io/ +oauthSettings: + uriFormat: https://{domain}/oauth2/callback + authMethod: POST +tags: +- tag: dag +- tag: data +- tag: data-pipelines +``` + + +`plural/receipes/dagster-aws.yaml`: + +```yaml +name: dagster-aws +description: Installs dagster on an aws eks cluster +provider: AWS +primary: true +dependencies: +- repo: bootstrap + name: aws-k8s +- repo: ingress-nginx + name: ingress-nginx-aws +- repo: postgres + name: aws-postgres +oidcSettings: + uriFormat: https://{domain}/oauth2/callback + authMethod: POST + domainKey: hostname +sections: +- name: dagster + configuration: + - name: dagsterBucket + type: BUCKET + documentation: s3 bucket for storing dagster logs + default: dagster + - name: hostname + type: DOMAIN + documentation: fqdn on which to deploy your dagster instance + items: + - type: TERRAFORM + name: aws + - type: HELM + name: dagster +``` ### Glue Files From e9118921963859f295ff8ace708a52f5d812ebf6 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 10:50:34 +0200 Subject: [PATCH 12/37] explain repository.yaml and recipe.yaml --- pages/adding-new-application/index.md | 47 +++++++++++++++++++-------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 87634c7d..d19114fe 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -6,7 +6,7 @@ description: >- guide outlines the process. --- -The two main functionalities that make the apps in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. +The two main functionalities that make the applications in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. When you install an app from the marketplace into your cluster a number of things happen along the way across different components of Plural's architecture. A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. @@ -15,7 +15,8 @@ A good unerstanding of an app's journey into your cluster will go a long way if In this section we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. -Next, the user can install an application using the `plural bundle install ` CLI command. +Next, the user can install an application bundle using the `plural bundle install ` CLI command. +> Most applications come with more than bundle, one for each targeted cloud provider. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. @@ -245,7 +246,7 @@ See [Templating](###Templating) for how powerful this additional templating laye The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, -if their deployment cannot be achieved by means of the artifact's Helm chart in a convenient manner. +if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. > One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. @@ -253,17 +254,9 @@ if their deployment cannot be achieved by means of the artifact's Helm chart in The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). On the top-level directory of each artifact you'll also find a`repository.yaml`. -The `repository.yaml` and recipe YAMLs are of particular importance and integral part of Plural's artifact packaging format. +The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format. `repository.yaml`: -The metadata in this file informs the Plural API about the application this artifact encapsulates. -From its name, category and description, over where it can find the icon and docs to display in the marketplace, -over the notes template to prompt after installation, -to upstream repositories and homepage if applicable. - -`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. -Behind the scenes, every `plural bundle install` triggers the OIDC client creation, so you can use your Plural user to access an applications frontend. - ```yaml name: dagster description: A data orchestration platform for the development, production, and observation of data assets. @@ -283,9 +276,19 @@ tags: - tag: data-pipelines ``` +The metadata in this file informs the Plural API about the application this artifact envelopes. +From its name, category and description, over where it can find the icon and docs to display in the marketplace, +the notes template to prompt after installation, to upstream repositories and homepage if applicable. + +`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. +> Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. + This happens, because every Client needs to be created before a `plural build` which then inputs the client info into the helm chart. + +The `private` flag controls whether the artifact's bundles are published publicly or privately. (#TODO what does this mean for the end user exactly, how does one get access to public repositories) +Once the artifact makes it into the `main` branch of the [Plural artifact repository](https://github.com/pluralsh/plural-artifacts) on a successful PR the Github Actions workflow will publish it and its components. -`plural/receipes/dagster-aws.yaml`: +`plural/receipes/dagster-aws.yaml`: ```yaml name: dagster-aws description: Installs dagster on an aws eks cluster @@ -319,6 +322,24 @@ sections: name: dagster ``` +The recipe file ties a bundle together, with one dedicated recipe per cloud provider. +It contains the bundle's parameter signature, metadata and dependency information that the Plural API needs to know. +Let's step through this file. + +- `provider` defines the targeted cloud provider of this recipe. +- The `primary` flag ... (#TODO ?) +- The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. + > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. +- Similar to `oauthSettings` in the `repository.yaml` `oidcSettings` specifies the same configuration at the bundle level. +- `sections[0].configuration` defines the user-provided values to prompt for during installation . + This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). + The Plural API has a built-in type checker that will validate the string's format passed to the configuration parameter against its type, e.g. to guarantee a valid domain name. + For examples on types check other Plural artifacts. + The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. + > A bundle can technically have multiple sections, but this feature's not yet used. +- `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories respectively that are part of this bundle. + + ### Glue Files Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. From c3d07b2256a63f789499ce84df5b0f13bba160d4 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:54:42 +0200 Subject: [PATCH 13/37] explain deps.yaml --- pages/adding-new-application/index.md | 93 +++++++++++++++++++++++++-- 1 file changed, 86 insertions(+), 7 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index d19114fe..38dd7d82 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -242,7 +242,55 @@ Another supported use case is to pass output from other Plural deployed applicat or configuration that you can query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. See [Templating](###Templating) for how powerful this additional templating layer can be. +Plural leverages dependency tracking of applications to achieve a high degree of resource efficiency and modularity. +Dependencies between artifacts are defined in the recipe files (see below). +At the level of the Helm charts and Terraform modules the cross-application dependencies are tracked in a dedicated `deps.yaml` at each chart's or module's top-level directory. +For example, for Dagster's Helm chart you would list required dependencies on: +- the `bootstrap` application's Helm chart +- the `ingress-nginx` application's Helm chart +- the `postrges` operator application's Helm chart +as well as optional dependencies the Dagster application's own Terraform modules to convey the intent that those are installed before the Helm chart. + +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + application: true + description: Deploys dagster crafted for the target cloud +spec: + breaking: true + dependencies: + - type: helm + name: bootstrap + repo: bootstrap + version: '>= 0.5.1' + - type: helm + name: ingress-nginx + repo: ingress-nginx + version: ">= 0.1.2" + - type: helm + name: postgres + repo: postgres + version: ">= 0.1.6" + - type: terraform + name: aws + repo: dagster + version: '>= 0.1.0' + optional: true + - type: terraform + name: azure + repo: dagster + version: '>= 0.1.0' + optional: true + - type: terraform + name: gcp + repo: dagster + version: '>= 0.1.0' + optional: true +``` + ### Terraform + The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, @@ -250,7 +298,26 @@ if their deployment cannot be achieved through the artifact's Helm chart in a co > One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. +Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about the order of dependencies on other modules. +``` +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + description: dagster aws setup + version: 0.1.2 +spec: + dependencies: + - name: aws-bootstrap + repo: bootstrap + type: terraform + version: '>= 0.1.1' + providers: + - aws +``` + + ### Plural Files + The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). On the top-level directory of each artifact you'll also find a`repository.yaml`. @@ -323,14 +390,15 @@ sections: ``` The recipe file ties a bundle together, with one dedicated recipe per cloud provider. -It contains the bundle's parameter signature, metadata and dependency information that the Plural API needs to know. +It informs the Plural API about the bundle's parameter signature, metadata, dependencies and sequence order of installations and upgrades. Let's step through this file. - `provider` defines the targeted cloud provider of this recipe. - The `primary` flag ... (#TODO ?) - The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. + This fullfills a > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. -- Similar to `oauthSettings` in the `repository.yaml` `oidcSettings` specifies the same configuration at the bundle level. +- Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. - `sections[0].configuration` defines the user-provided values to prompt for during installation . This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). The Plural API has a built-in type checker that will validate the string's format passed to the configuration parameter against its type, e.g. to guarantee a valid domain name. @@ -340,14 +408,25 @@ Let's step through this file. - `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories respectively that are part of this bundle. -### Glue Files - -Namely, a `deps.yaml` file which lists the dependencies of the Helm chart or Terraform module, and the `values.yaml.tpl` and `terraform.tfvars` file for Helm and Terraform respectively. +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + description: dagster aws setup + version: 0.1.2 +spec: + dependencies: + - name: aws-bootstrap + repo: bootstrap + type: terraform + version: '>= 0.1.1' + providers: + - aws -During a `plural build` inside the deployment repository the `values.yaml.tpl` and `terraform.tfvars` files of the artifact are run through the Plural templating engine, which is similar to that of Helm, and are used to generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module. +``` -### Templating +## Templating The next example is a snippet of the `values.yaml.tpl` file for Grafana: From c31e82567f9b67f508e2ad155e73f8bbd5047fcf Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:56:22 +0200 Subject: [PATCH 14/37] rm unfinished sentence --- pages/adding-new-application/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 38dd7d82..e149d019 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -396,7 +396,6 @@ Let's step through this file. - `provider` defines the targeted cloud provider of this recipe. - The `primary` flag ... (#TODO ?) - The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. - This fullfills a > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. - Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. - `sections[0].configuration` defines the user-provided values to prompt for during installation . From 581558cc0a9d4d4dd2d330ec09fe063db663742d Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:57:36 +0200 Subject: [PATCH 15/37] be more precise --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index e149d019..cae93a01 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -399,7 +399,7 @@ Let's step through this file. > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. - Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. - `sections[0].configuration` defines the user-provided values to prompt for during installation . - This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). + This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in templating expressions in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). The Plural API has a built-in type checker that will validate the string's format passed to the configuration parameter against its type, e.g. to guarantee a valid domain name. For examples on types check other Plural artifacts. The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. From 943d13e152de0e8d12931d6e8c0a61aff0ab386b Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:58:39 +0200 Subject: [PATCH 16/37] fix sentence --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index cae93a01..00ded60c 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -400,7 +400,7 @@ Let's step through this file. - Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. - `sections[0].configuration` defines the user-provided values to prompt for during installation . This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in templating expressions in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). - The Plural API has a built-in type checker that will validate the string's format passed to the configuration parameter against its type, e.g. to guarantee a valid domain name. + The Plural API has a built-in type checker that will validate any passed string's format against its type, e.g. to guarantee a valid domain name. For examples on types check other Plural artifacts. The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. > A bundle can technically have multiple sections, but this feature's not yet used. From b73ffd53598628ad3f42a8f77c54373bb994b8ca Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:59:20 +0200 Subject: [PATCH 17/37] reorder --- pages/adding-new-application/index.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 00ded60c..a2224fbf 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -401,11 +401,10 @@ Let's step through this file. - `sections[0].configuration` defines the user-provided values to prompt for during installation . This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in templating expressions in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). The Plural API has a built-in type checker that will validate any passed string's format against its type, e.g. to guarantee a valid domain name. - For examples on types check other Plural artifacts. + For examples on available types check other Plural artifacts. The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. - > A bundle can technically have multiple sections, but this feature's not yet used. - `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories respectively that are part of this bundle. - +> A bundle can technically have multiple sections, but this feature's not yet used. ```yaml apiVersion: plural.sh/v1alpha1 From 57f67fd7a17dffa41e7d912baf07cdfca421aad7 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 13:59:49 +0200 Subject: [PATCH 18/37] wording --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index a2224fbf..49131e2c 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -403,7 +403,7 @@ Let's step through this file. The Plural API has a built-in type checker that will validate any passed string's format against its type, e.g. to guarantee a valid domain name. For examples on available types check other Plural artifacts. The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. -- `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories respectively that are part of this bundle. +- `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories that are part of this bundle. > A bundle can technically have multiple sections, but this feature's not yet used. ```yaml From f85bd3b79948c9c5da99046986063cf4d871a3ce Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 14:11:01 +0200 Subject: [PATCH 19/37] wording --- pages/adding-new-application/index.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 49131e2c..5894adc8 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -230,9 +230,9 @@ Let's disect this artifact's structure. ### Helm The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. -Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions of them (if they're actively maintained, allow for required customization and fit Plural's quality standards) -as well as other helper charts, e.g. from Plural's module-library (#TODO: insert link). -If any additional resources are necessary, they can be added and templated in the same manner as in any other Helm chart. +Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) +as well as other helper charts, e.g. from Plural's [module-library](https://github.com/pluralsh/module-library). +If any additional resources are necessary, they can be added and templated in the same manner as with any other Helm chart. Any default chart parametrization goes into your standard `values.yaml` file, most prominently resource requirements or limits, labels, annotations, entrypoint customizations, and so on. One thing that is unique about a Plural artifact's Helm chart is the ability to template in values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. @@ -249,6 +249,7 @@ For example, for Dagster's Helm chart you would list required dependencies on: - the `bootstrap` application's Helm chart - the `ingress-nginx` application's Helm chart - the `postrges` operator application's Helm chart + as well as optional dependencies the Dagster application's own Terraform modules to convey the intent that those are installed before the Helm chart. ```yaml @@ -298,7 +299,7 @@ if their deployment cannot be achieved through the artifact's Helm chart in a co > One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. -Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about the order of dependencies on other modules. +Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules. ``` apiVersion: plural.sh/v1alpha1 kind: Dependencies @@ -344,8 +345,8 @@ tags: ``` The metadata in this file informs the Plural API about the application this artifact envelopes. -From its name, category and description, over where it can find the icon and docs to display in the marketplace, -the notes template to prompt after installation, to upstream repositories and homepage if applicable. +It will store its name, category and description, where it can find the icon and docs to display in the marketplace, +the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. `oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. > Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. From efcf620274f8986a584911740bbd6193559a48e7 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 15:44:19 +0200 Subject: [PATCH 20/37] explain primary --- pages/adding-new-application/index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 5894adc8..f1c13abc 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -344,6 +344,10 @@ tags: - tag: data-pipelines ``` +private recipe +private repository +sharing a private app between different plural accounts + The metadata in this file informs the Plural API about the application this artifact envelopes. It will store its name, category and description, where it can find the icon and docs to display in the marketplace, the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. @@ -395,7 +399,7 @@ It informs the Plural API about the bundle's parameter signature, metadata, depe Let's step through this file. - `provider` defines the targeted cloud provider of this recipe. -- The `primary` flag ... (#TODO ?) +- For every artifact one of the recipes can be marked as `primary` which will make it possible to install with simply a `plural bundle install ` (leaving out the ``). - The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. - Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. @@ -421,7 +425,6 @@ spec: version: '>= 0.1.1' providers: - aws - ``` From 9f3cc29a0f3e1b303042d64fc0ca7d1857ced1dc Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Mon, 9 Oct 2023 16:00:28 +0200 Subject: [PATCH 21/37] restructure --- .../application-artifact-package.md | 283 ++++++++++++ pages/adding-new-application/index.md | 417 ------------------ .../install-and-configure.md | 133 ++++++ pages/adding-new-application/templating.md | 130 ++++++ 4 files changed, 546 insertions(+), 417 deletions(-) create mode 100644 pages/adding-new-application/application-artifact-package.md create mode 100644 pages/adding-new-application/install-and-configure.md create mode 100644 pages/adding-new-application/templating.md diff --git a/pages/adding-new-application/application-artifact-package.md b/pages/adding-new-application/application-artifact-package.md new file mode 100644 index 00000000..13186f48 --- /dev/null +++ b/pages/adding-new-application/application-artifact-package.md @@ -0,0 +1,283 @@ +# Plural Application Artifact + +As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. +Some extra configuration files are necessary in the applications artifact for Plural to be able to understand +- the Helm charts and Terraform modules dependencies to run them through its templating engine +- dependencies on other Plural artifacts +- platform specific components and infrastructure configurations +- as well as Plural's own package and version specs. + +As an example Dagster's artifact `tree` would look like this: + +```console +$ pwd +~/Repos/plural-artifacts/dagster +$ tree . +. +├── Pluralfile +├── helm +│ └── dagster +│ ├── Chart.lock +│ ├── Chart.yaml +│ ├── README.md +│ ├── charts +│ │ ├── config-overlays-0.1.1.tgz +│ │ ├── dagster-1.4.10.tgz +│ │ ├── postgres-0.1.16.tgz +│ │ └── test-base-0.1.10.tgz +│ ├── deps.yaml +│ ├── runbooks +│ │ └── scaling-manual.xml +│ ├── templates +│ │ ├── _helpers.tpl +│ │ ├── oidc.yaml +│ │ ├── runbooks.yaml +│ │ └── secret.yaml +│ ├── values.yaml +│ └── values.yaml.tpl +├── plural +│ ├── docs +│ │ ├── private-ingress.md +│ │ └── user-code.md +│ ├── icons +│ │ ├── dagster-primary-mark.png +│ │ └── dagster.png +│ ├── notes.tpl +│ ├── recipes +│ │ ├── dagster-aws.yaml +│ │ ├── dagster-azure.yaml +│ │ └── dagster-gcp.yaml +│ └── tags +│ ├── helm +│ │ └── dagster.yaml +│ └── terraform +│ ├── aws.yaml +│ ├── azure.yaml +│ └── gcp.yaml +├── repository.yaml +├── terraform +│ ├── aws +│ │ ├── deps.yaml +│ │ ├── iam.tf +│ │ ├── main.tf +│ │ ├── outputs.tf +│ │ ├── postgres.tf +│ │ ├── terraform.tfvars +│ │ └── variables.tf +│ ├── azure +│ │ ├── deps.yaml +│ │ ├── main.tf +│ │ ├── terraform.tfvars +│ │ └── variables.tf +│ └── gcp +│ ├── deps.yaml +│ ├── main.tf +│ ├── outputs.tf +│ ├── terraform.tfvars +│ └── variables.tf +└── vendor_images.yaml +``` + +Let's disect this artifact's structure. + + +### Helm + +The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. +Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) +as well as other helper charts, e.g. from Plural's [module-library](https://github.com/pluralsh/module-library). +If any additional resources are necessary, they can be added and templated in the same manner as with any other Helm chart. +Any default chart parametrization goes into your standard `values.yaml` file, most prominently resource requirements or limits, labels, annotations, entrypoint customizations, and so on. + +One thing that is unique about a Plural artifact's Helm chart is the ability to template in values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. +This enables us to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. +The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. +Another supported use case is to pass output from other Plural deployed applications that live in the same cluster, +or configuration that you can query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. +See [Templating](###Templating) for how powerful this additional templating layer can be. + +Plural leverages dependency tracking of applications to achieve a high degree of resource efficiency and modularity. +Dependencies between artifacts are defined in the recipe files (see below). +At the level of the Helm charts and Terraform modules the cross-application dependencies are tracked in a dedicated `deps.yaml` at each chart's or module's top-level directory. +For example, for Dagster's Helm chart you would list required dependencies on: +- the `bootstrap` application's Helm chart +- the `ingress-nginx` application's Helm chart +- the `postrges` operator application's Helm chart + +as well as optional dependencies the Dagster application's own Terraform modules to convey the intent that those are installed before the Helm chart. + +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + application: true + description: Deploys dagster crafted for the target cloud +spec: + breaking: true + dependencies: + - type: helm + name: bootstrap + repo: bootstrap + version: '>= 0.5.1' + - type: helm + name: ingress-nginx + repo: ingress-nginx + version: ">= 0.1.2" + - type: helm + name: postgres + repo: postgres + version: ">= 0.1.6" + - type: terraform + name: aws + repo: dagster + version: '>= 0.1.0' + optional: true + - type: terraform + name: azure + repo: dagster + version: '>= 0.1.0' + optional: true + - type: terraform + name: gcp + repo: dagster + version: '>= 0.1.0' + optional: true +``` + +### Terraform + +The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. +For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. +Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, +if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. +> One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. + This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. + +Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules. +``` +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + description: dagster aws setup + version: 0.1.2 +spec: + dependencies: + - name: aws-bootstrap + repo: bootstrap + type: terraform + version: '>= 0.1.1' + providers: + - aws +``` + + +### Plural Files + +The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). +On the top-level directory of each artifact you'll also find a`repository.yaml`. + +The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format. + +`repository.yaml`: +```yaml +name: dagster +description: A data orchestration platform for the development, production, and observation of data assets. +category: DATA +private: false +docs: plural/docs +icon: plural/icons/dagster-primary-mark.png +notes: plural/notes.tpl +gitUrl: https://github.com/dagster-io/dagster +homepage: https://dagster.io/ +oauthSettings: + uriFormat: https://{domain}/oauth2/callback + authMethod: POST +tags: +- tag: dag +- tag: data +- tag: data-pipelines +``` + +private recipe +private repository +sharing a private app between different plural accounts + +The metadata in this file informs the Plural API about the application this artifact envelopes. +It will store its name, category and description, where it can find the icon and docs to display in the marketplace, +the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. + +`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. +> Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. + This happens, because every Client needs to be created before a `plural build` which then inputs the client info into the helm chart. + +The `private` flag controls whether the artifact's bundles are published publicly or privately. (#TODO what does this mean for the end user exactly, how does one get access to public repositories) +Once the artifact makes it into the `main` branch of the [Plural artifact repository](https://github.com/pluralsh/plural-artifacts) on a successful PR the Github Actions workflow will publish it and its components. + + +`plural/receipes/dagster-aws.yaml`: +```yaml +name: dagster-aws +description: Installs dagster on an aws eks cluster +provider: AWS +primary: true +dependencies: +- repo: bootstrap + name: aws-k8s +- repo: ingress-nginx + name: ingress-nginx-aws +- repo: postgres + name: aws-postgres +oidcSettings: + uriFormat: https://{domain}/oauth2/callback + authMethod: POST + domainKey: hostname +sections: +- name: dagster + configuration: + - name: dagsterBucket + type: BUCKET + documentation: s3 bucket for storing dagster logs + default: dagster + - name: hostname + type: DOMAIN + documentation: fqdn on which to deploy your dagster instance + items: + - type: TERRAFORM + name: aws + - type: HELM + name: dagster +``` + +The recipe file ties a bundle together, with one dedicated recipe per cloud provider. +It informs the Plural API about the bundle's parameter signature, metadata, dependencies and sequence order of installations and upgrades. +Let's step through this file. + +- `provider` defines the targeted cloud provider of this recipe. +- For every artifact one of the recipes can be marked as `primary` which will make it possible to install with simply a `plural bundle install ` (leaving out the ``). +- The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. + > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. +- Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. +- `sections[0].configuration` defines the user-provided values to prompt for during installation . + This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in templating expressions in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). + The Plural API has a built-in type checker that will validate any passed string's format against its type, e.g. to guarantee a valid domain name. + For examples on available types check other Plural artifacts. + The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. +- `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories that are part of this bundle. +> A bundle can technically have multiple sections, but this feature's not yet used. + +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Dependencies +metadata: + description: dagster aws setup + version: 0.1.2 +spec: + dependencies: + - name: aws-bootstrap + repo: bootstrap + type: terraform + version: '>= 0.1.1' + providers: + - aws +``` + diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index f1c13abc..135e6c9f 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -10,423 +10,6 @@ The two main functionalities that make the applications in Plural's marketplace When you install an app from the marketplace into your cluster a number of things happen along the way across different components of Plural's architecture. A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. -## Install and Configure an Application - -In this section we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. - -When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. -Next, the user can install an application bundle using the `plural bundle install ` CLI command. -> Most applications come with more than bundle, one for each targeted cloud provider. -The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. -These inputs are saved in the `context.yaml` file. - -For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: - -```console -$ pwd -~/Repos/deployment-repository -$ tree -L 1 . -. -├── bootstrap -├── cert-manager -├── console -├── dagster -├── postgres -├── terraform.tfstate -├── workspace.yaml -└── context.yaml -``` - -The `workspace.yaml` might look like this: -```yaml -apiVersion: plural.sh/v1alpha1 -kind: ProjectManifest -metadata: - name: pluraldev -spec: - cluster: pluraldev - bucket: pluraldevsh-tf-state - project: "123456765432" - provider: aws - region: us-east-2 - owner: - email: plural-dev@pluraldev.sh - network: - subdomain: dev.plural.sh - pluraldns: false - bucketPrefix: pluraldev - context: {} -``` - -And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. -```yaml -apiVersion: plural.sh/v1alpha1 -kind: Context -spec: - bundles: - - repository: dagster - name: dagster-aws - - repository: plural - name: plural-aws - - repository: console - name: console-aws - - repository: bootstrap - name: aws-efs - - repository: cert-manager - name: cert-manager-aws - - repository: ingress-nginx - name: ingress-nginx-aws - buckets: - - pluraldev-pluraldev-dagster - domains: - - console.dev.plural.sh - - dagster.dev.plural.sh - configuration: - bootstrap: - vpc_name: pluraldev - cert-manager: {} - console: - admin_name: admin - console_dns: console.dev.plural.sh - ... - repo_url: git@github.com:pluralsh/deployment-repo.git - dagster: - dagsterBucket: pluraldev-pluraldev-dagster - hostname: dagster.dev.plural.sh - ingress-nginx: {} - postgres: - wal_bucket: pluraldev-pluraldev-postgres-wal -``` - - -Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. -The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. -During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. - -For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: - -```console -$ pwd -~/Repos/deployment-repository/dagster -$ tree . -. -├── build.hcl -├── deploy.hcl -├── diff.hcl -├── helm -│ └── dagster -│ ├── Chart.lock -│ ├── Chart.yaml -│ ├── charts -│ │ └── dagster-0.1.44.tgz -│ ├── default-values.yaml -│ ├── templates -│ │ ├── NOTES.txt -│ │ ├── application.yaml -│ │ ├── license.yaml -│ │ └── secret.yaml -│ └── values.yaml -├── manifest.yaml -├── output.yaml -└── terraform - ├── aws - │ ├── deps.yaml - │ ├── iam.tf - │ ├── main.tf - │ ├── outputs.tf - │ ├── postgres.tf - │ └── variables.tf - ├── main.tf - └── outputs.tf -``` - -Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. -Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. - -## Plural Application Artifact - -As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. -Some extra configuration files are necessary in the applications artifact for Plural to be able to understand -- the Helm charts and Terraform modules dependencies to run them through its templating engine -- dependencies on other Plural artifacts -- platform specific components and infrastructure configurations -- as well as Plural's own package and version specs. - -As an example Dagster's artifact `tree` would look like this: - -```console -$ pwd -~/Repos/plural-artifacts/dagster -$ tree . -. -├── Pluralfile -├── helm -│ └── dagster -│ ├── Chart.lock -│ ├── Chart.yaml -│ ├── README.md -│ ├── charts -│ │ ├── config-overlays-0.1.1.tgz -│ │ ├── dagster-1.4.10.tgz -│ │ ├── postgres-0.1.16.tgz -│ │ └── test-base-0.1.10.tgz -│ ├── deps.yaml -│ ├── runbooks -│ │ └── scaling-manual.xml -│ ├── templates -│ │ ├── _helpers.tpl -│ │ ├── oidc.yaml -│ │ ├── runbooks.yaml -│ │ └── secret.yaml -│ ├── values.yaml -│ └── values.yaml.tpl -├── plural -│ ├── docs -│ │ ├── private-ingress.md -│ │ └── user-code.md -│ ├── icons -│ │ ├── dagster-primary-mark.png -│ │ └── dagster.png -│ ├── notes.tpl -│ ├── recipes -│ │ ├── dagster-aws.yaml -│ │ ├── dagster-azure.yaml -│ │ └── dagster-gcp.yaml -│ └── tags -│ ├── helm -│ │ └── dagster.yaml -│ └── terraform -│ ├── aws.yaml -│ ├── azure.yaml -│ └── gcp.yaml -├── repository.yaml -├── terraform -│ ├── aws -│ │ ├── deps.yaml -│ │ ├── iam.tf -│ │ ├── main.tf -│ │ ├── outputs.tf -│ │ ├── postgres.tf -│ │ ├── terraform.tfvars -│ │ └── variables.tf -│ ├── azure -│ │ ├── deps.yaml -│ │ ├── main.tf -│ │ ├── terraform.tfvars -│ │ └── variables.tf -│ └── gcp -│ ├── deps.yaml -│ ├── main.tf -│ ├── outputs.tf -│ ├── terraform.tfvars -│ └── variables.tf -└── vendor_images.yaml -``` - -Let's disect this artifact's structure. - - -### Helm - -The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. -Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) -as well as other helper charts, e.g. from Plural's [module-library](https://github.com/pluralsh/module-library). -If any additional resources are necessary, they can be added and templated in the same manner as with any other Helm chart. -Any default chart parametrization goes into your standard `values.yaml` file, most prominently resource requirements or limits, labels, annotations, entrypoint customizations, and so on. - -One thing that is unique about a Plural artifact's Helm chart is the ability to template in values from other parts of the infrastructure, that cannot be known ahead of deployment time, in the dedicated `values.yaml.tpl` file. -This enables us to parametrize values for resources that depend on application components that do not live in the cluster, but in your cloud account and that are deployed with terraform and not helm. -The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. -Another supported use case is to pass output from other Plural deployed applications that live in the same cluster, -or configuration that you can query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. -See [Templating](###Templating) for how powerful this additional templating layer can be. - -Plural leverages dependency tracking of applications to achieve a high degree of resource efficiency and modularity. -Dependencies between artifacts are defined in the recipe files (see below). -At the level of the Helm charts and Terraform modules the cross-application dependencies are tracked in a dedicated `deps.yaml` at each chart's or module's top-level directory. -For example, for Dagster's Helm chart you would list required dependencies on: -- the `bootstrap` application's Helm chart -- the `ingress-nginx` application's Helm chart -- the `postrges` operator application's Helm chart - -as well as optional dependencies the Dagster application's own Terraform modules to convey the intent that those are installed before the Helm chart. - -```yaml -apiVersion: plural.sh/v1alpha1 -kind: Dependencies -metadata: - application: true - description: Deploys dagster crafted for the target cloud -spec: - breaking: true - dependencies: - - type: helm - name: bootstrap - repo: bootstrap - version: '>= 0.5.1' - - type: helm - name: ingress-nginx - repo: ingress-nginx - version: ">= 0.1.2" - - type: helm - name: postgres - repo: postgres - version: ">= 0.1.6" - - type: terraform - name: aws - repo: dagster - version: '>= 0.1.0' - optional: true - - type: terraform - name: azure - repo: dagster - version: '>= 0.1.0' - optional: true - - type: terraform - name: gcp - repo: dagster - version: '>= 0.1.0' - optional: true -``` - -### Terraform - -The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. -For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. -Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, -if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. -> One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. - This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. - -Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules. -``` -apiVersion: plural.sh/v1alpha1 -kind: Dependencies -metadata: - description: dagster aws setup - version: 0.1.2 -spec: - dependencies: - - name: aws-bootstrap - repo: bootstrap - type: terraform - version: '>= 0.1.1' - providers: - - aws -``` - - -### Plural Files - -The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). -On the top-level directory of each artifact you'll also find a`repository.yaml`. - -The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format. - -`repository.yaml`: -```yaml -name: dagster -description: A data orchestration platform for the development, production, and observation of data assets. -category: DATA -private: false -docs: plural/docs -icon: plural/icons/dagster-primary-mark.png -notes: plural/notes.tpl -gitUrl: https://github.com/dagster-io/dagster -homepage: https://dagster.io/ -oauthSettings: - uriFormat: https://{domain}/oauth2/callback - authMethod: POST -tags: -- tag: dag -- tag: data -- tag: data-pipelines -``` - -private recipe -private repository -sharing a private app between different plural accounts - -The metadata in this file informs the Plural API about the application this artifact envelopes. -It will store its name, category and description, where it can find the icon and docs to display in the marketplace, -the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. - -`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. -> Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. - This happens, because every Client needs to be created before a `plural build` which then inputs the client info into the helm chart. - -The `private` flag controls whether the artifact's bundles are published publicly or privately. (#TODO what does this mean for the end user exactly, how does one get access to public repositories) -Once the artifact makes it into the `main` branch of the [Plural artifact repository](https://github.com/pluralsh/plural-artifacts) on a successful PR the Github Actions workflow will publish it and its components. - - -`plural/receipes/dagster-aws.yaml`: -```yaml -name: dagster-aws -description: Installs dagster on an aws eks cluster -provider: AWS -primary: true -dependencies: -- repo: bootstrap - name: aws-k8s -- repo: ingress-nginx - name: ingress-nginx-aws -- repo: postgres - name: aws-postgres -oidcSettings: - uriFormat: https://{domain}/oauth2/callback - authMethod: POST - domainKey: hostname -sections: -- name: dagster - configuration: - - name: dagsterBucket - type: BUCKET - documentation: s3 bucket for storing dagster logs - default: dagster - - name: hostname - type: DOMAIN - documentation: fqdn on which to deploy your dagster instance - items: - - type: TERRAFORM - name: aws - - type: HELM - name: dagster -``` - -The recipe file ties a bundle together, with one dedicated recipe per cloud provider. -It informs the Plural API about the bundle's parameter signature, metadata, dependencies and sequence order of installations and upgrades. -Let's step through this file. - -- `provider` defines the targeted cloud provider of this recipe. -- For every artifact one of the recipes can be marked as `primary` which will make it possible to install with simply a `plural bundle install ` (leaving out the ``). -- The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. - > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. -- Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. -- `sections[0].configuration` defines the user-provided values to prompt for during installation . - This is basically the signature of the bundle, it contains all required user-provided parameters that can be used in templating expressions in the `values.yaml.tpl` or in the terraform module (e.g. in the `.tfvars` file). - The Plural API has a built-in type checker that will validate any passed string's format against its type, e.g. to guarantee a valid domain name. - For examples on available types check other Plural artifacts. - The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. -- `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories that are part of this bundle. -> A bundle can technically have multiple sections, but this feature's not yet used. - -```yaml -apiVersion: plural.sh/v1alpha1 -kind: Dependencies -metadata: - description: dagster aws setup - version: 0.1.2 -spec: - dependencies: - - name: aws-bootstrap - repo: bootstrap - type: terraform - version: '>= 0.1.1' - providers: - - aws -``` - ## Templating diff --git a/pages/adding-new-application/install-and-configure.md b/pages/adding-new-application/install-and-configure.md new file mode 100644 index 00000000..1c547bbf --- /dev/null +++ b/pages/adding-new-application/install-and-configure.md @@ -0,0 +1,133 @@ +# Install and Configure an Application + +In this section we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. + +When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. +Next, the user can install an application bundle using the `plural bundle install ` CLI command. +> Most applications come with more than bundle, one for each targeted cloud provider. +The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. +These inputs are saved in the `context.yaml` file. + +For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: + +```console +$ pwd +~/Repos/deployment-repository +$ tree -L 1 . +. +├── bootstrap +├── cert-manager +├── console +├── dagster +├── postgres +├── terraform.tfstate +├── workspace.yaml +└── context.yaml +``` + +The `workspace.yaml` might look like this: +```yaml +apiVersion: plural.sh/v1alpha1 +kind: ProjectManifest +metadata: + name: pluraldev +spec: + cluster: pluraldev + bucket: pluraldevsh-tf-state + project: "123456765432" + provider: aws + region: us-east-2 + owner: + email: plural-dev@pluraldev.sh + network: + subdomain: dev.plural.sh + pluraldns: false + bucketPrefix: pluraldev + context: {} +``` + +And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Context +spec: + bundles: + - repository: dagster + name: dagster-aws + - repository: plural + name: plural-aws + - repository: console + name: console-aws + - repository: bootstrap + name: aws-efs + - repository: cert-manager + name: cert-manager-aws + - repository: ingress-nginx + name: ingress-nginx-aws + buckets: + - pluraldev-pluraldev-dagster + domains: + - console.dev.plural.sh + - dagster.dev.plural.sh + configuration: + bootstrap: + vpc_name: pluraldev + cert-manager: {} + console: + admin_name: admin + console_dns: console.dev.plural.sh + ... + repo_url: git@github.com:pluralsh/deployment-repo.git + dagster: + dagsterBucket: pluraldev-pluraldev-dagster + hostname: dagster.dev.plural.sh + ingress-nginx: {} + postgres: + wal_bucket: pluraldev-pluraldev-postgres-wal +``` + + +Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. +The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. +During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. +More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. + +For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: + +```console +$ pwd +~/Repos/deployment-repository/dagster +$ tree . +. +├── build.hcl +├── deploy.hcl +├── diff.hcl +├── helm +│ └── dagster +│ ├── Chart.lock +│ ├── Chart.yaml +│ ├── charts +│ │ └── dagster-0.1.44.tgz +│ ├── default-values.yaml +│ ├── templates +│ │ ├── NOTES.txt +│ │ ├── application.yaml +│ │ ├── license.yaml +│ │ └── secret.yaml +│ └── values.yaml +├── manifest.yaml +├── output.yaml +└── terraform + ├── aws + │ ├── deps.yaml + │ ├── iam.tf + │ ├── main.tf + │ ├── outputs.tf + │ ├── postgres.tf + │ └── variables.tf + ├── main.tf + └── outputs.tf +``` + +Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. +Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. diff --git a/pages/adding-new-application/templating.md b/pages/adding-new-application/templating.md new file mode 100644 index 00000000..d4723be7 --- /dev/null +++ b/pages/adding-new-application/templating.md @@ -0,0 +1,130 @@ +# Templating + +The next example is a snippet of the `values.yaml.tpl` file for Grafana: + +```yaml +grafana: + admin: + password: + { { dedupe . "grafana.grafana.admin.password" (randAlphaNum 14) } } + user: admin + ingress: + tls: + - hosts: + - { { .Values.hostname } } + secretName: grafana-tls + hosts: + - { { .Values.hostname } } +``` + +In the above example, the hostname a for Grafana that is saved in the `context.yaml` will be set in the ingress for Grafana by `{{ .Values.hostname }}`. It also showcases `dedupe`, which is one of the templating functions available in the Plural CLI. + +We are using the `dedupe` function so that a new random password for the Grafana admin is not generated if it has already been set. The reason `grafana.grafana.admin.password` is specified for the path, is because the CLI will create a wrapper Helm chart named `grafana` in a user's installation workspace. Please see [this section](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/#overriding-values-from-a-parent-chart) of the Helm docs for an explanation on how to pass values to a subchart. + +The next snippet shows a part of the `terraform.tfvars` for the AWS bootstrap terraform module: + +```hcl +vpc_name = {{ .Values.vpc_name | quote }} +cluster_name = {{ .Cluster | quote }} +``` + +Except for the user's application inputs from the `context.yaml` and the aforementioned `dedupe` function, Plural includes a lot of other values and functions that make it possible to simplify otherwise complex application configurations. + +### Templating reference + +Along with the standard functions available in the Go templating language, the following Plural specific functions can be used. + +##### Functions: + +| Function | Input (type) | Returned value | Description | +| --------------- | -------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| fileExists | Path (string) | Boolean | Checks if a file exists | +| pathJoin | Parts (\[]string) | String | Joins parts of a path | +| repoRoot | X | String | Returns the path of the installation repository | +| repoName | X | String | Returns the base path of repoRoot | +| repoUrl | X | String | Returns the `remote.origin.url` of the installation repository | +| branchName | X | String | Returns the name of the branch used for the installation repository | +| dumpConfig | X | String, Error | Returns the Plural CLI config | +| dumpAesKey | X | String, Error | Returns the AES keys used for the installation repository | +| readFile | Path (strong) | String | Returns the contents of a file as a string | +| readLine | Prompt (string) | String, Error | Prompts the user for input and returns the inputted string | +| readPassword | Prompt (string) | String, Error | Prompts the user for a password input and returns the inputted string | +| readLineDefault | Prompt, Default (string) | string, error | Prompts the user for input and returns the inputted string or the default value if input is empty | +| homeDir | Parts (\[]string) | string, error | Returns the path of the user's home directory with the path parts appended to it | +| knownHosts | X | string, error | Returns the contents of the `.ssh/known_hosts` file in the user's home directory | +| probe | Object (interface{}), Path (string) | interface{}, error | Checks if the object exists at the input path | +| dedupe | Object (interface{}), Path, Value (string) | string | Returns Object if it exists at the input Path, otherwise returns Value as a string | +| dedupeObj | Object (interface{}), Path (string), Value (interface{}) | interface{} | Returns Object if it exists at the input Path, otherwise returns Value as an interface{}. Used for complex data types | +| namespace | Name (string) | string | Returns the namespace for an application | +| secret | Namespace, Name (string) | map\[string]interface{} | Returns the content of a Kubernetes secret | +| importValue | Tool, Path (string) | string | Returns the value from another tool at the given path. An example use case is to use a value from the Terraform output in the Helm chart | +| toYaml | Value (interface{}) | string, error | Formats the input value as YAML | +| eabCredential | Cluster, Provider (string) | `api.EabCredential`, error | Creates a new set of EAB credentials in the Plural API for when using cert-manager with the Plural DNS | + +##### Template values: + +| Path | Description | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| .Values | The (recipe) values a user inputted for the given application | +| .Configuration | Allows for getting values from other applications stored in the `context.yaml` file. For example, to use the hostname configured for Grafana in another application `{{ .Configuration.grafana.hostname }}` can be used. | +| .License | The License for this application installation | +| .Config | The Plural CLI configuration object. Please see below for a detailed explanation | +| .OIDC | The OIDC configuration object for an application. Please see below for a detailed explanation | +| .Region | The cloud region configured for the workspace of the installation | +| .Project | The cloud project configured for the workspace of the installation | +| .Cluster | The name of the cluster configured for the workspace of the installation | +| .Provider | The name of the cloud provider configured for the workspace of the installation | +| .Context | The context map of the Plural cloud provider config for the workspace of the installation | +| .SMTP | The SMTP configuration of the workspace. Please see below for a detailed explanation | +| .Applications.HelmValues | Function that can be used to get the Helm values from another application. This differs from `.Configuration` as this can access all Helm values for an application, not only those stored in the `context.yaml`. For example, to use the generated Grafana admin password in another application you can do `{{ $grafanaValues := .Applications.HelmValues "grafana" }}` then `{{ $grafanaValues.grafana.grafana.admin.password }}`. | +| .Applications.TerraformValues | Function that can be used to get the Terraform outputs from another application. Usage is similar to `.Applications.HelmValues`. For example, to get the Terraform outputs of the bootstrap application you can use `{{ $bootstrapOutputs := .Applications.TerraformValues "bootstrap" }}` | +| .Acme.KeyId | The ACME key ID for to the application when using Plural DNS | +| .Acme.Secret | The ACME secret for to the application when using Plural DNS | + +##### `.Config` values: + +| Path | Description | +| ----------------------- | --------------------------------------------------------------- | +| .Config.Email | The user email address of the current CLI user | +| .Config.Token | The Plural token of the current CLI user | +| .Config.NamespacePrefix | The prefix to add to namespaces created for Plural applications | +| .Config.Endpoint | The Plural endpoint the CLI will use | +| .Config.LockProfile | | +| .Config.ReportErrors | If CLI errors should be reported back to Plural | +| .COnfig.metadata.Name | Name of the CLI config that is currently being used | + +##### `.OIDC` values: + +| Path | Description | +| ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| .OIDC.Id | The ID of the OIDC provider for the application | +| .OIDC.ClientId | The Client ID for the OIDC provider of the application | +| .OIDC.ClientSecret | The Client Secret for the OIDC provider of the application | +| .OIDC.RedirectUris | List of redirect URIs for the OIDC provider of the application | +| .OIDC.Bindings | List of bindings for the OIDC provider of the application. Please see below for a detailed explanation | +| .OIDC.Configuration.Issuer | The Issuer of the OIDC provider for the application | +| .OIDC.Configuration.AuthorizationEndpoint | The Authorization endpoint of the OIDC provider for the application | +| .OIDC.Configuration.TokenEndpoint | The Token endpoint of the OIDC provider for the application | +| .OIDC.Configuration.JwksUri | The JWKS URI of the OIDC provider for the application | +| .OIDC.Configuration.UserinfoEndpoint | The endpoint exposing user info of the OIDC provider for the application | + +##### `.OIDC.Bindings` values: + +| Path | Description | +| ------------------------- | --------------------------------------------------------------------- | +| .OIDC.Bindings.User.Id | The ID of the user that's bound to the application's OIDC provider | +| .OIDC.Bindings.User.Email | The Email of the user that's bound to the application's OIDC provider | +| .OIDC.Bindings.User.Name | The Name of the user that's bound to the application's OIDC provider | +| .OIDC.Bindings.Group.Id | The ID of the group that's bound to the application's OIDC provider | +| .OIDC.Bindings.Group.Name | The Name of the group that's bound to the application's OIDC provider | + +##### `.SMTP` values: + +| Path | Description | +| -------------- | --------------------------------------------- | +| .SMTP.Service | | +| .SMTP.Server | The SMTP server configured for the workspace | +| .SMTP.Port | The SMTP port configured for the workspace | +| .SMTP.Sender | The SMTP sender to use for the application | +| .SMTP.User | The username used to login to the SMTP server | +| .SMTP.Password | The password used to login to the SMTP server | From 77ad4920a68d1adc89923465098361c1b3eed889 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 14:09:30 +0200 Subject: [PATCH 22/37] publishing page --- pages/adding-new-application/publishing.md | 37 ++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 pages/adding-new-application/publishing.md diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md new file mode 100644 index 00000000..8d95827b --- /dev/null +++ b/pages/adding-new-application/publishing.md @@ -0,0 +1,37 @@ +# Publishing a Plural Artifact + +Every Plural account holder can act as a publisher to publish artifacts and in that way contribute open source applications to Plural's public marketplace. +If you don't want to publish an application publicly straight away and make it available to everybody you can first test it by releasing it privately whereby only users of your Plural account can install it. + +## Create a publisher +To publish an artifact you need to create a publisher with your Plural account admin first. + +1. Click on your profile on the bottom left on `app.plural.sh`. +2. Select `create publisher`. +3. Enter your information. +4. Hit save. + +## Publish an Artifact with the Plural CLI +When you're done with implementing your artifact and you have created a publisher you're ready to publish it. + +1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts). +2. Locate the `Pluralfile`. +3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. The arbument is case sensitive. E.g. if your publisher is called `Mypublisher`, it should look like this: + +``` +REPO dagster +ATTRIBUTES Mypublisher repository.yaml + +TF terraform/* +HELM helm/* +RECIPE plural/recipes/* +``` + +4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. + +## Public vs Private Artifacts +Artifacts can be published with a `private: true` or `private: false` setting in the `repository.yaml` (the `private` flag inside each recipe YAML needs to match the `private` flag in the `repository.yaml`). +Bundles from private artifacts can only be downloaded and installed by users from the same account the publisher was created in. +Artifacts created, and published privately by you, are not visible in the public marketplace. +By default only Plural developers can publish artifacts publicly. +If you want to publish your artifact publicly, too, you need to open a PR in the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts), so the Plural team can review your code. From 5dabce128facecde274b486cf6ea73970fd709c0 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 15:01:21 +0200 Subject: [PATCH 23/37] correct info about publishing --- .../application-artifact-package.md | 8 ++----- pages/adding-new-application/publishing.md | 22 +++++++++++++------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/pages/adding-new-application/application-artifact-package.md b/pages/adding-new-application/application-artifact-package.md index 13186f48..2f11e5e7 100644 --- a/pages/adding-new-application/application-artifact-package.md +++ b/pages/adding-new-application/application-artifact-package.md @@ -198,10 +198,6 @@ tags: - tag: data-pipelines ``` -private recipe -private repository -sharing a private app between different plural accounts - The metadata in this file informs the Plural API about the application this artifact envelopes. It will store its name, category and description, where it can find the icon and docs to display in the marketplace, the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. @@ -210,8 +206,8 @@ the notes template to prompt after installation, as well as links to any upstrea > Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. This happens, because every Client needs to be created before a `plural build` which then inputs the client info into the helm chart. -The `private` flag controls whether the artifact's bundles are published publicly or privately. (#TODO what does this mean for the end user exactly, how does one get access to public repositories) -Once the artifact makes it into the `main` branch of the [Plural artifact repository](https://github.com/pluralsh/plural-artifacts) on a successful PR the Github Actions workflow will publish it and its components. +The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push`. +It should be set to the same value as the `private` flag in inside the `repository.yaml`. `plural/receipes/dagster-aws.yaml`: diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md index 8d95827b..61f0d2a4 100644 --- a/pages/adding-new-application/publishing.md +++ b/pages/adding-new-application/publishing.md @@ -16,22 +16,30 @@ When you're done with implementing your artifact and you have created a publishe 1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts). 2. Locate the `Pluralfile`. -3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. The arbument is case sensitive. E.g. if your publisher is called `Mypublisher`, it should look like this: +3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. E.g. if your publisher is called `mypublisher`, it should look like this: ``` REPO dagster -ATTRIBUTES Mypublisher repository.yaml +ATTRIBUTES mypublisher repository.yaml TF terraform/* HELM helm/* RECIPE plural/recipes/* ``` -4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. +4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. E.g. for our dagster example +```console +$ plural apply -f Pluralfile +Setting attributes for dagster +upload progress 100% |███████████████████████████████████████████████████████████████| ( 0B/4.8 kB, ) +pushing terraform terraform/aws +upload progress 100% |███████████████████████████████████████████████████████████████| ( 0B/1.7 kB, ) +pushing helm helm/dagster...✓ +pushing recipe plural/recipes/dagster-aws.yaml✓ +``` ## Public vs Private Artifacts -Artifacts can be published with a `private: true` or `private: false` setting in the `repository.yaml` (the `private` flag inside each recipe YAML needs to match the `private` flag in the `repository.yaml`). -Bundles from private artifacts can only be downloaded and installed by users from the same account the publisher was created in. -Artifacts created, and published privately by you, are not visible in the public marketplace. -By default only Plural developers can publish artifacts publicly. +Artifacts can be published with a `private: true` or `private: false` setting in the `repository.yaml`. +Bundles from private artifacts are only visible for, and can only be downloaded and installed by, users from the same account the publisher was created in. If you want to publish your artifact publicly, too, you need to open a PR in the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts), so the Plural team can review your code. +Artifacts published publicly by you are also visible in the public marketplace. From 669c6dc952ae51e4b1131dc78322a7e836c154bc Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 15:05:38 +0200 Subject: [PATCH 24/37] format --- pages/adding-new-application/publishing.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md index 61f0d2a4..505ca6d3 100644 --- a/pages/adding-new-application/publishing.md +++ b/pages/adding-new-application/publishing.md @@ -1,9 +1,10 @@ # Publishing a Plural Artifact -Every Plural account holder can act as a publisher to publish artifacts and in that way contribute open source applications to Plural's public marketplace. +Every Plural account holder can act as a publisher to publish artifacts and in that way contribute open source applications to Plural's marketplace. If you don't want to publish an application publicly straight away and make it available to everybody you can first test it by releasing it privately whereby only users of your Plural account can install it. ## Create a publisher + To publish an artifact you need to create a publisher with your Plural account admin first. 1. Click on your profile on the bottom left on `app.plural.sh`. @@ -12,6 +13,7 @@ To publish an artifact you need to create a publisher with your Plural account a 4. Hit save. ## Publish an Artifact with the Plural CLI + When you're done with implementing your artifact and you have created a publisher you're ready to publish it. 1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts). @@ -39,6 +41,7 @@ pushing recipe plural/recipes/dagster-aws.yaml✓ ``` ## Public vs Private Artifacts + Artifacts can be published with a `private: true` or `private: false` setting in the `repository.yaml`. Bundles from private artifacts are only visible for, and can only be downloaded and installed by, users from the same account the publisher was created in. If you want to publish your artifact publicly, too, you need to open a PR in the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts), so the Plural team can review your code. From 0a9d511bb8e20292b96bf0550a7a065079a2a887 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 16:48:43 +0200 Subject: [PATCH 25/37] test --- .../application-artifact-package.md | 14 +- .../background-app-install.md | 135 ++++++++++++++++++ 2 files changed, 144 insertions(+), 5 deletions(-) create mode 100644 pages/adding-new-application/background-app-install.md diff --git a/pages/adding-new-application/application-artifact-package.md b/pages/adding-new-application/application-artifact-package.md index 2f11e5e7..0708f708 100644 --- a/pages/adding-new-application/application-artifact-package.md +++ b/pages/adding-new-application/application-artifact-package.md @@ -1,6 +1,10 @@ -# Plural Application Artifact +--- +title: Plural Application Artifact Structure +description: >- + In this guide we will lay out how a Plural artifact is constructed. +--- -As mentioned above, the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. +As mentioned in (Background on Application Installations)[/adding-new-application/background-app-install], the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. Some extra configuration files are necessary in the applications artifact for Plural to be able to understand - the Helm charts and Terraform modules dependencies to run them through its templating engine - dependencies on other Plural artifacts @@ -81,7 +85,7 @@ $ tree . Let's disect this artifact's structure. -### Helm +## Helm The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) @@ -144,7 +148,7 @@ spec: optional: true ``` -### Terraform +## Terraform The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. @@ -171,7 +175,7 @@ spec: ``` -### Plural Files +## Plural Files The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`). On the top-level directory of each artifact you'll also find a`repository.yaml`. diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md new file mode 100644 index 00000000..ccb2cca7 --- /dev/null +++ b/pages/adding-new-application/background-app-install.md @@ -0,0 +1,135 @@ +--- +title: Background on Application Installation +description: >- + In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. +--- + +When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. +Next, the user can install an application bundle using the `plural bundle install ` CLI command. +> Most applications come with more than bundle, one for each targeted cloud provider. +The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. +These inputs are saved in the `context.yaml` file. + +For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: + +```console +$ pwd +~/Repos/deployment-repository +$ tree -L 1 . +. +├── bootstrap +├── cert-manager +├── console +├── dagster +├── postgres +├── terraform.tfstate +├── workspace.yaml +└── context.yaml +``` + +The `workspace.yaml` might look like this: +```yaml +apiVersion: plural.sh/v1alpha1 +kind: ProjectManifest +metadata: + name: pluraldev +spec: + cluster: pluraldev + bucket: pluraldevsh-tf-state + project: "123456765432" + provider: aws + region: us-east-2 + owner: + email: plural-dev@pluraldev.sh + network: + subdomain: dev.plural.sh + pluraldns: false + bucketPrefix: pluraldev + context: {} +``` + +And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. +```yaml +apiVersion: plural.sh/v1alpha1 +kind: Context +spec: + bundles: + - repository: dagster + name: dagster-aws + - repository: plural + name: plural-aws + - repository: console + name: console-aws + - repository: bootstrap + name: aws-efs + - repository: cert-manager + name: cert-manager-aws + - repository: ingress-nginx + name: ingress-nginx-aws + buckets: + - pluraldev-pluraldev-dagster + domains: + - console.dev.plural.sh + - dagster.dev.plural.sh + configuration: + bootstrap: + vpc_name: pluraldev + cert-manager: {} + console: + admin_name: admin + console_dns: console.dev.plural.sh + ... + repo_url: git@github.com:pluralsh/deployment-repo.git + dagster: + dagsterBucket: pluraldev-pluraldev-dagster + hostname: dagster.dev.plural.sh + ingress-nginx: {} + postgres: + wal_bucket: pluraldev-pluraldev-postgres-wal +``` + + +Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. +The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. +During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. +More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. + +For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: + +```console +$ pwd +~/Repos/deployment-repository/dagster +$ tree . +. +├── build.hcl +├── deploy.hcl +├── diff.hcl +├── helm +│ └── dagster +│ ├── Chart.lock +│ ├── Chart.yaml +│ ├── charts +│ │ └── dagster-0.1.44.tgz +│ ├── default-values.yaml +│ ├── templates +│ │ ├── NOTES.txt +│ │ ├── application.yaml +│ │ ├── license.yaml +│ │ └── secret.yaml +│ └── values.yaml +├── manifest.yaml +├── output.yaml +└── terraform + ├── aws + │ ├── deps.yaml + │ ├── iam.tf + │ ├── main.tf + │ ├── outputs.tf + │ ├── postgres.tf + │ └── variables.tf + ├── main.tf + └── outputs.tf +``` + +Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. +Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. From 243751520852e3da2398d96a2b27ed5de6da9bdc Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:02:53 +0200 Subject: [PATCH 26/37] restructure, proof read --- .../background-app-install.md | 16 +- pages/adding-new-application/guide.md | 3 +- pages/adding-new-application/index.md | 137 +----------------- .../install-and-configure.md | 133 ----------------- ...ackage.md => plural-artifact-structure.md} | 34 +++-- pages/adding-new-application/publishing.md | 4 +- pages/adding-new-application/templating.md | 11 +- src/NavData.tsx | 6 +- 8 files changed, 47 insertions(+), 297 deletions(-) delete mode 100644 pages/adding-new-application/install-and-configure.md rename pages/adding-new-application/{application-artifact-package.md => plural-artifact-structure.md} (85%) diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md index ccb2cca7..f4cf5196 100644 --- a/pages/adding-new-application/background-app-install.md +++ b/pages/adding-new-application/background-app-install.md @@ -2,15 +2,16 @@ title: Background on Application Installation description: >- In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. + A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. --- When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application bundle using the `plural bundle install ` CLI command. -> Most applications come with more than bundle, one for each targeted cloud provider. +> Most applications come with more than one bundle, one for each targeted cloud provider. The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. -For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: +For example, let's have a look how the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like: ```console $ pwd @@ -27,7 +28,7 @@ $ tree -L 1 . └── context.yaml ``` -The `workspace.yaml` might look like this: +The `workspace.yaml` would look like this: ```yaml apiVersion: plural.sh/v1alpha1 kind: ProjectManifest @@ -48,7 +49,7 @@ spec: context: {} ``` -And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. +And the `context.yaml` like this. In the `spec.configuration` section you can see the recorded user input for each artifact. ```yaml apiVersion: plural.sh/v1alpha1 kind: Context @@ -88,13 +89,12 @@ spec: wal_bucket: pluraldev-pluraldev-postgres-wal ``` - -Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. +Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in the deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. +More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` see ([Templating](/adding-new-application/templating)) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. -For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: +For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository would look like this: ```console $ pwd diff --git a/pages/adding-new-application/guide.md b/pages/adding-new-application/guide.md index 81bf6cc7..6f7503c9 100644 --- a/pages/adding-new-application/guide.md +++ b/pages/adding-new-application/guide.md @@ -1,8 +1,7 @@ --- title: Guide description: >- - This guide is for anyone who wants to make an open-source application available on the - Plural marketplace. + This guide is for anyone who wants to make an open-source application available on the Plural marketplace and create a new Plural artifact. --- To add your own application, you'll need the Helm chart for your application and corresponding Terraform. diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 135e6c9f..4e6f8e34 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -6,138 +6,7 @@ description: >- guide outlines the process. --- -The two main functionalities that make the applications in Plural's marketplace work are dependency tracking between DevOps tools (Helm and Terraform) and templating. -When you install an app from the marketplace into your cluster a number of things happen along the way across different components of Plural's architecture. -A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. +If you want to contribute your own application to the Plural marketplace we invite you to create and publish your own artifacts. +In this section we guide you through the ins and outs of Plural's application artifacts, how you can build your own with the tools that Plural provides you, and how you can publish them for others to use. - -## Templating - -The next example is a snippet of the `values.yaml.tpl` file for Grafana: - -```yaml -grafana: - admin: - password: - { { dedupe . "grafana.grafana.admin.password" (randAlphaNum 14) } } - user: admin - ingress: - tls: - - hosts: - - { { .Values.hostname } } - secretName: grafana-tls - hosts: - - { { .Values.hostname } } -``` - -In the above example, the hostname a for Grafana that is saved in the `context.yaml` will be set in the ingress for Grafana by `{{ .Values.hostname }}`. It also showcases `dedupe`, which is one of the templating functions available in the Plural CLI. - -We are using the `dedupe` function so that a new random password for the Grafana admin is not generated if it has already been set. The reason `grafana.grafana.admin.password` is specified for the path, is because the CLI will create a wrapper Helm chart named `grafana` in a user's installation workspace. Please see [this section](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/#overriding-values-from-a-parent-chart) of the Helm docs for an explanation on how to pass values to a subchart. - -The next snippet shows a part of the `terraform.tfvars` for the AWS bootstrap terraform module: - -```hcl -vpc_name = {{ .Values.vpc_name | quote }} -cluster_name = {{ .Cluster | quote }} -``` - -Except for the user's application inputs from the `context.yaml` and the aforementioned `dedupe` function, Plural includes a lot of other values and functions that make it possible to simplify otherwise complex application configurations. - -### Templating reference - -Along with the standard functions available in the Go templating language, the following Plural specific functions can be used. - -##### Functions: - -| Function | Input (type) | Returned value | Description | -| --------------- | -------------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -| fileExists | Path (string) | Boolean | Checks if a file exists | -| pathJoin | Parts (\[]string) | String | Joins parts of a path | -| repoRoot | X | String | Returns the path of the installation repository | -| repoName | X | String | Returns the base path of repoRoot | -| repoUrl | X | String | Returns the `remote.origin.url` of the installation repository | -| branchName | X | String | Returns the name of the branch used for the installation repository | -| dumpConfig | X | String, Error | Returns the Plural CLI config | -| dumpAesKey | X | String, Error | Returns the AES keys used for the installation repository | -| readFile | Path (strong) | String | Returns the contents of a file as a string | -| readLine | Prompt (string) | String, Error | Prompts the user for input and returns the inputted string | -| readPassword | Prompt (string) | String, Error | Prompts the user for a password input and returns the inputted string | -| readLineDefault | Prompt, Default (string) | string, error | Prompts the user for input and returns the inputted string or the default value if input is empty | -| homeDir | Parts (\[]string) | string, error | Returns the path of the user's home directory with the path parts appended to it | -| knownHosts | X | string, error | Returns the contents of the `.ssh/known_hosts` file in the user's home directory | -| probe | Object (interface{}), Path (string) | interface{}, error | Checks if the object exists at the input path | -| dedupe | Object (interface{}), Path, Value (string) | string | Returns Object if it exists at the input Path, otherwise returns Value as a string | -| dedupeObj | Object (interface{}), Path (string), Value (interface{}) | interface{} | Returns Object if it exists at the input Path, otherwise returns Value as an interface{}. Used for complex data types | -| namespace | Name (string) | string | Returns the namespace for an application | -| secret | Namespace, Name (string) | map\[string]interface{} | Returns the content of a Kubernetes secret | -| importValue | Tool, Path (string) | string | Returns the value from another tool at the given path. An example use case is to use a value from the Terraform output in the Helm chart | -| toYaml | Value (interface{}) | string, error | Formats the input value as YAML | -| eabCredential | Cluster, Provider (string) | `api.EabCredential`, error | Creates a new set of EAB credentials in the Plural API for when using cert-manager with the Plural DNS | - -##### Template values: - -| Path | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| .Values | The (recipe) values a user inputted for the given application | -| .Configuration | Allows for getting values from other applications stored in the `context.yaml` file. For example, to use the hostname configured for Grafana in another application `{{ .Configuration.grafana.hostname }}` can be used. | -| .License | The License for this application installation | -| .Config | The Plural CLI configuration object. Please see below for a detailed explanation | -| .OIDC | The OIDC configuration object for an application. Please see below for a detailed explanation | -| .Region | The cloud region configured for the workspace of the installation | -| .Project | The cloud project configured for the workspace of the installation | -| .Cluster | The name of the cluster configured for the workspace of the installation | -| .Provider | The name of the cloud provider configured for the workspace of the installation | -| .Context | The context map of the Plural cloud provider config for the workspace of the installation | -| .SMTP | The SMTP configuration of the workspace. Please see below for a detailed explanation | -| .Applications.HelmValues | Function that can be used to get the Helm values from another application. This differs from `.Configuration` as this can access all Helm values for an application, not only those stored in the `context.yaml`. For example, to use the generated Grafana admin password in another application you can do `{{ $grafanaValues := .Applications.HelmValues "grafana" }}` then `{{ $grafanaValues.grafana.grafana.admin.password }}`. | -| .Applications.TerraformValues | Function that can be used to get the Terraform outputs from another application. Usage is similar to `.Applications.HelmValues`. For example, to get the Terraform outputs of the bootstrap application you can use `{{ $bootstrapOutputs := .Applications.TerraformValues "bootstrap" }}` | -| .Acme.KeyId | The ACME key ID for to the application when using Plural DNS | -| .Acme.Secret | The ACME secret for to the application when using Plural DNS | - -##### `.Config` values: - -| Path | Description | -| ----------------------- | --------------------------------------------------------------- | -| .Config.Email | The user email address of the current CLI user | -| .Config.Token | The Plural token of the current CLI user | -| .Config.NamespacePrefix | The prefix to add to namespaces created for Plural applications | -| .Config.Endpoint | The Plural endpoint the CLI will use | -| .Config.LockProfile | | -| .Config.ReportErrors | If CLI errors should be reported back to Plural | -| .COnfig.metadata.Name | Name of the CLI config that is currently being used | - -##### `.OIDC` values: - -| Path | Description | -| ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | -| .OIDC.Id | The ID of the OIDC provider for the application | -| .OIDC.ClientId | The Client ID for the OIDC provider of the application | -| .OIDC.ClientSecret | The Client Secret for the OIDC provider of the application | -| .OIDC.RedirectUris | List of redirect URIs for the OIDC provider of the application | -| .OIDC.Bindings | List of bindings for the OIDC provider of the application. Please see below for a detailed explanation | -| .OIDC.Configuration.Issuer | The Issuer of the OIDC provider for the application | -| .OIDC.Configuration.AuthorizationEndpoint | The Authorization endpoint of the OIDC provider for the application | -| .OIDC.Configuration.TokenEndpoint | The Token endpoint of the OIDC provider for the application | -| .OIDC.Configuration.JwksUri | The JWKS URI of the OIDC provider for the application | -| .OIDC.Configuration.UserinfoEndpoint | The endpoint exposing user info of the OIDC provider for the application | - -##### `.OIDC.Bindings` values: - -| Path | Description | -| ------------------------- | --------------------------------------------------------------------- | -| .OIDC.Bindings.User.Id | The ID of the user that's bound to the application's OIDC provider | -| .OIDC.Bindings.User.Email | The Email of the user that's bound to the application's OIDC provider | -| .OIDC.Bindings.User.Name | The Name of the user that's bound to the application's OIDC provider | -| .OIDC.Bindings.Group.Id | The ID of the group that's bound to the application's OIDC provider | -| .OIDC.Bindings.Group.Name | The Name of the group that's bound to the application's OIDC provider | - -##### `.SMTP` values: - -| Path | Description | -| -------------- | --------------------------------------------- | -| .SMTP.Service | | -| .SMTP.Server | The SMTP server configured for the workspace | -| .SMTP.Port | The SMTP port configured for the workspace | -| .SMTP.Sender | The SMTP sender to use for the application | -| .SMTP.User | The username used to login to the SMTP server | -| .SMTP.Password | The password used to login to the SMTP server | +We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program). \ No newline at end of file diff --git a/pages/adding-new-application/install-and-configure.md b/pages/adding-new-application/install-and-configure.md deleted file mode 100644 index 1c547bbf..00000000 --- a/pages/adding-new-application/install-and-configure.md +++ /dev/null @@ -1,133 +0,0 @@ -# Install and Configure an Application - -In this section we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. - -When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. -Next, the user can install an application bundle using the `plural bundle install ` CLI command. -> Most applications come with more than bundle, one for each targeted cloud provider. -The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. -These inputs are saved in the `context.yaml` file. - -For example, the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like this: - -```console -$ pwd -~/Repos/deployment-repository -$ tree -L 1 . -. -├── bootstrap -├── cert-manager -├── console -├── dagster -├── postgres -├── terraform.tfstate -├── workspace.yaml -└── context.yaml -``` - -The `workspace.yaml` might look like this: -```yaml -apiVersion: plural.sh/v1alpha1 -kind: ProjectManifest -metadata: - name: pluraldev -spec: - cluster: pluraldev - bucket: pluraldevsh-tf-state - project: "123456765432" - provider: aws - region: us-east-2 - owner: - email: plural-dev@pluraldev.sh - network: - subdomain: dev.plural.sh - pluraldns: false - bucketPrefix: pluraldev - context: {} -``` - -And the `context.yaml` like this. In the `spec.configuration` section you can see the user input parametrization of each artifact. -```yaml -apiVersion: plural.sh/v1alpha1 -kind: Context -spec: - bundles: - - repository: dagster - name: dagster-aws - - repository: plural - name: plural-aws - - repository: console - name: console-aws - - repository: bootstrap - name: aws-efs - - repository: cert-manager - name: cert-manager-aws - - repository: ingress-nginx - name: ingress-nginx-aws - buckets: - - pluraldev-pluraldev-dagster - domains: - - console.dev.plural.sh - - dagster.dev.plural.sh - configuration: - bootstrap: - vpc_name: pluraldev - cert-manager: {} - console: - admin_name: admin - console_dns: console.dev.plural.sh - ... - repo_url: git@github.com:pluralsh/deployment-repo.git - dagster: - dagsterBucket: pluraldev-pluraldev-dagster - hostname: dagster.dev.plural.sh - ingress-nginx: {} - postgres: - wal_bucket: pluraldev-pluraldev-postgres-wal -``` - - -Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in their deployment repository. -The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. -During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see below) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. - -For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository might look like this: - -```console -$ pwd -~/Repos/deployment-repository/dagster -$ tree . -. -├── build.hcl -├── deploy.hcl -├── diff.hcl -├── helm -│ └── dagster -│ ├── Chart.lock -│ ├── Chart.yaml -│ ├── charts -│ │ └── dagster-0.1.44.tgz -│ ├── default-values.yaml -│ ├── templates -│ │ ├── NOTES.txt -│ │ ├── application.yaml -│ │ ├── license.yaml -│ │ └── secret.yaml -│ └── values.yaml -├── manifest.yaml -├── output.yaml -└── terraform - ├── aws - │ ├── deps.yaml - │ ├── iam.tf - │ ├── main.tf - │ ├── outputs.tf - │ ├── postgres.tf - │ └── variables.tf - ├── main.tf - └── outputs.tf -``` - -Here you can see the wrapper Helm chart in `./helm/dagster` around the `./helm/dagster/charts/dagster-0.1.44.tgz`, i.e. the artifact's Helm chart that the Plural CLI downloads for you. -Similarly the wrapper Terraform module in `./terraform` will contain a local copy of the artifact's Terraform module inside `./terraform/aws`. diff --git a/pages/adding-new-application/application-artifact-package.md b/pages/adding-new-application/plural-artifact-structure.md similarity index 85% rename from pages/adding-new-application/application-artifact-package.md rename to pages/adding-new-application/plural-artifact-structure.md index 0708f708..e4e7aec1 100644 --- a/pages/adding-new-application/application-artifact-package.md +++ b/pages/adding-new-application/plural-artifact-structure.md @@ -1,17 +1,18 @@ --- -title: Plural Application Artifact Structure +title: Plural Artifact Structure description: >- In this guide we will lay out how a Plural artifact is constructed. --- -As mentioned in (Background on Application Installations)[/adding-new-application/background-app-install], the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user defined values for that installation. -Some extra configuration files are necessary in the applications artifact for Plural to be able to understand +As mentioned in [Background on Application Installations](/adding-new-application/background-app-install), +the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user provided values for that installation. +Some extra configuration files are necessary in the applications artifact for the Plural API to be able to understand: - the Helm charts and Terraform modules dependencies to run them through its templating engine - dependencies on other Plural artifacts - platform specific components and infrastructure configurations - as well as Plural's own package and version specs. -As an example Dagster's artifact `tree` would look like this: +As an example, Dagster's artifact `tree` would look like this: ```console $ pwd @@ -87,9 +88,9 @@ Let's disect this artifact's structure. ## Helm -The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the app's Kubernetes components into your cluster. -Many apps in Plural's marketplace define Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) -as well as other helper charts, e.g. from Plural's [module-library](https://github.com/pluralsh/module-library). +The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the Kubernetes components into your cluster. +Many artifacts define the Helm charts in terms of their upstream open source versions (if they're actively maintained, allow for required customization and fit Plural's quality standards) +as well as other helper charts, e.g. from Plural's [Module Library](/adding-new-application/module-library). If any additional resources are necessary, they can be added and templated in the same manner as with any other Helm chart. Any default chart parametrization goes into your standard `values.yaml` file, most prominently resource requirements or limits, labels, annotations, entrypoint customizations, and so on. @@ -98,17 +99,17 @@ This enables us to parametrize values for resources that depend on application c The ARN of an AWS role or bucket, or VPC subnet ids are common examples for this. Another supported use case is to pass output from other Plural deployed applications that live in the same cluster, or configuration that you can query from the Plural API, e.g. OIDC config if you're using Plural as an OIDC provider for your apps, too. -See [Templating](###Templating) for how powerful this additional templating layer can be. +See [Templating](/adding-new-application/templating) for how powerful this additional templating layer can be. -Plural leverages dependency tracking of applications to achieve a high degree of resource efficiency and modularity. +Plural leverages dependency tracking of applications to achieve a high degree of resource efficiency and deduplication. Dependencies between artifacts are defined in the recipe files (see below). -At the level of the Helm charts and Terraform modules the cross-application dependencies are tracked in a dedicated `deps.yaml` at each chart's or module's top-level directory. +Dependencies are also tracked between the Helm charts and Terraform modules of other applications in a dedicated `deps.yaml` file in each chart's or module's directory. For example, for Dagster's Helm chart you would list required dependencies on: - the `bootstrap` application's Helm chart - the `ingress-nginx` application's Helm chart - the `postrges` operator application's Helm chart -as well as optional dependencies the Dagster application's own Terraform modules to convey the intent that those are installed before the Helm chart. +as well as optional dependencies on Dagster's own Terraform modules to convey intent that those are installed before the Helm chart. ```yaml apiVersion: plural.sh/v1alpha1 @@ -151,10 +152,10 @@ spec: ## Terraform The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. -For each cloud provider that the artifact offers a bundle for there will be one under the related directory name. +For each cloud provider, that the artifact offers a bundle for, there will be one under the related directory name. Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. -> One peculiarity about the terraform modules is that at the very least they need to contain the Kubernetes namespace for your application. +> One peculiarity about the Terraform modules is that, at the very least, they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules. @@ -204,11 +205,11 @@ tags: The metadata in this file informs the Plural API about the application this artifact envelopes. It will store its name, category and description, where it can find the icon and docs to display in the marketplace, -the notes template to prompt after installation, as well as links to any upstream git repository or homepage if applicable. +the notes template to prompt after installation, as well as links to any upstream git repository or homepage (if applicable). `oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it. > Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. - This happens, because every Client needs to be created before a `plural build` which then inputs the client info into the helm chart. + This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart. The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push`. It should be set to the same value as the `private` flag in inside the `repository.yaml`. @@ -253,7 +254,7 @@ It informs the Plural API about the bundle's parameter signature, metadata, depe Let's step through this file. - `provider` defines the targeted cloud provider of this recipe. -- For every artifact one of the recipes can be marked as `primary` which will make it possible to install with simply a `plural bundle install ` (leaving out the ``). +- For every artifact one of the recipes can be marked as `primary` which will make it possible to simply install with a `plural bundle install ` (leaving out the ``). - The apps listed in `dependencies` tell Plural on which other Plural bundles this bundle depends on. > Most bundles depend on the installation of other Plural applications. For example, every bundle will at least depend on the bootstrap application that packages the cluster itself. - Similar to `oauthSettings` in the `repository.yaml`, `oidcSettings` in the recipe YAML should specify the same configuration at the bundle level. @@ -263,6 +264,7 @@ Let's step through this file. For examples on available types check other Plural artifacts. The Plural CLI will store the passed values in the according section in the `context.yaml` as discussed above. - `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories that are part of this bundle. + > A bundle can technically have multiple sections, but this feature's not yet used. ```yaml diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md index 505ca6d3..a8597fda 100644 --- a/pages/adding-new-application/publishing.md +++ b/pages/adding-new-application/publishing.md @@ -1,4 +1,6 @@ -# Publishing a Plural Artifact +--- +title: Publishing a Plural Artifact +--- Every Plural account holder can act as a publisher to publish artifacts and in that way contribute open source applications to Plural's marketplace. If you don't want to publish an application publicly straight away and make it available to everybody you can first test it by releasing it privately whereby only users of your Plural account can install it. diff --git a/pages/adding-new-application/templating.md b/pages/adding-new-application/templating.md index d4723be7..721d8d2f 100644 --- a/pages/adding-new-application/templating.md +++ b/pages/adding-new-application/templating.md @@ -1,6 +1,13 @@ -# Templating +--- +title: Templating +description: >- + The two main functionalities that make Plural work are dependency tracking between DevOps tools (Helm and Terraform) and templating. + This guide serves as a reference for how you can leverage this feature. +--- -The next example is a snippet of the `values.yaml.tpl` file for Grafana: +In [Plural Artifact Structure](/adding-new-application/plural-artifact-structure) we layed out how an artifact can be paremtrized. + +The example is a snippet of the `values.yaml.tpl` file for Grafana: ```yaml grafana: diff --git a/src/NavData.tsx b/src/NavData.tsx index 44970c23..d79a77bf 100644 --- a/src/NavData.tsx +++ b/src/NavData.tsx @@ -177,7 +177,11 @@ const rootNavData: NavMenu = deepFreeze([ href: '/adding-new-application', title: 'Contribute a New Application to the Plural Catalog', sections: [ - { href: '/adding-new-application/guide', title: 'Guide' }, + { href: '/adding-new-application/background-app-install', title: 'Background on Application Installations'}, + { href: '/adding-new-application/application-artifact-package', title: 'Plural Artifact Structure'}, + { href: '/adding-new-application/templating', title: 'Plural Artifact Templating'}, + { href: '/adding-new-application/guide', title: 'Guide on Creating a New Application Artifact' }, + { href: '/adding-new-application/publishing', title: 'Guide on Publishing a Plural Artifact'}, { href: '/adding-new-application/plural-custom-resources', title: 'Plural Custom Resources', From b63e519d37bead9df89d30bb5b149bc2f6575e55 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:04:46 +0200 Subject: [PATCH 27/37] shorten desc --- pages/adding-new-application/background-app-install.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md index f4cf5196..e8b616a1 100644 --- a/pages/adding-new-application/background-app-install.md +++ b/pages/adding-new-application/background-app-install.md @@ -1,10 +1,11 @@ --- -title: Background on Application Installation +title: Background on Application Installations description: >- - In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. A good unerstanding of an app's journey into your cluster will go a long way if you want to contribute your own application to the marketplace. --- +In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app. + When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application bundle using the `plural bundle install ` CLI command. > Most applications come with more than one bundle, one for each targeted cloud provider. From f993bd72afb8a0c62919d2bdb2f48cedbef3578c Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:06:20 +0200 Subject: [PATCH 28/37] fix parenthesis --- pages/adding-new-application/background-app-install.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md index e8b616a1..9b861ac4 100644 --- a/pages/adding-new-application/background-app-install.md +++ b/pages/adding-new-application/background-app-install.md @@ -93,7 +93,7 @@ spec: Next, the user would run `plural build` or `plural build --only ` which will create a wrapper Helm chart and Terraform module for that app under a dedicated directory in the deployment repository. The wrapper Helm chart and Terraform module depend on the application Helm chart(s) and Terraform module(s) it gets from the application's artifact repository via the Plural API. During this step the CLI will generate the `default-values.yaml` for the wrapper helm chart and `main.tf` for the wrapper Terraform module using its templating engine. -More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` see ([Templating](/adding-new-application/templating)) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. +More precisely, `default-values.yaml` will contain the interpolated templated values from its `values.yaml.tpl` (see [Templating](/adding-new-application/templating)) that are derived by injecting the values saved in the `context.yaml` at `spec.configuration`. For example, after the `plural build --only dagster` command, the `tree` of the built Dagster application in your deployment repository would look like this: From 1237d4e878a1ccfb5c36e7a897d3eaf5a168ba9b Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:07:57 +0200 Subject: [PATCH 29/37] highlight bounty --- pages/adding-new-application/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/index.md b/pages/adding-new-application/index.md index 4e6f8e34..38de0b4a 100644 --- a/pages/adding-new-application/index.md +++ b/pages/adding-new-application/index.md @@ -9,4 +9,4 @@ description: >- If you want to contribute your own application to the Plural marketplace we invite you to create and publish your own artifacts. In this section we guide you through the ins and outs of Plural's application artifacts, how you can build your own with the tools that Plural provides you, and how you can publish them for others to use. -We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program). \ No newline at end of file +> We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program). \ No newline at end of file From 147e0335a387b3c2447eeca41d05e7f6c95e2cb6 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:10:03 +0200 Subject: [PATCH 30/37] fix list --- pages/adding-new-application/publishing.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md index a8597fda..b57ea24f 100644 --- a/pages/adding-new-application/publishing.md +++ b/pages/adding-new-application/publishing.md @@ -7,7 +7,7 @@ If you don't want to publish an application publicly straight away and make it a ## Create a publisher -To publish an artifact you need to create a publisher with your Plural account admin first. +To publish an artifact you first need to create a publisher with your Plural account admin. 1. Click on your profile on the bottom left on `app.plural.sh`. 2. Select `create publisher`. @@ -21,7 +21,6 @@ When you're done with implementing your artifact and you have created a publishe 1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts). 2. Locate the `Pluralfile`. 3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. E.g. if your publisher is called `mypublisher`, it should look like this: - ``` REPO dagster ATTRIBUTES mypublisher repository.yaml @@ -30,7 +29,6 @@ TF terraform/* HELM helm/* RECIPE plural/recipes/* ``` - 4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. E.g. for our dagster example ```console $ plural apply -f Pluralfile From be4e5f61db9c12256928115b65e04645f1f6defb Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:15:55 +0200 Subject: [PATCH 31/37] match title with navdata --- pages/adding-new-application/guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/guide.md b/pages/adding-new-application/guide.md index 6f7503c9..93fb1c6d 100644 --- a/pages/adding-new-application/guide.md +++ b/pages/adding-new-application/guide.md @@ -1,5 +1,5 @@ --- -title: Guide +title: Guide on Creating a New Application Artifact description: >- This guide is for anyone who wants to make an open-source application available on the Plural marketplace and create a new Plural artifact. --- From 027d4a5530987399b7b41714bc5bc3e970d3f70a Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:18:32 +0200 Subject: [PATCH 32/37] fix indent --- pages/adding-new-application/background-app-install.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md index 9b861ac4..a5818fa7 100644 --- a/pages/adding-new-application/background-app-install.md +++ b/pages/adding-new-application/background-app-install.md @@ -8,7 +8,9 @@ In this guide we will lay out how your user provided values tie in with the depl When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under. Next, the user can install an application bundle using the `plural bundle install ` CLI command. + > Most applications come with more than one bundle, one for each targeted cloud provider. + The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. From aef9cd399ab7e93e43ab2993bf1e66412fbb5552 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:24:24 +0200 Subject: [PATCH 33/37] add info on node groups --- pages/adding-new-application/plural-artifact-structure.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pages/adding-new-application/plural-artifact-structure.md b/pages/adding-new-application/plural-artifact-structure.md index e4e7aec1..279d14bd 100644 --- a/pages/adding-new-application/plural-artifact-structure.md +++ b/pages/adding-new-application/plural-artifact-structure.md @@ -6,7 +6,7 @@ description: >- As mentioned in [Background on Application Installations](/adding-new-application/background-app-install), the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user provided values for that installation. -Some extra configuration files are necessary in the applications artifact for the Plural API to be able to understand: +Some extra configuration files are necessary in the application's artifact for the Plural API to be able to understand: - the Helm charts and Terraform modules dependencies to run them through its templating engine - dependencies on other Plural artifacts - platform specific components and infrastructure configurations @@ -153,8 +153,9 @@ spec: The `terraform` directory contains the app's provider-specific terraform modules that encapsulate all application components that do not (or cannot) live inside the cluster. For each cloud provider, that the artifact offers a bundle for, there will be one under the related directory name. -Most commonly you'd find object storage alongside their IAM resources, but sometimes also Kubernetes resources like service accounts, -if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. +Most commonly you'd find object storage alongside their IAM resources, or additional node groups, if your app needs a GPU. +Sometimes it will also include Kubernetes resources like service accounts, if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner. + > One peculiarity about the Terraform modules is that, at the very least, they need to contain the Kubernetes namespace for your application. This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step. From 5c207d1df893f599c1a08f0955f407fc8a4641da Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:26:11 +0200 Subject: [PATCH 34/37] try making those headers --- .../plural-artifact-structure.md | 23 ++----------------- 1 file changed, 2 insertions(+), 21 deletions(-) diff --git a/pages/adding-new-application/plural-artifact-structure.md b/pages/adding-new-application/plural-artifact-structure.md index 279d14bd..84411b0f 100644 --- a/pages/adding-new-application/plural-artifact-structure.md +++ b/pages/adding-new-application/plural-artifact-structure.md @@ -184,7 +184,7 @@ On the top-level directory of each artifact you'll also find a`repository.yaml`. The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format. -`repository.yaml`: +### `repository.yaml` ```yaml name: dagster description: A data orchestration platform for the development, production, and observation of data assets. @@ -213,10 +213,8 @@ the notes template to prompt after installation, as well as links to any upstrea This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart. The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push`. -It should be set to the same value as the `private` flag in inside the `repository.yaml`. - -`plural/receipes/dagster-aws.yaml`: +### `plural/receipes/dagster-aws.yaml` ```yaml name: dagster-aws description: Installs dagster on an aws eks cluster @@ -267,20 +265,3 @@ Let's step through this file. - `sections[0].items` lists the chart and module directories in the `helm` or `terraform` directories that are part of this bundle. > A bundle can technically have multiple sections, but this feature's not yet used. - -```yaml -apiVersion: plural.sh/v1alpha1 -kind: Dependencies -metadata: - description: dagster aws setup - version: 0.1.2 -spec: - dependencies: - - name: aws-bootstrap - repo: bootstrap - type: terraform - version: '>= 0.1.1' - providers: - - aws -``` - From 7d75c8ece80657ceb22d19978e45a89c4aefb247 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:26:48 +0200 Subject: [PATCH 35/37] typo --- pages/adding-new-application/plural-artifact-structure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/plural-artifact-structure.md b/pages/adding-new-application/plural-artifact-structure.md index 84411b0f..ff552f6f 100644 --- a/pages/adding-new-application/plural-artifact-structure.md +++ b/pages/adding-new-application/plural-artifact-structure.md @@ -214,7 +214,7 @@ the notes template to prompt after installation, as well as links to any upstrea The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push`. -### `plural/receipes/dagster-aws.yaml` +### `plural/recipes/dagster-aws.yaml` ```yaml name: dagster-aws description: Installs dagster on an aws eks cluster From 8efdfee7e3d0218918cfe7308b5a7afff74857ac Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:28:41 +0200 Subject: [PATCH 36/37] add link --- pages/adding-new-application/plural-artifact-structure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/adding-new-application/plural-artifact-structure.md b/pages/adding-new-application/plural-artifact-structure.md index ff552f6f..d0e210bd 100644 --- a/pages/adding-new-application/plural-artifact-structure.md +++ b/pages/adding-new-application/plural-artifact-structure.md @@ -212,7 +212,7 @@ the notes template to prompt after installation, as well as links to any upstrea > Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt. This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart. -The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push`. +The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push` or `plural apply` (see [Publishing](/adding-new-application/publishing)). ### `plural/recipes/dagster-aws.yaml` ```yaml From 5a524501a0857cb8849edc4e0bc606c57037e961 Mon Sep 17 00:00:00 2001 From: Hans Rauer Date: Tue, 10 Oct 2023 18:36:45 +0200 Subject: [PATCH 37/37] last fixes --- pages/adding-new-application/background-app-install.md | 2 +- pages/adding-new-application/publishing.md | 3 +-- src/NavData.tsx | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/pages/adding-new-application/background-app-install.md b/pages/adding-new-application/background-app-install.md index a5818fa7..f6ea7a30 100644 --- a/pages/adding-new-application/background-app-install.md +++ b/pages/adding-new-application/background-app-install.md @@ -11,7 +11,7 @@ Next, the user can install an application bundle using the `plural bundle instal > Most applications come with more than one bundle, one for each targeted cloud provider. -The CLI will then prompt the user for for inputs needed to setup that application, along with any dependencies of the application. +The CLI will then prompt the user for inputs needed to setup that application, along with any dependencies of the application. These inputs are saved in the `context.yaml` file. For example, let's have a look how the `tree` of a deployment repository, where we installed Dagster with `plural bundle install dagster dagster-aws` (among others), might look like: diff --git a/pages/adding-new-application/publishing.md b/pages/adding-new-application/publishing.md index b57ea24f..2b85f503 100644 --- a/pages/adding-new-application/publishing.md +++ b/pages/adding-new-application/publishing.md @@ -3,7 +3,7 @@ title: Publishing a Plural Artifact --- Every Plural account holder can act as a publisher to publish artifacts and in that way contribute open source applications to Plural's marketplace. -If you don't want to publish an application publicly straight away and make it available to everybody you can first test it by releasing it privately whereby only users of your Plural account can install it. +If you don't want to publish an application publicly straight away and make it available to everybody, you can first test it by releasing it privately whereby only users of your Plural account can install it. ## Create a publisher @@ -44,5 +44,4 @@ pushing recipe plural/recipes/dagster-aws.yaml✓ Artifacts can be published with a `private: true` or `private: false` setting in the `repository.yaml`. Bundles from private artifacts are only visible for, and can only be downloaded and installed by, users from the same account the publisher was created in. -If you want to publish your artifact publicly, too, you need to open a PR in the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts), so the Plural team can review your code. Artifacts published publicly by you are also visible in the public marketplace. diff --git a/src/NavData.tsx b/src/NavData.tsx index d79a77bf..482e7ae9 100644 --- a/src/NavData.tsx +++ b/src/NavData.tsx @@ -178,7 +178,7 @@ const rootNavData: NavMenu = deepFreeze([ title: 'Contribute a New Application to the Plural Catalog', sections: [ { href: '/adding-new-application/background-app-install', title: 'Background on Application Installations'}, - { href: '/adding-new-application/application-artifact-package', title: 'Plural Artifact Structure'}, + { href: '/adding-new-application/plural-artifact-structure', title: 'Plural Artifact Structure'}, { href: '/adding-new-application/templating', title: 'Plural Artifact Templating'}, { href: '/adding-new-application/guide', title: 'Guide on Creating a New Application Artifact' }, { href: '/adding-new-application/publishing', title: 'Guide on Publishing a Plural Artifact'},