diff --git a/README.md b/README.md index e388491c9..a89ecb853 100644 --- a/README.md +++ b/README.md @@ -2,139 +2,33 @@ [![codecov](https://codecov.io/gh/redhat-appstudio/application-service/branch/main/graph/badge.svg)](https://codecov.io/gh/redhat-appstudio/application-service) +## Overview -A Kubernetes operator to create and manage applications and control the lifecycle of applications. +A Kubernetes operator to create, manage and control the lifecycle of applications and components. -## Building & Testing -This operator provides a `Makefile` to run all the usual development tasks. If you simply run `make` without any arguments, you'll get a list of available "targets". +This repository is closely associated with the [application-api](https://github.com/redhat-appstudio/application-api/) repository, which contains the Kubernetes CRD definitions for the application-service specific resources - `Application`, `Component` and `ComponentDetectionQuery`. -To build the operator binary run: +## Documentation -``` -make build -``` +### ⚡ Project Info +* [HAS/application-service project information page](https://docs.google.com/document/d/1axzNOhRBSkly3M2Y32Pxr1MBpBif2ljb-ufj0_aEt74/edit?usp=sharing) - document detailing the project +* [Google Drive](https://drive.google.com/drive/u/0/folders/1pqESr0oc2ldtfj9RDx65vD_KdkgY_G9h) - information for people new to the project -To test the code: +### 🔥 Developers -``` -make test -``` +* [Build, Test and Deploy](./docs/build-test-and-deploy.md) - build, test and deploy the application-service +* [Serviceability](./docs/serviceability.md) - serviceability information like accessing and understanding logs, debugging, common problems and FAQs -**Note:** In order for the controller tests to run, follow the instructions for [installing the Pact tools](#installing-pact-tools) +### ⭐ Other Info -To build the docker image of the operator one can run: +* [Gitops-generation](./docs/gitops-generation.md) - information on the GitOps resource generation from application-service +* [Pact tests](./docs/pact-tests.md) - contract tests using a Pact framework (part of the unit tests) +* [OpenShift CI job artifacts](https://docs.ci.openshift.org/docs/how-tos/artifacts/) - Prow job executed by the CI system generates an artifacts directory, this document describes the contents of this directory and how they can be used to investigate the job steps -``` -make docker-build -``` +## Release -This will make a docker image called `controller:latest` which might or might not be what you want. To override the name of the image build, specify it in the `IMG` environment variable, e.g.: - -``` -IMG=quay.io/user/hasoperator:next make docker-build -``` - -To push the image to an image repository one can use: - -``` -make docker-push -``` - -The image being pushed can again be modified using the environment variable: -``` -IMG=quay.io/user/hasoperator:next make docker-push -``` - -## Installing Pact Tools - -The Pact tests in the controller package require pact tooling to be installed and on your path. Follow these instructions to do so: - -1. Change directory to an appropriate folder (e.g. `/usr/local`) -2. Run `curl -fsSL https://raw.githubusercontent.com/pact-foundation/pact-ruby-standalone/master/install.sh | bash` -3. Add the pact tools' bin folder (e.g. `/usr/local/pact/bin`) to your path to your shell PATH. Ensure all binary files within the `bin/` folder has executable permissions -4. Run `go install github.com/pact-foundation/pact-go@v1` to install the `pact-go` tool -5. Run `pact-go install` to validate that all of the necessary Pact tools are installed - -## Deploying the Operator - -The following section outlines the steps to deploy HAS on a physical Kubernetes cluster. - -### Setting up the AppStudio Build Service environment - -* Install `OpenShift GitOps` from the in-cluster Operator Marketplace. -* `oc -n openshift-gitops apply -f https://raw.githubusercontent.com/redhat-appstudio/infra-deployments/main/argo-cd-apps/base/build.yaml` - -As a user, upon creation of Component, Tekton resources would be created by the controller. - -To use auto generated image repository for the Component's image add `image.redhat.com/generate: "true"` annotation to the Component. - -If you wish to get 'working' PipelineRuns with user provided image repository, create an image pull secret and link it to the `pipeline` Service Account in the Component's namespace (in both `secrets` and `imagePullSecrets` sections). -See [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) for more information on how to create `Secrets` containing registry credentials. - - - -### Creating a GitHub Secret for HAS - -Before deploying the operator, you must ensure that a secret, `has-github-token`, exists in the namespace where HAS will be deployed. This secret must contain a key, `tokens`, whose value points to a comma separated list, without spaces, of key-value pairs of token names and tokens, delimited by a colon. - -For example, on OpenShift: - -Screenshot 2023-03-22 at 3 53 11 PM - -Or via command-line: - -```bash -application-service % kubectl create secret generic has-github-token --from-literal=tokens=token1:ghp_faketoken,token2:ghp_anothertoken,token3:ghp_thirdtoken -``` - -Any token that is used here must have the following permissions set: -- `repo` -- `delete_repo` - -In addition to this, each GitHub token must be associated with an account that has write access to the GitHub organization you plan on using with HAS (see next section). - - -### Using Private Git Repos - -HAS requires SPI to be set up in order to work with private git repositories. - -See [private-git-repos.md](docs/private-git-repos.md) for information on setting up HAS and SPI for use with private git repositories. - -### Deploying HAS - - -Once a secret has been created, simply run the following commands to deploy HAS: -``` -make install -make deploy -``` - -### Specifying Alternate GitHub org - -By default, HAS will use the `redhat-appstudio-appdata` org for the creation of GitOps repositories. If you wish to use your own account, or a different GitHub org, setting `GITHUB_ORG=` before deploying will ensure that an alternate location is used. - -For example: - -`GITHUB_ORG=fake-organization make deploy` would deploy HAS configured to use github.com/fake-organization. - -### Specifying Alternate Devfile Registry URL - -By default, the production devfile registry URL will be used for `ComponentDetectionQuery`. If you wish to use a different devfile registry, setting `DEVFILE_REGISTRY_URL=` before deploying will ensure that an alternate devfile registry is used. - -For example: - -`DEVFILE_REGISTRY_URL=https://myregistry make deploy` would deploy HAS configured to use https://myregistry. - -### Disabling Webhooks for Local Dev - -Webhooks require self-signed certificates to validate the resources. To disable webhooks during local dev and testing, export `ENABLE_WEBHOOKS=false` - -### Useful links: -* [HAS Project information page](https://docs.google.com/document/d/1axzNOhRBSkly3M2Y32Pxr1MBpBif2ljb-ufj0_aEt74/edit?usp=sharing) -* Every Prow job executed by the CI system generates an artifacts directory containing information about that execution and its results. This [document](https://docs.ci.openshift.org/docs/how-tos/artifacts/) describes the contents of this directory and how they can be used to investigate the steps by the job. -* For more information on the GitOps resource generation, please refer to the [gitops-generation](./docs/gitops-generation.md) documentation -* Contract testing using a Pact framework is part of unit tests. Follow [this documentation](pactTests.md) to learn more. +For more information on the application-service release policy, please read the release [guideline](./docs/release.md). ## Contributions -Please see our [CONTRIBUTING](./docs/CONTRIBUTING.md) for more information. +If you would like to contribute to application-service, please be so kind to read our [CONTRIBUTING](./docs/CONTRIBUTING.md) guide for more information. diff --git a/config/samples/binding/binding.yaml b/config/samples/binding/binding.yaml index c0543d4ab..a8d7fb207 100644 --- a/config/samples/binding/binding.yaml +++ b/config/samples/binding/binding.yaml @@ -1,4 +1,4 @@ -# API type definition is in github.com/redhat-appstudio/managed-gitops +# API type definition is in https://github.com/redhat-appstudio/application-api/tree/main/api/v1alpha1 apiVersion: appstudio.redhat.com/v1alpha1 kind: SnapshotEnvironmentBinding metadata: diff --git a/config/samples/environment/environment.yaml b/config/samples/environment/environment.yaml index 030ae3cbf..41d1cd182 100644 --- a/config/samples/environment/environment.yaml +++ b/config/samples/environment/environment.yaml @@ -1,4 +1,4 @@ -# API type definition is in github.com/redhat-appstudio/managed-gitops +# API type definition is in https://github.com/redhat-appstudio/application-api/tree/main/api/v1alpha1 apiVersion: appstudio.redhat.com/v1alpha1 kind: Environment metadata: diff --git a/config/samples/snapshot/snapshot.yaml b/config/samples/snapshot/snapshot.yaml index b0a14dd34..2a338f295 100644 --- a/config/samples/snapshot/snapshot.yaml +++ b/config/samples/snapshot/snapshot.yaml @@ -1,4 +1,4 @@ -# API type definition is in github.com/redhat-appstudio/managed-gitops +# API type definition is in https://github.com/redhat-appstudio/application-api/tree/main/api/v1alpha1 apiVersion: appstudio.redhat.com/v1alpha1 kind: Snapshot metadata: diff --git a/docs/build-test-and-deploy.md b/docs/build-test-and-deploy.md new file mode 100644 index 000000000..fb919f4cb --- /dev/null +++ b/docs/build-test-and-deploy.md @@ -0,0 +1,124 @@ +# Build, Test and Deploy + +## Build +This operator provides a `Makefile` to run all the usual development tasks. If you simply run `make` without any arguments, you'll get a list of available "targets". + +To build the operator binary run: + +``` +make build +``` + +To build the docker image of the operator one can run: + +``` +make docker-build +``` + +This will make a docker image called `controller:latest` which might or might not be what you want. To override the name of the image build, specify it in the `IMG` environment variable, e.g.: + +``` +IMG=quay.io/user/hasoperator:next make docker-build +``` + +To push the image to an image repository one can use: + +``` +make docker-push +``` + +The image being pushed can again be modified using the environment variable: +``` +IMG=quay.io/user/hasoperator:next make docker-push +``` + +## Test + +To test the code: + +``` +make test +``` + +**Note:** In order for the controller tests to run, follow the instructions for [installing the Pact tools](./installing-pact-tools.md) + +## Deploy + +The following section outlines the steps to deploy application-service on a physical Kubernetes cluster. + +### Deploying on a Local Cluster + +#### Creating a GitHub Secret for application-service + +Before deploying the operator, you must ensure that a secret `has-github-token`, exists in the namespace where application-service will be deployed. This secret must contain a key `tokens`, whose value points to a comma separated list without spaces of key-value pairs of token names and tokens, delimited by a colon. + +For example, on OpenShift: + +Screenshot 2023-03-22 at 3 53 11 PM + +Or via command-line: + +```bash +application-service % kubectl create secret generic has-github-token --from-literal=tokens=token1:ghp_faketoken,token2:ghp_anothertoken,token3:ghp_thirdtoken +``` + +Any token that is used here must have the following permissions set: +- `repo` +- `delete_repo` + +In addition to this, each GitHub token must be associated with an account that has write access to the GitHub organization you plan on using with application-service. + +#### Using Private Git Repos + +The application-service component requires SPI to be set up in order to work with private git repositories. + +Please refer to the [instructions](./private-git-repos.md) for information on setting up application-service and SPI for use with private git repositories. + +#### Deploy application-service + + +Once a secret has been created, simply run the following commands to deploy application-service: +``` +make install +make deploy +``` + +The application-service deployment can be further configured. Please refer to the sections below for your needs. + +#### Specifying Alternate GitHub org + +By default, application-service will use the `redhat-appstudio-appdata` org for the creation of GitOps repositories. If you wish to use your own account, or a different GitHub org, setting `GITHUB_ORG=` before deploying will ensure that an alternate location is used. + +For example: + +`GITHUB_ORG=fake-organization make deploy` would deploy application-service configured to use github.com/fake-organization. + +#### Specifying Alternate Devfile Registry URL + +By default, the production devfile registry URL will be used for `ComponentDetectionQuery`. If you wish to use a different devfile registry, setting `DEVFILE_REGISTRY_URL=` before deploying will ensure that an alternate devfile registry is used. + +For example: + +`DEVFILE_REGISTRY_URL=https://myregistry make deploy` would deploy application-service configured to use https://myregistry. + +### Deploying Locally + +#### Disabling Webhooks for Local Development + +Webhooks require self-signed certificates to validate the Kubernetes resources. To disable webhooks during local development and testing, export `ENABLE_WEBHOOKS=false` + +#### Setting the GitHub Token Environment variable + +Either of the Environment variable `GITHUB_AUTH_TOKEN` or `GITHUB_TOKEN_LIST` needs to be set. + +The `GITHUB_AUTH_TOKEN` variable is the legacy format and requires one token, example `GITHUB_AUTH_TOKEN=ghp_faketoken`. The `GITHUB_TOKEN_LIST` can take a list of tokens, example `GITHUB_TOKEN_LIST=token1:ghp_faketoken,token2:ghp_anothertoken`. + +#### Executing the application-service binary + +The application-service controller manager can be run locally on your development environment (example, laptop). For example, to build and run the executable manager: + +``` +make install +make build +./bin/manager +``` diff --git a/docs/installing-pact-tools.md b/docs/installing-pact-tools.md new file mode 100644 index 000000000..5fd0510a5 --- /dev/null +++ b/docs/installing-pact-tools.md @@ -0,0 +1,9 @@ +# Installing Pact Tools + +The Pact tests in the controller package require pact tooling to be installed and on your path. Follow these instructions to do so: + +1. Change directory to an appropriate folder (e.g. `/usr/local`) +2. Run `curl -fsSL https://raw.githubusercontent.com/pact-foundation/pact-ruby-standalone/master/install.sh | bash` +3. Add the pact tools' bin folder (e.g. `/usr/local/pact/bin`) to your path to your shell PATH. Ensure all binary files within the `bin/` folder has executable permissions +4. Run `go install github.com/pact-foundation/pact-go@v1` to install the `pact-go` tool +5. Run `pact-go install` to validate that all of the necessary Pact tools are installed diff --git a/pactTests.md b/docs/pact-tests.md similarity index 100% rename from pactTests.md rename to docs/pact-tests.md diff --git a/docs/private-git-repos.md b/docs/private-git-repos.md index a3381136e..904ea9408 100644 --- a/docs/private-git-repos.md +++ b/docs/private-git-repos.md @@ -1,8 +1,11 @@ # Using Private Git Repositories with HAS +Please follow the following instructions to install SPI and use with application-service. + +Note: SPI _cannot_ be used on a Red Hat OpenShift Local (formerly, CRC) cluster. ## Configuring SPI -In order to use HAS resources (e.g. `Applications`, `Components`, `ComponentDetectionQuery`) with private git repositories, SPI must be installed on the same cluster as HAS (TODO review HAS install instructions): +In order to use HAS resources (e.g. `Application`, `Component`, `ComponentDetectionQuery`) with private git repositories, SPI must be installed on the same cluster as HAS (TODO review HAS install instructions): 1) Clone the [SPI operator repo](https://github.com/redhat-appstudio/service-provider-integration-operator) and run the [make command](https://github.com/redhat-appstudio/service-provider-integration-operator/blob/main/docs/DEVELOP.md#running-in-cluster) corresponding to your target cluster type e.g. `make deploy_openshift` diff --git a/docs/release.md b/docs/release.md new file mode 100644 index 000000000..7f2f51865 --- /dev/null +++ b/docs/release.md @@ -0,0 +1,7 @@ +# Release + +The application-service repository does not do GitHub releases at the moment. Consumers of the application-service Go module are expected to update to the latest commit on a frequent basis. + +Any depreceation or breaking changes are documented as part of the code change in the PR or the function description. + +Since application-service is part of the AppStudio project, code commits to the application-service repository are automatically added to the [infra-deployments](https://github.com/redhat-appstudio/infra-deployments/tree/main/components/has) repository. diff --git a/docs/serviceability.md b/docs/serviceability.md new file mode 100644 index 000000000..9b807022f --- /dev/null +++ b/docs/serviceability.md @@ -0,0 +1,74 @@ +# Serviceability + +The serviceability document aims to help the local application-service developer and the Site Reliability Engineer (SRE) to access and service the application-service component. This document will help you understand how to access and understand the application-service logs, how to debug an application-service problem and provides a quick summary on the various questions that you might have regarding application-service. + +## Accessing the Logs + +### Deployed Locally +View the application-service controller logs in the terminal window where the executable manager is running. Example, `./bin/manager` will output the controller logs in the terminal. + +### Deployed on a Local Cluster +View the application-service controller logs by tailing the manager container log of the controller manager pod. The pod resides in the application-service namespace. Example, + +``` +oc logs -f application-service-application-service-controller-manager -c manager -n application-service +``` + +### Deployed on a Managed Cluster + +To access the application-service controller logs on either the AppStudio Staging or Production clusters, you would need access to CloudWatch service. To learn more about how to request access to CloudWatch and view the application-service logs, refer to the HAS Access Control [documentation](https://docs.google.com/document/d/1cK4XGKpXBEYOKfIqSiHuuCfsfHjElxhG9lrlEozzgVE/edit#heading=h.yxk6h5uvh57d). + +## Understanding the Logs +Each application-service controller logs their reconcile logic to the manager. The log message format is generally of format + +``` +{"level":"info","ts":"2023-08-31T19:59:21.144Z","msg":"Finished reconcile loop for user-tenant/devfile-sample-go-basic-development-binding-hr9nm","controller":"snapshotenvironmentbinding","controllerGroup":"appstudio.redhat.com","controllerKind":"SnapshotEnvironmentBinding","SnapshotEnvironmentBinding":{"name":"devfile-sample-go-basic-development-binding-hr9nm","namespace":"user-tenant"},"namespace":"user-tenant","name":"devfile-sample-go-basic-development-binding-hr9nm","reconcileID":"d5c8545b-957b-4f1a-b177-84a5d5f0d26c"} +``` + +To understand the AppStudio controller logging convention, refer to the Appstudio [ADR](https://github.com/redhat-appstudio/book/blob/main/ADR/0006-log-conventions.md) + +## Debugging + +- Insert break points at the controller functions to debug unit tests or to debug a local controller deployment, refer to the next section on how to set up a debugger +- When debugging unit and integration tests, remember that the mock clients used are hosting dummy data and mocking the API call and returns +- For debugging controller logs either from a local cluster or a managed cluster, gather the log and search for: + - the namespace the user belongs to. For example, if the user is on the project/namespace, search for `user-tenant` + - the resource name that the controller is reconciling. For example, if the `SnapshotEnvironmentBinding` resource is being reconciled, then search for `"name":"devfile-sample-go-basic-development-binding-hr9nm"` belonging to `"controllerKind":"SnapshotEnvironmentBinding"` + - the log message. For example, `"msg":"Finished reconcile loop for user-tenant/devfile-sample-go-basic-development-binding-hr9nm"`. You may search for the string `Finished reconcile loop for` in the application-service repository to track down the code logic that is emitting the log. Remember to look out for resource name concatenation and/or error wrapping that may not turn up in your code search. It is advised to exclude such strings from the code search for debugging purposes + +### How to debug on RHTAP or How to set up a debugger on VS Code + +For more information, on how to debug on RHTAP Staging or how to set up a debugger on VS Code for local deployment of the application-service controller, please refer to the [Debugging](https://docs.google.com/document/d/1dneldJepfnJ6LnESSYMIhKqmFgjMtf_om_Eud5NMDtU/edit#heading=h.lz54tm3le87l) section of the Education Module document. + +## Common Problems +- When deploying HAS locally or on a local cluster, a Github Personal Access Token is required as the application-service controller requires the token for pushing the resources to the GitOps repository. Please refer to the [instructions](../docs/build-test-and-deploy.md#setting-the-github-token-environment-variable) in the deploy section for more information +- When creating a `Component` from the `ComponentDetectionQuery`, remember to replace the generic application name `insert-application-name`, if the information is being used from a `ComponentDetectionQuery` status + +## FAQs +Q. Where can I view the application-service API types? + +A. The application-service API types and their corresponding webhooks are defined in the [redhat-appstudio/application-api](https://github.com/redhat-appstudio/application-api) repository. You can also find the API reference information in the [Book of AppStudio](https://redhat-appstudio.github.io/book/ref/index.html) website. + +Q. Where are the application-service controller logic located? + +A. Most of the application-service business logic like the reconcile functions are located in the [controllers](https://github.com/redhat-appstudio/application-service/tree/main/controllers) pkg. + +Q. What is the application-service release process? + +A. Since application-service is part of the AppStudio project, code commits to the application-service repository are automatically added to the [infra-deployments](https://github.com/redhat-appstudio/infra-deployments/tree/main/components/has) repository. + +Q. Where can I learn more about the devfile project? + +A. The devfile project is hosted on the [devfile/api](https://github.com/devfile/api) repository and more information about the project as well as the spec can be found on [devfile.io](https://devfile.io/). + +Q. How does `ComponentDetectionQuery` detect the component framework and runtime? + +A. `ComponentDetectionQuery` uses the go module [devfile/alizer](https://github.com/devfile/alizer) for component detection. For more information about the Alizer project, please head over to the Alizer repository. + +Q. How do I debug zombie processes? + +A. For more information on how zombie processes affect application-service controllers, please refer to the Troubleshooting [guide](https://docs.google.com/document/d/1yCFkFslhbdd8M_RarRhZcgx6gm9nr2JwDObxNtl4H-U/edit#heading=h.4brqv3sh6lq9). + +Q. How do I debug Rate Limiting? + +A. The GitHub Personal Access Tokens used by application-service controllers may be rate limited. For more information on how the GitHub PAT rate limiting affects application-service controllers, please refer to the Troubleshooting [guide](https://docs.google.com/document/d/1yCFkFslhbdd8M_RarRhZcgx6gm9nr2JwDObxNtl4H-U/edit#heading=h.3xnfno3qm3if). diff --git a/samples/hasapplication/hasapp.yml b/samples/hasapplication/hasapp.yml deleted file mode 100644 index 4c26c7fbc..000000000 --- a/samples/hasapplication/hasapp.yml +++ /dev/null @@ -1,9 +0,0 @@ -apiVersion: appstudio.redhat.com/v1alpha1 -kind: Application -metadata: - annotations: - name: application-sample - namespace: default -spec: - description: application definition for petclinic-app - displayName: petclinic diff --git a/samples/hasapplication/hascomp.yml b/samples/hasapplication/hascomp.yml deleted file mode 100644 index c6c708452..000000000 --- a/samples/hasapplication/hascomp.yml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: appstudio.redhat.com/v1alpha1 -kind: Component -metadata: - name: component-sample - namespace: default -spec: - componentName: sample-component-name - application: application-sample - source: - git: - url: https://github.com/devfile-samples/devfile-sample-code-with-quarkus - build: - containerImage: docker.io//foobar:dev-code-with-quarkus