diff --git a/.github/ISSUE_TEMPLATE/release_template.md b/.github/ISSUE_TEMPLATE/release_template.md index 0b254c13a4..5be03a1804 100644 --- a/.github/ISSUE_TEMPLATE/release_template.md +++ b/.github/ISSUE_TEMPLATE/release_template.md @@ -116,4 +116,4 @@ __Replace with links to all component tracking issues.__ | :red_circle: | Missed multiple milestones |

- + \ No newline at end of file diff --git a/README.md b/README.md index 4272f8c404..e74383a477 100644 --- a/README.md +++ b/README.md @@ -6,34 +6,32 @@ [![codecov](https://codecov.io/gh/opensearch-project/opensearch-build/branch/main/graph/badge.svg?token=03S5XZ80UI)](https://codecov.io/gh/opensearch-project/opensearch-build) - [Releasing OpenSearch](#releasing-opensearch) - - [Releases and Versions](#releases-and-versions) - - [Creating a New Version](#creating-a-new-version) - - [Onboarding a New Plugin](#onboarding-a-new-plugin) - - [Building and Testing an OpenSearch Distribution](#building-and-testing-an-opensearch-distribution) - - [Building from Source](#building-from-source) - - [Assembling a Distribution](#assembling-a-distribution) - - [Building Patches](#building-patches) - - [Min snapshots](#min-snapshots) - - [CI/CD Environment](#cicd-environment) - - [Build Numbers](#build-numbers) - - [Latest Distribution Url](#latest-distribution-url) - - [Testing the Distribution](#testing-the-distribution) - - [Checking Release Notes](#checking-release-notes) - - [Signing Artifacts](#signing-artifacts) - - [PGP](#pgp) - - [Windows](#windows) - - [Signing RPM artifacts](#signing-rpm-artifacts) - - [Making a Release](#making-a-release) - - [Releasing for Linux](#releasing-for-linux) - - [Releasing for FreeBSD](#releasing-for-freebsd) - - [Releasing for Windows](#releasing-for-windows) - - [Releasing for MacOS](#releasing-for-macos) - - [Utilities](#utilities) - - [Checking Out Source](#checking-out-source) - - [Cross-Platform Builds](#cross-platform-builds) - - [Sanity Checking the Bundle](#sanity-checking-the-bundle) - - [Auto-Generating Manifests](#auto-generating-manifests) - - [Deploying Infrastructure](#deploying-infrastructure) +- [Onboarding a New Plugin](#onboarding-a-new-plugin) +- [Building and Testing an OpenSearch Distribution](#building-and-testing-an-opensearch-distribution) + - [Building from Source](#building-from-source) + - [Assembling a Distribution](#assembling-a-distribution) + - [Building Patches](#building-patches) + - [Min snapshots](#min-snapshots) + - [CI/CD Environment](#cicd-environment) + - [Build Numbers](#build-numbers) + - [Latest Distribution Url](#latest-distribution-url) + - [Testing the Distribution](#testing-the-distribution) + - [Checking Release Notes](#checking-release-notes) + - [Signing Artifacts](#signing-artifacts) + - [PGP](#pgp) + - [Windows](#windows) + - [Signing RPM artifacts](#signing-rpm-artifacts) +- [Making a Release](#making-a-release) + - [Releasing for Linux](#releasing-for-linux) + - [Releasing for FreeBSD](#releasing-for-freebsd) + - [Releasing for Windows](#releasing-for-windows) + - [Releasing for MacOS](#releasing-for-macos) +- [Utilities](#utilities) + - [Checking Out Source](#checking-out-source) + - [Cross-Platform Builds](#cross-platform-builds) + - [Sanity Checking the Bundle](#sanity-checking-the-bundle) + - [Auto-Generating Manifests](#auto-generating-manifests) +- [Deploying Infrastructure](#deploying-infrastructure) - [Contributing](#contributing) - [Getting Help](#getting-help) - [Code of Conduct](#code-of-conduct) @@ -41,18 +39,9 @@ - [License](#license) - [Copyright](#copyright) -## Releasing OpenSearch +### Releasing OpenSearch -### Releases and Versions - -The OpenSearch project releases as versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. It [follows semantic versioning](https://opensearch.org/blog/technical-post/2021/08/what-is-semver/). Software, such as Data Prepper, clients, and the Logstash output plugin, are versioned independently of the OpenSearch Project. They also may have independent releases from the main project distributions. The OpenSearch Project may also release software under alpha, beta, release candidate, and generally available labels. The definition of when to use these labels is derived from [the Wikipedia page on Software release lifecycle](https://en.wikipedia.org/wiki/Software_release_life_cycle). Below is the definition of when to use each label. - -Release labels: - -* **Alpha** - The code is released with instructions to build. Built distributions of the software may not be available. Some features many not be complete. Additional testing and developement work is planned. Distributions will be postfixed with `-alphaX` where "X" is the number of the alpha version (e.g., "2.0-alpha1"). -* **Beta** - Built distributions of the software are available. All features are completed. Additional testing and developement work is planned. Distributions will be postfixed with `-betaX` where "X" is the number of the beta version (e.g., "2.0.0-beta1"). -* **Release Candidate** - Built distributions of the software are available. All features are completed. Code is tested and minimal validation remains. At this stage the software is potentially stable and will release unless signficant bugs emerge. Distributions will be postfixed with `-rcX` where "X" is the number of the release candidate version (e.g., "2.0.0-rc1"). -* **Generally Available** - Built distributions of the software are available. All features are completed and documented. All testing is completed. Distributions for generally available versions are not postfixed with an additional label (e.g., "2.0.0"). +See [release process](./RELEASE_PROCESS_OPENSEARCH.md) document for more information. ### Creating a New Version @@ -210,20 +199,6 @@ RPM artifacts are signed via a legacy shell script which uses a [macros template See [src/sign_workflow](./src/sign_workflow) for more information. -### Making a Release - -#### Releasing for Linux / Windows - -The Linux / Windows release is managed by a team at Amazon following [this release template](.github/ISSUE_TEMPLATE/release_template.md) (e.g. [opensearch-build#2649](https://github.com/opensearch-project/opensearch-build/issues/2649)). - -#### Releasing for FreeBSD - -The FreeBSD ports and packages for OpenSearch are managed by a community [OpenSearch Team](https://wiki.freebsd.org/OpenSearch) at FreeBSD. When a new release is rolled out, this team will update the port and commit it to the FreeBSD ports tree. Anybody is welcome to help the team by providing patches for [upgrading the ports](https://docs.freebsd.org/en/books/porters-handbook/book/#port-upgrading) following the [FreeBSD Porter's Handbook](https://docs.freebsd.org/en/books/porters-handbook/book/) instructions. - -#### Releasing for MacOS - -At this moment there's no official MacOS distribution. However, this project does support building and assembling OpenSearch for MacOS. See [opensearch-build#37](https://github.com/opensearch-project/opensearch-build/issues/37) and [#38](https://github.com/opensearch-project/opensearch-build/issues/38) for more details. - ### Utilities #### Checking Out Source diff --git a/RELEASE_PROCESS_OPENSEARCH.md b/RELEASE_PROCESS_OPENSEARCH.md new file mode 100644 index 0000000000..8fe5f8aa31 --- /dev/null +++ b/RELEASE_PROCESS_OPENSEARCH.md @@ -0,0 +1,325 @@ +- [OpenSearch Releas Process](#opensearch-release-process) + - [Release Terminology and Knowledge Center](#release-terminology-and-knowledge-center) + - [Types of Manifests](#types-of-manifests) + - [Build Workflows](#build-workflows) + - [Release Workflows](#release-workflows) + - [Releases and Versions](#releases-and-versions) + - [Creating a New Version](#creating-a-new-version) + - [Making a Release](#making-a-release) + - [Releasing for Linux and Windows](#releasing-for-linux-and-windows) + - [Releasing for FreeBSD](#releasing-for-freebsd) + - [Releasing for Windows](#releasing-for-windows) + - [Releasing for MacOS](#releasing-for-macos) + - [Release issue](#release-issue) + - [Release Manager](#release-manager) + - [Release Issue Update](#release-issue-update) + - [Version Increment](#version-increment) + - [Version Increment Core](#version-increment-core) + - [Release Branch](#release-branch) + - [Plugins Version Increment](#plugins-version-increment) + - [Distribution Build](#distribution-build) + - [Release Candidate](#release-candidate) + - [Final Release](#final-release) + - [Pre-Release](#pre-release) + - [Release Notes](#release-notes) + - [Release](#release) + - [Post-Release](#post-release) + +## OpenSearch Release Process + +### Release Terminology and Knowledge Center + +#### Types of Manifests + + +- Input Manifest: Uses as an initial input to drive the build workflow. These manifests are [located](https://github.com/opensearch-project/opensearch-build/tree/main/manifests) in build repo and gets created by [manifest workflow](https://github.com/opensearch-project/opensearch-build/tree/main/src/manifests_workflow). +Once the manifest is created the maintainers of the build repo approves and merges it to `main` branch. The entire build repo runs on single branch `main`. + +- Build Manifest: Output of the [build workflow](https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow), used as input to the [assemble workflow](https://github.com/opensearch-project/opensearch-build/tree/main/src/assemble_workflow). Sample [manifest](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/1.3.10/6126/linux/x64/tar/builds/opensearch-dashboards/manifest.yml). + + ##### Sample Generated Build Manifests + + | deb | rpm | tar | windows | + | --- | --- | --- | ------- | + | [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/deb/builds/opensearch/manifest.yml), [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/deb/builds/opensearch/manifest.yml) | [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/rpm/builds/opensearch/manifest.yml), [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/rpm/builds/opensearch/manifest.yml) | [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/tar/builds/opensearch/manifest.yml), [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/tar/builds/opensearch/manifest.yml) | [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/windows/x64/zip/builds/opensearch/manifest.yml) | + +- Bundle Manifest: The final output of the assemble workflow and manifest that is added to the final distribution, this has the commitID (can be used for reproducible builds ) and the artifact file links. + + ##### Sample Generated Bundle Manifests + + | deb | rpm | tar | windows | + | --- | --- | --- | ------- | + | [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/deb/dist/opensearch/manifest.yml), [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/deb/dist/opensearch/manifest.yml) | [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/rpm/dist/opensearch/manifest.yml), [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/rpm/dist/opensearch/manifest.yml) | [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/x64/tar/dist/opensearch/manifest.yml), [os-arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/linux/arm64/tar/dist/opensearch/manifest.yml) | [os-x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.11/7979/windows/x64/zip/dist/opensearch/manifest.yml) | + + +AUTOCUT issues: These are the issues created by automation with the distribution build and intesg-test failure, the automation detects the component failure and raises an issue in the respective plugin component. Sample [integ-test failure AUTOCUT issue](https://github.com/opensearch-project/k-NN/issues/914) and [distribution build failure AUTOCUT issue](https://github.com/opensearch-project/k-NN/issues/732). + + +#### Build Workflows + +| Wokflow | Description | +| ----------------------------------------------------------------------------------------------------------------------------- | ------------------- | +| [Check for Build](https://build.ci.opensearch.org/job/check-for-build/) | Workflow that peridically triggers the distribution workflows using parameterized Cron. | +| [OpenSearch Distribution Build](https://build.ci.opensearch.org/job/distribution-build-opensearch/) | Workflow that is responsible to build/assemble the OpenSearch and its components. | +| [OpenSearch Dashboards Distribution Build](https://build.ci.opensearch.org/job/distribution-build-opensearch-dashboards/) | Workflow that is responsible to build/assemble the OpenSearch Dashboards and its components. | +| [OpenSearch Integ Test](https://build.ci.opensearch.org/job/integ-test/) | Workflow that runs integ tests for OpenSearch and its components. | +| [OpenSearch Dashboards Integ Test](https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/) | Workflow that runs integ tests for OpenSearch Dashboards and its components. | +| [Performance Tests](https://build.ci.opensearch.org/job/perf-test/) | Workflow that runs Performance tests on a cluster created with a given version. | +| [BWC Tests](https://build.ci.opensearch.org/job/bwc-test/) | Workflow that runs backward compatibility tests on a cluster created with a given version. | +| [RPM Validation](https://build.ci.opensearch.org/job/rpm-validation/) | Workflow that validates the RPM distribution | +| [Docker Build](https://build.ci.opensearch.org/job/docker-build/) | Workflow that builds the OpenSearch and OpenSearch Dashboards docker images | +| [Docker Copy](https://build.ci.opensearch.org/job/docker-copy/) | Workflow that copies the created docker images to multiple DockerHub and ECR repositories | +| [Docker Scan](https://build.ci.opensearch.org/job/docker-scan/) | Workflow that checks vulnerabilities for a given docker image as an input. | + + +#### Release Workflows + +| Wokflow | Description | +| ---------------------------------------------------------------------------------------- | ------------------- | +| [Release Notes Tracker](https://build.ci.opensearch.org/job/release-notes-tracker/) | Workflow that identifies if a component has a release notes added based on the commit history. | +| [Promote Repos](https://build.ci.opensearch.org/job/distribution-promote-repos/) | Workflow that signs and promotes the APT/YUM repos to the production buckets accessed via the cloudfront. | +| [Promote artifacts](https://build.ci.opensearch.org/job/distribution-promote-artifacts/) | Workflow that signs and promotes all the release artifcats to the production buckets accessed via the cloudfront. | +| [Publish to Maven](https://build.ci.opensearch.org/job/publish-to-maven/) | Workflow that signs and publishes to the central maven repository. | +| [Docker Promotion](https://build.ci.opensearch.org/job/docker-promotion/) | Workflow that promotoes the docker images to production docker repositories. | +| [Validation Workflow](https://build.ci.opensearch.org/job/distribution-validation) | Workflow that validates the released distribution. | + + +#### Releases and Versions + +The OpenSearch project releases as versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. It [follows semantic versioning](https://opensearch.org/blog/technical-post/2021/08/what-is-semver/). Software, such as Data Prepper, clients, and the Logstash output plugin, are versioned independently of the OpenSearch Project. They also may have independent releases from the main project distributions. The OpenSearch Project may also release software under alpha, beta, release candidate, and generally available labels. The definition of when to use these labels is derived from [the Wikipedia page on Software release lifecycle](https://en.wikipedia.org/wiki/Software_release_life_cycle). Below is the definition of when to use each label. + +##### Release labels: + +* **Alpha** - The code is released with instructions to build. Built distributions of the software may not be available. Some features many not be complete. Additional testing and developement work is planned. Distributions will be postfixed with `-alphaX` where "X" is the number of the alpha version (e.g., "2.0-alpha1"). +* **Beta** - Built distributions of the software are available. All features are completed. Additional testing and developement work is planned. Distributions will be postfixed with `-betaX` where "X" is the number of the beta version (e.g., "2.0.0-beta1"). +* **Release Candidate** - Built distributions of the software are available. All features are completed. Code is tested and minimal validation remains. At this stage the software is potentially stable and will release unless signficant bugs emerge. Distributions will be postfixed with `-rcX` where "X" is the number of the release candidate version (e.g., "2.0.0-rc1"). +* **Generally Available** - Built distributions of the software are available. All features are completed and documented. All testing is completed. Distributions for generally available versions are not postfixed with an additional label (e.g., "2.0.0"). + +#### Creating a New Version + +Each new OpenSearch release process starts when any one component increments a version, typically on the `main` branch. For example, [OpenSearch#1192](https://github.com/opensearch-project/OpenSearch/pull/1192) incremented the version to 2.0. The [version check automation workflow](.github/workflows/versions.yml) will notice this change or it can be triggered [manually](https://github.com/opensearch-project/opensearch-build/actions/workflows/versions.yml), and make a pull request (e.g. [opensearch-build#514](https://github.com/opensearch-project/opensearch-build/pull/514)) that adds a new manifest (e.g. [opensearch-2.0.0.yml](manifests/2.0.0/opensearch-2.0.0.yml). After that's merged, a GitHub issue is automatically opened by [this workflow](.github/workflows/releases.yml) to make a new release using [this release template](.github/ISSUE_TEMPLATE/release_template.md) (e.g. [opensearch-build#566](https://github.com/opensearch-project/opensearch-build/issues/566)). Existing and new components [(re)onboard into every release](ONBOARDING.md) by submitting pull requests to each version's manifest. + +#### Making a Release + +##### Releasing for Linux and Windows + +The Linux / Windows release is managed by a team at Amazon following [this release template](.github/ISSUE_TEMPLATE/release_template.md) (e.g. [opensearch-build#2649](https://github.com/opensearch-project/opensearch-build/issues/2649)). + +##### Releasing for FreeBSD + +The FreeBSD ports and packages for OpenSearch are managed by a community [OpenSearch Team](https://wiki.freebsd.org/OpenSearch) at FreeBSD. When a new release is rolled out, this team will update the port and commit it to the FreeBSD ports tree. Anybody is welcome to help the team by providing patches for [upgrading the ports](https://docs.freebsd.org/en/books/porters-handbook/book/#port-upgrading) following the [FreeBSD Porter's Handbook](https://docs.freebsd.org/en/books/porters-handbook/book/) instructions. + +##### Releasing for MacOS + +At this moment there's no official MacOS distribution. However, this project does support building and assembling OpenSearch for MacOS. See [opensearch-build#37](https://github.com/opensearch-project/opensearch-build/issues/37) and [#38](https://github.com/opensearch-project/opensearch-build/issues/38) for more details. + +### Release issue + +This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section. + + +#### Release Manager + +The release managers to a specific OpenSearch release will be assigned through volunteer model. The request for release managers will be posted in [OpenSearch public slack workspace](https://opensearch.slack.com/archives/C0561HRK961) (under releases channel) and selected on first come first served (FCFS) model. +Note: The release managers should be maintainers of a repo under OpenSearch GitHub organization. + +#### Release Issue Update + +The release issue is created by an [automation workflow](https://github.com/opensearch-project/opensearch-build/blob/main/.github/workflows/releases.yml). Once the release manager is finalized, the release manager should be updating the created release issue. Sample [Release Issue 2.8.0](https://github.com/opensearch-project/opensearch-build/issues/3434) + +### Version Increment + +Versions are incremented as soon as development starts on a given version to avoid confusion. Following is the example that depicts the version increment. + +* OpenSearch: `main` = 2.0.0, `1.x` = 1.1.0, and `1.0` = 1.0.0 +* common-utils: `main` = 1.1.0.0, `1.0` = 1.0.0.0 + +#### Version Increment Core + +To ensure the version incrementation process is handled correctly, it is important to follow up with the core team. Currently, the version incrementation is being done manually by an individual from the core repositories. Sample OpenSearch Version Increment [PR](https://github.com/opensearch-project/OpenSearch/pull/7864/). + +#### Release Branch + +Not applicable for patch release. This step requires that every team participating in a release has their release branch created for the corresponding release by the date listed on this step. The distribution build job will also start using the release branch to create release candidate instead of `.x` branches. + +#### Plugins Version Increment + +The next step is to identify the pending plugin version increment pull requests. The creation of plugin version increment pull requests is automated with workflows running in the build repo [OpenSearch Version Increment Workflow](https://github.com/opensearch-project/opensearch-build/blob/main/.github/workflows/os-increment-plugin-versions.yml), +[OpenSearch Dashboards Version Increment Workflow](https://github.com/opensearch-project/opensearch-build/blob/main/.github/workflows/osd-increment-plugin-versions.yml). Sample plugin version increment [PR](https://github.com/opensearch-project/job-scheduler/pull/398). + +The goal is to get these pull requests merged so that all the plugins are in sync with the core version. It's not always just raising a pull request and getting it merged. There are CI checks for every repo that get triggered when a pull request is raised. These CI checks test against the dependencies. Therefore, before the version increment pull requests are merged, the release manager needs to ensure that the version increment is done for dependencies, built, and pushed to Maven and S3. This is a key step for the release manager to coordinate with the plugin teams, address the dependencies, and get the version increment pull requests merged. + +### Distribution Build + +Refer the [Build Workflows](#build-workflows) section to get the details about the distribution build jobs. This section covers how to sequentially +handle the distribution build job during the release. + +Following is the order of execution for the distribution build: + +##### OpenSearch +``` +OpenSearch +OpenSearch + common-utils + job-scheduler +OpenSearch + common-utils + job-scheduler + performance-analyzer +OpenSearch + common-utils + job-scheduler + performance-analyzer + security +OpenSearch + common-utils + job-scheduler + ml-commons + performance-analyzer + security +All components (which are ready after completion of version increment) +``` + +##### OpenSearch Dashboards +``` +OpenSearch Dashboards +All components (which are ready after completion of version increment) +``` + +### Release Candidate + +The automated build workflows will create release candidates with the finalized commit and publish the docker images and associated distribution artifacts for testing. Now once all the version increment PR’s are completed and all the components are part of the input manifest, now its time to generate the RC. + + +#### Sample Build details: + +| OpenSearch | OpenSearch Dashboards | +|----------|----------| +| [build_7858](https://build.ci.opensearch.org/job/distribution-build-opensearch/7848/console) | [build_6126](https://build.ci.opensearch.org/job/distribution-build-opensearch-dashboards/6126/) | + +The distribution build number denotes the RC, now with the above example finalized RC’s are `OS: 7848`, `OSD: 6126`. + +#### Integ Test Tar: + +| Integ Test Tar | x64 | arm64 | +|----------|----------|----------| +| OpenSearch | [x64](https://build.ci.opensearch.org/job/integ-test/4906/console) | [arm64](https://build.ci.opensearch.org/job/integ-test/4897/console) | +| OpenSearch Dashboards | [x64](https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/3531/console) | [arm64](https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/3530/console) | + +#### Integ Test RPM: + +| Integ Test RPM | x64 | arm64 | +|----------|----------|----------| +| OpenSearch | [x64](https://build.ci.opensearch.org/job/integ-test/4908/console) | [arm64](https://build.ci.opensearch.org/job/integ-test/4907/console) | +| OpenSearch Dashboards | [x64](https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/3532/console) | [arm64](https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/3533/console) | + +#### Docker Build and Scan: + +| Docker | build | scan | +|----------|----------|----------| +| OpenSearch | [Build](https://build.ci.opensearch.org/job/docker-build/3371/) | [Scan](https://build.ci.opensearch.org/job/docker-scan/1558/artifact/scan_docker_image.txt) | +| OpenSearch Dashboards | [Build](https://build.ci.opensearch.org/job/docker-build/3370/) | [Scan](https://build.ci.opensearch.org/job/docker-scan/1557/artifact/scan_docker_image.txt) | + + +#### Docker freeze RC: +This to ensure that [check-for-build.jenkinsfile](https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/check-for-build.jenkinsfile) wont re-build periodically and override the docker, the RC docker is created with build number. + +| Docker Freeze | copy | +|----------|----------| +| OpenSearch | [docker-copy](https://build.ci.opensearch.org/job/docker-copy/466/console) | +| OpenSearch Dashboards | [docker-copy](https://build.ci.opensearch.org/job/docker-copy/467/console) | + +#### Performance Test +| test_workload | sample_run | +|----------|----------| +|nyc_taxis |[perf-test](https://build.ci.opensearch.org/job/perf-test/4682/) | + + +#### Windows Integ Test + +Currently, the manually tested Windows zip is being evaluated and approved by the plugin teams for sign off. In order to test the Windows distribution, two instances need to be created: one with security features enabled and another without security. Afterward, API calls should be tested by launching the OpenSearch and Dashboard processes through direct execution of the `.bat` file. + +#### Broadcast and Communication + +Broadcast the release candidate in OpenSearch public slack workspace and the release GitHub issue using below format to gather votes. Following is the sample message that can be posted on the release issue. + + ###### The artifacts can be downloaded from: + + * OpenSearch - #Build-number (Note: Windows version does not have performance analyzer plugin) + * arm64 [[manifest](manifest.yml)] [[tar](opensearch-vesion-linux-arm64.tar.gz)] [[rpm](opensearch-vesion-linux-arm64.rpm)][[deb](opensearch-version-linux-arm64.deb)] + * x64 [[manifest](manifest.yml)] [[tar](opensearch-version-linux-x64.tar.gz)] [[rpm](opensearch-version-linux-x64.rpm)] [[deb](opensearch-version-linux-x64.deb)] [[windows](opensearch-version-windows-x64.zip)] + * OpenSearch Dashboards - #Build-number + * arm64 [[manifest](manifest.yml)] [[tar](opensearch-dashboards-vesion-linux-arm64.tar.gz)] [[rpm](opensearch-dashboards-vesion-linux-arm64.rpm)][[deb](opensearch-dashboards-version-linux-arm64.deb)] + * x64 [[manifest](manifest.yml)] [[tar](opensearch-dashboards-version-linux-x64.tar.gz)] [[rpm](opensearch-dashboards-version-linux-x64.rpm)] [[deb](opensearch-dashboards-version-linux-x64.deb)] [[windows](opensearch-dashboards-version-windows-x64.zip)] + + +As a release manager, it is essential to ensure the successful completion of all the above mentioned jobs. In the event of failures during integration tests or scans, the release manager should collaborate with the plugin teams and initiate a re-run to ensure that all jobs are executed successfully. +Post all the job related failures in the Release issue, Sample [post](https://github.com/opensearch-project/opensearch-build/issues/3331#issuecomment-1550461519). + +Note: Sometimes the integ-test jobs are flaky and might not pass due to several reasons with the plugin, in that case coordinate with the respective plugin team and get a manual sign off. Sample [manual sign off](https://github.com/opensearch-project/opensearch-build/issues/3331#issuecomment-1552191673). +All the failed logs are in s3 accessed through the cloudfront. Sample [link](https://github.com/opensearch-project/opensearch-build/issues/3331#issuecomment-1552148546) + + +In order to exclude the release from running periodically, at this point it is necessary for the release manager to lock the input manifest and update the `check-for-build.jenkins` to remove it from the scheduled execution, sample [PR](https://github.com/opensearch-project/opensearch-build/pull/3523/files). + + +### Final Release + +Release the artifacts to production distribution channels, update the website and inform the community of the release. + +#### Pre-Release + +Get the Go / No-Go votes from project management committee (PMC). Stage the release artifacts for production publishing process + +##### Promote Repos + +| Repo | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) | +| ---- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | +| YUM | [yum-os](https://build.ci.opensearch.org/job/distribution-promote-repos/46/) | [yum-osd](https://build.ci.opensearch.org/job/distribution-promote-repos/48/) | +| APT | [apt-os](https://build.ci.opensearch.org/job/distribution-promote-repos/) | [apt-osd](https://build.ci.opensearch.org/job/distribution-promote-repos/49/) | + + +##### Promote artifacts + + + +| Artifacts | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Windows | [os-windows-zip-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/233/) | [osd-windows-zip-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/234/) | +| Debian | [os-deb-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/235/), [os-deb-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/236/) | [osd-deb-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/237/), [osd-deb-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/238/) | +| TAR | [os-tar-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/243/), [os-tar-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/246/) | [osd-tar-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/244/), [osd-tar-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/245/) | +| RPM | [os-rpm-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/239/), [os-rpm-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/240/) | [osd-rpm-arm64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/241/), [osd-rpm-x64](https://build.ci.opensearch.org/job/distribution-promote-artifacts/242/) | + + +#### Release Notes + +Coordinate with the plugin teams and create a consolidates release notes. Sample [PR](https://github.com/opensearch-project/opensearch-build/pull/3532). Release manager can check if a plugin team has created a release notes or not using the [release notes tracker tool](https://github.com/opensearch-project/opensearch-build/tree/main/src/release_notes_workflow). Sample [run](https://build.ci.opensearch.org/job/release-notes-tracker/). + + +#### Release + +Release the artifacts to production distribution channels, update the website and inform the community of the release. + +##### Maven Promotion +Promote OpenSearch to maven, trigger the publish-to-maven worfklow, sample [run](https://build.ci.opensearch.org/job/publish-to-maven/17/console). + +##### Docker Promotion +Publish the images to docker and ECR, trigger the docker promotion workflow, samplr [run](https://build.ci.opensearch.org/job/docker-promotion/32/console) + +##### Website Changes +Coordinate with the documentation website team to ensure the changes are in place. + +##### Release Validation +Use the validation workflow to validate the published artifcats, sample [run](https://build.ci.opensearch.org/job/distribution-validation/3/console) + +#### Post Release + +Once the release is completed following are the activities that needs to be completed by the release manager: + +##### Release Tags +Create release tags for each OpenSearch and Dashboard components. +Sample [OpenSearch](https://build.ci.opensearch.org/job/distribution-release-tag-creation/78/), [OpenSearch Dashboards](https://build.ci.opensearch.org/job/distribution-release-tag-creation/77/) workflow runs. + +##### Input Manifest Update + +Replace refs in input manifest with tags and remove checks. The refs can be identified from the bundle or build manifest, sample [PR](https://github.com/opensearch-project/opensearch-build/pull/3534) that updates the input manifest. + +##### OpenSearch Build Release notes + +Generate distribution release notes for opensearch-build repository, sample [1.3.10](https://github.com/opensearch-project/opensearch-build/releases/tag/1.3.10) release details. + +##### Check for build Jenkinsfile update +Lower the frequency of builds for the release version of OpenSearch and/or OpenSearch Dashboards, sample [PR](https://github.com/opensearch-project/opensearch-build/pull/3523). + +##### Retrospective Issue +Create an issue for a retrospective, solicit feedback, and publish a summary, sample created [retro issue](https://github.com/opensearch-project/opensearch-build/issues/3535). + +##### Helm and Ansible Playbook release +Update and release the [Helm chart](https://github.com/opensearch-project/helm-charts) and [ansible playbook](https://github.com/opensearch-project/ansible-playbook) with the new OpenSearch and Dashboard version. +Sample helm chart [PR](https://github.com/opensearch-project/helm-charts/pull/431/files) and ansible [PR](https://github.com/opensearch-project/ansible-playbook/pull/131).