From dc671a76575a69ce77fdb98d31fd3a2d014f5fb3 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 27 Jul 2024 14:59:22 +0700 Subject: [PATCH 1/7] docs(self-hosted): external storage --- develop-docs/self-hosted/external-storage.mdx | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 develop-docs/self-hosted/external-storage.mdx diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx new file mode 100644 index 0000000000000..5d9ea9164b3f5 --- /dev/null +++ b/develop-docs/self-hosted/external-storage.mdx @@ -0,0 +1,86 @@ +--- +title: Self Hosted External Storage +sidebar_title: External Storage +sidebar_order: 90 +--- + +In some cases, storing Sentry data on-disk is not really something people can do. Sometimes, it's better if they can offload it into some bucket storage (like AWS S3 or Google Cloud Storage). + + + After changing configuration files, re-run the ./install.sh script, to rebuild and restart the containers. See the configuration section for more information. + + +## Sentry + +The Sentry service has a abstraction called "filestore" that handles storing attachment, sourcemap (release artifacts), and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. + +### Google Cloud Storage backend + +The configuration for GCS backend is pointed to `sentry.filestore.gcs.GoogleCloudStorage`. You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). + +```yaml +filestore.backend: "gcs" +filestore.options: + bucket_name: "..." +``` + +### S3 backend + + + Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + + +The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. + +```yaml +filestore.backend: 's3' +filestore.options: + bucket_acl: 'private' + default_acl: 'private' + access_key: '' + secret_key: '' + bucket_name: 'my-bucket' + region_name: 'auto' + endpoint_url: 'https://' # If you're not using AWS. + addressing_style: 'path' # For regular AWS S3, use "auto" or "virtual". For other S3-compatible API like MinIO or Ceph, use "path". + signature_version: 's3v4' +``` + +Refer to [botocore configuration](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) for valid configuration values. + +## Vroom + +Vroom is the service that handles profiling. By default the data for profiling is saved on local filesystem. On self-hosted deployment, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. + +### Google Cloud Storage backend + +You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). + +```bash +gs://my-bucket +``` + +### S3 backend + + + Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + + +```bash +# For regular AWS S3 +s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com + +# For other S3-compatible APIs +s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL +``` + +Additional environment variables should be provided: +- `AWS_ACCESS_KEY=foobar` +- `AWS_SECRET_KEY=foobar` +- `AWS_SESSION_TOKEN=foobar` (optional) + +Further explanation on the query string options: +- `region`: The AWS region for requests. +- `endpoint`: The endpoint URL (hostname only or fully qualified URI). +- `disableSSL`: A value of "true" disables SSL when sending requests. +- `s3ForcePathStyle`: A value of "true" forces the request to use path-style addressing. From 790cba94b49ae96d737a514644798706cc8b0f80 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Wed, 28 Aug 2024 14:28:48 +0700 Subject: [PATCH 2/7] Apply suggestions from code review Co-authored-by: Hubert Deng --- develop-docs/self-hosted/external-storage.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index 5d9ea9164b3f5..0f12c76dd0a4e 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -4,7 +4,7 @@ sidebar_title: External Storage sidebar_order: 90 --- -In some cases, storing Sentry data on-disk is not really something people can do. Sometimes, it's better if they can offload it into some bucket storage (like AWS S3 or Google Cloud Storage). +In some cases, storing Sentry data on-disk is not really something people can do. Sometimes, it's better to offload it into some bucket storage (like AWS S3 or Google Cloud Storage). After changing configuration files, re-run the ./install.sh script, to rebuild and restart the containers. See the configuration section for more information. @@ -12,7 +12,7 @@ In some cases, storing Sentry data on-disk is not really something people can do ## Sentry -The Sentry service has a abstraction called "filestore" that handles storing attachment, sourcemap (release artifacts), and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. +Sentry has an abstraction called "filestore" that handles storing attachments, sourcemaps (release artifacts), and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. ### Google Cloud Storage backend @@ -27,7 +27,7 @@ filestore.options: ### S3 backend - Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. @@ -50,7 +50,7 @@ Refer to [botocore configuration](https://botocore.amazonaws.com/v1/documentatio ## Vroom -Vroom is the service that handles profiling. By default the data for profiling is saved on local filesystem. On self-hosted deployment, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. +Vroom is the service that handles profiling. By default the data for profiling is saved on the local filesystem. On self-hosted deployments, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. ### Google Cloud Storage backend @@ -63,7 +63,7 @@ gs://my-bucket ### S3 backend - Although S3 support is available, it is not thoroughly tested and being used by Sentry SaaS internally. Therefore, it is not something that Sentry folks will offer very good support for it. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. ```bash From ad1cf1ce396ea6b057b458a0df0d86a8707ed735 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sun, 15 Sep 2024 07:27:38 +0700 Subject: [PATCH 3/7] feat(self-hosted): specify details on handling object retention for external storage See https://github.com/getsentry/self-hosted/issues/3315 --- develop-docs/self-hosted/external-storage.mdx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index 0f12c76dd0a4e..de76ad14ddbb6 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -14,6 +14,8 @@ In some cases, storing Sentry data on-disk is not really something people can do Sentry has an abstraction called "filestore" that handles storing attachments, sourcemaps (release artifacts), and replays. Filestore configuration for Sentry should be configured on the `sentry/config.yml` file. +**Important:** `sentry cleanup` command won't delete files that is stored on an external storage such as GCS or S3. You will have to configure your own cleanup mechanism by utilizing your storage provider's object retention configuration. This should be set accordingly to the `SENTRY_EVENTS_RETENTION_DAYS`, although you can set it as a different value than what's set on the Docker Compose file. + ### Google Cloud Storage backend The configuration for GCS backend is pointed to `sentry.filestore.gcs.GoogleCloudStorage`. You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). @@ -27,7 +29,7 @@ filestore.options: ### S3 backend - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. @@ -52,6 +54,8 @@ Refer to [botocore configuration](https://botocore.amazonaws.com/v1/documentatio Vroom is the service that handles profiling. By default the data for profiling is saved on the local filesystem. On self-hosted deployments, this should be done by overriding the `SENTRY_BUCKET_PROFILES` environment variable. It's also possible that additional environment variables should be added, depending on the backend of choice. +**Important:** `sentry cleanup` command won't delete files that is stored on an external storage such as GCS or S3. You will have to configure your own cleanup mechanism by utilizing your storage provider's object retention configuration. This should be set accordingly to the `SENTRY_EVENTS_RETENTION_DAYS`, although you can set it as a different value than what's set on the Docker Compose file. + ### Google Cloud Storage backend You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). @@ -63,7 +67,7 @@ gs://my-bucket ### S3 backend - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. ```bash From 85847b37b5594e1eb2d100f0a1edd4dba9fa83c2 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 28 Sep 2024 17:40:19 +0700 Subject: [PATCH 4/7] docs(self-hosted): better set up guide for gcs --- develop-docs/self-hosted/external-storage.mdx | 53 ++++++++++++++----- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index de76ad14ddbb6..249c5b6897a03 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -20,12 +20,27 @@ Sentry has an abstraction called "filestore" that handles storing attachments, s The configuration for GCS backend is pointed to `sentry.filestore.gcs.GoogleCloudStorage`. You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). +On your `sentry/config.yml` file, you will need to set the following: + ```yaml filestore.backend: "gcs" filestore.options: bucket_name: "..." ``` +If you set up via service account key, you will need to configure your `docker-compose.yml` file with the following: + +```yaml +x-sentry-defaults: &sentry-defaults + # ... + environment: + # The rest of the environment variables + GOOGLE_APPLICATION_CREDENTIALS: "/run/secrets/service_account.json" + volumes: + # The rest of the volumes + - "/path/to/service_account.json:/run/secrets/service_account.json:ro" +``` + ### S3 backend @@ -34,6 +49,8 @@ filestore.options: The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. +On your `sentry/config.yml` file, you will need to set the following: + ```yaml filestore.backend: 's3' filestore.options: @@ -60,8 +77,18 @@ Vroom is the service that handles profiling. By default the data for profiling i You will need to set `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For more information, refer to the [Google Cloud documentation for setting up authentication](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication). -```bash -gs://my-bucket +On your `docker-compose.yml` file, you will need to add the following (this assumes you are setting up via service account file): + +```yaml +services: + vroom: + environment: + # The rest of the environment variables + SENTRY_BUCKET_PROFILES: "gs://my-bucket" + GOOGLE_APPLICATION_CREDENTIALS: "/run/secrets/service_account.json" + volumes: + # The rest of the volumes + - "/path/to/service_account.json:/run/secrets/service_account.json:ro" ``` ### S3 backend @@ -70,19 +97,21 @@ gs://my-bucket Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. -```bash -# For regular AWS S3 -s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com +On your `docker-compose.yml` file, you will need to add the following: -# For other S3-compatible APIs -s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL +```yaml +services: + vroom: + environment: + # The rest of the environment variables + SENTRY_BUCKET_PROFILES: "s3://my-bucket?awssdk=v1®ion=us-west-1&endpoint=amazonaws.com" + # For other S3-compatible APIs + SENTRY_BUCKET_PROFILES: "s3://my-bucket?awssdk=v1®ion=any-region&endpoint=minio.yourcompany.com&s3ForcePathStyle=true&disableSSL" + AWS_ACCESS_KEY: "foobar" + AWS_SECRET_KEY: "foobar" + AWS_SESSION_TOKEN: "foobar" # (optional) ``` -Additional environment variables should be provided: -- `AWS_ACCESS_KEY=foobar` -- `AWS_SECRET_KEY=foobar` -- `AWS_SESSION_TOKEN=foobar` (optional) - Further explanation on the query string options: - `region`: The AWS region for requests. - `endpoint`: The endpoint URL (hostname only or fully qualified URI). From f792e657ce5425d78f82f2aa8372d981ab7a2f31 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 28 Sep 2024 19:09:19 +0700 Subject: [PATCH 5/7] Update external-storage.mdx --- develop-docs/self-hosted/external-storage.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index 249c5b6897a03..b51f53a668fc7 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -44,7 +44,7 @@ x-sentry-defaults: &sentry-defaults ### S3 backend - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experimental and not officially supported. The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. From 34c2e682587d6e8250c5c5fe0803d49cfe059b86 Mon Sep 17 00:00:00 2001 From: Reinaldy Rafli Date: Sat, 28 Sep 2024 19:09:25 +0700 Subject: [PATCH 6/7] Update external-storage.mdx --- develop-docs/self-hosted/external-storage.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index b51f53a668fc7..0be0f0305a286 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -94,7 +94,7 @@ services: ### S3 backend - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experiemental and not officially supported. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experimental and not officially supported. On your `docker-compose.yml` file, you will need to add the following: From 8b37b6f21b07b6ea67f20fa92a6686c8ff213552 Mon Sep 17 00:00:00 2001 From: Hubert Deng Date: Tue, 1 Oct 2024 13:11:51 -0700 Subject: [PATCH 7/7] add warning that external storage docs are unmaintained --- develop-docs/self-hosted/external-storage.mdx | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/develop-docs/self-hosted/external-storage.mdx b/develop-docs/self-hosted/external-storage.mdx index 0be0f0305a286..8e3772bcb664d 100644 --- a/develop-docs/self-hosted/external-storage.mdx +++ b/develop-docs/self-hosted/external-storage.mdx @@ -6,6 +6,9 @@ sidebar_order: 90 In some cases, storing Sentry data on-disk is not really something people can do. Sometimes, it's better to offload it into some bucket storage (like AWS S3 or Google Cloud Storage). + + The docs below are contributed by the community. Sentry does not officially provide support for external storage backends that extend beyond our typical docker compose setup. Follow these guidelines at your own risk as these docs are not maintained nor kept up to date! + After changing configuration files, re-run the ./install.sh script, to rebuild and restart the containers. See the configuration section for more information. @@ -44,7 +47,7 @@ x-sentry-defaults: &sentry-defaults ### S3 backend - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experimental and not officially supported. + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS. Therefore, it is experimental and not officially supported. The configuration for S3-compatible backend is pointed to `sentry.filestore.s3.S3Boto3Storage`. @@ -93,8 +96,8 @@ services: ### S3 backend - - Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS internally. Therefore, it is experimental and not officially supported. + + Although S3 support is available, it is not thoroughly tested and is not used by Sentry SaaS. Therefore, it is experimental and not officially supported. On your `docker-compose.yml` file, you will need to add the following: