❇️ Central OpenSearch

[jacobwoffenden]

User Story

As a Observability Platform product engineer
I want to offer OpenSearch for publishing logs
So that we have a viable alternative to shipping EKS logs to CloudWatch

User Type(s)

Observability Platform engineering and Observability Platform tenants

Value

This isn't based on any feedback from tenants, but I am concerned that shipping to CloudWatch Logs might not scale for Cloud Platform or Data Platform's EKS clusters.

Assumptions / Hypothesis / Questions / Unknowns

Hypothesis

If we... [do a thing]
Then... [this will happ]

Proposal

Build a central OpenSearch cluster that uses IAM (sigv4) authentication to write data, similar to how we offer Amazon Managed Prometheus

Each tenant role can only write to specific index

We can then create a Grafana data source which is scoped to that index and uses data source permissions to control access

Unknowns

Potential pitfalls that could cause the story to expand beyond its original scope. Ideally this section will remain blank.

Definition of done

• spike timeboxed
• options investigated
• new stories raised based on outcome of spike

Reference

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html

https://docs.fluentbit.io/manual/pipeline/outputs/opensearch

How to write good user stories