Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DISCUSSION] Should flagd support S3-compatible storage providers #864

Closed
agardnerIT opened this issue Aug 26, 2023 · 9 comments
Closed
Labels
enhancement New feature or request Needs Triage This issue needs to be investigated by a maintainer

Comments

@agardnerIT
Copy link
Contributor

agardnerIT commented Aug 26, 2023

Requirements

Can / should flagd be extended to support s3-compatible storage services.

These would include:

  • S3 (obviously)
  • DigitalOcean spaces
  • MinIO
  • Cloudflare R2
  • any other existing or future S3 compatible provider

While flagd already supports the Authorisation header, the difficulty seems to be that S3 auth is built from a dynamically generated (HMAC) header. Presumably calculating this would mean enhancements to flagd at the core.

Signature Timeouts

Amazon's documentation states that:

References

@agardnerIT agardnerIT added enhancement New feature or request Needs Triage This issue needs to be investigated by a maintainer labels Aug 26, 2023
@beeme1mr
Copy link
Member

Thanks for starting the discussion @agardnerIT. Let's leave this open for a bit to gauge interest. For those who are interested, please add a comment describing your use case and why an S3-compatible sync provider would address it.

@agardnerIT
Copy link
Contributor Author

I'm going to close this as there is no immediate interest / need. We can re-open if someone else is interested in future.

My usecase was to store / retrieve flag values from cheap object storage like S3 and R2 etc.

@austindrenski
Copy link
Member

Hey folks, just getting started with OpenFeature and flagd after reading about it in the CNCF announcement blog post over the holidays, and I've pretty quickly found myself yearning for built-in support for S3-compatible storage.

I was very excited to learn of this project, and I suspect there will be many more newcomers in the coming months looking to trial flagd in non-critical production workloads (as many have with otelcol over the last few years).

Far from a blocker to getting started, but in-the-box support would certainly help lower the barrier to entry for curious onlookers such as myself.

@beeme1mr
Copy link
Member

beeme1mr commented Jan 2, 2024

Hey @austindrenski, thanks for letting us know you would like an S3-compatible sync. It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

FYI @toddbaert

@beeme1mr
Copy link
Member

beeme1mr commented Jan 2, 2024

It looks like Open Policy Agent has done something similar already.

open-policy-agent/opa#5489

@beeme1mr
Copy link
Member

beeme1mr commented Jan 2, 2024

I'm going to reopen this discussion.

@beeme1mr beeme1mr reopened this Jan 2, 2024
@austindrenski
Copy link
Member

austindrenski commented Jan 3, 2024

You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

That's actually what I tried after posting here earlier and, after some tinkering, I got things working without making the bucket entirely public (i.e. anonymous access within my VPC, no public access outside of it) by using Block all public access + a bucket policy along the lines of:

// sharing for future readers, but be sure to consult your friendly
// neighborhood SecOps team before yeeting this anywhere near prod!
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-***"
        }
      },
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:s3:::open-feature/flagd/*"
    }
  ]
}

It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. [...]

+1, skimmed through the sync providers earlier and came to the same conclusion of an s3 sync provider wrapping the http sync provider, but implementing the auth part without a direct dependency on (at least) github.com/aws/aws-sdk-go-v2/config is where I stumbled out of my depth.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

Really appreciate you looking into this, on whatever timetable you're able to.

I think finding a good answer for how to handle auth for AWS et al. will be important if flagd plans to support other storage backends in the future, which isn't to say it needs to, but given the success of otelcol and otelcol-contrib, I'm guessing there may be a similar push for broad, plug-and-play support from flagd (or some future flagd-contrib distro).

For example, while we can use S3-based flag definitions with a little bit of work today, it would be a neat option in the long-term to integrate with DynamoDB (or some other general object/document store) for read/write scenarios. (This is presumably far out of scope for now, but the building blocks here are pretty exciting.)

@toddbaert
Copy link
Member

You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

That's actually what I tried after posting here earlier and, after some tinkering, I got things working without making the bucket entirely public (i.e. anonymous access within my VPC, no public access outside of it) by using Block all public access + a bucket policy along the lines of:

// sharing for future readers, but be sure to consult your friendly
// neighborhood SecOps team before yeeting this anywhere near prod!
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-***"
        }
      },
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:s3:::open-feature/flagd/*"
    }
  ]
}

It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. [...]

+1, skimmed through the sync providers earlier and came to the same conclusion of an s3 sync provider wrapping the http sync provider, but implementing the auth part without a direct dependency on (at least) github.com/aws/aws-sdk-go-v2/config is where I stumbled out of my depth.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

Really appreciate you looking into this, on whatever timetable you're able to.

I think finding a good answer for how to handle auth for AWS et al. will be important if flagd plans to support other storage backends in the future, which isn't to say it needs to, but given the success of otelcol and otelcol-contrib, I'm guessing there may be a similar push for broad, plug-and-play support from flagd (or some future flagd-contrib distro).

For example, while we can use S3-based flag definitions with a little bit of work today, it would be a neat option in the long-term to integrate with DynamoDB (or some other general object/document store) for read/write scenarios. (This is presumably far out of scope for now, but the building blocks here are pretty exciting.)

Glad to hear you had a degree of success!

Agreed on all counts. @beeme1mr and I had a brief chat about this, and basically we came to the exact same conclusions, and share the same questions and goals.

@beeme1mr
Copy link
Member

beeme1mr commented Aug 2, 2024

I've created an issue to add s3 support.
#1376

@beeme1mr beeme1mr closed this as completed Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Needs Triage This issue needs to be investigated by a maintainer
Projects
None yet
Development

No branches or pull requests

4 participants