Refreshing credentials in SharedCredentials

[zendesk-duncanboynton]

This pull request introduces a new feature to the SharedCredentials class, enabling opt-in credential refreshing. This functionality allows credentials to be automatically refreshed when they are near expiration, based on user-defined settings.

This functionality is crucial for long-running applications that would otherwise need to use unconventional or impractical methods to refresh credentials (e.g., using cat to access PATH variables during refresh). With this feature, SharedCredentials can handle refreshing directly, simplifying the process.

PR #1619 proposed this feature but was closed due to being an opt-out implementation, which posed a potential breaking change. This PR addresses those concerns by making it opt-in and maintaining backward compatibility.

[mullermp]

This is an interesting idea. I will discuss with the team.

[zendesk-duncanboynton]

@alextwoods @mullermp I'd greatly appreciate another look- I did my best to refine my changes and incorporate your comments

[alextwoods]

Does this actually refresh credentials? Aws.shared_config I don't think will re-load/parse the config files, so I'm not sure you would actually get new credentials from this.

[mullermp]

This would have to be SharedConfig.new, always, right?

[mullermp]

Can I ask more about your use case? Is it not possible to write your own custom CredentialProvider that sources your credentials from a dynamic source? Why are your credentials on disk (meant to be static) changing?

[zendesk-duncanboynton]

Can I ask more about your use case? Is it not possible to write your own custom CredentialProvider that sources your credentials from a dynamic source? Why are your credentials on disk (meant to be static) changing?

@mullermp Our use case involves long-running applications that require dynamic, temporary credentials for an IAM Role. These credentials need periodic refreshing to maintain security and access.

[mullermp]

Have you looked at ProcessCredentials? You can have some process that is responsible for returning credentials, and that process is responsible for vending new and rotated credentials. Shared config is for static credentials typically.

[zendesk-duncanboynton]

@mullermp that works but if the credentials are in a json file we have to do something like Aws::ProcessCredentials.new('cat $SHARED_CREDENTIALS_JSON') which is not a good experience

[mullermp]

Hmm. I see these as functionally equivalent. If you're writing out new rotated credentials on disk, you can write them to a json and echo them in process credentials. That's functionally equivalent to re-reading configuration in Ruby with parsing (which may be more expensive). Why do you feel it's a bad experience?

[alextwoods]

A slightly different question - how is the credentials file getting updated with the new temporary IAM credentials? I assume there is some process running which manages that.

Also - RE: using the ProcessCredentials as suggested by @mullermp - I believe that would only refresh if the credentials have an expiration time on them (in which case that expiration will be used to manage when the refresh happens).

[derricklam-zendesk]

A slightly different question - how is the credentials file getting updated with the new temporary IAM credentials?

Hi @alextwoods and @mullermp, the credentials file is refreshed by a secret sidecar. The issue we are attempting to solve is the ability for CredentialProvider to consume a credentials filepath. ex:

Aws::ProcessCredentials.new("/path/to/creds/file")

It looks like SharedCredentials supports a path parameter today, but ProcessCredentials does not. Would it be possible for ProcessCredentials to accept a path as well?

[mullermp]

The process that is executed can be literally whatever you want at whatever path you want. For your use case, are you not able to do simply echo path/to/creds.json?

[derricklam-zendesk]

Right, Aws::ProcessCredentials.new('cat $SHARED_CREDENTIALS_JSON') is our existing approach today

However, we are looking to make this small improvement so that we don't have to run process cat or echo

[alextwoods]

I think what I'm trying to understand a little bit is how your process updates the shared credentials file.

It sounds like you are using short term/temporary IAM credentials which should come with an expiration. A limitation of the shared credentials file (eg .aws/credentials) used by the SharedCredentials is that is does NOT support expiration. The refreshing of the credentials used by the SDK should be based on that expiration, which the ProcessCredentials do support automatically (as long as the expiration is included in the returned json). This seems like a better solution than relying on a configured refresh interval (which is easy to misconfigure and may not get updated correctly if you update the expiration times for your temporary credentials).

[mullermp]

I agree with Alex. I'm also wondering what you mean by "small improvement" - process credentials calling echo or cat is much faster than parsing an entire shared config file in Ruby. I actually think this would be a performance hit. We also discussed this with SDK teams internally and decided this is not something we should support in SDKs.

[mullermp]

I'm going to close this after further discussions with SDK engineers. We believe ProcessCredentials is the correct way to support this use case both from a practical and performance standpoint.