aws - add credential caching for aws assume role sessions

Add caching for AWS AssumeRole session credentials are not requested
for every single request. Sessions are valid for 15m by default but without
caching that did not matter. This will speed up requests and stop users from
hitting rate-limiting issues with the STS (session token service).

Fixes #37787

CHANGELOG.next.asciidoc

@@ -56,6 +56,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Support build of projects outside of beats directory {pull}36126[36126]
 - Support Elastic Agent control protocol chunking support {pull}37343[37343]
 - Upgrade elastic-agent-libs to v0.7.5. Removes obsolete "Treating the CommonName field on X.509 certificates as a host name..." deprecation warning for 8.0. {pull}37755[37755]
+- aws: Add credential caching for `AssumeRole` session tokens. {issue}37787[37787]
 
 *Auditbeat*
 

x-pack/libbeat/common/aws/credentials.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/url"
+	"time"
 
 	"github.com/aws/aws-sdk-go-v2/service/sts"
 
@@ -44,6 +45,13 @@ type ConfigAWS struct {
 	FIPSEnabled          bool              `config:"fips_enabled"`
 	TLS                  *tlscommon.Config `config:"ssl" yaml:"ssl,omitempty" json:"ssl,omitempty"`
 	DefaultRegion        string            `config:"default_region"`
+
+	// The duration of the role session. Defaults to 15m when not set.
+	AssumeRoleDuration time.Duration `config:"assume_role.duration"`
+
+	// AssumeRoleExpiryWindow will allow the credentials to trigger refreshing prior to the credentials
+	// actually expiring. If expiry_window is less than or equal to zero, the setting is ignored.
+	AssumeRoleExpiryWindow time.Duration `config:"assume_role.expiry_window"`
 }
 
 // InitializeAWSConfig function creates the awssdk.Config object from the provided config
@@ -154,8 +162,15 @@ func addAssumeRoleProviderToAwsConfig(config ConfigAWS, awsConfig *awssdk.Config
 		if config.ExternalID != "" {
 			aro.ExternalID = awssdk.String(config.ExternalID)
 		}
+		if config.AssumeRoleDuration > 0 {
+			aro.Duration = config.AssumeRoleDuration
+		}
+	})
+	awsConfig.Credentials = awssdk.NewCredentialsCache(stsCredProvider, func(options *awssdk.CredentialsCacheOptions) {
+		if config.AssumeRoleExpiryWindow > 0 {
+			options.ExpiryWindow = config.AssumeRoleExpiryWindow
+		}
 	})
-	awsConfig.Credentials = stsCredProvider
 }
 
 // addStaticCredentialsProviderToAwsConfig adds a static credentials provider to the current AWS config by using the keys stored in Beats config

x-pack/libbeat/docs/aws-credentials-config.asciidoc

@@ -15,6 +15,9 @@ To configure AWS credentials, either put the credentials into the {beatname_uc}
 * *fips_enabled*: Enabling this option instructs {beatname_uc} to use the FIPS endpoint of a service. All services used by {beatname_uc} are FIPS compatible except for `tagging` but only certain regions are FIPS compatible. See https://aws.amazon.com/compliance/fips/ or the appropriate service page, https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html, for a full list of FIPS endpoints and regions.
 * *ssl*: This specifies SSL/TLS configuration. If the ssl section is missing, the host's CAs are used for HTTPS connections. See <<configuration-ssl>> for more information.
 * *default_region*: Default region to query if no other region is set. Most AWS services offer a regional endpoint that can be used to make requests. Some services, such as IAM, do not support regions. If a region is not provided by any other way (environment variable, credential or instance profile), the value set here will be used.
+* *assume_role.duration*: The duration of the requested assume role session. Defaults to 15m when not set. AWS allows a maximum session duration between 1h and 12h depending on your maximum session duration policies.
+* *assume_role.expiry_window*: The expiry_window will allow refreshing the session prior to its expiration.
+  This is beneficial to prevent expiring tokens from causing requests to fail with an ExpiredTokenException.
 
 [float]
 ==== Supported Formats