[AWS][Billing] Support Configurable Group By Types

[BenB196]

Enhancement

Proposed commit message

WHAT: Implements the ability to configure the AWS Billing modules "Group By" values. Previously these were hardcoded to DIMENSION and TAG. But AWS supports defining any combo of these values.
WHY: Expands the use-cases that the AWS Billing module can support.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Given that the PR makes some configuration changes, I've made an attempt at keeping things backwards compatible. Let me know if there is a better way of doing this.
  • The default group by types align with the old, DIMENSION & TAG setup.
  • The old config values group_by_dimension_keys & group_by_tag_keys have been marked as deprecated, and are only referenced if the new config values group_by_primary_keys & group_by_secondary_keys are unset.
• The integration tests for the module are a bit light (what is there did pass), but I wasn't really able to understand how this framework works to expand it.
  • I did a majority of the testing locally, and I'll cover the configs I used to validate this PR below in the "How to test this PR locally" section.
• The majority of this PR can be described as two things, either:
  • Simple find/replace to make naming more "generic".
  • Updating the key mapping logic to be more "correct".
  • (An additional change made in this PR was to update the supported values of supportedDimensionKeys as they were last touched ~4 years ago and were out-of-date)

How to test this PR locally

1. Ensure you have access to an AWS account and have the ability to configure an AWS access method that works with Metricbeat and that has all required permissions.
2. Build Metricbeat from main and run it with a config. This run will be used as the "baseline" for validating this PR.

Base Config used for Testing

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"

3. Build Metricbeat from this PR and run it with the following configs:

Base Config used for Testing (no changes to data should be observed)

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"

New Config with two dimension keys

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
    group_by_primary_type: "DIMENSION"
    group_by_tag_keys:
      - "REGION"
    group_by_secondary_type: "DIMENSION"

New Config with two dimension keys (more keys)

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "USAGE_TYPE"
    group_by_primary_type: "DIMENSION"
    group_by_tag_keys:
      - "REGION"
      - "SERVICE"
    group_by_secondary_type: "DIMENSION"

New Config with two tag keys

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "aws:createdBy"
    group_by_primary_type: "TAG"
    group_by_tag_keys:
      - "Name"
    group_by_secondary_type: "TAG"

New Config using new config values, with two dimensions

metricbeat.modules:
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_primary_keys: #<----
      - "SERVICE"
      - "REGION"
    group_by_primary_type: "DIMENSION"
    group_by_secondary_keys: #<----
      - "LINKED_ACCOUNT"
    group_by_secondary_type: "DIMENSION"

Related issues

• Closes [Metricbeat] AWS Billing Module accept pairs of group_by values #34193

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @BenB196? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 5 min 34 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b aws-billing-support-dynamic-group-by-types upstream/aws-billing-support-dynamic-group-by-types
git merge upstream/main
git push upstream aws-billing-support-dynamic-group-by-types

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b aws-billing-support-dynamic-group-by-types upstream/aws-billing-support-dynamic-group-by-types
git merge upstream/main
git push upstream aws-billing-support-dynamic-group-by-types

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)