[receiver/cloudfoundry]: move datapoint level attributes to resource level

[odubajDT]

Description:
Introduce a feature gate enable copying envelope tags to the metrics as resource attributes instead of datapoint attributes.

Link to tracking Issue: #34824

[crobert-1]

Just a high level review this round, I'll review again once naming is a bit more sorted out 👍

[crobert-1]

To be able to conform to OTel metrics formatting with resource attributes, we need to try to create more of a hierarchy here with resource metrics, rather than a flat layout as we have without resource attributes here.

If two metrics share the same resource attributes, they should be in the same resource metric. However, the way the logic currently is written, each metric is in its own resource metric, regardless of whether or not it shares identical resource attributes with other metrics.

I could see two main ways of doing this:

1. When converting envelopes to metrics, check if a resource metric already exists with the parsed resource attributes. If yes, insert the new metric into the existing resource metric.
2. After all envelopes have been converted to metrics, deduplicate the resource metrics to merge them.

I think option 1. may be more straight forward and clear to follow the logic of, but I'd welcome more input here.

[odubajDT]

good point, will adapt using the 1. option

[odubajDT]

adapted to the new logic, please have a look if you have time. Still adding some more tests, but not done with those yet, so please keep that i mind for now

[crobert-1]

As @jriguera pointed out in the original issue, not all of the attributes can be considered resource attributes. We should have a predefined list of resource attributes that are copied when the feature gate is enabled.

When the feature gate is enabled the predefined attributes are copied to resource attributes. The rest of the attributes will be datapoint attributes as they are today. The predefined list will only be resource attributes, they will not be both resource attributes and datapoint attributes.

     -> org.cloudfoundry.index: Str(4b406f26-f1ec-4d66-8b92-f09ab6105d46)
     -> org.cloudfoundry.ip: Str(10.0.4.8)
     -> org.cloudfoundry.job: Str(compute)
     -> org.cloudfoundry.instance_group: Str(compute)
     -> org.cloudfoundry.origin: Str(system_metrics_agent)
     -> org.cloudfoundry.source_id: Str(system_metrics_agent)

@jriguera From your comment on the original issue it sounded like you don't believe the above attributes should be considered resource attributes. Is that correct? If so, can you expand your reasoning? My understanding is that since they describe the resource the metrics are coming from they can be considered resource attributes. Source

[odubajDT]

ok so the list above is the full list of resource attributes, is that correct?

[odubajDT]

AFAIK, this should be the full list of resource attributes, please correct me if I am missing something

-> org.cloudfoundry.index
-> org.cloudfoundry.ip
-> org.cloudfoundry.deployment
-> org.cloudfoundry.id
-> org.cloudfoundry.job
-> org.cloudfoundry.product
-> org.cloudfoundry.instance_group
-> org.cloudfoundry.instance_id
-> org.cloudfoundry.origin
-> org.cloudfoundry.system_domain
-> org.cloudfoundry.source_id

[odubajDT]

adapted the logic, here is the list (for now) of resource attributes

[odubajDT]

@crobert-1 would you have time to take another look?

[crobert-1]

@crobert-1 would you have time to take another look?

I was hoping to hear back from @jriguera to try and get a consensus around which attributes should be moved to resource attributes here. If we don't hear back soon we can move this forward. 👍

[jriguera]

I will continue with my reviewing tomorrow

[odubajDT]

I will continue with my reviewing tomorrow

Hi @jriguera, any other comments?

[odubajDT]

cc @jriguera