[exporter/prometheusremotewriteexporter] Retry on 5xx status codes using cenkalti/backoff client

[vasireddy99]

Description: Fix the Retry on 5xx status code using cenkalti/backoff package and added unit test.

Thanks to @rapphil for proposing the solution.

Link to tracking Issue: #20304

Testing: Added unit tests for retryOn5xx and noRetryOn4xx

Prometheus compliance test results -

--- PASS: TestRemoteWrite (20.10s)
    --- PASS: TestRemoteWrite/otelcollector (0.01s)
        --- PASS: TestRemoteWrite/otelcollector/NameLabel (10.05s)
        --- PASS: TestRemoteWrite/otelcollector/Invalid (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/RepeatedLabels (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/Retries400 (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/SortedLabels (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/Up (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/JobLabel (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/HonorLabels (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/EmptyLabels (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/Counter (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/InstanceLabel (10.06s)
        --- PASS: TestRemoteWrite/otelcollector/Ordering (17.10s)
        --- PASS: TestRemoteWrite/otelcollector/Histogram (10.02s)
        --- PASS: TestRemoteWrite/otelcollector/Gauge (10.02s)
        --- PASS: TestRemoteWrite/otelcollector/Headers (10.02s)
        --- PASS: TestRemoteWrite/otelcollector/Retries500 (10.03s)
        --- PASS: TestRemoteWrite/otelcollector/Staleness (10.03s)
        --- PASS: TestRemoteWrite/otelcollector/Summary (10.03s)
        --- PASS: TestRemoteWrite/otelcollector/Timestamp (10.03s)
PASS
ok      github.com/prometheus/compliance/remote_write   20.368s

[dmitryax]

Do you want to remove exporterhelper.WithRetry from the factory, given that it's handled separately now?

[vasireddy99]

Do you want to remove exporterhelper.WithRetry from the factory, given that it's handled separately now?

I would leave the setting for now and can take the follow up on this. As I don't want to disable retries in this PR.

[rapphil]

@vasireddy99 I think what @dmitryax suggested is reasonable and valid to include in the scope of this PR: we never return a retryable error and this component cannot handle retries in the helper level, therefore we should remove this capability of this component here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/factory.go#L60

[vasireddy99]

@vasireddy99 I think what @dmitryax suggested is reasonable and valid to include in the scope of this PR: we never return a retryable error and this component cannot handle retries in the helper level, therefore we should remove this capability of this component here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/factory.go#L60

Makes sense, I agree with that to, but wanted to discuss and also double nthe usage. Will commit the changes

[vasireddy99]

Commited the changes.
This Failure is not related to the PR- quick note.

[dmitryax]

@vasireddy99, doesn't seem like anything new was pushed, just merged main

[vasireddy99]

@vasireddy99, doesn't seem like anything new was pushed, just merged main

Thanks @dmitryax, Please review

[vasireddy99]

This PR #24729 shall also be review/merged after this PR is merged. Just adding the note.

[codecov]

Codecov Report

Patch coverage is 87.75% of modified lines.

Files Changed	Coverage
exporter/prometheusremotewriteexporter/factory.go	ø
exporter/prometheusremotewriteexporter/exporter.go	87.75%

📢 Thoughts on this report? Let us know!.