[jaeger-v2] Rethink components configuration to align with OTEL

[yurishkuro]

Right now several PRs that are trying to add support for standard storage backends to v2 are reusing the same configuration for storage as we had in v1. Now is a good time to rethink that configuration, in particular:

• better align with standard components of OTEL Collector. For example, the gRPC storage ([jaeger-v2] Add support for GRPC storarge #5228) is conceptually very similar to OTLP Exporter - they both send data over gRPC to a remote service. Our configuration is pretty slim, while OTLP Exporter's configuration has a lot more features, there is no reason why we cannot support them (would be even better if we could simply reuse OTEL's configuration utilities for gRPC)
• our storage configs have been accumulating over time, we can see if there are better logical groupings. For example, ES config is a flat list of everything, we can group many flags into sub-types so that the YAML config looks a lot more structured than using long field names like priority_dependencies_template
• ensure that we have sensible defaults. For example, in [jaeger-v2] add cassandra e2e integration tests #5398 basic options had to be added to Cassandra configuration, even though in v1 many of them are already defaulted to these values, so we're clearly missing defaulting mechanism. It's somewhat complicated by the fact that the storage extension cannot actually define the defaults in CreateDefaultConfig because at that point we don't known which storage will be used.

Scope

• Cassandra storage config -- [jaeger-v2] Align Cassandra Storage Config With OTEL #5928
• Elasticsearch / OpenSearch storage config -- [Jaeger-V2] Add configurable index data layout and rollover frequency #5790
• Badger storage config -- [jaeger-v2] Align Badger Storage Config With OTEL #5926
• Memory storage config -- [jaeger-v2] Align Memory Storage Config With OTEL #5923
• Kafka config
• Adaptive sampling processor / extension config
• Query service config -- [jaeger-v2] Align Query Service Config With OTEL #5996
• Collector-specific config
• Utilize env var defaults available since OTEL v0.110 e.g.

  jaeger/cmd/jaeger/internal/integration/kafka_test.go

  Line 24 in c655e56

  // we can change the config to use topic: "${KAFKA_TOPIC:-jaeger-spans}"

Tasks / outcomes

For each of the above components, we need to ensure

• All config fields are tagged with mapstructure such that they can be addressable from YAML via "good" names
• Configs implement Validate() -- example
• Configs via YAML support the same defaults as in jaeger-v1
• Configs reuse standard elements of OTEL configs whenever possible, e.g. TLS, clientcfg, etc.
• Configuration looks "logical" and properly grouped, not flattened with long weird names (good example [Jaeger-V2] Add configurable index data layout and rollover frequency #5790)
• We create a migration guide document with a table that lists v1 CLI flags and the corresponding v2 config fields (example for Kafka)

How to work on this meta-issue

• pick a component / area
• create a dedicated ticket for it
• copy tasks / outcomes from above
• use the ticket for design discussions, updates, links to PRs

[mahadzaryab1]

@yurishkuro What is the status of this issue? Is there anything I can help with?

[yurishkuro]

yes, plenty. I just updated the description.

[mahadzaryab1]

@yurishkuro are the configs for v1 supposed to be in plugin/storage/*/options.go and the configs for v2 supposed to be in pkg/*/config.go?

[yurishkuro]

yes, that's accurate

[mahadzaryab1]

@yurishkuro how come some stores still have the v2 configuration living in plugin/storage/*? do we want to move those over?

[yurishkuro]

We are in the transition period. Moving them doesn't solve anything, having configs side by side is actually convenient. The long term goal is to only have v2 configurations.

Wwe started v2 by just wrapping v1 configs. For some backends we did preliminary work of cleaning up and making v2 the "primary" config, i.e. the Factory uses v2 config directly, but v1 config still drives the v1 cli flags and can be transformed into v2. We could continue using this pattern. The reason we want v2 configs to be primary is to ensure that all their settings are actually used by the storage implementation - there's less guarantee of that if v1 is primary and v2 is translated into v1.

But overall this ticket is about what v2 config look like, rather than where they are located.

[mahadzaryab1]

@yurishkuro sounds good! and where should the migration doc be created? is any public google doc fine? Or do we want to have them in a README in this repo?

[yurishkuro]

yes a google doc is fine

[mahadzaryab1]

@yurishkuro PR for memory storage component at #5925

[mahadzaryab1]

@yurishkuro PR for badger storage component at #5927

[mahadzaryab1]

@yurishkuro PR for Cassandra component ready for review at #5949.

[mahadzaryab1]

@yurishkuro PR for Query Service ready for review at #5998

[mahadzaryab1]

@yurishkuro we can link the following PRs to the issue description for badger:

• [jaeger-v2] Add Validation And Comments to Badger Storage Config #5927
• [jaeger-v2] Consolidate Options And NamespaceConfig For Badger Storage #5937

[mahadzaryab1]

@yurishkuro what is the Collector-specific config checklist item referring to?

[yurishkuro]

what is the Collector-specific config checklist item referring to?

if you run SPAN_STORAGE_TYPE=memory go run ./cmd/collector help there are a bunch of collector-specific flags printed that are not related to storage but to the collector itself. Most of them are related to http/grpc endpoints, but some could control the overall collector's behavior. Ideally we need to create a gdoc mapping all these flags to the corresponding OTEL config locations and if any are not supported decide what to do about them.

[mahadzaryab1]

what is the Collector-specific config checklist item referring to?

if you run SPAN_STORAGE_TYPE=memory go run ./cmd/collector help there are a bunch of collector-specific flags printed that are not related to storage but to the collector itself. Most of them are related to http/grpc endpoints, but some could control the overall collector's behavior. Ideally we need to create a gdoc mapping all these flags to the corresponding OTEL config locations and if any are not supported decide what to do about them.

@yurishkuro Got it! So for this part we just need to create a migration guide? No code changes?

[yurishkuro]

Perhaps, depends on what the comparison would show.

[mahadzaryab1]

@yurishkuro Thanks! Would you also be able to help me understand how to solve Utilize env var defaults one OTEL supports them in 0.110 ? How do we determine which things need to be defaulted? As well, where do these defaults needs to be utilized? Is it just in the docker compose setups?

[yurishkuro]

We had a number of nasty workarounds because we couldn't use defaults. For instance, Kafka e2e test needs to physically rewrite the config because we can't just have a var for encoding without default (it makes the config invalid by default). I think it's sufficient to just fix that to close the item, since we can add more vars/defaults incrementally as the needs arise.