Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Jaeger-V2] Add configurable index data layout and rollover frequency #5790

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

[Jaeger-V2] Add configurable index data layout and rollover frequency #5790

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
47c0a63
Add configurable index data layout and rollover frequency
JaredTan95 Sep 2, 2024
4ce1b97
remove indexprefix
JaredTan95 Sep 3, 2024
037f479
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 3, 2024
2e86cd2
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 6, 2024
cf818b8
fix prefix
JaredTan95 Sep 7, 2024
890daa3
fix prefix
JaredTan95 Sep 8, 2024
3e65f6b
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 8, 2024
f99515e
fix prefix
JaredTan95 Sep 8, 2024
b712a72
Merge branch 'configure-index-daltalayout-and-frequency' of https://g…
JaredTan95 Sep 8, 2024
b9cebd6
fix-opensearch-config
yurishkuro Sep 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions cmd/es-rollover/app/init/action.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,19 @@ type Action struct {

func (c Action) getMapping(version uint, templateName string) (string, error) {
mappingBuilder := mappings.MappingBuilder{
TemplateBuilder: es.TextTemplateBuilder{},
PrioritySpanTemplate: int64(c.Config.PrioritySpanTemplate),
PriorityServiceTemplate: int64(c.Config.PriorityServiceTemplate),
PriorityDependenciesTemplate: int64(c.Config.PriorityDependenciesTemplate),
PrioritySamplingTemplate: int64(c.Config.PrioritySamplingTemplate),
Shards: int64(c.Config.Shards),
Replicas: int64(c.Config.Replicas),
IndexPrefix: c.Config.IndexPrefix,
UseILM: c.Config.UseILM,
ILMPolicyName: c.Config.ILMPolicyName,
EsVersion: version,
TemplateBuilder: es.TextTemplateBuilder{},
Indices: c.Config.Indices,
UseILM: c.Config.UseILM,
ILMPolicyName: c.Config.ILMPolicyName,
EsVersion: version,
}

// TODO: does not have a separate flag for Indices
mappingBuilder.Indices.Spans.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Services.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Dependencies.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Sampling.Prefix = c.Config.IndexPrefix

return mappingBuilder.GetMapping(templateName)
}

Expand Down
28 changes: 16 additions & 12 deletions cmd/es-rollover/app/init/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"github.com/spf13/viper"

"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
cfg "github.com/jaegertracing/jaeger/pkg/es/config"
)

const (
Expand All @@ -23,12 +24,7 @@ const (
// Config holds configuration for index cleaner binary.
type Config struct {
app.Config
Shards int
Replicas int
PrioritySpanTemplate int
PriorityServiceTemplate int
PriorityDependenciesTemplate int
PrioritySamplingTemplate int
cfg.Indices
}

// AddFlags adds flags for TLS to the FlagSet.
Expand All @@ -43,10 +39,18 @@ func (*Config) AddFlags(flags *flag.FlagSet) {

// InitFromViper initializes config from viper.Viper.
func (c *Config) InitFromViper(v *viper.Viper) {
c.Shards = v.GetInt(shards)
c.Replicas = v.GetInt(replicas)
c.PrioritySpanTemplate = v.GetInt(prioritySpanTemplate)
c.PriorityServiceTemplate = v.GetInt(priorityServiceTemplate)
c.PriorityDependenciesTemplate = v.GetInt(priorityDependenciesTemplate)
c.PrioritySamplingTemplate = v.GetInt(prioritySamplingTemplate)
c.Indices.Spans.Shards = v.GetInt(shards)
c.Indices.Services.Shards = v.GetInt(shards)
c.Indices.Dependencies.Shards = v.GetInt(shards)
c.Indices.Sampling.Shards = v.GetInt(shards)

c.Indices.Spans.Replicas = v.GetInt(replicas)
c.Indices.Services.Replicas = v.GetInt(replicas)
c.Indices.Dependencies.Replicas = v.GetInt(replicas)
c.Indices.Sampling.Replicas = v.GetInt(replicas)

c.Indices.Spans.Priority = v.GetInt(prioritySpanTemplate)
c.Indices.Services.Priority = v.GetInt(priorityServiceTemplate)
c.Indices.Dependencies.Priority = v.GetInt(priorityDependenciesTemplate)
c.Indices.Sampling.Priority = v.GetInt(prioritySamplingTemplate)
}
12 changes: 6 additions & 6 deletions cmd/es-rollover/app/init/flags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@ func TestBindFlags(t *testing.T) {
require.NoError(t, err)

c.InitFromViper(v)
assert.Equal(t, 8, c.Shards)
assert.Equal(t, 16, c.Replicas)
assert.Equal(t, 300, c.PrioritySpanTemplate)
assert.Equal(t, 301, c.PriorityServiceTemplate)
assert.Equal(t, 302, c.PriorityDependenciesTemplate)
assert.Equal(t, 303, c.PrioritySamplingTemplate)
assert.Equal(t, 8, c.Indices.Spans.Shards)
assert.Equal(t, 16, c.Indices.Spans.Replicas)
assert.Equal(t, 300, c.Indices.Spans.Priority)
assert.Equal(t, 301, c.Indices.Services.Priority)
assert.Equal(t, 302, c.Indices.Dependencies.Priority)
assert.Equal(t, 303, c.Indices.Sampling.Priority)
}
8 changes: 4 additions & 4 deletions cmd/esmapping-generator/app/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import (
type Options struct {
Mapping string
EsVersion uint
Shards int64
Replicas int64
Shards int
Replicas int
IndexPrefix string
UseILM string // using string as util is being used in python and using bool leads to type issues.
ILMPolicyName string
Expand Down Expand Up @@ -40,12 +40,12 @@ func (o *Options) AddFlags(command *cobra.Command) {
esVersionFlag,
7,
"The major Elasticsearch version")
command.Flags().Int64Var(
command.Flags().IntVar(
&o.Shards,
shardsFlag,
5,
"The number of shards per index in Elasticsearch")
command.Flags().Int64Var(
command.Flags().IntVar(
&o.Replicas,
replicasFlag,
1,
Expand Down
9 changes: 5 additions & 4 deletions cmd/esmapping-generator/app/flags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,9 @@ func TestOptionsWithDefaultFlags(t *testing.T) {

assert.Equal(t, "", o.Mapping)
assert.Equal(t, uint(7), o.EsVersion)
assert.Equal(t, int64(5), o.Shards)
assert.Equal(t, int64(1), o.Replicas)
assert.Equal(t, 5, o.Shards)
assert.Equal(t, 1, o.Replicas)

assert.Equal(t, "", o.IndexPrefix)
assert.Equal(t, "false", o.UseILM)
assert.Equal(t, "jaeger-ilm-policy", o.ILMPolicyName)
Expand All @@ -44,8 +45,8 @@ func TestOptionsWithFlags(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, "jaeger-span", o.Mapping)
assert.Equal(t, uint(7), o.EsVersion)
assert.Equal(t, int64(5), o.Shards)
assert.Equal(t, int64(1), o.Replicas)
assert.Equal(t, 5, o.Shards)
assert.Equal(t, 1, o.Replicas)
assert.Equal(t, "test", o.IndexPrefix)
assert.Equal(t, "true", o.UseILM)
assert.Equal(t, "jaeger-test-policy", o.ILMPolicyName)
Expand Down
21 changes: 15 additions & 6 deletions cmd/esmapping-generator/app/renderer/render.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (

"github.com/jaegertracing/jaeger/cmd/esmapping-generator/app"
"github.com/jaegertracing/jaeger/pkg/es"
cfg "github.com/jaegertracing/jaeger/pkg/es/config"
"github.com/jaegertracing/jaeger/plugin/storage/es/mappings"
)

Expand All @@ -25,14 +26,22 @@ func GetMappingAsString(builder es.TemplateBuilder, opt *app.Options) (string, e
return "", err
}

indexOpts := cfg.IndexOptions{
Prefix: opt.IndexPrefix,
Shards: opt.Shards,
Replicas: opt.Shards,
}
mappingBuilder := mappings.MappingBuilder{
TemplateBuilder: builder,
Shards: opt.Shards,
Replicas: opt.Replicas,
EsVersion: opt.EsVersion,
IndexPrefix: opt.IndexPrefix,
UseILM: enableILM,
ILMPolicyName: opt.ILMPolicyName,
Indices: cfg.Indices{
Spans: indexOpts,
Services: indexOpts,
Dependencies: indexOpts,
Sampling: indexOpts,
},
EsVersion: opt.EsVersion,
UseILM: enableILM,
ILMPolicyName: opt.ILMPolicyName,
}
return mappingBuilder.GetMapping(opt.Mapping)
}
Expand Down
29 changes: 27 additions & 2 deletions cmd/jaeger/config-elasticsearch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,35 @@ extensions:
backends:
some_storage:
elasticsearch:
index_prefix: "jaeger-main"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: we may want to rethink if we still want a single field in the config for index prefix, which would apply if the individual index options didn't override it explicitly. This would simplify the configuration while maintaining the flexibility.

indices:
JaredTan95 marked this conversation as resolved.
Show resolved Hide resolved
spans:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
services:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
dependencies:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
sampling:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
another_storage:
elasticsearch:
index_prefix: "jaeger-archive"
indices:
spans:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: is the services index being built at all when the storage is in the archive mode? I don't think it should be, but in this configuration services.prefix: "jaeger-main" would be the default, and the archive storage is not told in any way that it is indeed archive.

prefix: "jaeger-archive"

receivers:
otlp:
Expand Down
30 changes: 27 additions & 3 deletions cmd/jaeger/config-opensearch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,35 @@ extensions:
backends:
some_storage:
opensearch:
index_prefix: "jaeger-main"

indices:
spans:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
services:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
dependencies:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
sampling:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
another_storage:
opensearch:
index_prefix: "jaeger-main"
indices:
spans:
prefix: "jaeger-archive"

receivers:
otlp:
Expand Down
Loading