[UA] Low hanging fruit to improve reindexing performance

[jloleysens]

Today when upgrading indices through reindexing we make the following requests to (1) create an index and (2) request a reindex:

Index creation request

      createIndex = await esClient.indices.create({
        index: newIndexName,
        body: {
          settings,
          mappings,
        },
      });

Regardless of current index settings, create indices with:

• number_of_replicas: 0 (docs)
• refresh_interval: -1 (docs) turn off refresh since we don't expect to be servicing search on the new index at this time

After reindexing is done, before creating aliases, we can use the index update API to make the settings the same as the original index (or default)

Reindexing request

    const startReindexResponse = await esClient.reindex({
      refresh: true,
      wait_for_completion: false,
      body: {
        source: { index: indexName },
        dest: { index: reindexOp.attributes.newIndexName },
      },
    });

• slices: auto - (docs) possibly less low hanging fruit but could speed up reindexing by handling (more) shards in parallel. "Indexing performance scales linearly across available resources with the number of slices"

This has a few potential issues we should consider:

• No longer 1 task: 1 task that splits into N tasks. Completed tasks will by reported by the "parent" task returned from the original request. How much impact will this have on existing code? It appears that "cancel" logic will remain unchanged but the contents of the "reindex" task my be quite different.
• "Reindexing from remote clusters does not support manual or automatic slicing." - will this be an issue for UA?

Resources

• Tune for indexing speed

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)