Add action to create index from a source index #118890
Conversation
Add new action that create and index from a source index, copying settings and mappings from the source index. This was refactored out of ReindexDataStreamIndexAction
elasticsearchmachine
added
the
needs:triage
| .andThenApply(ignored -> new ReindexDataStreamIndexAction.Response(destIndexName)) | ||
| .addListener(listener); | ||
| } | ||
|
|
||
| private void setBlockWrites(String sourceIndexName, ActionListener<AcknowledgedResponse> listener) { | ||
| logger.debug("Setting write block on source index [{}]", sourceIndexName); | ||
| final Settings readOnlySettings = Settings.builder().put(IndexMetadata.INDEX_BLOCKS_WRITE_SETTING.getKey(), true).build(); | ||
| var updateSettingsRequest = new UpdateSettingsRequest(readOnlySettings, sourceIndexName); | ||
| client.admin().indices().updateSettings(updateSettingsRequest, new ActionListener<>() { |
There was a problem hiding this comment.
Add block-writes and read-only to the source and dest indices was originally done with an UpdateSettingsRequest. While this works, I noticed that ILM used the AddIndexBlock api which seems a bit safer, so I used this as well.
There was a problem hiding this comment.
LGTM. I assume there's a rest action coming that will pretty directly use this new action, and that kibana will use that. Just for my own sake, the parts of reindexing an index for upgrade are:
- add write block to the source index
- delete destination index (if it exists)
- create destination index <== This new action
- reindex from source to dest
- undo blocks on dest
- delete source (not currently being done)
So kibana gets to take advantage of creating the destination index, including overriding mappings and copying over source settings. But they'll continue to be responsible for the rest themselves (for now anyway).
parkertimmins
added
auto-backport
elasticsearchmachine
added
the
needs:triage
parkertimmins
added
>enhancement
:Data Management/Data streams
elasticsearchmachine
added
Team:Data Management
|
Hi @parkertimmins, I've created a changelog YAML for you. |
|
Pinging @elastic/es-data-management (Team:Data Management) |
|
From Kibana's side this action would replace the following code with one API call: Brief description of this code: we request source index mappings and settings ( fwiw If destination index exists already (for whatever reason) we just continue to call reindex from source -> dest (no deletion attempt). Aside: we planned on introducing this improvement. The idea is that we'll create the destination index with With ES's new create from source action available via rest it would be a nice-to-have if we could provide these "overrides" to the create action directly, or perhaps if the API just did this by default? WDYT? |
|
@jloleysens Looking through what Kibana does, it appears I missed how it filters out some of the deprecated settings. Thanks for linking this. Currently, this action will fail if the index already exists. Since we delete the destination index if it exists in the reindex-data-stream code, this seemed like the correct thing to do. (There has been some debate on this point, and I'm uncertain which way is better). I am inclined to not put any defaults in the override as I think this api could be useful more generally and don't want unexpected defaults for other users. Though I could see adding a boolean parameter (something like What do you think? |
💚 Backport successful
|
Add new action that creates an index from a source index, copying settings and mappings from the source index. This was refactored out of ReindexDataStreamIndexAction.
|
@jloleysens After some discussion, I think we should actually not provide the |
* Add action to create index from a source index (#118890) Add new action that creates an index from a source index, copying settings and mappings from the source index. This was refactored out of ReindexDataStreamIndexAction. * block-writes cannot be added after read-only Fix bug in ReindexDataStreamIndexAction. If the source index has both a block-writes and is read-only, these must be updated on the destination index. If read-only is set first, the block-writes cannot be added because settings cannot be modified.
Add CreateIndexFromSourceAction which creates a new index using the settings and mappings from a source index.
This already existed within ReindexDataStreamIndexAction, but was factored into it's own action, so can be used for migrating non-data stream indices.
The request takes source and destination index names, along with settings and mappings overrides. The destination index is then created with the settings and mappings from the source. Private and non-copyable settings are not added to the source index. If the request contains setting or mapping overrides, these are merged with the source settings and mappings before index creation.