Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Lucene HNSW merge optimization #785

Closed
jmazanec15 opened this issue Mar 1, 2023 · 1 comment
Closed

Support Lucene HNSW merge optimization #785

jmazanec15 opened this issue Mar 1, 2023 · 1 comment
Labels
Enhancements Increases software capabilities beyond original client specifications k-NN v2.8.0

Comments

@jmazanec15
Copy link
Member

Description

In Lucene, in apache/lucene#12050, an optimization was added to create a new segment HNSW graph by initializing the graph from the largest graph without any deletes from the collection of segments-to-be-merged. Also, see apache/lucene#11354 for related benchmarks. This will speed up merge operations for larger graphs.

This change will go out in Lucene 96, which we will pick up by default - we wont have to make any code changes. Once we are able to test this change in OpenSearch, we should run performance tests to measure benefit.

@jmazanec15 jmazanec15 added Enhancements Increases software capabilities beyond original client specifications v2.8.0 labels Mar 1, 2023
@vamshin vamshin added the k-NN label Mar 20, 2023
@martin-gaievski
Copy link
Member

With 2.8 k-NN received Lucene 9.6 with the mentioned fix, performance test is completed so we can check on changed numbers. Perf results added to release issue: #905

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancements Increases software capabilities beyond original client specifications k-NN v2.8.0
Projects
None yet
Development

No branches or pull requests

3 participants