Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-indexing: directly support multi-root projects #598

Open
keynmol opened this issue Jun 15, 2023 · 0 comments
Open

Auto-indexing: directly support multi-root projects #598

keynmol opened this issue Jun 15, 2023 · 0 comments

Comments

@keynmol
Copy link
Contributor

keynmol commented Jun 15, 2023

To reduce the number of jobs being scheduled, I propose we move multi-root support into the tool itself, peraps as a separate command that delegates to regular index

For example, in this repo: https://github.com/in28minutes/spring-boot-examples we are currently inferencing 1 indexer job per folder.

That's quite wasteful, not only because of CPU cycles, but also because of lack of persistent dependency cache.

Instead, we can add index-batch [options] root1 root2 root3 command which can index each root independently and possible in parallel, and then merge the resulting SCIP indexes.

Merging should be done in simplest way possible - basically concatenating lists of documents, assuming that subfolders are independent of each other. This can be lossy, but good enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants