Please ask whether the document data supports increment？

[xiaolibuzai-ovo]

If my document continues to grow, is there a way to increment the vector data instead of reloading it each time?🤔

[SeeknnDestroy]

Hi @xiaolibuzai-ovo,

Could you please clarify if you're asking about incrementally updating the vector data in the AutoQueryEngine as new documents are added, without the need to reprocess the entire dataset? This would help in providing a more accurate response.

Best,
Talha

[xiaolibuzai-ovo]

Yes, I want to update incrementally instead of processing all the data at once

[SeeknnDestroy]

Absolutely, autollm supports incremental updates of your document store using LanceDB. When you process your first set of documents, they are stored in the LanceDB vector store, which by default creates a .lancedb directory in your current working directory.

Here's how it works:

1. First Set of Documents:

   from autollm import AutoQueryEngine, read_files_as_documents
   
   # Processing and storing the first set of documents
   documents1 = read_files_as_documents(input_dir="your/first/document/path")
   query_engine = AutoQueryEngine.from_defaults(documents=documents1)
   # At this point, documents1 are stored in './.lancedb'

2. Adding More Documents:

   # Adding a second set of documents
   documents2 = read_files_as_documents(input_dir="your/second/document/path")
   query_engine = AutoQueryEngine.from_defaults(documents=documents2)
   # Now, documents2 are added to the existing store at './.lancedb'

By using the same lancedb_uri and lancedb_table_name (which are set to "./.lancedb" and "vectors" by default), the new documents are incrementally added to the existing store. This allows you to query across both the original and the newly added documents without reprocessing everything."

[xiaolibuzai-ovo]

Thanks for your help. Is there a complete documentation of autollm to see some of its features in detail

[SeeknnDestroy]

Not yet, but its in our roadmap for sure 🚀 We tried to cover most of the things in our 🌟features, click on the 👉 more buttons to see detailed usage. And please don't hesitate to ask if you need further help.

[SeeknnDestroy]

And at the end, you con just use AutoQueryEngine.from_defaults() and not give any documents to conenct the existing vector store to ask questions as you wish :)

[xiaolibuzai-ovo]

If I need to append vector data, can I just execute this code?

from autollm import read_files_as_documents

read_files_as_documents(input_dir="xxx/docs")

Whether

AutoQueryEngine.from_defaults(docs) 

should be executed, In this append operation

[SeeknnDestroy]

Hi @xiaolibuzai-ovo,

For appending vector data in autollm, executing read_files_as_documents(input_dir="xxx/docs") is a correct step to read documents from your specified directory and convert them into a format suitable for processing.

However, to integrate these newly read documents into your vector database, you indeed need to execute AutoQueryEngine.from_defaults(docs) after reading the files. This step is necessary to ensure that your AutoQueryEngine is updated with the new vector data derived from the documents you've just read.

So, the process would be:

1- Read the files as documents, as its converting the data into suitable format to add our vector database.
2- Update the AutoQueryEngine with these documents.
Here's the sequence:

from autollm import read_files_as_documents, AutoQueryEngine

# Read documents from the specified directory
docs = read_files_as_documents(input_dir="xxx/docs")

# Update the AutoQueryEngine with the new documents
auto_query_engine = AutoQueryEngine.from_defaults(docs)

This will append the new vector data from the documents in "xxx/docs" to your existing vector database instance.

[xiaolibuzai-ovo]

okey, I got it, Thank you