Add support for CrateDB to LangChain LLM framework

[amotl]

About

Discussing the patch to add support for CrateDB to LangChain, to be submitted upstream. Do not merge.

What's inside

• Support for CrateDB's FLOAT_VECTOR / KNN_MATCH functionality through LangChain's vector store subsystem.
• Support for loading documents from CrateDB through LangChain's document loader subsystem.
• Support for managing "chat" history using LangChain's conversational memory subsystem.

Documentation

• integrations/providers/cratedb.mdx

Notebooks

• integrations/vectorstores/cratedb.ipynb
• integrations/document_loaders/cratedb.ipynb
• integrations/document_loaders/sqlalchemy.ipynb
• integrations/memory/cratedb_chat_message_history.ipynb

Backlog

• See comments below.
• Break out SQLDatabaseLoader documentation into separate patch.
• Review all FIXME and TODO remarks.
• Bring documentation up to speed. Have a look at those blueprints:
  • For New Contributors: Update Integration Documentation langchain-ai/langchain#15664
  • https://python.langchain.com/docs/integrations/vectorstores/faiss
  • docs: Updated MongoDB Chat history example notebook to use LCEL format. langchain-ai/langchain#15750
  • docs: Updated SQLite example to use LCEL and SQLChatMessageHistory langchain-ai/langchain#16094
  • docs: Updated integration docs structure for chat/anthropic langchain-ai/langchain#16268

/cc @matriv, @seut, @marijaselakovic, @karynzv: You may also want to have a review on it? Thanks!

[kneth]

I believe that I am not able to provide a meaningful review without a deeper understanding of LangChain.

Said that, I believe it is important work and it is great to see that we are getting closer to complete it.

[kneth]

Can we create an issue to track the work?

[amotl]

I believe that I am not able to provide a meaningful review without a deeper understanding of LangChain.

At LangChain and CrateDB, we are maintaining a set of educational resources to learn a bit about their dynamic duo. That may spark your interest?

[amotl]

This document is the main entrypoint that reflects details about CrateDB on the LangChain documentation, in its major LangChain Providers section. Please review it correspondingly, and suggest relevant edits. Thanks!

[amotl]

Hi.

I believe it is important work and it is great to see that we are getting closer to complete it.

The fundamental implementation (vector store, document loader, chat history) as implemented within this genesis patch has been converged into a dedicated package langchain-cratedb. 🎄

• PyPI: https://pypi.org/project/langchain-cratedb/
• Repository: https://github.com/crate/langchain-cratedb/
• Documentation (local): https://cratedb.com/docs/guide/integrate/langchain/
• Documentation (remote): https://python.langchain.com/docs/integrations/providers/cratedb/
• Examples: https://github.com/crate/cratedb-examples/tree/main/topic/machine-learning/llm-langchain

We will follow up on this fundamental implementation and add new features and documentation as we go. This PR and feature branch will be closed/deleted in January 2025.

With kind regards,
Andreas.