Integrate VoyageAI Vectorizer and Reranker class

.github/workflows/run_tests.yml

@@ -61,6 +61,7 @@ jobs:
         GCP_LOCATION: ${{ secrets.GCP_LOCATION }}
         GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
         COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
+        VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
         MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
         AZURE_OPENAI_API_KEY: ${{secrets.AZURE_OPENAI_API_KEY}}
         AZURE_OPENAI_ENDPOINT: ${{secrets.AZURE_OPENAI_ENDPOINT}}
@@ -81,6 +82,7 @@ jobs:
         GCP_LOCATION: ${{ secrets.GCP_LOCATION }}
         GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
         COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
+        VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
         MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
         AZURE_OPENAI_API_KEY: ${{secrets.AZURE_OPENAI_API_KEY}}
         AZURE_OPENAI_ENDPOINT: ${{secrets.AZURE_OPENAI_ENDPOINT}}

README.md

@@ -194,6 +194,7 @@ Integrate with popular embedding providers to greatly simplify the process of ve
 - [Cohere](https://www.redisvl.com/api/vectorizer.html#coheretextvectorizer)
 - [Custom](https://www.redisvl.com/api/vectorizer.html#customtextvectorizer)
 - [GCP VertexAI](https://www.redisvl.com/api/vectorizer.html#vertexaitextvectorizer)
+- [VoyageAI](https://www.redisvl.com/api/vectorizer/html#voyageaitextvectorizer)
 - [HuggingFace](https://www.redisvl.com/api/vectorizer.html#hftextvectorizer)
 - [Mistral](https://www.redisvl.com/api/vectorizer/html#mistralaitextvectorizer)
 - [OpenAI](https://www.redisvl.com/api/vectorizer.html#openaitextvectorizer)

docs/api/reranker.rst

@@ -12,3 +12,15 @@ CohereReranker
 .. autoclass:: CohereReranker
    :show-inheritance:
    :members:
+
+
+VoyageAIReranker
+================
+
+.. _voyageaireranker_api:
+
+.. currentmodule:: redisvl.utils.rerank.voyageai
+
+.. autoclass:: VoyageAIReranker
+   :show-inheritance:
+   :members:

docs/api/vectorizer.rst

@@ -62,6 +62,18 @@ CohereTextVectorizer
    :members:
 
 
+VoyageAITextVectorizer
+====================
+
+.. _voyageaitextvectorizer_api:
+
+.. currentmodule:: redisvl.utils.vectorize.text.voyageai
+
+.. autoclass:: VoyageAITextVectorizer
+   :show-inheritance:
+   :members:
+
+
 CustomTextVectorizer
 ====================
 
@@ -71,4 +83,4 @@ CustomTextVectorizer
 
 .. autoclass:: CustomTextVectorizer
    :show-inheritance:
-   :members:
+   :members:

docs/user_guide/getting_started_01.ipynb

@@ -413,7 +413,7 @@
    "source": [
     "## Creating `VectorQuery` Objects\n",
     "\n",
-    "Next we will create a vector query object for our newly populated index. This example will use a simple vector to demonstrate how vector similarity works. Vectors in production will likely be much larger than 3 floats and often require Machine Learning models (i.e. Huggingface sentence transformers) or an embeddings API (Cohere, OpenAI). `redisvl` provides a set of [Vectorizers](https://www.redisvl.com/user_guide/vectorizers_04.html#openai) to assist in vector creation."
+    "Next we will create a vector query object for our newly populated index. This example will use a simple vector to demonstrate how vector similarity works. Vectors in production will likely be much larger than 3 floats and often require Machine Learning models (i.e. Huggingface sentence transformers) or an embeddings API (Cohere, OpenAI, VoyageAI). `redisvl` provides a set of [Vectorizers](https://www.redisvl.com/user_guide/vectorizers_04.html#openai) to assist in vector creation."
    ]
   },
   {

docs/user_guide/rerankers_06.ipynb

@@ -13,6 +13,7 @@
     "\n",
     "- A re-ranker that uses pre-trained [Cross-Encoders](https://sbert.net/examples/applications/cross-encoder/README.html) which can use models from [Hugging Face cross encoder models](https://huggingface.co/cross-encoder) or Hugging Face models that implement a cross encoder function ([example: BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base)).\n",
     "- The [Cohere /rerank API](https://docs.cohere.com/docs/rerank-2).\n",
+    "- The [VoyageAI /rerank API](https://docs.voyageai.com/docs/reranker).\n",
     "\n",
     "Before running this notebook, be sure to:\n",
     "1. Have installed ``redisvl`` and have that environment active for this notebook.\n",
@@ -196,7 +197,7 @@
    "source": [
     "### Rerank documents with CohereReranker\n",
     "\n",
-    "Below we will use the `CohereReranker` to rerank and truncate the list of\n",
+    "Below we will use the `CohereReranker` to rerank and also truncate the list of\n",
     "documents above based on relevance to the initial query."
    ]
   },
@@ -306,7 +307,100 @@
     "for result, score in zip(results, scores):\n",
     "    print(score, \" -- \", result)"
    ]
-  }
+  },
+
+  {
+    "cell_type": "markdown",
+    "metadata": {},
+    "source": [
+     "### Using the VoyageAI Reranker\n",
+     "\n",
+     "To initialize the VoyageAI reranker you'll need to install the voyaeai library and provide the right VoyageAI API Key."
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 32,
+    "metadata": {
+     "metadata": {}
+    },
+    "outputs": [],
+    "source": [
+     "#!pip install voyageai"
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 33,
+    "metadata": {
+     "metadata": {}
+    },
+    "outputs": [],
+    "source": [
+     "import getpass\n",
+     "\n",
+     "# setup the API Key\n",
+     "api_key = os.environ.get(\"VOYAGE_API_KEY\") or getpass.getpass(\"Enter your VoyageAI API key: \")"
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 34,
+    "metadata": {
+     "metadata": {}
+    },
+    "outputs": [],
+    "source": [
+     "from redisvl.utils.rerank import VoyageAIReranker\n",
+     "\n",
+     "reranker = VoyageAIReranker(model=\"rerank-lite-1\", limit=3, api_config={\"api_key\": api_key})",
+     "# Please check the available models at https://docs.voyageai.com/docs/reranker"
+    ]
+   },
+    {
+    "cell_type": "markdown",
+    "metadata": {},
+    "source": [
+     "### Rerank documents with VoyageAIReranker\n",
+     "\n",
+     "Below we will use the `VoyageAIReranker` to rerank and also truncate the list of\n",
+     "documents above based on relevance to the initial query."
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 35,
+    "metadata": {
+     "metadata": {}
+    },
+    "outputs": [],
+    "source": [
+     "results, scores = reranker.rank(query=query, docs=docs)"
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 36,
+    "metadata": {
+     "metadata": {}
+    },
+    "outputs": [
+     {
+      "name": "stdout",
+      "output_type": "stream",
+      "text": [
+       "0.796875  --  Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.\n",
+       "0.578125  --  Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas.\n",
+       "0.5625  --  Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274.\n"
+      ]
+     }
+    ],
+    "source": [
+     "for result, score in zip(results, scores):\n",
+     "    print(score, \" -- \", result)"
+    ]
+   }
+
  ],
  "metadata": {
   "kernelspec": {

docs/user_guide/vectorizers_04.ipynb

@@ -13,7 +13,8 @@
     "3. Vertex AI\n",
     "4. Cohere\n",
     "5. Mistral AI\n",
-    "6. Bringing your own vectorizer\n",
+    "6. VoyageAI\n",
+    "7. Bringing your own vectorizer\n",
     "\n",
     "Before running this notebook, be sure to\n",
     "1. Have installed ``redisvl`` and have that environment active for this notebook.\n",
@@ -503,6 +504,76 @@
    ]
   },
   {
+    "cell_type": "markdown",
+    "metadata": {},
+    "source": [
+     "### VoyageAI\n",
+     "\n",
+     "[VoyageAI](https://dash.voyageai.com/) allows you to implement language AI into your product. The `VoyageAITextVectorizer` makes it simple to use RedisVL with the embeddings models at VoyageAI. For this you will need to install `voyageai`.\n",
+     "\n",
+     "```bash\n",
+     "pip install voyageai\n",
+     "```"
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 11,
+    "metadata": {},
+    "outputs": [],
+    "source": [
+     "import getpass\n",
+     "# setup the API Key\n",
+     "api_key = os.environ.get(\"VOYAGE_API_KEY\") or getpass.getpass(\"Enter your VoyageAI API key: \")"
+    ]
+   },
+   {
+    "cell_type": "markdown",
+    "metadata": {},
+    "source": [
+     "\n",
+     "Special attention needs to be paid to the `input_type` parameter for each `embed` call. For example, for embedding \n",
+     "queries, you should set `input_type='query'`; for embedding documents, set `input_type='document'`. See\n",
+     "more information [here](https://docs.voyageai.com/docs/embeddings)"
+    ]
+   },
+   {
+    "cell_type": "code",
+    "execution_count": 12,
+    "metadata": {},
+    "outputs": [
+     {
+      "name": "stdout",
+      "output_type": "stream",
+      "text": [
+       "Vector dimensions:  1024\n",
+       "[0.015814896672964096, 0.046988241374492645, -0.00518248463049531, -0.05383478105068207, -0.015586535446345806, -0.0837097093462944, 0.03744547441601753, -0.007797810714691877, 0.00717928446829319, 0.06857716292142868]\n",
+       "Vector dimensions:  1024\n",
+       "[0.006725038401782513, 0.01441393606364727, -0.030212024226784706, -0.06782275438308716, -0.021446991711854935, -0.07667966187000275, 0.01804908737540245, -0.015767497941851616, -0.02152789570391178, 0.049741245806217194]\n"
+      ]
+     }
+    ],
+    "source": [
+     "from redisvl.utils.vectorize import VoyageAITextVectorizer\n",
+     "\n",
+     "# create a vectorizer\n",
+     "vo = VoyageAITextVectorizer(\n",
+     "    model=\"voyage-law-2\",  # Please check the available models at https://docs.voyageai.com/docs/embeddings\n",
+     "    api_config={\"api_key\": api_key},\n",
+     ")\n",
+     "\n",
+     "# embed a search query\n",
+     "test = vo.embed(\"This is a test sentence.\", input_type='query')\n",
+     "print(\"Vector dimensions: \", len(test))\n",
+     "print(test[:10])\n",
+     "\n",
+     "# embed a document\n",
+     "test = vo.embed(\"This is a test sentence.\", input_type='document')\n",
+     "print(\"Vector dimensions: \", len(test))\n",
+     "print(test[:10])"
+    ]
+   },
+   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [