feat: Add tfidf

.env.example

@@ -1 +1 @@
-COHERE_API_KEY=
+COHERE_API_KEY=

LICENSE

@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

docs/examples/hybrid-layer.ipynb

@@ -4,53 +4,53 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Semantic Router: Hybrid Layer"
+    "# Semantic Router: Hybrid Layer\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The Hybrid Layer in the Semantic Router library can improve  making performance particularly for niche use-cases that contain specific terminology, such as finance or medical. It helps us provide more importance to  making based on the keywords contained in our utterances and user queries."
+    "The Hybrid Layer in the Semantic Router library can improve making performance particularly for niche use-cases that contain specific terminology, such as finance or medical. It helps us provide more importance to making based on the keywords contained in our utterances and user queries.\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Getting Started"
+    "## Getting Started\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We start by installing the library:"
+    "We start by installing the library:\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
-    "!pip install -qU semantic-router==0.0.11"
+    "#!pip install -qU semantic-router==0.0.11"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We start by defining a dictionary mapping s to example phrases that should trigger those s."
+    "We start by defining a dictionary mapping s to example phrases that should trigger those s.\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from semantic_router.schema import Route\n",
+    "from semantic_router.route import Route\n",
     "\n",
     "politics = Route(\n",
     "    name=\"politics\",\n",
@@ -69,7 +69,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let's define another for good measure:"
+    "Let's define another for good measure:\n"
    ]
   },
   {
@@ -81,7 +81,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -114,58 +114,92 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now we initialize our embedding model:"
+    "Now we initialize our embedding model:\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [],
    "source": [
     "import os\n",
-    "from semantic_router.encoders import CohereEncoder\n",
+    "from semantic_router.encoders import CohereEncoder, BM25Encoder, TfidfEncoder\n",
     "from getpass import getpass\n",
     "\n",
     "os.environ[\"COHERE_API_KEY\"] = os.environ[\"COHERE_API_KEY\"] or getpass(\n",
     "    \"Enter Cohere API Key: \"\n",
     ")\n",
     "\n",
-    "encoder = CohereEncoder()"
+    "dense_encoder = CohereEncoder()\n",
+    "# sparse_encoder = BM25Encoder()\n",
+    "sparse_encoder = TfidfEncoder()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`."
+    "Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`.\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\u001b[32m2024-01-08 16:50:29 INFO semantic_router.utils.logger Creating embeddings for all routes...\u001b[0m\n"
+     ]
+    }
+   ],
    "source": [
     "from semantic_router.hybrid_layer import HybridRouteLayer\n",
     "\n",
-    "dl = HybridRouteLayer(encoder=encoder, routes=routes)"
+    "dl = HybridRouteLayer(\n",
+    "    dense_encoder=dense_encoder, sparse_encoder=sparse_encoder, routes=routes\n",
+    ")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'politics'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "dl(\"don't you love politics?\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'chitchat'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "dl(\"how's the weather today?\")"
    ]
@@ -174,7 +208,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "---"
+    "---\n"
    ]
   }
  ],