数据增强之Retriever.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "34e0f9f9-e805-4744-a38f-828904ecb7c5",
   "metadata": {},
   "source": [
    "# 介绍完整的LTESR数据增强业务流\n",
    "## L(Loader)\n",
    "## T(Transformer)\n",
    "## E(Embedding)\n",
    "## S(Strore in vector)\n",
    "## R(Retrieve)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "cb5e5fe4-b4a5-44ae-b6d1-2943568ba53a",
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (143452483.py, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  Cell \u001b[1;32mIn[28], line 2\u001b[1;36m\u001b[0m\n\u001b[1;33m    pip install -q pydantic==1.10\u001b[0m\n\u001b[1;37m        ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "pip install -q docarray tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1e68535b-e69b-4ac7-a101-1f37f6acab1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    " #这个向量数据库有bug\n",
    "# from langchain_community.vectorstores import DocArrayInMemorySearch\n",
    "\n",
    "#换一个新的数据库， pip install chromadb\n",
    "from langchain.vectorstores import Chroma \n",
    "\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.runnables import RunnableParallel, RunnablePassthrough\n",
    "\n",
    "#这里不写chat_models ,直接从langchain_openai里调ChatOpenAI也OK\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_openai.embeddings import OpenAIEmbeddings\n",
    "api_key = os.getenv(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "72f5df40-5988-4728-8d97-6d46b11401a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorstore = DocArrayInMemorySearch.from_texts(\n",
    "#     [\"harrison worked at kensho\", \"bears like to eat honey\"],\n",
    "#     embedding=OpenAIEmbeddings(),\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "51fe1b6e-5778-4d63-974d-a6eb73600eaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorstore = Chroma.from_texts(\n",
    "    [\"harrison worked at kensho\", \"bears like to eat honey\",\"jerry likes jenny\",\"today is a good day\"],\n",
    "    OpenAIEmbeddings(base_url = \"https://apejhvxcd.cloud.sealos.io/v1\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c703e61-ea3f-4d29-ac2a-408cc594cd60",
   "metadata": {},
   "source": [
    "## as_retriever() 是一个重要的函数，向量数据库通过这个函数创建一个检索器，以供调用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9186b663-890a-41c3-bc33-52e9ac4c2531",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = vectorstore.as_retriever()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fc2d7b78-ee49-478e-ae6a-0cad418d26d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"Answer the question based only on the following context:\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "26361e2d-fe24-4423-ba63-5ffb735f63a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "model = ChatOpenAI(\n",
    "    openai_api_key=api_key,\n",
    "    temperature = 0.5,\n",
    "    base_url = \"https://apejhvxcd.cloud.sealos.io/v1\"\n",
    ")\n",
    "\n",
    "output_parser = StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff04519-d9e3-400b-9cde-a693ec36d4ff",
   "metadata": {},
   "source": [
    "### retriever以向量数据库的形式检索相关文档，RunnablePassthrough()将传入的数据原样传给下一个组件;\n",
    "### 字典包含两个键值对，每个键值对都表示一个可运行的组件。\n",
    "### 在LCEL语法中，字典格式可以用作RunnableParallel的输入，用于同时执行多个可运行组件，并将它们的输出作为一个字典返回。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "985387b3-b2cd-4008-b44c-44b2c62e711d",
   "metadata": {},
   "outputs": [],
   "source": [
    "setup_and_retrieval = RunnableParallel(\n",
    "    {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e9485860-a4da-4a50-93a3-4f2c44369b31",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = setup_and_retrieval | prompt | model | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5e44676e-4bb8-4b36-abcb-bff8b0e5dcd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Harrison worked at Kensho.\n"
     ]
    }
   ],
   "source": [
    "response = chain.invoke(\"where did harrison work?\")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f789c77-69de-4a5e-8899-b6b1574eda89",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}