cookbooks/end2end_application/rag/qa_system_1_dataset.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 企业级问答系统 —— 离线知识生产流程\n",
    "\n",
    "## 一、背景介绍\n",
    "\n",
    "智能问答系统是当前应用范围最广，最容易落地的大模型应用。通常来说，智能问答系统包括以下五部分：\n",
    "1. Query理解\n",
    "2. 混合检索召回\n",
    "3. 语义排序\n",
    "4. 答案生成\n",
    "5. 追问生成\n",
    "\n",
    "AppBuilder在问答系统沉淀了最佳实践流程，并以极易用的组件化的方式对外开放，可以使用页面操作 + 低代码的方式快速搭建自己的问答系统。\n",
    "\n",
    "<img src=\"./qa_system/qa_flow.png\" alt=\"drawing\" width=\"1000\"/>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、实操流程\n",
    "\n",
    "整体使用流程包括以下三个环节：\n",
    "\n",
    "1. 登录[百度智能云千帆AppBuilder官网](https://cloud.baidu.com/product/AppBuilder)创建账户，\n",
    "\n",
    "<img src=\"./qa_system/product.png\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "2. 在[百度智能云千帆AppBuilder控制台](https://console.bce.baidu.com/ai_apaas/dialogHome)右侧头像栏『API密钥』页面获取密钥，并复制。\n",
    "\n",
    "<img src=\"./qa_system/token.jpg\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "<img src=\"./qa_system/get_token.png\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "3. 引用AppBuilder SDK代码，调用Dataset API相关接口，创建数据集，并增、查、删除文档。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 环境准备\n",
    "\n",
    "首先需要安装AppBuilder-SDK代码库，若已在开发环境安装，则可跳过此步。\n",
    "\n",
    "**注意：**: appbuilder-sdk 的python版本要求 3.9+"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Collecting appbuilder-sdk\n",
      "  Using cached appbuilder_sdk-0.6.0-py3-none-any.whl.metadata (768 bytes)\n",
      "Requirement already satisfied: requests in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (2.31.0)\n",
      "Requirement already satisfied: proto-plus==1.22.3 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (1.22.3)\n",
      "Requirement already satisfied: pydantic==2.6.4 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (2.6.4)\n",
      "Requirement already satisfied: numpy in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (1.24.4)\n",
      "Requirement already satisfied: SQLAlchemy~=1.4.50 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (1.4.52)\n",
      "Requirement already satisfied: urllib3<2.0.0 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (1.26.18)\n",
      "Requirement already satisfied: tenacity in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (8.2.3)\n",
      "Requirement already satisfied: pandas in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (2.2.1)\n",
      "Requirement already satisfied: openpyxl in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (3.1.2)\n",
      "Requirement already satisfied: pymochow>=1.1.2 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (1.1.2)\n",
      "Requirement already satisfied: parameterized in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from appbuilder-sdk) (0.9.0)\n",
      "Requirement already satisfied: protobuf<5.0.0dev,>=3.19.0 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from proto-plus==1.22.3->appbuilder-sdk) (4.25.3)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pydantic==2.6.4->appbuilder-sdk) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pydantic==2.6.4->appbuilder-sdk) (2.16.3)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pydantic==2.6.4->appbuilder-sdk) (4.10.0)\n",
      "Requirement already satisfied: orjson in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pymochow>=1.1.2->appbuilder-sdk) (3.9.15)\n",
      "Requirement already satisfied: future in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from pymochow>=1.1.2->appbuilder-sdk) (0.18.2)\n",
      "Requirement already satisfied: et-xmlfile in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from openpyxl->appbuilder-sdk) (1.1.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pandas->appbuilder-sdk) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pandas->appbuilder-sdk) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from pandas->appbuilder-sdk) (2024.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from requests->appbuilder-sdk) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from requests->appbuilder-sdk) (3.6)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from requests->appbuilder-sdk) (2024.2.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/chengmo/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas->appbuilder-sdk) (1.16.0)\n",
      "Using cached appbuilder_sdk-0.6.0-py3-none-any.whl (335 kB)\n",
      "Installing collected packages: appbuilder-sdk\n",
      "Successfully installed appbuilder-sdk-0.6.0\n"
     ]
    }
   ],
   "source": [
    "!python3 -m pip install appbuilder-sdk"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 引入AppBuilder-SDK，并设置TOKEN\n",
    "\n",
    "**注意：** 请使用刚才在AppBuilder平台上新建的个人TOKEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AppBuilder 模块导入成功！\n",
      "您的AppBuilder Token为：your_appbuilder_token\n"
     ]
    }
   ],
   "source": [
    "# 引入os模块，引入appbuilder 模块\n",
    "import os\n",
    "import appbuilder\n",
    "\n",
    "# 设置appbuilder的token密钥，从页面上复制粘贴token，覆盖此处的 \"your_appbuilder_token\"\n",
    "os.environ['APPBUILDER_TOKEN'] = \"your_appbuilder_token\"\n",
    "\n",
    "print(\"AppBuilder 模块导入成功！\")\n",
    "print(\"您的AppBuilder Token为：{}\".format(os.environ['APPBUILDER_TOKEN']))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 创建全新知识库\n",
    "\n",
    "我们从零开始创建一个新的知识库，并使用它来支持问答系统。我们将其命名为**说明书知识库**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "知识库创建成功！ 知识库的ID为：c368a982-699f-4114-9e9a-c08b76c07f92， 知识库的名称为： 说明书知识库\n"
     ]
    }
   ],
   "source": [
    "# 创建全新知识库\n",
    "dataset = appbuilder.console.Dataset.create_dataset(\"说明书知识库\")\n",
    "\n",
    "print(\"知识库创建成功！ 知识库的ID为：{}， 知识库的名称为： {}\".format(\n",
    "    dataset.dataset_id,\n",
    "    dataset.dataset_name))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 上传文档到知识库\n",
    "\n",
    "在notebook的同级目录`./qa_system`下，我们提前准备了两篇文档：\n",
    "- `./qa_system/平板电脑说明书.pdf`\n",
    "- `./qa_system/显示器说明书.pdf`\n",
    "\n",
    "我们将这两篇文档上传到知识库中，用于后续问答应用的私域知识检索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文档上传到知识库成功！知识库中的文档ID为：['c396c715-66d8-49cb-af9d-af0bd08a16e2', '817d29bb-be72-420b-bee1-c1d9b1116826']\n"
     ]
    }
   ],
   "source": [
    "file_1 = \"./qa_system/平板电脑说明书.pdf\"\n",
    "file_2 = \"./qa_system/显示器说明书.pdf\"\n",
    "\n",
    "document_infos = dataset.add_documents([\n",
    "    file_1, file_2\n",
    "])\n",
    "\n",
    "print(\"文档上传到知识库成功！知识库中的文档ID为：{}\".format(document_infos.document_ids))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 查找知识库中的文档\n",
    "\n",
    "问答系统中的知识库，常用增、删、查操作，我们刚才演示了增加文档的方法，下面演示如何查找文档。\n",
    "\n",
    "我们获取[百度智能云千帆AppBuilder控制台-我的知识](https://console.bce.baidu.com/ai_apaas/dataset)页面中，我们刚才创建的知识库中的文档列表。\n",
    "\n",
    "<img src=\"./qa_system/doc_list.png\" alt=\"drawing\" width=\"1000\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文档列表获取成功！\n",
      "第二个的文档是: 平板电脑说明书.pdf, 文档ID是: c396c715-66d8-49cb-af9d-af0bd08a16e2, 文档字数是: 8568\n"
     ]
    }
   ],
   "source": [
    "doc_list = dataset.get_documents(page=1,limit=10)\n",
    "print(\"文档列表获取成功！\")\n",
    "print(\"第二个的文档是: {}, 文档ID是: {}, 文档字数是: {}\".format(\n",
    "    doc_list.data[1].name, doc_list.data[1].id, doc_list.data[1].word_count))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 删除知识库中的文档\n",
    "\n",
    "下面我们演示如何删除知识库中的文档。我们尝试删除 『平板电脑说明书.pdf』 这篇文档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文档: 平板电脑说明书.pdf, 文档ID: c396c715-66d8-49cb-af9d-af0bd08a16e2 删除成功!\n",
      "文档删除成功，请刷新『知识库』前端页面，观察是否已无『平板电脑说明书.pdf』文档\n"
     ]
    }
   ],
   "source": [
    "need_delete_doc_name = \"平板电脑说明书.pdf\"\n",
    "need_delete_doc_id = \"\"\n",
    "\n",
    "doc_list = dataset.get_documents(page=1,limit=10)\n",
    "for doc in doc_list.data:\n",
    "    if doc.name == need_delete_doc_name:\n",
    "        need_delete_doc_id = doc.id\n",
    "        break\n",
    "\n",
    "if need_delete_doc_id == \"\":\n",
    "    print(\"未找到指定文档，请检查 2.4 步骤是否正确上传，以及2.5步骤中是否展示了正确的文档列表\")\n",
    "else:\n",
    "    dataset.delete_documents([need_delete_doc_id])\n",
    "    print(\"文档: {}, 文档ID: {} 删除成功!\".format(\n",
    "        need_delete_doc_name,\n",
    "        need_delete_doc_id\n",
    "    ))\n",
    "\n",
    "print(\"文档删除成功，请刷新『知识库』前端页面，观察是否已无『{}』文档\".format(need_delete_doc_name))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行完上述步骤后，前端页面刷新，指定文档已经被成功删除\n",
    "\n",
    "<img src=\"./qa_system/delete_doc.png\" alt=\"drawing\" width=\"1000\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "当前创建的知识库的ID为：c368a982-699f-4114-9e9a-c08b76c07f92， 知识库的名称为： 说明书知识库\n"
     ]
    }
   ],
   "source": [
    "# 再一次确认当前知识库的 Name 与 ID，方便后续 企业级问答系统 —— 在线问答流程 的使用\n",
    "\n",
    "print(\"当前创建的知识库的ID为：{}， 知识库的名称为： {}\".format(\n",
    "    dataset.dataset_id,\n",
    "    dataset.dataset_name))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、总结\n",
    "\n",
    "AppBuilder 为 基于RAG的 企业问答系统提供了 `appbuilder.console.dataset` API，可以方便的以代码态进行知识库的管理，实现基本的增删改查业务需求。\n",
    "\n",
    "本示例展示了如何从零创建知识库，并实现增、查、改的代码，并可以观察SDK与前端页面的联动。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}