-
Notifications
You must be signed in to change notification settings - Fork 477
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
505c3ef
commit 909a428
Showing
14 changed files
with
674 additions
and
125 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Xinference is a powerful tool that supports function calling and is fully compatible with OpenAI's Tool Call API. It allows you to seamlessly integrate and utilize the functionality of OpenAI's tools within your own projects." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Preparation\n", | ||
| "\n", | ||
| "First, you need to install Xinference:\n", | ||
| "```shell\n", | ||
| "pip install xinference\n", | ||
| "```\n", | ||
| "\n", | ||
| "Currently, only ChatGLM3 model is available for tool calls. Here, we install a chatglm cpp for fast inference on Mac. If you are not using a Mac, please follow the install instruction: https://github.com/li-plus/chatglm.cpp#python-binding\n", | ||
| "\n", | ||
| "```shell\n", | ||
| "CMAKE_ARGS=\"-DGGML_METAL=ON\" pip install -U chatglm-cpp --no-cache-dir\n", | ||
| "```\n", | ||
| "\n", | ||
| "Then, start the Xinference server by the following command:\n", | ||
| "```shell\n", | ||
| "xinference\n", | ||
| "```\n", | ||
| "\n", | ||
| "The Xinference server will be started:\n", | ||
| "\n", | ||
| "```shell\n", | ||
| "2023-11-02 16:04:55,278 xinference 38878 INFO Xinference successfully started. Endpoint: http://127.0.0.1:9997\n", | ||
| "2023-11-02 16:04:55,280 xinference.core.supervisor 38878 INFO Worker 127.0.0.1:32187 has been added successfully\n", | ||
| "2023-11-02 16:04:55,281 xinference.deploy.worker 38878 INFO Xinference worker successfully started.\n", | ||
| "```\n", | ||
| "\n", | ||
| "Finally, we launch a ChatGLM3 model for tool calls.\n", | ||
| "```shell\n", | ||
| "xinference launch -u my_tool_model -n chatglm3 -f ggmlv3 -q q4_0\n", | ||
| "```\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Function calling" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "This is an exciting game that showcases the function calling feature. Our objective is to call the AI and obtain a set of lottery numbers to determine if we are lucky enough to win the lottery. To get started, we first need to define a simple function for generating lottery numbers." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import random\n", | ||
| "\n", | ||
| "list_red = list(range(1, 34))\n", | ||
| "list_blue = list(range(1, 17))\n", | ||
| "\n", | ||
| "\n", | ||
| "def get_lucky_lottery(num):\n", | ||
| " total = []\n", | ||
| " for _ in range(num):\n", | ||
| " res_red = random.sample(list_red, 6)\n", | ||
| " res_blue = random.sample(list_blue, 1)\n", | ||
| " total.append(res_red + res_blue)\n", | ||
| " return total" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Then, we describe two functions as a dict, more details refers to https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "tools = [\n", | ||
| " {\n", | ||
| " \"type\": \"function\",\n", | ||
| " \"function\": {\n", | ||
| " \"name\": \"get_lucky_lottery\",\n", | ||
| " \"description\": \"生成双色球彩票号码\",\n", | ||
| " \"parameters\": {\n", | ||
| " \"type\": \"int\",\n", | ||
| " \"properties\": {\"num\": {\"description\": \"生成的彩票号码组数\"}},\n", | ||
| " \"required\": [\"num\"],\n", | ||
| " },\n", | ||
| " },\n", | ||
| " },\n", | ||
| " {\n", | ||
| " \"type\": \"function\",\n", | ||
| " \"function\": {\n", | ||
| " \"name\": \"lottery_draw\",\n", | ||
| " \"description\": \"开奖双色球\",\n", | ||
| " \"parameters\": {\n", | ||
| " \"type\": \"object\",\n", | ||
| " \"properties\": {},\n", | ||
| " },\n", | ||
| " },\n", | ||
| " },\n", | ||
| "]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Define a function to call above two functions according to the response." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import functools\n", | ||
| "import json\n", | ||
| "\n", | ||
| "FUNCTION_TABLE = {\n", | ||
| " \"get_lucky_lottery\": get_lucky_lottery,\n", | ||
| " \"lottery_draw\": functools.partial(get_lucky_lottery, 1),\n", | ||
| "}\n", | ||
| "\n", | ||
| "\n", | ||
| "def handle_response(completion):\n", | ||
| " assert \"tool_calls\" == completion.choices[0].finish_reason\n", | ||
| " func_name = completion.choices[0].message.tool_calls[0].function.name\n", | ||
| " func_args = completion.choices[0].message.tool_calls[0].function.arguments\n", | ||
| " func_kwargs = json.loads(func_args)\n", | ||
| " return FUNCTION_TABLE.get(func_name)(**func_kwargs)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Finally, let's ask the LLM to generate 5 group lottery numbers and check if we win the game." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import openai\n", | ||
| "\n", | ||
| "client = openai.Client(api_key=\"not empty\", base_url=f\"http://127.0.0.1:9997/v1\")\n", | ||
| "completion = client.chat.completions.create(\n", | ||
| " model=\"my_tool_model\",\n", | ||
| " messages=[{\"role\": \"user\", \"content\": \"帮我生成5组双色球号码\"}],\n", | ||
| " tools=tools,\n", | ||
| ")\n", | ||
| "print(f\"Lottery numbers: {handle_response(completion)}\")\n", | ||
| "completion = client.chat.completions.create(\n", | ||
| " model=\"my_tool_model\", messages=[{\"role\": \"user\", \"content\": \"开奖\"}], tools=tools\n", | ||
| ")\n", | ||
| "print(f\"Lottery draw: {handle_response(completion)}\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "No lucky, I lose the game.\n", | ||
| "\n", | ||
| "```\n", | ||
| "Lottery numbers: [[30, 9, 22, 26, 17, 27, 14], [8, 6, 27, 30, 21, 20, 13], [4, 33, 9, 32, 27, 22, 14], [19, 1, 30, 4, 28, 13, 7], [16, 7, 23, 17, 8, 30, 12]]\n", | ||
| "Lottery draw: [[29, 10, 27, 30, 15, 6, 1]]\n", | ||
| "```" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "language_info": { | ||
| "name": "python" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.