-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LLAMA_CPP notebook with Qwen-1.5-7B-Chat #901
Merged
ilya-lavrenov
merged 4 commits into
openvinotoolkit:master
from
vshampor:example_notebook_qwen
May 7, 2024
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,292 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "b2cb321e-2c20-45ea-93e0-a3bd16e5a120", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## QWEN model inference w/ OpenVINO's LLAMA_CPP plugin" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "2bb9b46a-d9c5-42dc-8e50-6700180aad0c", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "This notebook illustrates the usage of the `LLAMA_CPP` plugin for OpenVINO, which enables the `llama.cpp`-powered inference of corresponding GGUF-format model files via OpenVINO API. The user flow will be demonstrated on an LLM inferencing task with the Qwen-7B-Chat model in Chinese. This notebook executes direct Linux shell commands, and therefore should be run in a Linux environment." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "77a75ee1-6e0b-4c75-9af0-24989658ea81", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The first step is to procure a GGUF file that you would like to execute. Below the required dependencies and repositories are checked out and the Qwen-7B-Chat model is exported from HuggingFace's `transformers` into a GGUF file:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "64317459-fe47-4835-b1ae-e239301cde17", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!pip install transformers[torch]\n", | ||
| "!pip install tiktoken\n", | ||
| "!huggingface-cli download Qwen/Qwen1.5-7B-Chat-GGUF qwen1_5-7b-chat-q5_k_m.gguf --local-dir . --local-dir-use-symlinks False" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "ed2e6a45-a4e1-4478-b251-a1bb52a845da", | ||
| "metadata": { | ||
| "scrolled": true | ||
| }, | ||
| "source": [ | ||
| "The `LLAMA_CPP` plugin must be built from [sources](https://github.com/openvinotoolkit/openvino_contrib/modules/llama_cpp_plugin) in a standard [OpenVINO extra module user flow](https://github.com/openvinotoolkit/openvino_contrib/#how-to-build-openvino-with-extra-modules):" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ff88d765-5257-4aa4-9373-4e1207197bf9", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!apt-get install -y cmake python3.8-dev build-essential\n", | ||
| "!git clone https://github.com/openvinotoolkit/openvino_contrib\n", | ||
| "!git clone --recurse-submodules https://github.com/openvinotoolkit/openvino\n", | ||
| "\n", | ||
| "!pip install --upgrade pip \n", | ||
| "!pip install --upgrade build setuptools wheel\n", | ||
| "!pip install -r openvino/src/bindings/python/wheel/requirements-dev.txt\n", | ||
| "\n", | ||
| "# Add -DLLAMA_CUBLAS=1 to the cmake line below build the plugin with the CUDA backend.\n", | ||
| "# The underlying llama.cpp inference code will be executed on CUDA-powered GPUs on your host.\n", | ||
| "!cmake -B build -DCMAKE_BUILD_TYPE=Release -DOPENVINO_EXTRA_MODULES=../openvino_contrib/modules/llama_cpp_plugin -DENABLE_PLUGINS_XML=ON -DENABLE_LLAMA_CPP_PLUGIN_REGISTRATION=ON -DENABLE_PYTHON=1 -DPYTHON_EXECUTABLE=`which python3.8` -DENABLE_WHEEL=ON openvino #-DLLAMA_CUBLAS=1\n", | ||
| "\n", | ||
| "!cmake --build build --parallel `nproc` -- llama_cpp_plugin pyopenvino ie_wheel" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "458258b7-2e00-44b3-b3ee-e73238a47019", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "After the build, the plugin binaries should be installed into the same directory as the rest of the OpenVINO plugin binaries, and the plugin itself should be registered in the `plugins.xml` file in the same directory. In our case, we configured the build above to generate a Python `.whl`, which will place the necessary binaries and files to their required locations automatically after an installation into a virtual environment:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ea92d639-4f1e-414b-af95-7b82b98f117a", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# You may want to restart your kernel after this cell.\n", | ||
| "!pip install --force-reinstall ./build/wheels/openvino-2024.1.0-cp39-cp39-linux_x86_64.whl" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "f8f9a117-6302-4e7b-8570-a6fecffcf104", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now the actual inferencing pipeline in Python can be executed. Load the GGUF file as if it were an OpenVINO-supported serialized file format by passing the path to it into the `.compile_model` call and explicitly specify the `LLAMA_CPP` as the target pseudo-\"device\" so that the `LLAMA_CPP` plugin code would be used instead of the regular OpenVINO code paths:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "77d741bc-9674-4da0-b5c6-01ee95693128", | ||
| "metadata": { | ||
| "scrolled": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import openvino as ov\n", | ||
| "ov_model = ov.Core().compile_model(\"qwen1_5-7b-chat-q5_k_m.gguf\", \"LLAMA_CPP\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "76785d5e-e6f5-46b8-9ff8-4436ac5e67c0", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The models loaded through the `LLAMA_CPP` plugin flow from GGUF expose two primary inputs - `input_ids` and `position_ids`, with the same semantics as the corresponding model inputs in the original PyTorch representation of the models in the HuggingFace repository. Additionally, `attention_mask` and `beam_idx` inputs are exposed for drop-in compatibility with existing OpenVINO example pipelines, but these inputs are left unused since the `llama.cpp` execution model either does not require or does not expose these inputs." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "1e5531e8-84c7-4730-8036-3f35eb49ab84", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "ov_model.inputs" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "356a09b3-d8b1-4bf7-b74e-a3df922915d5", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Since this a chat version of the Qwen model, the user input should be formatted in a model and language-specific way. Below is some utility code that will be used to format the input prompt to the model." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ed042d99-8e22-42c8-ae7e-53719b7ca222", | ||
| "metadata": { | ||
| "scrolled": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "DEFAULT_SYSTEM_PROMPT_CHINESE = \"\"\"\\\n", | ||
| "你是一个乐于助人、尊重他人以及诚实可靠的助手。在安全的情况下，始终尽可能有帮助地回答。 您的回答不应包含任何有害、不道德、种族主义、性别歧视、有毒、危险或非法的内容。请确保您的回答在社会上是公正的和积极的。\n", | ||
| "如果一个问题没有任何意义或与事实不符，请解释原因，而不是回答错误的问题。如果您不知道问题的答案，请不要分享虚假信息。另外，答案请使用中文。\\\n", | ||
| "\"\"\"\n", | ||
| "\n", | ||
| "model_configuration = {\n", | ||
| " \"model_id\": \"Qwen/Qwen-7B-Chat\",\n", | ||
| " \"remote\": True,\n", | ||
| " \"start_message\": f\"<|im_start|>system\\n {DEFAULT_SYSTEM_PROMPT_CHINESE }<|im_end|>\",\n", | ||
| " \"history_template\": \"<|im_start|>user\\n{user}<im_end><|im_start|>assistant\\n{assistant}<|im_end|>\",\n", | ||
| " \"current_message_template\": '\"<|im_start|>user\\n{user}<im_end><|im_start|>assistant\\n{assistant}',\n", | ||
| " \"stop_tokens\": [\"<|im_end|>\", \"<|endoftext|>\"]\n", | ||
| "}\n", | ||
| "\n", | ||
| "model_name = model_configuration[\"model_id\"]\n", | ||
| "start_message = model_configuration[\"start_message\"]\n", | ||
| "history_template = model_configuration.get(\"history_template\")\n", | ||
| "current_message_template = model_configuration.get(\"current_message_template\")\n", | ||
| "stop_tokens = model_configuration.get(\"stop_tokens\")\n", | ||
| "tokenizer_kwargs = model_configuration.get(\"tokenizer_kwargs\", {})\n", | ||
| "\n", | ||
| "def convert_history(history):\n", | ||
| " text = start_message + \"\".join([\"\".join([history_template.format(num=round, user=item[0], assistant=item[1])]) \n", | ||
| " for round, item in enumerate(history[:-1])])\n", | ||
| " text += \"\".join([\"\".join([current_message_template.format(num=len(history) + 1, user=history[-1][0], assistant=history[-1][1])])])\n", | ||
| " return text" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "2b543737-eb59-453c-99f5-236fa162ada3", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now we can build the input tokens representing the user prompt to the model. The prompt is adjusted to conform to the expected chatbot template using the utility functions defined earlier and then tokenized using the corresponding tokenizer.\n", | ||
| "\n", | ||
| "**Note:** Although the GGUF file contains the vocabulary and the tokenizer information, the tokenization and detokenization steps are currently not part of the `LLAMA_CPP` flow. These steps of the LLM processing pipeline should be executed using other means (e.g. in Python you can instantiate the required tokenizers from the `transformers` library if these are available for your model, and optionally convert these to OpenVINO representation separately to be executed with other plugins such as 'CPU')." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "41f70a67-729c-4f3b-810c-d273d455e49e", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "user_prompt = \"孙悟空是谁?\"\n", | ||
| "# user_prompt = \"太阳为什么是黄色的?\"\n", | ||
| "\n", | ||
| "formatted_input_prompt = convert_history([[user_prompt, \"\"]])\n", | ||
| "\n", | ||
| "from transformers import AutoTokenizer\n", | ||
| "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n", | ||
| "\n", | ||
| "initial_prompt_tokens = tokenizer(formatted_input_prompt, return_tensors=\"np\", **tokenizer_kwargs).input_ids" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "3c523b75-4acf-46cc-abb3-ff312321ff3d", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The initial prompt tokens are fed through the model to provide the context for the response generation. Note that the regular OpenVINO Python API is used to supply inputs and receive outputs:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "fa601c84-33d4-44a3-a27f-83f08eebe407", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The response tokens are generated one-by-one, conditioned on all previous tokens (within a certain window) by the KV-cache internally maintained by `llama.cpp`, until an end-of-sequence token is encountered as output or the maximum generated token count is reached." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "cdbf4d60-519d-4c1d-862f-3796c362154c", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import numpy as np\n", | ||
| "sequence_length = len(initial_prompt_tokens[0])\n", | ||
| "position_ids = np.arange(0, sequence_length).reshape(initial_prompt_tokens.shape)\n", | ||
| "\n", | ||
| "infer_request = ov_model.create_infer_request()\n", | ||
| "infer_request.set_tensors({\"input_ids\": ov.Tensor(initial_prompt_tokens), \"position_ids\": ov.Tensor(position_ids)})\n", | ||
| "infer_request.infer()\n", | ||
| "logits = infer_request.get_tensor(\"logits\").data\n", | ||
| "\n", | ||
| "curr_token_ids = np.argmax(logits[:, -1, :], axis=1).reshape([1, 1])\n", | ||
| "\n", | ||
| "MAX_TOKENS_GENERATED = 256\n", | ||
| "STOP_TOKENS = [tokenizer(st, return_tensors=\"np\").input_ids[0][0] for st in model_configuration[\"stop_tokens\"]]\n", | ||
| "\n", | ||
| "curr_tokens_generated = 0\n", | ||
| "last_token_id = curr_token_ids[0][0]\n", | ||
| "\n", | ||
| "response_tokens = []\n", | ||
| "next_position_id = sequence_length - 1\n", | ||
| "\n", | ||
| "while (last_token_id not in STOP_TOKENS) and (curr_tokens_generated < MAX_TOKENS_GENERATED): \n", | ||
| " print(tokenizer.decode(last_token_id), end='')\n", | ||
| " curr_tokens_generated += 1\n", | ||
| " curr_position_ids = np.ndarray([1, 1], dtype=np.int64)\n", | ||
| " curr_position_ids[0][0] = next_position_id \n", | ||
| " next_position_id += 1\n", | ||
| " \n", | ||
| " infer_request.set_tensors({\"input_ids\": ov.Tensor(curr_token_ids), \"position_ids\": ov.Tensor(curr_position_ids)})\n", | ||
| " infer_request.infer()\n", | ||
| " curr_logits = infer_request.get_tensor(\"logits\").data\n", | ||
| " \n", | ||
| " curr_token_ids = np.argmax(curr_logits[:, -1, :], axis=1).reshape([1, 1])\n", | ||
| " last_token_id = curr_token_ids[0][0]\n", | ||
| "\n", | ||
| "infer_request.reset_state()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "7ba9e1a2-beda-426a-9979-0e1ab2970430", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Note the last line in the cell above - since the model inference is stateful, we should reset the model's internal state if we want to process new text inputs that are unrelated to the current chatbot interaction. " | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.9.5" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vshampor , I suggest to specify the high-level requirements for this notebooks at the beginning. Is it required to have NVidia GPU+CUDA? Will it work on Windows?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MaximProshin CUDA and GPU is not necessary in the basic version of the notebook execution, although the user may uncomment a CMake option to compile for CUDA support, in which case they are naturally expected to have CUDA and GPUs on board. Windows is not supported due to my using Linux paths and direct shell calls to build the plugin from source.