-
Notifications
You must be signed in to change notification settings - Fork 798
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
CVS-142136
- Loading branch information
Showing
5 changed files
with
1,296 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -452,6 +452,7 @@ MusicGen | |
Müller | ||
Nakayosi | ||
nano | ||
nanoLLaVA | ||
nar | ||
NAS | ||
natively | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
# Visual-language assistant with LLaVA Next and OpenVINO | ||
|
||
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/nano-llava-multimodal-chatbot/nano-llava-multimodal-chatbot.ipynb) | ||
|
||
nanoLLaVA is a "small but mighty" 1B vision-language model designed to run efficiently on edge devices. It uses [SigLIP-400m](https://huggingface.co/google/siglip-so400m-patch14-384) as Image Encoder and [Qwen1.5-0.5B](https://huggingface.co/Qwen/Qwen1.5-0.5B) as LLM. | ||
In this tutorial, we consider how to convert and run nanoLLaVA model using OpenVINO. Additionally, we will optimize model using [NNCF](https://github.com/openvinotoolkit/nncf) | ||
|
||
## Notebook contents | ||
The tutorial consists from following steps: | ||
|
||
- Install requirements | ||
- Download PyTorch model | ||
- Convert model to OpenVINO Intermediate Representation (IR) | ||
- Compress model weights using NNCF | ||
- Prepare Inference Pipeline | ||
- Run OpenVINO model inference | ||
- Launch Interactive demo | ||
|
||
In this demonstration, you'll create interactive chatbot that can answer questions about provided image's content. | ||
|
||
|
||
## Installation instructions | ||
This is a self-contained example that relies solely on its own code.</br> | ||
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start. | ||
For details, please refer to [Installation Guide](../../README.md). |
1,269 changes: 1,269 additions & 0 deletions
1,269
notebooks/nano-llava-multimodal-chatbot/nano-llava-multimodal-chatbot.ipynb
Large diffs are not rendered by default.
Oops, something went wrong.