Merge branch 'latest' into openvoice-weight-compression

.ci/check_notebooks.py

@@ -15,6 +15,7 @@
     Path("notebooks/openvino-tokenizers/openvino-tokenizers.ipynb"),  # cpu required for loading extensions
     Path("notebooks/sparsity-optimization/sparsity-optimization.ipynb"),  # cpu expected to be used
     Path("notebooks/s3d-mil-nce-text-to-video-retrieval/s3d-mil-nce-text-to-video-retrieval.ipynb"),  # only cpu
+    Path("notebooks/explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb"),  # device-agnostic
 ]
 
 

.ci/ignore_treon_docker.txt

@@ -71,4 +71,5 @@ notebooks/stable-diffusion-v3/stable-diffusion-v3.ipynb
 notebooks/pixart/pixart.ipynb
 notebooks/llm-rag-llamaindex/llm-rag-llamaindex.ipynb
 notebooks/minicpm-v-multimodal-chatbot/minicpm-v-multimodal-chatbot.ipynb
-notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb
+notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb
+notebooks/explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb

.ci/skipped_notebooks.yml

@@ -529,6 +529,11 @@
     - python:
         - '3.8'
         - '3.9'
+- notebook: notebooks/explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb
+  skips:
+    - python:
+        - '3.8'
+        - '3.9'
 - notebook: notebooks/minicpm-v-multimodal-chatbot/minicpm-v-multimodal-chatbot.ipynb
   skips:
     - os:

.ci/spellcheck/.pyspelling.wordlist.txt

@@ -3,6 +3,7 @@ abstractive
 accelerometers
 accelerometer
 acknowledgement
+ActivationMap
 activations
 adaptively
 adas
@@ -27,6 +28,7 @@ APIs
 Arcface
 argmax
 artstation
+arxiv
 ASPP
 ASR
 asr
@@ -87,6 +89,7 @@ Cifar
 cityscape
 Cityscapes
 classname
+ClassProbabilityMap
 CLI
 cli
 CLIP's
@@ -232,6 +235,7 @@ et
 Evol
 EVS
 eXplainable
+explainer
 Explainer
 facto
 fastcomposer
@@ -510,6 +514,7 @@ notus
 nsamples
 nsfw
 NSFW
+num
 numpy
 NumPy
 NPU
@@ -663,6 +668,7 @@ RealSR
 Realtime
 realtime
 rebase
+ReciproCAM
 redistributable
 RedPajama
 ReferenceNet
@@ -761,6 +767,7 @@ splitters
 SPS
 SQA
 SQuAD
+src
 SRT
 SSD
 SSDLite
@@ -884,6 +891,7 @@ virtualenv
 VisCPM
 ViT
 vit
+VITReciproCAM
 vits
 VITS
 vitt

notebooks/README.md

@@ -113,6 +113,7 @@
 
 ## Explainable AI
 
+- [OpenVINO™ Explainable AI Toolkit: Deep Dive notebook](./explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb)
 - [eXplainable AI (XAI) for OpenVINO™ IR Models](./explainable-ai-1-basic/explainable-ai-1-basic.ipynb)
 - [Language-Visual Saliency with CLIP and OpenVINO™](./clip-language-saliency-map/clip-language-saliency-map.ipynb)
 

notebooks/big-transfer-quantization/tensorflow-bit-image-classification-nncf-quantization.ipynb

@@ -55,7 +55,7 @@
     "if platform.system() != \"Windows\":\n",
     "    %pip install -q \"matplotlib>=3.4\" \"tensorflow_datasets>=4.9.0\"\n",
     "else:\n",
-    "    %pip install -q \"matplotlib>=3.4,<3.7\" \"tensorflow_datasets>=4.9.0<4.9.3\""
+    "    %pip install -q \"matplotlib>=3.4,<3.7\" \"tensorflow_datasets>=4.9.0,<4.9.3\""
    ]
   },
   {
@@ -642,7 +642,7 @@
    "cell_type": "code",
    "execution_count": 14,
    "metadata": {
-       "test_replace": {
+    "test_replace": {
      "for img_batch, _ in validation_dataset:": "for img_batch, _ in validation_dataset.take(2):"
     }
    },

notebooks/explainable-ai-1-basic/README.md

@@ -1,4 +1,5 @@
 # eXplainable AI (XAI) for OpenVINO™ IR Models
+
 [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb)
 
 The notebook will cover the following topics:
@@ -10,9 +11,7 @@ The notebook will cover the following topics:
 
 A saliency map is a visualization technique that highlights regions of the interest in an image from the model perspective. For example, it can be used to explain classification model predictions for a particular label. Here is an example of a saliency map that you will get in this notebook:
 
-<p align="center">
-    <img width="80%" src="https://github-production-user-asset-6210df.s3.amazonaws.com/17028475/343197195-ccb67c0b-c58e-4beb-889f-af0aff21cb66.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240626%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240626T141846Z&X-Amz-Expires=300&X-Amz-Signature=cbad9a773f45d590b47c4fc2cf0c0c3a09fc1f77961eaa934eb0d110d990e2b2&X-Amz-SignedHeaders=host&actor_id=17028475&key_id=0&repo_id=806947463"/>
-</p>
+![Saliency Map Example](https://github.com/user-attachments/assets/5557d79d-2e9a-4784-aa17-fea2931a1435)
 
 ## Installation Instructions
 

notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb

@@ -431,16 +431,19 @@
    "version": "3.10.13"
   },
   "openvino_notebooks": {
-   "imageUrl": "https://user-images.githubusercontent.com/36741649/127172572-1cdab941-df5f-42e2-a367-2b334a3db6d8.jpg",
+   "imageUrl": "https://github.com/user-attachments/assets/5557d79d-2e9a-4784-aa17-fea2931a1435",
    "tags": {
     "categories": [
      "First Steps",
      "Explainable AI"
     ],
     "libraries": [],
-    "other": [],
+    "other": [
+        "XAI"
+    ],
     "tasks": [
-     "Image Classification"
+        "Image Classification",
+        "Image-to-Image"
     ]
    }
   },

notebooks/explainable-ai-2-deep-dive/README.md

@@ -0,0 +1,35 @@
+# OpenVINO™ Explainable AI Toolkit: Classification Explanation
+
+**OpenVINO™ Explainable AI (XAI) Toolkit** provides a suite of XAI algorithms for visual explanation of
+[**OpenVINO™**](https://github.com/openvinotoolkit/openvino) Intermediate Representation (IR) models.
+
+Using **OpenVINO XAI**, you can generate **saliency maps** that highlight regions of interest in input images from the model's perspective. This helps users understand why complex AI models produce specific responses.
+
+This notebook shows an example how to use OpenVINO XAI.
+
+It depicts a heatmap with areas of interest where neural network (classification or detection) focuses before making a decision.
+
+Example: Saliency map for `flat-coated retriever` class for MobileNetV3 classification model:
+
+![Saliency Map Example](https://github.com/user-attachments/assets/5557d79d-2e9a-4784-aa17-fea2931a1435)
+
+## Notebook Contents
+
+The tutorial consists of the following steps:
+
+- Run explainer in Auto-mode
+- Specify preprocess and postprocess functions
+- Run explainer in White-box mode
+  - Insert XAI branch to use updated model in own pipelines
+- Run explainer in Black-box mode
+- Advanced: add label names and use them to save saliency maps instead of label indexes
+
+These are explainable AI algorithms supported by OpenVINO XAI :
+
+| Domain          | Task                 | Type      | Algorithm           | Links |
+|-----------------|----------------------|-----------|---------------------|-------|
+| Computer Vision | Image Classification | White-Box | ReciproCAM          | [arxiv](https://arxiv.org/abs/2209.14074) / [src](https://github.com/openvinotoolkit/openvino_xai/blob/releases/1.0.0/openvino_xai/methods/white_box/recipro_cam.py) |
+|                 |                      |           | VITReciproCAM  (for transformer models)     | [arxiv](https://arxiv.org/abs/2310.02588) / [src](https://github.com/openvinotoolkit/openvino_xai/blob/releases/1.0.0/openvino_xai/methods/white_box/recipro_cam.py) |
+|                 |                      |           | ActivationMap       | experimental / [src](https://github.com/openvinotoolkit/openvino_xai/blob/releases/1.0.0/openvino_xai/methods/white_box/activation_map.py) |
+|                   |                    | Black-Box | RISE                | [arxiv](https://arxiv.org/abs/1806.07421v3) / [src](https://github.com/openvinotoolkit/openvino_xai/blob/releases/1.0.0/openvino_xai/methods/black_box/rise.py) |
+|                 | Object Detection     |           | ClassProbabilityMap | experimental / [src](https://github.com/openvinotoolkit/openvino_xai/blob/releases/1.0.0/openvino_xai/methods/white_box/det_class_probability_map.py) |