Allow for OpenVINO models to be stored in a model subdirectory

[tomaarsen]

What does this PR do?

This PR allows users to load a model from a subdirectory via just file_name.

My goal:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

Where only the modeling files are in the openvino subfolder.
Result:

No OpenVINO files were found for sentence-transformers-testing/stsb-bert-tiny-openvino, setting `export=True` to convert the model to the OpenVINO IR. Don't forget to save the resulting model with `.save_pretrained()`
Framework not specified. Using pt to export the model.
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 17.5M/17.5M [00:01<00:00, 15.3MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1.50k/1.50k [00:00<?, ?B/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.36MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 712k/712k [00:00<00:00, 2.71MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 732/732 [00:00<?, ?B/s]
...

I.e. it starts exporting because it doesn't find any openvino file anywhere (because it only looks in the root directory).

For reference, this is exactly how I do it for ONNX via Optimum, and it works well there.

An alternative

So I tried using subfolder:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino_model.xml", subfolder="openvino")
print(model)

But then the rest of the files outside of the subfolder can't be read, and I get:

Could not infer whether the model was already converted or not to the OpenVINO IR, keeping `export=False`.
The library name could not be automatically inferred. If using the command-line, please provide the argument --library {transformers,diffusers,timm,sentence_transformers}. Example: `--library diffusers`.
Traceback (most recent call last):
  File "c:\code\sentence-transformers\demo_export_onnx.py", line 54, in <module>
    model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino_model.xml", subfolder="openvino")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\intel\openvino\modeling_base.py", line 373, in from_pretrained
    return super().from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\modeling_base.py", line 381, in from_pretrained
    library_name = TasksManager.infer_library_from_model(model_id, subfolder, revision, cache_dir, token=token)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\exporters\tasks.py", line 1901, in infer_library_from_model
    library_name = cls._infer_library_from_model_name_or_path(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\optimum\exporters\tasks.py", line 1863, in _infer_library_from_model_name_or_path       
    raise ValueError(
ValueError: The library name could not be automatically inferred. If using the command-line, please provide the argument --library {transformers,diffusers,timm,sentence_transformers}. Example: `--library diffusers`.

I'm also pretty sure that I can't specify the library anywhere, so this is a dead end. Ideally, I want it to work equivalently to ONNX anyways.

The fix

The simple fix that I've applied for now is also look for openvino files in subdirectories. After all, if there's an exported openvino file anywhere, you probably want to inform the user rather than re-export a model.

After the fix, I get this:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

Compiling the model to CPU ...
<optimum.intel.openvino.modeling.OVModelForFeatureExtraction object at 0x0000024A2057FD10>

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Note

This issue will likely prevent me from including UKPLab/sentence-transformers#2712 into the next Sentence Transformers release, sadly. That PR adds OpenVINO and ONNX backends to Sentence Transformers, but currently it's not possible to load saved OpenVINO files due to this issue.

P.s. I have no plans to further update sentence-transformers-testing/stsb-bert-tiny-openvino, I'll be using it in my own tests as well.

cc @echarlaix @helena-intel

• Tom Aarsen

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[tomaarsen]

P.s. the original problematic snippet:

from optimum.intel import OVModelForFeatureExtraction

model_id = "sentence-transformers-testing/stsb-bert-tiny-openvino"
model = OVModelForFeatureExtraction.from_pretrained(model_id, file_name="openvino/openvino_model.xml")
print(model)

does work with optimum-intel 1.18.3, it's related to the new "setting export=True when needed".

• Tom Aarsen