Use NNCF in transformers + optimum-intel ?

[tigerinus]

I read that to use NNCF in transformers, patch is needed as described in https://github.com/openvinotoolkit/nncf/blob/develop/third_party_integration/huggingface_transformers/README.md

But it also seems that NNCF is already being used in optimum-intel as described in https://huggingface.co/docs/optimum/intel/optimization_ov

Does it mean to use NNCF I need to do both patching and making sure it uses optimum-intel library?

Thanks.

[MaximProshin]

@tigerinus , no, it's either of two ways. I would say Intel-optimum is recommended way to do it for HF models while the patch is more an example to demonstrate how it could be done directly.

[tigerinus]

I see.

What are the types of text-generation models supported by the post-training compression described at https://huggingface.co/docs/optimum/intel/optimization_ov#post-training-optimization ?

[alexsu52]

What type of models are you interested in? NNCF is general solution for model quantization and don't have known limitation by supporting text-generarion models. If you will face any problems, please, file an issue on github.

FYI: The transformer patch was removed in #2381