IQA-PyTorch v0.1.13 Release Notes

📢 Major Changes

📦 Model Weights Migration

• All weights moved to Hugging Face: Now available at chaofengc/IQA-PyTorch-Weights
  • ⚡ Enables faster and more stable downloads
  • 🤝 Allows community contributions for model weights
  • 🌏 For users in mainland China: Use the Hugging Face mirror with:

    export HF_ENDPOINT=https://hf-mirror.com

⚡ Performance Improvements

• Enhanced Metric Efficiency: All metrics now process 1080×800 images with:
  • ⏱️ Sub-second inference time on NVIDIA V100 GPU
  • 💾 Memory efficient: Most metrics require <6GB GPU memory. Exceptions: qalign and qalign_8bit
  • 📊 Detailed results available in efficiency benchmark

📚 Development Updates

• Dataset Organization: Split information consolidated into meta CSV files
  • 🔍 Available at chaofengc/IQA-PyTorch-Datasets-metainfo

🐛 Bug Fixes

• 🔧 Fixed weight loading functionality (5bc4df8)
• 🔄 Removed imgaug for NumPy 2.x compatibility (336831)
• 🛠️ Resolved LIQE num_patch issue (5a3b3e3) - Thanks @avermilov

✨ New Features

📊 New Metrics

• Add perceptual color difference metric msswd proposed in MS-SWD (ECCV2024). (3835ad9)
• Added lpips+ and lpips-vgg+ (b57dede)
• Implemented piqe metric (5b5afb3)
• Added MATLAB-compatible niqe_matlab and brisque_matlab (065c2b7)
• Introduced qalign_8bit and qalign_4bit (e225d8f)

🛠️ Enhanced Functionality

• Added approximate score ranges for metrics (cbba398)
• Implemented optional input validation (0aa6f71)

⚡ Performance & Usability Improvements

🚀 Speed Optimizations

• 📥 Faster loading through simplified imports and lazy loading with class mapper cache
• ⚡ 5x GPU performance boost for nrqm and pi metrics
• 🔋 Enhanced efficiency for ilniqe and piqe through matrix operations

💻 Command Line Interface

• 🔧 Improved CLI syntax:

  pyiqa [metric_name(s)] -t [image_path or dir] -r [image_path or dir] --device cuda --verbose

📖 Documentation

• 📚 Comprehensive documentation updates and improvements

👥 Contributors

Thank you to our new contributors for their valuable input:

• 🌟 @jaycecd: Score range improvements (#180)
• 🌟 @Luciennnnnnn: Cache strategy enhancement (#192)
• 🌟 @RistoAle97: Typo fixes (#197)

📝 For complete details, see the Full Changelog

pyiqa v.0.1.12

❗Important Change: Remove set_random_seed in test process.

In this version, set_random_seed has been removed due to its potential negative impact on the training process. For metrics that were previously tested on multiple crops, we have adopted a uniform cropping approach to ensure consistent results. This method aims to cover the entire image as much as possible, providing a fairer evaluation.
Please note that this change may lead to slight differences in the following metrics. Below is a comparison of results on sample images:

Metrics	I03.bmp	I04.bmp	I06.bmp	I08.bmp	I19.bmp	average
ahiq	-0.1059	0.5855	0.7066	0.5176	0.2566	0.3921
v0.1.12	-0.0996	0.5952	0.7331	0.5710	0.2196	0.4039
hyperiqa	0.1572	0.7397	0.7056	0.7204	0.2898	0.5226
v0.1.12	0.1692	0.7270	0.6972	0.7130	0.3105	0.5234
maniqa	0.2518	0.6176	0.4449	0.5197	0.2309	0.430
v0.1.12	0.2320	0.6033	0.4573	0.5201	0.2670	0.415
tres	27.86	85.95	87.19	95.53	80.02	75.31
v0.1.12	25.67	85.64	85.31	96.49	77.49	74.12

📢 Fix Bugs

1. Fix bug in inference of liqe. 0508684

✨ New features

1. Improve codes to allow training of lpips, dists, pieapp 036f2cc
2. Add distributed training. 18f123e

🛠️ Improvements

1. Update demo codes. c8ae6a7
2. Add alert to make README more readable. bee2c14

🍻 New Contributors

• Add ARNIQA by @LorenzoAgnolucci at #158
• Update requirements @shakedel by at #164

Many thanks to the valuable contributions 🤗 !

Full Changelog: v0.1.11...v0.1.12

pyiqa v0.1.11

📢 Fix Bugs

1. 🐛 fix topiq_nr-face multiple inference error
2. 📝 update distributed train

✨ New features

You can now use pyiqa in terminal like this:

# list all available metrics
pyiqa -ls

# test with default settings
pyiqa [metric_name(s)] --target [image_path or dir] --ref [image_path or dir]

1. ✨ add wadiqam pretrained models
2. ✨ add qalign
3. 🔨 add scandir_images func
4. 🚩 add inception_score
5. 🚩 add console entry point with pyiqa command

🛠️ Improvements

1. add star-history
2. 🔧 set seed for every forward in test mode

🍻 New Contributors

• Add UNIQUE by @zwx8981 at #146

Many thanks to the valuable contributions 🤗 !

Full Changelog: v0.1.10...v0.1.11

pyiqa v0.1.10

📢 Fix Bugs

1. fix vertical flip in data transforms 872e6ca
2. fix bug of fid calculation for different sizes 2da824d
3. fix maniqa hyperparameter error 33a2829
4. fix device handling 6b1547f

✨ New features

1. Add huggingface link to download datasets 031fafd
2. Add liqe liqe_mix by @zwx8981
3. Add nima-koniq and nima-spaq bd385c7

🛠️ Improvements

1. Improve training pipeline 43fb392
2. Add more error messages in fid c36d31c
3. Improve error message for FR and demo codes 921bd75

🍻 New Contributors

• Improve SSIM numerical stability by transforming operations into float64 by @Luciennnnnnn in #125
• Add LIQE by @zwx8981 in #127
  Many thanks to their valuable contributions 🤗 !

Full Changelog: v0.1.8...v0.1.10

pyiqa v0.1.8

📢 Fix Bugs

1. Fix lower_better for FID be5d8c7
2. Change mad option test_y_channel to be false by default cf84ffb
3. Set read rgb to true in inference model to force read RGB image ad93cd4

✨ New features

1. Add topiq_fr, topiq_nr, topiq_nr-face, topiq_iaa etc. introduced by our paper TOPIQ
2. Add st-lpips by @abhijay9 in #93. Many thanks ❤️ !
3. Add laion_aes introduced by LAION-Aesthetics_Predictor
4. Add datasets PIQ2023, GFIQA f46faae

🛠️ Improvements

1. Add documents: https://iqa-pytorch.readthedocs.io
2. Update to torchvision>=0.13, torch>=1.12
3. Improve dataset api 6fbae36
4. Meta information files for training can be download automatically now.
5. Update evaluation protocol and results 2770a7e
6. Update results of maniqa 7ee5ea6
7. Add assertion for brisque to force gray input 4e2c707

New Contributors

• @abhijay9 made their first contribution in #93

Full Changelog: v0.1.7...v0.1.8

pyiqa v0.1.7

📢 Important Changes & Bug Fix

1. Fix verbose option in FID 49b2297
2. Fix grad backpropagation for as_loss=True e027618
3. Fix niqe with gray scale input 212ecef

✨ New features

1. Add metric uranker b142d1c
2. Add metric maniqa-koniq, maniqa-kadid fe95923
3. Add metric clipscore for image-caption matching ecb3e5e
4. Add metric entropy to calculate gray scale image entropy like matlab 5f6d4fb
5. Add pytest cases for results calibration, datasets loading and gradient backward. bc5e135

🛠️ Improvements

1. Recursively find images in folder for FID calculation d7ade54
2. Add metric output doc in model cards

Hotfix of NRQM & PI

🚨 Hotfix: NRQM & PI Calculation Bug Resolved

We have identified and resolved a critical bug in the NRQM calculation within our toolbox. The issue stemmed from the SSIM (Structural Similarity Index) function, where only the structure similarity score was being utilized. We apologize for any inconvenience this may have caused.

As a result of this bug, the PI, which is determined by the formula PI = 0.5 * (10 - NRQM + NIQE), was also affected.

With this hotfix, we have:

• Updated the NRQM calculation to correctly incorporate all relevant components of the SSIM function.
• Adjusted the PI calculation to reflect the corrected NRQM values.

We strongly recommend using the latest release to benefit from these crucial fixes.

If you encounter any issues or have further questions, please don't hesitate to reach out to our support team. Thank you for your understanding and continued support.

IQA-PyTorch v0.1.6

⚠️ Important Changes & Bug Fix

1. Fix OOM on GPU for NRQM fcb7f6e
2. Fix device problem for gradient calculation 08f8850
3. Fix bug for small image test of NIQE 8d7462d
4. Fix default dataset config 1baa70e
5. Fix clip installation error 53d176f
6. feat: add psnry for y colorspace; ssimc for RGB ssim

New features

1. Add CNNIQA, TreS
2. Pass forward argument to inference model
3. Add loss reduction when using metric as loss f03d7f1

Improvements

1. Update benchmark results
2. Update clipiqa+

IQA-PyTorch v0.1.5

⚠️ Fix bugs

1. Fix FID bug
2. Fix read meta info error in livechallenge.
3. Fix shape error for NRQM
4. Fix bug in nancov
5. Add missing requirements package
6. Fix link for lpips squeeze net version

New features

1. Add MANIQA, AHIQ pretrained weights
2. Add metric_mode option for list_models
3. Add new metrics: FID, MANIQA
4. Enable image path as inputs. See demo codes in README
5. Add as_loss option to enable gradient backpropagation for metric. Default False.

Improvements

1. Use epoch instead of iteration in lr scheduler
2. Add clean_state_dict before loading pretrain model

IQA-PyTorch v0.1.4

New features

1. Add new metrics: FID, MANIQA
2. Enable image path as inputs. See demo codes in README
3. Add as_loss option to enable gradient backpropagation for metric. Default False.

Fix bugs

1. Fix rmse error
2. Fix benchmark test with PieAPP

Improvements

1. Disable gradient calculation by default for convenience.
2. Add filter2 function to matlab utils
3. Add reduction option to EMDLoss
4. Add crop_border option to PSNR, SSIM