documentation

Welcome

Welcome to the NRMetricFramework documentation! This directory provides function documentation for the NRMetricFramework GitHub repository.

Overview

No-reference (NR) metrics are algorithms that predict the quality of an image or video, using only the pixel values. This repository takes an inclusive philosophy. We want to also support metrics that analyze some information from compressed bit-streams, but the necessary support functions are not yet implemented.

This repository does not contain images and videos. See the subjective datasets page for information on where to download suitable datasets.

Tutorials / Demos

• Demo #1 — get started
• Demo #2 — code NR parameters
• Demo #3 — train NR metrics
• Demo #4 — create new subjective dataset variable
• IEEE Transactions on Broadcasting Why NR metrics for image and video quality lack accuracy and reproducibility
• Machine Learning: Tools — predicting image and video quality with machine learning — new!
• Machine Learning: Feature Selection — machine learning feature selection — new!

References

• Dataset Structure — variables that describe subjective datasets
• Publications — papers that describe this repository
• Subjective Datasets — list of subjective datasets for no reference metric research
• Updates — release notes; updates that change code behavior
• Wishlist — future capabilities that are desired but not yet implemented

Main Functions

• analyze_NRpars.m
• import_dataset.m
• calculate_NRpars.m
• compromise_NRpars.m
• ci_NRpars.m and ci_calc.m
• peek_NRpars.m
• residual_NRpars.m — new!

Exporting Functions

• export_dataset.m
• export_NRpars.m

Media Functions

• convert_media.m — convert videos to uncompressed AVI
• display_xyt.m and display_color_xyt.m — shown an image or video
• read_avi.m — read an uncompressed AVI file
• read_media.m — read a dataset's media
• rgb2ycbcr_double.m — color space conversion: RGB to YCbCr
• write_avi.m — write a video to an uncompressed AVI file
• ycbcr2rgb_double.m — color space conversion: YCbCr to RGB

Image Processing Functions

• divide_100_blocks.m — specify 100 regions, roughly equal in size
• filter_sobel.m — Sobel filter
• filter_si_hv_adapt.m — large edge spatial information (SI) filter
• filter_smooth.m — bandpass filter with characteristics similar to SI filter used by filter_si_hv_adapt.m
• image_scale.m — scale image to monitor used during subjective test
• st_statistic.m
• valid_region_search_nosafety.m

Subjective Test Analysis

• analyze_lab2lab.m

NR Metrics and Reports

This repository includes (1) code for NR metrics developed by various organizations, and (2) reports that analyze their performance. This introduction defines the ⭐ ⭐ ⭐ ⭐ ⭐ scale used in the reports and tables below. Generally, 1-star metrics are very innacurate, 2-star metrics are promising, 3-star metrics perform consistently across 10+ datasets, 4-star metrics are as accurate as one person, and 5-star metrics are as accurate as a 6 person pilot test.

NR Metric Sawatch

The Sawatch NR metric was developed by NTIA using the NRMetricFramework repository. Sawatch estimates mean opinion score (MOS) via a linear combination of other NR metric parameters that supply root cause analysis (RCA). An online demo of Sawatch version 2 is available here.

Metric Name	Goal	Rating	Notes
Sawatch version 3	MOS	⭐ ⭐ ⭐	NR metric training method demonstrated

Metric Name	Metric Group	Goal	Rating
S-WhiteLevel	Auto Enhancement	RCA	⭐ ⭐ ⭐
S-BlackLevel	Auto Enhancement	RCA	⭐ ⭐
S-Blockiness	Blockiness	RCA	⭐ ⭐
S-Blur	Blur	RCA	⭐ ⭐ ⭐
S-FineDetail	Fine Detail	RCA	⭐ ⭐ ⭐
S-ColorNoise	Peculiar Color	RCA	⭐ ⭐
S-SuperSaturated	Peculiar Color	RCA	⭐ ⭐
S-Pallid	Peculiar Color	RCA	⭐ ⭐
S-PanSpeed	Pan Speed	RCA	⭐ ⭐
S-Jiggle	Pan Speed	RCA	⭐ ⭐
S-dipIQ	dipIQ	MOS	⭐ ⭐

Other NR Metrics

The following pages provide objective and factual information on the performance of NR metrics from other organizations. These reports analyze the metric's performance on diverse media from modern camera systems. This is often outside of the metric's intended scope. See Introduction for details.

This section also includes NR metrics we developed that are not included in the current version of NR metric Sawatch.

Metric Name	Goal	Rating	Notes
2stepQA-NR	MOS	⭐ ⭐	NR constrained variant of 2stepQA, outliers and invalid values prevent ⭐ ⭐ ⭐ rating
BRISQUE	MOS	⭐
Curvelet QA	MOS	❓	4 variants, technical issues mar performance and prevent analyses
JP2KNR	MOS	⭐	Code produces errors, content dependencies, possible inspiration for RCA
LBP	MOS	⭐	Not intended for MOS estimation, possible inspiration for RCA
Log-BIQA	MOS	⭐
NIQE	MOS	⭐ ⭐	Re-training tools available
NR-IQA-CDI	MOS	⭐	Variants Mean, Standard deviation, and Skewness
NR-IQA-CDI	MOS	⭐ ⭐	Variants Kurtosis and Entropy: possible inspiration for RCA
NSS	MOS	⭐	3 variants, outliers mar performance
OG-IQA	MOS	⭐ ⭐	Partial analysis, possible inspiration for RCA
PIQE	MOS	⭐
SpEED-NR	MOS	⭐ ⭐	NR constrained variant of Speed-QA, outliers mar performance

Metric Name	Goal	Rating	Impairment	Notes
ADMD	RCA	⭐	Uneven illumination
AGWN	RCA	⭐	Noise
AllBorderWeight	RCA	⭐	Contours, borders
BorderWeight	RCA	⭐	Contours, borders
CPBD	RCA	⭐ ⭐	Blur/Sharpness
dipIQ	RCA	❓	Coding	dipIQ is intended for ORD, not RCA
Entropy_Noise	RCA	⭐	Noise
HVS-MaxPol	RCA	⭐ ⭐	Blur/Sharpness	4 variants, trained on 7 datasets, outliers prevent ⭐ ⭐ ⭐ rating
JNB	RCA	⭐ ⭐	Blur/Sharpness	Performance marred by resolution dependencies
MaxPol	RCA	⭐ ⭐	Blur/Sharpness	Aka Synthetic-MaxPol, invalid values prevent ⭐ ⭐ ⭐ rating
Munsell_Red	RCA	⭐	Red present	Complex relationship between metric and datasets
NR-PWN	RCA	⭐ ⭐	Noisiness	Performance marred by dataset dependencies
TDME	RCA	⭐ ⭐ ⭐	Contrast enhancement
TDMEC	RCA	⭐ ⭐ ⭐	Contrast enhancement
Viqet-Sharpness	RCA	⭐ ⭐ ⭐	Blur Sharpness

Metric Name	Goal	Rating	Notes
dipIQ	ORD	❓	NR metric training method, statistics for ORD proposed

Acknowledgements

If you use this repository in your research or product development, please reference this GitHub repository and the paper listed below:

Margaret H. Pinson, Philip J. Corriveau, Mikołaj Leszczuk, and Michael Colligan, "Open Software Framework for Collaborative Development of No Reference Image and Video Quality Metrics," Human Vision and Electronic Imaging (HVEI), Jan. 2020.

The first version of the code in this repository was designed and made available by:

• The Institute for Telecommunication Sciences (ITS), which is the research and engineering branch of the National Telecommunications and Information Administration (NTIA), an agency of the U.S. Department of Commerce (DOC)
• The Public Safety Communications Research (PSCR) Division of the National Institute for Standards and Technology (NIST), an agency of the U.S. Department of Commerce (DOC)

This repository was inspired by discussions and work conducted in the Video Quality Experts Group (VQEG), especially the efforts of the No Reference Metrics (NORM) project and the Video and Image Models for consumer content Evaluation (VIME) project.