Skip to content

Commit

Permalink
single-image-test: added accuracy workaround for computing CPU refere…
Browse files Browse the repository at this point in the history
…nce and adjust the NRMSE and PSNR metrics (#26020)

### Details:
Small improvements for SIT:
- Disable [dynamic quantization
feature](https://github.com/openvinotoolkit/openvino/blob/master/src/inference/include/openvino/runtime/properties.hpp#L574),
which is used by default by CPU pipeline to generate the reference
outputs and affects accuracy for some particular models
 - Increase number of decimals for reported NRMSE metric to improve
- For PSNR metric report fails only when the accuracy decrease not when
it increase above the target

### Tickets:
 - *[CVS-143420](https://jira.devtools.intel.com/browse/CVS-143420)*
  • Loading branch information
beleiuandrei authored Oct 2, 2024
1 parent 0b21860 commit cddcfe8
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions src/plugins/intel_npu/tools/single-image-test/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1200,7 +1200,8 @@ bool computeRRMSE(const ov::Tensor& output, const ov::Tensor& reference) {

double rrmseLoss = sqrt(error / sum);

std::cout << "RRMSE loss : " << rrmseLoss << " RRMSE threshold : " << FLAGS_rrmse_loss_threshold << std::endl;
std::cout << "RRMSE loss : " << std::fixed << std::setprecision(4) << rrmseLoss
<< " RRMSE threshold : " << FLAGS_rrmse_loss_threshold << std::endl;
return rrmseLoss <= FLAGS_rrmse_loss_threshold;
}

Expand Down Expand Up @@ -1267,7 +1268,8 @@ bool computeNRMSE(const ov::Tensor& output, const ov::Tensor& reference) {
double nrmseLoss =
sqrt(error / size) / std::max(0.001f, std::max(maxOutput - minOutput, maxReference - minReference));

std::cout << "NRMSE loss : " << nrmseLoss << " NRMSE threshold : " << FLAGS_nrmse_loss_threshold << std::endl;
std::cout << "NRMSE loss : " << std::fixed << std::setprecision(4) << nrmseLoss
<< " NRMSE threshold : " << FLAGS_nrmse_loss_threshold << std::endl;
return nrmseLoss <= FLAGS_nrmse_loss_threshold;
}

Expand Down Expand Up @@ -1319,7 +1321,7 @@ bool testPSNR(const TensorMap& outputs, const TensorMap& references, const int d

auto result = utils::runPSNRMetric(actOutput, refOutput, dstHeight, dstWidth, scaleBorder, normalizedImage);

if (std::fabs(result - FLAGS_psnr_reference) > FLAGS_psnr_tolerance) {
if (FLAGS_psnr_reference - result > FLAGS_psnr_tolerance) {
std::cout << "Absolute difference between actual value " << result << " and reference value "
<< FLAGS_psnr_reference << " larger then tolerance " << FLAGS_psnr_tolerance << std::endl;
return false;
Expand Down

0 comments on commit cddcfe8

Please sign in to comment.