Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WebNN EP] Support axes and fix some validation for Resize #21952

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Honry
Copy link
Contributor

@Honry Honry commented Sep 2, 2024

  • Supports arbitrary axes for Resize opset 18+
  • Check all inputs and attributes more carefully

@Honry
Copy link
Contributor Author

Honry commented Sep 2, 2024

@fdwr, @guschmue, PTAL, thanks!

Copy link
Contributor

@fdwr fdwr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Wanming.

onnxruntime/core/providers/webnn/builders/helper.h Outdated Show resolved Hide resolved
onnxruntime/core/providers/webnn/builders/helper.h Outdated Show resolved Hide resolved
onnxruntime/core/providers/webnn/builders/helper.h Outdated Show resolved Hide resolved
return false;
}

if (has_axes && num_of_sizes != 2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ONNX Resize technically has no notion of NCHW or NHWC - it simply has a list of axes, and dimension semantics are irrelevant (especially now that WebNN supports any axes, layout concepts and limitations matter much less now). Additionally ONNX supports 1D/2D/3D/4D... inputs (whereas ironically WebNN's resample2D doesn't actually support 2D input tensors), but it's unclear that this code currently handles 2D input tensors?

I think it would be cleaner (and more robust) if instead of checking a hard-coded axes sizes and explicit values, that we just normalized the sizes and scales up-front, expanding using the ONNX axes and deriving the WebNN axes from those. Then any other less common but legal cases from ONNX models (like 1D cases which are a subset of 2D, or a 3D case with scales of {1,2,2} where the 1.0 scale is actually irrelevant) should just work too.

e.g.

if (input_rank > 4) {
  LOGS(logger, ERROR) << "WebNN resample does not support input rank more than 4D.";
  return false;
}

... if using scales ...

// Expand the scales to get the full scales per axis.
std::vector<float> webnn_scales;
if (has_axes) {
  webnn_scales = GetScales(...);
} else {
  webnn_scales.assign(input_rank, 1.0f);
  ExpandIntoAxes(/*inout*/ webnn_scales, scales, onnx_axes);
}
ResizeRightAligned(/*inout*/ webnn_scales, 4, 1.0f);

// Determine which axes are actually functional, retaining at least 2 axes.
// Remove any axes where scales == 1.
webnn_axes.resize(4);
std::iota(webnn_axes.begin(), webnn_axes.end(), 0);
for (size_t i = webnn_axes.size(); i-- > 0 && webnn_axes.size() > 2; ) {
  if (webnn_scales[i] == 1.0f) {
    webnn_axes.erase(i);
    webnn_scales.erase(i);
  }
}

... else if instead using sizes ...

// Expand the size to get the full output sizes per axis.
std::vector<uint32_t> webnn_sizes;
if (has_axes) {
  webnn_sizes = GetSizes(...);
} else {
  webnn_sizes = input_shape;
  ExpandIntoAxes(/*inout*/ webnn_sizes, sizes, onnx_axes);
}
ResizeRightAligned(/*inout*/ webnn_sizes, 4, 1u);

// Determine which sizes are actually functional, retaining at least 2 axes.
// Remove any axes where output size equals the original input size.
webnn_axes.resize(4);
std::iota(webnn_axes.begin(), webnn_axes.end(), 0);
for (size_t i = webnn_axes.size(); i-- > 0 && webnn_axes.size() > 2; ) {
  if (webnn_sizes[i] == input_sizes[i]) {
    webnn_axes.erase(i);
    webnn_sizes.erase(i);
  }
}

// At this point, axes has 2 elements, and either scales or sizes has 2 elements.

...
// Ensure the input (which may be 1D/2D/3D) is coerced up to WebNN's required 4D.
ResizeRightAligned(/*inout*/ input_shape_4D_with_leading_ones, 4, 1u);

... reshape(input, input_shape_4D_with_leading_ones);

ExpandIntoAxes is a simple templated function equivalent to:

for (size_t i = 0; i < axes.size(); ++i) {
  output[axes[i]] = input[i]
}

ResizeRightAligned is a simple function that inserts leading values:

some_vector.insert(some_vector.begin(), new_size - some_vector.size(), fill_value);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good proposal, this is actually a requirement for supporting Resize for input shape less than 4D, currently implementation only supports 4D input.

Besides, in onnxruntime, layout transformation will transpose sizes and scales as well if it is NHWC input layout, (I fixed a minor bug to restrict it to only transpose 4D sizes and scales, https://github.com/microsoft/onnxruntime/pull/21954/files, PTAL :) ), this increases the complexity a bit, so can we move this requirement to a follow-up?

@fdwr
Copy link
Contributor

fdwr commented Oct 5, 2024

/azp run ONNX Runtime Web CI Pipeline

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

- Supports arbitrary axes for Resize opset 18+
- Check all inputs and attributes more carefully
Copy link
Contributor Author

@Honry Honry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fdwr, thanks for your comments, I've addressed them % the support for input shape < 4D. PTAL, thanks!

onnxruntime/core/providers/webnn/builders/helper.h Outdated Show resolved Hide resolved
onnxruntime/core/providers/webnn/builders/helper.h Outdated Show resolved Hide resolved
return false;
}

if (has_axes && num_of_sizes != 2) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good proposal, this is actually a requirement for supporting Resize for input shape less than 4D, currently implementation only supports 4D input.

Besides, in onnxruntime, layout transformation will transpose sizes and scales as well if it is NHWC input layout, (I fixed a minor bug to restrict it to only transpose 4D sizes and scales, https://github.com/microsoft/onnxruntime/pull/21954/files, PTAL :) ), this increases the complexity a bit, so can we move this requirement to a follow-up?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants