-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WebNN EP] Support axes and fix some validation for Resize #21952
base: main
Are you sure you want to change the base?
Conversation
Honry
commented
Sep 2, 2024
- Supports arbitrary axes for Resize opset 18+
- Check all inputs and attributes more carefully
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Wanming.
return false; | ||
} | ||
|
||
if (has_axes && num_of_sizes != 2) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ONNX Resize technically has no notion of NCHW or NHWC - it simply has a list of axes, and dimension semantics are irrelevant (especially now that WebNN supports any axes, layout concepts and limitations matter much less now). Additionally ONNX supports 1D/2D/3D/4D... inputs (whereas ironically WebNN's resample2D doesn't actually support 2D input tensors), but it's unclear that this code currently handles 2D input tensors?
I think it would be cleaner (and more robust) if instead of checking a hard-coded axes sizes and explicit values, that we just normalized the sizes and scales up-front, expanding using the ONNX axes and deriving the WebNN axes from those. Then any other less common but legal cases from ONNX models (like 1D cases which are a subset of 2D, or a 3D case with scales of {1,2,2} where the 1.0 scale is actually irrelevant) should just work too.
e.g.
if (input_rank > 4) {
LOGS(logger, ERROR) << "WebNN resample does not support input rank more than 4D.";
return false;
}
... if using scales ...
// Expand the scales to get the full scales per axis.
std::vector<float> webnn_scales;
if (has_axes) {
webnn_scales = GetScales(...);
} else {
webnn_scales.assign(input_rank, 1.0f);
ExpandIntoAxes(/*inout*/ webnn_scales, scales, onnx_axes);
}
ResizeRightAligned(/*inout*/ webnn_scales, 4, 1.0f);
// Determine which axes are actually functional, retaining at least 2 axes.
// Remove any axes where scales == 1.
webnn_axes.resize(4);
std::iota(webnn_axes.begin(), webnn_axes.end(), 0);
for (size_t i = webnn_axes.size(); i-- > 0 && webnn_axes.size() > 2; ) {
if (webnn_scales[i] == 1.0f) {
webnn_axes.erase(i);
webnn_scales.erase(i);
}
}
... else if instead using sizes ...
// Expand the size to get the full output sizes per axis.
std::vector<uint32_t> webnn_sizes;
if (has_axes) {
webnn_sizes = GetSizes(...);
} else {
webnn_sizes = input_shape;
ExpandIntoAxes(/*inout*/ webnn_sizes, sizes, onnx_axes);
}
ResizeRightAligned(/*inout*/ webnn_sizes, 4, 1u);
// Determine which sizes are actually functional, retaining at least 2 axes.
// Remove any axes where output size equals the original input size.
webnn_axes.resize(4);
std::iota(webnn_axes.begin(), webnn_axes.end(), 0);
for (size_t i = webnn_axes.size(); i-- > 0 && webnn_axes.size() > 2; ) {
if (webnn_sizes[i] == input_sizes[i]) {
webnn_axes.erase(i);
webnn_sizes.erase(i);
}
}
// At this point, axes has 2 elements, and either scales or sizes has 2 elements.
...
// Ensure the input (which may be 1D/2D/3D) is coerced up to WebNN's required 4D.
ResizeRightAligned(/*inout*/ input_shape_4D_with_leading_ones, 4, 1u);
... reshape(input, input_shape_4D_with_leading_ones);
ExpandIntoAxes
is a simple templated function equivalent to:
for (size_t i = 0; i < axes.size(); ++i) {
output[axes[i]] = input[i]
}
ResizeRightAligned
is a simple function that inserts leading values:
some_vector.insert(some_vector.begin(), new_size - some_vector.size(), fill_value);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good proposal, this is actually a requirement for supporting Resize
for input shape less than 4D, currently implementation only supports 4D input.
Besides, in onnxruntime, layout transformation will transpose sizes
and scales
as well if it is NHWC input layout, (I fixed a minor bug to restrict it to only transpose 4D sizes
and scales
, https://github.com/microsoft/onnxruntime/pull/21954/files, PTAL :) ), this increases the complexity a bit, so can we move this requirement to a follow-up?
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
/azp run ONNX Runtime Web CI Pipeline |
Azure Pipelines successfully started running 1 pipeline(s). |
- Supports arbitrary axes for Resize opset 18+ - Check all inputs and attributes more carefully
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fdwr, thanks for your comments, I've addressed them % the support for input shape < 4D. PTAL, thanks!
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/webnn/builders/impl/resize_op_builder.cc
Outdated
Show resolved
Hide resolved
return false; | ||
} | ||
|
||
if (has_axes && num_of_sizes != 2) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good proposal, this is actually a requirement for supporting Resize
for input shape less than 4D, currently implementation only supports 4D input.
Besides, in onnxruntime, layout transformation will transpose sizes
and scales
as well if it is NHWC input layout, (I fixed a minor bug to restrict it to only transpose 4D sizes
and scales
, https://github.com/microsoft/onnxruntime/pull/21954/files, PTAL :) ), this increases the complexity a bit, so can we move this requirement to a follow-up?