Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy #4126

jantonguirao · 2022-04-11T07:05:39Z

Signed-off-by: Joaquin Anton janton@nvidia.com

Description
This PR extends Resize with new functionality:

New boolean parameter antialias, which enables an antialiasing filter when downscaling with linear and cubic interpolation modes.
New axes argument, allowing to select a subset of dimensions that sizes, roi and scales refer to.
New keep_aspect_ratio_policy argument, determining how to interpret the value of sizes. Three policies supported:
- "stretch" (default): The output is resized to the target size, disregarding the original aspect ratio
- "not_larger" : The output is resized so that no extent is larger than the provided size, while keeping its aspect ratio
- "not_smaller" : The output is resized so that no extent is smaller than the provided size, while keeping its aspect ratio

Motivation and Context

Why is this change required? What problem does it solve?

Some libraries apply a downscaling antialiasing (e.g. PIL) while others don't (e.g. OpenCV). Having such an option allows the user to be aligned with either. This options will be crucial to be able to port the data preprocessing pipeline from image classification models (resnet), which is currently described as a Python function using PIL resize. For more background see: https://github.com/onnx/working-groups/blob/main/preprocessing/notebooks/resize-antialias/resize-antialias.ipynb
Previous version of resize forces the user to provide a scale or extent for dimensions that are not meant to be resize. This makes it hard to use in some contexts, such as image resize, where the "channel" dimension should not be resized.
Resizing data while keeping aspect ratio is currently hard to achieve, requiring a complicated pipeline for a trivial and quite common operation. We can benefit to have such "keep aspect ratio" semantics, for example for image vision applications.

Signed-off-by: Joaquin Anton <janton@nvidia.com>

onnx/defs/tensor/defs.cc

mzient · 2022-05-04T08:19:15Z

onnx/defs/tensor/defs.cc

            AttributeProto::STRING,
            std::string("round_prefer_floor"))
        .Attr(
            "extrapolation_value",
            "When coordinate_transformation_mode is \"tf_crop_and_resize\" and x_original is outside the range [0, length_original - 1], this value is used as the corresponding output value. Default is 0.0f.",
            AttributeProto::FLOAT,
            static_cast<float>(0))
+        .Attr(
+            "antialias",
+            "If set to 1, \"linear\" and \"cubic\" interpolation modes will use an antialiasing filter when downscaling. "


This wording is likely to go stale when a new interpolation method is added. We could say that it is not applicable to "nearest" interpolation method.

That is true. On the other hand, there is a chance that a new interpolation method is added that doesn't support antialiasing.

Signed-off-by: Joaquin Anton <janton@nvidia.com>

p-wysocki · 2022-06-17T12:08:26Z

onnx/backend/test/case/node/resize.py

+def round_half_up(x: float) -> int:
+    """
+    Computes the nearest integer value, rounding halfway cases up
+    """
+    fractional = x - int(x)
+    if fractional == 0.5:
+        return int(x) + 1
+    else:
+        return round(x)


Is this function necessary? Unless you are going for very explicit and clear code, the same could be acheived by int(x + 0.5), maybe enclosed in a lambda.

Good point. I'll take your suggestion

p-wysocki · 2022-06-17T12:10:48Z

docs/Changelog.md

+<dt><tt>antialias</tt> : int (default is 0)</dt>
+<dd>If set to 1, "linear" and "cubic" interpolation modes will use an antialiasing filter when downscaling. Antialiasing is achieved by stretching the resampling filter by a factor max(1, 1 / scale), which means that when downsampling, more input pixels contribute to an output pixel.</dd>
+<dt><tt>axes</tt> : list of ints</dt>
+<dd>If provided, it specifies a subset of axes that 'roi', 'scales' and 'sizes' refer to. If not provided, all axes are assumed [0, 1, ..., r-1], where r = rank(data). Non-specified dimensions are interpreted as non-resizable. Negative value means counting dimensions from the back. Accepted range is [-r, r-1], where r = rank(data). Behavior is undefined if an axis is repeated.</dd>


The user could be notified about a possible undefined behaviour. Maybe throwing an error (like when both scales and sizes are given in Inputs) or giving a warning would be beneficial?

I've added some more error checking

onnx/defs/tensor/defs.cc

Signed-off-by: Joaquin Anton <janton@nvidia.com>

onnx/defs/tensor/utils.cc

Signed-off-by: Joaquin Anton <janton@nvidia.com>

…xel_for_nn Signed-off-by: Joaquin Anton <janton@nvidia.com>

Signed-off-by: Joaquin Anton <janton@nvidia.com>

yuanyao-nv · 2022-06-24T18:37:04Z

Should we be more precise in the ONNX spec about what antialias filter to use for various interpolation methods? For example, by giving the precise mathematical formula in cases of possible ambiguity.

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao · 2022-06-27T09:50:11Z

Should we be more precise in the ONNX spec about what antialias filter to use for various interpolation methods? For example, by giving the precise mathematical formula in cases of possible ambiguity.

@yuanyao-nv There is this definition:

Antialiasing is achieved by stretching the resampling filter by a factor max(1, 1 / scale), which means that when downsampling, more input pixels contribute to an output pixel.

which is basically extending the neighborhood region by the inverse of the scale factor, resampling the interpolation filter weights accordingly. Do you think we should phrase it differently?

Regarding being mathematically explicit, the formulas are in resize.py (cubic_coeffs_antialias, linear_coeffs_antialias). I feel that to express it mathematically in the documentation, we would need to also express all the formulas for the different interpolation methods with all the variants, which might be hard to do concisely

yuanyao-nv · 2022-06-29T17:41:38Z

Thanks for the reply @jantonguirao .
Do you know if this is the same antialias definition used in TensorFlow and PyTorch ? In the TF definition, they mention some hat/tent filter for bilinear.

jantonguirao · 2022-06-29T21:04:16Z

Thanks for the reply @jantonguirao . Do you know if this is the same antialias definition used in TensorFlow and PyTorch ? In the TF definition, they mention some hat/tent filter for bilinear.

It seems to be the same filter as those two frameworks.

Regarding Tensorflow:
becomes a hat/tent filter function with radius 1 when downsampling.
Yes, this is the same filter. Also known as triangular filter.

Regarding Pytorch:
They mention being close to what PIL does. PIL applies antialiasing with the same approach as I am suggesting here. This is a triangular filter for the case of bilinear interpolation.

I made a demo comparing this proposal against OpenCV (no antialiasing) and Pillow (antialiasing): Notebook here. As you can see, the formulas I am proposing match what Pillow is doing.

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jcwchen · 2022-07-26T16:08:25Z

onnx/defs/tensor/utils.cc

+  bool hasSizesInput = (3 < ctx.getNumInputs()) && (ctx.getInputType(3) != nullptr);
+
+  if (hasScalesInput + hasSizesInput != 1) {
+    fail_shape_inference("Either `sizes` or `scales` must be provided, but not both of them");


FYI this line causes shape inference failures with some of ONNX Model Zoo models. Please check this issue: #4380. Thanks!

…4126) * [WIP] Add Resize-16: Antialiasing filter Signed-off-by: Joaquin Anton <janton@nvidia.com> * Update to version 17 Signed-off-by: Joaquin Anton <janton@nvidia.com> * Add 'keep_aspect_ratio_policy' and 'axes' Signed-off-by: Joaquin Anton <janton@nvidia.com> * Add shape inference implementation Signed-off-by: Joaquin Anton <janton@nvidia.com> * Fix version converter Signed-off-by: Joaquin Anton <janton@nvidia.com> * Workaround windows build issue Signed-off-by: Joaquin Anton <janton@nvidia.com> * Fixes Signed-off-by: Joaquin Anton <janton@nvidia.com> * Code review fixes Signed-off-by: Joaquin Anton <janton@nvidia.com> * Apply code review suggestions Signed-off-by: Joaquin Anton <janton@nvidia.com> * Move new Resize to opset 18 Signed-off-by: Joaquin Anton <janton@nvidia.com> * Correct error checking Signed-off-by: Joaquin Anton <janton@nvidia.com> * Adjust converter to opset 18 Signed-off-by: Joaquin Anton <janton@nvidia.com> * Update DomainToVersionRange Signed-off-by: Joaquin Anton <janton@nvidia.com> * Apply clang-format Signed-off-by: Joaquin Anton <janton@nvidia.com> * Fix shape inference test Signed-off-by: Joaquin Anton <janton@nvidia.com> * Fix flake8 issue Signed-off-by: Joaquin Anton <janton@nvidia.com> * Delete generated test test_resize_downsample_sizes_nearest_tf_half_pixel_for_nn Signed-off-by: Joaquin Anton <janton@nvidia.com> * Check for repeated axes Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao requested review from a team as code owners April 11, 2022 07:05

jantonguirao changed the title ~~[WIP] Add Resize-16: Antialiasing filter~~ Add Resize-16: Antialiasing filter Apr 11, 2022

[WIP] Add Resize-16: Antialiasing filter

e509ae7

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao force-pushed the resize_16 branch from d196632 to 256b91f Compare April 13, 2022 17:29

Update to version 17

ab4215a

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao force-pushed the resize_16 branch from 256b91f to ab4215a Compare April 13, 2022 17:31

jantonguirao changed the title ~~Add Resize-16: Antialiasing filter~~ Add Resize-117: Antialiasing filter Apr 13, 2022

jantonguirao changed the title ~~Add Resize-117: Antialiasing filter~~ Add Resize-17: Antialiasing filter Apr 13, 2022

gramalingam added the topic: operator Issues related to ONNX operators label Apr 26, 2022

jantonguirao added 2 commits April 28, 2022 10:31

Merge remote-tracking branch 'upstream/main' into resize_16

77cf303

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Merge remote-tracking branch 'upstream/main' into resize_16

c77135d

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao force-pushed the resize_16 branch 8 times, most recently from d119e54 to e50bcf6 Compare May 2, 2022 16:46

Add 'keep_aspect_ratio_policy' and 'axes'

2997aa4

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao force-pushed the resize_16 branch from e50bcf6 to 2997aa4 Compare May 2, 2022 16:49

jantonguirao changed the title ~~Add Resize-17: Antialiasing filter~~ Add Resize-17: Antialiasing, axes and keep_aspect_ratio_policy May 2, 2022

jantonguirao added 4 commits May 3, 2022 16:39

Add shape inference implementation

80106c6

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Merge remote-tracking branch 'upstream/main' into resize_16

048fbb2

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Fix version converter

d330975

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Workaround windows build issue

947533a

Signed-off-by: Joaquin Anton <janton@nvidia.com>

mzient reviewed May 4, 2022

View reviewed changes

onnx/defs/tensor/defs.cc Show resolved Hide resolved

mzient reviewed May 4, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/main' into resize_16

1cedf71

Signed-off-by: Joaquin Anton <janton@nvidia.com>

p-wysocki reviewed Jun 17, 2022

View reviewed changes

Apply code review suggestions

1e0fe55

Signed-off-by: Joaquin Anton <janton@nvidia.com>

gramalingam reviewed Jun 21, 2022

View reviewed changes

onnx/defs/tensor/utils.cc Outdated Show resolved Hide resolved

jantonguirao added 10 commits June 22, 2022 10:29

Merge remote-tracking branch 'upstream/main' into resize_16

9e93acc

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Move new Resize to opset 18

84342fe

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Correct error checking

9142aab

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Adjust converter to opset 18

11788d6

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Update DomainToVersionRange

2f9dbbb

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Apply clang-format

d38e854

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Fix shape inference test

578fb3a

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Fix flake8 issue

0469292

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Delete generated test test_resize_downsample_sizes_nearest_tf_half_pi…

ab31c31

…xel_for_nn Signed-off-by: Joaquin Anton <janton@nvidia.com>

Merge remote-tracking branch 'upstream/main' into resize_16

40ada40

Signed-off-by: Joaquin Anton <janton@nvidia.com>

jantonguirao changed the title ~~Add Resize-17: Antialiasing, axes and keep_aspect_ratio_policy~~ Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy Jun 22, 2022

jantonguirao added 2 commits June 27, 2022 10:46

Merge remote-tracking branch 'upstream/main' into resize_16

9f0ec2e

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Check for repeated axes

bd43f72

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Merge remote-tracking branch 'upstream/main' into resize_16

110279f

Signed-off-by: Joaquin Anton <janton@nvidia.com>

gramalingam approved these changes Jun 29, 2022

View reviewed changes

jantonguirao added 2 commits July 6, 2022 09:19

Merge remote-tracking branch 'upstream/main' into resize_16

d41daa4

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Merge remote-tracking branch 'upstream/main' into resize_16

258a5fa

Signed-off-by: Joaquin Anton <janton@nvidia.com>

gramalingam merged commit 500c3f4 into onnx:main Jul 11, 2022

jcwchen mentioned this pull request Jul 26, 2022

New Resize check for sizes and scales breaks some existing models #4380

Closed

jcwchen reviewed Jul 26, 2022

View reviewed changes

gramalingam mentioned this pull request Oct 12, 2022

Enable partial data propagation for Resize's sizes #4582

Merged

seddonm1 mentioned this pull request Nov 30, 2022

to work with onnx 1.13 rc, implement ver 18 reduce and optioanl ops, … microsoft/onnxruntime#13765

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy #4126

Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy #4126

Uh oh!

jantonguirao commented Apr 11, 2022 •

edited

Loading

Uh oh!

Uh oh!

mzient May 4, 2022

Uh oh!

jantonguirao May 4, 2022

Uh oh!

p-wysocki Jun 17, 2022

Uh oh!

jantonguirao Jun 17, 2022

Uh oh!

p-wysocki Jun 17, 2022

Uh oh!

jantonguirao Jun 17, 2022

Uh oh!

Uh oh!

Uh oh!

yuanyao-nv commented Jun 24, 2022

Uh oh!

jantonguirao commented Jun 27, 2022 •

edited

Loading

Uh oh!

yuanyao-nv commented Jun 29, 2022

Uh oh!

jantonguirao commented Jun 29, 2022 •

edited

Loading

Uh oh!

jcwchen Jul 26, 2022

Uh oh!

Uh oh!

Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy #4126

Add Resize-18: Antialiasing, axes and keep_aspect_ratio_policy #4126

Uh oh!

Conversation

jantonguirao commented Apr 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mzient May 4, 2022

Choose a reason for hiding this comment

Uh oh!

jantonguirao May 4, 2022

Choose a reason for hiding this comment

Uh oh!

p-wysocki Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

jantonguirao Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

p-wysocki Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

jantonguirao Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuanyao-nv commented Jun 24, 2022

Uh oh!

jantonguirao commented Jun 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuanyao-nv commented Jun 29, 2022

Uh oh!

jantonguirao commented Jun 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcwchen Jul 26, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jantonguirao commented Apr 11, 2022 •

edited

Loading

jantonguirao commented Jun 27, 2022 •

edited

Loading

jantonguirao commented Jun 29, 2022 •

edited

Loading