Skip to content

add DeformConv-19 #4783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 23, 2023
Merged

add DeformConv-19 #4783

merged 8 commits into from
Mar 23, 2023

Conversation

yuanyao-nv
Copy link
Contributor

@yuanyao-nv yuanyao-nv commented Jan 18, 2023

Description

Add new op DeformConv to opset 19.

Motivation and Context

Deformable convolution (introduced in https://arxiv.org/abs/1703.06211 and https://arxiv.org/abs/1811.11168) is a popular variant of the convolutional neural network used in situations where scale, pose, viewpoints, etc. deform parts of the image. Instead of sampling using an evenly spaced kernel, it adds offsets to the sampling locations.

Related issues: #2395, #2665, #2485

Torchvision implementation: https://pytorch.org/vision/stable/_modules/torchvision/ops/deform_conv.html

@yuanyao-nv yuanyao-nv requested review from a team as code owners January 18, 2023 01:06
@gramalingam gramalingam added the topic: operator Issues related to ONNX operators label Jan 25, 2023
Copy link
Contributor

@p-wysocki p-wysocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe new operators now require a reference implementation as well. Reference evaluator can be found at https://github.com/onnx/onnx/tree/main/onnx/reference.

@yuanyao-nv
Copy link
Contributor Author

Hi @xadupre , can I ask whether there's a way to call the python implementation of an existing op when writing it for a new op in onnx/reference/ops? For example in this case, I would like to make use of the existing implementation for GridSample and Conv. Thanks in advance!

@xadupre
Copy link
Contributor

xadupre commented Feb 14, 2023

Page https://onnx.ai/onnx/api/reference.html shows an example which uses an implementation of new custom operator. The same mechanism can be used to replace an existing op. Is that what you are looking for?

@yuanyao-nv
Copy link
Contributor Author

yuanyao-nv commented Feb 14, 2023 via email

from onnx.reference.ops.op_cos import Cos
from onnx.reference.ops.op_cosh import Cosh
from onnx.reference.ops.op_cum_sum import CumSum
from onnx.reference.ops.op_deform_conv import DeformConv

Check notice

Code scanning / CodeQL

Unused import

Import of 'DeformConv' is not used.
@yuanyao-nv
Copy link
Contributor Author

Hi @gramalingam @p-wysocki , do you have any more comments or questions about this PR?

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
yuanyao-nv and others added 2 commits March 22, 2023 21:26
@liqunfu
Copy link
Collaborator

liqunfu commented Mar 23, 2023

Thanks for this PR @yuanyao-nv ! feel free merging it anytime. I have 2 follow up questions on implementation of this op in ORT:
(1) I see you added 2 references to DeformConv models. I wonder is there any new models use the op after these 2 publications. I searched for any uses of pytorch DeformConv2d (https://pytorch.org/vision/main/generated/torchvision.ops.DeformConv2d.html) but did not find any use of it on internet.
(2) Now we have many ops extended to 3D and ND, any examples of using DeformDonv for 3D data is very helpful.
There is a plan to add DeformConv CPU kernels to ORT. It will be very helpful if you have any pointers to above 2 questions. thanks!

@yuanyao-nv yuanyao-nv merged commit bc4b4f3 into onnx:main Mar 23, 2023
@yuanyao-nv
Copy link
Contributor Author

yuanyao-nv commented Mar 23, 2023

Hi @liqunfu , to answer your questions:

  1. Besides the original papers, there are some other models mentioned in the issues I linked as well, such as Kernel Transformer Network https://arxiv.org/abs/1812.03115, and https://arxiv.org/abs/1911.08947.
  2. Examples of 3D case: https://arxiv.org/abs/2004.02803 https://arxiv.org/abs/1911.01898

@liqunfu
Copy link
Collaborator

liqunfu commented Mar 23, 2023

Hi @liqunfu , to answer your questions:

  1. Besides the original papers, there are some other models mentioned in the issues I linked as well, such as Kernel Transformer Network https://arxiv.org/abs/1812.03115.
  2. Examples of 3D case: https://arxiv.org/abs/2004.02803 https://arxiv.org/abs/1911.01898

Thank you very much @yuanyao-nv !

broune pushed a commit to broune/onnx that referenced this pull request May 6, 2023
* add DeformConv

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* fix lint error

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* add clarification on dimensionality

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* rename function

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* RUFF fix

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* add node about dimension support

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

* revert accidental pybind11 change

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>

---------

Signed-off-by: Yuan Yao <yuanyao@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: operator Issues related to ONNX operators
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants