[Paddle-TRT] custom operator support generating plugin automatically #58976

ming1753 · 2023-11-14T04:19:14Z

PR types

Others

PR changes

Others

Description

Pcard-71501

本PR的目的是在写了自定义算子(custom opetator)后，只需提供尽可能少的代码就能使自定义算子进trt，即自动生成对应的trt plugin。

自定义算子提供了kenel实现，可以直接在plugin中复用
getOutputDimensions通过SetTrtInferShapeFn指定
supportsFormatCombination通过配置生成
- 格式：{"<dtype1>[:format1]+<dtype2>[:format2]+...", "<dtype3>[:format3]+<dtype4>[:format4]+..."}
- 依次列出所有输入和输出的dtype和format，format可缺省，缺省时为LINEAR
- 比如一个有2个输入1个输出，第1个输入支持fp32和fp16，第二个输入只支持int32的自定义算子，config可以写成：{"float32:LINEAR+int32:LINEAR+float32:LINEAR", "float16:LINEAR+int32:LINEAR+float16:LINEAR"}

Example：

PD_BUILD_OP(custom_op)
    .Inputs({"X"})
    .Outputs({"Out"})
    .SetKernelFn(PD_KERNEL(paddle_gap_forward))
    .Attrs({"test_attr1: std::vector<int>", "test_attr2: int"})
    .SetInferShapeFn(PD_INFER_SHAPE(InferShape))
    .SetInferDtypeFn(PD_INFER_DTYPE(InferDtype))
    .SetTrtInferShapeFn(PD_TRT_INFER_SHAPE(getOutputDimensions))
    .SetTrtSupportsFormatConfig({"float32:LINEAR+float32:LINEAR"});

paddle-bot · 2023-11-14T04:19:18Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

yuanlehome · 2023-11-22T07:30:19Z

paddle/phi/api/ext/op_meta_info.h

@@ -27,6 +27,10 @@ limitations under the License. */
 #include "paddle/utils/none.h"
 #include "paddle/utils/optional.h"

+#ifdef PADDLE_WITH_TENSORRT
+#include "NvInfer.h"


这个.h在inference demo仓库编译用户模块的时候，能找到吗

TensorRT的头文件，用户指定TENSORRT_ROOT后是可以找到的。这个问题会在demo的README.me中补充说明一下。

yuanlehome · 2023-11-22T07:33:17Z

paddle/phi/api/ext/op_meta_info.h

+  // NOTE(HongyuJia): Used to be compatible with the 2.0.1 released
+  // interface, and will be deprecated in the future
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(const bool&);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(const int&);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(const float&);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(const int64_t&);
+
+  // NOTE(HongyuJia): Used to be compatible with the 2.1 released
+  // interface, but not recommended
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(std::string);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(std::vector<int>);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(std::vector<float>);
+  PD_SPECIALIZE_TrtSupportsFormateCallHelper_FOR_ATTR(std::vector<std::string>);


这些为了兼容的，你这里是否有必要再加上？

我认为还是有必要的，不然在存量的自定义算子中，会出现为了使用trt还要把const int的属性改为int。

yuanlehome · 2023-11-22T07:35:16Z

inference demo仓库里也需要提供demo示例

yuanlehome · 2023-11-22T07:36:32Z

需要在PR描述里把所有不支持的情况以及TODO暴露一下，比如win上有bug之类的

XieYunshen

LGTM for set_tests_properties(test_inference_gap_setup PROPERTIES TIMEOUT 180)

zhiqiu

LGTM for const_cast

yuanlehome

LGTM

vivienfanghuagood · 2023-11-24T14:40:43Z

paddle/phi/api/ext/op_meta_info.h

+                pos_nbInputs_nbOutputs, in_out, attrs, pargs..., arg);     \
+      } catch (paddle::bad_any_cast&) {                                    \
+        PD_THROW(                                                          \
+            "Attribute cast error in custom operator TrtSupportsFormate "  \


这里换行有点奇怪

这里断行是和原自定义算子的SetKernelFn, SetInferDtypeFn, SetInferDtypeFn中保持一致写的

vivienfanghuagood · 2023-11-24T14:43:03Z

paddle/phi/api/ext/op_meta_info.h

+
+  // end: base template
+  template <typename T>
+  struct TrtSupportsFormateCallHelper<TypeTag<T>> {


Formate -> format？

vivienfanghuagood · 2023-11-24T14:47:49Z

test/custom_op/inference_gap.cc

+  return dimsOutput;
+}
+
+bool supportsFormatCombination(


这个函数写起来过于繁琐，增加用户理解成本，可以考虑使用代码生成：
https://github.com/NVIDIA-AI-IOT/tensorrt_plugin_generator/blob/main/tpg/generate.py

… auto_plugin_v2

yuanlehome

LGTM

Aurelius84

LGTM for const_cast

XiaoguangHu01

LGTM

…addlePaddle#58976) * [Paddle-TRT] custom operator support generating plugin automatically

ming1753 added 3 commits November 9, 2023 07:43

[Paddle-TRT] custom operator support generating plugin automatically

2a36ef5

[Paddle-TRT] custom operator support generating plugin automatically

4025faf

modify func name

7de2225

ming1753 added 6 commits November 14, 2023 05:03

fix bug

7141eae

fix address change bug

e01b822

add test

3bf3b90

modify test

a5c08fb

add test

a17ccee

disable windows test

e3fc460

vivienfanghuagood requested review from weishengying and YuanRisheng November 22, 2023 07:02

yuanlehome reviewed Nov 22, 2023

View reviewed changes

skip window CI

54385a4

XieYunshen previously approved these changes Nov 23, 2023

View reviewed changes

YuanRisheng previously approved these changes Nov 24, 2023

View reviewed changes

zhiqiu previously approved these changes Nov 24, 2023

View reviewed changes

Ligoml previously approved these changes Nov 24, 2023

View reviewed changes

yuanlehome previously approved these changes Nov 24, 2023

View reviewed changes

vivienfanghuagood reviewed Nov 24, 2023

View reviewed changes

modify func name

fcfc024

ming1753 dismissed stale reviews from yuanlehome, Ligoml, zhiqiu, and YuanRisheng via fcfc024 November 25, 2023 03:53

ming1753 dismissed XieYunshen’s stale review via fcfc024 November 25, 2023 03:53

ming1753 added 6 commits November 28, 2023 13:23

generate supportsFormatCombination

a9ac49a

add TensorRT version macro

8ebdca0

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

4c47593

… auto_plugin_v2

fix bug

0527d6d

modify func name

3d07727

modify func name

e7c2a74

vivienfanghuagood approved these changes Dec 4, 2023

View reviewed changes

YuanRisheng approved these changes Dec 4, 2023

View reviewed changes

XieYunshen approved these changes Dec 4, 2023

View reviewed changes

Ligoml approved these changes Dec 4, 2023

View reviewed changes

yuanlehome approved these changes Dec 4, 2023

View reviewed changes

Aurelius84 approved these changes Dec 4, 2023

View reviewed changes

XiaoguangHu01 approved these changes Dec 4, 2023

View reviewed changes

Wangzheee merged commit ad9c6ef into PaddlePaddle:develop Dec 4, 2023

SigureMo pushed a commit to gouzil/Paddle that referenced this pull request Dec 5, 2023

[Paddle-TRT] custom operator support generating plugin automatically (P…

75c1a5e

…addlePaddle#58976) * [Paddle-TRT] custom operator support generating plugin automatically

[Paddle-TRT] custom operator support generating plugin automatically #58976

[Paddle-TRT] custom operator support generating plugin automatically #58976

Uh oh!

Conversation

ming1753 commented Nov 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Description

Uh oh!

paddle-bot bot commented Nov 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuanlehome commented Nov 22, 2023

Uh oh!

yuanlehome commented Nov 22, 2023

Uh oh!

XieYunshen left a comment

Choose a reason for hiding this comment

Uh oh!

zhiqiu left a comment

Choose a reason for hiding this comment

Uh oh!

yuanlehome left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuanlehome left a comment

Choose a reason for hiding this comment

Uh oh!

Aurelius84 left a comment

Choose a reason for hiding this comment

Uh oh!

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ming1753 commented Nov 14, 2023 •

edited

Loading