Add FusedMultiTransformer fuse pass for GPT3 #45907
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
552a2c5 to
0617335
Compare
b3602sss
reviewed
Sep 21, 2022
paddle/fluid/framework/ir/pass.cc
Outdated
| if (!graph->Has(kPassRecorder)) { | ||
| graph->Set<PassRecorder>(kPassRecorder, new PassRecorder); | ||
| } | ||
| graph->Get<PassRecorder>(kPassRecorder).insert(Type()); | ||
|
|
||
| if(graph->IsMainGraph() and "graph_viz_pass"!=Type() and "graph_to_program_pass"!=Type()) { |
| int fusion_count{0}; | ||
| auto handler = [&](const GraphPatternDetector::subgraph_t& subgraph, | ||
| Graph* g) { | ||
| GET_IR_NODE_FROM_SUBGRAPH(input0, input0, fused_multi_transformer_pattern); |
There was a problem hiding this comment.
需要主动调用IsCompat(subgraph, graph)， AddOpCompat才能生效
qingqing01
previously approved these changes
Oct 17, 2022
| @@ -0,0 +1,416 @@ | |||
| // Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved. | |||
… add_fused_multi_transformer_pass_fleetx
qingqing01
previously approved these changes
Oct 19, 2022
LielinJiang
previously approved these changes
Oct 19, 2022
4e1aba3
1cd0d85 to
409cf3a
Compare
qingqing01
approved these changes
Oct 20, 2022
LielinJiang
approved these changes
Oct 20, 2022
raindrops2sea
approved these changes
Oct 20, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Describe
Add FusedMultiTransformer fuse pass for GPT3
add passes:
fused_multi_transformer_encoderfused_multi_transformer_decoderfused_multi_transformer_encoder_fuse_qkvfused_multi_transformer_decoder_fuse_qkvfused_multi_transformer_encoder
匹配GPT encoder部分的Transformer Layer(包含multt-head attention和feed forward部分)
转化为
FusedMultiTransformer融合OP，如图中红框部分即为一层，用于推理加速fused_multi_transformer_decoder
decoder结构与encoder类似，多了
Cache KV的处理，如图中红框部分转换为带
TimeStep输入的FusedMultiTransformer融合OP，用于解码fused_multi_transformer_encoder/decoder_fuse_qkv
fused_multi_transformer_encoder/decoder_fuse_qkv结构与fused_multi_transformer_encoder/decoder类似，差别为Multi-Head Attention例QKV是concat成一个Tensor计算的，计算完后通过split分开，如图中红框所示add multi-devices passes:
multi_devices_fused_multi_transformer_encoder_fuse_qkvmulti_devices_fused_multi_transformer_decoder_fuse_qkvmulti-devices passes是在上述pass的基础上插入了
c_identity和c_allreduce_sum卡间通信OPadd subgraph pass support for the 6 passes
add
IR_NODE_UNLINKfor unlinking nodes in Graphfix
Var->persistentcheck segment fault when Variable is nullTODO