[megatron] refactor: support MLATransformerConfig abstraction for DeepSeek V3 #1836
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ETOgaosion
reviewed
Jun 4, 2025
| **override_transformer_config_kwargs, | ||
| ) | ||
| transformer_config = MLATransformerConfig(**args) | ||
| # Common parallel state parameters |
There was a problem hiding this comment.
Thanks for fixing this~
What about making an abstraction of using a _get_mla_transformer_config as a basic function for future models?
|
Could you consider my PR to your repo? |
Done! Thanks for your updates. |
|
And maybe need a merge after approving? qaq Also maybe rebase and merge, not Squash and merge~ |
sry, i am not familiar with this. its ok now ? |
|
Sure, successfully, thanks a lot! |
|
@ISEEKYAN Could you please help to review this refactor and make a double check~ |
sure |
|
@jinqinn It seams that the error you encountered has been fix in the latest main branch. |
ETOgaosion
approved these changes
Jun 10, 2025
yellowbee686
pushed a commit
to yellowbee686/verl
that referenced
this pull request
Jun 10, 2025
…pSeek V3 (volcengine#1836) I encountered an error when training DeepSeek V3 with the latest code due to the TransformerConfig not including q_lora_rank, which is required for DeepSeek V3. #### Error Message ``` (TaskRunner pid=1256989) File "/workspace/verl/verl/single_controller/base/megatron/worker.py", line 69, in _init_hf_config_and_tf_config (TaskRunner pid=1256989) tf_config = hf_to_mcore_config(hf_config, dtype) (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/registry.py", line 131, in hf_to_mcore_config (TaskRunner pid=1256989) return MODEL_CONFIG_CONVERTER_REGISTRY[model](hf_config, dtype) (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/config_converter.py", line 210, in hf_to_mcore_config_dpskv3 (TaskRunner pid=1256989) args = _get_base_transformer_config( (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/config_converter.py", line 85, in _get_base_transformer_config (TaskRunner pid=1256989) return TransformerConfig(**base_config) (TaskRunner pid=1256989) TypeError: TransformerConfig.__init__() got an unexpected keyword argument 'q_lora_rank' ``` #### Solution The `hf_to_mcore_config_dpskv3` function should directly create an `MLATransformerConfig` instance instead of going through `_get_base_transformer_config()`, since DeepSeek V3 uses Multi-Latent Attention (MLA) which requires MLA-specific parameters. --------- Co-authored-by: ETOgaosion <gaoziyuan19@mails.ucas.ac.cn>
whatadayG
pushed a commit
to whatadayG/verl
that referenced
this pull request
Sep 5, 2025
…pSeek V3 (volcengine#1836) I encountered an error when training DeepSeek V3 with the latest code due to the TransformerConfig not including q_lora_rank, which is required for DeepSeek V3. #### Error Message ``` (TaskRunner pid=1256989) File "/workspace/verl/verl/single_controller/base/megatron/worker.py", line 69, in _init_hf_config_and_tf_config (TaskRunner pid=1256989) tf_config = hf_to_mcore_config(hf_config, dtype) (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/registry.py", line 131, in hf_to_mcore_config (TaskRunner pid=1256989) return MODEL_CONFIG_CONVERTER_REGISTRY[model](hf_config, dtype) (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/config_converter.py", line 210, in hf_to_mcore_config_dpskv3 (TaskRunner pid=1256989) args = _get_base_transformer_config( (TaskRunner pid=1256989) File "/workspace/verl/verl/models/mcore/config_converter.py", line 85, in _get_base_transformer_config (TaskRunner pid=1256989) return TransformerConfig(**base_config) (TaskRunner pid=1256989) TypeError: TransformerConfig.__init__() got an unexpected keyword argument 'q_lora_rank' ``` #### Solution The `hf_to_mcore_config_dpskv3` function should directly create an `MLATransformerConfig` instance instead of going through `_get_base_transformer_config()`, since DeepSeek V3 uses Multi-Latent Attention (MLA) which requires MLA-specific parameters. --------- Co-authored-by: ETOgaosion <gaoziyuan19@mails.ucas.ac.cn>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I encountered an error when training DeepSeek V3 with the latest code due to the TransformerConfig not including q_lora_rank, which is required for DeepSeek V3.
Error Message
Solution
The
hf_to_mcore_config_dpskv3function should directly create anMLATransformerConfiginstance instead of going through_get_base_transformer_config(), since DeepSeek V3 uses Multi-Latent Attention (MLA) which requires MLA-specific parameters.