File another issue #413 to trace FSDP2 for MoE models.

1. Qwen3-30B-A3B is obviously slower than Qwen3-32B, especially on the refit process or when using hf-tp-plan with dtensor tp > 1.
2. DeepseekV2ForCausalLM using fsdp2 will fail on the following error on model.layers.0.self_attn.rotary_emb.cos_cached, said v.shape=torch.Size([2048, 64]) and self.reference_model_buffers[k].shape=torch.Size([163840, 64]) shape mismatch in self.use_reference_model().

• DeepseekV2ForCausalLM should use vllm==0.8.5 and add this line https://github.com/vllm-project/vllm/blob/v0.9.0/vllm/v1/attention/backends/mla/common.py#L494 to vllm for test.

@yuki-666 what were the parameters of your gemma2 run? I can't seem to get it to run correctly:

uv run examples/run_grpo_math.py policy.model_name=google/gemma-2-2b-it logger.wandb_enabled=True cluster.gpus_per_node=8 +policy.generation.vllm_cfg.load_format=auto

@terrykong Thanks very much for pointing out this!

I tested with almost the same script as yours before this commit fdb565c.
After this commit, load_format of vllm during training is default set to dummy, only specific models will change it through nemo_rl/models/huggingface/common.py. The param policy.generation.vllm_cfg.load_format is removed from yaml and has no effect even if we pass it.

It is fixed now, and other models won't be affect since they don't need special handle on load_format.

Thanks @jgerh , have updated from your suggestions.

Thanks for the quick fix @yuki-666 . Gemma2 seems to be okay now from a quick run: