The release notes from 0.10.0 look good. Particularly this:

RLHF Support: new RPC methods for runtime weight reloading (https://github.com/vllm-project/vllm/pull/20096) and config updates (https://github.com/vllm-project/vllm/pull/20095), logprobs mode for selecting which stage of logprobs to return (https://github.com/vllm-project/vllm/pull/21398).

We need to be careful to select the post-processing logprobs ones for our case. The RPC methods should be useful as well.

Blocking merging #543 until vllm is updated to 0.10.0 here and merged

We need to be careful to select the post-processing logprobs ones for our case.

w/ vllm==0.10.0, logprobs_mode is using "raw_logprobs" as default, which is the same as our previous usage.
@SahilJain314 do you think I need to add logprobs_mode="raw_logprobs" when init vllm worker?

@yuki-666 lets force init to always use raw_logprobs in case the default in vllm is changed in future releases.

raw logprobs are fine for now (same as v1). But are actually incorrect for top-k, top-p sampling (even though it appears as though it would work correctly). Both the logprobs from vllm and the training framework are incorrect since they haven't been processed, which introduces error. (I believe the claim from #773 that v1 is fine is incorrect)

We have 2 options for how to mitigate this:

• use the post-processing from feat: [1/2] Top-k and Top-p support for dtensor worker with vLLM V0 when TP==1 #773 on both vllm raw_logprobs and megatron training
• use vllm post-processed logprobs and apply the processing to just megatron.

I'm leaning strongly towards the latter as it leaves less room for us to make mistakes and not notice. We should track this in #69.

@yuki-666 I see in some of the plots that the new vllm version is much more 'spiky' in logprob error. This is fine (spikes are occasional) but would you mind also adding plots with the logprob error clamped to a max value of 1.2? I'd like to confirm that the typical value has not increased at all.

Also want to just leave a comment that 0.10.0 version may fix this issue I've observed in gemma runs

vllm-project/vllm#19449