Shall we add a supported model list and raise error if the model is not in the list?

Try to avoid using log_probs_from_logits_response_rmpad because there is an unpad op inside. unpad is a cuda-blocking op. Instead, we can directly use unpad input_ids from the input

I think this list depends on transformers lib. No sure where to get this list. I didn't find any doc about the feature in transformers.

I think this list depends on transformers lib. No sure where to get this list. I didn't find any doc about the feature in transformers.

Simply add potential models in the CI. If the model passes CI, then add to the supported list. I guess we can target

• Llama
• Mistral
• QWen
• Gemma

Try to avoid using log_probs_from_logits_response_rmpad because there is an unpad op inside. unpad is a cuda-blocking op. Instead, we can directly use unpad input_ids from the input

Sure, I will write a new API for unpad input_ids

Simply add potential models in the CI. If the model passes CI, then add to the supported list. I guess we can target

Shall we add the test_transformers.py to CI? I didn't do it as I think it only depends on the transformers version and flash_attn version.

So, I guess the goal for the CI is to test whether the latest transformers + flash_attn would break our implementation

Simply add potential models in the CI. If the model passes CI, then add to the supported list. I guess we can target

Shall we add the test_transformers.py to CI? I didn't do it as I think it only depends on the transformers version and flash_attn version.

So, I guess the goal for the CI is to test whether the latest transformers + flash_attn would break our implementation

After this PR, we should set a minimum version of transformers

input_ids_rmpad, indices, cu_seqlens, max_seqlen_in_batch = unpad_input(
ValueError: too many values to unpack (expected 4)
When using hh-rlhf with rmpadding=True

Could you try flash-attn < 2.7? I guess flash-attn 2.7 changes the API. @PeterSH6 We should either enforce the version in setup.py or adapt the API for flash-attn >= 2.7

Or we can move the unpad logic into verl in case future breaks.

Or we can move the unpad logic into verl in case future breaks.

Thanks for your help! :)
My flash_attn version is 2.7.2. I will try to downgrade it.

I think we should deal with both cases: greater than 2.7 and lower.

A utility function would be enough for the two cases

from verl.utils.torch_functional import prepare_input_for_rmpad
ImportError: cannot import name 'prepare_input_for_rmpad' from 'verl.utils.torch_functional' (/verl_0111/verl/utils/torch_functional.py)

I find that thie function is not used. And not in the codebase. Maybe should just delete this line?

Sure, simply delete it. Sorry for the inconvenience

It's OK. And there is another error I need help.
File "/home/verl_0111/verl/single_controller/ray/base.py", line 399, in func
return getattr(self.worker_dict[key], name)(*args, **kwargs)
File "/home/verl_0111/verl/single_controller/base/decorator.py", line 404, in inner
return func(*args, **kwargs)
File "/home/verl_0111/verl/workers/fsdp_workers.py", line 853, in compute_rm_score
token_level_scores = self._expand_to_token_level(data, scores)
File "/home/verl_0111/verl/workers/fsdp_workers.py", line 775, in _expand_to_token_level
token_level_scores[torch.arange(batch_size), eos_mask_idx] = scores
RuntimeError: shape mismatch: value tensor of shape [128, 2048] cannot be broadcast to indexing result of shape [128]

You are using an actual reward model?

Could you toggle off reward model remove-padding for now? I guess we lack CI for model-based reward function.

OK. Thanks~~~

Also, I guess you have to change AutoModelForTokenClassification to AutoModelForSequenceClassification in order to make non-rmpad version work. We will fix this as soon as possible and add a CI for this case.