The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

It should always be the case indeed.
The built-in set isn't ordered right?
Is vllm faster when you pass 1 prompt with n=N outputs, than with N times the same prompt for n=1?

It should always be the case indeed. The built-in set isn't ordered right? Is vllm faster when you pass 1 prompt with n=N outputs, than with N times the same prompt for n=1?

Unfortunately, set is not ordered. Yes vllm can share the prefill for the n generations so it is faster, I profiled around 1.5x faster with the changes in this PR at 2k max_completion_length.

Nice!! I am surprised, I expected a smaller speedup given that the prefix should already be reused since #2757.

We should probably do the same with tranformers generation in a future PR, if it makes sense.

Anyway, can you just add comment somewhere to explain why we do this?

My guess is it's an easier optimization for vllm to understand that single prompt has multiple generations than sending the same prompt multiple times from the #2776 refactor.

@qgallouedec, without diving into the codebase of vllm, I would assume that the prefix cache is only used to compare a new batch of prompts with previously processed prompts. The system prompt is shared across all prompts, so this is cached and reused for all batches, whereas a new batch of prompts would first all need to have their prefill calculated and entered into the cache before vllm could identify that there are num_generations of the prompts are exactly the same. Hence you get some improvement when you move from prompt*num_generations to n generations for each prompt.

Let me know if you would like me to clarify.

Thanks Ed! Actually I meant adding a comment in the code to concisely explain why we merge the prompts. Something like Since 'prompts' contains 'num_generations' duplicates, we first take unique prompts, and generate num_generations outputs for each one. This is faster than generating outputs for each duplicate prompt individually..