Motivation

When evaluating SFT/DPO trained models, vLLM accelerated inference with a single LoRA adapter is often used, but this feature isn’t supported in the source code. Therefore, I added a few lines of code to enable this simple functionality.

Modification

Added a lora_path in opencompass/models/vllm.py and utilized it during generate.

Use cases (Optional)

Now we can use LoRA vLLM inference as shown in the code below.

models = [
dict(
type=VLLM,
abbr='Llama3_8B_LoRA_SFT',
path='llama-3-8b-instruct',
model_kwargs=dict(tensor_parallel_size=2, dtype='bfloat16', seed=0, max_model_len=4096, enable_lora=True,),
max_out_len=100,
max_seq_len=4096,
batch_size=32,
lora_path="Llama3_8B_LoRA_checkpoints/checkpoint-1250/",
generation_kwargs=dict(temperature=0.0, top_p=0.8, max_tokens=1024,),
stop_words=['<|end_of_text|>', '<|eot_id|>'],
run_cfg=dict(num_gpus=2),
)
]