Skip to content

Pass params to vLLM model creation to improve flexibility #1136

@sergiopaniego

Description

@sergiopaniego

I've been playing with the vLLM (VLLMModel) support and generated a sample.

model = VLLMModel(model_id=model_name)

The support is super nice but it could benefit from some additional flexibility.
Other model clients like HfApiModel and OpenAIServerModel already provide client_kwargs where you can add the params for the client.
It could be interesting to extend this functionality to VLLMModel. For example, it would enable configurations like:

model = VLLMModel(model_id=model_name, client_kwargs={"max_model_len": 65536})
...

I already have a fix proposal developed 😄

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions