🗝️ Fix type hint in vLLM client #3205
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR aims to fix an incorrect type hint in the vLLM client while also temporarily disabling the communicator initialization and cleanup routines.
- Fix the type hint in the generate method from list[list[str]] to list[list[int]].
- Comment out communicator initialization and atexit registration which could affect client resource management.
Comments suppressed due to low confidence (1)
trl/extras/vllm_client.py:94
- Commenting out the communicator initialization and cleanup may lead to resource management issues if left in production. If this change is temporary, consider adding a comment explaining the rationale or removing the dead code altogether.
# self.init_communicator()
| @@ -140,7 +139,7 @@ def generate( | |||
| min_p: float = 0.0, | |||
| max_tokens: int = 16, | |||
| guided_decoding_regex: Optional[str] = None, | |||
| ) -> list[list[str]]: | |||
| ) -> list[list[int]]: | |||
There was a problem hiding this comment.
The updated type hint now indicates the function returns list[list[int]], which differs from the previous list[list[str]]. Please verify that model completions are meant to be represented as integers.
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
Line 301 in f4813ae
There was a problem hiding this comment.
Pull Request Overview
This PR fixes an incorrect type hint in the vLLM client. The key change is updating the annotated return type of the generate() function from list[list[str]] to list[list[int]].
Comments suppressed due to low confidence (1)
trl/extras/vllm_client.py:143
- The updated type hint indicates that the function should return a list of lists of ints. Please verify that the actual returned values match this new type to avoid potential type errors.
) -> list[list[int]]:
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.