Add endpoint for file support, purely to speed up processing of input_embeds. #2797
Conversation
There was a problem hiding this comment.
You can add a test case here https://github.com/sgl-project/sglang/blob/main/test/srt/test_input_embeddings.py
|
@merrymercy It's based on Gemma 2 2b where I launch it with the following While it's faster with the new endpoint I can't quit get it as close as how the model can perform on simple implementation. Is there's anything else I can do to get faster response? Perhaps there's something I overlooked while making the changes in this PR, I'm especially curious about whether fastapi implementation made in sglang can be improved. |
|
I skipped checking for param from file name because using the overlap scheduler the sampling params are ignored anyway. |
|
Could you rebase and fix the lint? Sorry for the inconvenience created by our recent refactor #2996 |
Motivation
While input_embeds is now supported, I find that for what I'm using it for, normally requesting through /generate wasn't enough. I was spending around 2.5 seconds on average until the request was even handled in tokenizer manager. With this in mind, I suspect that the main issue came around possible overhead and such. To overcome said issue, I figured sending said input_embeds as a file likely would reduce the issue, doing so I managed to shaved off around 2 seconds from my use case.
Please contribute or comment your idea of a proper unit test for this application.
Modifications
Added endpoint in server.py to handle direct file transfer.
Checklist