jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

OAI: Add cancellation with inline load

Browse source 
When the request is cancelled, cancel the load task. In addition,
when checking if a model container exists, also check if the model
is fully loaded.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-09-11 00:08:55 -04:00
parent b9e5693c1b
commit e00eb09ef3
2 changed files with 14 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

8
endpoints/OAI/utils/completion.py
View file

@ -112,8 +112,12 @@ async def _stream_collector(
async def load_inline_model(model_name: str, request: Request):
"""Load a model from the data.model parameter"""
# Return if the model container already exists
if model.container and model.container.model_dir.name == model_name:
# Return if the model container already exists and the model is fully loaded
if (
model.container
and model.container.model_dir.name == model_name
and model.container.model_loaded
):
return
# Inline model loading isn't enabled or the user isn't an admin