jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Model: Bypass lock checks when shutting down

Browse source 
Previously, when a SIGINT was emitted and a model load is running,
the API didn't shut down until the load finished due to waitng for
the lock. However, when shutting down, the lock doesn't matter since
the process is being killed anyway.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-08-03 16:05:34 -04:00
parent 65c16f2a7c
commit 2a33ebbf29
3 changed files with 15 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

4
common/model.py
View file

@ -43,11 +43,11 @@ def load_progress(module, modules):
yield module, modules
async def unload_model(skip_wait: bool = False):
async def unload_model(skip_wait: bool = False, shutdown: bool = False):
"""Unloads a model"""
global container
await container.unload(skip_wait=skip_wait)
await container.unload(skip_wait=skip_wait, shutdown=shutdown)
container = None

2
common/signals.py
View file

@ -29,7 +29,7 @@ async def signal_handler_async(*_):
"""Internal signal handler. Runs all async code to shut down the program."""
if model.container:
await model.unload_model(skip_wait=True)
await model.unload_model(skip_wait=True, shutdown=True)
if model.embeddings_container:
await model.unload_embedding_model()