jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Model: Add fixes for kwargs and add note for migration

Browse source 
One goal is to try migrating away from kwargs and use the ModelLoadRequest
instead. However, Pydantic doesn't support async validators making
parsing of the inline config impossible due to its use of aiofiles.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-04-21 22:39:07 -04:00
parent 93854a3107
commit d26260b332
1 changed files with 4 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

7
common/model.py
View file

@ -46,7 +46,6 @@ def load_progress(module, modules):
yield module, modules
# TODO: Change this to be inline with config.yml
async def apply_inline_overrides(model_dir: pathlib.Path, **kwargs):
"""Sets overrides from a model folder's config yaml."""
@ -105,10 +104,12 @@ async def load_model_gen(model_path: pathlib.Path, **kwargs):
container = None
# Model_dir is already provided
# TODO: Isolate the root cause
kwargs.pop("model_dir")
if "model_dir" in kwargs:
kwargs.pop("model_dir")
# Merge with config and inline defaults
# TODO: Figure out a way to do this with Pydantic validation
# and ModelLoadRequest. Pydantic doesn't have async validators
kwargs = {**config.model_defaults, **kwargs}
kwargs = await apply_inline_overrides(model_path, **kwargs)