jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

API: Add inline exception for dummy models

Browse source 
If an API key sends a dummy model, it shouldn't error as the server
is catering to clients that expect specific OAI model names. This
is a problem with inline model loading since these names would error
by default. Therefore, add an exception if the provided name is in the
dummy model names (which also doubles as inline strict exceptions).

However, the dummy model names weren't configurable, so add a new
option to specify exception names, otherwise the default is gpt-3.5-turbo.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-11-17 21:12:38 -05:00
parent b94c646210
commit bd9e78e19e
5 changed files with 53 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

17
common/config_models.py
View file

@ -141,14 +141,25 @@ class ModelConfig(BaseConfigModel):
False,
description=(
"Allow direct loading of models "
"from a completion or chat completion request (default: False)."
"from a completion or chat completion request (default: False).\n"
"This method of loading is strict by default.\n"
"Enable dummy models to add exceptions for invalid model names."
),
)
use_dummy_models: Optional[bool] = Field(
False,
description=(
"Sends dummy model names when the models endpoint is queried.\n"
"Enable this if the client is looking for specific OAI models."
"Sends dummy model names when the models endpoint is queried. "
"(default: False)\n"
"Enable this if the client is looking for specific OAI models.\n"
),
)
dummy_model_names: List[str] = Field(
default=["gpt-3.5-turbo"],
description=(
"A list of fake model names that are sent via the /v1/models endpoint. "
'(default: ["gpt-3.5-turbo"])\n'
"Also used as bypasses for strict mode if inline_model_loading is true."
),
)
model_name: Optional[str] = Field(