jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Model + API: GPU split updates and fixes

Browse source 
For the TP loader, GPU split cannot be an empty array. However,
defaulting the parameter to an empty array makes it easier to calculate
the device list. Therefore, cast an empty array to None using
falsy comparisons at load time.

Also add draft_gpu_split to the load request.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-02-15 21:50:14 -05:00
parent 304df16543
commit 9f649647f0
2 changed files with 10 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

7
endpoints/core/types/model.py
View file

@ -63,7 +63,10 @@ class DraftModelLoadRequest(BaseModel):
default=None,
examples=[1.0],
)
draft_cache_mode: Optional[str] = None
draft_gpu_split: Optional[List[float]] = Field(
default_factory=list,
examples=[[24.0, 20.0]],
)
class ModelLoadRequest(BaseModel):
@ -94,7 +97,7 @@ class ModelLoadRequest(BaseModel):
gpu_split_auto: Optional[bool] = None
autosplit_reserve: Optional[List[float]] = None
gpu_split: Optional[List[float]] = Field(
default=None,
default_factory=list,
examples=[[24.0, 20.0]],
)
rope_scale: Optional[float] = Field(