jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

OAI: Add API-based model loading/unloading and auth routes

Browse source 
Models can be loaded and unloaded via the API. Also add authentication
to use the API and for administrator tasks.

Both types of authorization use different keys.

Also fix the unload function to properly free all used vram.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2023-11-14 01:17:19 -05:00
parent 47343e2f1a
commit b625bface9
11 changed files with 195 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

22
config_sample.yml
View file

@ -1,8 +1,14 @@
model_dir: "D:/models"
model_name: "this_is_a_exl2_model"
max_seq_len: 4096
gpu_split: "auto"
rope_scale: 1.0
rope_alpha: 1.0
no_flash_attention: False
low_mem: False
# Network options
network:
host: "0.0.0.0"
port: 8012
# Only used if you want to initially load a model
model:
model_dir: "D:/models"
model_name: "airoboros-mistral2.2-7b-exl2"
max_seq_len: 4096
gpu_split: "auto"
rope_scale: 1.0
rope_alpha: 1.0
no_flash_attention: False
low_mem: False