jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Model + Config: Remove low_mem option

Browse source 
Low_mem doesn't work in exl2 and it was an experimental option to
begin with. Keep the loading code commented out in case it gets fixed
in the future.

A better alternative is to use 8bit cache which works and helps save
VRAM.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2023-12-03 01:07:42 -05:00
parent 109e4223e0
commit c67c9f6d66
2 changed files with 4 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

3
config_sample.yml
View file

@ -45,9 +45,6 @@ model:
# Disable Flash-attention 2. Set to True for GPUs lower than Nvidia's 3000 series. (default: False)
no_flash_attention: False
# Enable low vram optimizations in exllamav2 (default: False)
low_mem: False
# Enable 8 bit cache mode for VRAM savings (slight performance hit). Possible values FP16, FP8. (default: FP16)
cache_mode: FP16