jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Async: Add option to use Uvloop/Winloop

Browse source 
These are faster event loops for asyncio which should improve overall
performance. Gate these under an experimental flag for now to stress
test these loops.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-07-24 18:56:28 -04:00
parent 71de3060bb
commit 5c082b7e8c
6 changed files with 85 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

5
config_sample.yml
View file

@ -62,6 +62,11 @@ developer:
# This can save a few MBs of VRAM, but has a risk of errors. Use at your own risk.
#cuda_malloc_backend: False
# Enable Uvloop or Winloop (default: False)
# Make the program utilize a faster async event loop which can improve performance
# NOTE: It's recommended to enable this, but if something breaks, turn this off.
#uvloop: False
# Options for model overrides and loading
# Please read the comments to understand how arguments are handled between initial and API loads
model: