jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Model: Dynamically scale generate_window

Browse source 
Allows for adjustment of reservation space at the end of the context
before rolling it. This should be scaled as a model's max_seq_len
goes up.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-01-24 01:26:38 -05:00 • committed by Brian Dashore
parent b14c5443fd
commit 740b0215dd
1 changed files with 3 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

4
backends/exllamav2/model.py
View file

@ -554,7 +554,9 @@ class ExllamaV2Container:
token_healing = unwrap(kwargs.get("token_healing"), False)
max_tokens = unwrap(kwargs.get("max_tokens"), 150)
stream_interval = unwrap(kwargs.get("stream_interval"), 0)
generate_window = min(unwrap(kwargs.get("generate_window"), 512), max_tokens)
generate_window = max(
unwrap(kwargs.get("generate_window"), 512), max_tokens // 8
)
# Sampler settings
gen_settings = ExLlamaV2Sampler.Settings()