jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Templates: Support list style chat_template keys

Browse source 
HuggingFace updated transformers to provide templates in a list for
tokenizers. Update to support this new format. Providing the name
of a template for the "prompt_template" value in config.yml will also
look inside the template list.

In addition, log if there's a template exception, but continue model
loading since it shouldn't shut down the application.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-04-07 11:17:55 -04:00
parent 5bb4995a7c
commit 46ac3beea9
3 changed files with 78 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

4
config_sample.yml
View file

@ -107,7 +107,9 @@ model:
# Possible values FP16, FP8, Q4. (default: FP16)
#cache_mode: FP16
# Set the prompt template for this model. If empty, chat completions will be disabled. (default: Empty)
# Set the prompt template for this model. If empty, attempts to look for the model's chat template. (default: None)
# If a model contains multiple templates in its tokenizer_config.json, set prompt_template to the name
# of the template you want to use.
# NOTE: Only works with chat completion message lists!
#prompt_template: