jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Templates: Revert to load metadata on runtime

Browse source 
Metadata is generated via a template's module. This requires a single
iteration through the template. If a template tries to access a passed
variable that doesn't exist, it will error.

Therefore, generate the metadata at runtime to prevent these errors
from happening. To optimize further, cache the metadata after the
first generation to prevent the expensive call of making a template
module.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-08-17 11:30:50 -04:00
parent 617ac12150
commit b4752c1e62
2 changed files with 19 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

4
endpoints/OAI/utils/chat_completion.py
View file

@ -181,7 +181,9 @@ def _create_stream_chunk(
def _append_template_metadata(data: ChatCompletionRequest):
"""Adding metadata is a one-time process."""
template_metadata = model.container.prompt_template.metadata
template_metadata = model.container.prompt_template.extract_metadata(
data.template_vars
)
# Stop strings
if isinstance(data.stop, str):