jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

OAI: Add separator style fallback

Browse source 
Some models may return None for separator style with FastChat. Fall
back to LLAMA2 if this is the case.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2023-12-01 23:30:19 -05:00
parent ae69b18583
commit 61f6e51fdb
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
OAI/utils.py
View file

@ -16,6 +16,7 @@ from typing import Optional, List
try:
import fastchat
from fastchat.model.model_adapter import get_conversation_template
from fastchat.conversation import SeparatorStyle
_fastchat_available = True
except ImportError:
_fastchat_available = False
@ -116,6 +117,9 @@ def get_chat_completion_prompt(model_path: str, messages: List[ChatCompletionMes
)
conv = get_conversation_template(model_path)
if conv.sep_style is None:
conv.sep_style = SeparatorStyle.LLAMA2
for message in messages:
msg_role = message.role
if msg_role == "system":