jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Docs: Update tool calling

Browse source 
For new variables and format.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-07-05 21:43:04 -04:00
parent 871f71c4e7
commit 1c3f84151f
1 changed files with 31 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

74
docs/10.-Tool-Calling.md
View file

@ -32,7 +32,7 @@ For example, if you are using a Llama 3.1 Family model you can simply modify you
```yaml
model:
...
prompt_template: chatml_with_headers_tool_calling
prompt_template: tool_calls/chatml_with_headers
```
If loading via `/v1/model/load`, you would also need to specify a tool-supporting `prompt_template`.
@ -40,7 +40,6 @@ If loading via `/v1/model/load`, you would also need to specify a tool-supportin
## Tool Template Variables
- `tools`: Tools object.
- `tools_json`: Tools object as a JSON string.
## Creating a Tool Calling Prompt Template
@ -56,29 +55,36 @@ Here's how to create a TabbyAPI tool calling prompt template:
```jinja
{# Metadata #}
{% set stop_strings = ["<|im_start|>", "<|im_end|>"] %}
{% set message_roles = ['system', 'user', 'assistant', 'tool'] %}
{% set tool_start = "<|tool_start|>" %}
{% set tool_end = "<|tool_end|>" %}
{%- set stop_strings = ["<|im_start|>", "<|im_end|>"] -%}
{%- set tool_start = "<|tool_start|>" -%}
{# Optional Metadata #}
{%- set tool_end = "<|tool_end|>" -%}
```
`tool_start` and `tool_end` should be selected based on which model you decide to use. For example, [Groq's Tool calling models](https://huggingface.co/Groq/Llama-3-Groq-70B-Tool-Use) expects `<tool_call>` and `</tool_call>` while [Llama3 FireFunctionV2's](https://huggingface.co/fireworks-ai/llama-3-firefunction-v2) model expects only `functools` to start the call, without a `tool_end`
2. Define an `initial_system_prompt`:
While the name of your `inital_system_prompt` can vary, it's purpose does not. This inital prompt is typically a simple instruction set followed by accessing the `tools_json` variable. This will contain the function specification the user provided to the `tools` endpoint in their client when the chat completion request. Inside the template we can call this like so: `{{ tools_json }}`.
While the name of your `inital_system_prompt` can vary, it's purpose does not. This initial prompt is typically a simple instruction set followed by accessing the `tools` template variable.
Note: Depending on the model you are using, it's possible your model may expect a special set of tokens to surround the function specifications. Feel free to surround `tools_json` with these tokens.
This will contain the function specification the user provided to the `tools` endpoint in their client when the chat completion request. Inside the template we can call this like so: `{{ tools | tojson }}`.
> [!NOTE]
> Depending on the model you are using, it's possible your model may expect a special set of tokens to surround the function specifications. Feel free to surround `tools_json` with these tokens.
> [!NOTE]
> To get a JSON representation of the tools variable, use `| tojson(indent=2)` in the assignment
```jinja
{% set initial_system_prompt %}
Your instructions here...
Available functions:
{{ tools_json }}
{{ tools | tojson(indent=2) }}
{% endset %}
```
You'll then want to make sure to provide this to the model in the first message it recieves. Here is a simple example:
You'll then want to make sure to provide this to the model in the first message it receives. Here is a simple example:
```jinja
{%- if loop.first -%}
@ -88,11 +94,11 @@ Here's how to create a TabbyAPI tool calling prompt template:
{{ content }}{{ eos_token }}
```
3. Handle messages with the `tool` role:
4. Handle messages with the `tool` role:
After a tool call is made, a *well behaved* client will respond to the model with a new message containing the role `tool`. This is a response to a tool call containing the results of it's execution.
The simplest implementation of this will be to ensure your `message_roles` list within your prompt template contains `tool`. Further customization may be required for models that expect specific tokens surrounding tool reponses. An example of this customization is the Groq family of models from above. They expect special tokens surrounding their tool responses such as:
The simplest implementation of this will be to ensure your `message_roles` list within your prompt template contains `tool`. Further customization may be required for models that expect specific tokens surrounding tool responses. An example of this customization is the Groq family of models from above. They expect special tokens surrounding their tool responses such as:
```jinja
{% if role == 'tool' %}
@ -100,51 +106,29 @@ Here's how to create a TabbyAPI tool calling prompt template:
{% endif %}
```
4. Preserve tool calls from prior messages:
5. Preserve tool calls from prior messages:
When creating a tool calling `prompt_template`, ensure you handle previous tool calls from the model gracefully. Each `message` object within `messages` exposed within the `prompt_template` could also contain `tool_calls_json`. This field will contain tool calls made by the assistant in previous turns, and must be handled appropriatly so that the model understands what previous actions it has taken (and can properly identify what tool response ID belongs to which call).
When creating a tool calling `prompt_template`, ensure you handle previous tool calls from the model gracefully. Each `message` object within `messages` exposed within the `prompt_template` could also contain `tool_calls`.
This will require using the `tool_start` (and possibly `tool_end`) from above to wrap the `tool_call_json` like so:
This field will contain tool calls made by the assistant in previous turns, and must be handled appropriately so that the model understands what previous actions it has taken (and can properly identify what tool response ID belongs to which call).
This will require using the `tool_start` (and possibly `tool_end`) from above to wrap the `tool_call` object like so:
```jinja
{% if 'tool_calls_json' in message and message['tool_calls_json'] %}
{{ tool_start }}{{ message['tool_calls_json'] }}{{ tool_end }}
{% if 'tool_calls' in message and message['tool_calls'] %}
{{ tool_start }}{{ message['tool_calls'] | tojson(indent=2) }}{{ tool_end }}
{% endif %}
```
5. Handle tool call generation:
6. Add the generation prompt check at the end:
```jinja
{% set tool_reminder %}
Available Tools:
{{ tools_json }}
Tool Call Format Example:
{{ tool_start }}{{ example_tool_call }}
Prefix & Suffix: Begin tool calls with {{ tool_start }} and end with {{ tool_end }}.
Argument Types: Use correct data types for arguments (e.g., strings in quotes, numbers without).
{% endset %}
{% if tool_precursor %}
{{ start_header }}system{{ end_header }}
{{ tool_reminder }}{{ eos_token }}
{{ start_header }}assistant{{ end_header }}
{{ tool_precursor }}{{ tool_start }}
{% else %}
{% if add_generation_prompt %}
{{ start_header }}assistant{{ end_header }}
{% endif %}
```
This clever bit of temporal manipulation allows us to slip in a reminder as a system message right before the model generates a tool call, but after it writes the `tool_start` token. This is possible due to TabbyAPI revisitng the `prompt_template` after a `tool_start` token is detected. Here's how it works:
- We detect `tool_precursor`, which signals the model is about to generate a tool call.
- We then inject a system message with our `tool_reminder`.
- Finally, we initialize an assistant message using `tool_precursor` as the content.
This creates the illusion that the model just happened to remember the available tools and proper formatting right before generating the tool call. It's like giving the model a little nudge at exactly the right moment, enhancing its performance without altering what the user sees.
When creating your own tool calling `prompt_template`, it's best to reference the default `chatml_with_headers_tool_calling.jinja` template as a starting point.
> [!NOTE]
> When creating your own tool calling template, it's best to reference the default `chatml_with_headers` template as a starting point.
## Support and Bug Reporting
For bugs, please create a detailed issue with the model, prompt template, and conversation that caused it. Alternatively, join our [Discord](https://discord.gg/sYQxnuD7Fj) and ask for Storm.