1344726936
Actually commit the edits. Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
49 lines
1.5 KiB
Markdown
49 lines
1.5 KiB
Markdown
# Overview
|
|
|
|
Clients can send sampler parameters that users are unaware of, or even not send any parameters at all!
|
|
|
|
Sampler overrides is tabbyAPI's flexible proxy for adding or forcing sampler values before generation.
|
|
# Setting up
|
|
|
|
All supported samplers are located in `sampler_overrides/sample_preset.yml`.
|
|
|
|
> [!NOTE]
|
|
> Sampler overrides can also be switched at runtime via the `/v1/sampling/override` endpoint set. Please read the [API docs](https://theroyallab.github.io/tabbyAPI/#operation/list_sampler_overrides_v1_sampling_override_list_get) for more information.
|
|
|
|
Steps:
|
|
1. Create a new YML in the `sampler_overrides` folder or duplicate `sample_preset.yml`
|
|
2. Adjust the samplers you want to override using the examples below
|
|
3. Save and rename the file (if you haven't already)
|
|
4. Open `config.yml` and inside the `sampling` block, set the `sampler_override` key to the preset name from step 3
|
|
# Examples
|
|
|
|
Let's say a client doesn't send a value for `top_p`:
|
|
|
|
```yml
|
|
top_p:
|
|
override: 0.7
|
|
force: False
|
|
```
|
|
|
|
This override will change the default fallback of top_p from a neutral value of 1.0 to 0.7.
|
|
|
|
Now, let's say a client forces a value that you don't like (ex. `top-k: 30`):
|
|
|
|
```yml
|
|
top_k:
|
|
override: 0
|
|
force: True
|
|
```
|
|
|
|
This override forces top_k to be 0 no matter what the client sends.
|
|
|
|
Finally, let's say a client sends a list, but you want to add something else:
|
|
|
|
```yml
|
|
stop:
|
|
override: ["a"]
|
|
force: False
|
|
additive: True
|
|
```
|
|
|
|
Now this override appends the given stop string to whatever the client sends.
|