jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

API + Model: Add speculative ngram decoding

Browse source 
Speculative ngram decoding is like speculative decoding without the
draft model. It's not as useful because it only decodes on predictable
sequences, but it depends on the usecase.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-03-13 23:32:11 -04:00
parent 2ebefe8258
commit efc01d947b
3 changed files with 32 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

5
common/sampling.py
View file

@ -123,6 +123,10 @@ class BaseSamplerRequest(BaseModel):
default_factory=lambda: get_default_sampler_value("grammar_string"),
)
speculative_ngram: Optional[bool] = Field(
default_factory=lambda: get_default_sampler_value("speculative_ngram"),
)
# Aliased variables
typical: Optional[float] = Field(
default_factory=lambda: get_default_sampler_value("typical", 1.0),
@ -268,6 +272,7 @@ class BaseSamplerRequest(BaseModel):
"negative_prompt": self.negative_prompt,
"json_schema": self.json_schema,
"grammar_string": self.grammar_string,
"speculative_ngram": self.speculative_ngram,
}
return {**gen_params, **kwargs}