kingbri ab10b263fd Model: Add override base seq len ... Some models (such as mistral and mixtral) set their base sequence length to 32k due to assumptions of support for sliding window attention. Therefore, add this parameter to override the base sequence length of a model which helps with auto-calculation of rope alpha. If auto-calculation of rope alpha isn't being used, the max_seq_len parameter works fine as is. Signed-off-by: kingbri <bdashore3@proton.me>		2023-12-20 00:45:39 -05:00
..
types	Model: Add override base seq len	2023-12-20 00:45:39 -05:00
utils.py	API: Add route for draft model list	2023-12-19 23:45:53 -05:00