tabbyAPI-ollama/common
kingbri 0eedc8ca14 API: Switch from request ID middleware to depends
Middleware runs on both the request and response. Therefore, streaming
responses had increased latency when processing tasks and sending
data to the client which resulted in erratic streaming behavior.

Use a depends to add request IDs since it only executes when the
request is run rather than expecting the response to be sent as well.

For the future, it would be best to think about limiting the time
between each tick of chunk data to be safe.

Signed-off-by: kingbri <bdashore3@proton.me>
2024-07-22 12:19:46 -04:00
..
args.py Tree: Format 2024-05-26 15:48:18 -07:00
auth.py Auth: Fix key check coalesce 2024-07-19 10:08:57 -04:00
concurrency.py API + Model: Add blocks and checks for various load requests 2024-05-25 21:16:14 -04:00
config.py Tree: Update to cleanup globals 2024-03-12 23:59:30 -04:00
downloader.py Tree: Swap from map to list comprehensions 2024-05-25 21:16:14 -04:00
gen_logging.py API: Add ability to use request IDs 2024-07-21 21:01:05 -04:00
logger.py API: Add HuggingFace downloader 2024-04-29 01:15:02 -04:00
model.py Tree: Ruff lint 2024-07-08 15:06:28 -04:00
networking.py API: Switch from request ID middleware to depends 2024-07-22 12:19:46 -04:00
sampling.py Tree: Swap from map to list comprehensions 2024-05-25 21:16:14 -04:00
signals.py Signal: Fix signal handlers for uvicorn 2024-03-16 23:23:31 -04:00
templating.py Templates: Migrate to class 2024-04-21 23:28:14 -04:00
transformers_utils.py Tree: Add transformers_utils 2024-04-20 00:07:39 -04:00
utils.py Sampling: Copy over iterable overrides 2024-05-17 21:38:28 -04:00