API: Fix blocking iterator execution

Run these iterators on the background thread. On startup, the API
spawns a background thread as needed to run sync code on without blocking
the event loop.

Use asyncio's run_thread function since it allows for errors to be
propegated.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-03-16 22:31:50 -04:00 committed by Brian Dashore
parent 7fded4f183
commit 2755fd1af0
4 changed files with 56 additions and 24 deletions

View file

@ -1,7 +1,7 @@
"""Completion utilities for OAI server."""
from asyncio import CancelledError
import pathlib
from asyncio import CancelledError
from fastapi import HTTPException
from typing import Optional