Config: Add experimental torch cuda malloc backend

This option saves some VRAM, but does have the chance to error out.
Add this in the experimental config section.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri 2024-02-14 21:44:04 -05:00
parent 664e2c417e
commit 949248fb94
3 changed files with 16 additions and 1 deletions

View file

@ -140,3 +140,8 @@ def add_developer_args(parser: argparse.ArgumentParser):
type=str_to_bool,
help="Disables API request streaming",
)
developer_group.add_argument(
"--cuda-malloc-backend",
type=str_to_bool,
help="Disables API request streaming",
)