jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Tree: Format

Browse source 
Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-08-19 22:58:25 -04:00
parent bd3aa5bb04
commit 4036c70d75
2 changed files with 8 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/01.-Getting-Started.md
View file

@ -121,7 +121,7 @@ NOTE:
Here are ways to install exllamav2: Here are ways to install exllamav2:
1. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended) 1. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended)
1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu12` and `cp311` corresponds to CUDA 12.1 and python 3.11 1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu121` and `cp311` corresponds to CUDA 12.1 and python 3.11
2. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2` 2. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2`
2. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration. 2. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration.
3. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source) 3. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source)

14
start.py
View file

@ -11,11 +11,12 @@ import traceback
from shutil import copyfile from shutil import copyfile
from typing import List from typing import List
has_uv = subprocess.run( has_uv = (
["uv", "--version"], subprocess.run(
stdout=subprocess.DEVNULL, ["uv", "--version"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
stderr=subprocess.DEVNULL ).returncode
).returncode == 0 == 0
)
start_options = {} start_options = {}
@ -149,7 +150,7 @@ def migrate_start_options(start_options: dict):
# Migrate gpu_lib key # Migrate gpu_lib key
gpu_lib = start_options.get("gpu_lib") gpu_lib = start_options.get("gpu_lib")
if (gpu_lib == "cu121" or gpu_lib == "cu118"): if gpu_lib == "cu121" or gpu_lib == "cu118":
print("GPU lib key is legacy, migrating to cu12") print("GPU lib key is legacy, migrating to cu12")
start_options["gpu_lib"] = "cu12" start_options["gpu_lib"] = "cu12"
migrated = True migrated = True
@ -165,7 +166,6 @@ def run_pip(command: List[str]):
if __name__ == "__main__": if __name__ == "__main__":
# Create an argparser and add extra startup script args # Create an argparser and add extra startup script args
# Try creating a full argparser if pydantic is installed # Try creating a full argparser if pydantic is installed
# Otherwise, create an abridged one solely for startup # Otherwise, create an abridged one solely for startup