jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Start: Migrate options from cu121/118 to cu12

Browse source 
This encapsulates more cuda versions and makes install easier for
new users.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-08-19 22:25:30 -04:00
parent 1344726936
commit 30a3cd75cf
6 changed files with 35 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/01.-Getting-Started.md
View file

@ -47,7 +47,7 @@ To get started, make sure you have the following installed on your system:
1. On Windows: `.\venv\Scripts\activate`
2. On Linux: `source venv/bin/activate`
3. Install the pyproject features based on your system:
1. Cuda 12.x: `pip install -U .[cu121]`
1. Cuda 12.x: `pip install -U .[cu12]`
2. ROCm 5.6: `pip install -U .[amd]`
4. Start the API by either
1. Run `start.bat/sh`. The script will check if you're in a conda environment and skip venv checks.
@ -98,7 +98,7 @@ There are a couple ways to update TabbyAPI:
These scripts exit after running their respective tasks. To start TabbyAPI, run `start.bat` or `start.sh`.
2. **Manual** - Install the pyproject features and update dependencies depending on your GPU:
1. `pip install -U .[cu121]` = CUDA 12.x
1. `pip install -U .[cu12]` = CUDA 12.x
2. `pip install -U .[amd]` = ROCm 6.0
If you don't want to update dependencies that come from wheels (torch, exllamav2, and flash attention 2), use `pip install .` or pass the `--nowheel` flag when invoking the start scripts.
@ -121,7 +121,7 @@ NOTE:
Here are ways to install exllamav2:
1. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended)
1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu121` and `cp311` corresponds to CUDA 12.1 and python 3.11
1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu12` and `cp311` corresponds to CUDA 12.1 and python 3.11
2. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2`
2. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration.
3. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source)