jalr/tabbyAPI-ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Docs: Update getting started guide

Browse source 
Add downloader options and edit some points.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri 2025-02-18 12:14:52 -05:00
parent 830301b2b4
commit d6b8c7db4b
1 changed files with 35 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

62
docs/01.-Getting-Started.md
View file

@ -5,15 +5,12 @@ To get started, make sure you have the following installed on your system:
- [Python 3.x](https://www.python.org/downloads/release/python-3117/) (preferably 3.11) with pip
- Do NOT install python from the Microsoft store! This will cause issues with pip.
- Alternatively, you can use miniconda if it's present on your system.
- Alternatively, you can use miniconda or uv if it's present on your system.
> [!NOTE]
> Prefer a video guide? Watch the step-by-step tutorial on [YouTube](https://www.youtube.com/watch?v=03jYz0ijbUU)
> [!NOTE]
> You can install [miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) on your system which will give you the benefit of having both python and conda!
> [!WARNING]
> CUDA and ROCm aren't prerequisites because torch can install them for you. However, if this doesn't work (ex. DLL load failed), install the CUDA toolkit or ROCm on your system.
>
@ -39,26 +36,34 @@ To get started, make sure you have the following installed on your system:
1. Follow the on-screen instructions and select the correct GPU library.
2. Assuming that the prerequisites are installed and can be located, a virtual environment will be created for you and dependencies will be installed.
4. The API should start with no model loaded
4. The API should start with no model loaded. Please read more to see how to download a model.
### For Advanced Users
> [!NOTE]
> TabbyAPI has recently switched to use pyproject.toml. These instructions may look different than before.
1. Follow steps 1-2 in the [For Beginners](#for-beginners) section
2. Create a python environment through venv:
5. Follow steps 1-2 in the [For Beginners](#for-beginners) section
6. Create a python environment through venv:
1. `python -m venv venv`
2. Activate the venv
1. On Windows: `.\venv\Scripts\activate`
2. On Linux: `source venv/bin/activate`
3. Install the pyproject features based on your system:
7. Install the pyproject features based on your system:
1. Cuda 12.x: `pip install -U .[cu121]`
2. ROCm 5.6: `pip install -U .[amd]`
4. Start the API by either
8. Start the API by either
1. Run `start.bat/sh`. The script will check if you're in a conda environment and skip venv checks.
2. Run `python main.py` to start the API. This won't automatically upgrade your dependencies.
## Download a Model
TabbyAPI includes a built-in Hugging Face downloader that works via both the API and terminal. You can use the following command to download a repository with a specific branch revision:
`.\Start.bat download <repo name> --revision <branch>`
Example with Turboderp's Llama 3.1 8B quants:
`.\Start.bat download turboderp/Qwen2.5-VL-7B-Instruct-exl2 --revision 4.0bpw`
Alternatively, running `main.py` directly can also trigger the downloader. For additional options, run `.\Start.bat download --help`
## Configuration
Loading solely the API may not be your optimal usecase. Therefore, a config.yml exists to tune initial launch parameters and other configuration options.
@ -73,19 +78,19 @@ You can also access the configuration parameters under [2. Configuration](https:
## Where next?
1. Take a look at the [usage docs](https://github.com/theroyallab/tabbyAPI/wiki/03.-Usage)
2. Get started with [community projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects): Find loaders, UIs, and more created by the wider AI community. Any OAI compatible client is also supported.
9. Take a look at the [usage docs](https://github.com/theroyallab/tabbyAPI/wiki/03.-Usage)
10. Get started with [community projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects): Find loaders, UIs, and more created by the wider AI community. Any OAI compatible client is also supported.
## Updating
There are a couple ways to update TabbyAPI:
1. **Update scripts** - Inside the update_scripts folder, you can run the following scripts:
11. **Update scripts** - Inside the update_scripts folder, you can run the following scripts:
1. `update_deps`: Updates dependencies to their latest versions.
2. `update_deps_and_pull`: Updates dependencies and pulls the latest commit of the Github repository.
These scripts exit after running their respective tasks. To start TabbyAPI, run `start.bat` or `start.sh`.
2. **Manual** - Install the pyproject features and update dependencies depending on your GPU:
12. **Manual** - Install the pyproject features and update dependencies depending on your GPU:
1. `pip install -U .[cu121]` = CUDA 12.x
2. `pip install -U .[amd]` = ROCm 6.0
@ -108,11 +113,11 @@ NOTE:
Here are ways to install exllamav2:
1. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended)
13. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended)
1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu121` and `cp311` corresponds to CUDA 12.1 and python 3.11
2. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2`
1. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration.
3. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source)
14. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2`
2. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration.
15. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source)
## Other installation methods
@ -123,21 +128,24 @@ These are short-form instructions for other methods that users can use to instal
### Conda
1. Install [Miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) with python 3.11 as your base python
2. Create a new conda environment `conda create -n tabbyAPI python=3.11`
3. Activate the conda environment `conda activate tabbyAPI`
4. Install optional dependencies if they aren't present
16. Install [Miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) with python 3.11 as your base python
17. Create a new conda environment `conda create -n tabbyAPI python=3.11`
18. Activate the conda environment `conda activate tabbyAPI`
19. Install optional dependencies if they aren't present
1. CUDA via
1. CUDA 12 - `conda install -c "nvidia/label/cuda-12.2.2" cuda`
1. CUDA 12 - `conda install -c "nvidia/label/cuda-12.4.1" cuda`
2. Git via `conda install -k git`
5. Clone TabbyAPI via `git clone https://github.com/theroyallab/tabbyAPI`
6. Continue installation steps from:
20. Clone TabbyAPI via `git clone https://github.com/theroyallab/tabbyAPI`
21. Continue installation steps from:
1. [For Beginners](#for-beginners) - Step 3. The start scripts detect if you're in a conda environment and skips the venv check.
2. [For Advanced Users](#For-advanced-users) - Step 3
### Docker
> [!NOTE]
> If you are planning to use custom versions of dependencies such as dev ExllamaV2, make sure to build the Docker image yourself!
1. Install Docker and docker compose from the [docs](https://docs.docker.com/compose/install/
2. Install the Nvidia container compatibility layer
1. For Linux: [Nvidia container toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)