Add local AI and Qdrant vector backends
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Local Models
|
||||
|
||||
RA-H does not run model weights directly. You run a local OpenAI-compatible model server, then RA-H calls that server over HTTP.
|
||||
|
||||
The supported local profile is intentionally narrow:
|
||||
|
||||
- Utility LLM: Qwen3 4B or the closest tested Qwen3 4B runtime equivalent
|
||||
- Embeddings: Qwen3 Embedding 0.6B
|
||||
- Embedding dimensions: `1024`
|
||||
- Runtime options behind the same contract: Ollama or llama.cpp
|
||||
|
||||
OpenAI remains the default supported path. Local mode is for users who are comfortable installing a model runtime, pulling model weights, starting local server processes, and managing rebuilds when embedding settings change.
|
||||
|
||||
## Core Env
|
||||
|
||||
```bash
|
||||
LLM_PROFILE=openai-compatible
|
||||
LLM_BASE_URL=http://127.0.0.1:11434/v1
|
||||
LLM_MODEL=qwen3:4b
|
||||
|
||||
EMBEDDING_PROFILE=openai-compatible
|
||||
EMBEDDING_BASE_URL=http://127.0.0.1:11434/v1
|
||||
EMBEDDING_MODEL=qwen3-embedding:0.6b
|
||||
EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run doctor:local-ai
|
||||
```
|
||||
|
||||
If you change embedding provider, model, dimensions, or vector backend after data exists, run:
|
||||
|
||||
```bash
|
||||
npm run rebuild:embeddings
|
||||
```
|
||||
|
||||
## Vector Storage
|
||||
|
||||
Qwen3 creates vectors. sqlite-vec or Qdrant stores and searches those vectors.
|
||||
|
||||
You do not need Qdrant just because models are local. Use Qdrant when sqlite-vec is unavailable or unreliable on your platform, especially Alpine/musl Docker images, Windows ARM64, or other native-extension-hostile environments.
|
||||
|
||||
SQLite remains the source-of-truth database. Qdrant stores only derived vector indexes.
|
||||
Reference in New Issue
Block a user