ra-h-os/LOCAL-MODELS.md

Local Models

RA-H does not run model weights directly. You run a local OpenAI-compatible model server, then RA-H calls that server over HTTP.

The supported local profile is intentionally narrow:

• Utility LLM: Qwen3 4B or the closest tested Qwen3 4B runtime equivalent
• Embeddings: Qwen3 Embedding 0.6B
• Embedding dimensions: 1024
• Runtime options behind the same contract: Ollama or llama.cpp

OpenAI remains the default supported path. Local mode is for users who are comfortable installing a model runtime, pulling model weights, starting local server processes, and managing rebuilds when embedding settings change.

Core Env

Ollama:

LLM_PROFILE=openai-compatible
LLM_BASE_URL=http://127.0.0.1:11434/v1
LLM_MODEL=qwen3:4b

EMBEDDING_PROFILE=openai-compatible
EMBEDDING_BASE_URL=http://127.0.0.1:11434/v1
EMBEDDING_MODEL=qwen3-embedding:0.6b
EMBEDDING_DIMENSIONS=1024

llama.cpp:

LLM_PROFILE=openai-compatible
LLM_BASE_URL=http://127.0.0.1:8080/v1
LLM_MODEL=qwen3-4b

EMBEDDING_PROFILE=openai-compatible
EMBEDDING_BASE_URL=http://127.0.0.1:8081/v1
EMBEDDING_MODEL=qwen3-embedding-0.6b
EMBEDDING_DIMENSIONS=1024

Run:

npm run doctor:local-ai

If you change embedding provider, model, dimensions, or vector backend after data exists, run:

npm run rebuild:embeddings

Vector Storage

Qwen3 creates vectors. sqlite-vec or Qdrant stores and searches those vectors.

You do not need Qdrant just because models are local. Use Qdrant when sqlite-vec is unavailable or unreliable on your platform, especially Alpine/musl Docker images, Windows ARM64, or other native-extension-hostile environments.

SQLite remains the source-of-truth database. Qdrant stores only derived vector indexes.