SyslogSolution/ra-h-os
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align schema and MCP indexing guidance

Browse Source 
- document node and chunk embedding storage
- clarify MCP context and retrieval behavior
- keep unrelated onboarding code edits unstaged

Generated with Claude Code
This commit is contained in:
“BeeRad”
2026-04-20 17:07:24 +10:00
parent 0775b9c4b7
commit c0c896cf8b
5 changed files with 274 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -33,7 +33,7 @@ Current contract:
- node quality comes from `title`, `description`, `source`, `metadata`, and explicit `edges`
- direct node lookup first for specific-node intent
- `getContext` for orientation and `retrieveQueryContext` for broader current-turn grounding
- standalone MCP writes node data, but the app owns chunking and embeddings
- standalone MCP writes node data, but the app owns chunking and embeddings: `nodes.source` becomes readable `chunks`, node-level vectors in `vec_nodes`, and passage vectors in `vec_chunks`
---
docs/0_overview.md
+3 -2
View File
@@ -53,6 +53,7 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a
- **Flexible pane system:** explicit `1 / 2 / 3` pane workspace with right-edge chat
- **Single assistant:** one built-in RA-H runtime grounded in your graph and skills
- **Retrieval split:** direct lookup for specific nodes, broader retrieval when the turn actually needs graph context
- **Indexed retrieval:** source text becomes readable chunks, full-text indexes, and semantic vectors
- **MCP server:** connect Claude Code and other external agents to the same graph
- **Skills:** markdown procedures that shape graph work and agent behavior
- **Extraction tools:** website, YouTube, and PDF ingestion paths
@@ -62,8 +63,8 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a
| Doc | Description |
|-----|-------------|
| [Architecture](./1_architecture.md) | Single-agent runtime, tools, and system design |
| [Schema](./2_schema.md) | Database schema, node/edge structure |
| [Context](./3_context.md) | How context flows through the system |
| [Schema + Search](./2_schema.md) | Database schema, indexing, and retrieval surfaces |
| [Grounding](./3_context.md) | How graph grounding flows through the system |
| [Tools & Skills](./4_tools-and-guides.md) | Available tools, skill system |
| [UI](./6_ui.md) | Component structure, panels, views |
| [Voice](./7_voice.md) | Voice interface (STT/TTS) |
docs/2_schema.md
+252 -23
View File
@@ -1,8 +1,36 @@
# RA-H Schema
# RA-H Schema + Search
RA-H stores the graph in SQLite. SQLite is not just a file dump. It gives RA-H one local structured store for nodes, edges, source chunks, metadata, full-text indexes, and semantic-vector lookup tables.
That matters because an agent can ask the database for likely matches instead of rereading every markdown file from scratch on every turn.
## Core Mental Model
```
User source text
|
v
nodes row
title, description, source, metadata
|
+--> nodes_fts exact node text lookup
+--> vec_nodes whole-node semantic lookup
|
v
chunks rows
readable source slices
|
+--> chunks_fts exact passage lookup
+--> vec_chunks passage-level semantic lookup
```
## Core Tables
### `nodes`
One durable graph artifact: an idea, source, person, project, decision, transcript, note, or other atomic thing.
Important fields:
- `id`
- `title`
- `description`
@@ -11,10 +39,19 @@
- `metadata`
- `chunk_status`
- `event_date`
- `embedding`
- `embedding_updated_at`
- `embedding_text`
- `created_at`
- `updated_at`
`title` names the thing. `description` says what it is and why it belongs. `source` preserves the original or canonical long-form text.
### `edges`
Explicit relationships between nodes.
Important fields:
- `id`
- `from_node_id`
- `to_node_id`
@@ -23,7 +60,13 @@
- `source`
- `created_at`
`edges.explanation` is the human-readable reason the connection exists. `edges.context` is structured JSON metadata, not the main user-facing explanation.
### `chunks`
Readable slices of `nodes.source`.
Shape:
- `id`
- `node_id`
- `chunk_idx`
@@ -32,42 +75,219 @@
- `metadata`
- `created_at`
### `dimension_migration_snapshots`
- Stores one-time snapshots of legacy dimension data before dropping the old tables.
- Exists for auditability and migration verification only.
`chunks.text` is normal prose. A person can read it. For long source material like transcripts, books, articles, and PDFs, chunks let RA-H find the relevant passage instead of handing the whole source to the model.
## Search / Retrieval
### `vec_chunks`
- `nodes_fts` indexes title, description, and source for full-text lookup.
Machine-readable semantic vectors for chunks.
Shape:
- `chunk_id`
- `embedding FLOAT[1536]`
`vec_chunks` is a separate sqlite-vec virtual table. It is table-like, but it is optimized for vector similarity search rather than normal text inspection.
The join point is:
```sql
chunks.id = vec_chunks.chunk_id
```
`chunks.text` is the readable passage. `vec_chunks.embedding` is a long numeric fingerprint of that passage's meaning. The raw vector is not useful prose for a normal user to read.
Concrete live example from the April 20 audit:
- node `5816`: `From IDEs to AI Agents with Steve Yegge`
- chunk `108055`: `chunk_idx = 0`
- chunk text starts with `[0.1s] Tell me about your levels.`
- `vec_chunks` has a matching row where `chunk_id = 108055`
- that row stores a 1536-number embedding for semantic comparison
### `vec_nodes`
Machine-readable semantic vectors for whole nodes.
Shape:
- `node_id`
- `embedding FLOAT[1536]`
The join point is:
```sql
nodes.id = vec_nodes.node_id
```
RA-H also stores node-level embedding data on the node row:
- `nodes.embedding`: the vector BLOB stored on the node
- `nodes.embedding_updated_at`: when the node embedding was last generated
- `nodes.embedding_text`: the readable text RA-H used to make that embedding
Current node-level embedding text includes:
- title
- description
- up to the first 6000 source characters
- a context placeholder, currently usually `none`
- optional AI-generated analysis
Node-level vector search finds likely relevant nodes as whole records. Chunk-level vector search finds likely relevant passages inside long source text.
### `nodes_fts` and `chunks_fts`
FTS means full-text search.
- `nodes_fts` indexes node title, description, and source.
- `chunks_fts` indexes chunk text.
- `vec_nodes` stores node-level vectors.
- `vec_chunks` stores chunk-level vectors.
- Full-text search and vector search are separate surfaces:
- FTS uses `nodes_fts` / `chunks_fts`
- semantic/vector retrieval uses `vec_nodes` / `vec_chunks`
- retrieval can combine them, but they should not be described as the same thing
Full-text search is good when the words matter: names, quotes, exact phrases, product names, error messages, or near-literal recall.
### `dimension_migration_snapshots`
Audit-only historical table for the old dimensions removal.
- It stores one-time snapshots of legacy dimension data before dropping old tables.
- It is not part of the live organizing model.
## Schema Diagram
```
nodes
id
title
description
source
embedding_text
embedding
chunk_status
|
| nodes.id = edges.from_node_id / edges.to_node_id
v
edges
explanation
nodes.id --------------------> vec_nodes.node_id
whole-node semantic vector
nodes.id --------------------> chunks.node_id
readable source slices
|
| chunks.id = vec_chunks.chunk_id
v
vec_chunks
passage-level semantic vector
nodes -----------------------> nodes_fts
title + description + source text index
chunks ----------------------> chunks_fts
passage text index
```
## Ingestion And Indexing Flow
```
Save node
|
+--> store title, description, source, metadata in nodes
|
+--> update nodes_fts
|
+--> build node embedding input
| title + description + source preview + optional analysis
|
+--> write nodes.embedding + nodes.embedding_text
|
+--> write vec_nodes row
|
+--> split source into chunks
|
+--> write readable chunks rows
|
+--> update chunks_fts
|
+--> generate one embedding per chunk
|
+--> write vec_chunks rows
|
v
set chunk_status = chunked
```
Important distinction:
- `chunked` means readable chunk rows exist.
- `vectorized` means matching vector rows also exist in `vec_chunks`.
`chunk_status = 'chunked'` should not be documented as proof that every chunk has a matching vector row unless vector coverage has been checked.
## Search Methods
```
User asks something
|
+--> specific known node?
| use direct node lookup / queryNodes
|
+--> exact words, names, quotes, errors?
| use FTS over nodes_fts or chunks_fts
|
+--> similar meaning with different words?
| use vector search over vec_nodes or vec_chunks
|
+--> source passage question?
| search chunks, then ground answer in matching passages
|
v
combine the strongest evidence with graph neighbors when useful
```
RA-H uses more than one search path because the paths answer different questions.
Full-text search asks: "where do these words appear?"
Semantic search asks: "what has similar meaning even if the words differ?"
Graph traversal asks: "what is connected to this thing, and why?"
## Why SQLite Beats A Loose Markdown Folder For Retrieval
A markdown folder can be a useful source or export layer. It is not the same thing as an indexed local graph.
SQLite gives RA-H:
- one structured local store for nodes, edges, chunks, metadata, and indexes
- fast indexed lookup instead of scanning every file every time
- FTS for literal text and quote-like recall
- embeddings for conceptually similar retrieval
- edges for explicit graph context after retrieval
- integrity checks and repair surfaces for operational confidence
The practical difference: an agent can retrieve a small, relevant slice of the graph quickly, then reason over that slice. With loose markdown, the agent either guesses which files to read or spends tokens scanning too much raw text.
## Embedding Lifecycle
- `nodes.source` is the canonical long-form field for chunking and chunk embeddings.
- Creating or changing `nodes.source` must put the node back through the app-owned chunk pipeline so the `chunks` rows, `chunks_fts`, and `vec_chunks` state reflect the latest source.
- Standalone MCP can write `nodes.source`, but it does not directly create `chunks` or vector rows. The app later processes those pending nodes.
- Deleting a node must remove dependent chunk rows and must not leave stale node/chunk search or vector state behind.
- Node-level embeddings are a separate surface from chunk embeddings. The contract for what feeds the node-level embedding must be explicit, and updates to those fields must trigger a fresh node-level embedding run.
- Integrity and degraded-mode checks must cover both search surfaces and embedding-related write surfaces, not just top-level node reads.
- Creating or changing `nodes.source` puts the node back through the app-owned chunk pipeline.
- The app pipeline creates or refreshes `chunks`, `chunks_fts`, and `vec_chunks`.
- Standalone MCP can write `nodes.source`, but it does not directly create chunks or vector rows. The app later processes pending nodes.
- Node-level embeddings are separate from chunk embeddings and write to `nodes.embedding`, `nodes.embedding_text`, and `vec_nodes`.
- Deleting a node must remove dependent chunk rows and stale search/vector state.
## Edge Contract
## Current Live Coverage Caveat
- `edges.explanation` is now a top-level field and should be treated as the human-readable reason the connection exists.
- `edges.context` still exists as structured JSON for inferred type, confidence, and creation metadata.
- docs should not describe edge context JSON as if it is the only user-facing explanation surface.
The April 20 audit found current ingestion healthy for recent nodes, but historical vector-table coverage incomplete.
Current status to document carefully:
- recent nodes since `2026-04-16` that have chunks are represented in `vec_chunks`
- older rows can have chunks without matching `vec_chunks`
- some older nodes can have `nodes.embedding` without being present in `vec_nodes`
- successful transcript retrieval may come from vector search, FTS, or text fallback
Historical universal vector coverage is a separate backfill/repair task, not something this docs page should imply is already complete.
## Important Constraints
- `dimensions` and `node_dimensions` are no longer canonical tables.
- New installs should never create them.
- Existing installs migrate by snapshotting old dimension data, then dropping the legacy tables.
- FTS repair and integrity handling are now operational concerns. Do not describe automatic live rebuild behavior as normal product behavior.
- Existing installs migrate by snapshotting old dimension data, then dropping legacy tables.
- The removed `contexts` table and old context capsule are not live product doctrine.
- FTS repair and integrity handling are operational concerns. Do not describe automatic live rebuild behavior as normal product behavior.
## Common Queries
@@ -90,3 +310,12 @@ FROM nodes
ORDER BY updated_at DESC
LIMIT 25;
```
Check chunk/vector coverage:
```sql
SELECT COUNT(*) AS missing_vectors
FROM chunks c
LEFT JOIN vec_chunks v ON v.chunk_id = c.id
WHERE v.chunk_id IS NULL;
```
docs/8_mcp.md
+16 -1
View File
@@ -17,6 +17,7 @@ Important runtime distinction:
- `createEdge` is a post-confirmation execution tool. Agents should propose likely edges first and only write them after the user explicitly confirms.
- `queryNodes` searches title, description, and source.
- `dimensions` are removed from the MCP contract.
- `contexts` and context-capsule tools are removed from the MCP contract.
## Behavior Split
@@ -64,7 +65,21 @@ Write:
- Keep writeback prompts terse and selective. The goal is not to ask constantly whether every useful sentence should be saved.
- Do not assume MCP node creation immediately produces chunks or embeddings. The canonical contract is:
- write node data first
- app-owned pipeline later creates chunks, FTS rows, and vectors
- app-owned pipeline later creates readable chunks, FTS rows, `vec_nodes`, and `vec_chunks`
- Do not use `chunk_status = 'chunked'` as a promise that every chunk has a row in `vec_chunks`. Chunking and vector coverage are related but distinct.
## Search And Indexing Notes
MCP users should understand the same retrieval split as the app:
- direct node lookup searches node title, description, and source
- full-text search can match literal words in nodes or chunks
- chunk search can find passages inside long source text
- `vec_nodes` can find semantically similar whole nodes when node-level vectors exist
- `vec_chunks` can find semantically similar passages when chunk-level vectors exist
- standalone MCP does not generate embeddings itself
If an external agent creates or updates a node through standalone MCP while the app is closed, the node can exist before its chunks and vectors do. The app-owned pipeline processes that later.
## Memory-File Rule
docs/README.md
+2 -2
View File
@@ -14,7 +14,7 @@
| Doc | Description |
|-----|-------------|
| [Overview](./0_overview.md) | What RA-H OS is and what contract it shares with the main app |
| [Schema](./2_schema.md) | Current SQLite contract |
| [Schema + Search](./2_schema.md) | Current SQLite contract, indexing, and retrieval surfaces |
| [Tools & Skills](./4_tools-and-guides.md) | MCP tools and skill system |
| [Logging & Evals](./5_logging-and-evals.md) | Logs, evals, and debugging surfaces |
| [UI](./6_ui.md) | Current pane and focus model |
@@ -60,7 +60,7 @@ Add to your `~/.claude.json`:
If you publish a newer MCP release and need clients to use it immediately, bump the pinned version here and restart the client. Do not assume plain `npx ra-h-mcp-server` always refreshes instantly.
Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path.
Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source: readable `chunks`, full-text indexes, `vec_nodes`, and `vec_chunks`. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path.
## Questions?