From c0c896cf8ba33c58453b463d731aa0d0eedebcbc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CBeeRad=E2=80=9D?= Date: Mon, 20 Apr 2026 17:07:24 +1000 Subject: [PATCH] docs: align schema and MCP indexing guidance - document node and chunk embedding storage - clarify MCP context and retrieval behavior - keep unrelated onboarding code edits unstaged Generated with Claude Code --- README.md | 2 +- docs/0_overview.md | 5 +- docs/2_schema.md | 275 +++++++++++++++++++++++++++++++++++++++++---- docs/8_mcp.md | 17 ++- docs/README.md | 4 +- 5 files changed, 274 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index f51f8a6..00c30e0 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Current contract: - node quality comes from `title`, `description`, `source`, `metadata`, and explicit `edges` - direct node lookup first for specific-node intent - `getContext` for orientation and `retrieveQueryContext` for broader current-turn grounding -- standalone MCP writes node data, but the app owns chunking and embeddings +- standalone MCP writes node data, but the app owns chunking and embeddings: `nodes.source` becomes readable `chunks`, node-level vectors in `vec_nodes`, and passage vectors in `vec_chunks` --- diff --git a/docs/0_overview.md b/docs/0_overview.md index 2c3f30b..f1ab38d 100644 --- a/docs/0_overview.md +++ b/docs/0_overview.md @@ -53,6 +53,7 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a - **Flexible pane system:** explicit `1 / 2 / 3` pane workspace with right-edge chat - **Single assistant:** one built-in RA-H runtime grounded in your graph and skills - **Retrieval split:** direct lookup for specific nodes, broader retrieval when the turn actually needs graph context +- **Indexed retrieval:** source text becomes readable chunks, full-text indexes, and semantic vectors - **MCP server:** connect Claude Code and other external agents to the same graph - **Skills:** markdown procedures that shape graph work and agent behavior - **Extraction tools:** website, YouTube, and PDF ingestion paths @@ -62,8 +63,8 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a | Doc | Description | |-----|-------------| | [Architecture](./1_architecture.md) | Single-agent runtime, tools, and system design | -| [Schema](./2_schema.md) | Database schema, node/edge structure | -| [Context](./3_context.md) | How context flows through the system | +| [Schema + Search](./2_schema.md) | Database schema, indexing, and retrieval surfaces | +| [Grounding](./3_context.md) | How graph grounding flows through the system | | [Tools & Skills](./4_tools-and-guides.md) | Available tools, skill system | | [UI](./6_ui.md) | Component structure, panels, views | | [Voice](./7_voice.md) | Voice interface (STT/TTS) | diff --git a/docs/2_schema.md b/docs/2_schema.md index eb8a0dc..0621435 100644 --- a/docs/2_schema.md +++ b/docs/2_schema.md @@ -1,8 +1,36 @@ -# RA-H Schema +# RA-H Schema + Search + +RA-H stores the graph in SQLite. SQLite is not just a file dump. It gives RA-H one local structured store for nodes, edges, source chunks, metadata, full-text indexes, and semantic-vector lookup tables. + +That matters because an agent can ask the database for likely matches instead of rereading every markdown file from scratch on every turn. + +## Core Mental Model + +``` +User source text + | + v +nodes row + title, description, source, metadata + | + +--> nodes_fts exact node text lookup + +--> vec_nodes whole-node semantic lookup + | + v +chunks rows + readable source slices + | + +--> chunks_fts exact passage lookup + +--> vec_chunks passage-level semantic lookup +``` ## Core Tables ### `nodes` + +One durable graph artifact: an idea, source, person, project, decision, transcript, note, or other atomic thing. + +Important fields: - `id` - `title` - `description` @@ -11,10 +39,19 @@ - `metadata` - `chunk_status` - `event_date` +- `embedding` +- `embedding_updated_at` +- `embedding_text` - `created_at` - `updated_at` +`title` names the thing. `description` says what it is and why it belongs. `source` preserves the original or canonical long-form text. + ### `edges` + +Explicit relationships between nodes. + +Important fields: - `id` - `from_node_id` - `to_node_id` @@ -23,7 +60,13 @@ - `source` - `created_at` +`edges.explanation` is the human-readable reason the connection exists. `edges.context` is structured JSON metadata, not the main user-facing explanation. + ### `chunks` + +Readable slices of `nodes.source`. + +Shape: - `id` - `node_id` - `chunk_idx` @@ -32,42 +75,219 @@ - `metadata` - `created_at` -### `dimension_migration_snapshots` -- Stores one-time snapshots of legacy dimension data before dropping the old tables. -- Exists for auditability and migration verification only. +`chunks.text` is normal prose. A person can read it. For long source material like transcripts, books, articles, and PDFs, chunks let RA-H find the relevant passage instead of handing the whole source to the model. -## Search / Retrieval +### `vec_chunks` -- `nodes_fts` indexes title, description, and source for full-text lookup. +Machine-readable semantic vectors for chunks. + +Shape: +- `chunk_id` +- `embedding FLOAT[1536]` + +`vec_chunks` is a separate sqlite-vec virtual table. It is table-like, but it is optimized for vector similarity search rather than normal text inspection. + +The join point is: + +```sql +chunks.id = vec_chunks.chunk_id +``` + +`chunks.text` is the readable passage. `vec_chunks.embedding` is a long numeric fingerprint of that passage's meaning. The raw vector is not useful prose for a normal user to read. + +Concrete live example from the April 20 audit: +- node `5816`: `From IDEs to AI Agents with Steve Yegge` +- chunk `108055`: `chunk_idx = 0` +- chunk text starts with `[0.1s] Tell me about your levels.` +- `vec_chunks` has a matching row where `chunk_id = 108055` +- that row stores a 1536-number embedding for semantic comparison + +### `vec_nodes` + +Machine-readable semantic vectors for whole nodes. + +Shape: +- `node_id` +- `embedding FLOAT[1536]` + +The join point is: + +```sql +nodes.id = vec_nodes.node_id +``` + +RA-H also stores node-level embedding data on the node row: +- `nodes.embedding`: the vector BLOB stored on the node +- `nodes.embedding_updated_at`: when the node embedding was last generated +- `nodes.embedding_text`: the readable text RA-H used to make that embedding + +Current node-level embedding text includes: +- title +- description +- up to the first 6000 source characters +- a context placeholder, currently usually `none` +- optional AI-generated analysis + +Node-level vector search finds likely relevant nodes as whole records. Chunk-level vector search finds likely relevant passages inside long source text. + +### `nodes_fts` and `chunks_fts` + +FTS means full-text search. + +- `nodes_fts` indexes node title, description, and source. - `chunks_fts` indexes chunk text. -- `vec_nodes` stores node-level vectors. -- `vec_chunks` stores chunk-level vectors. -- Full-text search and vector search are separate surfaces: - - FTS uses `nodes_fts` / `chunks_fts` - - semantic/vector retrieval uses `vec_nodes` / `vec_chunks` - - retrieval can combine them, but they should not be described as the same thing + +Full-text search is good when the words matter: names, quotes, exact phrases, product names, error messages, or near-literal recall. + +### `dimension_migration_snapshots` + +Audit-only historical table for the old dimensions removal. + +- It stores one-time snapshots of legacy dimension data before dropping old tables. +- It is not part of the live organizing model. + +## Schema Diagram + +``` +nodes + id + title + description + source + embedding_text + embedding + chunk_status + | + | nodes.id = edges.from_node_id / edges.to_node_id + v +edges + explanation + +nodes.id --------------------> vec_nodes.node_id + whole-node semantic vector + +nodes.id --------------------> chunks.node_id + readable source slices + | + | chunks.id = vec_chunks.chunk_id + v + vec_chunks + passage-level semantic vector + +nodes -----------------------> nodes_fts + title + description + source text index + +chunks ----------------------> chunks_fts + passage text index +``` + +## Ingestion And Indexing Flow + +``` +Save node + | + +--> store title, description, source, metadata in nodes + | + +--> update nodes_fts + | + +--> build node embedding input + | title + description + source preview + optional analysis + | + +--> write nodes.embedding + nodes.embedding_text + | + +--> write vec_nodes row + | + +--> split source into chunks + | + +--> write readable chunks rows + | + +--> update chunks_fts + | + +--> generate one embedding per chunk + | + +--> write vec_chunks rows + | + v +set chunk_status = chunked +``` + +Important distinction: +- `chunked` means readable chunk rows exist. +- `vectorized` means matching vector rows also exist in `vec_chunks`. + +`chunk_status = 'chunked'` should not be documented as proof that every chunk has a matching vector row unless vector coverage has been checked. + +## Search Methods + +``` +User asks something + | + +--> specific known node? + | use direct node lookup / queryNodes + | + +--> exact words, names, quotes, errors? + | use FTS over nodes_fts or chunks_fts + | + +--> similar meaning with different words? + | use vector search over vec_nodes or vec_chunks + | + +--> source passage question? + | search chunks, then ground answer in matching passages + | + v +combine the strongest evidence with graph neighbors when useful +``` + +RA-H uses more than one search path because the paths answer different questions. + +Full-text search asks: "where do these words appear?" + +Semantic search asks: "what has similar meaning even if the words differ?" + +Graph traversal asks: "what is connected to this thing, and why?" + +## Why SQLite Beats A Loose Markdown Folder For Retrieval + +A markdown folder can be a useful source or export layer. It is not the same thing as an indexed local graph. + +SQLite gives RA-H: +- one structured local store for nodes, edges, chunks, metadata, and indexes +- fast indexed lookup instead of scanning every file every time +- FTS for literal text and quote-like recall +- embeddings for conceptually similar retrieval +- edges for explicit graph context after retrieval +- integrity checks and repair surfaces for operational confidence + +The practical difference: an agent can retrieve a small, relevant slice of the graph quickly, then reason over that slice. With loose markdown, the agent either guesses which files to read or spends tokens scanning too much raw text. ## Embedding Lifecycle - `nodes.source` is the canonical long-form field for chunking and chunk embeddings. -- Creating or changing `nodes.source` must put the node back through the app-owned chunk pipeline so the `chunks` rows, `chunks_fts`, and `vec_chunks` state reflect the latest source. -- Standalone MCP can write `nodes.source`, but it does not directly create `chunks` or vector rows. The app later processes those pending nodes. -- Deleting a node must remove dependent chunk rows and must not leave stale node/chunk search or vector state behind. -- Node-level embeddings are a separate surface from chunk embeddings. The contract for what feeds the node-level embedding must be explicit, and updates to those fields must trigger a fresh node-level embedding run. -- Integrity and degraded-mode checks must cover both search surfaces and embedding-related write surfaces, not just top-level node reads. +- Creating or changing `nodes.source` puts the node back through the app-owned chunk pipeline. +- The app pipeline creates or refreshes `chunks`, `chunks_fts`, and `vec_chunks`. +- Standalone MCP can write `nodes.source`, but it does not directly create chunks or vector rows. The app later processes pending nodes. +- Node-level embeddings are separate from chunk embeddings and write to `nodes.embedding`, `nodes.embedding_text`, and `vec_nodes`. +- Deleting a node must remove dependent chunk rows and stale search/vector state. -## Edge Contract +## Current Live Coverage Caveat -- `edges.explanation` is now a top-level field and should be treated as the human-readable reason the connection exists. -- `edges.context` still exists as structured JSON for inferred type, confidence, and creation metadata. -- docs should not describe edge context JSON as if it is the only user-facing explanation surface. +The April 20 audit found current ingestion healthy for recent nodes, but historical vector-table coverage incomplete. + +Current status to document carefully: +- recent nodes since `2026-04-16` that have chunks are represented in `vec_chunks` +- older rows can have chunks without matching `vec_chunks` +- some older nodes can have `nodes.embedding` without being present in `vec_nodes` +- successful transcript retrieval may come from vector search, FTS, or text fallback + +Historical universal vector coverage is a separate backfill/repair task, not something this docs page should imply is already complete. ## Important Constraints - `dimensions` and `node_dimensions` are no longer canonical tables. - New installs should never create them. -- Existing installs migrate by snapshotting old dimension data, then dropping the legacy tables. -- FTS repair and integrity handling are now operational concerns. Do not describe automatic live rebuild behavior as normal product behavior. +- Existing installs migrate by snapshotting old dimension data, then dropping legacy tables. +- The removed `contexts` table and old context capsule are not live product doctrine. +- FTS repair and integrity handling are operational concerns. Do not describe automatic live rebuild behavior as normal product behavior. ## Common Queries @@ -90,3 +310,12 @@ FROM nodes ORDER BY updated_at DESC LIMIT 25; ``` + +Check chunk/vector coverage: + +```sql +SELECT COUNT(*) AS missing_vectors +FROM chunks c +LEFT JOIN vec_chunks v ON v.chunk_id = c.id +WHERE v.chunk_id IS NULL; +``` diff --git a/docs/8_mcp.md b/docs/8_mcp.md index edb1da9..f0299b3 100644 --- a/docs/8_mcp.md +++ b/docs/8_mcp.md @@ -17,6 +17,7 @@ Important runtime distinction: - `createEdge` is a post-confirmation execution tool. Agents should propose likely edges first and only write them after the user explicitly confirms. - `queryNodes` searches title, description, and source. - `dimensions` are removed from the MCP contract. +- `contexts` and context-capsule tools are removed from the MCP contract. ## Behavior Split @@ -64,7 +65,21 @@ Write: - Keep writeback prompts terse and selective. The goal is not to ask constantly whether every useful sentence should be saved. - Do not assume MCP node creation immediately produces chunks or embeddings. The canonical contract is: - write node data first - - app-owned pipeline later creates chunks, FTS rows, and vectors + - app-owned pipeline later creates readable chunks, FTS rows, `vec_nodes`, and `vec_chunks` +- Do not use `chunk_status = 'chunked'` as a promise that every chunk has a row in `vec_chunks`. Chunking and vector coverage are related but distinct. + +## Search And Indexing Notes + +MCP users should understand the same retrieval split as the app: + +- direct node lookup searches node title, description, and source +- full-text search can match literal words in nodes or chunks +- chunk search can find passages inside long source text +- `vec_nodes` can find semantically similar whole nodes when node-level vectors exist +- `vec_chunks` can find semantically similar passages when chunk-level vectors exist +- standalone MCP does not generate embeddings itself + +If an external agent creates or updates a node through standalone MCP while the app is closed, the node can exist before its chunks and vectors do. The app-owned pipeline processes that later. ## Memory-File Rule diff --git a/docs/README.md b/docs/README.md index d317d07..07cfd9f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,7 +14,7 @@ | Doc | Description | |-----|-------------| | [Overview](./0_overview.md) | What RA-H OS is and what contract it shares with the main app | -| [Schema](./2_schema.md) | Current SQLite contract | +| [Schema + Search](./2_schema.md) | Current SQLite contract, indexing, and retrieval surfaces | | [Tools & Skills](./4_tools-and-guides.md) | MCP tools and skill system | | [Logging & Evals](./5_logging-and-evals.md) | Logs, evals, and debugging surfaces | | [UI](./6_ui.md) | Current pane and focus model | @@ -60,7 +60,7 @@ Add to your `~/.claude.json`: If you publish a newer MCP release and need clients to use it immediately, bump the pinned version here and restart the client. Do not assume plain `npx ra-h-mcp-server` always refreshes instantly. -Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path. +Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source: readable `chunks`, full-text indexes, `vec_nodes`, and `vec_chunks`. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path. ## Questions?