docs: align schema and MCP indexing guidance

- document node and chunk embedding storage
- clarify MCP context and retrieval behavior
- keep unrelated onboarding code edits unstaged

Generated with Claude Code

README.md

@@ -33,7 +33,7 @@ Current contract:
 - node quality comes from `title`, `description`, `source`, `metadata`, and explicit `edges`
 - direct node lookup first for specific-node intent
 - `getContext` for orientation and `retrieveQueryContext` for broader current-turn grounding
-- standalone MCP writes node data, but the app owns chunking and embeddings
+- standalone MCP writes node data, but the app owns chunking and embeddings: `nodes.source` becomes readable `chunks`, node-level vectors in `vec_nodes`, and passage vectors in `vec_chunks`
 
 ---
 

docs/0_overview.md

@@ -53,6 +53,7 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a
 - **Flexible pane system:** explicit `1 / 2 / 3` pane workspace with right-edge chat
 - **Single assistant:** one built-in RA-H runtime grounded in your graph and skills
 - **Retrieval split:** direct lookup for specific nodes, broader retrieval when the turn actually needs graph context
+- **Indexed retrieval:** source text becomes readable chunks, full-text indexes, and semantic vectors
 - **MCP server:** connect Claude Code and other external agents to the same graph
 - **Skills:** markdown procedures that shape graph work and agent behavior
 - **Extraction tools:** website, YouTube, and PDF ingestion paths
@@ -62,8 +63,8 @@ Both versions follow the same core graph contract. The Mac app adds packaging, a
 | Doc | Description |
 |-----|-------------|
 | [Architecture](./1_architecture.md) | Single-agent runtime, tools, and system design |
-| [Schema](./2_schema.md) | Database schema, node/edge structure |
+| [Schema + Search](./2_schema.md) | Database schema, indexing, and retrieval surfaces |
-| [Context](./3_context.md) | How context flows through the system |
+| [Grounding](./3_context.md) | How graph grounding flows through the system |
 | [Tools & Skills](./4_tools-and-guides.md) | Available tools, skill system |
 | [UI](./6_ui.md) | Component structure, panels, views |
 | [Voice](./7_voice.md) | Voice interface (STT/TTS) |

docs/2_schema.md

@@ -1,8 +1,36 @@
-# RA-H Schema
+# RA-H Schema + Search
+
+RA-H stores the graph in SQLite. SQLite is not just a file dump. It gives RA-H one local structured store for nodes, edges, source chunks, metadata, full-text indexes, and semantic-vector lookup tables.
+
+That matters because an agent can ask the database for likely matches instead of rereading every markdown file from scratch on every turn.
+
+## Core Mental Model
+
+```
+User source text
+      |
+      v
+nodes row
+  title, description, source, metadata
+      |
+      +--> nodes_fts      exact node text lookup
+      +--> vec_nodes      whole-node semantic lookup
+      |
+      v
+chunks rows
+  readable source slices
+      |
+      +--> chunks_fts     exact passage lookup
+      +--> vec_chunks     passage-level semantic lookup
+```
 
 ## Core Tables
 
 ### `nodes`
+
+One durable graph artifact: an idea, source, person, project, decision, transcript, note, or other atomic thing.
+
+Important fields:
 - `id`
 - `title`
 - `description`
@@ -11,10 +39,19 @@
 - `metadata`
 - `chunk_status`
 - `event_date`
+- `embedding`
+- `embedding_updated_at`
+- `embedding_text`
 - `created_at`
 - `updated_at`
 
+`title` names the thing. `description` says what it is and why it belongs. `source` preserves the original or canonical long-form text.
+
 ### `edges`
+
+Explicit relationships between nodes.
+
+Important fields:
 - `id`
 - `from_node_id`
 - `to_node_id`
@@ -23,7 +60,13 @@
 - `source`
 - `created_at`
 
+`edges.explanation` is the human-readable reason the connection exists. `edges.context` is structured JSON metadata, not the main user-facing explanation.
+
 ### `chunks`
+
+Readable slices of `nodes.source`.
+
+Shape:
 - `id`
 - `node_id`
 - `chunk_idx`
@@ -32,42 +75,219 @@
 - `metadata`
 - `created_at`
 
-### `dimension_migration_snapshots`
+`chunks.text` is normal prose. A person can read it. For long source material like transcripts, books, articles, and PDFs, chunks let RA-H find the relevant passage instead of handing the whole source to the model.
-- Stores one-time snapshots of legacy dimension data before dropping the old tables.
-- Exists for auditability and migration verification only.
 
-## Search / Retrieval
+### `vec_chunks`
 
-- `nodes_fts` indexes title, description, and source for full-text lookup.
+Machine-readable semantic vectors for chunks.
+
+Shape:
+- `chunk_id`
+- `embedding FLOAT[1536]`
+
+`vec_chunks` is a separate sqlite-vec virtual table. It is table-like, but it is optimized for vector similarity search rather than normal text inspection.
+
+The join point is:
+
+```sql
+chunks.id = vec_chunks.chunk_id
+```
+
+`chunks.text` is the readable passage. `vec_chunks.embedding` is a long numeric fingerprint of that passage's meaning. The raw vector is not useful prose for a normal user to read.
+
+Concrete live example from the April 20 audit:
+- node `5816`: `From IDEs to AI Agents with Steve Yegge`
+- chunk `108055`: `chunk_idx = 0`
+- chunk text starts with `[0.1s] Tell me about your levels.`
+- `vec_chunks` has a matching row where `chunk_id = 108055`
+- that row stores a 1536-number embedding for semantic comparison
+
+### `vec_nodes`
+
+Machine-readable semantic vectors for whole nodes.
+
+Shape:
+- `node_id`
+- `embedding FLOAT[1536]`
+
+The join point is:
+
+```sql
+nodes.id = vec_nodes.node_id
+```
+
+RA-H also stores node-level embedding data on the node row:
+- `nodes.embedding`: the vector BLOB stored on the node
+- `nodes.embedding_updated_at`: when the node embedding was last generated
+- `nodes.embedding_text`: the readable text RA-H used to make that embedding
+
+Current node-level embedding text includes:
+- title
+- description
+- up to the first 6000 source characters
+- a context placeholder, currently usually `none`
+- optional AI-generated analysis
+
+Node-level vector search finds likely relevant nodes as whole records. Chunk-level vector search finds likely relevant passages inside long source text.
+
+### `nodes_fts` and `chunks_fts`
+
+FTS means full-text search.
+
+- `nodes_fts` indexes node title, description, and source.
 - `chunks_fts` indexes chunk text.
-- `vec_nodes` stores node-level vectors.
+
-- `vec_chunks` stores chunk-level vectors.
+Full-text search is good when the words matter: names, quotes, exact phrases, product names, error messages, or near-literal recall.
-- Full-text search and vector search are separate surfaces:
+
-  - FTS uses `nodes_fts` / `chunks_fts`
+### `dimension_migration_snapshots`
-  - semantic/vector retrieval uses `vec_nodes` / `vec_chunks`
+
-  - retrieval can combine them, but they should not be described as the same thing
+Audit-only historical table for the old dimensions removal.
+
+- It stores one-time snapshots of legacy dimension data before dropping old tables.
+- It is not part of the live organizing model.
+
+## Schema Diagram
+
+```
+nodes
+  id
+  title
+  description
+  source
+  embedding_text
+  embedding
+  chunk_status
+   |
+   | nodes.id = edges.from_node_id / edges.to_node_id
+   v
+edges
+  explanation
+
+nodes.id --------------------> vec_nodes.node_id
+                                 whole-node semantic vector
+
+nodes.id --------------------> chunks.node_id
+                                 readable source slices
+                                  |
+                                  | chunks.id = vec_chunks.chunk_id
+                                  v
+                                vec_chunks
+                                  passage-level semantic vector
+
+nodes -----------------------> nodes_fts
+                                 title + description + source text index
+
+chunks ----------------------> chunks_fts
+                                 passage text index
+```
+
+## Ingestion And Indexing Flow
+
+```
+Save node
+  |
+  +--> store title, description, source, metadata in nodes
+  |
+  +--> update nodes_fts
+  |
+  +--> build node embedding input
+  |       title + description + source preview + optional analysis
+  |
+  +--> write nodes.embedding + nodes.embedding_text
+  |
+  +--> write vec_nodes row
+  |
+  +--> split source into chunks
+  |
+  +--> write readable chunks rows
+  |
+  +--> update chunks_fts
+  |
+  +--> generate one embedding per chunk
+  |
+  +--> write vec_chunks rows
+  |
+  v
+set chunk_status = chunked
+```
+
+Important distinction:
+- `chunked` means readable chunk rows exist.
+- `vectorized` means matching vector rows also exist in `vec_chunks`.
+
+`chunk_status = 'chunked'` should not be documented as proof that every chunk has a matching vector row unless vector coverage has been checked.
+
+## Search Methods
+
+```
+User asks something
+  |
+  +--> specific known node?
+  |       use direct node lookup / queryNodes
+  |
+  +--> exact words, names, quotes, errors?
+  |       use FTS over nodes_fts or chunks_fts
+  |
+  +--> similar meaning with different words?
+  |       use vector search over vec_nodes or vec_chunks
+  |
+  +--> source passage question?
+  |       search chunks, then ground answer in matching passages
+  |
+  v
+combine the strongest evidence with graph neighbors when useful
+```
+
+RA-H uses more than one search path because the paths answer different questions.
+
+Full-text search asks: "where do these words appear?"
+
+Semantic search asks: "what has similar meaning even if the words differ?"
+
+Graph traversal asks: "what is connected to this thing, and why?"
+
+## Why SQLite Beats A Loose Markdown Folder For Retrieval
+
+A markdown folder can be a useful source or export layer. It is not the same thing as an indexed local graph.
+
+SQLite gives RA-H:
+- one structured local store for nodes, edges, chunks, metadata, and indexes
+- fast indexed lookup instead of scanning every file every time
+- FTS for literal text and quote-like recall
+- embeddings for conceptually similar retrieval
+- edges for explicit graph context after retrieval
+- integrity checks and repair surfaces for operational confidence
+
+The practical difference: an agent can retrieve a small, relevant slice of the graph quickly, then reason over that slice. With loose markdown, the agent either guesses which files to read or spends tokens scanning too much raw text.
 
 ## Embedding Lifecycle
 
 - `nodes.source` is the canonical long-form field for chunking and chunk embeddings.
-- Creating or changing `nodes.source` must put the node back through the app-owned chunk pipeline so the `chunks` rows, `chunks_fts`, and `vec_chunks` state reflect the latest source.
+- Creating or changing `nodes.source` puts the node back through the app-owned chunk pipeline.
-- Standalone MCP can write `nodes.source`, but it does not directly create `chunks` or vector rows. The app later processes those pending nodes.
+- The app pipeline creates or refreshes `chunks`, `chunks_fts`, and `vec_chunks`.
-- Deleting a node must remove dependent chunk rows and must not leave stale node/chunk search or vector state behind.
+- Standalone MCP can write `nodes.source`, but it does not directly create chunks or vector rows. The app later processes pending nodes.
-- Node-level embeddings are a separate surface from chunk embeddings. The contract for what feeds the node-level embedding must be explicit, and updates to those fields must trigger a fresh node-level embedding run.
+- Node-level embeddings are separate from chunk embeddings and write to `nodes.embedding`, `nodes.embedding_text`, and `vec_nodes`.
-- Integrity and degraded-mode checks must cover both search surfaces and embedding-related write surfaces, not just top-level node reads.
+- Deleting a node must remove dependent chunk rows and stale search/vector state.
 
-## Edge Contract
+## Current Live Coverage Caveat
 
-- `edges.explanation` is now a top-level field and should be treated as the human-readable reason the connection exists.
+The April 20 audit found current ingestion healthy for recent nodes, but historical vector-table coverage incomplete.
-- `edges.context` still exists as structured JSON for inferred type, confidence, and creation metadata.
+
-- docs should not describe edge context JSON as if it is the only user-facing explanation surface.
+Current status to document carefully:
+- recent nodes since `2026-04-16` that have chunks are represented in `vec_chunks`
+- older rows can have chunks without matching `vec_chunks`
+- some older nodes can have `nodes.embedding` without being present in `vec_nodes`
+- successful transcript retrieval may come from vector search, FTS, or text fallback
+
+Historical universal vector coverage is a separate backfill/repair task, not something this docs page should imply is already complete.
 
 ## Important Constraints
 
 - `dimensions` and `node_dimensions` are no longer canonical tables.
 - New installs should never create them.
-- Existing installs migrate by snapshotting old dimension data, then dropping the legacy tables.
+- Existing installs migrate by snapshotting old dimension data, then dropping legacy tables.
-- FTS repair and integrity handling are now operational concerns. Do not describe automatic live rebuild behavior as normal product behavior.
+- The removed `contexts` table and old context capsule are not live product doctrine.
+- FTS repair and integrity handling are operational concerns. Do not describe automatic live rebuild behavior as normal product behavior.
 
 ## Common Queries
 
@@ -90,3 +310,12 @@ FROM nodes
 ORDER BY updated_at DESC
 LIMIT 25;
 ```
+
+Check chunk/vector coverage:
+
+```sql
+SELECT COUNT(*) AS missing_vectors
+FROM chunks c
+LEFT JOIN vec_chunks v ON v.chunk_id = c.id
+WHERE v.chunk_id IS NULL;
+```

docs/8_mcp.md

@@ -17,6 +17,7 @@ Important runtime distinction:
 - `createEdge` is a post-confirmation execution tool. Agents should propose likely edges first and only write them after the user explicitly confirms.
 - `queryNodes` searches title, description, and source.
 - `dimensions` are removed from the MCP contract.
+- `contexts` and context-capsule tools are removed from the MCP contract.
 
 ## Behavior Split
 
@@ -64,7 +65,21 @@ Write:
 - Keep writeback prompts terse and selective. The goal is not to ask constantly whether every useful sentence should be saved.
 - Do not assume MCP node creation immediately produces chunks or embeddings. The canonical contract is:
   - write node data first
-  - app-owned pipeline later creates chunks, FTS rows, and vectors
+  - app-owned pipeline later creates readable chunks, FTS rows, `vec_nodes`, and `vec_chunks`
+- Do not use `chunk_status = 'chunked'` as a promise that every chunk has a row in `vec_chunks`. Chunking and vector coverage are related but distinct.
+
+## Search And Indexing Notes
+
+MCP users should understand the same retrieval split as the app:
+
+- direct node lookup searches node title, description, and source
+- full-text search can match literal words in nodes or chunks
+- chunk search can find passages inside long source text
+- `vec_nodes` can find semantically similar whole nodes when node-level vectors exist
+- `vec_chunks` can find semantically similar passages when chunk-level vectors exist
+- standalone MCP does not generate embeddings itself
+
+If an external agent creates or updates a node through standalone MCP while the app is closed, the node can exist before its chunks and vectors do. The app-owned pipeline processes that later.
 
 ## Memory-File Rule
 

docs/README.md

@@ -14,7 +14,7 @@
 | Doc | Description |
 |-----|-------------|
 | [Overview](./0_overview.md) | What RA-H OS is and what contract it shares with the main app |
-| [Schema](./2_schema.md) | Current SQLite contract |
+| [Schema + Search](./2_schema.md) | Current SQLite contract, indexing, and retrieval surfaces |
 | [Tools & Skills](./4_tools-and-guides.md) | MCP tools and skill system |
 | [Logging & Evals](./5_logging-and-evals.md) | Logs, evals, and debugging surfaces |
 | [UI](./6_ui.md) | Current pane and focus model |
@@ -60,7 +60,7 @@ Add to your `~/.claude.json`:
 
 If you publish a newer MCP release and need clients to use it immediately, bump the pinned version here and restart the client. Do not assume plain `npx ra-h-mcp-server` always refreshes instantly.
 
-Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path.
+Run RA-H once first so the database exists. The standalone MCP server can write nodes without the app running, but the app owns chunking and embedding from node source: readable `chunks`, full-text indexes, `vec_nodes`, and `vec_chunks`. See [MCP docs](./8_mcp.md) for the full install, verify, memory-file, and troubleshooting path.
 
 ## Questions?