# RAG Pipeline Design

> Applies to: ChengOS v0.1.0+ ｜ Last verified: 2026-08-12 ｜ Source: `crates/cheng-nodes/src/nodes/builtin/rag/constants.rs`, `crates/cheng-nodes/src/nodes/builtin/rag/nodes`, `crates/cheng-nodes/src/nodes/builtin/rag/services`, `crates/cheng-vector`

RAG in ChengOS is five nodes and two pipelines. The indexing pipeline runs when documents
change; the retrieval pipeline runs on every question. They share nothing at runtime except the
knowledge base they read and write.

```text
INDEXING                                  RETRIEVAL
 source text                               user query
     │                                         │
 rag/chunker      ──chunks_artifact──►     rag/retriever  ──documents_artifact──►
     │                                         │                    rag/formatter
 rag/document_indexer                          │                          │
     │                                         │                       ai/llm
     ▼                                         ▼
  knowledge base (Qdrant collection) ◄─────────┘
```

## Configuration nodes

Three of the RAG nodes take config objects rather than loose fields, produced by dedicated
nodes: `ai/qdrant_config` (where the vector store is), `ai/embedding_config` (which embedding
model, and optionally a reranker), and `ai/llm` config for the steps that need a model.

Wiring config as an object rather than as twenty inputs is what lets one config node serve both
the indexer and the retriever — and that matters, because **the embedding model used to index
must be the model used to retrieve**. Sharing one config node makes the mismatch impossible.

## Chunking

`rag/chunker` splits text into **parent** and **child** chunks. Four modes:

| Mode | Splits by |
|---|---|
| `structural` (default) | Markdown headings |
| `fixed` | Fixed character windows |
| `semantic` | Paragraph semantic boundaries |
| `hybrid` | Structural, then semantic within sections |

Sizes are measured in **characters, not tokens**, and that is a deliberate design decision:
the chunker is a string splitter, not a tokenizer, and character counts are predictable,
testable, and language-agnostic. Token counts are not — one CJK character is roughly 1–2 tokens
while one English word is roughly 1.

| Constant | Value |
|---|---|
| Default parent chunk | 384 chars |
| Default child chunk | 384 chars |
| Max embedding chunk (hard limit) | 512 chars |
| Parent overlap ratio | 0.1 |
| Child overlap ratio | 0.2 |
| Min / max chunk size | 50 / 10000 chars |

The 512-character ceiling is a safety clamp applied at runtime: most embedding models cap at
512 tokens, and 512 characters leaves headroom in every language. A child chunk larger than
that is clamped, not rejected.

**Output is an artifact, not an array.** The chunker returns `chunks_artifact` plus bounded
previews (`parent_preview`, `child_preview`, `preview_truncated`) rather than the full chunk
list inline. A large document produces thousands of chunks; passing them through node outputs
would blow up execution records and any LLM context they touch. The inline `parent_chunks` /
`child_chunks` fields still exist but are empty unless you set `legacy_inline_chunks` — treat
them as deprecated. Symmetrically, the input accepts `content_artifact` /
`content_artifact_uri` in preference to inline `content`.

## Indexing

`rag/document_indexer` writes vectors into a knowledge base. Two modes:

| Mode | Vectors written |
|---|---|
| `standard` (default) | `child_vec` only |
| `enhanced` | `parent_vec` + `child_vec` + `context_vec` |

Named vectors with different dimensions live in the same Qdrant point:

| Vector | Dimension | Role |
|---|---|---|
| `parent_vec` | 384 | Small, fast — used for pre-filtering |
| `child_vec` | 1024 | Large, accurate — the primary match |
| `context_vec` | 256 | Lightweight contextual signal |

`enhanced` mode exists to feed the `ultimate` retrieval mode; if you are not going to retrieve
in `ultimate`, `standard` costs less and indexes faster.

With `generate_context: true` (enhanced mode), an LLM writes a short contextual description for
each chunk — the Contextual Retrieval technique — which is what `context_vec` embeds. This
requires an `llm_config` and costs one model call per chunk, so it is off by default.

The indexer takes `kb_id`, not a collection name: the collection is resolved from the database
and validated against the workspace, so a workflow cannot index into another workspace's
collection by guessing a name. Prefer wiring `chunks_artifact` from the chunker; supplying
`content` directly makes the indexer chunk internally with defaults.

**Idempotency**: pass `version`, and re-indexing an unchanged document returns `skipped: true`
rather than rewriting every vector. This is what makes it safe to put the indexing workflow on
a [schedule](/docs/exec-scheduling-triggers).

## Retrieval

`rag/retriever` has three modes, and the difference between them is the whole cost/quality
dial of the system.

| Mode | Pipeline | Default limit | P50 target |
|---|---|---|---|
| `economy` (default) | BM25 + `child_vec` | 5 | 100 ms |
| `power` | `parent_vec` pre-filter → `child_vec` + BM25 → RRF fusion | 10 | 120 ms |
| `ultimate` | HyDE + Contextual Retrieval + reranker | 20 | 150 ms |

**Economy** runs a keyword search and a vector search and returns the union — cheap, and good
enough when queries share vocabulary with the corpus.

**Power** first narrows the candidate set with the cheap `parent_vec` (pre-filter limit 50),
then runs `child_vec` and BM25 over what survives, then fuses the two ranked lists with
**Reciprocal Rank Fusion**:

```text
RRF score = Σ  1 / (k + rank_i)        k = 60
```

RRF combines rankings, not scores, which is why it can merge a BM25 list and a cosine-similarity
list without normalising incompatible score scales.

**Ultimate** adds HyDE (the LLM writes a hypothetical answer and embeds *that* instead of the
question, which matches the corpus's phrasing better), uses the contextual descriptions written
at index time, and reranks the top candidates (reranker top-k 10). It needs `llm_config`, an
`enhanced`-indexed knowledge base, and a reranker in the embedding config — supported reranker
models include Cohere `rerank-multilingual-v3.0`, Jina `jina-reranker-v2-base-multilingual`,
and Qwen3 rerank at 0.6b / 4b / 8b.

Two other inputs matter:

- **`workspace_scope`** — `ambient` follows the runtime workspace; `pinned` locks this retriever
  to one configured workspace. Pin it when a shared workflow must always answer from one
  reference corpus regardless of who runs it.
- **`document_filter`** — a list of document ids to restrict the search to.

`query` accepts either a plain string or a chat context, so a retriever can be wired straight to
a chat input node without an adapter in between.

Each result carries `score`, `text`, `document_id`, `chunk_index`, the `context` description if
one exists, and `retrieval_method` — which method actually found it. That last field is the one
to look at when tuning: it tells you whether BM25 or the vector search is doing the work.

Like the chunker, the retriever returns a `documents_artifact` and bounded previews.

## Formatting the prompt

`rag/formatter` turns query plus retrieved documents into a prompt. Templates: `qa` (standard
question answering), `summary` (document summarisation), or `custom` with placeholders
`{query}`, `{documents}`, `{count}`. Toggle `include_scores` and `include_sources` to decide
whether the model sees relevance scores and document ids — include sources when you want the
answer to cite, omit them when you do not want the model narrating its retrieval.

Prefer wiring `documents_artifact` from the retriever; the inline `documents` input is a
compatibility path.

## Managing knowledge bases

`rag/kb_manager` handles the housekeeping: `list` (default), `info`, and `delete`. `delete`
removes the collection — it is the one destructive node in the RAG set, so keep it out of
agent-facing workflows.

## A working pipeline

```text
Indexing:
  io/read_file → rag/chunker → rag/document_indexer
                    ▲                  ▲
       ai/embedding_config ────────────┘
       ai/qdrant_config    ────────────┘
       workspace/knowledge_base (select) → kb_id

Answering:
  chat/input → rag/retriever → rag/formatter → ai/llm → chat/output
```

Start in `economy` + `standard`. Move to `power` when recall is the problem; move to
`ultimate` + `enhanced` only when you have measured that reranking earns its latency and cost.

## Next steps

- [Knowledge Bases & Workspace Context](/docs/data-knowledge-base) — creating the target collection.
- [Storage, Redis & Vector Configuration](/docs/config-storage-redis) — running Qdrant.
- [LLM & Model Configuration](/docs/config-llm-models) — embedding and generation models.