# Semantic Routing with Decision Branch

> Applies to: ChengOS v0.2.0+ ｜ Last verified: 2026-09-25 ｜ Source: `crates/cheng-nodes/src/nodes/builtin/ai/decision_branch.rs`, `config/decision_calibration.toml`, `crates/cheng-llm/tests/fixtures/decision/`, `scripts/decision/cascade_eval.py`

A single general agent with every tool attached works, but it is slow, expensive, and holds more
privilege than most turns need. **Semantic routing** puts an `ai/decision_branch` in front: each
turn is sent to a focused sub-flow — email, calendar, weather — when one clearly applies, and to
the general agent otherwise.

This guide is about building that router well. The providers behind it are covered in
[Decision Providers & TypeSafe Jev](/docs/decision-providers-and-jev); every node field is in the
[Decision Branch Node](/docs/nodes-decision) reference.

## The shape of a router

```text
chat/input ──► ai/decision_branch
                 ├─ route_email     ──► email agent      (mail tools only)
                 ├─ route_calendar  ──► calendar agent   (calendar tools only)
                 ├─ route_weather   ──► weather lookup
                 ├─ fallback_out    ──► general agent    (the workflow as it was)
                 └─ decision        ──► optional: log / trace the routing summary
```

Three properties make this safe to add to an existing workflow:

- **The context is delivered unchanged.** Whichever port fires carries the same
  `AgentContext` the node received. History is never rewritten; the only addition is a bounded
  summary under `metadata["decision_route"]`.
- **Unselected routes skip themselves.** Every `route_<id>` port except the chosen one is
  `null`, so downstream nodes on other routes auto-skip — no condition nodes needed.
- **Everything unclear goes to the fallback.** Ordinary conversation, ambiguous or multi-intent
  requests, low confidence, and provider failures all reach `fallback_out`. Wire it to the agent
  you had before, and the router is an optimization: a misroute or outage degrades to today's
  behavior, never to a broken turn.

## Writing routes

Each route has three fields:

| Field | Rules | Seen by the provider |
|---|---|---|
| `id` | 1–64 characters: lowercase letters, digits, `_`, `-`. Names the port `route_<id>`. `__fallback__` is reserved. | Yes, as the choice ID |
| `label` | Up to 80 characters | **No** — UI only |
| `description` | 1–2000 characters; one or two sentences | **Yes** — this is what the model reads |

**Write descriptions as capabilities, not keywords.** "weather" is a poor description;
"Check current weather, temperature, or rain forecasts for a place" is a good one. The model
chooses by comparing the request with these sentences, so vague or overlapping descriptions
produce low-confidence answers that fall back.

**Keep routes disjoint.** If two routes could both handle a request, the provider splits its
probability between them and neither clears the threshold. Merge them, or sharpen the
difference in the descriptions.

**Stay within the limits.** At most **16** routes, and at most **15** on Laya (the fallback
takes one option). Laya's English model also has a small instruction budget; eight routes with
the default instruction already exceed it, and the editor reports the refusal.

**Edges follow IDs, not positions.** Reordering routes never moves an edge. Renaming an `id`
does — its port changes name.

The **Routing Instruction** (under advanced settings) replaces the default:
*"Choose the route that can satisfy the user's current request with the least privilege. Choose
a route only when its capability is required. If the request needs more than one route, or none
clearly matches, choose the fallback route."* Keep tool-operation instructions on the focused
agents, not here.

## Thresholds and failures

**Threshold Mode**

| Mode | Behavior |
|---|---|
| **Calibrated** (default) | Apply the measured profile for this provider, model, and language. No profile means the turn falls back. |
| **None** | Route on the raw answer. Only invalid, out-of-set, timed-out, or failed decisions fall back. Required for the LLM baseline. |

**Failure Mode** — **Fallback** (default) sends provider errors and missing configuration to
`fallback_out`; **Fail closed** fails the node with a typed error instead. Use fail-closed only
when running the general agent on an unrouted turn would itself be wrong.

**Timeout** — empty uses the provider default (1500 ms for Jev and Laya; adaptive for the LLM
baseline). A missed deadline follows the failure mode and never stalls the turn.

## Sticky routing for follow-ups

"Reply to him and say yes" cannot be routed on its own; it only makes sense after the previous
turn. With **Route Memory = Sticky** (the default):

- The previous turn's route is sent to the provider as context.
- In **calibrated** mode the provider is also asked whether the request is a referential
  follow-up. The previous route is **held** only when that probability and the previous route's
  own probability both clear the profile's hold thresholds (reason `sticky_hold`).
- A confident explicit fallback always wins over sticky; a held route is never forced.

Memory is kept per workflow, node, and conversation for six hours, **in the API process.** In a
multi-instance deployment or after a restart a follow-up may reach the general agent instead of
being held — safe by construction, never a wrong route. Laya is never sent the previous route,
so sticky routing has no effect on it. Set Route Memory to **Off** to route every turn on its
own.

## A two-stage cascade

A cheap local model can decide the easy turns and hand only the hard ones to a stronger one:

```text
stage 1: decision_branch (Laya, Calibrated, Uncertain Decisions = Separate port)
           ├─ route_*       ──► the focused sub-flows
           ├─ fallback_out  ──► general agent          (confident "none of these")
           └─ uncertain_out ──► stage 2
stage 2: decision_branch (LLM baseline, Threshold Mode = None, same routes)
           ├─ route_*       ──► the same sub-flows
           └─ fallback_out  ──► general agent
```

With **Uncertain Decisions = Separate port**, only turns that were below the threshold or had no
profile go to `uncertain_out`. Confident fallbacks and provider failures still go straight to
`fallback_out`. Give stage 2 exactly the same route IDs and descriptions.

Measure the cascade on your own deployment before quoting latency or accuracy: both depend on
your hardware, the chat model, and your routes. `scripts/decision/cascade_eval.py` replays the
labeled `routing-eval-v2` set through this exact shape against your chat model endpoint, and
`crates/cheng-llm/tests/fixtures/decision/` holds the recorded runs it compares against.

## Reading the decision summary

The `decision` output port (and `metadata["decision_route"]`) carries a text-free summary:
selected route, reason, provider, model, language, calibration profile, probabilities, latency.
The `reason` tells you why a turn went where it did:

| Reason | Meaning |
|---|---|
| `selected` | A route cleared the threshold (or threshold mode is None) |
| `sticky_hold` | The previous route was held for a follow-up |
| `fallback_selected` | The provider confidently chose "none of these" |
| `uncertain_fallback` / `below_threshold` | The answer was not confident enough |
| `uncalibrated` | Calibrated mode, but no profile for this provider, model, and language |
| `empty_request` | There was no user text to route |
| `timeout`, `authentication`, `rate_limited`, `unavailable`, `out_of_set`, … | Provider failure, handled by the failure mode |

A high share of `uncalibrated` means you are using a model or language without a profile.
Many `below_threshold` turns usually mean overlapping descriptions.

## Checklist

- `fallback_out` is connected to an agent that can handle anything.
- Every description says what the route *does*, in a sentence.
- No two routes overlap; multi-intent requests are expected to fall back.
- The provider, model, and language combination has a calibration profile — or threshold mode
  is None and you accept uncalibrated routing.
- Using Jev means routed request text leaves the deployment. If that is not acceptable, use Laya.

## Next steps

- [Decision Providers & TypeSafe Jev](/docs/decision-providers-and-jev) — credentials, Laya, profiles.
- [Decision Branch Node](/docs/nodes-decision) — the complete field reference.
- [ReAct Agent Guide](/docs/agent-react-guide) — building the agents behind each route.