Semantic Routing with Decision Branch

Applies to: ChengOS v0.2.0+ ｜ Last verified: 2026-09-25 ｜ Source: crates/cheng-nodes/src/nodes/builtin/ai/decision_branch.rs, config/decision_calibration.toml, crates/cheng-llm/tests/fixtures/decision/, scripts/decision/cascade_eval.py

A single general agent with every tool attached works, but it is slow, expensive, and holds more privilege than most turns need. Semantic routing puts an ai/decision_branch in front: each turn is sent to a focused sub-flow — email, calendar, weather — when one clearly applies, and to the general agent otherwise.

This guide is about building that router well. The providers behind it are covered in Decision Providers & TypeSafe Jev; every node field is in the Decision Branch Node reference.

The shape of a router

chat/input ──► ai/decision_branch
                 ├─ route_email     ──► email agent      (mail tools only)
                 ├─ route_calendar  ──► calendar agent   (calendar tools only)
                 ├─ route_weather   ──► weather lookup
                 ├─ fallback_out    ──► general agent    (the workflow as it was)
                 └─ decision        ──► optional: log / trace the routing summary

Three properties make this safe to add to an existing workflow:

• The context is delivered unchanged. Whichever port fires carries the same AgentContext the node received. History is never rewritten; the only addition is a bounded summary under metadata["decision_route"].
• Unselected routes skip themselves. Every route_<id> port except the chosen one is null, so downstream nodes on other routes auto-skip — no condition nodes needed.
• Everything unclear goes to the fallback. Ordinary conversation, ambiguous or multi-intent requests, low confidence, and provider failures all reach fallback_out. Wire it to the agent you had before, and the router is an optimization: a misroute or outage degrades to today's behavior, never to a broken turn.

Writing routes

Each route has three fields:

Write descriptions as capabilities, not keywords. "weather" is a poor description; "Check current weather, temperature, or rain forecasts for a place" is a good one. The model chooses by comparing the request with these sentences, so vague or overlapping descriptions produce low-confidence answers that fall back.

Keep routes disjoint. If two routes could both handle a request, the provider splits its probability between them and neither clears the threshold. Merge them, or sharpen the difference in the descriptions.

Stay within the limits. At most 16 routes, and at most 15 on Laya (the fallback takes one option). Laya's English model also has a small instruction budget; eight routes with the default instruction already exceed it, and the editor reports the refusal.

Edges follow IDs, not positions. Reordering routes never moves an edge. Renaming an id does — its port changes name.

The Routing Instruction (under advanced settings) replaces the default: "Choose the route that can satisfy the user's current request with the least privilege. Choose a route only when its capability is required. If the request needs more than one route, or none clearly matches, choose the fallback route." Keep tool-operation instructions on the focused agents, not here.

Thresholds and failures

Threshold Mode

Failure Mode — Fallback (default) sends provider errors and missing configuration to fallback_out; Fail closed fails the node with a typed error instead. Use fail-closed only when running the general agent on an unrouted turn would itself be wrong.

Timeout — empty uses the provider default (1500 ms for Jev and Laya; adaptive for the LLM baseline). A missed deadline follows the failure mode and never stalls the turn.

Sticky routing for follow-ups

"Reply to him and say yes" cannot be routed on its own; it only makes sense after the previous turn. With Route Memory = Sticky (the default):

• The previous turn's route is sent to the provider as context.
• In calibrated mode the provider is also asked whether the request is a referential follow-up. The previous route is held only when that probability and the previous route's own probability both clear the profile's hold thresholds (reason sticky_hold).
• A confident explicit fallback always wins over sticky; a held route is never forced.

Memory is kept per workflow, node, and conversation for six hours, in the API process. In a multi-instance deployment or after a restart a follow-up may reach the general agent instead of being held — safe by construction, never a wrong route. Laya is never sent the previous route, so sticky routing has no effect on it. Set Route Memory to Off to route every turn on its own.

A two-stage cascade

A cheap local model can decide the easy turns and hand only the hard ones to a stronger one:

stage 1: decision_branch (Laya, Calibrated, Uncertain Decisions = Separate port)
           ├─ route_*       ──► the focused sub-flows
           ├─ fallback_out  ──► general agent          (confident "none of these")
           └─ uncertain_out ──► stage 2
stage 2: decision_branch (LLM baseline, Threshold Mode = None, same routes)
           ├─ route_*       ──► the same sub-flows
           └─ fallback_out  ──► general agent

With Uncertain Decisions = Separate port, only turns that were below the threshold or had no profile go to uncertain_out. Confident fallbacks and provider failures still go straight to fallback_out. Give stage 2 exactly the same route IDs and descriptions.

Measure the cascade on your own deployment before quoting latency or accuracy: both depend on your hardware, the chat model, and your routes. scripts/decision/cascade_eval.py replays the labeled routing-eval-v2 set through this exact shape against your chat model endpoint, and crates/cheng-llm/tests/fixtures/decision/ holds the recorded runs it compares against.

Reading the decision summary

The decision output port (and metadata["decision_route"]) carries a text-free summary: selected route, reason, provider, model, language, calibration profile, probabilities, latency. The reason tells you why a turn went where it did:

A high share of uncalibrated means you are using a model or language without a profile. Many below_threshold turns usually mean overlapping descriptions.

Checklist

• fallback_out is connected to an agent that can handle anything.
• Every description says what the route does, in a sentence.
• No two routes overlap; multi-intent requests are expected to fall back.
• The provider, model, and language combination has a calibration profile — or threshold mode is None and you accept uncalibrated routing.
• Using Jev means routed request text leaves the deployment. If that is not acceptable, use Laya.

Next steps

• Decision Providers & TypeSafe Jev — credentials, Laya, profiles.
• Decision Branch Node — the complete field reference.
• ReAct Agent Guide — building the agents behind each route.