The Jev Decision System

Applies to: ChengOS v0.2.0+ ｜ Last verified: 2026-09-26 ｜ Source: crates/cheng-llm/src/decision.rs, crates/cheng-llm/src/decision_service.rs, crates/cheng-nodes/src/nodes/builtin/ai/decision_branch.rs, config/decision_calibration.toml, crates/cheng-llm/tests/fixtures/decision/

ChengOS can put a small, fast decision model in front of your agents. Before any large language model runs, the decision model reads the user's latest request and picks one of the routes you declared on the canvas — "email", "weather", "code" — or none of them. Each route leads to an agent that carries only the tools it needs; everything unclear goes to the general agent you already had.

This page explains why the system exists, how it is built, and what it measured. To build a router, read Semantic Routing with Decision Branch. For credentials and providers, read Decision Providers & TypeSafe Jev.

Why a decision system

One general agent with every tool attached works, but every turn pays for it:

• Context. The definitions of every tool are sent on every turn. With mail, web, and file tools attached that is about 4.3k–5.2k input tokens before the conversation even starts.
• Accuracy. More tools means more chances to pick the wrong one.
• Privilege. A request about the weather runs in an agent that could also send email.

The usual alternative, a single wrapper tool that lists the others on demand (Tool Hub), trades context for an extra round trip — and in the measurements below it did not save context at all on turns that actually needed a tool.

A decision model solves the problem one step earlier: it chooses the path before the chat model sees anything. It never answers the user, never calls a tool, and can only choose among routes the workflow author declared.

How it works

Every routed turn passes two independent decision points:

user message
  -> ai/decision_branch        choose a route (or the fallback) — decides context
  -> focused agent, or the general agent on fallback
  -> concrete tool call, if any
  -> approval policy           execute, ask the user, or deny — decides permission

The first decision only makes the context smaller. The second one — the deterministic approval policy — is still the only thing that grants permission. A route can never unlock a tool call that the approval policy would stop.

Inside ChengFlow the parts are:

Decision providers are deliberately not chat providers: they never appear in a chat model picker, and a chat model becomes a decision provider only through the explicit LLM-baseline option.

Design rules

These hold for every provider and every workflow:

• The router is an optimization, not a gate. Ordinary conversation, ambiguous or multi-intent requests, low confidence, timeouts, and provider outages all go to fallback_out. Wired to your existing agent, a misroute or outage degrades to today's behavior, never to a broken turn.
• The answer is always one declared ID. An out-of-set or malformed answer is rejected before it can name a port.
• The conversation is delivered unchanged. The chosen port carries the exact context the node received; history is never rewritten. Only a bounded summary is attached under metadata["decision_route"].
• The provider sees as little as possible. Only the latest request, the route descriptions, the routing instruction, and — for follow-ups — the previous route ID. No system prompts, no tool definitions, no credentials.
• Probabilities count only when calibrated. A threshold is trusted only for the exact provider, model, and language it was measured on. Without a profile a turn falls back.
• Edges follow IDs. Reordering routes never moves an edge; unselected routes skip themselves downstream.

What it measured

The go/no-go evaluation ran on 2026-09-23 against routing-eval-v2: 157 labeled requests (90 English, 67 Chinese) covering short and long single-intent requests, chit-chat, ambiguous and multi-intent requests, referential follow-ups ("make it shorter"), new topics after a previous route, and route lists of 3, 8, and 16. The decision model was jev-1.13.0.

The full interactive report — charts, per-category breakdown, threshold sweep — is published here: Jev semantic routing evaluation (Chinese). Its key figures:

Routing decisions, all 157 requests. A wrong route is the only error that loses capability — the focused agent lacks the tool it needs. A safe fallback just sends the turn to the general agent.

End-to-end completion, 101 three-route requests (email / weather / code, backed by the real mail, web, and file_ops_hub tools). A tool-needing request counts when the right tool is called; chit-chat and ambiguous requests count when no tool is called.

Median context on the 63 tool-needing requests (input tokens seen by the answering model, all turns; Jev's own ~462 input tokens are billed by TypeSafe and enter no LLM context):

What the numbers say:

• Jev made one mistake, on the same multi-intent request every model got wrong (#098, "email the code review results to the team"). Multi-intent phrasing is the residual risk.
• Chat models can route, but quality depends on the model — and they report no probabilities, so no threshold can catch their unsure answers. They remain an explicit, uncalibrated option.
• Tool Hub does not save context on tool-needing turns: 59%–62% of them need an extra discovery turn whose result alone is ~4k tokens, and 14–19 of 63 tool-needing requests were answered without discovering any tool.
• Calibration is stable. Every select threshold from 0.60 to 0.75 gives 156/157; the shipped profile uses 0.70. Follow-ups are held on the previous route when the follow-up probability is ≥ 0.75 and the previous route's own probability is ≥ 0.30 — real follow-ups scored 0.79–1.00, new topics at most 0.04.

Choosing a provider

Jev is the default because it is the most accurate, the fastest hosted option, and ships with calibration. Choose Laya when request text must stay on your own hosts: it is conservative — it falls back far more often than Jev and never gets the previous route, so sticky follow-ups do not apply — but it answers in milliseconds. A two-stage cascade (Laya first, a local chat model only for its uncertain turns) recovers 7–9 correct routes per scenario over Laya alone while calling the chat model on about 46% of turns. Neither local option approaches Jev; both keep your data at home.

Pairing routing with context compaction

Routing shrinks the first turn. Long runs grow through tool results instead, and a focused agent can be tuned for that too. Starting with the release after v0.2.0, each agent/react_agent takes an optional compaction preset — standard (today's behavior), aggressive, or lenient — so the agent behind each route can compact the way its work needs:

chat/input → ai/decision_branch ─route_email──► agent/react_agent  (standard)
                                ├route_code───► agent/react_agent  (lenient)
                                └route_data───► agent/react_agent  (aggressive)

The router itself is untouched; the preset is set on the agent, never inferred from a route name. On the describe → call workload the aggressive preset's early archival of already-used tool schemas cut cumulative request tokens by 34.8% in the offline corpus. It stays limited to aggressive until it is measured on real replays.

Limits and what's next

• The evaluation scored first-turn tool selection, not complete multi-step tasks, and the follow-up history was synthesized.
• Groups such as the 16-route cases (28 requests) are directional only. Re-measure before changing routes, language, or model version — a calibration profile applies only to what it was measured on.
• Route memory for follow-ups is kept in the API process for six hours. In a multi-instance deployment or after a restart, a follow-up may reach the general agent instead — safe, never a wrong route.
• Multi-intent requests go to the fallback by design. Serving them with more than one focused route is not part of the first release.
• Using a decision model to raise approval requirements for risky tool calls has been designed but deliberately deferred: the deterministic approval policy remains the only permission boundary.

Next steps

• Semantic Routing with Decision Branch — build and tune a router.
• Decision Providers & TypeSafe Jev — credentials, Laya, calibration profiles.
• Decision Branch Node — every node field.
• Tool-call Approval Policy — the permission boundary behind every route.