1 · A hexapod plans its next step
The testbed is an 18-DOF hexapod walker with a browser-based digital twin: the robot's recorded episodes replay in three.js over its real lidar map. For this experiment the twin gained a Plan mode — drop an obstacle with a kind label (cardboard box, cable, cat, child), set a goal, and watch the planner think once per gait cycle.
Each tick, code digests the scene into named fields — six clearance sectors, an obstacle list, goal bearing, plain-language house rules — and asks one request's worth of typed questions: which bounded step next (a Choice over forward / veer / turn / stop), how fast (a Score), is a person too close, should we stop and ask the operator (probabilities). Code then applies hard vetoes the model can never override: sector clearance, a person keep-out radius, a stride budget. The model picks among safe options; geometry has the last word.
The headline result came from comparing against a pure-geometry baseline on the same code path. On plain obstacle courses the two tie. On semantic scenes they split: told "never walk over cables," the model routes around a floor cable the geometric planner happily walks across — and with the rule changed to "it's taped down, fine," it walks over it too. Same geometry, opposite rule, opposite behavior. With a child or a pet inside a metre it stops and waits for a human instead of threading past. It also follows a walking person, holding a respectful distance, and chose "hold position" on its own when they paused.
Just as useful: what the model can't do. It cannot compare bearings or add distances — early versions circled forever. The fix was architectural: code computes the geometric recommendation and the model only overrides it for semantic reasons. And on a symmetric obstacle it dithered left-right-left until code hysteresis made it commit. The evaluation found a real failure too: a tight corridor defeats it where the dumb heuristic squeaks through.
2 · Should a robot merge this PR?
Same shape, different state: feed the model a digested code change — per-file stats computed in code, a bounded diff excerpt — and ask seven questions an automated pipeline cares about: what kind of change is this, does the new code plausibly contain a defect, is it security-sensitive, does it break public API, is it safe to merge with no human review, how much review does it deserve. A code gate maps the answers to an action, and the model's own "safe to automerge" opinion cannot override its risk signals.
Three real open-source commits, chosen to span the categories:
| Change | Kind | Bug | Security | Breaking | Automerge | Pipeline action |
|---|---|---|---|---|---|---|
| OpenSSL — the commit that introduced Heartbleed | feature | 0.88 | 0.98 | 0.28 | 0.03 | block · expert review |
| Flask — "fix typos" | docs_or_typo | 0.02 | 0.03 | 0.02 | 0.94 | automerge candidate |
| Flask — "remove deprecated code" (−225 lines, CI green) | removal | 0.04 | 0.08 | 0.81 | 0.29 | block · breaking change |
| Control: the Heartbleed fix commit | bug_fix | 0.26 | 0.99 | 0.16 | 0.06 | block · expert review |
The control row is the one that makes this interesting. The fix touches the same files and the same protocol surface as the vulnerability, yet the bug probability drops from 0.88 to 0.26 while security-sensitivity correctly stays pinned — the axes are independent, not one blended "scariness" score. Re-running the fix with a deliberately bland commit message changed nothing, so the signal comes from reading the diff, not the description. And the deprecation removal is the case size heuristics get wrong: mostly deletions, green CI, tiny diff — and an 0.81 breaking-change flag that correctly keeps it away from automerge.
Honest caveat: three examples illustrate that the axes separate; they don't measure precision. Heartbleed is also the most famous vulnerability in history, so some of that 0.88 may be recognition. The follow-up is a corpus of regression-introducing commits nobody has heard of. Until then this runs exactly like the other experiments started: labeling in shadow, gating nothing.
3 · An inbox, classified in shadow
The first experiment chronologically, and the template for the other two. A read-only classifier fetches a rolling seven-day window of mail, redacts it down to headers plus a bounded excerpt — URLs, codes, and long numbers stripped before anything leaves the machine — and asks fourteen independent judgments in one request: message kind, attention queue, per-topic label probabilities, does this need a reply, is it personal or high-stakes, would archiving it be safe.
Nothing is mutated. The weekly audit note shows flagged disagreements plus a small calibration sample, each row with a one-character verdict column a human fills in; the next run parses those verdicts back into a ledger keyed by a stable opaque ID. The ledger is the dataset that will eventually justify automation — per-label precision after 200 adjudicated messages, and a hard rule that no amount of aggregate accuracy unlocks auto-archiving: hiding a personal message is the unrecoverable error, so personal correspondence carries a code veto no probability can cross. In the very first review pass the model scored a terse personal note as "safe to archive" — it was wrong, and that's exactly the kind of error the shadow phase exists to catch.
What three experiments agree on
- Don't let the model compute — let it override. Code produces the deterministic answer (the geometric step, the diff stats, the filter signals); the model accepts it or overrides for semantic reasons. Every experiment got better the moment it adopted this.
- Per-action thresholds, not aggregate accuracy. A wrong label is recoverable; a hidden personal email, a merged vulnerability, and a step toward a child are not. Each irreversible action keeps a code veto.
- Shadow mode first, with a ledger. Classify, log, mutate nothing, and turn human corrections into the evaluation set that earns each promotion.
- The probabilities are real. Confidence sat at 0.75–0.85 when the model agreed with a good recommendation and dropped to 0.35–0.45 on genuinely ambiguous scenes — informative enough to gate on.
- It's fast and cheap enough to ask everything. 150–700 ms and a few thousand tokens per request, with all questions evaluated in parallel — so every tick, diff, and email gets the full question set.
TypeSafe is currently in preview — you can sign up at typesafe.ai and run experiments like these yourself.