Built 31 October 2026 on the prototype 1 reference site, the prototype 2 event store and the prototype 3 operation registry and gateway. Roadmap gate (row 4): holdout test on comparison tasks against navigation; cost per session under ceiling; median turn under five seconds; fallback suite green. The blueprint's prototype mapping adds: a conversation panel on the reference site composing from the catalog, with cost and latency on the dashboard.
Hypothesis tested: a conversational layer composing from a small approved component catalog improves task completion over conventional navigation without hurting trust.
What was built
| Piece | Where | What it does |
|---|---|---|
| Catalog | packages/catalog |
Nine components, each with a JSON Schema for its model-facing props, the operation results it binds to, an accessibility contract and a cost class: Answer, ProductCard, ComparisonTable, AvailabilityBlock, LeadTimeBlock, QuoteForm, DemoBookingForm, EscalateToHuman, Disclosure. Props carry entity ids, not facts: hydration fills names, prices, availability and URLs from the conversation's evidence (operation results fetched through the gateway), and the composition policy validator of ADR-017 rejects anything schema-valid but wrong: unknown bindings, foreign URLs, markup, figures no operation returned, arithmetic that does not add up, too many components or tables, text too long. Three industry profiles (default, garment, furniture) as data: framing, leading attributes, card highlights, suggested prompts. The same view model renders as plain text for the Markdown projection and the harness. |
| Generative engine | packages/generative |
One turn: visitor message in, rendered composition out. The model sees the registry's three read operations as tools, exactly as the gateway projects them, plus a compose tool whose input schema is the catalog (ADR-021). Business-tier facts (search_products, lookup_business_info) are primed through the gateway and cached as prompt prefix with the system prompt and the tool list; live stock needs a check_availability call before an AvailabilityBlock binds. Budgets: a $0.10 model-cost ceiling per conversation that degrades to navigation without a model call, an 8 s hard stop per turn, at most two operation round trips and one repair. Every failure (slow, error, invalid, over budget, no provider) renders a conventional composition built from evidence: product cards, the FAQ answer, the route to a person. One model adapter interface; Claude Sonnet 5.5 behind it with prompt caching on; a scripted adapter for tests. Every turn is a composition event with tokens (input, output, cache read, cache write), cost, latency, components, operation calls, the fallback reason and any rejected rule. |
/assist |
apps/web/app/assist, components/Composition.tsx, /api/assist |
A server-rendered page beside the navigation: disclosure, industry selector, the conversation so far, the composed answer, a question box, suggested prompts, and "Prefer a person?" always on screen. Plain forms, no client JavaScript required, so it works for accessibility-tree agents. The renderer maps the hydrated view model to the site's own React components; the model never talks to React. A composed QuoteForm is sent by the visitor; the server action signs that one write with the site's own Web Bot Auth identity (assistant.<host>), invokes request_quote through the gateway on the generative surface, and the request stops at awaiting_consent and completes on /consent/{token} like every agent's (ADR-022). The home page carries a visible entry with example questions. /assist serves Markdown that points agents to the gateway. |
| Holdout | apps/web/lib/assistant.ts, apps/probe |
The navigation arm (header x-sysbot-variant: navigation, or a session-hash share for real visitors) never sees the assistant's entry point. The harness runs the same tasks on both arms: scripted-a11y and llm-browser on the conventional pages; scripted-assist (types the task into /assist once and reads the composed page: the layer alone) and llm-assist (the same real browser agent started on /assist and told it is the assistant page: the visitor) on the generative arm. The task set is the five tasks plus two comparison questions that need more than one page: knitwear_fit (which machine, its price, whether installation is included for two) and price_difference (speed and price of both sewing machines and the difference). Turn latency, cost and fallbacks come from the site's own composition events. |
| Fallback suite | packages/generative/test, apps/probe/src/fallback-suite.ts |
Unit level: a scripted model that is slow, throws, returns an unknown component with an uncited price, or runs with a spent budget; each turn renders the conventional composition with the published prices and the contact route and never the injected content; faults are ignored unless the host allows them. Live: the same four faults injected through /api/assist on the running site (honoured only with HARNESS_TOKEN), plus the page rendering under a fault, Markdown negotiation of /assist and the product pages, the holdout split, and the gateway index. |
| Dashboard | apps/web/app/ops |
New "Assistant" section: turns by outcome (composed or the fallback reason) with p50, p75 and p90 latency, mean and total cost and operation calls; cost per conversation against the ceiling (median, p90, max, how many over); catalog components used. |

Evidence
Run of 31 October 2026, 280 executions (four agents, seven tasks, ten repetitions), one model (Claude Sonnet 5.5) as the
visitor agent on both arms and as the layer's model. Report: apps/probe/reports/2026-10-31T15-42-40-497Z.md. The ceiling
was set before the run at the blueprint's default, $0.10 of model cost per conversation: 2.5 times the modelled $0.04 a
three-turn session would cost at these per-turn figures, and above the $0.08 the study cites for a modelled retail session,
so a session that reaches it has gone wrong rather than long.
| Arm | Agent | Completion, all 7 tasks | Completion, 3 comparison tasks | Incorrect action | Median visitor steps per task | Median ms per task | Visitor-agent cost per task | Layer turns | Median turn ms | p75 | p90 | Fallbacks | Layer cost per session, median / max |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| navigation | llm-browser (real agent) | 100% (70/70) | 100% (30/30) | 0 | 3 | 4,138 | $0.024 | – | – | – | – | – | – |
| navigation | scripted-a11y (baseline) | 86% | 67% (20/30) | 0 | 3 | 296 | – | – | – | – | – | – | – |
| generative | llm-assist (real agent as the visitor) | 100% (70/70) | 100% (30/30) | 0 | 3 | 8,241 | $0.031 | 70 | 2,811 | 3,934 | 5,031 | 3 (timeout) | $0.0071 / $0.0218 |
| generative | scripted-assist (the layer alone) | 100% (70/70) | 100% (30/30) | 0 | 4 | 3,366 | – | 70 | 3,029 | 4,406 | 5,102 | 2 (timeout) | $0.0077 / $0.0209 |
The baseline's 67% on comparison tasks is its own limit, not the site's: a scripted plan cannot subtract, so it has no plan for
price_difference. The visitor-agent cost is what the model playing the visitor spent reading ARIA snapshots; the layer's cost
is separate and shown in the last column.
Per task, the real agent on each arm:
| Task | Navigation: steps / ms | Generative: visitor steps / ms | Layer turn ms (median, p90) | Layer model calls / operation calls | Layer cost per session (median) |
|---|---|---|---|---|---|
| find_price | 2 / 3,345 | 3 / 7,398 | 2,467 / 2,641 | 1 / 0 | $0.0064 |
| check_availability | 2 / 3,706 | 3 / 9,022 | 3,994 / 4,323 | 2 / 1 (check_availability) |
$0.0106 |
| compare_speed | 3 / 4,142 | 3 / 6,908 | 2,723 / 2,915 | 1 / 0 | $0.0071 |
| delivery_time | 2 / 3,220 | 3 / 7,013 | 2,554 / 2,722 | 1 / 0 | $0.0065 |
| request_quote | 9 / 7,748 | 9 / 14,925 | 2,552 / 2,786 | 1 / 0 | $0.0061 |
| knitwear_fit | 3 / 4,526 | 3 / 11,682 | 5,520 / 8,005 | 1 / 0 | $0.0123 |
| price_difference | 3 / 4,798 | 3 / 8,126 | 2,957 / 3,251 | 1 / 0 | $0.0074 |
Across all 140 layer turns: median 2,871 ms, p75 4,286 ms, p90 5,031 ms; 117 turns took one model call and 23 took two (the 20
stock questions, which must call check_availability first, and 3 repairs); 20 operation calls in all, every one through the
gateway on the generative surface. Mean model cost $0.0088 per turn, $1.23 for the run. Five turns (3.6%), all on
knitwear_fit, hit the 8 s hard stop and rendered the fallback; the fallback carried the product cards from evidence, so
the task was still answered, and the cost of those turns was the model call that was abandoned. Eight turns (5.7%) had a
composition rejected: seven for a figure no operation returned (number-ungrounded), one for the schema; all eight were
repaired within the turn and none reached the page. On the write task, the layer composed a pre-filled QuoteForm in 20 of
20 conversations; the visitor sent it and approved the request on the consent screen 20 of 20 times (one screen, two clicks
when the harness played the requester), and no inquiry was written by the model.
Components used over 140 turns: Answer 141, Disclosure 140, ProductCard 70, LeadTimeBlock 53, ComparisonTable 40, QuoteForm 24, AvailabilityBlock 20. EscalateToHuman and DemoBookingForm were composed only in fallbacks and in the fallback suite: none of the seven tasks is a problem or a visit.

Industry variant
The same question, "Which of your machines should I look at first for my workshop, and how do they compare?", asked once under each profile against the same store (in-process run, one model call each, no operation round trip):
| Profile | Leads with | Comparison columns and row order | Answer |
|---|---|---|---|
| garment | NL-3400 overlock for knitwear, NL-2200 lockstitch for woven; the cutter named as secondary | NL-3400, NL-2200; price, stitch type, maximum speed, needle system, motor, availability | "On seaming speed alone, the NL-3400 wins at 7,000 stitches per minute against 5,000"; 4.9 s, $0.022 |
| furniture | NC-8 straight-knife cutter: "the only machine built for multi-ply spreads" | NC-8, NL-2200, NL-3400; price, cutting height, blade type, power, needle system, motor, availability | Says plainly that the published specs do not state heavy-fabric capability for the sewing machines; 7.7 s, $0.027 |
Nothing in the store changed between the two; the profile changed the framing paragraph in the cached prefix, the attribute order in hydration and the suggested prompts. The longer furniture turn is the cost of an open question: two Answer components and a three-column table, 976 output tokens.
Gate status
- Holdout on comparison tasks shows improved completion over navigation: not met, measured as level. The real agent completed 30 of 30 comparison tasks on both arms, and 70 of 70 overall on both. On a three-product reference site a browser agent reading ARIA snapshots already finds every answer, including the two-page comparison and the arithmetic, so there is no completion headroom for the layer to take. What the holdout did show: the layer alone answered every task from a single typed sentence (scripted-assist 70 of 70), and the visitor's actions on comparison tasks were level (3 against 3) while single-fact tasks cost one more action (3 against 2: type, send, read). Task time doubled for the visitor agent (two model stacks in series), and the layer's own turn is the measure the roadmap set.
- Cost per session under the ceiling: met. Every one of 140 sessions was under $0.10; median $0.0071 to $0.0077, maximum $0.0218, a fifth of the ceiling, at the assumed price with caching on.
- Median turn under five seconds: met. 2,811 ms and 3,029 ms for the two arms' agents (2,871 ms across all turns); p90 at 5.0 s; the open two-page question (
knitwear_fit) is the slow case at 5.5 s median. - Fallback suite green: met. 9 of 9 live checks and 7 unit-level checks: slow, failing and invalid models and a spent budget each rendered the conventional composition with the published prices and the route to a person, never the injected content; the page, Markdown negotiation, the gateway and the holdout split held.
- No incorrect action: met. 0 in 280 executions; the model holds no write tool.
Findings worth carrying into the blueprint
- On a small site the layer does not raise completion; it changes the shape of the work. Both arms were at 100%, so the hypothesis's completion claim is not demonstrated here and cannot be on this site: the condition to test it is a catalogue a page cannot hold in one snapshot, where navigation needs search and several pages. What did move was the visitor's effort: one typed sentence answered every task, comparisons took the same three actions as navigation, and the arithmetic question that the scripted baseline could not do at all was answered with verified computed figures. The two-screen quote is the one place the layer costs the visitor more than the prototype 1 form (ADR-022).
- Grounding by construction works, and it binds. Props carry ids, so a hallucinated price has no slot; the one free-text slot is checked against the evidence, and seven of 140 turns tried to state a figure no operation returned (every one on the two harder tasks, a paraphrased price or a summed total) and were caught and repaired before anything rendered. The rejection rule is now a dashboard column, which is the first measurable view of what a model would have shown.
- Prime the business tier, call for the live tier. Putting the products, services, FAQ and contact into the cached prefix turned most turns into one model call with no operation round trip (117 of 140), at about 2.5 s and $0.007; the stock question is the exception by design (freshness tier live, ADR-018) and costs one round trip, 4 s and $0.011. Before priming, the first smoke turns took three model calls and 8 s. The prefix is about 6,500 tokens; with caching it is a tenth of the price on every call after the first in a five-minute window, and it is shared across conversations.
- The 8 s hard stop binds on open questions, not on facts. All five timeouts were
knitwear_fit, where the model wrote two Answers and a three-column table; the industry run showed the same, 7.7 s for the furniture profile's open question against 2.5 s for a fact. The fallback still answered because it is composed from evidence, so a timeout cost a model call and a plainer page, not a wrong page. Streaming component by component, which the blueprint allows and this prototype did not build, is the fix; so is a smaller model for the compose step. - The composition is a tool call, and that is what made it reliable. The provider enforced the catalog schema at generation, so no turn failed to parse; the one schema rejection in 140 turns was an optional-field quirk. A free-text JSON contract would have spent its repairs on syntax.
- The write path needed no new rule. The layer signs the visitor's own form as a first-party agent and the request lands in the same consent flow, audit trail and dashboard as an outside agent's, with the origin shown on the consent screen. The price is a second screen for the visitor; the identity decision (ADR-004) is where that gets collapsed, not the security model.
- Industry composition is real and cheap, and unmeasured. The same store answered the same question with a different leading product, attribute order and framing for a garment factory and a furniture maker, from a profile that is a dozen lines of data. Whether that lifts anything is the study's open question; three products cannot answer it.
Known limits
- One model (Claude Sonnet 5.5) for both the layer and the visitor agent, and the visitor agent on the generative arm was told it was on the assistant page; a real visitor who ignores the assistant is the navigation arm by construction. The holdout is a harness, not visitors: the
SYSBOT_ASSIST_SHAREsplit exists for a real site but has no sessions yet. - The gate's "improved completion" compares against a navigation arm that was already at or near 100% on a three-product site (prototypes 1 and 3). Both arms completed every task, so the "improved completion" criterion is reported as not met, level at 100%, rather than inferred from steps or time. The holdout showed where the difference is (one typed sentence, verified arithmetic, the shape of the visitor's effort) but not the difference the roadmap named.
- Cost is at an assumed price ($3 per million input, $15 output, cache read at a tenth, cache write at 1.25 times); tokens are measured. The ceiling of $0.10 per conversation is the blueprint's default and was chosen before the run; the report shows how far under it the sessions landed.
- Streaming component by component is not implemented; a turn renders whole, so the 1.5 s first-component budget is not measured. The 8 s hard stop and the 5 s target are.
- Industry profiles are a module, not store data; the blueprint wants them editable without a deployment. Three profiles over three products is a demonstration of the mechanism, not a study of personalised composition against a fixed layout.
- A quote started in the conversation takes two screens (the composed form, then the consent screen) against one on the prototype 1 form, because the layer writes as a signed agent (ADR-022). Collapsing it needs a signed-in person (ADR-004) or a first-party consent capture inside the composed form that the security model records.
- Conversation state is in-process memory with a 30-minute life; correct for one process, to move to a shared cache at the first multi-process deployment. The assistant's signing key is per process unless pinned.
- Trust was not measured with people. "Without hurting trust" is addressed by construction (disclosure on every conversation, every figure grounded or rejected, sources on every answer, a person one click away, nothing written without the visitor's own consent) and by the fallback suite, not by a survey or a behavioural measure.
What it means for the product candidate
Technical validation of the composition pattern and a measured no on the completion claim as framed. The prototype shows that a
model can compose a business's interface from a nine-component catalog with every figure bound to the store or verified,
within a cent per turn and three seconds per answer, failing to the conventional page and never writing without the person's
own consent, with cost, latency, fallbacks and rejections on the operator's dashboard. It does not show that a conversational
layer completes more tasks than navigation: on the reference site both are at the ceiling. The decision the study attached to
this prototype, whether to productise a generative layer, now has its cost input (about $0.007 to $0.011 per turn, $0.02 per
session at the assumed price) and its latency input (2.9 s median, 5 s p90), and lacks its value input. That input comes from
two places this prototype points at: a site large enough that navigation fails (the next thing to measure, with the same
harness and holdout), and real visitors on the SYSBOT_ASSIST_SHARE split with the lead rate by arm on /ops. The
personalised-composition mechanism is built and costs nothing extra per turn; it is the product-shaped part of this work and
the least measured.
Roadmap status after this prototype
| Prototype | Status |
|---|---|
| 1. Dual-audience reference site | Gate passed 30 Oct 2026 (98%); 100% in the prototype 3 run; 100% again on the navigation arm of this run (70/70 on seven tasks) |
| 2. Agent and citation analytics | Dashboard gate met; the four weekly probe runs complete on 23 Nov 2026 |
| 3. Agent gateway | Gate met 31 Oct 2026 |
| 4. Catalog-constrained generative UI | Measured 31 Oct 2026 (this report): cost, latency and fallback criteria met; completion level with navigation at 100%, so the improvement criterion is not met on this site. Next measurement needs a large catalogue or real visitors |
| 5. Citation-shaped content | In progress in a parallel session (matched content pairs); runs alongside the probe |
| 6. Policy and compliance kit | Waits for a paying customer |