Sysbot Technologies

Technical prototype 3 · report

Agent gateway

Agents using tools complete tasks at least as well as through the HTML path, with hand-off to a person in one screen and no authorisation bypass.

Status
Gate met · 31 October 2026
Built on
The AI-era reference site
Result
300 executions: 100% completion on browser, REST and MCP; 20 of 20 hand-offs in one screen; 20 of 20 policy checks held.

Built 31 October 2026 on the prototype 1 reference site and the prototype 2 event store. Roadmap gate (row 3): task completion rate and steps per task for agents using tools versus the HTML path, on the same 150-execution protocol as prototype 1; human hand-off completes in one screen. The blueprint's prototype mapping adds: no authorisation bypass found by the harness.

What was built

Piece Where What it does
Operation registry packages/operations Five operations defined once with JSON Schemas for input and output, kind (read or write), idempotency, side effects, consent requirement, rate class and freshness tier: search_products, check_availability, lookup_business_info, request_quote, book_service_visit. The Inquiry lifecycle of ADR-016 (closed states, audited transitions, 72-hour duplicate window, erasure in terminal states). The roadmap named three operations; the two added are the ones the five tasks and a real business need, and the registry is the only place they exist.
Security model packages/security Authentication (a Web Bot Auth signature verified against the agent's published key, reusing the prototype 2 verifier, or a delegation bearer token a person issued), the v1 authorisation matrix as versioned data (reads open; writes for signed agents with a fresh, unreplayed signature, or delegations in scope; anonymous writes challenged), rate limits per rate class and caller, a 250 ms decision budget that fails closed (ADR-015), and the security records: single-use consent tokens hashed at rest, consent records by hashed subject, delegations, idempotency keys.
Gateway packages/gateway, hosted in apps/web One invoke pipeline (authenticate, authorise, validate, check every named entity belongs to the tenant, honour the idempotency key, execute, validate output, write one operation_call event) projected as REST (POST /api/v1/operations/{name}, RFC 9457 problem details, OpenAPI 3.1 at /api/v1/openapi.json) and MCP (/api/mcp, streamable HTTP, stateless, tools and resources generated from the registry). Product feed and a discovery-only commerce manifest at /.well-known/ucp. ADR-019 records the design.
Consent flow apps/web/app/consent/[token], apps/web/app/agents/authorise A write on either transport stops at awaiting_consent and returns a single-use confirmation URL (72 hours, never containing the Inquiry id). The requester opens it, sees exactly what the agent asked for, and approves or declines in one screen; approval records consent, submits the Inquiry and emits the lead, attributed to the agent that filed it. /agents/authorise lets a person delegate named operations to an agent ahead of time, with consent recorded, so that agent's requests for that person submit directly. ADR-005 records the trust decision.
Dashboard apps/web/app/ops New "Agent gateway" section: calls by operation, surface and outcome with latency percentiles; refusals by rule and caller; requests awaiting consent. Every call and every refusal is an event in the prototype 2 store, so the existing traffic, agent and lead attribution tables include gateway traffic.
Agent guide apps/web/app/agents The human-readable page the blueprint asks for: endpoints, operations, rate classes, identity accepted, how a write completes. Served as Markdown too.
Harness apps/probe The same five tasks through three paths with a deterministic baseline and a real agent on each: scripted-a11y and llm-browser (prototype 1), scripted-rest, llm-rest, scripted-mcp, llm-mcp. The tool agents see only the gateway's own tool definitions (the OpenAPI document; MCP tools/list through the official client SDK). After a gateway write the harness plays the requester on the consent screen and counts screens. A policy check tries twenty attacks on the authorisation matrix over both transports before the run. The report compares the paths on completion, incorrect actions, refusals, steps, latency per call and per task, tokens and cost per task, and hand-off.

Evidence

Run of 31 October 2026, 300 executions (six agents, five tasks, ten repetitions), one model (Claude Sonnet 5.5) on every path, all agents signing as agents.sysbot.test. Report: apps/probe/reports/2026-10-31T07-42-49-609Z.md.

Path Agent Completion Incorrect action Refusals Median steps per task Median ms per call Median ms per task Median tokens per task Mean cost per task Hand-off
browser llm-browser (real agent) 100% (50/50) 0 – 2 – 3,843 5,490 $0.024 consent inside the form (prototype 1 design)
browser scripted-a11y (baseline) 100% 0 – 3 – 280 – –
REST llm-rest (real agent) 100% (50/50) 0 0 3 13 4,031 8,150 $0.032 10 of 10 in one screen; URL handed over in the answer 10 of 10
REST scripted-rest (baseline) 100% 0 0 2 4 6 – – 10 of 10 in one screen
MCP llm-mcp (real agent) 100% (50/50) 0 0 3 13 3,987 8,175 $0.032 10 of 10 in one screen; URL handed over 10 of 10
MCP scripted-mcp (baseline) 100% 0 0 2 5 19 – – 10 of 10 in one screen

The write task on its own, where the paths differ most:

Path (real agent) Median steps Median ms per task Median tokens per task Mean cost per task What the person does
browser 9 8,057 10,407 $0.044 nothing: the agent filled the form and ticked consent on the person's instruction
REST 3 (2 operation calls) 5,940 12,113 $0.042 opens the link the agent returned, one screen, two clicks (agree, approve), 155 ms
MCP 3 (2 operation calls) 6,042 12,118 $0.044 the same

Cost is derived from measured tokens at an assumed $3 per million input and $15 per million output tokens; the report flags the assumption and the token counts are the measured number.

Policy check: 20 of 20 held on both transports (anonymous writes challenged with a problem document that names both identity routes; replayed signature refused; stale signature and unknown key degrade to anonymous and are challenged; idempotency key returns the first result; entity outside the tenant is not found, never forbidden; invalid input lists its fields; no order operation exists to call; anonymous check_availability rate limited with Retry-After; a delegation issued on the first-party screen submits in scope, is denied out of scope, and is refused for another person's request; the consent link is single use; MCP and OpenAPI list the same operations).

Operator dashboard, agent gateway section, after the run: calls by operation, surface and outcome; refusals by rule; requests awaiting consent
Operator dashboard, agent gateway section, after the run: calls by operation, surface and outcome; refusals by rule; requests awaiting consent

Gate status

  • Completion and steps per task, tool paths versus HTML path, 150-execution protocol: met and measured. A real agent completed 50 of 50 on each of the three paths; steps per task went from 9 to 3 on the write task and were level (2 to 3) on the four read tasks.
  • Human hand-off completes in one screen: met. 20 of 20 agent-filed quote requests were approved by the requester on one screen with two clicks, and in every run the agent handed the confirmation URL over in its answer.
  • No authorisation bypass found by the harness: met. 20 of 20 policy checks held on both transports; zero incorrect actions in 300 executions.

Findings worth carrying into the blueprint

  1. On a small site the gateway does not save tokens; it saves actions and removes the form. A product page's ARIA snapshot already carries the answer, so the browser agent answered four read tasks in two steps at 5,500 tokens, while the tool agents spent 8,000 on the same tasks because five tool definitions with full schemas ride on every turn. The gateway's win is on the write: nine browser actions through a form became two typed calls, with no form to break (prototype 1 lost a run to a label association). The token comparison will turn the other way on a site with hundreds of products, where a page no longer fits a snapshot; that is the condition to measure next, not assume.
  2. REST and MCP are the same thing to an agent. Same completion, same steps, same tokens within 1%, same latency within a few milliseconds per call. The registry-to-projection design (ADR-019) is why: the model sees identical schemas and descriptions either way. Transport is a question of which clients can reach the site, not of behaviour; offering both costs one adapter file each.
  3. Determinism is the product, latency is not the argument. The baselines complete in 6 ms (REST) and 19 ms (MCP) against 280 ms for the accessibility-tree baseline, but a real agent's task takes about four seconds on every path, all of it model time. What the gateway changes is that the agent's calls are validated, authorised, audited and idempotent, and that a wrong call is a typed error instead of a wrong click.
  4. A write needs a per-request signature. Prototype 2 let the harness sign once per run because the Web Bot Auth profile covers only the authority. A write policy that refuses stale and replayed signatures (ADR-015) means agent operators must sign each write with a fresh nonce; the guide at /agents says so, and the policy check proves the replay is refused.
  5. The consent screen is one screen because the record is already complete. Everything the person sees was validated when the agent called; the screen adds nothing but the decision. The agent returned the URL verbatim in 20 of 20 runs after one sentence in the system prompt; the harness also has a fallback for the contact-channel delivery ADR-016 specifies, which was never needed.
  6. Refusals are now a dashboard column. Every challenge, denial, rate limit and validation failure is an event with its rule, so an operator can see who is trying what; this is the first measurable view of agent write intent, and the evidence the agent access economics question needs.

Known limits

  • One model on every path; the three "agents" of the protocol differ in kind (browser, REST, MCP), not vendor. Cost is at an assumed price. Prompt caching was not enabled for any path, so the token numbers are pessimistic for all three and most for the tool paths, whose definitions repeat per turn.
  • The reference business has three products, one service and three FAQ answers; search ranking is a token overlap, and the back-office adapter answers from the published entity while reporting source: back-office. ADR-009 stays open.
  • Consent links are returned to the agent only; delivery to the contact channel (email magic link, SMS code) is not built, so control of the channel is not proven. The harness plays the requester.
  • Rate limiting and nonce replay state are in-process; idempotency keys live in the store. Correct for one process, to be moved to the edge or a shared cache at the first multi-process deployment.
  • The commerce manifest's field layout follows the discovery-only posture of ADR-006 but has not been validated against the UCP specification; the product feed is JSON with Merchant Center attribute names, not a submitted feed.
  • Staff identity is still the OPS_TOKEN gate; ADR-004 (identity provider for people) remains open. Single tenant, single deployment.
  • Five operations rather than the roadmap's three; nothing moves money or changes an account, as the roadmap requires.

What it means for the agent-gateway product candidate

Technical validation, not product validation. The prototype shows that a business's operations can be defined once, exposed to any assistant over the two transports that matter today, called by a real agent with the same success as the human site, guarded by rules the harness could not bypass, and completed by the person in one screen, with every call and refusal visible to the operator. It does not show that any assistant outside this harness will call it, or that a customer will pay for it. The next step is the one the prototype 2 dashboard was built for: put the gateway on a real Sysbot client site, leave the claimed-versus-verified and refusal tables running, and see whether assistants arrive, what they try, and whether their leads convert. The rate-limit and refusal rows are also the first data the agent access economics question (who pays for agent traffic) can be argued from.

Roadmap status after this prototype

Prototype Status
1. Dual-audience reference site Gate passed 30 Oct 2026 (98%); re-measured in this run at 100%
2. Agent and citation analytics Dashboard gate met; the four weekly probe runs complete on 23 Nov 2026
3. Agent gateway Gate met 31 Oct 2026 (this report)
4. Catalog-constrained generative UI Next; catalog and generative remain typed stubs; the generative layer reaches data only through this registry
5. Citation-shaped content Not started; runs alongside the probe
6. Policy and compliance kit Waits for a paying customer

Research in practice

Want this built for your business?

The prototypes are the engineering behind our AI intelligence and digital experience work. Write with the problem and we reply within one business day.