Built 31 October 2026 on the prototype 1 reference site and the prototype 2 citation probe. Roadmap gate (row 5): a measurable difference in citation frequency between matched versions across five assistants after four weeks. Hypothesis under test, from Shift 1: front-loaded, dated, statistic-rich pages earn more assistant citations than the same facts written conventionally.
What was built
| Piece | Where | What it does |
|---|---|---|
| Matched content pairs | packages/citation-experiment/src/topics.ts |
Ten topics, one per buyer question. Each topic records its facts once (a quotable statement, the tokens that must appear in both versions, and a first-party source where the fact is a statistic) and carries two hand-written versions. The shaped version has an answer-first summary, a visible publication date, and every sourced statistic as a citedFact block with its source and date. The plain version has the same facts in conventional prose: context first, the answer in the middle, no summary, no visible date, statistics inside sentences without a source. Topics come from the five questions the probe already asked, extended to ten: delivery time, NL-2200 price, overlock choice, on-site service, lockstitch supplier, installation included, service contract cost, cutting machine, direct drive versus clutch motor, warranty and returns. |
| Equivalence tests | packages/citation-experiment/test |
Both versions of every pair contain every marker of every fact; both contain exactly the same set of numbers (prices, percentages, counts, years in prose); the shaped summary carries the answer fact and the plain version's opening block does not; every sourced fact is cited verbatim in the shaped version and never in the plain one; all twenty documents validate against the document schema. |
| Seeded assignment | packages/citation-experiment/src/assign.ts, assignment.json |
Each topic has two neutral URL and title slots. A seeded coin (mulberry32, seed 20261031) decides which slot carries the shaped version, and a seeded shuffle fixes the listing order of all twenty pages. The record is committed; a test regenerates it from the seed and requires equality, so the assignment is auditable and cannot drift. Six of ten topics landed shaped in the second slot. |
| Publication | apps/web |
The twenty pages are ordinary store documents (a new optional experiment field on the document schema names the experiment, topic and variant), seeded beside the reference business and served at /guides/<slug> with the usual HTML, JSON-LD and Accept: text/markdown negotiation; a new /guides index lists them in the recorded order. The presentation variables are bound to the variant in every projection: the plain version shows no date in the page, the Markdown front matter or the JSON-LD, and no answer-first description; the shaped version shows all three. Nothing else differs between the two routes. |
| Probe extension | apps/probe/src/citation |
citation.config.json names the experiment. On every run the probe asks the matched question of each pair (a question that is also a prototype 2 query is asked once and scored both ways) and scores each version: cited (its URL among the answer's citations, with rank), mentioned (its title in the answer text), surfaced (retrieved but not cited). The outcome per pair-answer is shaped, plain, both or neither. The report adds a per-pair table per assistant, an aggregate per assistant for the run, and the same aggregate over every run so far, with the shaped-minus-plain citation rate and an exact McNemar test on the discordant pairs. Pair observations are posted to the dashboard as citation events with the matched question and the detail in data.experiment. |
| Schedule | unchanged | The weekly job (GitHub Actions Mondays 03:30 UTC, or the launchd agent Mondays 09:00 local) runs npm run citations, which now runs the experiment because the config names it. No manual step. |

Method
- Unit of observation. One assistant answering one matched question in one run, scored against both versions of the pair. Ten pairs, five assistants and four weekly runs give 200 paired answers; the baseline run adds ten more per assistant that ran.
- Difference measure. Shaped citation rate minus plain citation rate, in percentage points, over all paired answers, and per assistant.
- Significance. McNemar's exact test on the discordant pairs (answers that cited only the shaped version, b, against answers that cited only the plain version, c). It is the appropriate test for paired yes/no outcomes and makes no assumption about the topics. The report prints the two-sided p and reads it in one line: with fewer than six discordant pairs it says the data cannot tell; at p below 0.05 it says the difference is unlikely to be chance.
- Controls. Facts recorded once and checked by test; versions assigned to neutral slots at random from a recorded seed; both versions published the same day on the same site with the same template; questions unbranded, so a mention is not handed to the assistant; the Markdown projection of a page does not count as a citation of it.
- Secondary measures. Mean rank among citations when cited; title mentions; surfaced-but-not-cited; competitors cited on the matched questions.
Evidence
Baseline run, 31 October 2026, Claude only (ChatGPT, Perplexity, Gemini and Grok skipped for want of keys), fifteen
questions: the five prototype 2 queries and the ten matched questions, of which three overlap. Report:
apps/probe/citations/2026-10-31.md.
| Measure | Claude (claude-sonnet-5-5), 31 Oct 2026 |
|---|---|
| Matched answers scored | 10 of 10, no errors |
| Shaped version cited | 0 of 10 |
| Plain version cited | 0 of 10 |
| Difference | +0 points; no discordant pairs, so no McNemar test yet |
| Title mentions, shaped / plain | 0 / 0 |
| Surfaced without citation, shaped / plain | 0 / 0 |
| Citations per matched answer | 61 in all, at least 4 per answer |
| Prototype 2 queries (series since 30 Oct) | cited in 0 of 5, unchanged |
| Competitors cited on the matched questions | vssewingmachine.com (4), dir.indiamart.com (4), indiamart.com (3), prizzisewing.com (2), tradeindia.com (2) |
| Median time per question | about 10 s |
The scoring was verified on real answers in the sense that matters at this stage: every answer was parsed, every citation list was non-empty (61 citations across the ten matched answers), no citation and no mention was wrongly credited to either version, and the per-pair, per-run and cumulative tables rendered from live data. The detection path (a cited URL matching a version's path, with rank, title mention, surfaced) is covered by unit tests against synthetic answers, including the subdomain, trailing-slash and Markdown-projection cases.
Gate status
- Measurable difference across five assistants after four weeks: started, not yet met. The experiment is published, scheduled and baselined. The four scheduled runs are Mondays 2, 9, 16 and 23 November 2026; the gate can be judged on 23 November 2026, from the "All runs so far" section of that day's report, which will state the cumulative shaped and plain citation counts, the difference in points, the discordant pairs and the McNemar p, per assistant and overall.
- Two conditions decide whether the number means anything. First, the other four assistant keys must be present
in the schedule's environment, or the gate is measured on one assistant. Second, and more important, the reference
site is served on localhost under a fictional domain. No assistant can retrieve, let alone cite, a page it cannot
reach, so until the site is on a public host the experiment measures zero for both versions and the four runs prove
only that the pipeline is unattended. The prototype 2 report carries the same caveat; the fix is the same, and it is
the next step below. Done on 31 October 2026: the site is deployed to https://northwind.sysbot.in and the probe
config names that host (
docs/publishing/reference-site.md), so the four scheduled runs measure real pages.
Findings worth carrying into the blueprint
- Equivalence is a test, not a claim. Writing two versions of the same facts by hand drifts within minutes; the number-set test caught a capitalised marker on the first run and would catch a statistic that appears in only one version. Any content system that promises "the same facts, differently presented" needs this check in its build.
- The presentation variables have to be bound to the content, not the template. Hiding dates for the plain
version meant touching the page, the Markdown front matter and the JSON-LD; a variable that leaks through one
projection is not controlled. The
experimentfield on the document is the mechanism; the renderers read it. - Random assignment needs a record, and the record needs a test. The seed alone is not evidence; the committed assignment plus a test that regenerates it is. Six of ten shaped versions landed in the second slot, which is what a fair coin looks like.
- A title the question already contains is not a mention the page earned. On the first baseline run Claude's answer to the warranty question "mentioned" the plain version, because its title, "Warranty and returns on industrial sewing machines", is a substring of the question. The scorer now applies the prototype 2 rule for brands to page titles (no credit when the question hands over the words), with a test; the baseline was rerun. Slot titles for future pairs should not echo the question.
- A matched question that is also a brand question is asked once. The probe de-duplicates, so extending the series to ten pairs added ten questions per assistant per run, not fifteen; at about ten seconds a question the run with five assistants stays under fifteen minutes.
- The cumulative table is the deliverable. Each weekly report carries the whole series so far, so the gate is read from one file on the day, and the difference measure and its significance are computed the same way every week by code that is tested, not by hand at the end.
Known limits
- Only Claude ran; the other adapters are exercised only when their keys exist. One assistant cannot meet a gate that names five.
- The site was not publicly reachable when the baseline ran, so the baseline measures a fictional domain. The site
was deployed to https://northwind.sysbot.in on 31 October 2026 (
docs/publishing/reference-site.md); the four scheduled runs measure it. Reachable is not indexed: discovery by the assistants' search indexes takes days to weeks. - Two near-duplicate pages per topic on one site may be deduplicated by an assistant's search; both versions are self-canonical by design. If assistants systematically cite only one page per site, the "both" outcome will be rare and the test still works on the discordant pairs.
- The first-party statistics on the pages are the fictional business's own measurements, in the same spirit as the thread-waste figure on the existing guide; they are there to test presentation, not to inform anyone.
- Freshness is held constant (both versions published the same day); the study's "keep key pages fresh" lever is not part of this experiment.
- The dashboard shows pair observations as citation rows keyed by the matched question; it has no experiment view of its own.
What it means for the product candidates
Nothing yet. The prototype shows that citation-shaped content can be specified as a controlled experiment inside the content store, published without a second template, and measured by the same unattended probe that already runs. The hypothesis itself is untested until the pages are reachable by the assistants. When they are, the same four weeks answer the question the study's Shift 1 raises for every customer: whether front-loaded, dated, sourced pages earn citations that conventional pages do not, and by how much.
Roadmap status after this prototype
| Prototype | Status |
|---|---|
| 1. Dual-audience reference site | Gate passed 30 Oct 2026 (98%); re-measured at 100% in the prototype 3 run |
| 2. Agent and citation analytics | Dashboard gate met; the four weekly probe runs complete on 23 Nov 2026 |
| 3. Agent gateway | Gate met 31 Oct 2026 |
| 4. Catalog-constrained generative UI | In progress |
| 5. Citation-shaped content | Built, published, scheduled and baselined 31 Oct 2026 (this report); gate judged 23 Nov 2026 |
| 6. Policy and compliance kit | Waits for a paying customer |