After 500+ A/B tests, I started testing the interface no human sees
Modern web products expose a second interface to crawlers, answer engines, and agents. I call it the shadow interface, and propose a testable funnel for evaluating it.
One product. Two interfaces.
Supported OpenSquid facts shown as a controlled illustration, not benchmark output.
The Shadow Funnel
Select a stage to inspect the question, silent failure, and reproducible test.
Can the consumer access a stable response?
Infrastructure, authentication, redirects, or directives block the journey before meaning is available.
Inspect the unauthenticated HTTP response, status, canonical path, and crawler directives.
Between 2021 and today, I helped deliver more than 500 A/B and multivariate experiments. The work trained me to see a product as a sequence of decisions: Can someone see the value, understand it, trust it, and act?
Then I started building systems for a different kind of visitor.
Crawlers, answer engines, retrieval systems, and browser agents assemble an interface from server responses, rendered DOM, accessibility semantics, JSON-LD, entity relationships, and retrievable passages. That interface can fail while the screenshot looks perfect.
The thesis is simple: a website now has two conversion funnels, and most teams measure only the human one. The interactive lab above shows the distinction between the visible surface and the structured representation a machine can resolve, retrieve, verify, and potentially refer.
I call that second representation the shadow interface.
The experiment I was not running
Traditional experimentation begins after a person reaches the product: the interface earns attention, creates understanding and trust, then enables an action. That path still matters, but it is no longer the only one.
An answer engine may inspect a product before a person knows it exists. A browser agent may build a shortlist. A retrieval system may isolate one paragraph from its layout. If that machine journey fails early, the human session may never happen.
I had spent years testing the visible interface while treating the machine representation as metadata. It is also interface state for a different consumer.
What I mean by a shadow interface
A shadow interface is the actionable model a machine constructs from a product surface. It is not one artifact. Different consumers combine different parts of the HTML, DOM, accessibility semantics, JSON-LD, screenshots, APIs, and retrieved text.
I use shadow funnel for the six-stage journey through that interface. Perception parity asks whether a person and a named machine consumer can reach the same supported conclusion. Representation parity asks whether the underlying artifacts encode required facts consistently. The latter is an engineering proxy for the former, not a synonym.
A product can look consistent while its representations disagree. A visible card may state a license that JSON-LD omits. A comparison table may associate a metric with a product while an extracted passage loses that relationship.
Your screenshot is not your product's machine interface.
Accessibility technology, automated testing, browser agents, retrieval systems, and answer engines have different needs, but they share one failure class: meaning exists in one representation and disappears in another.
What is new here, and what is not
The component practices already exist. Technical SEO tests crawling and rendering. Structured-data engineering tests entities. Accessibility testing compares visual and semantic interaction. Information retrieval measures evidence recovery. Agent evaluation measures task completion and unsupported output.
The proposed contribution is their synthesis: order representation failures as one journey, then govern the facts crossing those representations as a platform contract. The rule is one supported fact, one owner, explicit adapters. Each load-bearing fact gets a canonical source, declared representation requirements, and gates that detect drift.
The label earns its keep only if this sequence exposes dependencies or gaps that isolated audits miss. If an unordered checklist explains the same failures just as well, the new terminology adds no diagnostic value and should be discarded.
Two funnels, one product
The human funnel is familiar:
See → Understand → Trust → Act
The shadow funnel has six stages:
Fetch → Render → Resolve → Retrieve → Verify → Refer
| Stage | The machine must be able to | Common silent failure |
|---|---|---|
| Fetch | Access a stable response | Authentication, directives, or infrastructure block the consumer |
| Render | Perceive the meaningful content | Critical content exists only after unsupported client execution |
| Resolve | Identify entities and relationships | Conflicting identifiers split one entity into several |
| Retrieve | Isolate the relevant evidence | Generic or fragmented copy loses the answer-bearing passage |
| Verify | Connect claims to support | Visible claims and structured claims disagree |
| Refer | Cite, recommend, route, or hand off | Canonicals, actions, or attribution paths are missing |
The shadow funnel can precede, influence, or run beside the human funnel. An answer engine may create the first impression before a visitor sees the site. An agent may exclude a product because it cannot recover a capability that is obvious in the UI. The failure happens before the visible funnel begins.
Why screenshots lie
Teams review what they can see: design frames, browser QA, staging URLs, and analytics after client code runs. A machine consumer may receive a different artifact at a different time.
A server can return a shell while client code renders the offer and injects JSON-LD. Crawlers, agents, and assistive technologies may each receive a different subset of the final state. A screenshot collapses those differences into one reassuring image.
Shadow-interface testing therefore compares the initial response, rendered DOM, accessibility semantics, structured graph, retrievable passages, and available actions. The useful question is not merely, "Is the schema valid?" It is, "Can this consumer reach the same supported conclusion a person can?"
A design system for meaning
My frontend-platform work taught me that visual consistency does not survive through documentation alone. It survives when teams encode decisions into reusable APIs, constraints, tests, and migration paths.
Machine meaning needs the same treatment.
If a product name, capability, license, author, or result is independently authored in the UI, JSON-LD, agent index, metadata, and documentation, those representations will drift. The solution is not a larger content checklist. The solution is a shared domain model with explicit adapters.
The following TypeScript is an illustrative parity contract, not the API of a published package. It shows the part the smaller sketch omitted: how a build gate can distinguish a missing fact, a changed fact, and an unsupported fact.
type Surface = "ui" | "jsonld" | "agent"
type FactId =
| "product.name"
| "product.license"
| "product.author"
| "product.capability.mcp"
type CanonicalFact = {
value: string
requiredIn: Surface[]
evidence: string
}
type Representation = {
surface: Surface
facts: Record<string, string>
}
const canonical: Record<FactId, CanonicalFact> = {
"product.name": {
value: "OpenSquid",
requiredIn: ["ui", "jsonld", "agent"],
evidence: "package metadata"
},
"product.license": {
value: "MIT",
requiredIn: ["ui", "jsonld", "agent"],
evidence: "LICENSE"
},
"product.author": {
value: "https://smlee.dev/#person",
requiredIn: ["jsonld", "agent"],
evidence: "canonical Person @id"
},
"product.capability.mcp": {
value: "MCP runtime",
requiredIn: ["ui", "jsonld", "agent"],
evidence: "public documentation"
}
}
function validateRepresentationParity(
source: Record<FactId, CanonicalFact>,
representations: Representation[]
) {
const failures: string[] = []
for (const [id, fact] of Object.entries(source)) {
for (const surface of fact.requiredIn) {
const observed = representations.find(
representation => representation.surface === surface
)?.facts[id]
if (observed === undefined) {
failures.push(${surface} is missing ${id})
} else if (observed !== fact.value) {
failures.push(
${surface} changed ${id}: expected "${fact.value}", received "${observed}"
)
}
}
}
for (const representation of representations) {
for (const id of Object.keys(representation.facts)) {
if (!(id in source)) {
failures.push(${representation.surface} introduced unsupported ${id})
}
}
}
return failures
}
const failures = validateRepresentationParity(canonical, [
{ surface: "ui", facts: extractFactsFromRenderedHtml(html) },
{ surface: "jsonld", facts: extractFactsFromJsonLd(graph) },
{ surface: "agent", facts: extractFactsFromAgentIndex(index) }
])
if (failures.length) throw new Error(failures.join("\n"))
The extractor names are deliberately unimplemented placeholders, and they hide most of the production engineering. They must inspect the artifacts each consumer actually receives, not simply call the same serializer three times. A useful gate renders the page, reads the emitted JSON-LD, reads the built agent index, and compares their recovered facts with the canonical contract.
A production implementation also needs surface-specific parsers, normalization rules, relationship extraction, provenance, and versioned fixtures. Without those pieces, the parity function validates only hand-authored maps and proves little about the shipped interface.
The exact adapters differ by system. The architectural principle does not.
One supported fact should have one owner. Human-facing components and machine-facing representations should derive from it. Tests should fail when an adapter drops a required relationship, changes a supported value, or introduces a claim with no canonical evidence.
This is the layer where design systems, structured data, accessibility, and AI reliability meet. Each has to preserve supported meaning across representations.
The 15-minute shadow-interface test
You do not need a new analytics platform to find the first failures. Start with one important page and one supported question it should answer.
Minute 1 to 3: Fetch. Request the page without a browser. Check status, canonical behavior, directives, and whether the response contains meaningful content.
Minute 4 to 6: Render. Compare the initial response with the hydrated DOM. Identify claims, actions, or structured data that exist only after client execution.
Minute 7 to 9: Resolve. List the primary entities and their identifiers. Check whether the same Person, Organization, Product, or Article is represented consistently across the page.
Minute 10 to 11: Retrieve. Extract the smallest passage that answers the supported question. If the passage loses its subject or evidence when removed from the layout, retrieval is fragile.
Minute 12 to 13: Verify. Compare visible claims with structured claims. Confirm that metrics, relationships, and descriptions are supported by the page.
Minute 14 to 15: Refer. Check whether the consumer can cite a stable URL, attribute the claim, and route a person to the appropriate next action.
The result is not a universal score. Different machine consumers have different capabilities. The useful output is a representation gap you can reproduce and assign.
Download the reusable Shadow Interface 15-minute test, version 0.1.
What to test first
Full parity across every fact and representation is usually a poor first target. Start with facts and actions whose failure can change eligibility, attribution, trust, or completion: entity identity, canonical URLs, the primary offer, critical constraints, supporting evidence, and the next action.
Prioritize each candidate by four questions: Does it affect a consequential decision? How often does it change? Across how many representations is it repeated? How costly would silent drift be? High-impact, high-change facts deserve canonical ownership and release gates. Lower-impact facts can use scheduled sampling rather than blocking every deployment.
Scope the work to named consumers. A documentation retrieval adapter might preserve section titles, canonical URLs, and subject-bearing passages. A commerce product-page adapter might prioritize stable product identifiers, current offer state, and visible supporting evidence. A browser agent may additionally need accessible names and deterministic action states. Build only the adapters that serve an observed journey or defined evaluation target.
What would make this scientifically established
The Shadow Funnel is a working model. Testing whether its stages are distinct and useful requires a study designed to prove it wrong.
I would preregister a benchmark across commerce, SaaS, publishing, and documentation sites. Blinded annotators would identify the facts, relationships, evidence, and actions a person can verify. The same tasks would run against initial HTML, rendered DOM, accessibility semantics, JSON-LD, retrieved passages, and agent-available actions.
Results should remain a vector: visible-fact recall, relationship precision, action-completion success, and unsupported-claim rate. The study should publish agreement levels, model versions, prompts, rendering conditions, and collection dates.
The ordered funnel must also outperform an unordered checklist built from the same SEO, accessibility, and structured-data checks. It gains support only if its stages explain or predict held-out outcomes beyond that baseline. Inconsistent stages, indistinguishable variables, or equal checklist performance would count against it. Scientific establishment would still require independent replication.
Research foundations for the method, not the model
These references support how the proposed benchmark could measure its variables, not whether the model is valid. The design should build on established methods rather than inventing convenient measurements. Online experiment design can follow Kohavi, Longbotham, Sommerfield, and Henne's practical guide to controlled web experiments. Retrieval evaluation can use the gain-based measures defined by Järvelin and Kekäläinen. Entity matching methodology can draw from Peter Christen's Data Matching. Structured representation collection should follow the JSON-LD 1.1 recommendation, while accessibility-semantics interpretation should be grounded in the W3C Core Accessibility API Mappings.
What changes for product teams
Shadow-interface failures originate across the stack. Product defines claims and actions. Design establishes visible hierarchy. Frontend determines rendering and semantics. Backend owns data contracts. Platform teams create reusable adapters. Content provides retrievable evidence. AI teams evaluate what models recover.
Assign one accountable owner while distributing implementation across those teams. Encode the highest-priority requirements in component APIs, architecture decisions, automated tests, and release gates, as teams already do for accessibility and performance.
The next experiment surface
After 500+ experiments, I still care about human judgment, comprehension, and trust. But the visible interface is no longer where every journey starts.
A machine may encounter the product first, assemble a comparison, choose evidence, or hand the task to a person. That journey has stages, and its failures can be tested.
The shadow interface is a product surface. It is time we experimented on it.