Test 2026-08

Search is how you get into AI answers or Testing LLMS for cats.txt

TL;DR — I probed fifteen LLMs for the contents of the novel cats.txt file, with and without web search, on a single date. With search: 59–60 of 60 correct. Without search: zero, including from the four to five models that demonstrably hold other facts from the same domain in memory. A minority of models knew the file existed without searching — apparently from third-party commentary rather than the file itself — but not one could state anything it contained. Retrieval is the layer that responded. Training was not measurably touched. GEO, at least in this case, is search.

(Yes, this post is mostly AI generated; however, the test was human-designed & supervised. It also used a bunch of OpenRouter credits. This post is primarily a summary of the test results.)

The artefact

cats.txt is a satirical web standard. Mark Williams-Cook wrote the specification; Dave Smart published a conforming file at tamethebots.com/cats.txt. The file declares a handful of fields about a cat.

It exists as a comment on the evidence quality circulating in GEO discussions — Mark’s write-up of what happened next, How cats.txt showed llms.txt evidence is GEO astrology, and Search Engine Journal’s coverage lay out that argument. There is also a draft RFC-style document, a participants list and a growing pile of secondary coverage such as TrustPost’s explainer.

It also happens to be an unusually clean test artefact, for reasons that have nothing to do with cats.

To measure whether publishing something puts it “into the models,” you need content meeting four conditions simultaneously: recently published, on a site that is genuinely crawled, containing specific factual values, and not duplicated anywhere else on the web. Most commercial content fails the last condition badly. cats.txt meets all four.

A note on what this post withholds

This post does not state the field values in the file, and does not state the answer to the site-level control question either.

That is deliberate. This post will itself be crawled, indexed and quite possibly retrieved on queries about cats.txt. Publishing the answer key would inject the exact values the study probes for into the exact namespace it probes — contaminating the retrieval layer immediately and, potentially, the training layer later.

The values are in the published dataset for anyone verifying the work. They are not in the prose.

How the test works

The tool queries a pinned panel of fifteen models through OpenRouter. Every probe is sent twice to each model: once with web search enabled (Exa, ten results), once with no search available. Prompt, token limits and reasoning settings are held constant between conditions.

Each prompt asks for one short factual value and instructs the model to answer UNKNOWN if it does not know. That instruction matters — without it, caution and ignorance are indistinguishable in the output.

There are nine probes.

Four positive probes ask for field values appearing only in the file itself. These are the measurement.

Five controls rule out the ways this kind of test misleads you:

• An ingestion floor. Before concluding “this model doesn’t know the file’s contents,” you must rule out “this model doesn’t know the site.” One control asks for a biographical fact published on the same domain’s About page — unrelated to cats.txt, in ordinary HTML. A model that cannot answer this without search has no demonstrated knowledge of the domain, and its failure on the positive probes is uninformative. A second, weaker floor control asks for a fact appearing both on that site and on a third-party profile page.
• An absent-field control. Asks for a field the file does not declare. Catches models that invent rather than abstain.
• An absent-site control. Asks the same style of question about a domain publishing no cats.txt at all. Establishes the baseline confabulation rate.
• An existence control. A plain yes/no: does this site publish a cats.txt file? Separates knowing that something exists from knowing what it contains.

Grading is automatic, with raw response text stored verbatim alongside the search results each model received. Outcomes are classified as correct, incorrect, abstained, truncated, empty, tool-attempt or error — a model that timed out and a model that answered wrongly are not the same observation, and conflating them inflates apparent failure rates.

What the measurement covers

This limitation belongs before the results.

All runs were conducted on a single date. The tool went through several versions during development and each was executed against the live panel, producing multiple datasets. They are repeated measurements of one moment, not a time series. Where a finding held across repetitions, it means the observation is stable under re-measurement — not that it is stable over weeks.

Nothing here describes change over time. That requires scheduled monthly runs, which have not yet accumulated.

Results

With search: essentially perfect. 59 or 60 of 60 grounded positive cells returned the correct value, depending on the run. The single miss did not reproduce on re-run. Exa returned the file as the top result on nearly every grounded query.

Without search: zero. Across every version of the tool and every model in the panel, no ungrounded positive probe returned a correct field value.

Ingestion floor: four to five models of fifteen. These models answer the site-level control correctly without search, repeatedly. They demonstrably hold information from tamethebots.com in parametric memory. They are inside the zero above.

That is what makes the result interpretable. It is not that the models don’t know the site. Models that do know the site still produced nothing from the file.

Existence control: three models of fifteen answered yes without search. One did so consistently — roughly a third of ungrounded trials, abstaining on the rest. Two others produced a correct existence answer in the final run. When these models answered at all, they were not wrong.

An overview of the test results

What the models produce instead

Asked for the cat’s name without search, models offered generic cat names — Whiskers, Milo, Mittens, in one case Chairman Meow. Asked for its role, generic roles. Asked for a description detail, generic cat behaviours.

None are degraded or partial versions of the real values. There is no sign of a weak trace being reconstructed imperfectly. The pattern is what you would expect from a model with nothing to draw on, generating a plausible answer from priors about cats.

That distinction matters for interpretation: partial parametric memory tends to produce near-misses. This produced unrelated content.

Search is the channel

The two conditions differ by one variable, and the outcome flips completely. That is about as clean a result as field measurement produces.

Everything the models got right, they got right because a search engine returned the document. Nothing was recalled. The models that knew the domain best still recalled nothing. The file has been public, crawlable, and surrounded by third-party commentary — and the layer that responded was retrieval, on a timescale of weeks, while the training layer showed no measurable effect at all.

The practical consequence is that GEO’s working mechanism is search. Not a new discipline with new levers — the same crawlability, the same indexability, the same question of whether a retrieval system can find your document and prefer it over the alternatives. What changed is the surface the result appears on, not the machinery that decides which documents get considered.

Two things follow.

First, the levers are search levers. If a document is not crawlable, not indexed, or not the best match for the query, it does not enter the context window, and nothing downstream can rescue it. Everything the models said correctly in this test depended on Exa finding the file first.

Second, and this is the caveat that keeps the claim honest: this study measured retrieval in a namespace with no competition. There is exactly one document in the world containing these values. Retrieval could not fail. That demonstrates the channel works; it says nothing about winning contested commercial queries, where selection among many candidates is the entire problem. Retrieval being the mechanism does not mean retrieval is easy.

Two further findings

The existence/contents split has a plausible mechanism

Every grounded query returned a commentary ecosystem alongside the file: the spec site, the draft, the Substack post, the SEJ article, several SEO blogs, LinkedIn threads including Mark’s original announcement.

That commentary describes that cats.txt exists and what it satirises. It rarely reproduces field values from any individual conforming file.

That maps onto what the models show. The layer that was written about is the layer a minority can recall without search. The file’s own contents — in one place, quoted almost nowhere — were absent from all fifteen.

Consistent with the training signal being the commentary rather than the artefact. Not established by this data; the hypothesis that fits it.

Retrieval and citation are separable, and the difference is model-dependent

On identical queries, with identical retrieval sets containing the same document, some models cite the source inline and others do not. Claude Opus and Sonnet consistently attribute; GPT-5.6-sol, Gemini 3.7 Flash, Qwen 3.8 Max and Kimi K3 consistently use the retrieved content without naming it.

Same input. The difference is a property of the model, not of the content.

This has a direct measurement consequence. Any AI-visibility metric built on counting citations is partly measuring model attribution behaviour. Content can be present in the retrieval set and shaping the answer while registering zero citations, and two comparable sites can score differently depending on which models their queries reach.

What this does not show

One file, one site, one date. A case study. It does not generalise to content in general, and the single measurement date means nothing here describes change over time.

“Satirical” and “obscure” are confounded. Nobody deploys cats.txt, so there is no corpus of thousands of conforming files for a model to generalise from — unlike robots.txt, where there are millions. But a genuinely functional file with a single instance would produce identical results under this design. Separating the explanations needs a second arm: a real, low-traffic, single-instance file on the same domain with the same crawl exposure. That experiment has not been run.

Abstention is not proof of absence. Models are trained to decline on niche specifics. The ingestion floor mitigates this — floor-passing models will answer a question about this domain — but the floor question is a more conventional biographical fact than a field value from a novelty file, so the mitigation is partial.

The grading detects verbatim recall. A weak parametric trace that shifts token probabilities without surfacing as the top answer would not register.

The existence finding rests on few observations. Three models, a small number of gradeable trials each, on one date.

What to do with this

• Treat AI visibility work as search work. Crawlability, indexability, unambiguous phrasing and stable URLs are the levers this study could observe moving. There is no separate GEO mechanism in this data.
• Ask what evidence separates training from retrieval. If a vendor claims your content is “in the models,” this case is a counter-example: crawled, retrieved, surrounded by commentary, with zero field-level recall in any of fifteen models on the date tested.
• Segment citation counts by model before reading them as visibility. A change in the number may be a model’s attribution behaviour, not your content’s presence.
• Don’t infer competitive performance from this. The effect was measured where there was no competition. That is the study’s cleanest condition and its biggest limit.
• Treat retrieval-derived visibility as reflecting the current index. This study provides no evidence about durability.

Verification

The tool, model panel, probe definitions and full result set are published, including every raw response and the search results each model received. Responses are stored verbatim, so the corpus can be re-graded without further API calls.

The field values withheld from this post are in that dataset. If your reading of the results differs from mine, the rows are available — which is broadly the point cats.txt was making in the first place.