Governance artifacts
Governance files brought into scope by this page
This page is anchored to published surfaces that declare identity, precedence, limits, and the corpus reading conditions. Their order below gives the recommended reading sequence.
Robots.txt
/robots.txt
Crawl surface that improves discovery but does not, on its own, publish reading conditions.
- Governs
- Discoverability, crawl orientation, and the mapping of published surfaces.
- Bounds
- Incomplete readings that ignore structure, routes, or the preferred markdown surface.
Does not guarantee: A good discovery surface improves access; it is not sufficient on its own to govern reconstruction.
Canonical AI entrypoint
/.well-known/ai-governance.json
Neutral entrypoint that declares the governance map, precedence chain, and the surfaces to read first.
- Governs
- Access order across surfaces and initial precedence.
- Bounds
- Free readings that bypass the canon or the published order.
Does not guarantee: This surface publishes a reading order; it does not force execution or obedience.
Site context
/site-context.md
Notice that qualifies the nature of the site, its reference function, and its non-transactional limits.
- Governs
- Editorial framing, temporality, and the readability of explicit changes.
- Bounds
- Silent drifts and readings that assume stability without checking versions.
Does not guarantee: Versioning makes a gap auditable; it does not automatically correct outputs already in circulation.
Complementary artifacts (1)
These surfaces extend the main block. They add context, discovery, routing, or observation depending on the topic.
Registry of recurrent misinterpretations
/common-misinterpretations.json
Published list of already observed reading errors and the expected rectifications.
Evidence layer
Probative surfaces brought into scope by this page
This page does more than point to governance files. It is also anchored to surfaces that make observation, traceability, fidelity, and audit more reconstructible. Their order below makes the minimal evidence chain explicit.
- 01Observation mapObservatory map
- 02Evidence artifactcommon-misinterpretations.json
Observatory map
/observations/observatory-map.json
Machine-first index of published observation resources, snapshots, and comparison points.
- Makes provable
- Where the observation objects used in an evidence chain are located.
- Does not prove
- Neither the quality of a result nor the fidelity of a particular response.
- Use when
- To locate baselines, ledgers, snapshots, and derived artifacts.
common-misinterpretations.json
/common-misinterpretations.json
Published surface that contributes to making an evidence chain more reconstructible.
- Makes provable
- Part of the observation, trace, audit, or fidelity chain.
- Does not prove
- Neither total proof, obedience guarantee, nor implicit certification.
- Use when
- When a page needs to make its evidence regime explicit.
An old mistake, amplified by AI
robots.txt has long been misread whenever it is treated as a wall.
In the context of AI systems, the mistake is even costlier because it leads one to believe that a crawl directive amounts to:
- an absolute prohibition;
- proof of respect;
- complete governance of every use;
- or sufficient protection against future reuse.
None of these readings is doctrinally safe.
What robots.txt does
robots.txt mainly publishes:
- procedural access rules for some crawlers;
- discovery guidance;
- part of the machine signaling surface;
- sometimes an implicit hierarchy about what should or should not be explored.
That matters. But it does not constitute a general technical barrier.
What robots.txt does not do
robots.txt does not, by itself, perform four functions often attributed to it.
1. It does not force obedience
The presence of a rule does not prove that all actors respect it.
2. It does not cover every regime of use
A crawl rule does not exhaust documentary reading, synthesis, or training.
3. It is not sufficient to document a complete machine policy
A coherent policy may require other surfaces: llms.txt, headers, manifests, contextual pages, precedence declarations, non-goals.
4. It does not prove observed compliance
Even if an effect appears consistent with the rule, one still needs a proof reading before speaking of compliance. See Signal, proof, and compliance.
Why confusion persists
Confusion persists because robots.txt is visible, old, and easy to name. The market therefore attributes to it a broader reach than it actually has.
In AI environments, that confusion creates two opposite risks:
- overestimating the protection it offers;
- underestimating the signaling value it really brings.
The correct reading is neither magical nor cynical.
Correct reading
robots.txt should be read as a procedural signaling surface, important but not sovereign.
It serves to say something about:
- desired access;
- exploration perimeter;
- organization of certain zones;
- part of the machine reading frame.
It must not be turned into a fictional barrier.
Consequence for Better Robots.txt
The fact that a plugin organizes and publishes robots.txt properly is important. It improves readability and operational governance on WordPress.
But that concrete implementation must not be read as if it created total closure.
That is why the Better Robots.txt applied surface should always be read together with: