LLM perception drift measurement protocol
This protocol turns “the brand is misunderstood” into reproducible observation. It separates variation, isolated error, persistent drift and legitimate source disagreement.
1. Define the object and invariants
Establish the entity, identifiers, current state and dimensions: identity, category, scope, differentiation, relations, time, source authority, sentiment and recommendability. Date and bound the canon. Keep external reputation claims separate.
2. Build prompt families
Use identification, generic discovery, comparison, problem, verification, risk, alternatives and history. Include both brand-named and generic prompts. Avoid leading questions designed to force the expected canon.
3. Fix experimental conditions
Record product and model, date and time, account or session, language, region, browsing, memory, available system prompt, accessible sampling parameters, protocol version and corpus version. Explicitly state what consumer interfaces do not expose.
4. Sample
One answer is insufficient. Repeat material prompt families within a defined window and across systems when risk justifies it. Increase repetition and condition diversity with claim severity and decision proximity. Do not merge languages or browsing modes.
5. Capture outputs and sources
Preserve full answer, citations, links, source order and warnings. When sources are not visible, record “source not observable” rather than inventing retrieval. Sensitive claims require export, capture or a hashed log rather than memory.
6. Code dimensions
Classify each dimension as faithful, partial, contradictory, not observable, legitimate external, or inconclusive. Add a drift type using the brand representation taxonomy and severity using the brand safety matrix when required.
7. Separate sentiment and fidelity
Code sentiment independently. Output may be positive and false, negative and faithful, neutral and dangerous. Sentiment never replaces category, attribution or time analysis.
8. Calculate useful gaps
Possible metrics include correct identity rate, category stability, scope preservation, differentiator omission, relation accuracy, temporal freshness, correctly attributed claims, recommendability by intent, and cross-model or cross-language divergence.
Always surface critical incidents beside aggregate scores. A 90% score may conceal a serious false accusation in the remaining 10%.
9. Establish the baseline
A baseline is a dated observation set, not a permanent portrait. It documents behaviour before correction, redesign, rebrand or campaign. Preserve existing errors rather than cleaning history retrospectively.
10. Escalation thresholds
Monitor isolated low-impact gaps. Diagnose repeated category or scope gaps. Correct entity confusion quickly. Escalate severe unattributed claims. Apply source classes to official-external conflict. Re-observe before claiming improvement.
11. Correct the competent layer
Interventions may target canon, information architecture, entity relations, freshness, schema, scope pages, third-party sources, redirects or provider reporting. Do not publish generic positive content for a precise attribution failure.
12. Re-observe and qualify causality
Replay the protocol under comparable conditions. Record changes, non-changes and alternatives such as model updates, new sources, variability, region or seasonality.
Use restrained outcomes: no observed change; variation compatible with hypothesis; repeated improvement after correction; strong association without isolated causality; or inconclusive.
Deliverables
The protocol produces a prompt inventory, output log, baseline, gap matrix, drift taxonomy, source map, correction plan and re-observation report.
It does not produce a promise of control. Its value is making the gap measurable, contestable and revisable.