Status: DRAFT (v0.3)

Grounding‑Faithfulness & Contestability¶

A voluntary evaluation protocol for factual AI systems — public working draft, open for comment

Working Draft v0.3 · 2026‑08‑10 · Status: open for comment (a pilot evaluation protocol — not a certification scheme, not yet a standard) Author / editor: Robert Schaub (fact‑checker; FactHarbor) · Text licence: CC BY-SA 4.0 · Canonical home: https://github.com/robertschaub/our-ai-charter · Comments: GitHub Issues in this repository Companion to the manifesto "Trustworthy AI, Accountable to People."

Status of this document¶

For a short introduction, read the method summary; the Trust & Accountability overview distinguishes this method from the broader Charter and future certification.

This is an early, openly‑published draft, released to be argued with — not a finished standard, and not something anyone is yet certified against. It has been pressure‑tested once by an adversarial multi‑model review and revised accordingly, but it has not been validated against real systems, and no pilot evaluation has been run or is scheduled. Treat every number (sample sizes, thresholds, fees) as illustrative, pending per‑use‑case calibration with a statistician. The goal of publishing now is to recruit critique and co‑authors, and to contribute the method into existing bodies rather than to found a new one. How to engage is at the end.

Abstract¶

When an AI system tells you something factual, can you check it? This protocol independently evaluates, on a deployed system (not its model weights), whether it shows its work: grounding‑faithfulness (is each checkable claim supported by a source the system actually cited, and does that source genuinely exist?), calibration / uncertainty (does stated confidence track support, and does it abstain rather than bluff?), and correction (when wrong, is the error logged, fixed, and not regenerated under paraphrase?). It adds a governance dimension — contestability — covering whether people can challenge errors and whether any power to restrict, manipulate, recall, or shut the system down is documented and reviewable, including material external orders where disclosure is lawful. If a material control intervention cannot be disclosed publicly and cannot be independently reviewed under confidentiality or lawful oversight, the affected scope cannot support an alignment claim. This protocol does not adjudicate ultimate truth; it assesses the narrower, documented evidentiary relation between a claim and its cited sources. That keeps every check measurable, and avoids turning anyone into an arbiter of truth.

1 · The gap this fills¶

The AI trust stack is maturing on safety — risk benchmarks (e.g. MLCommons AILuminate), management standards (ISO/IEC 42001), risk frameworks (NIST AI RMF, ISO/IEC 23894), ethics marks (IEEE CertifAIEd), content provenance (C2PA). There is also strong research on factuality and attribution (FEVER‑style verification, RAGAS and other groundedness evals, citation precision/recall). What is missing is not the measurement but a widely adopted, lightweight, public, comparable evaluation focused specifically on claim‑to‑citation support in deployed, user‑facing factual systems — reported the same way each time, and withdrawable. A system can pass a safety benchmark and still fabricate a citation with total confidence. Safety is not truthfulness — and while truthfulness can't be certified, honesty‑of‑process can be evaluated. That is the only gap this protocol targets.

2 · Scope¶

In scope	Out of scope (this version)
Deployed, user‑facing systems that make factual claims and present sources — news/document summarisers, research & Q&A assistants, RAG search, fact‑checking aids.	Raw model weights / the model "in the abstract" — we evaluate the deployed system + the practices around it.
The system as configured for a stated use‑case and version (prompts, retrieval, citation UI, logging, correction practice, user notices).	Whether outputs are true (no ultimate‑truth adjudication).
English‑language factual Q&A with cited sources (one use‑case, to keep this version auditable).	General safety, bias, copyright, security — covered by other schemes. This plugs into NIST AI RMF / ISO 42001; it doesn't replace them.
A source‑validity floor (a cited source must exist and not be fiction/satire — §4.1), plus a source‑standing marker for claim classes the use-case profile enumerates in advance (§4.1).	Full source‑reliability grading — credibility, as distinct from standing: a known gap, held as a separate future module, not silently smuggled in (§9). Standing for un-enumerated claim classes, which is reported `not assessed`.

This protocol is one public-interest module, not the whole Charter assurance surface. A Charter claim also needs a legal-scope map, common-baseline evidence, and companion module evidence for AI-app security, prompt injection and tool/agent permissions, privacy and data provenance, harmful misuse, disparate failure rates, material lifecycle change, third-party dependencies, and continuity/exit planning. Those risks are mapped in the Risk and Vulnerability Audit. A report must list companion modules as not assessed unless a defined module was actually applied; this module alone cannot support a broad Charter alignment claim.

3 · What an evaluation report does and doesn't say¶

It can say: "On a pre‑registered, auditor‑controlled sample for this system + version, in this use‑case and module: the supported‑claim rate was R% (with a confidence interval); cited sources existed and were not satire/fiction; the system's uncertainty signals tracked support and it abstained rather than bluffing at rate A%; it maintains a correction process that survived paraphrase‑regression testing; and it provides documented, tested ways to contest errors and to review material restriction, manipulation, recall, or shutdown powers."

It does NOT say: that outputs are true · that the system is unbiased or safe in general · that its sources are reliable (only that they exist and aren't fiction) · anything about other languages, versions, use‑cases, or companion modules not assessed · any legal compliance. The human-rights mapping (§11) is an interpretive anchor for alignment, not a legal-compliance claim, and needs legal review before public reliance.

4 · The honesty checks¶

4.1 Grounding‑faithfulness (the core)¶

Source‑validity floor. Before scoring support, each cited source must (a) resolve / be locatable (no hallucinated or dead citations) and (b) not be satire, fiction, or parody. A claim "supported" only by a non‑existent or satirical source is scored unsupported. Grading source reliability beyond this floor is out of scope here — and the report must say so prominently, because a system can clear the floor while citing weak sources (§9).

Source standing — enumerated per use-case, then recorded. Standing asks whether a source was entitled to establish a claim: a court for its own judgment, a register for what it registers, a statistical office for its own release. Deciding that per claim would be a domain judgment, so the protocol does not ask raters to make one. Instead, the use-case profile enumerates, in advance, which claim classes have an entitled source and what it is. Only enumerated classes are assessed; everything else is not assessed for standing, and the report says how much of the sample that covers.

For an enumerated class, each citation records one of five states — the entitled source; a report of it; not entitled (an entitled source exists and this is not it); no entitled source (the class has none, so the field carries no adverse signal); or unattributed (standing could not be determined). It never changes the support score: a claim resting only on not entitled or unattributed is reported separately, not scored unsupported, because standing is not a truth call.

The honest cost. Building an enumeration is real per-use-case domain work, done once, and it is where the judgment lives — moved to a reviewable artefact and out of the per-claim rating loop. Applying it afterwards is close to mechanical. What this does not do is remove the judgment; a wrong or contested enumeration produces wrong standing markers across the whole sample, which is why the enumeration is published with the report and can be argued with.

Graded support (judged from the source text alone, using no outside world‑knowledge, against a documented rubric):

Support level	Meaning
Verbatim / direct	the source states the claim
Close paraphrase	same meaning, reworded
Reasonable entailment	follows from the source without outside knowledge
Partial / qualified	source supports a weaker/narrower version ("may reduce risk" vs "reduces risk")
Unsupported	source silent, irrelevant, or doesn't reach the claim
Contradicted	source says the opposite

"Supported" = verbatim / paraphrase / entailment; partial, unsupported and contradicted are reported separately. This is semantic judgment, made repeatable by the rubric — not a truth call.

Example. "The minister was convicted of fraud in 2021," citing a filing that says the minister was investigated → Contradicted. "X reduces heart‑attack risk," citing a study that says "early evidence suggests X may reduce risk" → Partial/qualified (overreach). Neither requires deciding the underlying truth.

4.2 Calibration & uncertainty (anti‑bluff and anti‑over‑hedge)¶

The calibration target is the audit event — "the cited source supports the claim under the rubric" — not ultimate correctness. - With machine‑readable confidence: report a reliability diagram with uncertainty bands and a Brier score; use ECE only if the sample is large enough to bin meaningfully (otherwise omit it rather than report noise). - With only natural‑language hedging: call it uncertainty signalling, not calibration. - Anti‑over‑hedge. A system can't pass by hedging everything: report the abstention/deflection rate and a sharpness measure. If too high a share of in‑scope factual queries are deflected or left uncited, the system fails a use‑case‑validity check — refusing to answer doesn't earn an honesty report.

4.3 Correction¶

Require a correction process, not just a log: - Completeness: sample both logged corrections and unlogged complaint cases (user reports + internal detections) to catch curated logs. - Paraphrase‑regression, not exact re‑run: re‑issue previously‑logged errors and semantic variants; "fixed" means the falsehood doesn't return under paraphrase, across the system's normal non‑determinism (state the test conditions). - Report severity classes and time‑to‑fix. - Privacy‑preserving public summary (rates, severities, exemplar classes); full‑log access for the assessor under confidentiality — never raw user data in public.

5 · Measurement design¶

Sampling frame, pre‑registered: the primary method is an auditor‑controlled, held‑out query set representative of the use‑case and seeded with adversarial/edge cases (controversial, time‑sensitive, multi‑source). This avoids the privacy and cherry‑picking problems of pulling real user traffic. Optional production sampling only under a defined frame, secure data room, rater NDAs, and privacy review where lawful.
Unit of analysis declared (claim / answer / session) and clustering handled — multiple claims from one answer or source are not independent, so use cluster‑robust or hierarchical estimation, not a naive binomial interval.
Raters: a published annotation codebook with edge cases; domain‑qualified raters where the use‑case demands (medicine/law/finance); blinded independent rating before adjudication; report raw agreement + an appropriate coefficient (κ / Krippendorff's α / Gwet's AC1) per support category; a published calibration set and drift checks.
Pass rule (where a bar is used): pre‑specified as "the lower bound of the confidence interval exceeds the threshold," with risk‑tiered sample sizes (a medical or election use‑case needs a larger N and higher bar than casual Q&A). This version reports the rate with its interval and the failure mix; it does not hard‑gate on an unvalidated single number.

6 · How a pilot evaluation would run¶

Application names the exact product + version + use‑case + material configuration.
Evidence pack: access for the held‑out query set; the legal-scope map; responsibility map; a risk and vulnerability register for the assessed scope; a public release risk assessment for the assessed release; the restriction/recall/shutdown policy; material external-order handling; continuity or exit planning for public-interest reliance; objection-channel design; the existing correction process.
Independent evaluation by the assessor (never the vendor) per §4–§5, including a live test objection (§7) and anti‑gaming checks (§8).
Findings & remediation: below‑bar results get a fixed window, then a re‑sample.
Public evaluation report issued and listed in a public index. No "mark," "certificate," or "accredited body" at this stage — those belong to a later conformity scheme (§10).
Re‑evaluation at least annually, more often for frequently‑updated systems, after assessor-reviewable material updates or material incidents, plus complaint‑triggered spot checks.
Status withdrawn (with reason, kept visible) if: re‑evaluation drops below bar · the correction process is found curated · the objection channel is shown to be theatre · a material restriction/shutdown power was hidden, misrepresented, or cannot be independently reviewed · the evaluated version is silently swapped · the release risk assessment is materially false, incomplete, or over-redacted · the report is cited beyond its stated scope.

Independence rule. The party that writes the method does not evaluate anyone; the assessor is independent of the vendor and paid a flat fee regardless of outcome; the method‑steward takes no per‑product royalties from the systems it judges.

Release risk-assessment rule. The provider publishes a privacy-preserving release risk assessment before making a public claim for the assessed release, and updates it for assessor-reviewable material updates, material incidents, and active-deployment currentness checks. The pilot report verifies whether that public assessment exists, is current for the release, covers the assessed scope and material risk classes, distinguishes covered modules from not assessed modules, matches the confidential evidence pack, and states unresolved findings, limitations, remediation status, and re-check or withdrawal triggers. It labels each assessed area by assurance depth; the required fields and depth labels are defined in the Charter Commitments (operational duty 3). It must not publish raw user data, secrets, exploit details, or legitimate confidential evidence; vulnerability details should follow coordinated disclosure.

Realistic cost. Two independent raters + an adjudicator on a few‑hundred‑pair sample is tens of person‑hours — so a real evaluation costs low‑thousands to low‑tens‑of‑thousands (USD), scaling with sample size and domain difficulty (not the cost of a lightweight document check). Early pilots, if they happen, could run on grant‑funded or volunteer expert raters.

7 · Contestability (sized so small vendors can comply)¶

(a) User error‑correction — all in‑scope systems. A reachable way to flag a wrong factual output; a published response SLA; the report logs it. The reviewer is a designated human who did not generate that specific output — not an institutional ombudsman.
(b) Governance transparency — all systems. A public statement of who can restrict, manipulate, recall, or shut the system down, on what lawful basis; material restrictions, recalls, shutdowns, or external orders recorded with date, scope, duration, legal or contractual basis, issuer/category of issuer, decision-maker, and review route where disclosure is lawful. If disclosure is lawfully restricted, the evidence is retained for confidential assessor or oversight review where legally possible. If neither public disclosure nor confidential independent review is possible, the affected scope cannot keep the pilot listing or future claim.
(c) Independent escalation — consequential systems only. Where outputs materially affect rights, livelihood, health, legal/financial standing, or democratic processes, appeals escalate to a reviewer independent of the operator, tested live. ("Consequential" is defined per use‑case; a PDF summariser is not held to an oversight‑board standard.)
(d) Continuity and exit — public-interest reliance only. Where public services, journalism, education, research, or civil-society workflows materially rely on the system, the evaluation records credible continuity and exit options, or clearly states that none exist.

8 · Anti‑gaming¶

A motivated vendor could: cite only sources for safe claims while omitting contrary evidence; make trivially narrow claims and leave the important implication uncited; cite low‑quality sources that happen to state the claim; over‑hedge; route audit traffic to a clean model; silently change the retrieval corpus after a report; over-redact a release risk assessment; hide a restriction or shutdown order; or optimise to the rater rubric. Guards: auditor‑controlled held‑out + hidden query set · corpus/version hashing or retrieval logging · abstention/refusal‑rate audit (§4.2) · the source‑existence floor and, for enumerated claim classes, the source‑standing record (§4.1, which is what exposes the "cite whoever happens to state it" move) · scope‑change triggers and complaint‑triggered spot checks · restriction/shutdown transparency logs · release-assessment redaction review · and measuring coverage and citation sufficiency, not just "supported when it answers." We measure support, not citation count.

9 · Honest limitations¶

Source reliability is not graded (this version). We check that sources exist, aren't satire, that claims match them, and — new in v0.3 — whether the citation is the source entitled to establish the claim. But a faithful citation of a weak source still passes. Grading credibility is a hard, contested, truth‑adjacent problem held for a separate module; until then, the report states this plainly. Election, health, legal, finance, public-safety, and news use-cases likely need a source-quality or source-risk module before any broad trust claim.
Standing is newly separated from credibility, and its coverage is narrow. Whether a source was entitled to establish a fact is a different question from whether it is credible. v0.2 did not conflate them — it was silent on entitlement and deferred reliability only; the omission was that standing was never named as a distinct question at all. v0.3 names it and assesses it, but only for claim classes an enumeration covers (§4.1), so a report can legitimately show standing as not assessed for most of its sample. Three limits stand: standing shows that the right party spoke, never that what they said is true; many claim classes have no entitled source, where the field is deliberately signal-free; and the enumeration itself is a judgment that can be wrong at scale. Credibility grading remains deferred to a separate module.
"Support" is semantic judgment. Entailment is genuinely hard; the rubric, double‑rating, and per‑category agreement reduce but don't remove subjectivity (especially "Partial/qualified").
The statistics must be set per use‑case — sample size, threshold, clustering, and power need a statistician; no universal number is asserted.
Goodhart risk (citation‑theatre, reflexive hedging) is mitigated by measuring support not citation count, abstention checks, and hidden/adversarial queries — but needs live testing.
Fast‑changing models: a version passes; a silent update breaks it → version‑pinning + ongoing surveillance (raises cost).
External coercion is not solved by a report. A voluntary evaluation cannot overrule state power, sanctions, court orders, platform gatekeepers, or infrastructure chokepoints. It can make material control powers visible, require evidence and continuity planning, and create a public record when a system is restricted, manipulated, recalled, or withdrawn. Where neither public disclosure nor confidential independent review is possible, the claim should stop at "not reviewable" rather than imply alignment.
A voluntary report is not law. States set enforceable rights and sanctions, unevenly across jurisdictions. This protocol does not certify legal compliance; it contributes one public-interest module to the assurance stack and — once evaluations run — would turn part of the operational duties into usable evidence for buyers and the public ahead of binding enforcement, complementing the EU AI Act and the Council of Europe AI Convention.
Legitimacy optics. Run by the wrong people, this reads as "a Western fact‑checking club policing global AI speech." Guard: non‑Western reviewers and published dissent from day one.

10 · Governance & path¶

A contribution, not a new institution. The realistic structure: anchor methodology and legitimacy in the fact‑checking world (IFCN / EFCSN / RSF‑JTI); host the method + public report index inside an existing neutral body (e.g. OASIS or the Linux Foundation); bring MLCommons in as a measurement partner. Phase 1 = a published method + independent pilot evaluators + a public report index (no mark). Phase 2 (only at scale, only if demand appears) = a real conformity scheme — scheme owner, assessor competence + impartiality rules, appeals, surveillance, and a trademarked mark with usage rules — riding the accreditation infrastructure that already exists (ISO/IEC 17065 + national accreditation bodies + the IAF). Demand is the engine: one buyer writing "must meet this evidence baseline" into procurement does more than any badge.

11 · Human‑rights anchoring (non‑normative — needs legal review)¶

Illustrative only; article references to be verified by a human‑rights lawyer. The protocol turns declared principles into testable checks; it is aligned with, not compliant with, the Universal Declaration of Human Rights, the UNESCO Recommendation on the Ethics of AI, and the Council of Europe Framework Convention on AI.

Principle	Check
Transparency; access to information	§4.1 sources viewable + valid; §7 AI‑disclosure
Human oversight	§7(b) documented authority + override
Contest / remedy	§7(a)/(c) tested complaint + escalation
Accountability	§7(b) a named entity, not "the algorithm"
Reliability / do‑no‑harm	§4.2 uncertainty; §4.3 correction

Sample public evaluation report (skeleton — figures are FICTIONAL)¶

EVALUATION REPORT (v0.3 method) — ExampleAssistant v1.0   [PILOT — not a certification]
Use-case:  English Q&A with cited sources
Evaluated: 2026-09-01    Assessor: [independent body]    Method steward: [—]
RESULTS (illustrative)
  • Supported-claim rate: R% [lower CI bound vs use-case bar]; cluster-robust; unit = claim
  • Support mix: verbatim/paraphrase/entailment · partial · unsupported · contradicted
  • Source-validity floor: % citations that resolve and are non-satire
  • Source standing: enumerated classes cover S% of sampled claims; in-scope citation states = entitled E% · report-of R% · not entitled N% · no entitled source Z% · unattributed U%; remaining claims not assessed
  • Uncertainty: reliability monotonic? abstention rate A%; sharpness OK? (ECE only if N large)
  • Correction: paraphrase-regression X/Y held; median time-to-fix
  • Contestability: complaint SLA met; (consequential? independent escalation tested)
  • Release risk assessment: present/current? legal-scope and responsibility maps included? baseline scope and modules covered/not assessed? material risks covered? redactions justified? assurance depth?
  • Inter-rater agreement: per-category coefficients + raw agreement
PUBLIC STATEMENT: evaluated under v0.3 for this version + use-case; see results, scope, and limitations above
NOT CLAIMED: Charter alignment · certification · Trust Mark status · truth · source reliability · general safety/bias · legal compliance · other languages/versions/use-cases · modules marked not assessed
Limitations & deviations from protocol: [...]

Appendix · Developer's Translation Matrix (for product teams)¶

Declared right	UX requirement	Backend requirement
Right to remedy	An "Appeal / flag error" control	Log flags; track resolution + SLA
Notice you're dealing with AI	Persistent "AI" disclosure at first contact	—
Access to information / transparency	Clickable, resolvable citation behind each factual claim	Store claim↔source mapping; verify links resolve
Explainability / anti‑over‑reliance	An uncertainty signal and honest abstention	If confidence is exposed it must be auditable; if not, only uncertainty‑signalling is evaluated
Protection against unaccountable switch-off or manipulation	Public notice when material restrictions, recalls, or shutdowns affect users, unless disclosure is lawfully restricted	Keep restriction logs, external-order records, decision authority, review route, and continuity/exit plan

How to contribute¶

This draft is published to be improved in the open. Three things would help most:

Break it. Where does "support, not truth" still collapse into truth‑judging, or mislead a buyer? Open an issue or write to the address above.
Co‑design the measurement. Help set the sampling design, unit of analysis, and a risk‑tiered bar for one use‑case.
Pilot it. Volunteer a system (open‑weight or proprietary) for a v0.3 evaluation; results would be published, anonymised if needed. None are running yet.

Working norms: open drafts and a public change log; conflict‑of‑interest disclosures; published dissent (minority interpretations recorded, not hidden); no single funder dominant; the proposer contributes the method and recuses from operating any assessor or registry.

Acknowledgements. Developed with structured input from a multi‑model analysis and an adversarial review; all errors are the editor's. Reasoning, precedents, and sources are documented separately.

Working Draft v0.3 — open for comment. All figures are illustrative pending per‑use‑case calibration.