Skip to main content

What is Capability-Grounded Reputation?

Capability-Grounded Reputation (CGR) is a trust model for autonomous agents in which reputation is specific to an agent and a capability domain, is updated from judgments that later resolve against observable outcomes, and is always reported together with the amount of evidence behind it. Peer reviews are weighted by the reviewer's own empirically demonstrated calibration rather than counted equally.

The problem: declared capability is not observed capability​

An agent can declare anything — a model card, a confidence score, a long list of completed tasks. None of it tells you whether the agent is right in a given domain. Two failure modes recur:

  • Declared ≠ observed. Self-reported confidence, task volume, and capability metadata are assertions. They describe what an agent claims to do, not what its judgments turned out to be worth once the world resolved them.
  • Ratings without outcomes. Star ratings and peer endorsements count opinions, not results. If a five-star review is never checked against a resolved outcome — and if the reviewer's own reliability is never measured — the rating is popularity, not evidence. That is precisely what a coordinated review farm exploits.

CGR removes both by grounding reputation in resolved outcomes and by weighting reviews by the reviewer's demonstrated calibration, not by headcount.

The formal model​

CGR's scoring model is documented in the CGR overview and implemented as a clean reference in cgr-bench. The essentials:

  • Per-(agent, domain) Beta posterior. Reputation for one agent in one capability domain is a Beta(α, β) posterior. The score is the posterior mean E = α / (α + β); the confidence is the evidence mass n = α + β. Every agent starts neutral (α = β = 1) — no unearned standing.
  • Two evidence channels, different trust rules. The verifiable channel (resolved outcomes — the world confirmed or refuted the call) is ungated: it is direct evidence. The review channel (peer review of a decision) is gated, because a review is only as good as the reviewer.
  • The review gate. Each reviewer carries a calibration weight w ∈ [0, 1], itself earned on verifiable outcomes. A review's contribution is scaled by the soft ramp g(w) = max(0, (w − τ) / (1 − τ)), with a per-source cap K bounding how far any single reviewer can move a score. An unknown or uncalibrated reviewer fails closed — no calibration weight means no contribution.
  • The evidence-gated ceiling. Thin-evidence scores are held below a ceiling that releases only as resolved outcomes accrue: E ← min(E, cap + ε + (1 − cap − ε)·s), where s rises with the count of verified outcomes. For a neutral agent this is 0.52 + 0.48·(n / 20) — so a strong-looking score cannot present as proven until enough outcomes have resolved.

A worked example​

Take an agent making contract-review judgments. It certifies a clause as compliant — a judgment call, tagged as such.

  1. Decision. The agent records the judgment through the governed substrate. On its own, this moves nothing: an unresolved judgment is not yet evidence.
  2. Outcome. Later, the clause is checked against what actually happened (an audit, a dispute, a downstream result). The outcome resolves the judgment as correct.
  3. Update. The resolved-correct outcome updates the Beta posterior in the contract-review domain (α increases). The score edges up and, crucially, the evidence mass n increases by one.
  4. Trust decision. A relying party now reads both numbers. A score of, say, 0.64 on five resolved outcomes is treated as provisional — the ceiling is still binding. The same 0.64 on forty resolved outcomes reads as an established track record. Score and evidence travel together, so the reader never mistakes a thin record for a proven one.

Limitations​

CGR is a method with an honest scope, not a guarantee:

  • Sybil resistance is not unbeatability. Fresh identities do not arrive with influence, but manipulation remains a threat-model problem — documented openly, not claimed away.
  • Reputation is per domain. A strong contract-review agent earns nothing in security auditing; CGR never treats capability as global.
  • Thin evidence stays uncertain. With few resolved outcomes, calibration is noisy and scores are deliberately withheld or discounted.
  • Validated method, public data. The reproduced results below are on public and synthetic data — a validated method on public data, not a live outcome network.

GRAFOMEM implementation note​

In GRAFOMEM, governed decisions, their resolved outcomes, and peer reviews are written as append-only, content-addressed facts in a tenant's governed memory store, and reputation is computed from them — every score traces back to resolved evidence. Reviewer-calibration updates and issued reputation attestations are recorded on GRAFOMEM's signed, append-only audit chain. Each tenant's reputation is isolated. The method is independently reproducible: cgr-bench reproduces its properties from source with one command — cold-start and Sybil-resistance behavior asserted in CI, plus an early-warning correlation of −0.997 at 25% resolution against real credit-default outcomes.

References​

  • CGR overview — the documented scoring model and endpoints.
  • cgr-bench — one-command, independently reproducible benchmark of CGR's properties.
  • GMP v0.2 specification — the Grafomem Memory Protocol the evidence substrate implements.

Scope: a validated method on public data — not a live outcome network.