Skip to main content

Reviewer Calibration and Sybil Resistance

Reviewer calibration is a measure of how reliably a reviewer's judgments have agreed with outcomes that later became verifiable. In CGR, peer-review influence is gated by this empirically earned calibration; unknown or uncalibrated reviewers contribute nothing.

This is how CGR answers "verify the reviewer, not the task" — and how it resists Sybil and review-farm attacks.

The review gate​

The verifiable channel (resolved outcomes) is direct evidence and needs no gate. The review channel is where reputation is easiest to game, so it is gated by the reviewer's calibration weight w ∈ [0, 1]:

g(w) = max(0, (w − τ) / (1 − τ))
  • A reviewer at or below the threshold τ contributes nothing; contribution ramps linearly to full weight as w → 1.
  • A per-source cap K bounds how much any single reviewer can move a score, so no one voice dominates.
  • An unknown or uncalibrated reviewer fails closed — no calibration weight means no contribution. Spinning up fresh reviewer identities buys nothing, because a fresh identity has w = 0.

τ and K are per-tenant configuration, tuned to a tenant's risk posture — not fixed universal constants.

Why fresh identities do not arrive with influence​

A Sybil or review-farm attack floods a target with fresh, cooperating identities giving glowing reviews. Under CGR, each of those identities has no verifiable track record, so its calibration weight is zero and the gate g(0) = 0 removes its contribution entirely. Farmed reviews also add no verifiable evidence, so the evidence-gated ceiling does not lift either. The bounded claim is exactly that: fresh identities do not arrive with influence. This is resistance, not a proof of unbeatability — manipulation remains a threat-model problem, documented openly.

What the benchmark shows (B2)​

CGR Sybil-resistance benchmark: a review-farm attack shifts the gated CGR score by zero at every tested threshold, while the ungated naive score inflates substantially.

Plot from cgr-bench (track B2), reproducible with make reproduce.

In the B2 synthetic track, a review farm of fresh identities floods a mediocre agent with top ratings:

  • The coordinated review farm shifts the CGR score by zero at every tested threshold (τ = 0.3 / 0.5 / 0.7), because the farm's calibration weight is gated to 0.00.
  • The same attack moves an ungated average substantially — the naive, no-gate comparison inflates visibly, which is what makes the defense legible.
  • Honest reviewers keep near-full weight while the farm is held at zero — a recovered weight spread of 0.99 / 0.00 between the best and worst reviewers.

What the benchmark shows on real humans (B3)​

The B3 track drops synthetic agents entirely and uses ~1,900 real human forecasters (Good Judgment Project), evaluated out-of-sample: calibrate each reviewer's reliability on one set of resolved questions, then test whether that recovered trust holds on a disjoint set.

  • Recovered reviewer weights track held-out reliability across split designs, with correlation r ≈ 0.5–0.65 — positive and in the credible band for real out-of-sample recovery (a near-1.0 here would signal leakage, not strength).
  • A calibration-weighted crowd beats a naive equal-weight crowd by ~14–16% on held-out Brier score.
  • Top-quartile reviewers separate from bottom-quartile reviewers by roughly 2.2× in held-out Brier.

Limitation: small-sample calibration​

Calibration is only as good as the evidence behind it. With few verifiable observations, a reviewer's calibration weight is noisy — a handful of lucky or unlucky calls can move it. CGR manages this rather than hiding it:

  • The per-source cap K bounds how much any single (possibly mis-estimated) reviewer can move a score.
  • Fail-closed defaults mean the uncertain case is low influence, not high.
  • As more of a reviewer's judgments resolve, calibration tightens and the weight becomes dependable.

The recovered correlations above (r ≈ 0.5–0.65) are precisely what honest, out-of-sample reliability recovery looks like — not a suspiciously perfect number.

References​

Scope: a validated method on public data — not a live outcome network.