Mian Zhang

Public Release - 2026-05-28

How to attack an AI claim scientifically.

A mature claim does not only show success cases. It must also say what evidence would force it to change: the exact claim, the minimum failing case, the observed break, the missing receipt, and the boundary update.

Counterexample Challenge Worked Examples Claim-Evidence Table Evidence Map
A structured counterexample challenge map connecting claim, receipt, replay, counterexample, and boundary update.

Principle

Disagreement is useful only when it can be replayed.

Public AI work should not ask readers to accept a claim on authority. It should give readers a clean way to break the claim, reproduce the break, and force a narrower boundary if the break is valid.

The Counterexample Challenge is therefore not a debate format. It is a small public protocol for converting skepticism into evidence: precise claim, minimum case, expected behavior, observed behavior, missing proof, and repair target.

Packet

A valid counterexample has a shape.

FieldQuestionValid exampleInvalid example
claim_idWhich public claim is attacked?A row in the claim-evidence table."The project is wrong."
minimum_caseWhat is the smallest failing case?One task, one seed, one trace, one reproduced metric.A broad opinion or anecdote.
expected_behaviorWhat should the protocol predict?The declared metric, receipt, or gate outcome.A moving target after seeing results.
observed_behaviorWhat actually happened?Reproducible output with command, hash, and artifact pointer.Screenshot-only claims without replay path.
missing_receiptWhich evidence object is absent?Missing provenance, baseline, boundary, or failure receipt."It feels unconvincing."
boundary_updateHow should the claim shrink?New condition, excluded regime, or stricter artifact requirement.Demanding unrelated private material.
A schema card listing the minimum fields for a reproducible counterexample.
1

Formula counterexample

Find a bounded setting where the stated variable relation fails while the assumptions still hold.

2

Data leakage

Show that a score depends on information unavailable at the declared decision time.

3

Stronger baseline

Provide a simpler or standard baseline that beats the reported result under the same protocol.

4

Irreproducible step

Identify the exact command, artifact, dependency, or random seed that prevents replay.

5

Boundary overreach

Show that a public claim says more than its evidence object can support.

6

Credit leak

Show that repair work, exploratory output, or paper-only evidence was counted as performance credit.

Five counterexample types for evidence-gated AI claims.

Boundary

Not every attack is a counterexample.

A useful attack changes the evidence state. It either narrows a claim, finds a missing receipt, improves a baseline, reveals leakage, or makes reproduction fail in a specific place.

A vague attack does not change the evidence state. Identity arguments, status arguments, broad dislike, demands for private logs, or requests to expose unsafe operational details are not part of this public route.

Where to aim

Three flagship public surfaces.

SurfaceClaim typeBest attackPublic route
WisdomBenchLearning from failure can be measured longitudinally.Task-level leakage, stronger baseline, unstable metric.Papers Proof-Carrying ActionHigh-risk action should carry warrant and receipt objects.Missing receipt, credit leak, false action authority.GitHub Relational ObservabilityAdaptive systems need observable relation, boundary, and control debt.Unobserved relation, stale evidence, boundary overreach.Technology

Public commitment

If a counterexample is valid, the claim changes.

The point of a public evidence field is not to win arguments. The point is to keep the claim surface clean: supported claims stay, unsupported claims shrink, missing artifacts become work orders, and negative results become part of the record.

This is the same discipline used inside the system: no proof, no action; no closed receipt, no credit; no clean denominator, no performance claim.

Routes

Start with the public interfaces.

RouteUseBest readerLink
Challenge rulesHow to submit a scientific attack.Reviewers, engineers, skeptics.Open Worked examplesConcrete shapes of valid and invalid attacks.Reproducibility readers.Open Claim tableWhich claim is supported by which artifact.Technical readers.Open Pilot packetHow a serious pilot would be scoped.Potential partners.Open
A public call to inspect, reproduce, challenge, and update AI claims.