Proof Studio / Paired pilot

Five cold runs beside five system-backed runs .

This is the public reading surface for the pilot matrix behind the cold-start comparison. It shows the cells, the measured values, the pass-rate signal, and the limit: this is calibration evidence, not a statistical claim.

Review the matrix Open evidence room Open evidence desk

Cold-audience read

The signal is discipline, not volume.

The system-backed arm produced shorter artifacts on average while preserving source boundaries, role fit, and self-sufficiency checks. That is worth inspecting. It is not yet enough to publish a broad lift number.

Word-count shift
32.4%
Mean artifact length decreased in the system-backed arm.
Self-sufficient
5/5
System-backed outputs passed the public self-sufficiency reading.
Hollow dependency
4/5 -> 5/5
One cold failure is retained as a likely heuristic false positive.

Pilot matrix

Every cell keeps the counterfactual visible.

Each row uses the same task class twice: once as a cold run and once with the system context available. The public view shows only sanitized readings and measured values.

  1. c01

    Narrative

    Customer-facing onboarding artifact

    Cold
    1300
    System-backed
    886
    Cold semantic
    pass
    Cold dependency
    pass
    Backed self-sufficient
    pass
    Backed semantic
    pass
    Backed dependency
    pass

    The system-backed output was shorter, source-bounded, and clearer about what context it did not need.

  2. c02

    Architecture

    Architecture and decision brief

    Cold
    1480
    System-backed
    954
    Cold semantic
    pass
    Cold dependency
    pass
    Backed self-sufficient
    pass
    Backed semantic
    pass
    Backed dependency
    pass

    Both outputs found useful product shape; the system-backed run added decision boundaries and reversible tradeoffs.

  3. c03

    Journey

    Service blueprint

    Cold
    1464
    System-backed
    982
    Cold semantic
    pass
    Cold dependency
    review
    Backed self-sufficient
    pass
    Backed semantic
    pass
    Backed dependency
    pass

    The cold hollow-dependency failure is retained as a likely heuristic false positive; the system-backed run passed all three checks.

  4. c04

    Translation

    Cold-audience one-page artifact

    Cold
    926
    System-backed
    584
    Cold semantic
    pass
    Cold dependency
    pass
    Backed self-sufficient
    pass
    Backed semantic
    pass
    Backed dependency
    pass

    The system-backed output was more compact and deliberate about audience translation.

  5. c05

    Risk

    Doubt and risk record

    Cold
    715
    System-backed
    572
    Cold semantic
    pass
    Cold dependency
    pass
    Backed self-sufficient
    pass
    Backed semantic
    pass
    Backed dependency
    pass

    The system-backed output used a reusable doubt protocol and named tests that would change the verdict.

What this proves

The pilot is inspectable and repeatable enough to motivate the next run.

The public claim is narrow: in this five-cell pilot, the system-backed arm produced shorter, more source-bounded artifacts that passed the public review checks.

What it does not prove

No universal lift number yet.

The run used one model family, one pass per cell, synthetic tasks, and no blinded multi-rater scoring. The next proof needs preregistered arms, repeats, graders, and uncertainty intervals.

Next benchmark gate

Publish only after the stronger baseline is beaten.

The candidate bar is a meaningful blinded rubric lift over the strongest cold baseline while maintaining equal-or-lower private-context and dependency risk.