the record · exhibit 17 · loop 03, Refinement
The cleaner artifact lost to the better novel
Two finished books, blinded by coin flip and read whole by five evaluators in both orders: five verdicts to none, with the one dimension the old book swept recorded beside the win.
· principles 8, 10 of twelve
What it shows
Principle eight, at whole-book scale. Two finished versions, labelled by coin flip, read in full by four jurors in both orders and by a fifth act against act. Five verdicts to the rework, every margin clear and none decisive, with the dimension the old book swept recorded beside the win. The caveats are kept: one juror disclosed a breach of the no-comparison rule, and its ballot matched the four untainted ones.
The principles are numbered on the method page. Cuts in the excerpts below are marked with an ellipsis in brackets; the three kinds of cut are stated on the record’s front page.
From the record
# A/B Test Verdict — manuscript vs manuscript_v2 **Date:** 2026-08-23 **Result: `manuscript_v2` WINS — unanimous, 5–0, margin "clear" on every ballot.** ## Protocol **Phase 1 — independent evaluation (previous step).** Two isolated expert-reader agents each read one version in full, chapter by chapter, with no access to the other version or to any project material, and wrote truth-mode assessments (`manuscript/READER-ASSESSMENT.md`, `manuscript_v2/READER-ASSESSMENT.md`). Both scored their book **8/10 overall** with near-identical profiles — the versions are genuine peers, which is what makes the comparative phase meaningful. **Phase 2 — blind comparative A/B (this verdict).** Both books were copied into neutral directories under blinded labels assigned by coin flip: - **OAK = manuscript_v2** (36 chapters) - **ELM = manuscript** (34 chapters) Five evaluators worked blind and mutually isolated, each reading **every chapter of both books** (no sampling, no diff tools), keeping per-chapter notebooks, and verifying every logged inconsistency at both ends before charging it: | Evaluator | Protocol | Reading order | |---|---|---| | Juror 1 | Full read of both, dimension verdict | OAK first | | Juror 2 | Full read of both, dimension verdict | ELM first | | Juror 3 | Full read of both, dimension verdict | OAK first | | Juror 4 | Full read of both, dimension verdict | ELM first | | Head-to-head judge | Act-by-act alternating comparison, per-act winners + split | Alternating per act | Reading order was balanced 2/2 across jurors to cancel order/recency bias. No evaluator knew which version was which, that a "v1/v2" relationship existed, or what any other evaluator concluded. All operated in truth mode, calibrated against published literary fiction.
## The ballots
| Evaluator | Winner | Margin | Dimension tally (win–loss–tie) |
|---|---|---|---|
| Juror 1 | **OAK** (v2) | Clear | OAK 8 – ELM 4 – 1 tie |
| Juror 2 | **OAK** (v2) | Clear | OAK 9 – ELM 1 – 3 ties |
| Juror 3 | **OAK** (v2) | Clear | OAK 9 – ELM 1 – 3 ties |
| Juror 4 | **OAK** (v2) | Clear | OAK 10 – ELM 1 – 2 ties |
| Head judge | **OAK** (v2) | 55–45 | OAK 7 – ELM 3 – 3 ties; acts: OAK 3, ELM 1, tie 1 (Act 1 verbatim identical) |
## Dimension consensus (aggregated across all five ballots)
| Dimension | Consensus | Notes |
|---|---|---|
| Factual correctness & consistency | **manuscript (ELM), 5–0** | The one dimension v1 sweeps: 0 verified errors vs v2's 3–4 |
| Pacing | manuscript_v2 (3–1–1) | v2 merges v1's repeated disproof-loop chapters; Juror 1 dissents, preferring v1's ch10/11 split |
| Engagement | **manuscript_v2, 5–0** | The most gripping chapters in either book are v2-only |
| Immersion | manuscript_v2 (3–1–1) | Consulta procedure, nomenclator ciphers, the "light" program |
| Prose & voice | **Tie, 5–0** | ~75% identical text; divergent chapters of equal sentence-level quality |
| Characterization | **manuscript_v2, 5–0** | […] (an entire extra character with an arc), Okonkwo's peak, a fuller Pope |
| Plot structure & coherence | **manuscript_v2, 5–0** | v2 motivates the summons ("why this one hand, why now"); v1 asserts it |
| Dialogue | **manuscript_v2, 5–0** | […] the consulta ensemble, […] corridor question |
| Emotional impact | Near-tie (4 ties, 1 to v1) | Shared spine lands equally; […] |
| Thematic depth | **manuscript_v2, 5–0** | The machine/mortality-as-warrant axis is a second load-bearing theme v1 lacks entirely |
| World-building | **manuscript_v2, 5–0** | 2073 is load-bearing only in v2; in v1 the future is largely backdrop |
| Originality | **manuscript_v2, 5–0** | Machine-restored silences, the human seal on machine work |
| […] | | |1. **v2's exclusive machine/young-keeper arc is the best material in either book, and it is load-bearing.** […] v1 asserts what v2 stages. […] […] 4. **The flaws are asymmetric in kind.** v1's advantages are finish-level (zero defects, cleaner cross-references, a cleaner climactic bridge); v2's advantages are scope-level (a theme, a character, three set pieces, causal coherence). As Juror 1 put it: copyedit-class advantages against writing-class ones. As Juror 2 put it: "ELM is the cleaner artifact; OAK is the better novel." 5. **What kept the margin at "clear" rather than "decisive"** is entirely mechanical: v2's small set of verified production defects, all fixable without rewriting anything. Several evaluators stated that after one copyedit pass the result would approach decisive. 5. **What kept the margin at "clear" rather than "decisive"** is entirely mechanical: v2's small set of verified production defects, all fixable without rewriting anything. Several evaluators stated that after one copyedit pass the result would approach decisive. […] ## Caveats, for the record - Approximately three-quarters of the two texts are identical; all evaluators correctly refused to manufacture differences there, which is why prose/voice is a unanimous tie and margins are "clear," not "decisive." - Juror 3's first two verdict-writing attempts were interrupted by infrastructure failures (machine sleep, stream stall) after both books were fully read; it was resumed with context intact and completed normally. Juror 3 also disclosed running one early file-listing byte-comparison in breach of the no-mechanical-comparison rule; it disregarded the output and read every chapter in full. Its verdict matches the four untainted ballots, so the result does not hinge on it. - The blinding held: no evaluator identified which version was which, and the label assignment (v2=OAK, v1=ELM) was decided by coin flip and sealed in the scratchpad until after all ballots were returned.