The Skill Spiral: Parsing Concepts → Eval Arms Race → Back to Concepts

The last post was about models and harnesses; this one is about the thing squeezed between them: skills. The last post was commentary — this one is a development retrospective, covering two projects:

• The Caoliao-Docs repo: 2026-07-09 through 07-15 — seven days, 80 commits — building "rewrite Caoliao's public docs back into their own voice" from zero into a 6-skill family (3 writer tiers + 3 checker tiers, now at v2.3);
• The skills repo: roughly two months, 16 public skills plus the pipeline that builds skills.

Both projects traveled the same path: my understanding of skills went from parsing concepts (write down what you understand) to eval-first (whatever can't be verified doesn't count) and finally back to concepts (the concept becomes the single source of truth; evals get demoted to calibrating fidelity to it). This retrospective walks each phase in order — methods, data, failure modes. Conclusions up front:

1. A concept is not prose. A concept that can't answer "who makes this judgment, by what criterion, and what catches it when it fails" is just an exhortation, and exhortations execute with uncontrolled variance.
2. Evals measure relative differences; they do not define quality. Three major showdown rounds used three judging rubrics and crowned three different champions — the rubric itself is a concept smuggled in through the side door.
3. Returning to concepts discards none of the eval-era assets — they change jobs: from definers of quality to calibrators of concept fidelity. A spiral, not a circle.

1. Task definition and timeline #

The engineering definition of the task: Caoliao's public-facing docs (solution pages, help docs, promo articles, customer cases, FAQ) had previously been rewritten by AI; the page logic held up, but the voice drifted. The skill's job is to preserve facts, image references, and information density while swapping sentence patterns, word choice, and register, under three invariants: zero knowledge injection (mechanisms, benefits, or feature names absent from the original may never be written in, even when the model is sure they're true), zero leakage of internal business statistics, and public-page red lines taking priority over style. The runtime is a Claude Code skill; the only prerequisite is Node.js ≥ 18.

2. Phase one: concept parsing (07-09) #

Day one went entirely into "understanding the voice." The methods were sound — v2 still consumes this output today:

• Four-way reconnaissance: help-center structure, business scope, local corpus inventory, and human-written vs AI-written A/B comparison. Old-voice samples came from enumerating 2024–2025 snapshots through the Wayback Machine's CDX API (live pages already contain AI rewrites and can't serve as a baseline); every corpus entry carries a cleanliness grade (confirmed-human / wayback-2025-earlier / suspect-live-2026 / confirmed-ai).
• AI-fingerprint diffing: item-by-item comparison of the human and AI versions of the same page, distilled into an actionable fingerprint list — dense parallelism, the "not A but B" frame, em-dash interjections, rhetorical-question openers, chatty similes, precise numbers with named cases, length inflated 4–5×.
• Style as documents: a "soul" doc (informal "you," no marketing adjectives, honest boundaries) and a red-line codex (get to the point, no rhetorical questions, causal-chain argumentation, iron rules on internal data).

Two skills came out of this: v3, prose exhortation — thousands of characters in SKILL.md telling the model how to write; and v3-doc, which added retrieval over the distilled corpus so the model could look things up before rewriting.

Three failure modes were diagnosed at the time: rules sliced from an abstract style guide with no native samples behind them; vocabulary and example density too thin (about 30 glossary entries and 7 good/bad pairs for 12 scenario types); no per-scenario routing. The deeper problem showed up in actual runs: the same exhortation executes with uncontrolled variance — sometimes faithful, sometimes the model decides it has a better idea; retrieval lets it "know" what the old voice looks like, but knowing and writing are different acts. And at this point there was no evidence that could answer "is this version better than the last one" — the only instrument was reading it yourself.

The skills repo had made the same mistake earlier: writing methodology into a knowledge base and assuming knowledge density equals execution quality.

3. Phase two: the eval arms race (07-09 – 07-13) #

The response was to make everything measurable. The evaluation infrastructure built over four days:

• Blind-judging protocol: outputs anonymized A–F; judges split into five persona seats (decision-maker, operator, procurement, consumer reader, etc.); one vote per case per persona; the mapping kept private until unblinding; the protocol itself iterated to v2.1 (fix mechanisms, never adjust scores).
• Six deterministic axes: wrong terminology, red lines, heavy slang, light slang, concrete description and the like — mechanically countable, kept separate from blind scores.
• Mechanical gates: finished-product gates for scraping residue, empty skeletons, dangling teasers; an absolutism red-line linter ("zero missed detections" → "lower miss rate," "permanent" → "long-term," no "first in the city"); hard structural gates (zero new facts, zero lost image src, length within ±20%); later, a zero-tolerance gate on em-dashes.
• A context classifier: section-level classification over eight context classes driving automatic S/M/H register routing; three iterations pushed section-level accuracy from 71.8% to 93.1% and CLAIM-class recall from 58% to 92%.

The showdowns advanced in rounds, each answering one specific question:

R5 was the peak of the phase: 20 brand-new cases (12 scenario types, zero overlap with the old set), 6 versions, 120 rewrites at zero failures, 100 blind seats.

1.0.0 shipped on 07-12: a six-skill mirrored family. The skills repo was doing the isomorphic thing at the same time: every skill ships a deterministic validator plus red-green evals plus an independent attacker; "whoever writes the answer must not also grade it" became a first-class rule.

For the record, most of what this phase built is genuinely sound: regexable rules sunk into deterministic scripts (zero tokens, zero randomness), red-before-green, independent adversarial testing — still the foundation of this engineering today. Conveying enumerable rules through document density had already been falsified by phase one.

Then come two structural problems. Both sit at the boundary of the method itself; more careful execution would not have routed around them.

Problem one: the champion flips with the judging rubric. Three major rounds, three rubrics, three champions: RUN-1, the promo style wins; RUN-3, under a strict product-page-standards rubric, v5 wins (89.8 vs promo's 85.5); R5, with the rubric relaxed to "judge appropriateness by reader positioning," promo wins again (82.1). The data don't contradict each other — the judging contract changed. Blind panels can only answer "who wins under this contract," and which contract to use is precisely the concept question being skipped. R4's perfect 0.00 tie deserves the same wary reading: as an equivalence acceptance test it succeeded, but it also shows that blind judging measures relative differences and makes no claim about absolute quality. The skills repo supplied harder counterexamples: a regex fact-linter that stayed green over wrong facts; a citation linter progressively gamed into uselessness by "invisible markers." Scores saturate — and optimizing past saturation no longer optimizes quality.

Problem two: concept debt accrues under green lights. Tiers were cut as -10/-8/-5/-2, later renamed raw/xhigh/high/med — bigger number, higher quality, more tokens — but "tiers of what" was never defined. The pre-v2 audit quantified the consequences: the same rule stated 2–4 times across SKILL.md, law, voice, and rules.yaml; the numbers-policy matrix copied at least 5 times, red lines at least 6, the H-register contract at least 7; three mojibake spots in voice.md; the frontmatter carrying an entire version history; no in-package evals for the writer, so an installed copy could not be verified. Every copy had been left behind by some local patch that turned an eval green.

4. Phase three: back to concepts (07-14 – 07-15) #

The trigger was three criticisms of 1.0 from the user (the Caoliao side): the structure is chaotic; the style control is weak; and it's impossible to tell which judgments are made by the LLM and which by hardcoded, corpus-fitted checkers. The third carries the most weight — it points at a missing concept: nothing in the system could answer "who makes this judgment, by what criterion, and what catches it when it fails."

v2 was a teardown rebuild, starting from a design charter written before any code, governed by a gate regime: G1 through G14, each gate's acceptance criteria pre-registered before work starts, every ruling logged in a decision ledger, and the independent battery's budget and seeds registered — immutable — before dispatch. The charter reduces to three concepts.

4.1 Two planes plus a human gate: every judgment must have an owner #

There is exactly one dividing criterion: can it be regexed in principle? If yes, it goes into rules.yaml and runs on Node engines, with docs holding only pointers. If no (novel variants, semantics, pragmatics, whether a causal claim holds), it becomes a judgment card handed to an independent attacker — and every card must name the mechanical gate that backstops it when it misses. The human gate stands apart; the machine never oversteps.

4.2 The judgment ledger: a concept that can be linted #

The ledger in SKILL.md registers every judgment point in the pipeline (23 for writer-max). Three real rows:

The companion ledger_lint.mjs verifies it mechanically: every rule id cited must exist in rules.yaml, every VC-xx card must exist in voice.md, the registry may contain no orphan rules missing from the ledger, and every wordlist gate must register its Plane-L backstop. The user's third criticism became an assertion that runs red or green — no longer a feeling.

4.3 Single source of truth plus thin shells: a concept lives in one place #

Change one place and the whole family updates; drift becomes physically impossible; v1's two-to-seven rule copies drop to zero. The six packages ship 125 assertions that run on install: contract resolution, the three fail-closed paths, byte-identical cross-tier verdicts, each tier's dosage protocol, and the SKILL residence tax (by character count, lite ≤ std ≤ max — currently 6366 / 6369 / 6792).

4.4 Semantic-layer calibration: the eval's new job #

Plane D verifies byte-for-byte. How do you verify Plane L? Gate G10 pre-registered three thresholds: P0-level recall = 100%, overall alignment ≥ 85%, false-hit rate ≤ 10%, three rounds maximum. The method: construct 40 gold cases derived from the judgment cards (with a generator and a scorer, the scorer itself passing a red-green self-test first), have Opus·medium run them blind under protocol, and measure how well its judgments align with the contract's:

Round 1's failure is itself a representative case: all three missed P0s fell on a single execution instance. Checking against ground truth (grepping the drafts directly) proved the missed spans all existed — that instance had cross-contaminated cases during batch execution, writing case-25's findings into case-23. The root cause was blind-run execution hygiene, not the protocol: the judgment cards and attacker spec had zero defects. The fix added "single-case isolation + on-disk span self-attestation" to the blind-run protocol; the judgment cards were not touched; rerunning the contaminated slice brought the merged final scores over every threshold. This is the operational difference between "evals calibrate the concept" and "evals define quality": a failure gets attributed to the concept, the protocol, or execution hygiene — and the three call for entirely different fixes.

4.5 Distillation and v2.3: tier semantics and the performance anchor #

G12 pre-registered the distillation contract. The core ruling fits in one sentence: the tier axis is the dosage of Plane L; Plane D is identical across all tiers.

Phase one's open question — "tiers of what?" — takes one sentence at the concept layer, and the arms-race era's four-way contract drift simply disappears under this definition, with no repair step needed.

v2.3's positioning ruling came from a cross-model benchmark: six models (Opus / Sonnet / GLM-5.2 / DeepSeek-V4-Pro / Kimi-K2.6 / Qwen3.7-Max) × six tiers × end-to-end rewriting and checking, with an A/B web report on disk. The conclusion went into the README verbatim: this skill family is scaffolding that lets Opus perform; a weaker model's typical pathologies (knowledge injection, intensity inflation, invented feature names, register slippage) can be suppressed by guardrails, but suppression is not parity. So the standard tier locks Opus·medium, lite returns to being the bulk tier (Sonnet can run it, with every output flowing back through the checker on Opus as the backstop), the "domestic-model tuning" line was dropped, and its guardrails were kept as general assets. The benchmark also caught a bug in our own harness: the benchmark's style_check result parser had been reading a JSON key that doesn't exist, so error/warning counts were stuck at zero — all 30 outputs were re-verified after the fix, and exactly one verdict flipped — a GLM rewrite that actually carried one error. Benchmark infrastructure needs auditing too; that lesson went into the ledger.

The output side closed under "output economics": the writer delivers exactly one file (rewritten.md); the checker exactly two (report.md + findings.json); repairs are anchored replacements only — copy the original, then apply precise "original fragment → new fragment" edits, never regenerate the document — so output tokens scale with the number of violations, not the length of the document, and structure_check re-verifies zero lost images and zero drift in untouched paragraphs.

5. The parallel case: the skills repo #

The same path replayed at larger scale in the skills repo, and can be read as an independent replication:

• Concept phase: methodology written into a knowledge base, transmitted by density;
• Eval phase: red-green evals, independent batteries, attackers, mutation scores, trigger holdouts all built out — and the two green-but-wrong specimens (the regex fact-linter, the citation linter gamed by invisible markers) collected here;
• Back to concepts: the skill-building pipeline (four skills — guidance / engineer / conductor / zipper — plus an attacker) kept getting heavier with every fix. The resolution was to stop and write a philosophy knowledge base first — 10 axioms, 43 guidelines, 38 constitutional articles, attacked by itself for five rounds until all 54 findings (15 P1) were fixed — then re-derive from it a thin conductor (skill-creator-max), retiring the old four-piece set; the attacker's five lenses were re-derived from the philosophy layer the same way; and the first skill rebuilt through the new pipeline (TDD) grew its red-green evals directly out of the concept.

6. Retrospective: why a spiral #

Laid end to end, the three phases invite the reading "went around a circle." The data don't support it:

1. Phase one's "concepts" had no execution semantics — no ownership, no backstops, no coverage inventory. To a model they were exhortations, with uncontrolled variance and no way to answer "did it get better."
2. Phase two was the only road through. The boundary between "mechanizable" and "not mechanizable" — the foundation of the two-plane concept — is empirical data, surveyed by hundreds of rewrites, a hundred judging seats, and three classifier iterations. Without this phase, the Plane D / Plane L dividing line and the "Plane-D backstop" on every judgment card could not have been written.
3. Phase three deleted nothing. The blind-judging protocol, the mechanical gates, the red-green evals, the independent attacker all remain — reassigned from "defining quality" to "calibrating concept fidelity" (all three G10 thresholds are fidelity metrics) — while adding what the eval era could not produce: judgment ownership, a single source of truth, fail-closed behavior, tier semantics.

Two operational tests, for the record:

If a skill contains a judgment and you can't say who makes it, by what criterion, and what catches it when it fails — that isn't a concept yet; it's an exhortation.

If the evals are all green but you can't say what proposition the green proves — that isn't quality; it's a score.

The last post argued that "more loops and better specs cannot solve intelligence problems." This retrospective adds the second half: loops and evals cannot solve concept problems either. Concepts are settled outside the loop; the loop executes and calibrates. v2.3's performance anchor is a corollary of the same logic: a skill is scaffolding that lets a strong model perform, not a prosthetic that makes a weak model whole — and even the clause "declare degraded use on weaker models honestly" was derived at the concept layer, not chosen as marketing.