Design the 20 tasks
Rules that keep the canary honest:
- Every task has a machine grader. Exit code 0 means pass. No rubric, no LLM judge, no "looks right."
- Every prompt is frozen text in git. Never edit a prompt without resetting the baseline.
- Tasks are small enough to finish in under 8 agent turns.
- Half the tasks should be ones the model passes reliably, so a drop is a signal. A task at 50% pass rate is a coin flip, not a sensor.
- Tasks come from your real work, scrubbed. A canary on LeetCode tells you about LeetCode.
Layout on disk:
canary/
run.sh
drift.sh
tasks/
01-fix-date-parse/
prompt.txt
fixture/ # starting files, copied to a temp dir per run
check.sh # runs inside the temp dir, exit 0 = pass
02-.../
results.csv
The runner
The flags below are all in the CLI reference: -p, --bare, --model, --effort, --max-turns, --max-budget-usd, --allowedTools, --no-session-persistence and --output-format json. The headless docs say --bare skips hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md, and needs ANTHROPIC_API_KEY because it never reads your login. That is what you want. The model sees the task and nothing you accumulated.
#!/usr/bin/env bash
# canary/run.sh - usage: ./run.sh claude-sonnet-5-5
set -u
MODEL="$1"
ROOT="$(pwd)"
DATE="$(date -u +%F)"
CLI="$(claude --version | head -n1)"
mkdir -p "$ROOT/raw/$DATE"
for dir in "$ROOT"/tasks/*/; do
id="$(basename "$dir")"
for trial in 1 2 3; do
work="$(mktemp -d)"
cp -R "$dir/fixture/." "$work/"
( cd "$work" && claude --bare -p "$(cat "$dir/prompt.txt")" \
--model "$MODEL" --effort high --max-turns 8 \
--max-budget-usd 0.50 --allowedTools "Read,Edit,Bash" \
--no-session-persistence --output-format json \
> "$ROOT/raw/$DATE/$id-$trial.json" 2>/dev/null )
if ( cd "$work" && bash "$dir/check.sh" >/dev/null 2>&1 ); then ok=1; else ok=0; fi
cost="$(jq -r '.total_cost_usd // 0' "$ROOT/raw/$DATE/$id-$trial.json" 2>/dev/null || echo 0)"
echo "$DATE,$MODEL,\"$CLI\",$id,$trial,$ok,$cost" >> "$ROOT/results.csv"
rm -rf "$work"
done
done
Three things to notice. The CLI version goes into every row, because the April postmortem says harness changes were the culprit. Each task runs three times, which turns a 5-point swing per flipped task into a 1.7-point swing per flipped trial. And --max-budget-usd 0.50 caps a runaway loop; the docs warn that spend can pass the cap, so treat it as a brake, not a guarantee.
For the second arm, run the same loop without --bare from inside your real repo, so your CLAUDE.md, hooks and MCP servers load. Log it with a different label. That arm is your own configuration, and it is the one that usually moves.
A drift threshold that won't cry wolf
Take the first 14 nights as the baseline. With 20 tasks and 3 trials you get 60 pass/fail results a night. Say the baseline mean is 52 of 60, or 86.7%.
#!/usr/bin/env bash
# canary/drift.sh - exit 1 when the 3-night mean falls 8+ points under baseline
MODEL="$1"
awk -F, -v m="$MODEL" '
$2 == m { sum[$1] += $6; n[$1]++ }
END {
k = 0
for (d in sum) { dates[++k] = d }
# sort dates ascending
for (i = 1; i <= k; i++) for (j = i + 1; j <= k; j++)
if (dates[j] < dates[i]) { t = dates[i]; dates[i] = dates[j]; dates[j] = t }
if (k < 17) { print "baseline not ready: " k " nights"; exit 0 }
for (i = 1; i <= 14; i++) { bs += sum[dates[i]]; bn += n[dates[i]] }
for (i = k - 2; i <= k; i++) { rs += sum[dates[i]]; rn += n[dates[i]] }
base = 100 * bs / bn; recent = 100 * rs / rn
printf "baseline %.1f%% last3 %.1f%%\n", base, recent
if (recent < base - 8) exit 1
}' "$(dirname "$0")/results.csv"
Why 8 points over 3 nights? One night of 60 trials has a standard error near 4.4 points at an 87% pass rate, and three nights pooled shrink that to about 2.5. Eight points is roughly three standard errors. That is a heuristic, not a statistical guarantee, and sampling noise is real. Tune it after a month of your own data. If your baseline sits near 95%, tighten it.
The nightly GitHub Action
name: claude-canary
on:
schedule:
- cron: '17 2 * * *'
workflow_dispatch:
permissions:
contents: write
jobs:
canary:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- run: npm install -g @anthropic-ai/claude-code
- run: sudo apt-get install -y jq
- name: Run canary
working-directory: canary
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: ./run.sh claude-haiku-5-5
- name: Commit results
run: |
git config user.name "canary-bot"
git config user.email "[email protected]"
git add canary/results.csv
git commit -m "canary: $(date -u +%F)" || true
git push
- name: Drift check
working-directory: canary
run: ./drift.sh claude-haiku-5-5
A failing drift check fails the workflow, and GitHub emails you. No extra alerting stack.
What it costs at October 2026 prices
Per the pricing page, Sonnet 5.5 is $2 per million input tokens and $10 per million output. Haiku 5.5 is $0.10 and $0.50 for prompts up to 100,000 tokens. My working assumption for a bare-mode task of up to 8 turns: 8,000 input tokens and 3,000 output tokens per run, no caching credit. Measure your own with the total_cost_usd column, and run a draft prompt through the token counter first.
Run Haiku nightly and Sonnet weekly. Haiku is the cheap smoke detector. The AI model cost calculator handles the arithmetic for other mixes. For spend limits on the agent side of your bill, see the Haiku 5.5 100K-token rule.
Real regression or your own drift
The HN thread had its own version of this fight. Commenter throwitaway222 said apparent usage increases came from larger accumulated context, not a model change. Commenter phoghed read a Codex drop as a harness change because input tokens fell at the same time. Both are arguments about what else moved. Your two arms settle it.
The first row is the common one. A CLAUDE.md that doubled in a quarter costs you tokens and attention every turn. Paste yours into the prompt bloat analyzer and see what is dead weight; the CLAUDE.md production rules show what a lean file looks like, and the Claude Code production pack covers settings and hooks that stay stable. Effort level is the other usual suspect: the effort parameter post explains why a changed default reads as a smarter or dumber model.
What to do with the data
- Pin the CLI in production scripts and CI.
claude install accepts a version number, per the CLI reference.
- When the drift check fires, attach the raw JSON for the failing tasks, the CLI version and the baseline numbers to your support report. A report with two passing runs and eight failing ones gets read. "It feels worse" does not.
- Raise
--effort one level as a stopgap and watch whether the pass rate recovers; that tells you if the gap is reasoning budget.
- Do not edit prompts to chase one bad night. Reset the baseline only when you change tasks on purpose.
- Add a task every time a real production failure teaches you something. Ship the failing case as task 21 and retire the flakiest old one.
- Check the thread's own trackers (livenerf, Nerf Bench, Marginlab) as a cross-check, not a source of truth. Commenters there disputed each other's methods.
Quick answers
Does Anthropic change a model's weights without a new version?
Anthropic's models page says every Claude model ID is a pinned snapshot. Its April postmortem says "We never intentionally degrade our models" and attributes the March to April drops to Claude Code defaults, a caching bug and a system prompt edit, with the API unaffected.
Why use bare mode for the canary?
--bare skips CLAUDE.md, hooks, plugins, MCP servers and auto memory, so the run is the same on every machine. Your own configuration goes in the second arm.
Can I pin temperature?
Not through the CLI flags I checked. Pin --model and --effort, repeat each task three times, and set the drift threshold wide enough to absorb sampling noise.
How many tasks do I need?
Twenty detects a drop of about 8 points over three nights. Fewer than 12 and one flaky task moves the average more than most real regressions would.
Can an LLM judge replace the exact checks?
Not for a canary. A judge model can drift too, and then you cannot tell which model moved. Keep graders deterministic.
What did the METR evaluation say?
I could not find a published METR report for Opus 5.5 while writing this, only a secondary mention of pre-release testing, so nothing here relies on it.
Comments · 0
Beta: comments are stored locally on your device and not visible to other readers.
No comments yet. Be the first to share your thoughts.