Model Regression Canary: Nightly Claude Eval with Drift Alerts

A Hacker News thread asking whether Opus 5.5 had been nerfed reached 924 points, and almost nobody in it had a baseline. The Model Regression Canary gives you one: twenty frozen tasks run nightly at 02:17 UTC on a pinned model and effort level, code grades every answer, and a drift check compares the night with your baseline. When the pass rate drops 10 points, or three tasks that always passed start failing, a Telegram or Slack message names the tasks and what else changed that night, such as the effort level.

It is the finished, tested version of the WOWHOW guide Is Claude Nerfed? Build a 20-Task Eval Canary in an Afternoon: same task mix, graders that run code, nightly Action and drift alert. Node 22.18 or 24, no npm packages.

What You Get

• canary.mjs: runs each task three times on the Anthropic Messages API with output_config.effort pinned, takes a majority vote, and records pass or fail, tokens, cost, latency and every failed answer in results/<UTC>.json. It also drives any OpenAI-compatible endpoint, and claude -p in bare or your-config mode to separate model drift from harness drift.
• drift.mjs: compares a night with baseline.json and writes drift.md plus a chat-sized summary.
• 20 tasks, 40 fixtures: a correct and a regressed answer per task; scripts/smoke-test.sh (34 offline checks) proves every grader passes one and fails the other, on macOS and Linux.
• .github/workflows/canary.yml: 02:17 UTC cron, alerts on drift only, results committed back with permissions: contents: write.
• README, USAGE-GUIDE, VERIFY.md, prices checked on 15 October 2026, single-buyer licence.

The 20 tasks

The drift rule

Drift fires when the pass rate falls 10 or more points below the baseline, or when 3 or more stable tasks (passing at least 80% of baseline runs) flip to failing. Flaky tasks cannot trigger the flip rule; a task edited since the baseline is caught by hash. The guide's slower 3-night, 8-point rule is one flag away: --window 3 --metric trials --threshold 8.

Tested on 15 October 2026

Six live runs on openai/gpt-oss-120b (Groq free tier) passed 17 to 19 of 20 tasks at medium reasoning effort and 16 of 20 at low. Against the medium runs as baseline, the drift check fired on the low-effort runs by both rules and named effort: medium -> low as the change. The results files ship in the zip; VERIFY.md has every command and its output. No Anthropic call was made in testing: your first night is the first real Claude score.

Cost per night

At October 2026 list prices, three runs per task cost about $0.034 a night on Haiku 5.5 ($1.02 a month) and $0.68 on Sonnet 5.5. Haiku nightly plus Sonnet weekly is about $4 a month. Price your mix in the free Always-On AI Agent Cost Simulator.

How to Set It Up

• Unzip and run scripts/smoke-test.sh: expect 34 passed.
• Push it to a GitHub repo; add ANTHROPIC_API_KEY as an Actions secret, plus TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID or SLACK_WEBHOOK_URL.
• Run the workflow once by hand. Drift checks begin after 14 nights, or tomorrow if you pool three manual runs with scripts/set-baseline.sh --first 3.
• On an alert, read drift.md: regressed tasks, failed answers, what else changed.

What is not included

• API usage. You pay Anthropic or your provider for the calls.
• An LLM judge. Graders check only what code can check.
• A sandbox. Model-written code runs in a temp directory with an empty environment, Node's permission model and a 10-second limit: a guardrail, on a runner GitHub discards after each job.
• A dashboard. Results are JSON files in your repo and a Markdown report per night.

Notes

Other models: any Claude id in the price file works, Opus 5.5 and Fable 5.1 included; OpenAI-compatible models run with cost left blank. Your own tasks: one JSON file plus a fixture pair each; the best canary tasks come from your real failures. Spend: pair it with Claude Code Spend Guard for a hard daily cap on Claude Code sessions.