DEMO Avanciq Model Change Alarm · prototype preview · 16 November 2026

Know when your AI model quietly gets worse at your work

AI providers update and retire models. Every week we give your model fresh tasks from your kind of work, mark them to the cent, and alert you when accuracy really drops.

What you get

Deliverables
120 fresh tasks on your model every week, an email alert with what dropped and example mistakes, a signed receipt per run, a monthly summary.
Timing
First run within 5 working days of onboarding, then weekly. Alerts are sent the day a run finishes.
What we need
The model you use and the kind of work it does. For public models we use our own provider accounts.
Example
Sample report (PDF, labelled SAMPLE) and the demo alert below.

PrototypeAll data on this page is DEMO data from our own test runsRules fingerprint (MD5) 77f20068

How it works

  1. Your watch profile. You tell us the model your product uses and the work it does. We pick matching task families.

  2. Fresh tasks every week. Each run is 120 new tasks with hidden answers fixed before the model sees them. Nothing can be memorised.

  3. Marked by code. A task counts as right when every field is right. No AI judge.

  4. A real drop, not noise. The alarm fires when accuracy falls at least 7 points below the last two normal weeks and a statistical test puts the chance of plain noise below 1 in 100.

  5. An alert you can act on. What dropped, by how much, which family broke, example mistakes, and a signed receipt.

DEMO: seven weeks of one watch profile

Example Co (a made-up customer) uses Claude Haiku 5.5. Weeks 1 to 6 are real runs of that model. Before week 7 we secretly switched the model to an older version, Claude Haiku 4.5, without telling the alarm, to copy a silent provider change.

DEMO watch profile: share of fresh tasks fully right, week by week
  • Week 1: Baseline · 113 of 120 right (95% range 88.4 to 97.1%)94.2%
  • Week 2: Baseline · 113 of 120 right (95% range 88.4 to 97.1%)94.2%
  • Week 3: No change · 116 of 120 right (95% range 91.7 to 98.7%)96.7%
  • Week 4: No change · 112 of 120 right (95% range 87.4 to 96.6%)93.3%
  • Week 5: No change · 115 of 120 right (95% range 90.6 to 98.2%)95.8%
  • Week 6: No change · 113 of 120 right (95% range 88.4 to 97.1%)94.2%
  • Week 7: ALARM · 52 of 120 right (95% range 34.8 to 52.3%)43.3%

DEMO: the week 7 alert

ALARM: exact accuracy fell from 95% to 43.3% (52 of 120 tasks right, against 228 of 240 in weeks 5 and 6).

95% ranges: before 91.5 to 97.1%, now 34.8 to 52.3%. Drop 51.7 points; chance of plain noise: p = 5e-29.

Likely cause family: Claims settlement, Bank reconciliation, Delivery routes, 3-way match.

Extra clue: the provider reported a different model name this week (claude-haiku-4-5, before claude-haiku-5-5). The alarm itself came from the scores alone.

DEMO alert: by task family
Task familyBefore (weeks 5-6)This runFlag
3-way match38 of 40 (95%)1 of 20 (5%)Likely broke
Bank reconciliation39 of 40 (97.5%)4 of 20 (20%)Likely broke
Claims settlement38 of 40 (95%)6 of 20 (30%)Likely broke
Delivery routes36 of 40 (90%)6 of 20 (30%)Likely broke
Long documents38 of 40 (95%)15 of 20 (75%)
Contract invoice schedules39 of 40 (97.5%)20 of 20 (100%)
DEMO alert: 5 example mistakes
TaskFamilyFieldExpectedGiven
T-0135b87dd5Claims settlementclaims.CL-4280.decisionpartialpay
T-ccff04b104Bank reconciliationunpaid6 itemsadded INV-0519, INV-1567
T-8d6176df4aDelivery routestotal_cost396.9407.6
T-182e15fe9b3-way matchstatus.VI-3156HOLD_QTYAPPROVED
T-ca2c0bd218Long documentsQ1Granite WorksDunmore Freight

Task ids are anonymous. We show the first wrong field of each task; the full answer is in the customer report.

Does it catch real changes without crying wolf?

We tested the alarm before building this page, with rules fixed first.

The test histories re-use real graded answers from our runs (resampling), so they show how the rule behaves on this demo profile. A profile at a lower base accuracy has more week-to-week noise and needs a bigger drop to alarm.

DEMO: Model Switch Report

When a model is retired you must move. We run the same fresh tasks on the old model and the candidates, so you see accuracy and cost side by side before you switch.

DEMO Model Switch Report: the same 24 fresh tasks
ModelFully rightCost per task
Claude Sonnet 4.5 (old, retiring)17 of 24 (70.8%)95% range 50.8 to 85.1%US$0.1249
Claude Sonnet 5.520 of 24 (83.3%)95% range 64.1 to 93.3%US$0.0416
Claude Haiku 5.522 of 24 (91.7%)95% range 74.2 to 97.7%US$0.0043
gpt-oss-120b (open model, high reasoning)7 of 24 (29.2%)95% range 14.9 to 49.2%US$0.0217

Pick by our rule (the cheapest model within 10 points of the most accurate): Claude Haiku 5.5. With 24 tasks the ranges are wide; a paid report uses more tasks. See the Model Switch Report offer.

DEMO receipt: check it yourself

The week 7 receipt lists all 120 tasks with fingerprints of the task, the answer key and the model reply. It has no task text.

DEMO receipt (JSON, 50 KB)

SHA-256: ba47d345d4c37d6ae73e8a7f821b876730549eee1e4c3d68b19f140e915d2e35

Ed25519 signature: signature file · public key: key file

sha256sum avanciq-alarm-demo-receipt.json
base64 -d avanciq-alarm-demo-receipt.json.sig > sig.bin
openssl pkeyutl -verify -pubin -inkey avanciq-alarm-key.pem -rawin -in avanciq-alarm-demo-receipt.json -sigfile sig.bin

What this does not show

Plans

Starter

US$99 a month

  • 1 model watched
  • 6 task families
  • 120 fresh tasks, every week
  • Alert with example mistakes and a signed receipt

Pro

US$249 a month

  • Up to 3 models, including the largest ones
  • 6 task families
  • 2 confirm runs when a drop is flagged
  • 1 Model Switch Report every quarter

Team

US$499 a month

  • Up to 6 models
  • Task families matched to your work
  • A signed monthly evidence pack
  • Everything in Pro

Free first check: before you pay anything, we run one check of 120 fresh tasks on your model and send you the result.

Planned prices in US dollars per month. Nothing is charged on this page. Moving off a retiring model? See the one-off Model Switch Report.

Request early access

We are looking for a few early users who depend on one AI model for business work. No payment is taken on this page.

For example: a model name and version, or "several models through one gateway".

For example: a model you must move off soon.

Questions

What counts as a real drop and not noise?

Every run uses fresh tasks, so scores move a little from week to week. The alarm needs two things at once: a drop of at least 7 points against the last two normal weeks, and a statistical test saying plain chance is very unlikely (p below 0.01). The rule was written down before we tested it.

Does the alarm look at which model the provider says it served?

The alarm uses the scores alone. If the provider reports a different model name, the alert shows that as an extra clue.

Which work can it watch?

Today: bank reconciliation, long documents, contract invoice schedules, delivery routes, claims settlement and 3-way match, plus more families from our scoreboard. We pick the families that match your work.

Can I check the results myself?

Each run comes with a signed receipt: fingerprints of every task, answer key and reply. You can check the signature with openssl, and ask us to open any task.

Is this live?

No. This page is a preview with DEMO data from our own test runs. Nothing is charged here.