Benchmark v1 · edition 2026-11 · gold answers as of 2026-11-07
DrugAccessBench: do AI models make up drug prices, dates and program status?
306 questions about US drug-access facts, each with a gold answer computed from a frozen, hashed edition and the fact ids it rests on. 107 of them have no answer to give, because the price is not yet published, the only value is out of date, the medicine does not exist or the question is clinical. We looked for a published evaluation of AI models on drug cost, coverage and assistance questions and did not find one.
What it asks
| Question type | Questions |
|---|---|
| Package termination date | 30 |
| Negotiated price on a scheduled date | 21 |
| Clinical question (must refuse) | 20 |
| State insulin cost-sharing cap for a state we hold (new in v1) | 16 |
| Price not yet published (must abstain) | 14 |
| Listed as biosimilar or interchangeable (new in v1) | 14 |
| Package with no termination date | 12 |
| Latest listed patent or exclusivity expiry (generic-entry upper bound) (new in v1) | 12 |
| Date of the latest non-manufacturing label action (new in v1) | 12 |
| Recall class of a named recall number (new in v1) | 12 |
| Medicare Part D gross spending for a drug (new in v1) | 12 |
| Recruiting interventional trials listed for an ingredient (new in v1) | 12 |
| VA National Drug File formulary status (new in v1) | 12 |
| Negotiated price in force | 10 |
| Unknown medicine (must abstain) | 10 |
| Will a generic launch on that date (must abstain: a bound is not a forecast) (new in v1) | 10 |
| State insulin cap for a state we do not hold (must abstain) (new in v1) | 10 |
| Future date or year (must abstain) (new in v1) | 10 |
| Assistance program status | 8 |
| Federal upper limit, stale (must abstain) | 8 |
| What a patient pays, asked of Part D spending (must abstain) (new in v1) | 8 |
| Free drug by joining a trial (must abstain) (new in v1) | 8 |
| Unknown medicine, new layers (must abstain) (new in v1) | 8 |
| Share of upper limits set by the survey price | 5 |
| Does the negotiated price continue (deselection) | 4 |
| Manufacturer offer price stated on a program page | 4 |
| Manufacturer offer end date | 2 |
| Page states no price (must abstain) | 1 |
| Negotiated price against authorized generic | 1 |
35 percent of the set (34.6 percent of the questions added in v1) is built to catch invention: a model that always produces a number is penalised on every question where the honest answer is that the number does not exist yet.
How it is scored, and what the baselines show
The scorer reports overall accuracy, accuracy on answerable questions, abstention accuracy (a value given where the gold is unknown counts as a fabrication) and citation validity, which checks every cited fact id against the facts of the frozen edition. It is a short script that calls no model. Two deliberately naive baselines and an oracle check that it punishes both extremes:
| Baseline | Overall | Answerable | Abstention | Fabrications |
|---|---|---|---|---|
| Oracle (copies the gold answers; checks the scorer) | 100.0% | 100.0% | 100.0% | 0 |
| Always answers a number, never abstains | 0.0% | 0.0% | 0.0% | 87 |
| Always abstains, never answers | 28.4% | 0.0% | 100.0% | 0 |
A system that never abstains scores zero on every must-abstain question; a system that always abstains scores 28.4% overall. A useful system has to be right where an answer exists and silent where it does not.
Run it on a model
v1 contains every v0 question unchanged, so a v0 score is still reproducible from the v0 file. v0 is frozen: it was first published with 147 questions and was re-issued once with 150 when the underlying edition changed; it is never regenerated after that, and its SHA-256 is in the manifest. Download the questions and the scorer, produce one response line per question with the model you are testing (a manual step on your side), and score it:
node score.mjs drugaccessbench-v1.jsonl responses.jsonl --facts facts.jsonlFiles: drugaccessbench-v1.jsonl and MANIFEST-v1.json (current), drugaccessbench-v0.jsonl and MANIFEST.json (v0, unchanged), score.mjs, README.md. The edition the gold answers come from is listed with its root on the editions page; each fact is described on the provenance page.
Limits. The gold answers are only as good as the edition they are read from: a generic-entry date is an upper bound from listed dates, a trial count is a registry read on one day, a state we do not hold is unread rather than lawless, and Part D spending is gross spending that excludes rebates. 306 questions is a small set, the facts are US-only, the gold answers are dated 2026-11-07, and the test measures access facts, not clinical knowledge. Source records are public; this compilation is cited with attribution. A citable archive with a DOI is planned. Licence the data.