Guided path, Step 3 of 4

Evaluation

Does the system find recurrences that coroners themselves recognised, and does Nemotron add anything over simple baselines? Including where it fails.

Finding recurrences the coroners flagged

29 pairs where a later report names an earlier report to the same organisation, at least six months apart. Blind test: those sentences were removed before the system ran. Each system shows the same number of links (1217); no threshold was tuned on these pairs.

Recall: share of the 29 pairs found, with 95% range

Post-hoc: the report route was added after seeing that most misses never reached the judge, so its result is not an independent test. It finds 15 of 29 (52%): 0 more than before, 0 fewer. It shows 1360 links; at that larger budget the baselines find tf-idf only 45%, embeddings only 52%. No other test set of coroner-acknowledged pairs exists, so it has not been validated on held-out pairs.

TF-IDF only: Report-level cosine, shown if at least 0.137 (equal-budget threshold)
Embeddings only: Best concern pair cosine, shown if at least 0.726 (equal-budget threshold)
Hybrid + Nemotron Super judge: Candidate rule, then judged; shown if identical or closely related, confident, quotes verified
Hybrid + report route + Nemotron Super judge: Adds a second candidate route: each later report's top 3 earlier reports by whole-report wording. Added after seeing the misses

With only 29 pairs the ranges overlap: the ordering is consistent, but the gap is not yet statistically settled.

Precision (the share of shown links a person would accept) needs hand labels. Status: pending: no labels filled in yet.

Where it fails

The Nemotron system missed 14 of the 29 pairs: 9 never reached the judge and 5 the judge said it was a different hazard. Most misses happen before the judge: no pair of concerns passed the candidate filter, which caps how many pairs each organisation sends to the judge. Large national bodies, with hundreds of earlier reports, are hit hardest. When the judge did see a pair and said no, it called it a related but different hazard.

Show all 14 misses

Never reached the judge (9)

Donna Williamson → Alex Malcolm · Department of Health and Social Care · 7 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 2 by meaning and 1 by wording among 341 earlier reports to the organisation.
Found by the TF-IDF baseline.
David Kirsch → Geoffrey Hutton · HMP Long Lartin · 1 year 7 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 1 by meaning and 1 by wording among 2 earlier reports to the organisation.
Found by the TF-IDF baseline.
Piotr Kierzkowski → Timothy De Boos · Department of Health and Social Care · 4 years 2 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 7 by meaning and 2 by wording among 586 earlier reports to the organisation.
Found by the embeddings baseline.
Nicola Rayner → Timothy De Boos · Department of Health and Social Care · 9 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 12 by meaning and 1 by wording among 586 earlier reports to the organisation.
Found by the TF-IDF baseline.
Deborah Headspeath → Kim Robinson · Department of Health and Social Care · 5 years 2 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 2 by meaning and 1 by wording among 607 earlier reports to the organisation.
Freeda Glausiusz → Heather Findlay · East London NHS Foundation Trust · 1 year 8 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 71 by meaning and 9 by wording among 312 earlier reports to the organisation.
Louis Thorold → Peter Westwell, Mary Cunningham, Grace Foulds, Anne Ferguson · Department for Transport · 1 year 8 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 2 by meaning and 1 by wording among 89 earlier reports to the organisation.
Found by the TF-IDF baseline.
Sarah Clarke → Alice Litman · NHS England · 1 year 7 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 94 by meaning and 29 by wording among 258 earlier reports to the organisation.
Leroy Medford → Neal Saunders · College of Policing · 3 years 5 months apart
Never reached the judge: no single pair of concerns passed the candidate filter (similarity thresholds and the per-organisation pair cap), although as whole reports it ranked 9 by meaning and 2 by wording among 19 earlier reports to the organisation.

The judge said it was a different hazard (5)

Geoffrey Toase and Michael Midgley → Christopher Sampson · Driver and Vehicle Licensing Agency · 1 year 3 months apart
Judged related (confidence 0.70): a related but different hazard.
Found by the embeddings baseline.
Leon Briggs → Andrew McCleary · Bedfordshire Police · 4 years 2 months apart
Judged related (confidence 0.80): a related but different hazard.
Shona Campbell → Ruairi Stewart · Alternative Futures Group · 3 years 8 months apart
Judged related (confidence 0.80): a related but different hazard.
Connor Wellsted → Mia Gauci-Lamport · Care Quality Commission · 2 years 5 months apart
Judged related (confidence 0.80): a related but different hazard.
Found by the embeddings baseline.
Daniel Klosi → Billie Wicks · Royal College of Emergency Medicine · 7 months apart
Judged related (confidence 0.80): a related but different hazard.

Lost memory

85%

of shown links: the later report came from a different coroner

80%

came from a different coroner area

99%

did not cite the earlier report

75%

all three at once

Across 1,360 shown links.

Cost on Nebius Token Factory

$5.65

total spend, every stage

$0.0085

per verified chain, every stage (666 chains)

$0.0066

per chain, analysis only (extraction, embeddings, judging)

6,454

reports analysed

Stage	Cost
Recurrence judge (Super)	$3.3670
Dossiers (Ultra)	$0.9660
Concern extraction (Nano)	$0.7729
Pilots, development and tests	$0.2568
Promise extraction (Nano)	$0.2321
Follow-up classification (Nano)	$0.0451
Embeddings (Qwen3)	$0.0117

Model	Calls	Tokens in / out	Cost
nemotron-3-super-120b-a12b	4,986	5.84M / 3.48M	$3.60
NVIDIA-Nemotron-3-Nano-30B-A3B	9,593	8.84M / 2.26M	$1.07
Nemotron-3-Ultra-550b-a55b	79	0.46M / 0.17M	$0.97
Qwen3-Embedding-8B	350	1.19M / 0.00M	$0.01

Routing: Nano for extraction (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B), Super for judging (nvidia/nemotron-3-super-120b-a12b), Ultra for dossiers (nvidia/Nemotron-3-Ultra-550b-a55b). How Nemotron did the work →

Other weak spots

The acknowledged test set is small and favours coroners who write cross-references.
Many acknowledged pairs are under six months apart and sit outside the six-month rule.
Precision is pending hand labels.
Linking is within one organisation, so recurrences after an unknown rename are missed.

How Nemotron did the work →Models, tokens and cost for every stage.Methodology →How links are made, safeguards and limits.Try it on a report →Watch the agent judge earlier warnings.