What happened
01
The Texas Attorney General announced the settlement on 18 September 2024.
02
At least four Texas hospitals were sending patient data to Pieces.
03
The AG said its accuracy metrics may have misled hospitals.
04
Pieces must explain the limits of its products to hospital staff.
Why this is a design failure
A near-perfect accuracy claim makes busy clinicians trust summaries they should check.
The fair alternative
Report how errors were measured, and tell staff when to double-check.
Do
Show how accuracy was tested, next to the claim.
Don’t
Don’t advertise error rates you cannot reproduce.
Design guide for this case
If you're designing a clinical AI summary tool
Interface
Put a short "How we measured this" link next to every accuracy claim, and a visible "Check against source" note on each summary.
Words
In the content guide: "Never state an error rate we cannot reproduce. Say what was tested, on what data, and what the limits are."
Flow
Add a step where clinicians open the source note before signing off on a summary for high-risk items like medications or allergies.
Metrics
Don't count a headline error rate as proof of safety. Track how often clinicians catch and correct errors in real summaries.
Testing
Give 5 clinicians summaries with planted errors and record how many they catch, with and without the limits notice shown.
Before launch
01
Can every accuracy claim be reproduced by someone outside the team?
02
Does the tool tell staff when to double-check?
03
Are hospital staff told the product's known limits before use?