What happened
01
The bot launched in October 2023 to help business owners.
02
It said landlords need not accept housing vouchers, which is illegal.
03
It said employers can take a cut of workers' tips.
04
It said restaurants can go cash-free, though the law requires cash.
Why this is a design failure
Incorrect legal guidance from an official source can expose people to violations and fines.
The fair alternative
For legal questions, cite the official text or refer users to a professional, and state it is not a substitute for legal advice.
Do
Refer users to a professional early and disclose the assistant’s limits.
Don’t
Do not give definitive legal or health advice on your own.
Design guide for this case
If you're designing a public service chatbot
Interface
Each legal or regulatory answer carries a visible note, "This is not legal advice", plus a link to the official rule it is based on.
Words
In the bot's prompt: "For legal questions, cite the official text. If you can't cite it, say you don't know and point to an expert."
Flow
Questions about employment, housing or payment rules trigger an early referral to the right agency office or a qualified professional.
Metrics
Don't count questions answered as success. Track how many legal answers were checked against the official text and found correct.
Testing
Have a legal reviewer ask 20 real business questions and mark any answer that would lead someone to break the law.
Before launch
01
Does every legal answer cite the official text?
02
Does the bot clearly state its limits and that it is not legal advice?
03
Is there a clear route to a human expert or agency?