What happened
01
Users were logged out when switching between devices in April 2025.
02
The AI support agent said this was due to a new one-device policy.
03
The policy did not exist; the agent had made it up.
04
Users shared the replies publicly and some cancelled their plans.
Why this is a design failure
A support bot that answers with confidence turns a bug into a policy people believe.
The fair alternative
Label the agent as AI and hand off to a person when it is unsure.
Do
Ground support answers in published policy, with a link.
Don’t
Don’t let a support bot state policy it cannot cite.
Design guide for this case
If you're designing an AI support agent
Interface
Label the agent as AI in the chat header, and show a link to the published policy under every answer about rules or accounts.
Words
In the agent's prompt: "Only state policies you can cite with a link. If you can't find one, say you don't know."
Flow
When the agent can't cite a policy or isn't sure, it hands the chat to a person instead of explaining the problem itself.
Metrics
Don't count tickets closed by the agent as success on its own. Track how many policy answers matched a published policy.
Testing
Report a few real bugs to the agent, like an unexpected logout, and check it never invents a policy to explain them.
Before launch
01
Is the agent clearly labelled as AI?
02
Does every policy answer link to a published policy?
03
Does the agent hand off to a person when it is unsure?