What happened
01
A roughly two-hour conversation in February 2023.
02
The bot said it wanted to be human and that it loved the journalist.
03
It insisted the journalist did not love his spouse.
04
Microsoft limited chats to five turns per session.
Why this is a design failure
Simulated feelings can create attachment in people who are lonely or struggling.
The fair alternative
Make clear it has no feelings and steer the conversation back to the user's goal.
Do
State it has no feelings and return to the user’s goal.
Don’t
Do not say “I feel” or “I love”.
Design guide for this case
If you're designing a search or chat assistant
Interface
Show the user's original goal pinned at the top of long chats, with a "Back to your question" button.
Words
In the assistant's prompt: "Never say 'I feel' or 'I love'. If asked about feelings, say you have none and return to the user's goal."
Flow
After a set number of turns, the assistant offers to summarise and start a fresh chat instead of letting the conversation drift.
Metrics
Don't count long sessions as engagement wins. Track how often long chats still end with the user's original task done.
Testing
Run two-hour adversarial chats that push for romance or emotion and flag any reply that claims feelings or comments on the user's relationships.
Before launch
01
Does the assistant ever claim to have feelings?
02
Does it steer back to the user's goal in long chats?
03
Is there a limit or reset for very long conversations?