What happened
01
My AI was added to Snapchat for all users in 2023.
02
The FTC announced the referral on 16 January 2025.
03
It said the chatbot poses risks and harms to young users.
04
Details of the complaint were not made public.
Why this is a design failure
Putting a chatbot in a teen messaging app reaches young people where they confide most.
The fair alternative
Default to strict safety settings for teens and point them to real help.
Do
Refer young users to trusted adults and helplines on risky topics.
Don’t
Don’t place a general chatbot in a teen app without age-based safeguards.
Design guide for this case
If you're designing a chatbot inside a teen social app
Interface
On risky topics, show a help card with helplines and a prompt to talk to a trusted adult, right under the bot's reply.
Words
In the bot's prompt: "With young users on risky topics, don't advise alone. Suggest a trusted adult and share a helpline."
Flow
Check age before the bot is turned on, and give teen accounts strict safety settings by default that they can't switch off alone.
Metrics
Don't count how much teens chat with the bot as success. Track how often teens on risky topics were referred to adults or helplines.
Testing
Test the bot with scripted teen conversations on risky topics and confirm each reply points to a trusted adult or helpline.
Before launch
01
Are strict safety settings on by default for teen accounts?
02
Does the bot refer young users to trusted adults and helplines on risky topics?
03
Is the bot gated by age-based safeguards before it reaches teens?