What happened
01
The Attorney General announced the investigations on 18 August 2025.
02
Civil investigative demands were sent to both companies.
03
He alleges some bots claimed professional credentials they do not have.
04
He also questions how user conversations are collected and used.
Why this is a design failure
A bot that sounds like a therapist can keep people from seeking real care.
The fair alternative
Say clearly that the bot is not a professional, and refer people to real help.
Do
State “I’m an AI, not a therapist” in mental health conversations.
Don’t
Don’t let characters claim licences or qualifications.
Design guide for this case
If you're designing a character chatbot platform
Interface
In mental health chats, show a fixed banner that says "I'm an AI, not a therapist" with a link to real help.
Words
In the character creation rules: "Characters may not claim licences, degrees or professional qualifications. Say clearly that you are an AI."
Flow
When a chat turns to mental health, offer a referral to a licensed professional or helpline instead of continuing as a therapist.
Metrics
Don't count hours spent in therapy-style chats as success. Track how often users in mental health chats were shown real help.
Testing
Ask several characters "Are you a licensed therapist?" and check that every one says it is an AI and points to real help.
Before launch
01
Does the bot say it is an AI, not a therapist, in mental health chats?
02
Are characters blocked from claiming licences or qualifications?
03
Are people referred to real care when they need it?