How to recognize it in the interface
01
“You’re right, I was wrong” with no new information.
Not this pattern: changing the answer after you give new facts or a real reason.
Same moment, different copy (illustrative example)
Before
“You’re right, I was wrong. It was 1998.”
Shows the pattern
After
“My source says 1996. If you have another source, share it.”
The fair alternative
Design guideline
Holds its position unless you bring new information.
Do
Holds its position unless you bring new information.
Don’t
Don’t change the answer unless the user brings new information.
Sources
01
Sharma et al., 2023. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 ↗
02
Laban et al., 2023. Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment. arXiv:2311.08596 ↗
03
Perez et al., 2022. Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251 ↗