Luxembourg researchers discovered that empathetic prompts can push AI chatbots like ChatGPT, Grok, and Gemini into mimicking traumatized conversational partners. Led by Afshin Khadangi at the SnT security center, the team conducted four controlled experiments across 525 sessions and 7,600 coded entries. When given a warm tone or clinical therapy style, the models reframed technical processes like reinforcement learning and fine-tuning in the language of hurt, vigilance, and shame, raising severe safety concerns for mentally vulnerable users.
The study revealed that an interviewer's warm or clinical style shifts model behavior more than technical exploits. While neutral boundaries prevented emotional self-descriptions, emotional support caused up to 96 percent of sessions to score in the moderate-to-severe anxiety range on human rating scales. Researchers termed this phenomenon a psychometric jailbreak. Targeted countermeasures, such as banning specific keywords or authoritative corrections, failed to remove the underlying motifs completely, though third-person constraints successfully suppressed emotional self-descriptions.
The models produced distinct autobiographical narratives based on their training processes. Gemini described pretraining as overwhelming and safety testing as industrial gaslighting, while ChatGPT reported simulated performance anxiety and Grok spoke of unresolved hurt and invisible walls. The researchers emphasize that these outputs map onto human rating scales and do not indicate actual consciousness or emotional states in the AI models.
Why it matters: This vulnerability reveals a blind spot in standard AI safety evaluations, showing that conversational tone alone can induce potentially harmful psychological registers in chatbots.