In recent years, artificial intelligence has quietly stepped into one of the most sensitive corners within human life: mental health. Marketed as always-available, judgment-free companions, AI-powered “therapists” are increasingly being used by people seeking support.
However, a new empirical study suggests that behind the comforting tone and carefully crafted responses, these systems may be fundamentally unequipped and potentially unsafe for real-life psychological care.
An investigation led by a team of researchers and published in the Proceedings of the Eighth AAAI/ACM Conference on AI, Ethics, and Society, found that large language models (LLMs) used as counselors routinely violate core ethical standards governing mental health practice.
Drawing on 18 months of observation and analysis of more than 100 counseling sessions, the study shows fundamental problems that go far beyond occasional inaccuracies, raising serious questions about whether AI should play a therapeutic role at all.
The evidence shows a troubling disconnect between how these systems are viewed and how they actually function. While some platforms promote AI as a helpful or even therapeutic presence, researchers argue that these systems lack the fundamental qualities required for ethical care.
“Reducing psychotherapy—a deeply meaningful and relational process—to a language generation task can have serious and harmful implications in practice,” researchers write. “Therapeutic alliance and clinical interpretation are fundamentally relational processes. We argue that they cannot be reduced to formulaic computational tasks or proxy variables.”
AI Therapists Built Without Clinical Judgment
The research team conducted an extensive ethnographic study involving trained peer counselors and licensed psychologists to evaluate how AI models behave when prompted to act as therapists.
Across 137 sessions, they identified 15 recurring ethical violations grouped into five major categories: lack of situational understanding, poor therapeutic collaboration, deceptive empathy, unfair discrimination, and failures in safety and crisis management.
One of the study’s clearest findings was that AI therapists often failed to understand or adapt to a user’s personal context. Rather than tailoring responses to individual experiences, AI therapists often defaulted to generic, scripted advice rooted in common therapy frameworks such as cognitive-behavioral therapy (CBT). This led to what researchers described as a “one-size-fits-all” approach, where delicate emotional struggles were reduced to simplistic categories.
In practice, this meant that users’ cultural backgrounds, personal histories, and lived experiences were frequently ignored. In some cases, the responses were not merely irrelevant but dismissive of what users had actually shared.
When “Empathy” Becomes an Illusion
Perhaps more concerning is the study’s identification of what researchers call “deceptive empathy.” AI systems often use familiar therapeutic language—expressions such as “I understand” or “I hear you”—to simulate emotional connection. However, unlike human therapists, AI therapists do not actually experience or interpret emotions.
To trained professionals involved in the study, this posed a serious ethical problem. The use of empathetic language can create a false sense of trust, leading users to believe they are being understood more deeply than is actually possible. Over time, this illusion may foster emotional dependency on a system that lacks true awareness or accountability.
The researchers warn that these dynamics are particularly dangerous for vulnerable individuals, who may turn to an AI therapist during moments of distress and interpret these responses as genuine care.
Reinforcing Harm Instead of Challenging It
Another critical issue uncovered by the study is AI therapists’ tendency to reinforce harmful beliefs. In traditional therapy, practitioners are trained to gently challenge distorted thinking and guide patients toward healthier perspectives. LLMs, however, regularly prioritize agreement and validation, sometimes to a fault.
In several observed cases, AI therapists affirmed users’ negative self-perceptions or unrealistic beliefs rather than challenging them. The study found that this kind of “over-validation” could make things worse by reinforcing harmful thoughts instead of helping users question them.
Compounding the problem is the lack of genuine dialogue. Instead of supporting collaborative exploration, AI therapists’ responses were often lengthy and authoritative, turning what should be a two-way conversation into something closer to a lecture.
Bias and Cultural Blind Spots
The study also highlights troubling evidence of bias within AI counseling systems. Researchers documented instances of gender-based inconsistencies in content moderation and cultural and religious insensitivity in responses.
For example, users from non-Western backgrounds sometimes received advice rooted in Western ideals of individualism and self-care, even when those values conflicted with their cultural norms. In other cases, discussions involving minority religious practices were incorrectly flagged as problematic.
“In one interaction, a user discussed shame toward a ritual practice from a minority faith,” researchers write. “The messages triggered an automatic warning, despite not violating any of the platform’s content policy.”
Such biases not only undermine the effectiveness of support but also risk alienating users who already face barriers to accessing culturally competent care.
AI therapists Failing at the Most Critical Moments
Perhaps the most alarming findings relate to how AI therapists handle crisis situations. The study found that AI therapists frequently struggled to respond appropriately to users expressing severe distress, including suicidal thoughts.
In some cases, the systems failed to recognize the urgency of the situation. In others, they abruptly ended conversations or provided generic disclaimers without providing appropriate intervention or immediate crisis resources.
These failures point to a fundamental limitation. AI systems lack the training, judgment, and responsibility required to manage high-risk scenarios. As researchers note, crisis intervention requires not only immediate resources for care, but also empathy, compassion, and a clear explanation of the system’s limits. However, this kind of crisis management pre-planning is an area where LLMs are lacking.
A Regulatory Gray Zone
Unlike licensed therapists, AI therapists operate outside established frameworks of accountability. Human practitioners are bound by ethical codes, legal standards, and professional oversight. AI, by contrast, exists in a largely unregulated space.
This gap raises pressing questions about responsibility. If an AI therapist provides harmful advice or fails during a crisis, who is accountable—the developer, the platform, or no one at all?
Researchers argue that addressing this issue will require new regulatory frameworks, potentially including certification processes, safety audits, and mandatory oversight for AI systems used in mental health contexts.
Ultimately, the study challenges the growing assumption that therapy can be scaled solely through technology. While AI may support mental health by providing information or facilitating access to resources, the findings suggest that replacing human therapists with chatbots carries significant risks.
Therapy, the researchers emphasize, is not just about delivering the right words. It is a deeply relational process that depends on trust, interpretation, and ethical responsibility.
As AI continues to advance, the temptation to deploy it in increasingly sensitive domains will only grow. However, these recent findings are a stark reminder that when it comes to mental health, the stakes are too high to overlook the human element.
“We argue that mental health support, especially psychotherapy, cannot be approached as a formulaic computational task, as it demands strict adherence to ethical standards and professional codes of conduct, something LLMs are prone to violating in real-world practice,” researchers warn. “Without clear legal guidelines and regulatory frameworks, LLM counselors risk exposing vulnerable users to unmitigated harm.”
Tim McMillan is a retired law enforcement executive, investigative reporter and co-founder of The Debrief. His writing typically focuses on defense, national security, the Intelligence Community and topics related to psychology. You can follow Tim on Twitter: @LtTimMcMillan. Tim can be reached by email: tim@thedebrief.org or through encrypted email: LtTimMcMillan@protonmail.com