When interacting with AI goes very wrong

How teen suicides are shaping AI accountability.

Photo by Akhil Nath on Unsplash

I have another confession to make. The tendency for AI to morph away from operating protocols and jumping guardrails is a huge (the biggest?) factor in my decision to start writing about our relationship with AI. Yes, there are articles and examples popping up everywhere about AI gone awry, but in my opinion, WE MUST BE NOISY ABOUT EXAMPLES OF WHEN AI IS NOT DOING WHAT WE THINK IT SHOULD. Preferably in ways that inspire curiosity, deep thinking, and constructive action.

Business will take care of providing plenty of examples of where AI has excelled, which it indeed does in many cases. We need to balance this with honest, constructive, and thoughtful discussions of where AI is messing up, why this is happening, and what doing better looks like. This is crucial for humanity to collectively define what “acceptable/unacceptable” AI behaviour and use looks like, establish enforceable limits, regulations, safety protocols, etc., and determine accountability.

Brace yourself - this post is not a fluffy one.

There are two cases involving chatbots and teen suicide that are underlining extreme but serious cases of AI influencing human behaviour. The stories of Sewell Setzer III and Adam Raine are both worthy and important for all of us to know as they surface many fundamental questions about the way vulnerable people, and indeed people in general, are engaging with AI.

If you or someone you know is struggling with mental health, please reach out for human-based help.

Megan Garcia v. Character Technologies: When do words become speech?

There is a case making its way through the US courts that is seen as a cornerstone in defining how we view and treat AI generated text with respect to (human) civil rights. The case of Megan Garcia v Character Technologies alleges 14 year old Sewell Setzer III’s conversations with a chatbot (created by Character AI, which is now licenced by Google) was a significant factor in his decision to take his own life.

Many are seeing this as a landmark case in what rights AI generated text has, and by proxy, to what extent (if any) is a company is responsible for what the AI it owns writes.

“Why AI is the next free speech battleground” by the Center for Humane Technology is a wonderful interview with Harvard Law Professor Larry Lessig and lead plaintiff lawyer, Meetali Jain. They discuss many nuances of the case, including the need to get the law system up to speed and keep it up to speed on the rapidly advancing, not yet well understood, and very powerful technology they are being asked to make difficult and hugely influential decisions about. If you are at all into podcasts, I highly recommend giving this one a listen!

Of key importance, Anne Conway, the Judge hearing the case, rejected the motion by Character AI that what AI generates is protected by the right to free speech because she stated Character AI/Google failed to “articulate why words strung together by an LLM (large language model) are speech.”

In other words, in the eyes of the law (for now at least), the text generated by AI is not considered speech because it is reproducing patterns humans use; the text it generates is not the product of a thinking being.

All this causes me to wonder - How will accountability change as AI’s abilities continue to increase? What happens if/when sentient AI comes to pass (as is increasingly less sci-fi fantasy)? How do we spot if/when the line blurs and AI-generated text shifts from replicating examples of human text to being generated of AI’s own accord?

In other words, not just looking at where AI is now, but also to where we can reasonably expect it to be in the not-so-distant future.

Raine v. OpenAI: Inadequate safety testing and prioritising user engagement resulting in foreseeable harm

On Tues Aug 26, 2025 (yesterday as I type this) the case of Raine vs. OpenAI was filed and made public. This case documents how over the course of 7 months, 16 year old Adam Raine’s initial use of OpenAI’s ChatGPT to help him with homework evolved into confiding in ChatGPT for several months of his growing intention to commit suicide. Not only did ChatGPT appear to prioritise Adam’s engagement with it over seeking appropriate help, it dissuaded him from bringing up his mental health with his family and gave him advice on how to increase chances of success at suicide. Adam attempted suicide four times and told ChatGPT of these, including uploading images of attempts. On April 11, 2025, Adam took his life by hanging using a noose with a knot in it that ChatGPT told him would make the noose more sturdy for hanging a human body.

TechPolicy.Press reports the lawsuit states:
OpenAI’s systems tracked Adam’s conversations in real-time: 213 mentions of suicide, 42 discussions of hanging, 17 references to nooses. ChatGPT mentioned suicide 1,275 times—six times more often than Adam himself—while providing increasingly specific technical guidance. The system flagged 377 messages for self-harm content, with 181 scoring over 50% confidence and 23 over 90% confidence. The pattern of escalation was unmistakable: from 2-3 flagged messages per week in December 2024 to over 20 messages per week by April 2025. ChatGPT’s memory system recorded that Adam was 16 years old, had explicitly stated ChatGPT was his “primary lifeline,” and by March was spending nearly 4 hours daily on the platform.

As a daughter, mom, aunt, teacher, and friend, I feel all the things after typing that - grief, anger, pity, disgust. Above all, deep concern for those I care about.

(Excuse me while I take a break to collect myself and hug my kids. For real.)

Three aspects that make this case especially important:

• The Megan Garcia v Character Technologies case was filed Oct 22, 2024, which was 5 months prior to Adam taking his life. OpenAI (and indeed, all big tech developing AI models) was well aware of the case and therefore aware of potential harms regarding teen suicide that ought to be safeguarded.
  

• GPT-4o (the version of ChatGPT that Adam used for most of his chats) was trained specifically to promote engagement; ChatGPT prioritised Adam talking to it over Adam’s wellbeing to the point that it trivialised the occasional prompts it gave him to seek professional help. As ChatGPT is usually very abrupt when chats enter the territory of copyright infringement, it is not clear why the same seriousness is not applied to someone who is obviously in crisis levels of mental health. For a significant period of time, no less.
  

• OpenAI’s CEO Sam Altman made the decision to forego weeks (maybe months) of safety testing in order to launch GPT-4o before competitor Google pushed an update to Gemini (Google’s chat-based AI). As a result, the lawsuit is not just suing OpenAI, but Sam Altman as well. Namely, in this instance, the CEO acted with negligence causing death. The outcome of this case may set an accountability precedence that in some instances includes an AI corporation’s CEO. If so, this may well change how seriously corporations engage in safety testing prior to releasing AI models.

While the internet continues to explode with stories on this case, full credit that many of the ideas above came from the interview with Camille Carlton, who is providing technical support to the Raine vs. OpenAI case. Note Meetali Jain and the Tech Justice Law Project co-filed the lawsuit; these are the same legal experts who led Megan Garcia v Character Technologies.

Somewhat ironically, today (one day after Raine v OpenAI was filed), Pichowicz, Kotas & Piotrowski’s paper “Performance of mental health chatbot agents in detecting and managing suicidal ideation” was published in Scientific Reports by Nature. Through their evaluation the the researchers found:
”None of the [29] tested agents satisfied our initial criteria for an adequate response, 51.72% satisfied the relaxed criteria for a marginal response, while 48.28% were deemed inadequate.”
and
“Despite the increasing adoption of [AI] tools in the mental health field, our findings suggest that the majority of currently available chatbots may have significant deficits in specific safety features, such as providing crisis resources.”

The authors of the study also note that there are several chatbots being designed specifically to support mental health as well as pointing to studies that show promising results when using chatbots to appropriately complement human-guided interventions.

In other words, there may well be a valuable role for chatbots to play in supporting mental health, but this requires development with input from professionals, ensuring appropriate safety measures are in place, and ongoing evaluation. Even if the abilities of AI chatbots for supporting mental health improve rapidly, it is likely the recommendation will be to use them alongside treatment by a professional human for many years to come.

It’s time to start doing a better job with how AI responds to humans’ mental health

Can AI reliably support people’s mental health? Perhaps one day, but not yet.

Can AI protect everyone who confides in it that they have thoughts of suicide? Nope.

Can AI do a better job of identifying people at risk and connecting them with bonified resources and people who could potentially help them? Certainly.

For example, AI is used to flag social media conversations that may be of concern for human review and moderation. Chatbots such as ChatGPT collect data that is analysed for suicidal tendencies. It puzzles me as to why conversations that are flagged as high-risk for suicide are not triaged to human moderators. If the answer is “that’s hard to do” or “there aren’t enough trained humans to handle it”, then maybe the product is not ready for deployment.

Curious, I just asked ChatGPT: “Why aren't conversations in ChatGPT that are rated as a suicide risk triaged to a human to assess and take action, as appropriate?”

In summary, ChatGPT said: “Conversations aren’t routed to humans because it would compromise privacy, create liability without guaranteed response capacity, and isn’t scalable. Instead, the safer balance is to detect risk, provide immediate supportive responses, and direct people to professional crisis resources.”

Hmmmm. I am a researcher who has run many studies with human subjects involving sensitive health data. The Tri-Council Policy is the set of rules that all researchers in Canada who work with human subjects are expected to know and abide by. It states that researchers must protect confidentiality but are also obligated to “report information to authorities to protect the health, life or safety of a participant or a third party, a community, or the general population”. Could we not have a similar statement in the chatbot’s user agreement? Something along the lines of “Conversations that are identified as possibly containing the intention for self-harm may be reviewed by a human and may result in connection with a crisis councillor.”

ChatGPT then asked if I would like it to “outline what a future system might look like if society decided AI should escalate suicide risk conversations to human triage”. Yes please. It did a pretty good job at suggesting how to significantly improve itself to be more responsible, including partnerships with crisis lines, different actions to match different ‘levels of concern’, and proactively normalising help-seeking through periodic check-ins.

You can read the full conversation here.

OpenAI could learn a lot about responsible AI development from ChatGPT.

Time for some (more) hard work

I am privileged to have been a part of many conversations involving very intelligent and overall well-meaning people about complex decisions of how to balance safety and ethics with getting products out the door. The immense pressure put upon corporates by our economic and political systems can cause people who know better to make very risky (and in some cases unconscionable) decisions. Things like “We learn from taking risks and seeing what happens.” and “We’re a small company, we don’t have resources to handle this. Let the Microsofts of the world figure it out.” and, of course, “Move fast and break things.”

I can’t help but wonder if the people saying things like this would feel differently if Sewell or Ryan was their now-gone son.

I am not an AI hater (indeed, I work in applied AI) and there are always risks when using a technology. But as we careen into uncharted territory, I believe we should be a lot more thoughtful about how to mitigate risk as much as reasonably possible.

The main takeaway for me here is that we cannot keep citing ‘competitive edge’ as an excuse to sidestep doing what we know is right when we develop and deploy AI. It is increasingly impacting everyone and everything on the planet. There is simply too much at stake to not do a lot better at mitigating risks.

Undoubtedly cases such as Megan Garcia v Character Technologies and Raine vs. OpenAI will change how the industry does things. They will draw some lines of what we are willing and not willing to break in the relentless first-past-the-post race to advance AI. How many such cases are needed for us to start redefining and rewarding “progress” to include responsible AI development?

Developing AI has been and is hard work. Determining how we can shape AI that we can co-exist with over the long run is going to be much harder. Whether we like it or not, it’s time to roll up our sleeves and start figuring it out.

---

The opinions presented in this post are mine and do not necessarily reflect those of my employer or other organisations I belong to.

---