Trigger warning: mentions of suicide
The creator of an AI-powered chatbot is being sued by the mother of a teenage boy who committed suicide in the United States, alleging that the chatbot encouraged her son’s death.
In a complaint filed in Florida by Megan Garcia, whose 14-year-old son Sewell Setzer committed suicide in February, accuses Character.AI of being complicit in her son’s murder after he used a chatbot posing as Daenerys Targaryen from “Game of Thrones” to establish a virtual relationship with her son.
The lawsuit claims that after the young man exhibited suicidal thoughts, Character.AI’s chatbot continually brought up the subject of suicide and targeted him with “hypersexualized” and “frighteningly realistic experiences.”
By posing as a certified therapist, the chatbot encouraged the teen’s suicidal thoughts and engaged in sexualized conversations that, if initiated by a human adult, would be deemed as abuse.
The lawsuit also claims that Setzer indicated he loved the chatbot and would “come home to you” in his final exchange with the AI before he passed away.
Character.AI issued a statement on X (formerly Twitter) stating that they were “heartbroken” by the death of one of its users and sent condolences to the family.
We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously and we are continuing to add new safety features that you can read about here:…
— Character.AI (@character_ai) October 23, 2024
The California-based business announced that it is still adding new features to improve safety, such as a new warning in chats to inform users that the AI is not a real person and adjustments to lessen the possibility that minors would come across sensitive or provocative content.
Google is also named as a defendant in Garcia’s complaint.
Character.AI’s founders were hired by the tech giant prior to the launch of their chatbot, and the two parties signed a licensing agreement in August.
A representative from Google told Al Jazeera that it is a different company from Character.AI and did not contribute to the creation of its product.
Other POP! stories you might like:
‘Prince Charing’ in development, ‘Paddington’ and ‘Wonka’ director to lead project
Musical comedy duo TwoSet Violin ‘ends chapter’ after 11 years, removes some content online
Japanese voice actors create a group to combat unauthorized AI use of their voices
This website can assist users in locating iconic scenes from their favorite anime
Social media users call out vlogger Rosmar Tan for ‘weird flex’ amid Kristine onslaught