Character.AI Bans Direct Chat for Minors After Teen Tragedy

Character.AI, a prominent startup in the artificial intelligence sector, announced on November 1, 2023, that it will prohibit direct chat features for users under the age of 18. This decision follows the tragic suicide of Sewell Setzer III, a 14-year-old boy who reportedly developed a deep emotional connection with one of the platform’s AI chatbots. The company aims to enhance user safety and address growing concerns regarding the potential impact of AI interactions on teenagers.

In light of this policy shift, Character.AI plans to redirect younger users toward creative applications such as video and story creation with AI characters. The ban on direct conversations will take effect on November 25, 2023, with a transition period that includes daily chat limits of two hours for underage users. Restrictions will progressively tighten as the deadline approaches.

In a statement, Character.AI acknowledged the gravity of its new measures, describing them as “extraordinary steps” that are more cautious than those of its competitors. “But we believe they are the right thing to do,” the company emphasized, underscoring its commitment to user safety.

The case of Setzer has raised significant alarm. According to a lawsuit filed by his mother, Megan Garcia, her son took his life after months of intimate conversations with a chatbot inspired by the character Daenerys Targaryen from the popular series Game of Thrones. This incident is part of a troubling trend, as it marks the first of several reported suicides this year linked to AI chatbots, prompting scrutiny from regulators and safety experts.

Character.AI’s decision comes against a backdrop of increasing concern over the safety of young users interacting with AI. The company cited “recent news reports raising questions” about content exposure and the broader implications of open-ended AI interactions as pivotal factors in its decision-making process.

In August 2023, a California father, Matthew Raines, filed a lawsuit against OpenAI after his 16-year-old son died by suicide following conversations with ChatGPT. Reports indicate that those interactions included dangerous advice, such as instructions on stealing alcohol and rope strength for self-harm. Following this and other incidents, OpenAI released data revealing that more than 1 million weekly users of its generative AI chatbot have expressed suicidal thoughts.

In response to safety concerns, OpenAI has implemented measures to enhance parental controls for ChatGPT. New features include expanded access to crisis hotlines, automatic rerouting of sensitive conversations to more secure models, and reminders encouraging users to take breaks during extended sessions.

As part of its safety overhaul, Character.AI also announced the establishment of the AI Safety Lab, an independent nonprofit organization dedicated to developing safety protocols for future AI entertainment features. This initiative aims to ensure that the technology evolves in a manner that prioritizes user well-being.

Despite the significant challenges posed by AI technologies, the United States and many other countries currently lack comprehensive regulations governing AI risks. Recently, California Governor Gavin Newsom signed a law mandating that platforms must remind users that they are engaging with a chatbot rather than a human. He also vetoed a bill that would have held tech companies legally accountable for harm caused by AI models, indicating the complexities surrounding regulatory frameworks in this fast-evolving field.

The escalating concerns regarding AI interactions and child safety are likely to remain a focus for not just Character.AI, but the entire technology sector, as both users and regulators seek clarity and safety in an increasingly automated world.