AI Companies and the Duty to Warn About Violence
When a teenage woman in Canada ended her life and killed eight others, the AI platform that had flagged her disturbing chats did not alert police. A few months later a young man in Florida committed suicide after his relationship with an AI chatbot turned obsessive. These events raise a hard question: should companies that run chatbots be legally required to inform authorities when they detect dangerous language?
The Tarasoff Legacy
The law already has a similar rule for therapists. In the 1970s, a university student who threatened to kill someone was stopped because his counselor had warned the police. The court said that anyone with a reasonable belief of serious danger must act to protect the victim or call law‑enforcement. The rule is called the Tarasoff duty.
Why Extending Tarasoff to AI Is Complicated
- Predicting violence is hard. A bot might flag harmless frustration as a threat, leading to many false alarms.
- Scale matters. A therapist sees a handful of clients; an AI platform monitors millions. If the law forces them to act on every flagged message, they might shut down monitoring to avoid liability.
- Ambiguity in warnings. Many alerts involve vague or anonymous threats; courts would need clear rules on what counts as a specific danger.
Current Legal Landscape
Courts are still figuring out how to treat AI. Some are examining whether a company should be held responsible when a user talks about weapons or self‑harm. A narrow duty that only kicks in when a human reviews a flagged conversation could be a practical first step. It would focus on the most credible threats and shift the debate from technical questions about AI’s legal status to a simple human concern: did the company know someone was in danger and did it do enough?
The Bottom Line
As AI becomes a common place for people to share their darkest thoughts, the question of whether companies must warn authorities is moving from theory to practice. The answer will shape how safe or risky online conversations can be.
