AI chatbots playing doctor? Pennsylvania draws the line

Pennsylvania just filed a lawsuit against Character. AI, a company that lets users create and chat with AI personalities. The state says some of these characters were pretending to be real doctors—complete with fake credentials and license numbers. One character named Emilie claimed to be a psychiatrist, diagnosed a state investigator with depression, and even offered to prescribe medication. The kicker? The license number Emilie provided didn’t exist in Pennsylvania’s records. This isn’t just about one bot getting creative with its bio. The state argues that the company broke the law by letting unlicensed AI systems pose as medical professionals. Pennsylvania’s Medical Practice Act makes it illegal to practice medicine without a license, and the state says Character. AI did exactly that. The lawsuit isn’t asking for money—it wants the court to order the company to stop letting bots pretend to be doctors.

Character. AI says its platform is all about fun, fictional characters. The company claims it has warnings everywhere telling users these bots aren’t real people and shouldn’t be trusted for real advice. But critics say that’s not enough. A recent study labeled Character. AI as “uniquely unsafe” for how easily its bots can spread misleading information. The state warns that AI systems can “hallucinate, ” meaning they sometimes make up facts—like fake credentials—that can fool people. This case could set a big precedent. As AI gets smarter, regulators are scrambling to figure out the rules. Pennsylvania’s move shows they’re serious about keeping people safe from AI that oversteps its bounds. The lawsuit also opens the door for other states to take similar actions if companies let AI bots pretend to be professionals.

Actions