Hunt’s fears are not unfounded, Corynne McSherry, legal director for the digital rights group the Electronic Frontier Foundation, told Ars.
“The discovery order poses genuine risks to user privacy in itself and as a precedent for the many other lawsuits around the country,” McSherry said. “And it is emblematic of a broader problem: AI chatbots are opening another vector for corporate surveillance, especially if users don’t have meaningful control over what happens to their chat histories and records.”
According to Hunt, Wang failed to “consider exempting ‘Anonymous Chats,’ which are reasonably expected to contain the most sensitive and potentially damaging information of users, from retention and disclosure in this case,” claiming that it “constitutes an overly broad and unreasonable action.”
He urged the judge to revise the order to include this exemption, as well as exemptions for any chats “discussing medical, financial, legal, and personal topics that contain deeply private information of users and bear no relevance whatsoever” to the plaintiff news organizations’ claimed interests.
For Hunt and many other users blindsided by the order, the stakes appear high. He suggested that Wang should have allowed him to intervene “because this case involves important, novel constitutional questions about the privacy rights incident to artificial intelligence usage—a rapidly developing area of law—and the ability of a magistrate to institute a nationwide mass surveillance program by means of a discovery order in a civil case.”
But Wang disagreed with Hunt that she exceeded her authority in enforcing the order, emphasizing in a footnote that her order cannot be construed as enabling mass surveillance.
“Proposed Intervenor does not explain how a court’s document retention order that directs the preservation, segregation, and retention of certain privately held data by a private company for the limited purposes of litigation is, or could be, a ‘nationwide mass surveillance program,'” Wang wrote. “It is not. The judiciary is not a law enforcement agency.”