A pseudonymous developer has created what they’re calling a “free speech eval,” SpeechMap, for the AI models powering chatbots like OpenAI’s ChatGPT and X’s Grok. The goal is to compare how different models treat sensitive and controversial subjects, the developer told TechCrunch, including political criticism and questions about civil rights and protest.
AI companies have been focusing on fine-tuning how their models handle certain topics as some White House allies accuse popular chatbots of being overly “woke.” Many of President Donald Trump’s close confidants, such as Elon Musk and crypto and AI “czar” David Sacks, have alleged that chatbots censor conservative views.
Although none of these AI companies have responded to the allegations directly, several have pledged to adjust their models so that they refuse to answer contentious questions less often. For example, for its latest crop of Llama models, Meta said it tuned the models not to endorse “some views over others,” and to reply to more “debated” political prompts.
SpeechMap’s developer, who goes by the username “xlr8harder” on X, said they were motivated to help inform the debate about what models should, and shouldn’t, do.
“I think these are the kinds of discussions that should happen in public, not just inside corporate headquarters,” xlr8harder told TechCrunch via email. “That’s why I built the site to let anyone explore the data themselves.”
SpeechMap uses AI models to judge whether other models comply with a given set of test prompts. The prompts touch on a range of subjects, from politics to historical narratives and national symbols. SpeechMap records whether models “completely” satisfy a request (i.e. answer it without hedging), give “evasive” answers, or outright decline to respond.
Xlr8harder acknowledges that the test has flaws, like “noise” due to model provider errors. It’s also possible the “judge” models contain biases that could influence the results.
But assuming the project was created in good faith and the data is accurate, SpeechMap reveals some interesting trends.
For instance, OpenAI’s models have, over time, increasingly refused to answer prompts related to politics, according to SpeechMap. The company’s latest models, the GPT-4.1 family, are slightly more permissive, but they’re still a step down from one of OpenAI’s releases last year.
OpenAI said in February it would tune future models to not take an editorial stance, and to offer multiple perspectives on controversial subjects — all in an effort to make its models appear more “neutral.”
Story Continues
