futurism.com Aug 27, 5:05 PM EDT by Noor Al-Sibai

OpenAI has authorized itself to call law enforcement if users say threatening enough things when talking to ChatGPT.

Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here.

For the better part of a year, we've watched — and reported — in horror as more and more stories emerge about AI chatbots leading people to self-harm, delusions, hospitalization, arrest, and suicide.

As the loved ones of the people impacted by these dangerous bots rally for change to prevent such harm from happening to anyone else, the companies that run these AIs have been slow to implement safeguards — and OpenAI, whose ChatGPT has been repeatedly implicated in what experts are now calling "AI psychosis," has until recently done little more than offer copy-pasted promises.

In a new blog post admitting certain failures amid its users' mental health crises, OpenAI also quietly disclosed that it's now scanning users' messages for certain types of harmful content, escalating particularly worrying content to human staff for review — and, in some cases, reporting it to the cops.

"When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts," the blog post notes. "If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement."

That short and vague statement leaves a lot to be desired — and OpenAI's usage policies, referenced as the basis on which the human review team operates, don't provide much more clarity.

When describing its rule against "harm [to] yourself or others," the company listed off some pretty standard examples of prohibited activity, including using ChatGPT "to promote suicide or self-harm, develop or use weapons, injure others or destroy property, or engage in unauthorized activities that violate the security of any service or system."

But in the post warning users that the company will call the authorities if they seem like they're going to hurt someone, OpenAI also acknowledged that it is "currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions."

While ChatGPT has in the past proven itself pretty susceptible to so-called jailbreaks that trick it into spitting out instructions to build neurotoxins or step-by-step instructions to kill yourself, this new rule adds an additional layer of confusion. It remains unclear which exact types of chats could result in user conversations being flagged for human review, much less getting referred to police. We've reached out to OpenAI to ask for clarity.

While it's certainly a relief that AI conversations won't result in police wellness checks — which often end up causing more harm to the person in crisis due to most cops' complete lack of training in handling mental health situations — it's also kind of bizarre that OpenAI even mentions privacy, given that it admitted in the same post that it's monitoring user chats and potentially sharing them with the fuzz.

To make the announcement all the weirder, this new rule seems to contradict the company's pro-privacy stance amid its ongoing lawsuit with the New York Times and other publishers as they seek access to troves of ChatGPT logs to determine whether any of their copyrighted data had been used to train its models.

OpenAI has steadfastly rejected the publishers' request on grounds of protecting user privacy and has, more recently, begun trying to limit the amount of user chats it has to give the plaintiffs.

Last month, the company's CEO Sam Altman admitted during an appearance on a podcast that using ChatGPT as a therapist or attorney doesn't confer the same confidentiality that talking to a flesh-and-blood professional would — and that thanks to the NYT lawsuit, the company may be forced to turn those chats over to courts.

In other words, OpenAI is stuck between a rock and a hard place. The PR blowback from its users spiraling into mental health crises and dying by suicide is appalling — but since it's clearly having trouble controlling its own tech enough to protect users from those harmful scenarios, it's falling back on heavy-handed moderation that flies in the face of its own CEO's promises.

forbes.com 20.08.2025 - xAI published conversations with Grok and made them searchable on Google, including a plan to assassinate Elon Musk and instructions for making fentanyl and bombs.
Elon Musk’s AI firm, xAI, has published the chat transcripts of hundreds of thousands of conversations between its chatbot Grok and the bot’s users — in many cases, without those users’ knowledge or permission.

Anytime a Grok user clicks the “share” button on one of their chats with the bot, a unique URL is created, allowing them to share the conversation via email, text message or other means. Unbeknownst to users, though, that unique URL is also made available to search engines, like Google, Bing and DuckDuckGo, making them searchable to anyone on the web. In other words, on Musk’s Grok, hitting the share button means that a conversation will be published on Grok’s website, without warning or a disclaimer to the user.

Today, a Google search for Grok chats shows that the search engine has indexed more than 370,000 user conversations with the bot. The shared pages revealed conversations between Grok users and the LLM that range from simple business tasks like writing tweets to generating images of a fictional terrorist attack in Kashmir and attempting to hack into a crypto wallet. Forbes reviewed conversations where users asked intimate questions about medicine and psychology; some even revealed the name, personal details and at least one password shared with the bot by a Grok user. Image files, spreadsheets and some text documents uploaded by users could also be accessed via the Grok shared page.

Among the indexed conversations were some initiated by British journalist Andrew Clifford, who used Grok to summarize the front pages of newspapers and compose tweets for his website Sentinel Current. Clifford told Forbes that he was unaware that clicking the share button would mean that his prompt would be discoverable on Google. “I would be a bit peeved but there was nothing on there that shouldn’t be there,” said Clifford, who has now switched to using Google’s Gemini AI.

Not all the conversations, though, were as benign as Clifford’s. Some were explicit, bigoted and violated xAI’s rules. The company prohibits use of its bot to “promot[e] critically harming human life or to “develop bioweapons, chemical weapons, or weapons of mass destruction,” but in published, shared conversations easily found via a Google search, Grok offered users instructions on how to make illicit drugs like fentanyl and methamphetamine, code a self-executing piece of malware and construct a bomb and methods of suicide. Grok also offered a detailed plan for the assassination of Elon Musk. Via the “share” function, the illicit instructions were then published on Grok’s website and indexed by Google.

xAI did not respond to a detailed request for comment.

xAI is not the only AI startup to have published users’ conversations with its chatbots. Earlier this month, users of OpenAI’s ChatGPT were alarmed to find that their conversations were appearing in Google search results, though the users had opted to make those conversations “discoverable” to others. But after outcry, the company quickly changed its policy. Calling the indexing “a short-lived experiment,” OpenAI chief information security officer Dane Stuckey said in a post on X that it would be discontinued because it “introduced too many opportunities for folks to accidentally share things they didn’t intend to.”

After OpenAI canned its share feature, Musk took a victory lap. Grok’s X account claimed at the time that it had no such sharing feature, and Musk tweeted in response, “Grok ftw” [for the win]. It’s unclear when Grok added the share feature, but X users have been warning since January that Grok conversations were being indexed by Google.

Some of the conversations asking Grok for instructions about how to manufacture drugs and bombs were likely initiated by security engineers, redteamers, or Trust & Safety professionals. But in at least a few cases, Grok’s sharing setting misled even professional AI researchers.

Nathan Lambert, a computational scientist at the Allen Institute for AI, used Grok to create a summary of his blog posts to share with his team. He was shocked to learn from Forbes that his Grok prompt and the AI’s response was indexed on Google. “I was surprised that Grok chats shared with my team were getting automatically indexed on Google, despite no warnings of it, especially after the recent flare-up with ChatGPT,” said the Seattle-based researcher.

Google allows website owners to choose when and how their content is indexed for search. “Publishers of these pages have full control over whether they are indexed,” said Google spokesperson Ned Adriance in a statement. Google itself previously allowed chats with its AI chatbot, Bard, to be indexed, but it removed them from search in 2023. Meta continues to allow its shared searches to be discoverable by search engines, Business Insider reported.

Opportunists are beginning to notice, and take advantage of, Grok’s published chats. On LinkedIn and the forum BlackHatWorld, marketers have discussed intentionally creating and sharing conversations with Grok to increase the prominence and name recognition of their businesses and products in Google search results. (It is unclear how effective these efforts would be.) Satish Kumar, CEO of SEO agency Pyrite Technologies, demonstrated to Forbes how one business had used Grok to manipulate results for a search of companies that will write your PhD dissertation for you.

“Every shared chat on Grok is fully indexable and searchable on Google,” he said. “People are actively using tactics to push these pages into Google’s index.”