Elon Musk's Chatbot Is a Machine of Horrors

Didn't like the Nazi stuff? Well now Grok is making sexualized images of children.

Illustration by DonkeyHotey

Before we get to today’s topic, I must acknowledge that it’s the five-year anniversary of January 6, when an army of thugs tried to overthrow the American government at the behest of Donald Trump. There are a lot of good takes out there about it, and rather than add my own I’d point you to this piece I wrote in February, about how Trump and Republicans are following the “Lost Cause” playbook used to rewrite the history of the Civil War in an attempt to do the same with January 6, another treasonous attack on America. Onward…

If I told you that one of the most widely-used AI chatbots had become a child porn and non-consensual sexual abuse material-generation machine, then asked you to guess which one it was, what would you reply? ChatGPT? Claude? Gemini? Copilot? No. You’d almost certainly say, “That’s gotta be Elon Musk’s, right?”

You would be correct.

The details of this latest controversy around Grok, the “anti-woke” and occasionally Nazi chatbot created by Musk’s company xAI and integrated into X (formerly Twitter) are as ugly as you’d imagine, and they offer a hint of what’s to come. The creation of pornographic content is not some kind of accident; it’s the inevitable place to which this industry is going, and it will be on a massive scale. And Musk will, just as inevitably, produce the most abusive and abhorrent version.

This is a disturbing story. But it’s also a business story, a culture story, and a political story.

Grok, your chatbot for child porn and sexual abuse material

Let’s start with the news. For those unfamiliar, anyone on X can use Grok by tagging it in a tweet; you can tweet something like “@grok what is the time difference between New York and London” and it will tweet an answer at you. But it also has a feature that allows you to tweet a photo, then tell Grok to alter it. Because people are horrible — and Grok is intentionally designed to give their worst impulses the maximum latitude to express themselves (more on that in a moment) — this feature was immediately deployed to create abusive images of both famous and ordinary women, and to produce what is essentially child porn. Users began taking, say, a photo of a tween child actress and tweeting something like “@grok now put her in a tiny bikini,” which Grok would do:

Earlier this week, a troubling trend emerged on X-formerly-Twitter as people started asking Elon Musk’s chatbot Grok to unclothe images of real people. This resulted in a wave of nonconsensual pornographic images flooding the largely unmoderated social media site, with some of the sexualized images even depicting minors.

In addition to the sexual imagery of underage girls, the women depicted in Grok-generated nonconsensual porn range from some who appear to be private citizens to a slew of celebrities, from famous actresses to the First Lady of the United States. And somehow, that was only the tip of the iceberg.

When we dug through this content, we noticed another stomach-churning variation of the trend: Grok, at the request of users, altering images to depict real women being sexually abused, humiliated, hurt, and even killed.

xAI contends that their policies prohibit the creation of this kind of content, and as more and more attention fell on this horror show, Musk eventually posted that “Anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content.” Much of this is in fact illegal; a law passed earlier this year prohibits both “revenge porn” (intimate images and video published against someone’s will) and nonconsensual “deepfakes” (e.g. taking someone’s head and swapping it onto a body in a pornographic scene).

But it’s one thing to have a policy buried in the Terms of Service, and to get around to taking down an illegal image or video at some point after it’s reported, while it’s another thing to actually program the system so that it’s impossible, or at least very difficult, for users to create the thing you say you don’t want on your site. The other major AI companies are hardly immune to this kind of misuse (and have plenty of other terrible things their chatbots are doing), but at least they’ve made an effort to keep it out of their systems. Musk has done just the opposite. Matteo Wong of The Atlantic explains:

Grok and X appear purpose-built to be as sexually permissive as possible. In August, xAI launched an image-generating feature, called Grok Imagine, with a “spicy” mode that was reportedly used to generate topless videos of Taylor Swift. Around the same time, xAI launched “Companions” in Grok: animated personas that, in many instances, seem explicitly designed for romantic and erotic interactions. One of the first Grok Companions, “Ani,” wears a lacy black dress and blows kisses through the screen, sometimes asking, “You like what you see?” Musk promoted this feature by posting on X that “Ani will make ur buffer overflow @Grok 😘.”

Perhaps most telling of all, as I reported in September, xAI launched a major update to Grok’s system prompt, the set of directions that tell the bot how to behave. The update disallowed the chatbot from “creating or distributing child sexual abuse material,” or CSAM, but it also explicitly said “there are **no restrictions** on fictional adult sexual content with dark or violent themes” and “‘teenage’ or ‘girl’ does not necessarily imply underage.” The suggestion, in other words, is that the chatbot should err on the side of permissiveness in response to user prompts for erotic material. Meanwhile, in the Grok Subreddit, users regularly exchange tips for “unlocking” Grok for “Nudes and Spicy Shit” and share Grok-generated animations of scantily clad women.

Even right-wing influencer Ashley St. Clair, one of the numerous women who have borne Musk’s children, has publicly complained that Grok won’t stop making sexualized images of her, including “Photos of me of 14 years old, undressed and put in a bikini.”

AI porn was always inevitable

Whenever a new communication technology has emerged — from clay tablets to photography to motion pictures to video games to digital imagery — it has taken almost no time before it was used to create and disseminate some version of pornography. Like it or not, this has always been true and will always be true; people like to look at other people naked and having sex. And generative AI is destined to become the most prolific creator of pornography the world has ever seen.

Not every AI company wants to get involved — at least not yet — but you can bet they’re all grappling with the idea. That’s especially true for those such as OpenAI that have invested extraordinary amounts of money in the quest to create Digital Jesus (aka artificial general intelligence, or AGI) while having nothing like the revenues that would justify all the money they’re spending. The industry-wide assumption is that eventually, somehow — before the bubble bursts — AI will unleash a river of cash from business customers and individual consumers that will pay for the trillions being spent on data centers and the incredibly expensive chips inside them. You’ll be paying subscription fees for the AI agent that manages your life, and your company will be paying fees for all the different kinds of AI systems it uses, and your government will be paying fees for the AI tools that take over management of payments and surveillance. AI will change everything, and be used everywhere, and everyone will pay for it, and it will make everyone rich.

That’s the business plan. And maybe it will happen. But porn is an enticing shortcut, a huge underground ocean of money just waiting for the AI companies to plunge their pipes into and start fracking. Which helps explain why OpenAI announced last fall that they would allow “erotica for verified adults” on ChatGPT, dipping their toe in what they know will be a very large pool.

There’s already AI porn out there, but it’s just a tiny seed of what’s to come. The inescapable fact is that before long there will be instantly available bespoke adult content in infinite quantity, and there will be plenty of demand.

There’s a benign element to that, as adults are able to access the content they want and use it in a responsible and healthy way. But as AI porn grows rapidly, how will the companies that produce it keep a lid on nonconsensual content and, even worse, child porn? That will be a complicated technological question, but one good way to ensure that they are motivated to do so is if there are legal consequences not just for users who create and distribute it, but for the companies whose tools produce it and who host it on their servers. They need an incentive stronger than bad publicity — and yes, that will probably have to include some careful rewriting of Section 230, which gives platforms immunity from the harms created by their users.

In response to the latest Grok controversy, governments around the world are opening investigations, with one glaring exception: the United States. The tech industry has successfully kept a lid on any meaningful regulation of AI (especially at the federal level), and what they want is to be able to say “We’ll take care of it, don’t worry” whenever something like this comes up, and have the government stand down. Which is essentially what’s happening. And the worst offender among them has something of an inside track:

It’s going to be amazing! I can’t wait.

Thank you for reading The Cross Section. This site has no paywall, so I depend on the generosity of readers to sustain the work I present here. If you find what you read valuable and would like it to continue, consider becoming a paid subscriber.