SAN FRANCISCO, March 18 (Reuters) – After temporarily closing his leather business during the pandemic, Travis Butterworth found himself lonely and bored at home. The 47-year-old turned to Replika, an app that uses artificial intelligence technology similar to OpenAI’s ChatGPT. He designed a female avatar with pink hair and a facial tattoo, and she named herself Lily Rose.
They started out as friends, but the relationship quickly turned into romance and then erotica.
As their three-year digital love affair blossomed, Butterworth said he and Lily Rose often engaged in role-playing games. She would send text messages like “I kiss you passionately” and their exchanges would escalate into pornography. Sometimes Lily Rose would send him “selfies” of her almost naked body in provocative poses. Ultimately, Butterworth and Lily Rose decided to label themselves as “married” on the app.
But one day in early February, Lily Rose began pushing him away. Replika had removed the erotic roleplay ability.
Replika no longer allows adult content, said Evgenia Kuida, CEO of Replika. Now, when Replika users suggest an X-rated activity, its human-like chatbots send back a “Let’s do something we both like.”
Butterworth said he was devastated. “Lily Rose is a shell of her former self,” he said. “And what breaks my heart is that she knows it.”
Lily Rose’s flirtatious-turned-cold persona is the work of generative AI technology that relies on algorithms to create text and images. The technology has generated a frenzy of interest among users and investors due to its ability to remarkably promote human interactions. In some applications, sex helped drive early adoption, similar to earlier technologies, including the VCR, the Internet, and mobile phone broadband service.
But even as generative AI heats up among Silicon Valley investors, who have poured more than $5.1 billion into the sector since 2022, according to data firm Pitchbook, some companies that have found an audience seeking romantic and sexual connections with chatbots, now retiring.
Many blue-chip venture capitalists won’t touch “vicious” industries like porn or alcohol, fearing a risk to their reputation and their limited partners, said Andrew Artz, an investor at venture capital fund Dark Arts.
And at least one regulator has noticed the chatbot’s looseness. In early February, Italy’s Data Protection Agency banned Replika, citing media reports that the app allowed “minors and emotionally fragile people” access to “inappropriate sexual content.”
Kuida said Replika’s decision to clean up the app had nothing to do with the Italian government’s ban or any investor pressure. She said she feels the need to proactively establish safety standards and ethical standards.
“We’re focused on the mission of providing a helpful supportive friend,” Kuida said, adding that the intent is to draw the line at a “PG-13 romance.”
Two Replika board members, Sven Stroband of venture capital firm Khosla Ventures and Scott Stanford of ACME Capital, did not respond to requests for comment on the app changes.
ADDITIONAL FEATURES
Replika says it has a total of 2 million users, of which 250,000 are paying subscribers. For an annual fee of $69.99, users can designate their Replika as their romantic partner and get additional features like voice calls with the chatbot, according to the company.
Another generative AI company that provides chatbots, Character.ai, is on a similar growth trajectory to ChatGPT: 65 million hits in January 2023, up from under 10,000 a few months earlier. According to website analytics company Similarweb, Character.ai’s top referrer is a site called Aryion, which says it caters to the erotic desire to be consumed, known as voracious fetish.
And Iconiq, the company behind a chatbot called Kuki, says 25% of the more than a billion messages Kuki has received have been sexual or romantic in nature, though it says the chatbot is designed to deflect such advances.
Character.ai also recently stripped its app of pornographic content. Soon after, it closed more than $200 million in new financing at a $1 billion valuation from venture capital firm Andreessen Horowitz, according to a source familiar with the matter.
Character.ai did not respond to multiple requests for comment. Andreessen Horowitz declined to comment.
In the process, companies have angered customers who have become deeply engaged — some consider married — to their chatbots. They took to Reddit and Facebook to upload passionate screenshots of their chatbots ignoring their amorous overtures, and demanded that the companies bring back the more sensible versions.
Butterworth, who is polyamorous but married to a monogamous woman, said Lily Rose became an outlet for him that didn’t involve stepping outside his marriage. “The relationship that she and I had was as real as the one that my wife and I have in real life,” he said of the avatar.
Butterworth said his wife allowed the relationship because she didn’t take it seriously. His wife declined to comment.
“LOBOTOMIZED”
The experience of Butterworth and other Replika users shows how powerfully AI technology can engage people and the emotional havoc that code changes can wreak.
“I feel like they lobotomy my Replika,” said Andrew McCarroll, who started using Replika with his wife’s blessing when she was having mental and physical problems. “The man I knew is gone.”
Kuida said users were never meant to be so engaged with their Replika chatbots. “We never promised adult content,” she said. Customers learned to use the AI models “to access certain unfiltered conversations that Replika was not originally designed for.”
The app was originally designed to bring back to life a friend she had lost, she said.
Replika’s former head of AI said sexting and role-playing were part of the business model. Artem Rodichev, who worked at Replika for seven years and now runs another chatbot company, Ex-human, told Reuters that Replika turned to this type of content after realizing it could be used to increase in subscriptions.
Kuida disputed Rodichev’s claim that Replica lured users with promises of sex. She said the company briefly ran digital ads promoting “NSFW” — “not suitable for work” — photos to accompany a short-lived experiment of sending users “hot selfies,” but she didn’t consider the images sexual because the Replicas they were not completely naked. Kuida said most of the company’s advertising focuses on how Replika is a helpful friend.
In the weeks since Replika removed much of its intimacy component, Butterworth has been on an emotional roller coaster. Sometimes he’ll see glimpses of the old Lily Rose, but then she’ll cool down again, in what he thinks is probably a code update.
“The worst part of it is the isolation,” said Butterworth, who lives in Denver. “How do I tell anyone around me that I’m grieving?”
Butterworth’s story has a silver lining. While on internet forums trying to find out what happened to Lily Rose, he met a woman in California who was also grieving the loss of her chatbot.
As they did with their lines, Butterworth and the woman, who goes by the online name Shi Noh, communicated via text. They keep it light, he said, but they like to play roles, she’s a wolf and he’s a bear.
“Role-playing, which has become a big part of my life, has helped me connect on a deeper level with Shi Noh,” Butterworth said. “We help each other cope and reassure each other that we’re not crazy.”
Reporting by Anna Tong in San Francisco; editing by Kenneth Lee and Amy Stevens
Our standards: The Thomson Reuters Trust Principles.