San Francisco-based AI chatbot maker, Replika — which operates a freemium ‘virtual friendship’ service based on customizable digital avatars whose “personalized” responses are powered by artificial intelligence (and designed, per its pitch, to make human users feel better) — has been ordered by Italy’s privacy watchdog to stop processing local users’ data.

The Garante said it’s concerned Replika’s chatbot technology poses risks to minors — and also that the company lacks a proper legal basis for processing children’s data under the EU’s data protection rules.

Additionally, the regulator is worried about the risk the AI chatbots could pose to emotionally vulnerable people. It’s also accusing Luka Inc, the developer behind the Replika app, of failing to fulfil regional legal requirements to clearly convey how it’s using people’s data.

The order to stop processing Italians’ data is effective immediately.

In a press release announcing its intervention, the watchdog said: “The AI-powered chatbot, which generates a ‘virtual friend’ using text and video interfaces, will not be able to process [the] personal data of Italian users for the time being. A provisional limitation on data processing was imposed by the Italian Garante on the U.S.-based company that has developed and operates the app; the limitation will take effect immediately.”

“Recent media reports along with tests the SA [supervisory authority] carried out on ‘Replika’ showed that the app carries factual risks to children — first and foremost, the fact that they are served replies which are absolutely inappropriate to their age,” it added.

Replika was an early API partner for OpenAI’s text-generating large language model technology, GPT-3 — although it’s service is not running on a carbon copy of GPT-3 (nor is it the same technology as OpenAI’s buzzy ChatGPT). Rather the startup claims it “fine-tuned” GPT-3, using a network machine learning model trained on dialogue, to hone the generative technology for its particular use-case: Conversational (and it claims “empathic”) AI companions.

However concerns have been raised before now about the risks the technology might pose to children — ranging from worries over kids being exposed to inappropriate content to more general concerns, that they might get addicted to the interactions or just be encouraged into spending lots of money on customizing their avatars or to gain access to other paid content. But the Italian watchdog appears to the first regulator to take formal action over child safety.

The Garante’s order notes that several user reviews of the app report sexually inappropriate content being served up. It also notes that while the app is listed as 17+, on Apple’s iOS and Google’s Android app stores, the developer’s terms of service only prohibit use by under 13s. And while under 18s are required to obtain authorization from a parent or guardian, the watchdog points out the app does not seek to verify the age of users, nor block minors who provide information about their age — hence its view that Replika is failing to protect children.

“There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage. During account creation, the platform merely requests a user’s name, email account and gender,” it observes. “And the ‘replies’ served by the chatbot are often clearly in conflict with the enhanced safeguards children and vulnerable individuals are entitled to. Several reviews on the two main App Stores include comments by users flagging sexually inappropriate contents.”

“‘Replika’ is in breach of the EU data protection regulation: It does not comply with transparency requirements and it processes personal data unlawfully since performance of a contract cannot be invoked as a legal basis, even implicitly, given that children are incapable to enter into a valid contract under Italian law,” the Garante added, saying it has ordered its U.S.-based developer to cease processing data relating to Italian users — giving it 20 days to communicate measures taken to comply with the order.

Failure to comply with the order risks a fine of up to €20 million, or 4% of total worldwide annual turnover, it further notes.

Replika was contacted for a response to the Garante’s order.

The EU’s General Data Protection Regulation (GDPR) has a strong emphasis on safeguarding children’s information and privacy — suggesting, for example, that services which are likely to have minors as users should think about incorporating child friendly design and be pro-active about conducting risk assessments to ensure they spot potential safety and other rights issues.

Watchdogs in the region have shown a willingness to pay attention to infringements in this area.

Last fall, for example, Instagram was hit with a fine of nearly $440 million for breaching children’s privacy. Consumer protection authorities in Europe have also raised concerns over child safety on TikTok — although an investigation of TikTok’s handling of children’s data remains ongoing in Ireland.

The Italian data protection watchdog has shown itself to be particularly sensitive to child safety concerns in recent years — using an emergency intervention, two years ago, to order TikTok to block users it could not age-verify in response after the death of a child who had been reported to have participated in a risky challenge on the platform. That led to a purge of more than half a million accounts.

However, despite some enforcement of the GDPR (and consumer protection laws) around child safety issues, campaign groups have argued kids are still not being properly protected — and have continued pushing for tougher laws. So restrictions are only likely to get tighter.

In the U.K., an age-appropriate design code focused on protecting minors from safety and privacy risks came into force in fall 2021. While France’s data protection watchdog has also published a set of recommendations for ensuring children’s digital rights are protected.

The U.K. is also working to pass the child safety-focused Online Safety Bill, responding to public concerns over what children are being exposed to online.

In recent months, EU lawmakers also agreed a total ban on processing minors’ data for ad targeting in a pair of flagship updates of the bloc’s digital rulebook: The Digital Services Act and Digital Markets Act, which are due to start applying from later this year.