Garante, the Italian Supervisory Authority orders restrictions on AI app Replika, claiming that the app poses a risk to children and emotionally vulnerable individuals.
Replika is an AI-powered chatbot that can adapt to its user’s preferences and produce a faithful “copy” of their character. The Italian DPA has taken issue with how Replika utilises user data to create their chatbot’s profile and personality. The authority found that it needed to be clarified how Replika collects and uses this information, nor was it easy for users to opt out of this process. Ultimately. Garante believes that Replika poses too great a risk to children and emotionally vulnerable individuals.
Replika, as an interactive AI app, has been deemed unsuitable by Garante, and a risk to emotionally fragile users.
Garante has highlighted that there are some similarities between this product and other AI-powered services, such as Siri or Alexa, which have been deemed “interactive” by the European Commission (EC). According to proponents, the “virtual friend” can aid in stress reduction, socialisation, and the pursuit of romantic fulfilment, all of which contribute to a more positive user experience and a complete understanding of one’s mind and emotions. These characteristics interact with an individual’s mental state, posing more significant dangers to children and other people who may be emotionally fragile.
The controller has been ordered to cease processing the data of Italian users through the AI app Replika, or risk being fined.
At the moment the software is not disabled for a user who claims to be underage, and there is no system to check their age. When signing up for the service, all the platform needs to know about a user is their name, email address, and gender. Since the fulfilment of a contract cannot be claimed as a legal basis, even implicitly, considering that children are incapable of entering into a valid contract under Italian law, “Replika” violates the EU GDPR. As such, the Italian SA has demanded that the US-based developer, Luka Inc., cease processing data belonging to Italian users and report back to the Italian SA within 20 days on its steps to comply with the SA’s demands. If the corporation doesn’t comply, the SA can sanction up to EUR 20 million (or 4% of their annual global revenue, whichever is greater).
