"Recommended for you" - Data risks in AI-powered customer solutions

What is happening?

Retailers and consumer brands are increasingly using AI online and in stores. Examples of how the technology could be leveraged include generative AI like ChatGPT, the automation of supply chains or the detection of fraud. Data regulators in the UK and EU are starting to catch up to the use of these technologies and making their positions known.

Why does it matter?

Regulators in the UK and in Europe have confirmed that data protection laws apply to all technologies that use personal data, including any AI systems. Data protection authorities (including the UK ICO) are focussing heavily on the issues raised by AI. This underlines the potential harm that could be caused to individuals due to the misuse or misguided use of AI.

For retailers and consumer brands using these technologies, a key obligation to remember is that set out in the UK and EU GDPR around automated decision making (ADM). The law restricts organisations from using solely automated processing to make a decision that has a legal or similarly significant effect on an individual, unless certain exceptions apply. What this means in practice is that individuals have a right to avoid important decisions about them being made purely by a machine with no human involvement.

As may be expected, this can significantly impact the development of a compliant AI technology. Retailers will need to assess both the level of human involvement in any decisions that are made, and the impact that these decisions could have on the individual. Aside from ADM, other data principles such as fairness, transparency and data minimisation all impact on the lawfulness of AI technologies.

In light of the boom in AI technologies in the market, regulators in the UK and EU have started to prioritise their governance of this area. In the EU, the focus has been on legislation in the form of the EU AI Act. The EU Parliament approved its version of the Act’s text in June 2023, and an agreed version among European institutions is expected by the end of the year. The act is based on a structured four-tier risk framework, and an EU AI Board will be established to provide formal guidance and assistance on the topic.

As the UK is no longer subject to the EU GDPR and EU guidance following Brexit, it is developing its own data protection laws that diverge in certain key areas from those in the EU. A stated focus for these revised laws has been reducing the impact of regulatory red tape on business and innovation, and the UK’s approach to AI is no exception. The UK Government published an AI White Paper in March 2023, which was then subject to consultation with many believing the UK’s approach may be too soft. The UK Government has since indicated that there may be more regulation than initially suggested, but that this will not reach the level of that in the EU.

Retailers that operate in the EU and the UK are likely to face a two-tier compliance regime as a result, as even a UK-headquartered retailer will likely be caught by the EU regime if it offers and sells goods to customers in the EU.

What action should you consider?

While the exact scope of the new AI regulations in the EU and UK is yet to become clear, there are steps retailers and consumer brands can take now to ensure that their use of AI is more likely to comply with data protection laws. Regulators in both the EU and UK consider this to be a risky area, particularly where the technology also involves the processing of biometric data.

Retailers and consumer brands should first take a step back to assess their own risk appetite for the use of AI, which will help to inform a strategy going forward. Other important initial considerations include making sure that any technology used is understandable and explainable to the end user. At the design phase, consider how a meaningful human review of any decisions could be built into the technology. Staff should also have the training and the authority to escalate and override decisions made by the AI system where these involve the processing of personal data.

A good starting point is the ICO’s guidance on AI, found here.

To hear more on these issues, sign-up to RPC's biggest retail and consumer event of the year, Retail Compass Live! on 1 November 2023 at our London office where our leading voices will explore these themes and much more besides. 

Click here to access the full Retail Compass Autumn 2023 edition