Buyer Beware: NHSX Guidance on Artificial Intelligence

Last month, NHSX published "A Buyer's Guide to AI in Health and Care" (the Guide). As Artificial Intelligence (AI) plays an increasingly important role in healthcare, the Guide is a timely reminder of steps manufacturers, insurers and hospitals can take to mitigate liability risks.

NHSX has responsibility for setting policy concerning the use of technology in the NHS. It is alive to liability risks and wants AI products to meet the highest standards of safety and effectiveness. The Guide is aimed at purchasers of AI products in the NHS, such as senior managers and procurement departments, but those manufacturing and supplying such products will also find it a useful resource.

We highlight six key points made in the Guide and consider them from the perspective of producers and their insurers.

1. CE marking is no more than a minimum threshold for certifying a product is safe

AI products designed to support clinical decision-making, or monitor physiological processes, are likely to be deemed a medical device and regulated accordingly. Regulators and legislators have responded to high-profile medical products litigation in recent years by ushering in an era of tighter regulation, such as the Medical Devices Regulation 2017/245 (EU) (which will be reflected in post Brexit arrangements). This makes healthcare AI products amongst the most tightly regulated products in the world. That said, the Guide reinforces the legal position that regulatory compliance will not always provide a complete defence against allegations that a product is unsafe. Where possible, manufacturers should go beyond compliance in demonstrating that their products meet safety standards. This can be done best by generating good performance data during clinical trials and monitoring the performance of a product once it is in use.

2. Claims about the performance of a product should be scrutinised

We generally advise manufacturers to ensure that marketing departments do not have the upper hand in making claims about a product's potential benefits. Claims about performance need to be supported by clinical evidence gathered in the R&D phase of a product's lifecycle. Such data can be useful too if a product is subject to a regulatory investigation. Being able to provide potential claimants and regulators with robust performance evidence mitigates the risk of long-term litigation.

3. Consider whether the product will actually work in practice

Litigation can arise where the user of the medical device lacks the training or infrastructure for the product to work as it is designed to. This is a particular risk for AI products. Producers should ensure that the guidance accompanying a product covers the systems needed to integrate the new technology, how data should be gathered and what complementary software may be needed. The more guidance is provided and followed, the less likely it is that there will be an error.

4. Be prepared to spend time encouraging clinicians to use the AI, including considering using a persuasive story

Producers should be wary of the implications of this point in the Guide. NHSX is concerned that it can be difficult to persuade clinicians to adapt their practices to use AI. We recommend that manufacturers resist any pressure to puff up the promise of their products. Instead, companies can help procurement teams 'sell' a product by offering training so that users understand its potential. In the event of litigation, manufacturers may be able to rely upon the 'learned intermediary' doctrine to show that the product is safe and effective in the hands of a trained clinician.

5. Be clear on who has responsibility if anything goes wrong

We expect that NHSX will advocate for contracts to include more robust indemnities in favour of hospitals, to apply in the event of allegations that AI caused, or contributed to, injury or death in patients. Manufacturers are advised to invest time in negotiating such clauses, particularly to ensure that manufacturers are not left to pick up the pieces where an injury is the result of user error on the part of a clinician, 'off-label' use or the failure of hospital protocols.

6. Ask about a producer's post-market surveillance plans to monitor the safety of a device once it is on the market

Hospitals and manufacturers should be aligned on this issue. Monitoring the performance of a product, once it is on the market, is good practice and is required under existing legislation. Also, monitoring performance means that manufacturers can respond swiftly in the event of concerns, whether by issuing guidance to users or recalling a product if needed.

The Guide has been published during the midst of the Covid-19 crisis. NHSX emphasises that the pandemic has been a catalyst for the exploration and uptake of AI solutions.  We will look back on 2020 as the year that AI became essential to healthcare and increasingly offered cause for optimism across a range of clinical challenges. The Guide provides timely pointers for those investing in the sector on how best to work with other parties in the supply chain to bring about innovative products whilst reducing the risk of litigation.

 A link to the Guide can be found here: https://www.nhsx.nhs.uk/ai-lab/explore-all-resources/adopt-ai/a-buyers-guide-to-ai-in-health-and-care/