UK NICE updates its Evidence Standards Framework for data-driven digital health technologies

In August 2022, the National Institute for Health and Care Excellence (“NICE”), the body responsible for conducting health technology assessments in the UK, updated its Evidence Standards Framework (“ESF”) for digital health technologies (“DHTs”) to include important guidance relating to evidential standards for artificial intelligence (“AI”) and data-driven technologies.

What is the ESF?

First published in March 2019, the ESF is a set of evidence standards that DHTs should ideally meet before being considered for commissioning or procurement by the National Health Service (“NHS”). It is designed to assist NHS evaluators to identify DHTs that are clinically effective and that offer value to the health and care system. The framework can also be used by developers to understand how the NHS evaluates these technologies to inform a view on commissioning or procurement decisions. Compliance with the ESF does not replace the formal Medical Technologies Evaluation Programme whereby NICE selects and evaluates medical technologies in order to produce recommendations based on the extent to which the adoption of the medical technology by the NHS would offer potential patient and healthcare system benefits.

The ESF includes 21 standards that are arranged across five areas of a DHT lifecycle:

• design factors;
• value description;
• performance characteristics;
• value delivery; and
• deployment consideration.

The overarching objective of the ESF is to accelerate the update of high-quality innovations that healthcare professionals can embed in clinical workflows and patient self-care toolkits.

How has the ESF been updated?

NICE was funded by the NHS AI Lab, which is part of NHS England’s Transformation Directorate, to update the ESF in a way that gives specific consideration to data-driven DHTs. The updated framework seeks to:

• include evidence requirements for AI and data-driven technologies with adaptive algorithms;
• align classification with regulatory requirements; and
• make the ESF easier to use.

The updated ESF recognises that data-driven DHTs with fixed or adaptive machine-learning algorithms might have increased risks not seen for other technologies. For example, information provided by DHTs can be used to assist in treating, diagnosing, triaging or identifying early signs of a disease or condition or guiding follow-up diagnostics or treatment interventions. Accordingly, the updates made to the ESF include aspects that are more relevant to these technologies.

Although the general principles set out in the extant ESF are generally applicable to define the performance characteristics of, and deployment pathways for, DHTs, the updates have highlighted the following considerations as particularly relevant to data-driven DHTs:

• How the design of the DHT could (a) provide a specific positive impact on addressing health inequalities in the UK health and social care system or improving access to care amongst hard-to-reach populations; (b) promote equality, eliminate unlawful discrimination and foster good relations between people with protected characteristics according to the Equalities Act 2010. In this regard, the updated ESF explains that DHT developers should describe any actions taken in the design of the DHT to mitigate against algorithmic bias that could lead to unequal impacts between different groups of service users or people.
• How the design of the DHT embeds good data practices to afford high-quality data-driven DHTs. Accordingly, the updated ESF explains that any datasets used to train, validate or develop the DHT should be of a high quality. A dataset will be considered of high quality if the developer can provide information such as: which datasets (title, source, version) were used for training and validating the DHT; why such datasets were collected, and by what means (manual input, through monitors or other devices); and the diversity (demographics, age, clinically relevant subgroups) in these datasets used and how this reflects the intended target population for the DHT. This design consideration should take account of the MHRA guiding principles on good machine-learning practice for medical device development.
• How to manage risks posed by the continual evolution and adaptation of AI and machine learning algorithims. In this regard, the updated ESF requires developers and evaluators to agree a plan for measuring usage and changes in the DHT’s performance over time. Such reporting may include: how regularly the algorithms are expected to retrain, the sources of retraining data, and the processes in place to detect any impacts of planned changes or environmental factors that may impact performance.

The bigger picture

Consistent with the UK Government’s Life Sciences Vision and NHS England’s Long Term Plan, digital transformation is critical to the long-term sustainability of health and social care. The goal is to ensure that the health system is equipped to support and foster innovation. The updated ESF could assist efficient uptake of these technologies within the NHS. Moreover, the COVID-19 pandemic has shown that having the right DHTs at the NHS’s disposal can be as critical as having the right therapies to prevent and treat infectious agents. Specifically, it is recognised that whether through underpinning the initial operational planning, clinical research into treatments, and the rapid, highly targeted NHS COVID-19 vaccine roll-out, DHTs have played an essential but largely hidden role in how the health and care service has responded to the biggest public health crisis in a century.