FDA Reveals AI Development Cheat Sheet in Highly Anticipated Draft Guidances

DRAFT AI-ENABLED DEVICE GUIDANCE

The draft AI-enabled device guidance is the latest in a series of FDA resources taking a total product life cycle approach to developing and monitoring AI-enabled devices. The draft AI-enabled device guidance offers recommendations for the information and data that sponsors should include in marketing submissions for devices with AI-enabled software functions and provides general strategies for the design and development of AI-enabled devices.

While FDA is clear that its other guidance documents on devices remain applicable to AI-enabled devices, the draft AI-enabled device guidance offers more details on the kinds of information that sponsors of AI-enabled devices should include about their products in marketing submissions.

Among other things, the draft AI-enabled device guidance recommends that marketing submissions for AI-enabled devices address the following:

• Device Description. The device description should include information about the model’s inputs and outputs and how AI is used to achieve the device’s intended use. The intended users and use environment, clinical workflow, and degree of automation compared to the current standard of care should also be described. The device description should explain whether the device requires regular calibration or includes configurable elements.
• User Interface and Labeling. Marketing submissions should allow FDA to understand what device elements users interact with and what information is communicated to users of the device throughout the device’s workflow. This includes any alerts or other communications to the user about the risks or limitations of the AI-enabled device, information about the AI model and its performance, and descriptions of the interface and operational sequence of the device.

The recommendations suggest that one of FDA’s major concerns about AI-enabled devices is that users will not understand the limitations of the device and its AI functionality. FDA appears to be emphasizing that transparency within its labeling framework is part of the solution. Per the guidance, all users should be provided with information about the device’s AI models, its proper use, and its limitations in a format and at a reading level appropriate to the given user. FDA encourages sponsors to use model cards to concisely communicate key elements of an AI-enabled device to users and includes an example of a completed model card in the guidance. We underscore that the model card included in the guidance is an example and any model card developed by a manufacturer should be customized to the relevant device and context for use.

• Risk Assessment. The risk management file included in a marketing submission for an AI-enabled device should include a risk assessment of the device that takes into consideration the unique risks posed by AI. FDA urges designers to consider user tasks and knowledge tasks that occur at all stages of the device use – from installation to interpretation of the results – as part of the risk assessment.
• Data Management. In the draft AI-enabled device guidance, FDA repeatedly emphasizes the importance of data quality to developing and validating AI-enabled devices. Reflecting this, marketing submissions for AI-enabled devices should detail the sponsor’s data management practices. The marketing submission should explain how data is cleaned, processed, annotated, stored, used, and controlled. The submission should also describe how data has been or will be collected and should characterize the data used to develop and validate the AI-enabled device.

FDA notes in the draft AI-enabled device guidance that data quality is a key performance driver for AI models and can help mitigate and reduce bias. To this end, FDA advises sponsors to provide information on the independence of test data from data used to develop an AI model, reference standards used in device development, and the representativeness of data.

• Model Description and Development. Marketing submissions should include information about the characteristics of each AI model used in the device – distinct from descriptions of the device as a whole. Descriptions of the models should be included in the software description section of the submission and should address the inputs and outputs, model architecture, model parameters, and how the model was trained. When a device involves multiple models, FDA encourages use of charts or other diagrams to help map the relationship between the inputs, operation, and outputs of each model.

FDA repeatedly notes its concern that it perceives opacity in many AI models and that this can be a source of risk in medical devices. In line with the emphasis on transparency throughout the guidance, FDA expects sufficient information on AI models to permit FDA to understand the functionality of an AI-enabled device and to identify potential limitations, sources of bias, and other considerations for appropriate labeling.

• Validation. Validation information should provide objective evidence of an AI model’s performance with respect to the intended use. Validation should be performed using a dataset independent from that used to train or develop the model. FDA notes that validation will look different depending on the device and the underlying models, but submissions should provide information about the studies performed to validate the device’s performance.
• Device Performance Monitoring. Recognizing that AI-enabled devices can experience changes in performance over time, FDA urges sponsors of AI-enabled devices to proactively monitor and address changes in device performance. Sponsors that choose to employ performance monitoring to control risk and provide reasonable assurance of device safety and effectiveness should include information about such performance monitoring plans in their market submissions.
• Cybersecurity. FDA urges sponsors to address the cybersecurity risks of AI in their submissions. In addition to pointing sponsors to other FDA guidance on cybersecurity, the FDA flags AI risks that can be exacerbated by cybersecurity threats, including data poisoning, data leakage, overfitting, model bias, and performance drift. Marketing submissions should address AI-specific cybersecurity concerns in the cybersecurity/interoperability section.

The draft AI-enabled device guidance also includes recommendations on the information that public submission summaries should cover. FDA again encourages sponsors to consider using a model card to organize and present key information about an AI-enabled device. Finally, appendices to the guidance provider sponsors with further recommendations on transparency, performance validation, and device usability that can be used to inform a total product life cycle approach to device design. The appendices also include helpful examples of a model card and 510(k) summary for an AI-enabled device.

DRAFT AI DRUG GUIDANCE

FDA also issued its first guidance document specifically addressing the use of AI in drug and biologic product development. This guidance was developed collaboratively by FDA’s human and animal medical product centers, the Office of Inspections and Investigations, the Oncology Center of Excellence, and the Office of Combination Products. The draft guidance reflects a years-long effort by FDA to develop its approach to AI in this area. FDA has previously issued discussion papers, sponsored workshops, and solicited public comments related to the use of AI in drug development.

The draft AI drug guidance describes a risk-based framework for sponsors to assess and establish the credibility of an AI model for a specific context of use when the model will be used to produce information to support regulatory decision-making regarding safety, effectiveness, or quality for drugs or biological products. Context of use refers to how an AI model is used to address a certain question of interest. The draft AI drug guidance does not apply to AI uses in drug discovery or operations not impacting patient safety, drug quality, or the reliability of results from clinical or nonclinical studies. Instead, the draft guidance’s scope includes predicting patient outcomes, integrating and processing large datasets to develop clinical trial endpoints, and helping select manufacturing conditions.

FDA highlights three challenges unique to AI that inform the draft AI drug guidance:

• The reliability of outputs produced by AI models depends on the quality, size, and representativeness of datasets used to train the model.
• The complexity and opacity of many AI models makes it difficult to determine how models arrive at a given conclusion and necessitates transparency in the development of a model.
• The performance of many AI models may shift or change over time or across deployment environments.

In light of these challenges, FDA considers establishing an AI model’s credibility for a given use to be essential. Doing so requires thoughtful consideration in the development, training, and deployment of an AI model. Throughout the model’s life cycle, the model’s risks, limitations, and performance should be monitored and documented.

The seven-step framework described in the draft drug AI guidance is similar to assessment frameworks developed for medical devices. The first two steps of the framework described in the draft AI drug guidance ask the AI user to first define the question, decision, or concern to be addressed by the AI model and to consider the context of use (i.e., the specific role and scope of the AI model in answering the question of interest).

The next steps are to assess the risks of the contemplated AI model and to develop a plan to assess the model’s credibility in the context of use. The guidance includes detailed recommendations on the contents of a credibility assessment plan, although FDA acknowledges that whether a plan must be submitted to FDA will depend on how the sponsor engages with FDA, the AI model, and context of use. A credibility assessment plan should include a description of the model, its training, and the sponsor’s rationale for selecting the specific modeling approach. The plan should then detail how the model will be evaluated and include information on how the data used to evaluate the model was collected, the metrics used to measure the model’s performance, limitations of the model, and the sponsor’s rationale for this assessment approach.

Finally, a sponsor should execute the plan, document the results from the credibility assessment, and determine the adequacy of the AI model for the context of use. If the results do not support the credibility of the model for the context of use, FDA suggests several options for how a sponsor might respond. For example, the sponsor could downgrade the model’s influence in answering the question of interest by incorporating additional types of evidence. The sponsor might also increase the rigor of the assessment and further train the model, establish additional protocols to mitigate model risk, or alter the modeling approach. Beyond the seven-step framework, the draft AI drug guidance emphasizes the importance of continued monitoring of AI model performance to address model drift. The draft AI drug guidance lists options for interfacing with FDA on AI model development and encourages early and transparent engagement on these issues by sponsors and other interested parties.

*  *  *

For more information on these draft guidance documents or other recent FDA guidance on AI, contact the authors of this article or any other member of McDermott’s Food, Drug & Medical Device Regulatory Group.