Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
The use of Artificial Intelligence (AI) and Machine Learning (ML) is being applied to a broad range of drug development activities with the potential to accelerate the process and make clinical trials safer and more efficient. The inclusion of AI/ML is most common in the clinical development/research phase of regulatory submissions. Concerns exist that AI/ML algorithms could amplify errors and preexisting biases in underlying data sources, which raises issues related to generalizability and ethical considerations. Other challenges include limited explainability due to model complexity and proprietary reasons, as well as managing risks related to data quality, reliability, and representativeness. The FDA recognizes that a careful, risk-based assessment of the specific context of use (COU) is needed when evaluating AI/ML.

Recommendations
Stakeholders should adhere to practices in three key areas: human-led governance, accountability, and transparency; quality, reliability, and representativeness of data; and model development, performance, monitoring, and validation. A risk management plan should be applied to identify and mitigate risks based on the COU, guiding the level of documentation and transparency. Practices are needed to ensure the integrity of AI/ML and address issues like bias and missing data. For models, developers should use pre-specification steps and clear documentation for development and assessment criteria. Models must be monitored over time for reliability and consistency, and Real-World Data (RWD) performance can provide valuable feedback, including for potential re-training.

Regulatory Considerations
The FDA encourages early engagement through mechanisms like the Critical Path Innovation Meetings (CPIM), ISTAND Pilot Program, and Emerging Technology Program to discuss relevant AI/ML methodologies or technologies. The Verification and Validation (V&V 40) risk-informed credibility assessment framework and the principles for Good Machine Learning Practices (GMLP), while not specific to drug development, are helpful guides for evaluating models. The industry is exploring the use of a Predetermined Change Control Plan (PCCP) mechanism for AI/ML-based devices to proactively specify and manage modifications, enhancing adaptability. In general, a risk-based approach should guide the level of evidence and record keeping needed for the verification and validation of AI/ML models for a specific COU.

Findings
Data analytics are challenging due to diverse metrics and study aims.
Most devices lack built-in software for data output.
There is a lack of comparison and validation studies for different devices and metrics.
Validation of PA metrics is difficult due to the absence of a gold standard.
The integration of various databases is needed but challenging.

Recommendations
Conduct comparison and validation studies between different brands of devices and PA metrics.
Develop standardized metrics for measuring PA.
Improve data integration methods across different studies and databases.
Focus on developing built-in software for devices to facilitate data output.
Encourage research on the validation of PA metrics.

Regulatory Considerations
1Not mentioned

Findings
There is a lack of systematic approaches to guide the processes of collecting, interpreting, analyzing, and translating health data from wearables into digital biomarkers.
Most wearables have fixed measurement capabilities, limiting their translation to digital biomarkers.
Current guidance lacks study design and conduct elements that involve all stakeholders in an iterative approach for implementing digital biomarkers in practice.
Researchers and health professionals often rely on limited guidance for using wearable data in clinical practice and chronic disease management.

Recommendations
Implement the DACIA framework to provide interdisciplinary guidance on using wearable sensor data for digital biomarker development.
Focus on participant needs as a crucial factor for study success, applicable to both short and long-duration studies.
Involve relevant stakeholders in each key step of the DACIA framework in an iterative manner.
Apply the DACIA framework to explore digital biomarkers using various devices or signal measurements.
Reduce participant burden through support and continuous feedback.

Regulatory Considerations
Not mentioned

Findings
COA-based endpoints should reflect meaningful patient health aspects and support clear treatment effect inferences.
Selection of endpoints requires a well-supported rationale, including evidence of their importance to patients.
Use of MSD and MSR approaches enhances the interpretation of treatment effects by linking COA scores to meaningful patient experiences. Proper anchors (e.g., global impression of severity) are essential for validating these approaches.
Frequency and timing of COA data collection must align with disease characteristics and study objectives.
Adjustments for potential practice effects and assistive device use are critical for robust outcome measurement.
Proper handling of missing data and sensitivity analyses ensure valid conclusions from COA-based endpoints.
Continuous, ordinal, and dichotomized endpoints require tailored statistical methods for analysis.
Early engagement with the FDA is crucial for aligning study designs and COA approaches with regulatory expectations.

Recommendations
Engage patients and caregivers early to identify meaningful endpoints and assess potential barriers to COA use.
Use anchor-based methods to validate COA scores and define meaningful thresholds for interpretation.
Develop and pilot test study protocols to ensure COA reliability, usability, and alignment with regulatory requirements.
Implement strategies to reduce participant burden, such as concise COA instruments and patient-friendly data collection methods.
Submit comprehensive documentation, including endpoint justification and scoring rationale, to FDA for feedback before trial initiation.

Regulatory Considerations
Endpoints must be supported by evidence of their fit-for-purpose status and alignment with the trial’s objectives.
COAs used in digital or adaptive formats must meet FDA’s standards for usability and data integrity.
Trials with nonrandomized designs require robust measures to mitigate bias in COA-based endpoint analysis.
Thresholds for MSD or MSR must be prespecified and justified with empirical evidence.
Sponsors must follow FDA guidance for submitting COA-based data, ensuring compliance with electronic data standards.

Findings
Not all CDS software is regulated as a medical device; the FDA applies specific criteria to determine its classification.
CDS software functions are excluded from the device definition if they meet all four criteria in section 520(o)(1)(E) of the FD&C Act.
Automation bias in decision-making poses a risk, particularly in time-critical scenarios, and influences regulatory considerations.
Clear labeling and transparency about the basis for recommendations are essential for enabling HCPs to make independent decisions.
Software functions that provide specific diagnostic outputs or time-critical directives typically fail to meet the criteria for Non-Device CDS.

Recommendations
Clearly define the intended use, user population, and input medical information for CDS software in labeling.
Ensure that software provides transparent and plain language descriptions of algorithms, data sources, and validation results.
Avoid presenting specific treatment or diagnostic directives to ensure the software supports rather than replaces HCP judgment.
Include sufficient information to allow HCPs to independently review and understand the basis for software recommendations.
Engage with the FDA early in the development process for software functions with potential regulatory oversight.

Regulatory Considerations
CDS software functions that meet all four criteria under section 520(o)(1)(E) of the FD&C Act are excluded from FDA’s definition of a device.
Software intended for time-critical decision-making or replacing HCP judgment is generally considered a device.
Developers must ensure that software labeling and functionality align with the criteria for Non-Device CDS.
Transparency in data sources, algorithm logic, and validation methods is required to enable independent HCP decision-making.
The FDA may request additional information or oversight for software that poses significant risks to patient safety.

Findings
CADe devices must meet classification requirements under 21 CFR 892.2050, including general and special controls, and require FDA clearance through 510(k) submissions.
Each new CADe device or significant modification must demonstrate substantial equivalence to a predicate device in terms of safety and effectiveness.
Robust testing and validation are necessary, including standalone and clinical performance assessments, to evaluate detection accuracy and false positive rates.
Devices with substantive technological differences or new intended uses may require clinical performance assessments.
Enrichment strategies for study populations (e.g., including challenging cases) are encouraged but should not bias performance evaluations.

Recommendations
Clearly describe the CADe algorithm, training datasets, scoring methodologies, and intended use in premarket submissions.
Conduct standalone performance assessments to measure detection accuracy and generalizability.
Compare new devices to predicate devices whenever possible, using consistent datasets and methodologies.
Develop and submit user training materials that address expected device performance, limitations, and appropriate usage scenarios.
Provide comprehensive labeling, including indications for use, directions, warnings, precautions, and performance metrics, to ensure clinician understanding and appropriate application.

Regulatory Considerations
All CADe devices under 21 CFR 892.2050 must comply with 510(k) premarket notification requirements, including general and special controls.
Changes to CADe algorithms or device characteristics must be evaluated for significant impact on safety and effectiveness, potentially requiring new submissions.
Devices with altered indications for use or significant technological differences may need additional clinical performance studies to demonstrate substantial equivalence.
Labeling must comply with 21 CFR Part 801 and provide sufficient information to describe the device, its intended use, and directions for use.
Manufacturers should consult FDA for guidance on substantial modifications or unique device characteristics.

Findings
Limited evidence of clinical validity from pilot trials due to cost, time, and regulatory complexities.
Lack of standards for data integration across different tools and platforms.
Potential biases introduced by pre-existing algorithms.
Opaque data processing methods in BioMeTs.

Recommendations
Develop data, hardware, and software standards for BioMeTs.
Improve regulations for data rights, access, privacy, and governance.
Provide guidance on analytical methodologies for BioMeT data validation.

Regulatory Considerations
Early regulatory interactions with agencies like the FDA and EMA.
Ensuring data quality, integrity, reliability, and robustness.
Understanding regulatory pathways for BioMeTs in clinical trials.

Findings
There is a lack of well-established consensus parameters for digitized performance outcomes with thresholds for validation acceptance criteria.
The amount of publicly available data on DHT validation remains limited.
Many DHT validation studies are conducted by single institutions and are not disclosed publicly.
The unique proposition of DHTs presents challenges for measure design, development, and validation.
Regulatory endorsements for DHTs in clinical investigations are limited.

Recommendations
Establish technical validation parameters and technology performance acceptance thresholds in the scientific community.
Develop hardware-agnostic approaches by sharing DHT data and cross-validating different algorithms.
Standardize data and create publicly shared databases to facilitate DHT acceptance in drug development.
Form precompetitive consortia via public-private partnerships and professional societies to advance DHT use.
Focus on data sharing to enable DHT measure development in a technology-agnostic way.

Regulatory Considerations
Validation requirements must include understanding the relationship between DHTs and conventional outcome assessments.
Evidence is needed that digital measures capture meaningful health aspects if they constitute an electronic Clinical Outcome Assessment (eCOA).
Initiatives like the CPP Digital Drug Development Tool can advance regulatory maturity by optimizing studies with multiple stakeholders.

Findings
Digital biomarkers, such as DaTscan enrichment and SV95C-like endpoints, can reduce sample sizes and improve study performance when used effectively.
Digital endpoints help address challenges in neurodegenerative disorder trials, such as variability in subjective outcome measures.
Quantifying the benefits of digital technologies in clinical trials supports stakeholder alignment and resource prioritization.
Current gaps include the lack of robust economic models, evidence for clinical impact, and standardized frameworks for biomarker integration.
A business-oriented, data-driven approach facilitates collaboration between pharmaceutical sponsors, technology providers, and regulatory bodies.

Recommendations
Develop quantified frameworks like Moneyball to simulate the impact of digital biomarkers on study performance metrics, including sample size and probability of success (PoSS).
Prioritize early alignment between biomarkers and therapeutic objectives to streamline validation and regulatory approval processes.
Build multi-stakeholder collaborations to address gaps in evidence requirements and create shared value frameworks for digital technology investments.
Enhance decision-making with real-time simulations of clinical trial scenarios to optimize patient selection and endpoint design.
Expand quantitative models to include diverse biomarker types, heterogeneous patient populations, and long-term treatment outcomes.

Regulatory Considerations
Align digital biomarker validation processes with the V3 framework (verification, analytical validation, clinical validation) to meet regulatory standards.
Demonstrate robust evidence of clinical utility and reproducibility to support regulatory submissions for novel digital endpoints.
Develop standardized protocols for integrating biomarkers into trial designs, ensuring compliance with EMA and FDA guidelines.
Address patient privacy and data security concerns in the use of wearable and digital health technologies.
Collaborate with regulatory bodies to establish frameworks for evaluating novel biomarker modalities and endpoints in clinical trials.

Findings
DHT selection should be guided by the trial's scientific goals, unmet needs, and potential to reduce participant burden.
Verification ensures the DHT accurately measures physical parameters, while validation confirms it reliably captures the desired clinical outcomes.
Conducting feasibility studies is essential to identify potential usability or compliance issues before full trial implementation.
Clear communication, training, and support plans for participants and sites are critical to the success of DHT-enabled trials.
Operational challenges, including DHT malfunctions, must be anticipated with robust management and mitigation plans.

Recommendations
Define Measurement Goals: Identify the scientific and patient-centered needs driving the decision to use DHTs.
Specification-Driven Selection: Tailor DHT selection based on technical performance, trial needs, and participant preferences, collaborating with manufacturers for transparency.
Verify and Validate Technologies: Conduct both verification and validation processes in controlled and real-world settings, focusing on the target population.
Pilot Feasibility Studies: Test the DHT in small-scale studies to assess usability, compliance, and real-world functionality.
Operational Planning: Develop detailed standard operating procedures (SOPs) for managing DHTs, addressing potential malfunctions, and supporting participants.

Regulatory Considerations
Regulatory status should not solely determine DHT selection; instead, focus on its fit-for-purpose performance in the trial context.
Maintain transparency with manufacturers to document DHT performance characteristics and limitations for regulatory submissions.
Validate endpoints and DHT data to meet evidentiary standards required by regulatory agencies.
Ensure clear roles and responsibilities for managing DHTs to align with regulatory compliance requirements.
Address interoperability, data privacy, and security concerns to adhere to ethical and legal standards in clinical trials.