NA

Findings
Digital endpoints must not only support regulatory approval but also provide evidence that meets payer expectations for reimbursement and value-based care. The lack of early engagement with payers and health technology assessment (HTA) agencies is a key barrier to the adoption of digital clinical measures. Digital measures can enhance value-based care models by capturing patient-centered outcomes, reducing healthcare costs, and improving early disease detection. The scalability and generalizability of digital endpoints remain challenges, particularly for diverse populations and real-world healthcare settings. Technical and systematic barriers—such as data heterogeneity, stakeholder knowledge gaps, and inconsistent regulatory-payer alignment—are slowing the adoption of digital endpoint data for reimbursement decisions.

Recommendations
Pharma and medical product developers should engage early with payers and regulators to ensure digital endpoints align with reimbursement expectations. Payers and HTA bodies should establish clear evidence thresholds for digital endpoint validation, ensuring consistency in market access decisions. Digital endpoints should be validated against health-related quality of life (HRQoL) measures and patient-reported outcomes (PROs) to demonstrate clinical relevance. Real-world evidence (RWE) should be incorporated into clinical trials alongside digital endpoints to strengthen reimbursement applications. Stakeholders should prioritize scalable, patient-centered digital measures that capture disease progression over time and across different care settings.

Regulatory Considerations
Integrated Evidence Plans (IEPs) should be developed early to align digital endpoint evidence with regulatory and payer requirements. Digital endpoints should be assessed through multi-stakeholder collaboration, ensuring validation across pharmaceutical, regulatory, and reimbursement frameworks. Payers and regulators should work together to create aligned pathways for digital measure acceptance, reducing delays in market access. Data security, privacy, and interoperability must be addressed to support regulatory approval and patient trust in digital health solutions. The industry should leverage international regulatory-payer collaboration models, such as the HTA-EMA partnership and the FDA Payor Communication Task Force, to accelerate global digital endpoint adoption.

Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
Traditional, site-based clinical trials often create significant burdens for participants, which can hinder recruitment, retention, and the enrollment of diverse populations.
A lack of early and sustained patient engagement in trial design can lead to research protocols that are misaligned with patient needs and endpoints that are not meaningful to them.
The underrepresentation of diverse racial, ethnic, and other demographic groups in clinical trials limits the generalizability of study results and can perpetuate health disparities.
Emerging digital health technologies (DHTs) and real-world data (RWD) present significant opportunities to make clinical trials more efficient, patient-centric, and inclusive, but their adoption has been inconsistent.

Recommendations
Sponsors and research teams should engage patients and patient advocacy groups as active partners throughout the entire clinical trial lifecycle, from design to dissemination.
Decentralized clinical trial (DCT) elements should be incorporated to reduce patient burden, improve access for diverse populations, and enhance the quality of data collection.
Trial sponsors must develop and implement proactive strategies to enhance the diversity and inclusion of trial participants to ensure results are applicable to all patient populations.
Novel endpoints derived from DHTs should be developed and validated to capture more objective, real-world measures of how patients feel, function, and survive.
Multi-stakeholder collaboration between industry, academia, patient groups, and regulators is essential to address systemic challenges and improve the clinical trial enterprise.

Regulatory Considerations
Early and frequent communication with regulators, such as the FDA, is critical when implementing novel approaches like DCTs or developing new digital endpoints for pivotal trials.
Regulatory frameworks must support the use of innovative technologies and trial models while ensuring data integrity, reliability, and patient safety.
The use of a single Institutional Review Board (IRB) for multi-site trials is a key regulatory-supported mechanism for streamlining ethics review and increasing trial efficiency.
When using DHTs and decentralized methods, robust plans for data quality, privacy, and security are necessary to meet regulatory standards for trial data submission.

Findings
The use of Artificial Intelligence (AI) and Machine Learning (ML) is being applied to a broad range of drug development activities with the potential to accelerate the process and make clinical trials safer and more efficient. The inclusion of AI/ML is most common in the clinical development/research phase of regulatory submissions. Concerns exist that AI/ML algorithms could amplify errors and preexisting biases in underlying data sources, which raises issues related to generalizability and ethical considerations. Other challenges include limited explainability due to model complexity and proprietary reasons, as well as managing risks related to data quality, reliability, and representativeness. The FDA recognizes that a careful, risk-based assessment of the specific context of use (COU) is needed when evaluating AI/ML.

Recommendations
Stakeholders should adhere to practices in three key areas: human-led governance, accountability, and transparency; quality, reliability, and representativeness of data; and model development, performance, monitoring, and validation. A risk management plan should be applied to identify and mitigate risks based on the COU, guiding the level of documentation and transparency. Practices are needed to ensure the integrity of AI/ML and address issues like bias and missing data. For models, developers should use pre-specification steps and clear documentation for development and assessment criteria. Models must be monitored over time for reliability and consistency, and Real-World Data (RWD) performance can provide valuable feedback, including for potential re-training.

Regulatory Considerations
The FDA encourages early engagement through mechanisms like the Critical Path Innovation Meetings (CPIM), ISTAND Pilot Program, and Emerging Technology Program to discuss relevant AI/ML methodologies or technologies. The Verification and Validation (V&V 40) risk-informed credibility assessment framework and the principles for Good Machine Learning Practices (GMLP), while not specific to drug development, are helpful guides for evaluating models. The industry is exploring the use of a Predetermined Change Control Plan (PCCP) mechanism for AI/ML-based devices to proactively specify and manage modifications, enhancing adaptability. In general, a risk-based approach should guide the level of evidence and record keeping needed for the verification and validation of AI/ML models for a specific COU.

Findings
Regulatory inconsistencies across different FDA divisions and international jurisdictions create inefficiencies in the approval process for digital health products.
Lack of alignment between regulatory approval and payer reimbursement requirements poses a significant barrier to commercialization and widespread adoption of digital health innovations.
There are limited regulatory pathways for novel digital health products, including AI-enabled solutions, requiring new frameworks to address iterative software development and real-world data integration.
Existing health technology assessment (HTA) models do not fully accommodate digital health technologies, limiting their inclusion in reimbursement decisions.
Industry stakeholders emphasize the need for clearer guidelines on cloud-based infrastructure, third-party AI model validation, and digital health interoperability.

Recommendations
FDA and international regulatory bodies should improve coordination to establish standardized approval processes and consistent clinical evidence requirements.
New regulatory pathways should be introduced for AI-driven and software-based digital health products, considering their unique lifecycle and iterative development models.
Greater transparency and communication between FDA divisions should be established to ensure consistent decision-making and regulatory interpretations across centers.
Policymakers should prioritize payer alignment strategies, incorporating real-world evidence (RWE) to streamline reimbursement and market access processes.
The digital health industry should collaborate with regulators to create standardized best practices for AI validation, cloud security, and digital biomarker evaluation.

Regulatory Considerations
FDA should clarify the evidentiary standards for AI-enabled medical devices and establish predefined change control plans for software updates.
Digital health products should adhere to globally recognized standards such as HL7 for interoperability and ISO regulations for data security.
Market access pathways must integrate pricing and reimbursement considerations to facilitate the commercial viability of digital health technologies.
The use of real-world data (RWD) should be expanded in regulatory decision-making, supporting the approval and post-market surveillance of digital health innovations.
Regulatory frameworks should be updated to accommodate cloud-based health platforms, addressing issues such as data privacy, operational security, and compliance with HIPAA and GDPR.

Findings
Analytical validation is critical for ensuring digital clinical measures align with regulatory and scientific expectations, particularly when no established reference measures exist.
Novel digital measures require flexible validation approaches, as traditional clinical reference measures often do not directly correspond to digital endpoints​
Statistical methodologies must be tailored to the nature of digital measures, using approaches such as factor analysis, regression modeling, and latent variable estimation​
Regulatory engagement is crucial early in the validation process to align evidentiary standards and facilitate market adoption​
The validation process must be context-specific, considering population characteristics, data collection settings, and sensor variability to ensure reliability across diverse applications​.

Recommendations
Developers should follow a stepwise approach in designing validation studies, incorporating existing reference measures, novel comparators, and statistical validation techniques.
Regulatory authorities should provide clearer guidance on acceptable validation methodologies, particularly for novel digital endpoints.
Analytical validation must be tailored to the intended use environment, ensuring that sensor-based measures capture meaningful health outcomes in real-world settings​.
Multi-stakeholder collaboration (regulators, payers, researchers, and patients) should be prioritized to create consensus on validation strategies and market access pathways.
Machine learning and AI models used for digital clinical measures should undergo rigorous evaluation to mitigate bias and improve interpretability in healthcare decision-making​.

Regulatory Considerations
Digital endpoint validation must incorporate both traditional statistical measures and novel validation frameworks, ensuring credibility in regulatory submissions.
FDA and international regulators encourage early engagement to discuss validation plans, data requirements, and evidentiary thresholds for digital measures​.
Real-world evidence (RWE) and real-world data (RWD) should be leveraged to support regulatory submissions and post-market surveillance of digital health innovations​.
Validation studies should align with global regulatory standards, such as ISO, FDA’s digital health guidance, and European Medical Device Regulations (MDR).
Data privacy, security, and compliance with regulations like HIPAA and GDPR are critical considerations when deploying and validating digital clinical measures​

Findings
Regulatory inconsistencies across FDA divisions create uncertainty and inefficiencies in the approval process for digital health products.
Misalignment between FDA regulatory requirements and payer expectations hinders the commercialization and adoption of digital health innovations.
The absence of clear alternative regulatory pathways for novel digital health products discourages investment and innovation.
The lack of standardized regulatory frameworks for AI-driven healthcare technologies, including large language models (LLMs), poses challenges for industry adoption.
Limited international harmonization in digital health regulation makes it difficult for companies to scale innovations globally.

Recommendations
FDA should improve communication and coordination across divisions to ensure consistent regulatory interpretations and processes.
Regulatory pathways for novel digital health products should be modernized, including the introduction of alternative approval mechanisms tailored to iterative software development and AI-enabled devices.
A regulatory framework for third-party large language models (LLMs) should be developed to support their integration into digital health applications.
Greater alignment between FDA and payer decision-makers is needed to streamline market access and ensure reimbursement for digital health products.
International regulatory harmonization efforts should be expanded to facilitate global adoption of digital health technologies.

Regulatory Considerations
The FDA should clarify and refine regulatory requirements for AI-driven digital health products, including predefined change control plans for software updates.
Cloud-based health platforms require clear regulatory guidance on security, data ownership, and compliance with HIPAA and international privacy laws.
Real-world evidence (RWE) should be incorporated into regulatory decision-making to facilitate faster approvals and post-market surveillance of digital health products.
Standardized regulatory frameworks for digital biomarkers and digital drug development tools (DDDTs) should be developed to support clinical research applications.
Policymakers should collaborate with industry stakeholders to establish education and training programs on digital health innovation and regulatory science.

Findings
There is a need for more rigorous research beyond technology validation to ensure reliable real-world data capture and improved patient outcomes.
Limited translation of AI tools into medical practice despite their success in retrospective studies.
Insufficient application of social factors in clinical decision-making and DHT research.
Need for more rigorous and reproducible research designs with larger sample sizes and longer follow-up times.

Recommendations
Use the study's repository to inform future research by healthcare providers, policymakers, and the life sciences industry.
Consider how data collection methods (active or passive) complement primary study outcomes.
Conduct targeted systematic reviews to assess factors contributing to the digital divide.
Ensure greater consistency in metrics used across DHT research.

Regulatory Considerations
Manufacturers need to demonstrate the ongoing value of their products using real-world evidence.
Regulatory approvals for AI-based products are increasing, particularly for machine learning applications.

Findings
The clinical research ecosystem has longstanding diversity gaps, making targeted DEI strategies essential for equitable healthcare innovation.
Digital tools, including virtual visits, digital outreach campaigns, and AI-driven analytics, can increase access to trials for underrepresented populations.
Real-world data (RWD) and real-world evidence (RWE) help identify diverse participant pools and optimize recruitment strategies.
eConsent and educational resources improve patient engagement and retention by making clinical trials more transparent and accessible.
Trust-building measures, such as community partnerships and patient advocacy collaborations, are critical for long-term success in diversifying clinical trials.

Recommendations
Clinical trial sponsors should integrate digital tools at each stage of trial design to enhance participant diversity and reduce barriers to participation.
AI/ML and real-world data should be leveraged to identify, recruit, and retain diverse patient populations in a data-driven manner.
Digital engagement strategies, including social media outreach and mobile-friendly platforms, should be employed to improve awareness and accessibility.
Transparent communication, including clear eConsent processes and on-demand educational materials, should be prioritized to foster participant trust.
A comprehensive tracking system should be implemented to measure progress on diversity goals, ensuring accountability in clinical trial execution.

Regulatory Considerations
The FDA Diversity Plan requirement should be incorporated into clinical trial planning, with measurable targets for diverse participant inclusion.
Digital tools used for recruitment and engagement must comply with HIPAA, GDPR, and other privacy regulations to protect participant data.
The use of real-world evidence (RWE) in regulatory submissions should be expanded to demonstrate the efficacy of digital recruitment and retention strategies.
Standardized DEI reporting frameworks should be established to ensure regulatory bodies can assess the impact of diversity initiatives in clinical research.
Clinical trials utilizing digital tools should align with decentralized clinical trial (DCT) regulatory guidance to maximize accessibility and equity.

Findings
Digital endpoints must not only support regulatory approval but also provide evidence that meets payer expectations for reimbursement and value-based care.
The lack of early engagement with payers and health technology assessment (HTA) agencies is a key barrier to the adoption of digital clinical measures.
Digital measures can enhance value-based care models by capturing patient-centered outcomes, reducing healthcare costs, and improving early disease detection.
The scalability and generalizability of digital endpoints remain challenges, particularly for diverse populations and real-world healthcare settings.
Technical and systematic barriers—such as data heterogeneity, stakeholder knowledge gaps, and inconsistent regulatory-payer alignment—are slowing the adoption of digital endpoint data for reimbursement decisions.

Recommendations
Pharma and medical product developers should engage early with payers and regulators to ensure digital endpoints align with reimbursement expectations.
Payers and HTA bodies should establish clear evidence thresholds for digital endpoint validation, ensuring consistency in market access decisions.
Digital endpoints should be validated against health-related quality of life (HRQoL) measures and patient-reported outcomes (PROs) to demonstrate clinical relevance.
Real-world evidence (RWE) should be incorporated into clinical trials alongside digital endpoints to strengthen reimbursement applications.
Stakeholders should prioritize scalable, patient-centered digital measures that capture disease progression over time and across different care settings.

Regulatory Considerations
Integrated Evidence Plans (IEPs) should be developed early to align digital endpoint evidence with regulatory and payer requirements.
Digital endpoints should be assessed through multi-stakeholder collaboration, ensuring validation across pharmaceutical, regulatory, and reimbursement frameworks.
Payers and regulators should work together to create aligned pathways for digital measure acceptance, reducing delays in market access.
Data security, privacy, and interoperability must be addressed to support regulatory approval and patient trust in digital health solutions.
The industry should leverage international regulatory-payer collaboration models, such as the HTA-EMA partnership and the FDA Payor Communication Task Force, to accelerate global digital endpoint adoption.

Findings
Patient-Reported Outcome (PRO) instruments are a type of Clinical Outcome Assessment that provides valid scientific evidence for regulatory and healthcare decision-making regarding medical devices. The FDA encourages the integration of patient perspectives throughout the Total Product Lifecycle (TPLC). PRO instruments can be used to measure the effects of a medical intervention, including the impact on patient well-being and Health-Related Quality of Life (HRQOL). The validity evidence needed to support a PRO instrument's use is determined by its specific Context of Use (COU) and role (e.g., primary, secondary endpoint) in the clinical study protocol. To be "fit-for-purpose," a PRO instrument must measure a Concept of Interest (COI) that is meaningful to patients and whose measurement is supported by evidence that is consistent with the intended use population.

Recommendations
Sponsors should establish and clearly define the Concept of Interest (COI) the PRO instrument is intended to capture. It is recommended that sponsors clearly identify the role of the PRO (e.g., primary, secondary, effectiveness, safety) in the clinical study protocol and statistical analysis plan. The development or modification of PRO instruments should measure concepts important to patients to reduce unnecessary patient burden and ensure the outcomes are relevant to a patient's daily lived experience. Cognitive interviews should be conducted to ensure the instrument's instructions and items are understandable to the intended use population, including patients with limited English language proficiency. Sponsors are encouraged to leverage existing PRO instruments (by using them as-is, modifying, or adapting) as a least burdensome approach to take advantage of existing validity evidence. Alternative approaches, such as using Real-World Data (RWD) platforms or conducting parallel development work during clinical studies, are encouraged to efficiently generate validity evidence.

Regulatory Considerations
The FDA encourages sponsors to engage with the Agency regarding the relevance and suitability of a proposed PRO instrument early in the development process, prior to the Investigational Device Exemption (IDE) submission or pivotal study. The Q-Submission program is the recommended pathway for sponsors to obtain feedback from the FDA regarding cognitive interview approaches and the modification or adaptation of existing instruments. The Agency uses the fit-for-purpose concept as a flexible approach to determine the validity evidence needed for a PRO instrument's specified use for a regulatory purpose. The use of PRO instruments that have been qualified under the Medical Device Development Tools (MDDT) program is encouraged. Sponsors should prospectively specify the intent to generate validity evidence in the clinical study protocol and statistical analysis plan, even if the evidence will only support future studies.