Findings
The rapid advancement of sensor technology and connectivity has enabled high-frequency, longitudinal monitoring of physiological processes, yet the infrastructure for large-scale deployment remains resource-intensive. Current challenges include a lack of standardized terminology for digital decision-making tools and significant variability in environmental factors that affect sensor performance. Proprietary algorithms and device-specific barriers often hinder the verification and validation processes necessary for regulatory approval. Additionally, there is a distinct gap between granular digital features and their clinical relevance or meaningfulness to patients. Ethical concerns are emerging around data management, patient anxiety in psychiatric contexts, and the responsibility for addressing adverse events detected by remote monitoring.

Recommendations
Stakeholders should develop consensus-driven frameworks for standardized device performance reporting and environmental testing to streamline evaluations for specific contexts of use. The community should adopt a modular approach to data standards that bins requirements by concept of interest and disease-specific needs. Collaborative efforts between patients and developers are essential to bridge the gap between technical metrics and meaningful aspects of health. It is recommended to implement ""bring-your-own-device"" (BYOD) frameworks that ensure data reliability while supporting the inevitable evolution of technology during long-term studies. Researchers and clinicians must be trained in the ethical, legal, and social implications of digital health technology use, particularly regarding data privacy and the management of remote-detected safety signals.

Regulatory Considerations
Digital health technologies used to collect endpoints must meet high evidentiary requirements for validation, with complexity increasing when multiple sensors or complex software are bundled. Regulatory agencies like the FDA and EMA have established pathways for the qualification of drug development tools, including biomarkers and clinical outcome assessments. Integration of new draft guidance on remote health monitoring with existing regulatory workflows is necessary to reduce uncertainty in trial evaluations. While many digital health technologies do not qualify as medical devices unless they have a specific medical purpose, synergies between device risk assessments and drug trial data integrity frameworks should be explored. Early engagement with regulators remains a critical step for obtaining feedback on novel digital endpoints and ensuring the suitability of evidentiary support.

Findings
The traditional process of evaluating biomarkers within the context of a single drug development program is inefficient and creates uncertainty for sponsors. This case-by-case approach leads to redundant efforts, slows down the development of novel therapies, and hinders the broad adoption of promising scientific tools. There is a clear need for a centralized, collaborative pathway to formally validate biomarkers, which can de-risk drug development, encourage innovation, and make the process more predictable and cost-effective for all stakeholders.

Recommendations
Drug developers, academic researchers, and other stakeholders should proactively engage with the FDA through the formal Biomarker Qualification Program to validate biomarkers for specific contexts of use. It is recommended to form public-private partnerships and other collaborations to pool resources and data, which strengthens the evidence package for a biomarker's utility. Developers should use the qualification process to establish a biomarker's value early, making it a publicly available and reliable tool that can accelerate the development of multiple drug products.

Regulatory Considerations
The Biomarker Qualification Program provides a distinct regulatory pathway for establishing a biomarker's validity for a specific Context of Use (COU), separate from an individual Investigational New Drug (IND) or New Drug Application (NDA). The process involves a three-stage submission and review cycle: the Letter of Intent, the Qualification Plan, and the Full Qualification Package. Once qualified, a biomarker is publicly listed and can be incorporated into multiple drug development programs without the need for sponsors to re-submit and re-justify the validation data for that specific COU, streamlining subsequent regulatory reviews.

Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
Establishing cross-sector consortia does not guarantee success without a unified objective and stakeholder buy-in. A neutral, independent facilitator is a key element for successful governance in many collaborative platforms. Many consortia lack consistent methods for storing critical data, meeting minutes, and regulatory briefing packages, which creates barriers after project completion. Regulatory success depends heavily on the early development of a strategy that defines the necessary evidence to validate innovative methodologies. Successful examples include the qualification of biomarkers for polycystic kidney disease and type 1 diabetes, as well as imaging measures for Alzheimer’s disease.

Recommendations
Consortium members should develop an initial regulatory strategy during the project scoping and planning phases. Teams must explicitly define the context of use for any proposed tool to articulate exactly what decisions the output will inform. A robust data strategy should be implemented early, including formal agreements for data use, standardization, and sharing that remain in place in perpetuity. Consortia must prioritize sustainability plans to ensure data and active databases remain available for research and regulatory use after funding expires. Projects should integrate regulatory science expertise from the start to cover both EU and US frameworks.

Regulatory Considerations
Regulators require individual patient-level data that is fully curated, standardized, and presented through formal submissions like qualification applications. Formal regulatory endorsement ensures a tool can be trusted for consistent interpretation in drug development and marketing authorization evaluations. Early engagement with agencies such as the FDA and EMA is essential to gain feedback on novel methodologies and align study designs with regulatory expectations. Specific pathways like the EMA Qualification of Novel Methodologies and the FDA Qualification Process for Drug Development Tools should be utilized. Regulatory qualification may require ongoing access to databases to support the long-term use of the methodology.

Key Principles (Findings)
The central principle of the DDT Qualification Programs is to create a formal pathway for the FDA to conclude that a specific tool is well-suited for a particular Context of Use (COU) in drug development. A key finding, as reflected in the program's design, is that qualification de-risks drug development by allowing a tool to be used in any regulatory submission for its qualified COU without needing to be re-validated each time. The program is designed to foster stakeholder collaboration, encouraging the development of tools that can benefit the entire research community, thereby reducing the burden on individual sponsors.

Program Activities (Recommendations)
The structure of the DDT programs serves as a series of recommendations for tool developers:
Engage Early and Collaboratively: The programs are designed to provide a framework for early and ongoing scientific collaboration with the FDA to facilitate the development of new tools.
Follow a Staged Process: Developers are guided through a multi-stage process, typically involving a Letter of Intent, a Qualification Plan, and a Full Qualification Package, to systematically build the evidence needed for qualification.
Seek Public Qualification: The ultimate recommendation is to achieve public qualification for a DDT, which makes the tool available for broad use and integrates it into the regulatory review process, expediting future drug development.

Regulatory Considerations
The DDT Qualification Programs are a formal regulatory framework established under the 21st Century Cures Act. A "qualified" DDT has a specific regulatory status; it can be relied upon to have a specific interpretation and application in drug development and regulatory review for its stated Context of Use (COU). This qualification is publicly available and allows the tool to be included in Investigational New Drug (IND), New Drug Application (NDA), or Biologics License Application (BLA) submissions without the FDA needing to reconsider its suitability. This creates a more efficient and predictable regulatory compliance pathway for sponsors who use the qualified tool.

Findings
Long and Unpredictable Timelines: The COA Qualification Program is lengthy and unpredictable, with an average qualification time of six years. Nearly half of all submissions experience review times that exceed the FDA's own published targets.
Low Qualification and Uptake: As of October 2024, only seven COAs (8.1% of those listed) have been qualified, and only three of those have been used to support the benefit-risk assessment of new medicines. No COAs submitted after the passage of the 21st Century Cures Act in 2016 have been qualified.
Limited Regulatory Impact: Qualified COAs are consistently designated for "exploratory use" and have never been accepted as a primary endpoint in a clinical trial. In contrast, some non-qualified COAs have been used as key endpoints and included in drug labels, questioning the utility of the formal qualification pathway.
Discrepancy Between FDA Centers: There is a notable difference in how COAs are qualified between the drug (CDER/CBER) and device (CDRH) centers. The Kansas City Cardiomyopathy Questionnaire (KCCQ) was qualified by CDRH for use as a primary or secondary endpoint, while for drugs, it was only qualified as an "exploratory" measure.

Recommendations
Increase Transparency of Timelines: The FDA should publish its actual, historical review timelines for COA qualification so that drug developers can better plan and integrate these tools into their development programs.
Clarify the Use of Qualified COAs: The FDA should clearly articulate how and when qualified COAs can be used as primary or secondary endpoints to support regulatory decision-making and provide a clear pathway for updating a COA's status from "exploratory" to a key endpoint.
Publish Best Practices: Both sponsors and the FDA should be encouraged to publish their experiences with the qualification program to share best practices and learnings with the broader drug development community.
Create a List of Accepted Endpoints: The FDA should create and maintain a public list of qualified COAs that can be used as surrogate endpoints to support drug approval decisions, thereby increasing their utility and adoption.

Regulatory Considerations
"Qualified as a Measure" Ambiguity: The FDA's practice of qualifying COAs as "measures" for "exploratory use" creates regulatory uncertainty for sponsors, as it implies that significant additional evidence is still needed before the tool can be relied upon for a key endpoint.
Qualification is Not Required: The analysis shows that COAs can be accepted for regulatory decision-making and included in drug labels without going through the formal qualification program, suggesting that qualification is not a prerequisite for use as a reliable endpoint.
Unclear Path to Endpoint Progression: The current DDT guidance does not specify the process for upgrading a COA's qualification status (e.g., from exploratory to a primary endpoint) after additional data has been generated, which hinders its evolution and broader use.

Findings
The database provides a transparent and accessible way for the public to track the progress of various Drug Development Tools (DDTs) through the FDA's qualification pipeline. This includes biomarkers, clinical outcome assessments, and animal models. The information available, such as submission status and supporting documentation, offers insight into the types of tools being developed and the evidence required for their qualification. The platform reveals that a wide range of tools are in development across numerous therapeutic areas, highlighting active areas of research and innovation in drug development.

Recommendations
Stakeholders in the drug development ecosystem are encouraged to utilize this database to inform their research and development strategies. By reviewing the status of existing DDT submissions, sponsors can identify opportunities for collaboration, avoid duplicative efforts, and better understand the evidentiary requirements for tool qualification. Prospective tool developers should use the database to learn from successful submissions and to align their own development plans with FDA expectations.

Regulatory Considerations
This database is a direct implementation of the transparency provisions of the 21st Century Cures Act. The public availability of this information is intended to foster trust and collaboration in the DDT qualification process. By providing a clear view of the regulatory journey of various tools, the FDA aims to standardize the qualification process and encourage the development and use of novel, validated tools in drug development. Users of the database should be aware that the information reflects the status of a DDT at a particular point in time and that the qualification process is an iterative one.

Findings
The use of Artificial Intelligence (AI) and Machine Learning (ML) is being applied to a broad range of drug development activities with the potential to accelerate the process and make clinical trials safer and more efficient. The inclusion of AI/ML is most common in the clinical development/research phase of regulatory submissions. Concerns exist that AI/ML algorithms could amplify errors and preexisting biases in underlying data sources, which raises issues related to generalizability and ethical considerations. Other challenges include limited explainability due to model complexity and proprietary reasons, as well as managing risks related to data quality, reliability, and representativeness. The FDA recognizes that a careful, risk-based assessment of the specific context of use (COU) is needed when evaluating AI/ML.

Recommendations
Stakeholders should adhere to practices in three key areas: human-led governance, accountability, and transparency; quality, reliability, and representativeness of data; and model development, performance, monitoring, and validation. A risk management plan should be applied to identify and mitigate risks based on the COU, guiding the level of documentation and transparency. Practices are needed to ensure the integrity of AI/ML and address issues like bias and missing data. For models, developers should use pre-specification steps and clear documentation for development and assessment criteria. Models must be monitored over time for reliability and consistency, and Real-World Data (RWD) performance can provide valuable feedback, including for potential re-training.

Regulatory Considerations
The FDA encourages early engagement through mechanisms like the Critical Path Innovation Meetings (CPIM), ISTAND Pilot Program, and Emerging Technology Program to discuss relevant AI/ML methodologies or technologies. The Verification and Validation (V&V 40) risk-informed credibility assessment framework and the principles for Good Machine Learning Practices (GMLP), while not specific to drug development, are helpful guides for evaluating models. The industry is exploring the use of a Predetermined Change Control Plan (PCCP) mechanism for AI/ML-based devices to proactively specify and manage modifications, enhancing adaptability. In general, a risk-based approach should guide the level of evidence and record keeping needed for the verification and validation of AI/ML models for a specific COU.

Findings
Only a small number of novel digital measures have matured into regulatory qualification or efficacy endpoints.
Demonstrating that digital measures are meaningful to patients is a key challenge.
There is resistance from sponsors due to uncertainty about the value of DHT-derived endpoints in regulatory discussions.
Patient experiences are highly heterogeneous, making it difficult to generalize meaningful aspects of health.
Challenges exist in defining clinical significance and classifying digital measures as COAs vs biomarkers.

Recommendations
Engage patients and caregivers in facilitated discussions to incorporate their voices.
Determine the best method for gathering patient input on a case-by-case basis.
Engage patients to inform evidence needs, implementation, and value delivery.
Return summarized health data to participants to motivate and encourage communication with clinicians.

Regulatory Considerations
Understand the FDA's recent guidance on patient engagement in drug development.
Recognize the shift in evidence rigor required by the FDA for demonstrating meaningfulness.
Provide evidence that DHTs are usable, acceptable, and clinically relevant.
Utilize early engagement channels like CPIM and pre-LOI programs offered by the FDA.

Findings
Evaluating patient outcomes on a case-by-case basis within individual drug programs is an inefficient use of resources and creates regulatory unpredictability. This approach frequently leads to redundant efforts to validate the same assessment tools across different development programs. The lack of a standardized, transparent process for accepting Clinical Outcome Assessments (COAs) hinders the development and use of novel, patient-centric endpoints, ultimately slowing the delivery of therapies that address outcomes that matter most to patients.

Recommendations
Developers of COAs, including patient groups, academic researchers, and pharmaceutical sponsors, are encouraged to collaborate with the FDA through the qualification program. This engagement should occur early to ensure that the measures are developed with sufficient rigor to meet regulatory standards. Stakeholders should leverage the program to validate a wide range of COAs, particularly Patient-Reported Outcomes (PROs), making them publicly available to advance patient-focused drug development across the entire industry and reduce redundant validation work.

Regulatory Considerations
The COA Qualification Program offers a formal regulatory pathway for the FDA to review and accept a COA for a specific Context of Use (COU). This qualification is separate from the review of an individual drug application, making the validated tool accessible for any sponsor to use in their clinical trials without re-adjudicating the COA's fitness for that purpose. Qualification requires a comprehensive submission demonstrating the measure is well-defined and reliable, ensuring that it appropriately captures the patient's experience or functional status.

Findings
Limited number of drugs approved using DHT data for labeling claims.
Lack of clarity on definitions and regulatory pathways for DHT-derived endpoints.
Challenges in global studies due to varying definitions among regulatory authorities.
Fine line between using DHT-derived measures for therapy response and quality of life assessments.

Recommendations
Create clear definitions for DHT-derived tools and measures.
Define specific evidentiary criteria for DHT-based tools.
Leverage precompetitive public-private partnerships to advance DHT development.
Utilize existing regulatory pathways like the iSTAND pilot program.

Regulatory Considerations
Need for harmonized global definitions and pathways for DHT-derived measures.
Use of existing programs like the iSTAND pilot program to integrate new digital measures.
Clear guidance from FDA and EMA for qualifying biomarkers or COAs in drug development.

Findings
Regulatory frameworks differ across regions, with the FDA focusing on digital health technologies and the EU emphasizing digital methodologies under MDR.
Determining whether a DHT qualifies as a medical device depends on its intended use and functionality, necessitating region-specific evaluations.
Health authority engagement can occur through FDA pathways like Critical Path Innovation Meetings (CPIM) and EMA’s Innovation Task Force (ITF).
Verification and validation of DHTs are crucial to ensure reliability and compliance with regulatory requirements in clinical trials.
Cybersecurity and compliance with privacy laws, such as GDPR, are mandatory considerations for DHT implementation.

Recommendations
Engage Regulators Early: Utilize FDA, EMA, or MHRA pathways (e.g., CPIM, ITF) during early development to align on requirements and mitigate risks.
Conduct thorough assessments to determine if a DHT qualifies as a medical device under regional regulations.
Implement robust validation and verification processes to confirm that DHTs are fit-for-purpose in clinical investigations.
Ensure compliance with GDPR, HIPAA, and other relevant data protection standards to safeguard patient information.
Adhere to GCP guidelines, including the ALCOA+ principles, to maintain data credibility and patient safety throughout the trial.

Regulatory Considerations
FDA Regulations: Evaluate DHTs under the FDA’s framework for medical devices, including exemptions under 21 CFR Part 812 and the Digital Health Software Precertification Program.
EU MDR/IVDR: Comply with MDR for medical devices and IVDR for in-vitro diagnostics, ensuring alignment with Annex VIII for software classification.
UK MHRA Guidance: Reference MHRA’s flowcharts for determining if a software qualifies as a medical device and ensure compliance with UK-specific regulatory requirements.
Global Harmonization Efforts: Consider global standards, such as ICH E6 (R2) and GHTF/IMDRF guidelines, to align multinational clinical trials.
Leverage pathways like EMA’s qualification process for novel methodologies and FDA’s DDT qualification program for broader acceptance of digital endpoints.