Findings
The quality of evidence for the analytical validation of sensor-based digital health technologies (sDHTs), which is the evaluation of algorithms converting sensor data into a clinically interpretable measure, is often inconsistent and insufficient. The existing V3+ framework codifies the overall evaluation process, which includes verification, usability validation, analytical validation, and clinical validation. To improve the scientific rigor of analytical validation, a hierarchical framework for selecting reference measures is needed because not all potential reference measures are of equal quality. The framework classifies reference measures based on attributes that contribute to reduced measurement variability, with defining and principal measures being the most rigorous due to objective data acquisition and the ability to retain source data.

Recommendations
The proposed framework sequentially moves the investigator through four steps: (1) Compile preliminary information, including the digital clinical measure, context of use (COU), algorithm requirements, and sensor verification evidence . (2) Select an existing reference measure, develop a novel comparator, or identify a set of anchor measures, prioritizing measures with the highest scientific rigor (defining → principal → manual → reported) . (3) Consider the impact of the data collection environment to determine if the analytical validation study can be conducted in the intended use environment with the highest-order measure, or if in-lab validation is necessary, ensuring the results are generalizable . (4) Describe the rationale for key study design decisions to encourage transparency for evaluators, regulators, and payers . Investigators must justify passing over a higher-ranked reference measure, generally only acceptable if the higher-ranked measure poses unacceptable risk or is not applicable to the context of use.

Regulatory Considerations
The principles of the framework for analytical validation apply regardless of the regulatory status of the sDHT (regulated medical device, low-risk general wellness apps, or research product) or its intended use (clinical care or clinical research). The framework is intended to help investigators support the most rigorous claims regarding sDHT performance, which is important for acceptance by evaluators, peer-reviewers, regulators, and payers. The categorization of the digital clinical measure as a digital biomarker or an electronic clinical outcome assessment also does not change the framework's applicability.

Findings
The rapid advancement of sensor technology and connectivity has enabled high-frequency, longitudinal monitoring of physiological processes, yet the infrastructure for large-scale deployment remains resource-intensive. Current challenges include a lack of standardized terminology for digital decision-making tools and significant variability in environmental factors that affect sensor performance. Proprietary algorithms and device-specific barriers often hinder the verification and validation processes necessary for regulatory approval. Additionally, there is a distinct gap between granular digital features and their clinical relevance or meaningfulness to patients. Ethical concerns are emerging around data management, patient anxiety in psychiatric contexts, and the responsibility for addressing adverse events detected by remote monitoring.

Recommendations
Stakeholders should develop consensus-driven frameworks for standardized device performance reporting and environmental testing to streamline evaluations for specific contexts of use. The community should adopt a modular approach to data standards that bins requirements by concept of interest and disease-specific needs. Collaborative efforts between patients and developers are essential to bridge the gap between technical metrics and meaningful aspects of health. It is recommended to implement ""bring-your-own-device"" (BYOD) frameworks that ensure data reliability while supporting the inevitable evolution of technology during long-term studies. Researchers and clinicians must be trained in the ethical, legal, and social implications of digital health technology use, particularly regarding data privacy and the management of remote-detected safety signals.

Regulatory Considerations
Digital health technologies used to collect endpoints must meet high evidentiary requirements for validation, with complexity increasing when multiple sensors or complex software are bundled. Regulatory agencies like the FDA and EMA have established pathways for the qualification of drug development tools, including biomarkers and clinical outcome assessments. Integration of new draft guidance on remote health monitoring with existing regulatory workflows is necessary to reduce uncertainty in trial evaluations. While many digital health technologies do not qualify as medical devices unless they have a specific medical purpose, synergies between device risk assessments and drug trial data integrity frameworks should be explored. Early engagement with regulators remains a critical step for obtaining feedback on novel digital endpoints and ensuring the suitability of evidentiary support.

Findings
The traditional process of evaluating biomarkers within the context of a single drug development program is inefficient and creates uncertainty for sponsors. This case-by-case approach leads to redundant efforts, slows down the development of novel therapies, and hinders the broad adoption of promising scientific tools. There is a clear need for a centralized, collaborative pathway to formally validate biomarkers, which can de-risk drug development, encourage innovation, and make the process more predictable and cost-effective for all stakeholders.

Recommendations
Drug developers, academic researchers, and other stakeholders should proactively engage with the FDA through the formal Biomarker Qualification Program to validate biomarkers for specific contexts of use. It is recommended to form public-private partnerships and other collaborations to pool resources and data, which strengthens the evidence package for a biomarker's utility. Developers should use the qualification process to establish a biomarker's value early, making it a publicly available and reliable tool that can accelerate the development of multiple drug products.

Regulatory Considerations
The Biomarker Qualification Program provides a distinct regulatory pathway for establishing a biomarker's validity for a specific Context of Use (COU), separate from an individual Investigational New Drug (IND) or New Drug Application (NDA). The process involves a three-stage submission and review cycle: the Letter of Intent, the Qualification Plan, and the Full Qualification Package. Once qualified, a biomarker is publicly listed and can be incorporated into multiple drug development programs without the need for sponsors to re-submit and re-justify the validation data for that specific COU, streamlining subsequent regulatory reviews.

Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
Establishing cross-sector consortia does not guarantee success without a unified objective and stakeholder buy-in. A neutral, independent facilitator is a key element for successful governance in many collaborative platforms. Many consortia lack consistent methods for storing critical data, meeting minutes, and regulatory briefing packages, which creates barriers after project completion. Regulatory success depends heavily on the early development of a strategy that defines the necessary evidence to validate innovative methodologies. Successful examples include the qualification of biomarkers for polycystic kidney disease and type 1 diabetes, as well as imaging measures for Alzheimer’s disease.

Recommendations
Consortium members should develop an initial regulatory strategy during the project scoping and planning phases. Teams must explicitly define the context of use for any proposed tool to articulate exactly what decisions the output will inform. A robust data strategy should be implemented early, including formal agreements for data use, standardization, and sharing that remain in place in perpetuity. Consortia must prioritize sustainability plans to ensure data and active databases remain available for research and regulatory use after funding expires. Projects should integrate regulatory science expertise from the start to cover both EU and US frameworks.

Regulatory Considerations
Regulators require individual patient-level data that is fully curated, standardized, and presented through formal submissions like qualification applications. Formal regulatory endorsement ensures a tool can be trusted for consistent interpretation in drug development and marketing authorization evaluations. Early engagement with agencies such as the FDA and EMA is essential to gain feedback on novel methodologies and align study designs with regulatory expectations. Specific pathways like the EMA Qualification of Novel Methodologies and the FDA Qualification Process for Drug Development Tools should be utilized. Regulatory qualification may require ongoing access to databases to support the long-term use of the methodology.

Findings
Long and Unpredictable Timelines: The COA Qualification Program is lengthy and unpredictable, with an average qualification time of six years. Nearly half of all submissions experience review times that exceed the FDA's own published targets.
Low Qualification and Uptake: As of October 2024, only seven COAs (8.1% of those listed) have been qualified, and only three of those have been used to support the benefit-risk assessment of new medicines. No COAs submitted after the passage of the 21st Century Cures Act in 2016 have been qualified.
Limited Regulatory Impact: Qualified COAs are consistently designated for "exploratory use" and have never been accepted as a primary endpoint in a clinical trial. In contrast, some non-qualified COAs have been used as key endpoints and included in drug labels, questioning the utility of the formal qualification pathway.
Discrepancy Between FDA Centers: There is a notable difference in how COAs are qualified between the drug (CDER/CBER) and device (CDRH) centers. The Kansas City Cardiomyopathy Questionnaire (KCCQ) was qualified by CDRH for use as a primary or secondary endpoint, while for drugs, it was only qualified as an "exploratory" measure.

Recommendations
Increase Transparency of Timelines: The FDA should publish its actual, historical review timelines for COA qualification so that drug developers can better plan and integrate these tools into their development programs.
Clarify the Use of Qualified COAs: The FDA should clearly articulate how and when qualified COAs can be used as primary or secondary endpoints to support regulatory decision-making and provide a clear pathway for updating a COA's status from "exploratory" to a key endpoint.
Publish Best Practices: Both sponsors and the FDA should be encouraged to publish their experiences with the qualification program to share best practices and learnings with the broader drug development community.
Create a List of Accepted Endpoints: The FDA should create and maintain a public list of qualified COAs that can be used as surrogate endpoints to support drug approval decisions, thereby increasing their utility and adoption.

Regulatory Considerations
"Qualified as a Measure" Ambiguity: The FDA's practice of qualifying COAs as "measures" for "exploratory use" creates regulatory uncertainty for sponsors, as it implies that significant additional evidence is still needed before the tool can be relied upon for a key endpoint.
Qualification is Not Required: The analysis shows that COAs can be accepted for regulatory decision-making and included in drug labels without going through the formal qualification program, suggesting that qualification is not a prerequisite for use as a reliable endpoint.
Unclear Path to Endpoint Progression: The current DDT guidance does not specify the process for upgrading a COA's qualification status (e.g., from exploratory to a primary endpoint) after additional data has been generated, which hinders its evolution and broader use.

Findings
Clinically Acceptable Performance: A clinical study demonstrated that the weekly AFib burden estimates from the Apple AFib History Feature were in close agreement with a reference ECG patch, with an average difference of just 0.67%. The vast majority of measurements had paired differences within ±10% of the reference device.
Generalizable Across Subgroups: The device's accuracy was similar across various subgroups, including different sexes, races, ages, and skin tones.
Performance Post-Ablation is Uncertain: In a small subgroup of patients with a prior cardiac ablation, the device's performance, while still strong, showed slightly more variability and exceeded a pre-specified acceptance criterion. The study was not designed or powered to demonstrate equivalent performance in this specific group.
Technical Limitations Exist: The feature only provides a retrospective weekly estimate and does not give specific timestamps or durations of AFib episodes. It also does not detect other atrial tachyarrhythmias, like atrial flutter.

Recommendations
Appropriate Use: The document implicitly recommends using the tool precisely within its qualified context of use—as a secondary, not primary, endpoint for comparing AFib burden between study arms in cardiac ablation device trials.
Supplemental Data Collection: For studies involving patients who have had a prior ablation, it would be beneficial to assess the tool alongside other methods of determining AFib burden to better characterize its performance in this population.
Define Study-Specific Endpoints: Investigators using the tool are responsible for defining and justifying their specific study designs and what constitutes a clinically significant reduction in AFib burden.

Regulatory Considerations
MDDT Qualification: The Apple AFib History Feature is officially qualified by the FDA as a Medical Device Development Tool (MDDT), which reduces the burden on device developers, as they no longer need to independently justify its methodology for collecting weekly AFib burden estimates in their clinical studies.
Secondary Endpoint Only: A key limitation for its regulatory use is its qualification only as a secondary endpoint. It cannot, by itself, be used to evaluate the primary safety and effectiveness of cardiac ablation devices. This is partly because FDA typically requires the inclusion of any atrial tachyarrhythmia (not just AFib) for defining ablation success in pivotal studies.
Not a Replacement for Primary Endpoints: The tool's utility is intended to provide supplemental data and help better understand post-treatment AFib burden; it is not meant to replace more clinically well-defined primary endpoints.

Findings
The development and evaluation of medical devices require scientifically plausible and reliable tools for collecting data to support regulatory submissions. A lack of standardized, pre-vetted tools can lead to inefficiencies and unpredictability in the device development and review process. The qualification of development tools can be applied across a wide range of device areas, including cardiovascular, neurology, imaging, and cybersecurity. The evidence required for tool qualification must be robust enough to support its intended context of use.

Recommendations
Tool developers, medical device sponsors, research organizations, and academic institutions are encouraged to voluntarily submit proposals to the MDDT program to qualify their tools. Submissions should include a detailed description of the tool, a clearly defined context of use (COU), specific performance criteria, and a comprehensive plan for collecting evidence to validate the tool's performance and scientific plausibility. Collaboration in developing tools and supporting evidence is recommended to pool resources and increase the acceptance of qualified tools.

Regulatory Considerations
The MDDT program is a formal regulatory mechanism for the FDA to qualify tools that can be used to support assessments of medical device safety, effectiveness, or performance. Once a tool is qualified for a specific context of use, the FDA accepts assessments from that tool in support of regulatory submissions without needing to re-evaluate the tool's suitability. The program recognizes four main categories of tools: Non-clinical Assessment Models (NAM), Biomarker Tests (BT), Clinical Outcome Assessments (COA), and an "Other" category for tools that do not fit the primary classifications.

Findings
The guidance applies to four types of Clinical Outcome Assessments (COAs): Patient-Reported Outcomes (PROs), Observer-Reported Outcomes (ObsROs), Clinician-Reported Outcomes (ClinROs), and Performance Outcomes (PerfOs). A COA is considered fit-for-purpose when the validation evidence is sufficient to support its context of use (COU). To determine if a COA is fit-for-purpose, sponsors must clearly describe the Concept of Interest (COI) and the COU, and present sufficient evidence to support a clear rationale for the COA's proposed interpretation and use. The rationale for using a COA should include up to eight components, such as justification for the COA type, capturing the important parts of the COI, appropriate administration and scoring, minimal influence from irrelevant factors or measurement error, and correspondence with the Meaningful Aspect of Health (MAH). The most direct assessment of how a patient feels or functions (MAH) should be used as the COI whenever possible.

Recommendations
Sponsors should use the Roadmap to Patient-Focused Outcome Measurement to guide the selection, modification, or development of a COA. The process begins with understanding the disease/condition (including patient perspectives) and conceptualizing clinical benefits and risks (defining the MAH, COI, and COU). When feasible, existing COAs are generally preferred, especially for well-established COIs, as this approach is often the least burdensome. If an existing COA is modified or used in a different context, additional evidence (e.g., cognitive interviews, psychometric studies) must be collected to justify its fitness for the new context of use. For new COA development, sponsors should involve patients, document all steps, and generally avoid using the new COA for the first time in a registration (pivotal) trial; a standalone observational study or early phase trial is recommended for evaluation.

Regulatory Considerations
Sponsors are encouraged to interact early and throughout medical product development with the relevant FDA review division to ensure COAs are appropriate for the intended COU. Sponsors should communicate their proposed COA-based endpoint approach, including the MAH, COI, COA type/name/score, and the final COA-based endpoint, ideally using the suggested format. The type and amount of evidence required to support the rationale for a COA's use is weighed against the degree of uncertainty regarding that part of the rationale. For ClinROs, it is recommended to use an assessor masked to treatment assignment and study visit for primary endpoints, if feasible. FDA strongly discourages proxy-reported measures for concepts known only to the patient (e.g., pain) and recommends using an ObsRO to measure observable behaviors instead when the patient cannot self-report.

Recommendations
Clearly define the concept of interest and its context of use to ensure COAs align with trial objectives.
Use conceptual and measurement frameworks to communicate how COAs measure patient experiences and generate interpretable scores.
Leverage existing COAs where possible, modifying them only when justified, and document all modifications rigorously.
Ensure COAs are accessible and inclusive, incorporating features like large fonts, touch interfaces, or audio assistance for diverse populations.
Conduct early engagement with FDA to discuss COA selection, development, and validation plans.

Regulatory Considerations
Fit-for-purpose validation requires evidence of conceptual alignment, scoring reliability, and sensitivity to clinically meaningful changes.
Digital health technologies used for COAs must comply with FDA’s guidance on data integrity, usability, and technical performance.
COAs intended for regulatory submissions must be developed and validated before pivotal trials to avoid jeopardizing trial outcomes.
Modifications to COAs or scoring methods during trials necessitate justification and revalidation.
Sponsors should submit comprehensive documentation on COA development, including scoring algorithms and item tracking matrices.

Findings
Regulatory Acceptance is Complex: Gaining regulatory acceptance for endpoints derived from Digital Health Technologies (DHTs) is a lengthy, multifaceted, and costly process that requires a global strategy and early health authority consultation.
"Fit-for-Purpose" is Key: A DHT's clearance or approval as a medical device does not automatically ensure it is fit-for-purpose in a clinical trial; its intended use must align with the specific context of use (COU) in the study.
Meaningfulness is a Hurdle: Demonstrating the clinical meaningfulness of novel digital endpoints, especially for abstract concepts like cognitive decline in Alzheimer's Disease, remains a significant challenge for regulatory acceptance.
International Harmonization is Lacking: Differences in regulatory requirements for DHT validation between major health authorities can delay or prevent the successful implementation of digital measures in global clinical trials.
Technology Changes Pose Risks: Software and hardware updates to DHTs during a clinical trial can have significant implications, potentially invalidating study results if not managed through a predetermined change-control plan.

Recommendations
Engage Health Authorities Early and Often: Sponsors should conduct multiple consultations with major health authorities (e.g., FDA, EMA) early in the development process to align on the Concept of Interest (COI), COU, and the validation roadmap.
Develop a Comprehensive Regulatory Strategy: A global regulatory strategy should be an integral part of the overall development plan, tailored to the program's objectives and endpoint hierarchy.
Establish "Fit-for-Purpose" Criteria: Before selecting a DHT, sponsors should establish the minimum technical and performance specifications required for the specific COU to guide the selection of a fit-for-purpose device.
Create a Conceptual Framework: For novel endpoints, sponsors should develop a conceptual framework that visualizes how the DHT-derived measure relates to meaningful health concepts and patient experiences.
Plan for Change and Missing Data: Sponsors should establish predetermined change-control plans with manufacturers to manage DHT updates and create risk management plans to minimize and handle missing data from remote acquisition.

Regulatory Considerations
Distinct Pathways in US vs. EU: The US FDA uses a risk-based approach for DHTs that are medical devices, while in Europe, CE marking for the intended COU is generally expected by the EMA.
Qualification is an Option, Not a Requirement: Both the FDA and EMA offer voluntary qualification programs for Drug Development Tools (DDTs), which can validate a DHT for a specific COU across multiple drug programs, though the process is resource-intensive.
Scientific Advice for Individual Programs: For DHTs used within a single drug development program, engaging with health authorities through scientific advice meetings is a more targeted and confidential pathway for gaining feedback and agreement.
Data Privacy and Security are Paramount: Sponsors must ensure that the collection, transfer, and storage of personal data via DHTs comply with all applicable regulations, such as GDPR in the EU, including cybersecurity and data transfer measures.

Findings
The use of Artificial Intelligence (AI) and Machine Learning (ML) is being applied to a broad range of drug development activities with the potential to accelerate the process and make clinical trials safer and more efficient. The inclusion of AI/ML is most common in the clinical development/research phase of regulatory submissions. Concerns exist that AI/ML algorithms could amplify errors and preexisting biases in underlying data sources, which raises issues related to generalizability and ethical considerations. Other challenges include limited explainability due to model complexity and proprietary reasons, as well as managing risks related to data quality, reliability, and representativeness. The FDA recognizes that a careful, risk-based assessment of the specific context of use (COU) is needed when evaluating AI/ML.

Recommendations
Stakeholders should adhere to practices in three key areas: human-led governance, accountability, and transparency; quality, reliability, and representativeness of data; and model development, performance, monitoring, and validation. A risk management plan should be applied to identify and mitigate risks based on the COU, guiding the level of documentation and transparency. Practices are needed to ensure the integrity of AI/ML and address issues like bias and missing data. For models, developers should use pre-specification steps and clear documentation for development and assessment criteria. Models must be monitored over time for reliability and consistency, and Real-World Data (RWD) performance can provide valuable feedback, including for potential re-training.

Regulatory Considerations
The FDA encourages early engagement through mechanisms like the Critical Path Innovation Meetings (CPIM), ISTAND Pilot Program, and Emerging Technology Program to discuss relevant AI/ML methodologies or technologies. The Verification and Validation (V&V 40) risk-informed credibility assessment framework and the principles for Good Machine Learning Practices (GMLP), while not specific to drug development, are helpful guides for evaluating models. The industry is exploring the use of a Predetermined Change Control Plan (PCCP) mechanism for AI/ML-based devices to proactively specify and manage modifications, enhancing adaptability. In general, a risk-based approach should guide the level of evidence and record keeping needed for the verification and validation of AI/ML models for a specific COU.

Findings
While verification, analytical validation, and clinical validation have been well-established, usability validation has not been systematically incorporated into digital health technology evaluation.
Variability in device designs, patient populations, and regulatory environments creates barriers to widespread adoption of sensor-based digital health technologies.
Usability problems, such as poor user interfaces and technical errors, can lead to significant data loss in clinical trials and real-world applications.
While some guidance exists for usability in medical devices, there is no unified global standard for assessing usability in digital health products, leading to inconsistencies in implementation.
Stakeholders, including regulators, industry leaders, and researchers, recognize the need for usability validation to ensure the real-world effectiveness of digital health technologies.

Recommendations
Adopt the V3+ framework as a standardized method to ensure that usability is rigorously tested alongside verification, analytical validation, and clinical validation.
Establish clear protocols for usability testing, including use specification development, risk analysis, iterative formative evaluations, and summative evaluations.
Bring together regulators, technology developers, clinicians, and patients to create guidelines that ensure fit-for-purpose digital health solutions.
Work with regulatory agencies such as FDA, EMA, and MHRA to establish harmonized global standards for usability validation.
Encourage the publication of usability study results, including negative findings, to facilitate transparency and continuous improvement in digital health technologies.

Regulatory Considerations
Agencies like FDA and EMA increasingly require usability data for digital health technologies, but standardized methodologies are still evolving.
Usability validation should align with regulatory requirements for medical devices and digital biomarkers, ensuring clinical relevance and data integrity.
Digital health technologies must adhere to HIPAA, GDPR, and other data protection regulations while ensuring seamless usability.
Poor usability can lead to missing or unreliable data, which affects regulatory submissions and real-world evidence generation.
A consistent approach to usability evaluation is needed to support regulatory decision-making and digital health product approvals globally.