Findings
While verification, analytical validation, and clinical validation have been well-established, usability validation has not been systematically incorporated into digital health technology evaluation.
Variability in device designs, patient populations, and regulatory environments creates barriers to widespread adoption of sensor-based digital health technologies.
Usability problems, such as poor user interfaces and technical errors, can lead to significant data loss in clinical trials and real-world applications.
While some guidance exists for usability in medical devices, there is no unified global standard for assessing usability in digital health products, leading to inconsistencies in implementation.
Stakeholders, including regulators, industry leaders, and researchers, recognize the need for usability validation to ensure the real-world effectiveness of digital health technologies.

Recommendations
Adopt the V3+ framework as a standardized method to ensure that usability is rigorously tested alongside verification, analytical validation, and clinical validation.
Establish clear protocols for usability testing, including use specification development, risk analysis, iterative formative evaluations, and summative evaluations.
Bring together regulators, technology developers, clinicians, and patients to create guidelines that ensure fit-for-purpose digital health solutions.
Work with regulatory agencies such as FDA, EMA, and MHRA to establish harmonized global standards for usability validation.
Encourage the publication of usability study results, including negative findings, to facilitate transparency and continuous improvement in digital health technologies.

Regulatory Considerations
Agencies like FDA and EMA increasingly require usability data for digital health technologies, but standardized methodologies are still evolving.
Usability validation should align with regulatory requirements for medical devices and digital biomarkers, ensuring clinical relevance and data integrity.
Digital health technologies must adhere to HIPAA, GDPR, and other data protection regulations while ensuring seamless usability.
Poor usability can lead to missing or unreliable data, which affects regulatory submissions and real-world evidence generation.
A consistent approach to usability evaluation is needed to support regulatory decision-making and digital health product approvals globally.

Findings
Existing guidelines lack pragmatic application and systematic approach for device selection.
Device choice is dependent on measurement objectives, user population, and available resources.
Current frameworks do not systematically consider verification, validation, feasibility, and protocol design.
Rapid obsolescence of digital devices due to technological advancements.
Need to incorporate social/psychological factors into device selection.

Recommendations
Develop a practical guide with a systematic approach for selecting wearable devices.
Use five core criteria: continuous monitoring capability, device suitability and availability, technical performance, feasibility of use, and cost evaluation.
Prioritize feasibility of use to ensure user needs are incorporated into the selection process.
Adapt guide criteria to accommodate novel innovations.
Foster clarity and transparency in decision-making among researchers, HCPs, and device users.

Regulatory Considerations
Follow FDA guidance for digital health technology usage in clinical investigations.
Consider CTTI recommendations for improving clinical trial quality and efficiency.
Use ePRO Consortium's factors for device suitability in regulatory trials.
Apply international guidelines for specific measurements when available.

Findings
There is a lack of systematic approaches to guide the processes of collecting, interpreting, analyzing, and translating health data from wearables into digital biomarkers.
Most wearables have fixed measurement capabilities, limiting their translation to digital biomarkers.
Current guidance lacks study design and conduct elements that involve all stakeholders in an iterative approach for implementing digital biomarkers in practice.
Researchers and health professionals often rely on limited guidance for using wearable data in clinical practice and chronic disease management.

Recommendations
Implement the DACIA framework to provide interdisciplinary guidance on using wearable sensor data for digital biomarker development.
Focus on participant needs as a crucial factor for study success, applicable to both short and long-duration studies.
Involve relevant stakeholders in each key step of the DACIA framework in an iterative manner.
Apply the DACIA framework to explore digital biomarkers using various devices or signal measurements.
Reduce participant burden through support and continuous feedback.

Regulatory Considerations
Not mentioned

Findings
Standard assessments lack sensitivity in early stages of neurodegenerative diseases.
Challenges with the validity and quality of RMT measurements.
Issues related to equity and inclusion in deploying digital tools.
Importance of considering feasibility, acceptance, usability, and ecological validity of digital endpoints.

Recommendations
Develop regulatory strategies early on.
Ensure equity and inclusion in deploying digital tools.
Address challenges related to the validity and usability of digital endpoints.
Promote public-private partnerships to address privacy and security concerns.
Involve patients and stakeholders in the design and implementation of digital tools.

Regulatory Considerations
Acceptance of digital endpoints by regulatory authorities is crucial.
Validation with current gold standards and clinically meaningful legacy endpoints.
Ensure data security and privacy.

Findings
High rates of missing data in DHTs compared to traditional measures due to technical and software difficulties.
Practical issues such as unfamiliarity with platforms, connectivity difficulties, and lack of data visibility.
Specific technical issues like blocking of websites by firewalls and failed software updates leading to data loss.

Recommendations
Ensure appropriate workforce training for those involved in DHT deployment.
Conduct pilot evaluations in study sites to identify potential issues early.
Improve data visibility at both site and central coordinating team levels.
Implement robust feasibility testing before full-scale deployment.
Co-design DHTs with study staff and patients to enhance usability.

Regulatory Considerations
The FDA requires adequate training, education, and experience for those responsible for data capture using mobile technologies.
Ensure data integrity through oversight responsibilities as recommended by the Clinical Trials Transformation Initiative.

Findings
Quantitative imaging extracts numerical values from medical data that are subject to systematic error and random variation. The utility of these values depends on well-characterized performance and sufficient user information for interpretation. Performance specifications often change throughout the operating range of a device, such as volumetric reproducibility varying by structure size. Fully automated functions require more robust analytical validation than manual or semi-automated functions because they lack the opportunity for expert user correction. While phantoms serve as high-quality reference standards for ground truth, they are simplifications that may not fully reflect clinical performance.

Recommendations
Manufacturers should provide a detailed technical description of the quantitative imaging function, including the measurand, algorithm training paradigms, and level of automation. Performance specifications should incorporate objective reference values when available to allow for comparisons between subject and predicate devices. A sensitivity analysis should be conducted to determine the impact of sources of error like patient characteristics, image acquisition protocols, and image processing. Labeling must include clear instructions for user-performed quality assurance and specify any limitations where the function has been found ineffective. For automated devices, manufacturers should help users understand scenarios where the function might generate an incorrect output that is not easily identifiable.

Regulatory Considerations
The FDA recommends following a ten-step technical performance assessment process, ranging from defining the measurand to comparing statistical results against pre-defined acceptance criteria. Premarket submissions should include performance data demonstrating that the device meets claims regarding bias, precision, linearity, and limits of quantitation. Uncertainty should be reported in units of the measurand and cover the entire operating range of the function. Manufacturers are encouraged to use the Q-Submission process to address questions regarding regulatory status or specific requirements. Software implementation details should align with existing FDA guidance for the content of premarket software documentation.

Findings
Verification involves testing the accelerometer's technical specifications (e.g., accuracy and precision) through peer-reviewed studies.
Validation of the algorithm relies on "ground truth" data, gathered through infrared video recordings and manual scoring of movements.
Cross-validation was used to assess the algorithm's performance, with additional validation in independent samples planned.
The separation of verification and validation allows greater flexibility, enabling the algorithm's use with multiple accelerometer devices that meet minimum standards.

Recommendations
Conduct separate verification and validation processes to ensure the reliability of both the device and the algorithm.
Use peer-reviewed publications to document the performance of DHTs and their limitations.
Ensure validation includes testing with representative populations to confirm the algorithm’s utility across diverse contexts.
Promote industry-wide standards to facilitate scalability and regulatory acceptance of DHTs in clinical trials.

Regulatory Considerations
Ensure DHTs undergo rigorous verification to meet accuracy and precision standards documented in peer-reviewed studies.
Validate algorithms using empirical "ground truth" data to demonstrate their ability to measure clinically meaningful outcomes.
Align the design and validation of DHTs with regulatory expectations for reliable and transferable performance across devices.

Findings
The MS digital health space is still largely uncharted.
Balancing the needs and desires of different users when creating a digital solution is challenging.
Insufficient adherence to remote digital health solutions presents a challenge to long-term engagement.
Creating a digital solution that is both meaningful to end users and aligned with regulatory standards involves challenges and compromises.

Recommendations
Employ an iterative development process to continually refine digital health solutions.
Collaborate closely with healthcare professionals and patients throughout the design process.
Use behavioral science strategies to enhance user engagement and adherence.
Ensure that digital solutions are scientifically robust and meet regulatory standards.
Implement a prescription-based model to improve adherence and integration into clinical practice.

Regulatory Considerations
Conduct technical verification and clinical validation for each assessment in digital health solutions.
Ensure data privacy and cybersecurity measures are robust and comply with local regulations.
Maintain ongoing post-marketing surveillance to monitor safety and effectiveness.
Adapt solutions to meet diverse regulatory requirements across different geographies.

Findings
The selection of DHTs must align with the specific goals of the trial, focusing on unmet patient or scientific needs.
A specification-driven approach, rather than solely relying on a technology's regulatory status, ensures alignment with trial requirements.
Verification and validation are distinct processes; both are critical to confirm the reliability and clinical relevance of DHTs.
Pre-trial feasibility studies help identify potential issues, such as wear-time compliance or usability concerns, before full implementation.
DHTs can alter participant interactions and trial workflows, necessitating clear communication, training, and management plans.

Recommendations
Define Measurement Goals Before Selection: Ensure that the decision to use a DHT is based on unmet needs or the promise of reducing trial burdens.
Adopt a Specification-Driven Selection Process: Tailor DHT selection to technical performance, participant needs, and study-specific requirements.
Verify and Validate Technologies Thoroughly: Collaborate with manufacturers to ensure DHTs are tested in both controlled and real-world settings and validated for the target population.
Conduct Feasibility Studies: Test DHTs for tolerability, usability, and compliance within the specific trial context to identify and address issues early.
Prepare for Operational Challenges: Develop a robust management plan with standard operating procedures (SOPs) to address potential failures and ensure smooth implementation.

Regulatory Considerations
The regulatory status of a DHT should not solely drive its selection; instead, focus on its ability to meet trial specifications.
Ensure transparent collaboration with manufacturers to document DHT performance characteristics and limitations.
Validate endpoints and DHT data to align with evidentiary standards for regulatory submissions.
Use feasibility studies and SOPs to ensure that DHTs comply with regulatory and operational requirements during trials.

Findings
Developing a novel endpoint requires an iterative and patient-centered approach, beginning with defining the study population and relevant health aspects.
Concepts of interest (COIs) must be specific, measurable, and clinically meaningful, with input from patients and caregivers.
Endpoint validation includes defining meaningful change, ensuring content validity, and demonstrating the ability to detect change.
Digital tools must meet criteria for usability, analytic validity, and tolerability within the target population.
Regulatory engagement and alignment throughout the process are critical to endpoint acceptance.

Recommendations
Define the study population and context of use (COU) early to guide endpoint and technology selection.
Identify meaningful health aspects (MHA) and concepts of interest (COI) with input from patients and clinicians.
Select and validate DHTs based on performance, usability, and their ability to capture meaningful data.
Establish meaningful change thresholds and validate endpoints in real-world settings.
Engage with regulators at every stage to align endpoints with evidentiary and regulatory standards.

Regulatory Considerations
Define and validate meaningful change thresholds that reflect treatment benefits for regulatory acceptance.
Ensure DHTs meet analytic validity standards, including accuracy, reliability, and reproducibility.
Demonstrate content validity, ensuring that endpoints accurately reflect the intended COI across the full range of anticipated data.
Align with regulatory requirements to incorporate validated endpoints into pivotal trials.
Address usability, data privacy, and compliance concerns to meet regulatory and operational standards.

Findings
Lack of widely accepted tools for digital mobility assessment.
Challenges in technical and clinical validation due to multiple expertise requirements.
Inconsistent testing procedures and variations in norms.
Limitations of current mobility measurement methods.
Need for real-world mobility assessment.

Recommendations
Adopt best practices and innovate with standards and open access tools.
Ensure transparency through regular interaction with stakeholders.
Develop algorithms in an agnostic and fully documented manner.
Make data accessible through a digital data biobank.
Aim for regulatory approval with accurate real-world mobility measurement.

Regulatory Considerations:
Engage in early dialogue with regulatory authorities.
Understand different regulatory requirements based on context of use.
Focus on qualification of new methodologies for safety and efficacy.
Use DMOs to monitor disease progression and as surrogates for secondary endpoints.
Adopt a staged approach to regulatory qualification.

Findings
Value-Based Metrics: Digital endpoints should directly measure meaningful outcomes for patients, emphasizing health-related quality of life and real-world relevance.
Technical Validation: Validation must ensure device reliability, data security, and usability in real-world settings while addressing potential confounders like environmental variability.
Clinical Validation: Rigorous evaluation should assess tolerability, differences between patients and controls, repeatability, correlation with traditional metrics, and responsiveness to disease changes.
Regulatory Challenges: Clear guidelines for digital endpoints are lacking, but early engagement with FDA or EMA can streamline the qualification process.
Collaboration Needs: Greater collaboration across stakeholders, including patients, regulators, and industry, is essential to standardize methodologies and share data effectively.

Recommendations
Engage Early with Regulators: Begin discussions with agencies like FDA and EMA to align on endpoint requirements, definitions, and validation approaches.
Adopt Patient-Centric Design: Collaborate with patients and advocacy groups to ensure digital endpoints are relevant, tolerable, and user-friendly.
Standardize Validation Processes: Follow a structured framework that includes technical validation, clinical evaluation, and regulatory case-building.
Invest in Data Sharing and Harmonization: Create shared databases and metadata standards to integrate findings across studies and devices, reducing duplication.
Encourage Open Science: Promote transparency and collaboration among researchers, industry, and regulatory bodies to accelerate innovation.

Regulatory Considerations
Regulatory Alignment: Align endpoints with EMA and FDA guidance to meet evidentiary standards for qualification and integration into clinical trials.
Iterative Validation: Conduct iterative validation studies, integrating feedback from regulatory interactions and stakeholder collaborations.
Privacy and Compliance: Ensure data privacy and security compliance, particularly when leveraging wearable and mobile technologies for home-based monitoring.
Address Real-World Variability: Provide evidence that real-world variability does not significantly bias results and demonstrate endpoint reliability across diverse populations.
Build Regulatory Confidence: Use validated endpoints in exploratory or secondary roles initially to build evidence for their adoption as primary endpoints.