Findings
Different Digital Health Technologies (DHTs) estimate sleep staging using data from various sensor-based sources (e.g., EEG, actigraphy, ballistocardiography), each with different properties impacting the estimation. Sleep staging algorithms are often proprietary. DHTs interpret sleep staging at different time intervals, or epochs (e.g., polysomnography uses 30-second epochs). DHT vendors transmit data at different levels, ranging from epoch-level data to pre-calculated summary data (e.g., "total sleep time").

Recommendations
Method and Signals: Ask the vendor about their method of sleep monitoring and which signals are being recorded and used, and understand the strengths and limitations of the technology.
Granularity and Epochs: Inquire about the granularity of sleep data estimated (coarse to fine grain) and the epoch length used for sleep annotations, as this informs interpretation and comparability to other research.
Thresholds and Rules: Ask what rules and thresholds are set for confirming events like sleep onset and offset to ensure certainty in the data and inform future interpretation of results.
Data Level: To align with the Core Digital Measures of Sleep, epoch-level data is preferred for further analysis and comparison between measurement systems. If only summary data is offered, ask for a detailed description of the estimation process.
Algorithms and Evidence: Ask for evidence to support the validity and reliability of the estimated sleep stages, which may include peer-reviewed manuscripts, technical documentation, and conference abstracts.

Regulatory Considerations
While not a regulatory document, the recommendations emphasize the need for vendors to provide evidence for the validity and reliability of their proprietary sleep staging algorithms. This evidence, which can be found in peer-reviewed literature or technical documentation, is crucial for establishing confidence in the results arising from the technology, and can be used for inclusion in, for example, regulatory documents.

Findings
Analytical validation is critical for ensuring digital clinical measures align with regulatory and scientific expectations, particularly when no established reference measures exist.
Novel digital measures require flexible validation approaches, as traditional clinical reference measures often do not directly correspond to digital endpoints​
Statistical methodologies must be tailored to the nature of digital measures, using approaches such as factor analysis, regression modeling, and latent variable estimation​
Regulatory engagement is crucial early in the validation process to align evidentiary standards and facilitate market adoption​
The validation process must be context-specific, considering population characteristics, data collection settings, and sensor variability to ensure reliability across diverse applications​.

Recommendations
Developers should follow a stepwise approach in designing validation studies, incorporating existing reference measures, novel comparators, and statistical validation techniques.
Regulatory authorities should provide clearer guidance on acceptable validation methodologies, particularly for novel digital endpoints.
Analytical validation must be tailored to the intended use environment, ensuring that sensor-based measures capture meaningful health outcomes in real-world settings​.
Multi-stakeholder collaboration (regulators, payers, researchers, and patients) should be prioritized to create consensus on validation strategies and market access pathways.
Machine learning and AI models used for digital clinical measures should undergo rigorous evaluation to mitigate bias and improve interpretability in healthcare decision-making​.

Regulatory Considerations
Digital endpoint validation must incorporate both traditional statistical measures and novel validation frameworks, ensuring credibility in regulatory submissions.
FDA and international regulators encourage early engagement to discuss validation plans, data requirements, and evidentiary thresholds for digital measures​.
Real-world evidence (RWE) and real-world data (RWD) should be leveraged to support regulatory submissions and post-market surveillance of digital health innovations​.
Validation studies should align with global regulatory standards, such as ISO, FDA’s digital health guidance, and European Medical Device Regulations (MDR).
Data privacy, security, and compliance with regulations like HIPAA and GDPR are critical considerations when deploying and validating digital clinical measures​

Findings
CADe clinical performance studies must address key variables, including reader variability, disease prevalence, and device design differences.
Properly conducted MRMC studies are critical for assessing diagnostic effectiveness, incorporating both unaided and aided reading conditions.
Enriched datasets, while useful for stress testing, must be carefully designed to avoid bias and reflect intended use populations.
The truthing process (establishing reference standards) is essential to validate device performance claims and should be rigorously defined.
The FDA encourages pre-specification of hypotheses, statistical methods, and endpoints to ensure robust and interpretable results.

Recommendations
Design studies with representative patient populations and include diverse subgroups relevant to the device’s intended use.
Use validated statistical methods for MRMC analyses, reporting sensitivity, specificity, and receiver operating characteristic (ROC) curve metrics.
Develop and document a detailed truthing process for establishing reference standards, ensuring consistency and reliability.
Conduct stress testing with enriched datasets to evaluate device performance under challenging conditions but avoid overrepresenting certain subsets.
Submit a complete study protocol and statistical analysis plan, including sample size justification, randomization methods, and scoring techniques.

Regulatory Considerations
CADe devices classified under 21 CFR 892.2050 or 892.2070 must comply with premarket notification requirements, including performance testing and labeling.
Standalone performance assessments may suffice in certain scenarios, but clinical studies are often necessary for substantial equivalence determinations.
Use of foreign clinical data requires justification of its applicability to U.S. populations and medical practice.
FDA expects data integrity controls, such as firewalls and audit trails, to prevent tuning bias in test datasets reused across studies.
The FDA encourages early engagement (e.g., Pre-Submission requests) for feedback on study protocols and regulatory pathways.

Findings
Limited evidence of clinical validity from pilot trials due to cost, time, and regulatory complexities.
Lack of standards for data integration across different tools and platforms.
Potential biases introduced by pre-existing algorithms.
Opaque data processing methods in BioMeTs.

Recommendations
Develop data, hardware, and software standards for BioMeTs.
Improve regulations for data rights, access, privacy, and governance.
Provide guidance on analytical methodologies for BioMeT data validation.

Regulatory Considerations
Early regulatory interactions with agencies like the FDA and EMA.
Ensuring data quality, integrity, reliability, and robustness.
Understanding regulatory pathways for BioMeTs in clinical trials.

Findings
A universally applicable evidentiary standard for biomarker qualification is not feasible; the necessary level of evidence depends entirely on the specific Context of Use (COU). The framework emphasizes that the strength of evidence is evaluated based on the potential risk and benefit associated with the biomarker's intended application in drug development. The relationship between a biomarker and clinical outcomes must be robustly demonstrated, but there are no fixed quantitative criteria for this association. The overall confidence in a biomarker is derived from a combination of analytical validation, clinical validation, and the strength of the biological rationale.

Recommendations
Sponsors should clearly define the specific COU for the biomarker early in the development process, as this will dictate the required evidentiary support. It is recommended that sponsors engage with the FDA throughout the biomarker development and validation process to ensure alignment on the evidentiary requirements. Submissions for biomarker qualification should include a comprehensive package of evidence detailing the analytical validation (how well the test measures the biomarker) and the clinical validation (how well the biomarker relates to a clinical endpoint). Sponsors should provide a strong biological rationale for the biomarker's role in the disease process and its relevance to the proposed COU.

Regulatory Considerations
The FDA's evidentiary framework is designed to be a flexible, risk-based approach to biomarker qualification. The qualification is specific to the COU for which it was evaluated and does not imply acceptance for other uses. The framework is intended to support the use of biomarkers as Drug Development Tools (DDTs), which can include uses for patient selection, as surrogate endpoints, or to demonstrate a drug's mechanism of action. The level of regulatory scrutiny is proportional to the impact the biomarker will have on drug development and clinical decision-making. Qualified biomarkers can help to de-risk and streamline the drug development process.

Findings
The complexity of selecting appropriate technology due to an increasing array of devices and sensors.
Risks associated with choosing inappropriate MSTs, including susceptibility to missing data or erroneous data transmission.
The need for both manufacturers and clinical trial sponsors to ensure analytical validation supports MST use.

Recommendations
Identify a digital outcome that meets an unmet need for the planned trial or population.
Determine whether the technology is fit-for-purpose based on the measure, context of use, and classification as a medical device.
Ensure devices are reliable and reproducible for collecting required data.
Conduct statistical analysis according to a predefined analysis plan.
Consider adaptive designs to reduce resource requirements and increase study success.

Regulatory Considerations
Compliance with medical device classifications such as 510(k)s and CE marks.
Ensure devices and platforms comply with HIPAA, GDPR, and data privacy regulations.
Be aware of potential updates to technology or software that could impact trials.

Findings
The CardiacSense PPG device can reliably detect heart rate in various situations, but noise suppression during activity remains a challenge.
The study did not directly address the device's ability to detect atrial fibrillation in ambulatory patients, indicating a gap in current research.
Further studies are needed to confirm the device's effectiveness in detecting AF during ambulatory conditions.

Recommendations
Improve noise suppression technology to enhance the device's accuracy during motion.
Conduct further studies to validate the device's ability to detect atrial fibrillation in ambulatory patients.
Continue research to address the limitations identified in the current study.

Regulatory Considerations
Adherence to FDA guidance for new medical device applications is crucial.
Ensure compliance with regulatory standards for digital health technologies.

Findings
Significant changes to device design, basic principles of operation, or clinical protocols require prior FDA approval through an IDE supplement.
Developmental changes made in response to information gathered during a clinical investigation and minor protocol modifications may be implemented with a 5-day notice if they meet specified criteria.
Changes that do not affect the scientific soundness of the investigational plan, risk-benefit profile, or participant rights and safety can be reported in an IDE annual progress report.
Sponsors are responsible for conducting risk analyses and using credible information, such as design controls or peer-reviewed literature, to justify changes.
The FDA reserves the right to question the appropriateness of changes implemented without prior approval.

Recommendations
Conduct a risk analysis for any device or protocol change and ensure no new risks are introduced.
Use credible information, such as design control data, preclinical testing, or published literature, to assess the impact of proposed changes.
Submit IDE supplements for significant design or protocol changes that affect the validity of study data, participant safety, or investigational plan soundness.
Use the 5-day notice process for developmental changes that improve safety or effectiveness but do not represent significant design changes.
Report minor investigational plan changes, such as clarifying instructions or updating IRB information, in the IDE annual progress report.

Regulatory Considerations
IDE Supplements: Required for significant changes to device design, manufacturing processes, or protocols that may impact study data validity, risk-benefit analysis, or participant safety.
5-Day Notices: Applicable for developmental changes and protocol modifications that do not introduce new risks or affect study soundness.
Annual Progress Reports: Used for minor changes, including clarifications to instructions for use or informed consent materials, provided they do not affect participant safety or study data integrity.
FDA reserves the right to reclassify changes implemented under a 5-day notice or annual report if they determine the changes required prior approval.