Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
The use of Artificial Intelligence (AI) and Machine Learning (ML) is being applied to a broad range of drug development activities with the potential to accelerate the process and make clinical trials safer and more efficient. The inclusion of AI/ML is most common in the clinical development/research phase of regulatory submissions. Concerns exist that AI/ML algorithms could amplify errors and preexisting biases in underlying data sources, which raises issues related to generalizability and ethical considerations. Other challenges include limited explainability due to model complexity and proprietary reasons, as well as managing risks related to data quality, reliability, and representativeness. The FDA recognizes that a careful, risk-based assessment of the specific context of use (COU) is needed when evaluating AI/ML.

Recommendations
Stakeholders should adhere to practices in three key areas: human-led governance, accountability, and transparency; quality, reliability, and representativeness of data; and model development, performance, monitoring, and validation. A risk management plan should be applied to identify and mitigate risks based on the COU, guiding the level of documentation and transparency. Practices are needed to ensure the integrity of AI/ML and address issues like bias and missing data. For models, developers should use pre-specification steps and clear documentation for development and assessment criteria. Models must be monitored over time for reliability and consistency, and Real-World Data (RWD) performance can provide valuable feedback, including for potential re-training.

Regulatory Considerations
The FDA encourages early engagement through mechanisms like the Critical Path Innovation Meetings (CPIM), ISTAND Pilot Program, and Emerging Technology Program to discuss relevant AI/ML methodologies or technologies. The Verification and Validation (V&V 40) risk-informed credibility assessment framework and the principles for Good Machine Learning Practices (GMLP), while not specific to drug development, are helpful guides for evaluating models. The industry is exploring the use of a Predetermined Change Control Plan (PCCP) mechanism for AI/ML-based devices to proactively specify and manage modifications, enhancing adaptability. In general, a risk-based approach should guide the level of evidence and record keeping needed for the verification and validation of AI/ML models for a specific COU.

Findings
Existing guidelines lack pragmatic application and systematic approach for device selection.
Device choice is dependent on measurement objectives, user population, and available resources.
Current frameworks do not systematically consider verification, validation, feasibility, and protocol design.
Rapid obsolescence of digital devices due to technological advancements.
Need to incorporate social/psychological factors into device selection.

Recommendations
Develop a practical guide with a systematic approach for selecting wearable devices.
Use five core criteria: continuous monitoring capability, device suitability and availability, technical performance, feasibility of use, and cost evaluation.
Prioritize feasibility of use to ensure user needs are incorporated into the selection process.
Adapt guide criteria to accommodate novel innovations.
Foster clarity and transparency in decision-making among researchers, HCPs, and device users.

Regulatory Considerations
Follow FDA guidance for digital health technology usage in clinical investigations.
Consider CTTI recommendations for improving clinical trial quality and efficiency.
Use ePRO Consortium's factors for device suitability in regulatory trials.
Apply international guidelines for specific measurements when available.

Findings
The FDA considers electronic records and signatures equivalent to their paper counterparts when they meet the requirements of 21 CFR Part 11. Due to technological advances, electronic systems and digital health technologies (DHTs) are now integral to clinical trials, requiring a modern, risk-based approach to ensure data integrity. Sponsors remain ultimately responsible for the quality and integrity of all data submitted, even when using third-party IT service providers or data from real-world sources like EHRs. The authenticity, integrity, and confidentiality of electronic data are paramount and must be maintained through robust system controls throughout the data lifecycle.

Recommendations
Regulated entities should use a justified and documented risk-based approach to validate all electronic systems before and during a clinical trial, with the level of validation depending on the system's potential to impact participant safety and trial result reliability. Secure, computer-generated, time-stamped audit trails must be implemented to track the creation, modification, and deletion of all electronic records without obscuring original data. Robust logical and physical access controls are necessary to limit system access to authorized individuals. Entities should have written agreements with IT service providers that clearly define roles, responsibilities, and procedures for ensuring data security and long-term retention.

Regulatory Considerations
The requirements of 21 CFR Part 11 apply to all electronic records created, modified, or submitted to the FDA under predicate rules for clinical investigations, including those from foreign sites under an IND or IDE. While the FDA does not intend to assess the Part 11 compliance of external source systems like EHRs, data becomes subject to these regulations once transferred into the sponsor's electronic system. During inspections, the FDA will focus on system validation, data handling procedures, security protocols, audit trails, and documentation of sponsor oversight. Users must certify to the FDA that their electronic signatures are the legally binding equivalent of handwritten signatures.

Findings
Misclassification of wakefulness during sleep periods and issues with tracking outside main sleep bouts.
Bias in performance evaluation studies due to limited representation of diverse populations.
Hidden complexities in consumer-grade devices related to data access, fees, privacy, and security.

Recommendations
Carefully interpret study results based on wearable sleep-tracking technology data.
Address biases in study populations by including diverse cohorts.
Ensure proper preprocessing of data from consumer-grade devices.
Avoid inserting personally identifiable information in device settings.
Evaluate issues related to specific populations like minors.

Regulatory Considerations
Complexity of privacy laws across different countries.
Need for strategies to protect personal information in device settings.
Consideration of specific population issues, such as minors, in regulatory frameworks.

Findings
A proactive risk assessment is essential for optimizing study quality by identifying and mitigating risks to human subject protection and data integrity before and during a trial. Monitoring should be comprehensive, addressing not only likely risks identified initially but also less probable, high-impact risks and unanticipated issues that emerge. The effectiveness of a monitoring strategy depends on tailoring its timing, frequency, and methods to study-specific factors like complexity and site experience. Centralized monitoring, as part of a risk-based approach, can detect systemic issues like data omissions or protocol deviations more rapidly than traditional on-site visits alone.

Recommendations
Sponsors should formally document their risk assessment methodologies and ensure these assessments directly inform the creation and revision of monitoring plans. Monitoring plans must be detailed, outlining the study design, specific data sampling strategies, and clear protocols for escalating significant issues. When significant problems are identified, sponsors must conduct a timely root cause analysis and implement corrective and preventive actions. All monitoring activities, findings, and subsequent actions should be thoroughly documented and communicated to sponsor management, clinical site staff, and other relevant parties.

Regulatory Considerations
FDA regulations mandate sponsor oversight and proper monitoring but do not prescribe specific methods, providing the flexibility for sponsors to adopt a risk-based approach. The FDA may request a sponsor's risk assessment and monitoring plan documentation during an inspection. This guidance represents the Agency's current thinking and is nonbinding, allowing sponsors to use alternative approaches if they satisfy regulatory requirements. A key focus of monitoring should be to ensure critical trial processes, such as the maintenance of blinding, are protected to maintain overall data and trial integrity.

Findings
The adoption of BYOD in clinical trials has been accelerated by the COVID-19 pandemic and supportive regulatory guidance, which now recognize it as an acceptable means for remote data collection. Studies have shown high measure completion and equivalent data quality between provisioned devices and BYOD, supporting its use in diverse patient populations. Key challenges to BYOD implementation include ensuring data equivalence across a wide variety of personal devices, managing participant technical support, and addressing data privacy and security concerns. The choice between native apps and web-based solutions involves trade-offs in usability, data security, and operational complexity.

Recommendations
Sponsors should develop a clear BYOD strategy that considers the target patient population, the complexity of the required data collection, and the global regulatory landscape. A robust training and support plan is essential for both participants and site staff to ensure proper device use and troubleshooting. Sponsors should work with technology vendors to ensure their platforms are user-friendly, secure, and capable of handling data from a variety of devices. It is crucial to establish clear communication channels for participants to report technical issues and receive timely assistance.

Regulatory Considerations
Both the FDA and EMA have issued guidance that supports the use of BYOD in clinical trials, provided that data integrity, security, and privacy are maintained. Sponsors must be able to demonstrate the equivalence of data collected via BYOD with data from provisioned devices. All BYOD solutions must comply with relevant data protection regulations, such as GDPR and HIPAA. The regulatory submission should include a clear description of the BYOD strategy and a justification for its use in the trial.

Findings
High rates of missing data in DHTs compared to traditional measures due to technical and software difficulties.
Practical issues such as unfamiliarity with platforms, connectivity difficulties, and lack of data visibility.
Specific technical issues like blocking of websites by firewalls and failed software updates leading to data loss.

Recommendations
Ensure appropriate workforce training for those involved in DHT deployment.
Conduct pilot evaluations in study sites to identify potential issues early.
Improve data visibility at both site and central coordinating team levels.
Implement robust feasibility testing before full-scale deployment.
Co-design DHTs with study staff and patients to enhance usability.

Regulatory Considerations
The FDA requires adequate training, education, and experience for those responsible for data capture using mobile technologies.
Ensure data integrity through oversight responsibilities as recommended by the Clinical Trials Transformation Initiative.

Findings
Not all CDS software is regulated as a medical device; the FDA applies specific criteria to determine its classification.
CDS software functions are excluded from the device definition if they meet all four criteria in section 520(o)(1)(E) of the FD&C Act.
Automation bias in decision-making poses a risk, particularly in time-critical scenarios, and influences regulatory considerations.
Clear labeling and transparency about the basis for recommendations are essential for enabling HCPs to make independent decisions.
Software functions that provide specific diagnostic outputs or time-critical directives typically fail to meet the criteria for Non-Device CDS.

Recommendations
Clearly define the intended use, user population, and input medical information for CDS software in labeling.
Ensure that software provides transparent and plain language descriptions of algorithms, data sources, and validation results.
Avoid presenting specific treatment or diagnostic directives to ensure the software supports rather than replaces HCP judgment.
Include sufficient information to allow HCPs to independently review and understand the basis for software recommendations.
Engage with the FDA early in the development process for software functions with potential regulatory oversight.

Regulatory Considerations
CDS software functions that meet all four criteria under section 520(o)(1)(E) of the FD&C Act are excluded from FDA’s definition of a device.
Software intended for time-critical decision-making or replacing HCP judgment is generally considered a device.
Developers must ensure that software labeling and functionality align with the criteria for Non-Device CDS.
Transparency in data sources, algorithm logic, and validation methods is required to enable independent HCP decision-making.
The FDA may request additional information or oversight for software that poses significant risks to patient safety.

Findings
CADe clinical performance studies must address key variables, including reader variability, disease prevalence, and device design differences.
Properly conducted MRMC studies are critical for assessing diagnostic effectiveness, incorporating both unaided and aided reading conditions.
Enriched datasets, while useful for stress testing, must be carefully designed to avoid bias and reflect intended use populations.
The truthing process (establishing reference standards) is essential to validate device performance claims and should be rigorously defined.
The FDA encourages pre-specification of hypotheses, statistical methods, and endpoints to ensure robust and interpretable results.

Recommendations
Design studies with representative patient populations and include diverse subgroups relevant to the device’s intended use.
Use validated statistical methods for MRMC analyses, reporting sensitivity, specificity, and receiver operating characteristic (ROC) curve metrics.
Develop and document a detailed truthing process for establishing reference standards, ensuring consistency and reliability.
Conduct stress testing with enriched datasets to evaluate device performance under challenging conditions but avoid overrepresenting certain subsets.
Submit a complete study protocol and statistical analysis plan, including sample size justification, randomization methods, and scoring techniques.

Regulatory Considerations
CADe devices classified under 21 CFR 892.2050 or 892.2070 must comply with premarket notification requirements, including performance testing and labeling.
Standalone performance assessments may suffice in certain scenarios, but clinical studies are often necessary for substantial equivalence determinations.
Use of foreign clinical data requires justification of its applicability to U.S. populations and medical practice.
FDA expects data integrity controls, such as firewalls and audit trails, to prevent tuning bias in test datasets reused across studies.
The FDA encourages early engagement (e.g., Pre-Submission requests) for feedback on study protocols and regulatory pathways.

Findings
Traditional RCTs face high costs, long timelines, recruitment challenges, and lack of diversity.
Recruitment efficiency in stroke trials has decreased over the past 25 years.
Digital tools for stroke prevention often lack quality and interactive functionality.
Decentralized RCTs present challenges in data quality and require validation.
Regulatory and compliance requirements vary significantly across regions.

Recommendations
Adopt decentralized RCTs with a patient-centric approach.
Leverage digital technologies to improve trial efficiency and participant experience.
Ensure participant engagement and education in trial design.
Provide high-quality training and support for decentralized procedures.

Regulatory Considerations
Collaborate with regulatory agencies early in trial design.
Compliance with varying international standards is necessary.
Rapid evolution of technology outpaces regulatory changes.
Cross-border data standards and privacy rules must be observed.

Findings
The DTRA Best Practice Evaluation Rubric uses five dimensions to determine if a DCT practice should be considered a "best practice":
Evidence of Success: Requires measurable and demonstrable success using KPIs and tangible outcomes.
Improving Patient Experience: Must address the needs of patients, caregivers, and therapeutic experts, demonstrating improved experience and engagement.
Site Impact: Must consider the implications of adoption and the practical impact on site burden and working practices.
Operational and Technical Feasibility: Ensures operational and technical aspects (including ongoing support, security, integrity, scaling, and reuse) have been fully considered when deploying new technologies.
Regulatory & Ethical Compliance: Requires appropriate consideration of global and local regulations and guidance (e.g., ICH E6/E8, GDPR, HIPAA), including adherence to privacy, consent, and ethical safeguards.

Recommendations
A practice should demonstrate several key factors across the dimensions:
Patient-Centricity: Reduce patient burden by offering the option to reduce physical visits and enable greater patient empowerment and access to information. It should strive to increase the diversity of recruited patients while mitigating bias toward technologically literate patients.
Site Support: Achieve a net reduction in burden for sites, utilizing simple, intuitive technology with minimal, on-demand training. It must provide clarity of fiduciary responsibility and use technology to increase risk-based monitoring without sacrificing data integrity.
Technical Rigor: Have a clear problem statement and a thoroughly defined strategy to mitigate operational and technical risks. It should take a holistic approach and ensure the solution is fit for use for the specific patient population, aligning with data privacy and security standards.

Regulatory Considerations
Practices must ensure compliance with both global and local regulations and Health Authority guidance. Explicit attention must be given to aligning with ICH E6 (Good Clinical Practice) and privacy laws like GDPR and HIPAA. The design must protect stakeholders providing sensitive or personal data with safeguards to ensure ethical safety.