Findings
The document introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an Artificial Intelligence (AI) model's output when used to support regulatory decisions regarding drug safety, effectiveness, or quality. The framework outlines a 7-step process beginning with defining the question of interest and the Context of Use (COU). Credibility is defined as trust, established through evidence, in the AI model's performance for a particular COU. The credibility assessment is tailored to the AI model risk, which is a combination of model influence (the AI model's evidence contribution relative to other evidence) and decision consequence (the significance of an adverse outcome from an incorrect decision). The document highlights challenges with AI use, including variability in development datasets (training/tuning), the need for methodological transparency due to model complexity, difficulty in quantifying and interpreting uncertainty in model output, and the potential for performance change over time (data drift), which necessitates life cycle maintenance.

Recommendations
Sponsors and interested parties should define the question of interest and clearly define the COU, detailing the AI model's specific role and scope and whether other information will be used. They should assess the AI model risk (low, medium, or high) to ensure that subsequent credibility assessment activities (Step 4) are commensurate with that risk and tailored to the COU. For Step 4, the credibility assessment plan should include a description of the model, model development process (including inputs, architecture, feature selection, and rationale), and data used (training and tuning data). Development data must be deemed fit for use (relevant and reliable) to mitigate issues like algorithmic bias. The plan should also detail the model evaluation process using independent test data and include performance metrics with confidence intervals, an estimate of uncertainty, and a description of model limitations. Early engagement with the FDA is strongly encouraged to discuss model risk and the adequacy of the credibility assessment plan.

Regulatory Considerations
The risk-based credibility assessment framework is intended to help organize and document information for regulatory submissions. The required stringency of assessment activities and the level of documentation should be commensurate with the AI model risk. For AI models whose performance can change over time (e.g., in pharmaceutical manufacturing or postmarketing), sponsors must implement life cycle maintenance plans to monitor performance and manage changes in a risk-based manner. Changes to AI models should be evaluated through the manufacturer's change management system and may require re-execution of parts of the credibility assessment plan. Early engagement can be facilitated through formal meetings (e.g., Pre-IND) or other specialized programs listed in the guidance, such as the Center for Clinical Trial Innovation (C3TI), the Model-Informed Drug Development (MIDD) Paired Meeting Program, and the Emerging Technology Program (ETP) or Advanced Technologies Team (CATT).

Findings
Usability is a multi-domain concept that requires a combination of methods for evaluation. Evaluations fall into two types: formative (for design modification of prototypes) and summative (for demonstrating usability of the final product to a representative user sample). The user experience metrics fall into several domains, including: Satisfaction, Usefulness, Ease of use, Learnability, Efficiency, Memorability, Understandability, Actionability, Readability, and Use-errors. Metrics related to Satisfaction and Usefulness are always subjectively reported by users.

Recommendations
Developers should select metrics based on the specific usability-related domain being evaluated.
Subjective Data (e.g., Satisfaction, Usefulness): Capture through qualitative surveys, quantitative surveys (scales), interviews, focus groups, and think-aloud evaluations .
Objective Data (e.g., Ease of use, Use-errors): Capture through direct or indirect observation (e.g., counting steps/attempts, timing task completion), or by using data generated by the sDHT (e.g., error reports, timestamps, page load times).
Time-based Metrics: Evaluate Learnability (ease of first use), Efficiency (ease with experience), and Memorability (ease after non-use) by measuring ease of use at different points in time .
Information Presentation: If the sDHT presents clinical data or written information (instructions, warnings), evaluate Understandability, Actionability, and Readability .
Use-errors: Objectively capture the number, type, and recoverability of use-errors (actions, or lack thereof, that may result in harm) via observation and sDHT data, noting that "use-error" is preferred to "user-error".

Regulatory Considerations
While this guide does not reference regulatory bodies like the FDA, it is part of the V3+ framework and recommends that researchers prioritize essential documents like the use specification and use-related risk analysis before designing a usability study. Summative evaluations demonstrating usability against a representative user sample under intended use conditions are the standard for demonstrating product fitness.

Findings
Sites must establish effective data privacy and security plans, especially considering regional and global regulations like GDPR.
Risk mitigation is critical, including plans to address unanticipated issues and potential patient disengagement due to technology challenges.
Budgeting and contracting often involve additional considerations, such as storage, training, and technical support requirements for connected devices.
Sites require adequate training to ensure staff and patients are prepared to use connected devices efficiently.
Companion applications or services often play an essential role in device functionality and data transmission.

Recommendations
Develop a clear plan for data pathways, including storage, security, and regulatory compliance.
Establish detailed risk mitigation and management strategies to handle unexpected challenges.
Ensure comprehensive training programs for site staff and patients to enhance device usability.
Incorporate device storage and resource allocation into budgeting and contracting processes.
Facilitate effective communication with sponsors and vendors to resolve operational and technical issues promptly.

Regulatory Considerations
Ensure connected devices comply with CFR 21, Part 11, and other relevant data collection and transmission regulations.
Understand and adhere to local and regional data privacy laws, such as GDPR, when managing patient data.
Verify that appropriate licenses and regulatory approvals are in place for device data transmission and storage.
Assess and address shipping and handling regulations for devices, ensuring safe and compliant transportation.

Findings
For an sDHT to be considered fit-for-purpose, a researcher or healthcare provider must understand the alignment between the sDHT's intended use (What it does, who uses it, where/when/how) and their own context of use . Key information for this assessment comes from the developer's Use Specification (detailing hardware, software, accessories, training) and Use-Related Risk Analysis (detailing warnings, harms from use-errors, and risk avoidance) . Usability validation evidence should cover study objectives, protocols, participant characteristics, metrics, and collection methods.

Recommendations
Researchers/providers should use the checklist to:
The Basics: Compare the sDHT's intended use to their context of use; if there is substantial overlap, existing evidence may be sufficient.
Use Specification/Risk Analysis: Gather detailed descriptions of the sDHT's hardware, software, accessories, written materials, training, cautions, warnings, and potential harms from use-errors to update their own Use Specification and Use-Related Risk Analysis .
Existing Evidence: Access existing usability validation study results (objectives, methods, participant characteristics, metrics, etc.) to determine its applicability and generalizability to their context of use .
Collaboration: Consider establishing a collaborative relationship with the developer to provide feedback for next-generation sDHTs, ensure version control, and potentially collaborate on future usability validation studies .

Regulatory Considerations
The document notes that if the sDHT is a regulated medical device, the intended use statement should already capture the answers to the basic questions. The entire checklist is framed around the V3+ framework, which is designed to ensure the rigor necessary for a product to be considered fit-for-purpose by all stakeholders.

Findings
FDA considers electronic records and signatures to be equivalent to paper records and handwritten signatures when they meet the requirements of 21 CFR part 11. Advances in technology, including Digital Health Technologies (DHTs) and cloud computing, necessitate updated guidance on ensuring the authenticity, integrity, and confidentiality of electronic data in clinical investigations. Records submitted to the FDA under predicate rules (e.g., marketing applications) are subject to part 11. FDA does not intend to assess the compliance of external Real-World Data (RWD) sources like Electronic Health Record (EHR) systems with part 11, but the sponsor remains responsible for the quality and integrity of all submitted data.

Recommendations
Risk-Based Validation: Regulated entities should use a risk-based approach to validation for all electronic systems deployed, proportionate to the risks to participant safety and reliability of trial results. Validation must cover system functionality, trial-specific configurations, customizations, and interoperability.
Data Retention & Audit Trails: Electronic records must be retained for the applicable period in a secure and traceable manner. Audit trails must capture all changes (old/new value, user ID, date/time) and should be protected from modification.
Security & Access Controls: Logical and physical access controls (e.g., strong login credentials, multi-factor authentication) must limit system access to authorized users based on a documented risk assessment. Security safeguards (e.g., encryption, antivirus) must be in place to protect data at rest and in transit.
DHT Use: DHTs should be selected and validated to be fit for purpose. The data originator (person, system, or DHT itself) must be associated with every data element as part of the audit trail. The final location of source data for inspection is the durable electronic data repository, not the individual DHT.
Outsourcing: Regulated entities must have a written agreement with IT service providers (including for cloud computing) detailing roles, responsibilities, and the service provider's ability to provide data integrity and security safeguards. The sponsor must maintain oversight.

Regulatory Considerations
FDA does not certify electronic systems or signature methods; they are evaluated during inspection. Users of electronic signatures must submit a letter of non-repudiation to the FDA certifying that the electronic signature is the legally binding equivalent of a handwritten signature. Security breaches impacting participant safety or privacy should be reported to the IRB and FDA in a timely manner.

Findings
A Use-Related Risk Analysis (URRA) is essential for identifying potential use-errors (actions or lack of action that may result in harm) and their associated use-related hazards (source of potential harm) when using an sDHT. The analysis must focus on user interactions with the sDHT, including all user groups (end-users, clinicians, carepartners, researchers, and administrators). Critical tasks are defined as those use-errors that may result in serious harm.

Recommendations
Developers should follow these five steps for the Use-Related Risk Analysis:
Describe all user tasks: Identify the sequence of actions a user performs to achieve a goal, which can be derived from sources like a task analysis or formative evaluations.
Describe potential use-errors: Identify and document potential actions or lack of actions that may result in harm for each task, noting that "use-error" is preferable to "user-error".
Describe potential use-related hazards: Determine the source of potential harm resulting from each identified use-error.
Develop a plan to minimize or eliminate known risks: The preferred approach is inherent safety by design (eliminating the error). If not feasible, use protective measures (e.g., warnings) or, as a last resort, provide instructions to users. Identify methods to evaluate the effectiveness of the chosen mitigation strategy.
Keep it up to date: The URRA is a living document requiring ongoing updates throughout the sDHT development and usability validation process.

Regulatory Considerations
The URRA is presented as a fundamental step within the V3+ framework for ensuring device usability and minimizing risk, implicitly setting the groundwork for regulatory compliance related to device safety. The minimization of use-errors, particularly for critical tasks, is a central tenet of device development best practices.

Findings
Regulation exists to ensure the safety and effectiveness of digital health products and to protect the public from potential risks. Engaging with the FDA throughout product development, even though it may seem burdensome, offers valuable benefits such as shared understanding of requirements, faster outcomes, enhanced efficiency in the review process, and built trust with regulators and the public. Working with the FDA is crucial for understanding a device's risk classification and applicable regulatory requirements.

Recommendations
Developers should follow a three-step approach for successful interaction:
EARLY: Start interacting with the agency as early as possible in development, ensuring the intended use and some basic product functionalities are defined.
OFTEN: Maintain communication, especially if new product features, design changes, or changes to how the product will be used occur, to ensure the FDA's advice remains accurate.
TRANSPARENT: Be honest and upfront about the product, evidence, testing plans, and data.
For both "non-written" (meetings) and "written" communications, best practices include:
Preparation: Define the purpose, have specific goals and questions, and prepare a well-planned meeting package (including supporting documentation and data) in advance.
Format and Tone: Select the right type of interaction for the goal, use a professional tone, and communicate clearly, concisely, and with proper formatting.
Follow-up: Respond to all FDA requests promptly and accurately, as delays can result in regulatory action.

Regulatory Considerations
Manufacturers must be familiar with and in compliance with relevant FDA guidance and regulations. Developers should present their argument for a product's regulatory category but must understand that the FDA determines the final regulatory status and obligations. It is critical to avoid providing false or misleading claims or withholding important information, as failure to cooperate or address concerns raised by the FDA can lead to penalties or failure to clear/approve the product for marketing. All communications may be subject to Freedom of Information requests and could become public

Findings
The MDR/IVDR enhance the focus on cybersecurity for devices incorporating electronic programmable systems and software. Cybersecurity risk is inherently linked to patient safety and effectiveness; manufacturers must reduce all risks, including security risks with safety impacts, to an acceptable level. The management of security risks should be integrated into the product's overall Risk Management System. Due to the rapid change in the threat landscape, security maintenance is a critical, ongoing requirement across the entire product lifecycle. Other EU legislation, such as GDPR (data protection) and the NIS Directive (network security), also apply in parallel.

Recommendations
Manufacturers must follow a "Secure by design" strategy throughout the Design and Development phase, adopting a "Defense-in-Depth strategy". This includes:
Risk Management: Conduct a Security Risk Assessment (using techniques like Threat Modelling) to identify vulnerabilities and their potential impact on safety and effectiveness.
Risk Control: Prioritize mitigating risks in this order: eliminate/reduce risks through safe design; take adequate protection measures (e.g., encryption, authentication, alarms); provide information for safety and training .
Minimum IT Requirements: Clearly set out the minimum hardware, IT network, and IT security requirements for the device's operating environment and communicate these in the Instructions for Use. Devices should be as autonomous as possible in terms of security and avoid sole reliance on the operating environment.
Vigilance: Establish a robust Post-Market Surveillance (PMS) System to actively collect information, review data, and timely implement corrective actions (e.g., security updates/patches) for security vulnerabilities and incidents throughout the device's lifespan. Manufacturers must report all serious incidents and Field Safety Corrective Actions (FSCA).

Regulatory Considerations
Manufacturers must ensure that technical documentation includes information demonstrating conformity with all general safety and performance requirements, including justification and verification/validation of security solutions. Instructions for Use must include information on residual risks related to IT security and detailed instructions for secure installation, configuration, operation, and deployment of security updates. The entire process is a continuous, iterative cycle, requiring regular updates to technical documentation, risk management, and clinical evaluation

Findings
Human Factors Engineering (HFE) and Usability Engineering (UE) are fundamental for medical device safety and effectiveness. The HFE/UE process focuses on the interactions between people and devices, considering three major components: device users, device use environments, and the device user interface. The most important goal of this process is to minimize use-related hazards and risks. The FDA's HFE requirements are derived from the Quality System Regulation (QSR), specifically relating to Design Input (needs of the user and patient) and Design Validation (conformance to defined user needs). If risk analysis shows that use errors could lead to serious harm, HFE is explicitly required and must be submitted in premarket submissions (PMA, 510(k)).

Recommendations
Manufacturers should follow HFE/UE processes throughout the device development to improve design and minimize potential use errors. This involves an iterative process that runs parallel to product development. Key steps include:
User Research: Understand the intended users (e.g., professionals, patients, lay caregivers) and their characteristics (e.g., physical, cognitive abilities, experience).
Risk Analysis: Focus on potential use errors and identify critical tasks where errors could result in serious harm.
Formative Evaluation: Conduct evaluations during development to generate ideas for test scenarios, identify dangers early, and gather input for user interface improvements.
Design for Safety: Apply the hierarchy of risk control, prioritizing inherently safe design and protective measures (alarms, warnings) over instructions and training.
Usability Validation Testing: Conduct final summative testing with representative users under simulated real-world use conditions to demonstrate the device can be used safely and effectively.

Regulatory Considerations
The FDA recommends that manufacturers submit human factors data in premarket submissions for devices where risk analysis indicates that use errors could result in serious harm. The FDA has provided guidance on the content that should be included in these submissions, such as descriptions of intended users, use environments, user interface, risk analysis of use-related hazards, and results of validation studies. Manufacturers should also continue to monitor user interactions through postmarket surveillance and adverse event reporting.