Findings
AI-enabled medical devices require robust risk assessment to address data drift, bias, and transparency challenges.
The total product lifecycle (TPLC) approach is essential for managing AI-enabled devices, ensuring continuous oversight and updates.
There is a need for improved standardization in AI model validation and performance monitoring to ensure consistency in regulatory submissions.
Effective data management practices, including dataset representativeness and bias control, are critical for AI model development.
Cybersecurity vulnerabilities in AI-enabled medical devices must be proactively addressed to prevent risks to patient safety and data integrity.

Recommendations
AI-enabled device manufacturers should integrate Good Machine Learning Practice (GMLP) principles throughout the device lifecycle.
Marketing submissions should include comprehensive documentation of AI model development, validation, and performance monitoring.
Developers should implement transparency measures, such as model interpretability and explainability, to enhance user trust and understanding.
AI models must undergo rigorous bias evaluation to ensure equitable performance across diverse patient populations.
A predetermined change control plan (PCCP) should be established to allow safe and effective AI model updates post-market without additional FDA submissions.

Regulatory Considerations
FDA encourages early engagement through the Q-Submission Program for AI-enabled device manufacturers.
Compliance with FDA-recognized consensus standards, such as ANSI/AAMI/ISO 14971 for risk management, is recommended.
AI-enabled devices must meet labeling requirements, ensuring that users clearly understand model inputs, outputs, and performance metrics.
Post-market surveillance and continuous monitoring of AI model performance are necessary to ensure ongoing safety and effectiveness.
Cybersecurity measures must be included in regulatory submissions, detailing safeguards against data breaches and unauthorized model modifications.

Findings
Cybersecurity is an integral part of medical device safety and effectiveness, and manufacturers are responsible for addressing it throughout the entire device lifecycle. The FDA considers a device's cybersecurity as part of its benefit-risk assessment for both premarket and postmarket activities. A lack of robust cybersecurity controls can lead to patient harm, compromised device functionality, and breaches of data privacy. The dynamic nature of cybersecurity threats requires ongoing monitoring, risk management, and timely implementation of mitigation strategies.

Recommendations
Manufacturers should build cybersecurity into devices from the design phase ("secure by design") and conduct a thorough risk analysis to identify and mitigate potential vulnerabilities. Premarket submissions should include comprehensive documentation of the device's cybersecurity controls, a risk management plan, and a plan for postmarket surveillance and response. Manufacturers should establish a robust postmarket surveillance program to monitor for, identify, and address new cybersecurity threats in a timely manner. Clear and informative labeling is essential to help users understand and manage cybersecurity risks.

Regulatory Considerations
The FDA has the authority to take action against devices with inadequate cybersecurity that pose a risk to public health. The agency recommends that manufacturers use the Q-submission process to discuss specific cybersecurity questions related to their device submissions. Compliance with recognized standards and best practices for cybersecurity is strongly encouraged. Manufacturers must report certain cybersecurity incidents to the FDA as part of their postmarket reporting requirements. The FDA collaborates with other government agencies and stakeholders to promote a coordinated approach to medical device cybersecurity.

Findings
The goal of sDHT design is to create tools that are functional, intuitive, accessible, and enjoyable to use, moving beyond merely minimizing use-errors. Human-centered design (HCD) is the preferred term over user-centered design, emphasizing the impact on many user groups beyond just the end-users. "Users" encompass end-users (patients/participants), carepartners, clinicians, investigators, and administrators.

Recommendations
Developers of sDHTs should adhere to the following HCD principles:
Empathetic: Take time to deeply understand users' needs, behaviors, and emotions, capturing this in the use specification.
Holistic: Consider the entire end-to-end user journey, including hardware, software, accessories, packaging, instructions for use, and training.
Iterative: Employ an iterative approach to designing, prototyping, testing, and refining, using formative evaluations to identify use-errors and gather usability data, capturing this in the use-related risk analysis.
User-centric: Improve usability by capturing user feedback in real-world settings, gradually recruiting larger, more diverse samples that represent the intended use population.
Inclusive: Collaborate with individuals representing all user groups by hiring them as consultants or creating user advisory panels to influence design decisions (co-design).
Multidisciplinary: Ensure the development team includes colleagues from various disciplines to bring diverse perspectives and innovative solutions.

Regulatory Considerations
The document ties the HCD process to risk management and eventual validation by recommending that findings from formative evaluations (used to identify use-errors) be captured in a use-related risk analysis. The approach aligns with the principles of the overarching V3+ framework.

Findings
Usability is a multi-domain concept that requires a combination of methods for evaluation. Evaluations fall into two types: formative (for design modification of prototypes) and summative (for demonstrating usability of the final product to a representative user sample). The user experience metrics fall into several domains, including: Satisfaction, Usefulness, Ease of use, Learnability, Efficiency, Memorability, Understandability, Actionability, Readability, and Use-errors. Metrics related to Satisfaction and Usefulness are always subjectively reported by users.

Recommendations
Developers should select metrics based on the specific usability-related domain being evaluated.
Subjective Data (e.g., Satisfaction, Usefulness): Capture through qualitative surveys, quantitative surveys (scales), interviews, focus groups, and think-aloud evaluations .
Objective Data (e.g., Ease of use, Use-errors): Capture through direct or indirect observation (e.g., counting steps/attempts, timing task completion), or by using data generated by the sDHT (e.g., error reports, timestamps, page load times).
Time-based Metrics: Evaluate Learnability (ease of first use), Efficiency (ease with experience), and Memorability (ease after non-use) by measuring ease of use at different points in time .
Information Presentation: If the sDHT presents clinical data or written information (instructions, warnings), evaluate Understandability, Actionability, and Readability .
Use-errors: Objectively capture the number, type, and recoverability of use-errors (actions, or lack thereof, that may result in harm) via observation and sDHT data, noting that "use-error" is preferred to "user-error".

Regulatory Considerations
While this guide does not reference regulatory bodies like the FDA, it is part of the V3+ framework and recommends that researchers prioritize essential documents like the use specification and use-related risk analysis before designing a usability study. Summative evaluations demonstrating usability against a representative user sample under intended use conditions are the standard for demonstrating product fitness.

Findings
For an sDHT to be considered fit-for-purpose, a researcher or healthcare provider must understand the alignment between the sDHT's intended use (What it does, who uses it, where/when/how) and their own context of use . Key information for this assessment comes from the developer's Use Specification (detailing hardware, software, accessories, training) and Use-Related Risk Analysis (detailing warnings, harms from use-errors, and risk avoidance) . Usability validation evidence should cover study objectives, protocols, participant characteristics, metrics, and collection methods.

Recommendations
Researchers/providers should use the checklist to:
The Basics: Compare the sDHT's intended use to their context of use; if there is substantial overlap, existing evidence may be sufficient.
Use Specification/Risk Analysis: Gather detailed descriptions of the sDHT's hardware, software, accessories, written materials, training, cautions, warnings, and potential harms from use-errors to update their own Use Specification and Use-Related Risk Analysis .
Existing Evidence: Access existing usability validation study results (objectives, methods, participant characteristics, metrics, etc.) to determine its applicability and generalizability to their context of use .
Collaboration: Consider establishing a collaborative relationship with the developer to provide feedback for next-generation sDHTs, ensure version control, and potentially collaborate on future usability validation studies .

Regulatory Considerations
The document notes that if the sDHT is a regulated medical device, the intended use statement should already capture the answers to the basic questions. The entire checklist is framed around the V3+ framework, which is designed to ensure the rigor necessary for a product to be considered fit-for-purpose by all stakeholders.

Findings
The Library of Human Factors Resources compiles external documents related to human factors, human-centered design, and usability in the context of Digital Health Technologies (DHTs), especially sensor-based DHTs (sDHTs). The resources in the library are categorized by usability validation activity:
Use-related risk assessment (topics include user tasks, use-errors, use-related hazards, and risk mitigation).
Design considerations (topics relate to product design).
Formative evaluation (topics related to usability evaluation of a prototype product).
Summative evaluation (topics related to usability evaluation of a final-version or marketed product).

Recommendations
The page recommends that users, developers, and evaluators utilize the library's interactive index to find relevant documents to their DHT:
Use the Search by Activity tab to filter by one of the usability validation activities listed above.
Use the human factors topic column to further narrow the search.
Review the document name, issuing body, and product of focus columns to identify the most relevant document.
Click the hyperlink to access the selected document, then use the information in the relevant sections column to locate the specific topics of interest.

Regulatory Considerations
The library is a collection of resources for accelerating sDHT adoption, and it specifically includes documents like regulatory guidance and industry standards. This emphasizes the importance of understanding and applying human factors principles for validation and achieving regulatory acceptance for sDHTs.

Findings
The V3+ Use Specification must contain a detailed description of the user groups, use environments, and the sDHT user interface. The user groups include end-users (individuals from whom data is captured) as well as carepartners, clinicians, researchers, and administrators. Characteristics of users (e.g., demographics, literacy, physical/cognitive capabilities, disease characteristics) and use environments (e.g., temperature, network availability, clutter) must be considered for risk management .

Recommendations
Developers must follow these four steps to create the Use Specification:
Identify all user groups: Create a list of users, including sub-categories (e.g., different types of researchers), and describe the characteristics of each group (e.g., health literacy, physical capabilities) to create detailed descriptions of representative users.
Identify all likely use environments: Create a list of typical environments (e.g., Home, Hospitals) and describe their characteristics (e.g., temperature, noise, network availability), also considering "edge cases" (e.g., extreme weather).
Describe the sDHT user interface: Detail all aspects of the hardware and software (visual, auditory, tactile cues), accessories (e.g., packaging, chargers), and all written materials and training (e.g., instructions for use, helpdesk troubleshooting).
Keep it up to date: The Use Specification is a living document that requires ongoing updates and maintenance throughout the sDHT development and usability validation process.

Regulatory Considerations
The development of the Use Specification is presented as the foundational step for the usability validation component of the V3+ framework. This document directly informs the subsequent Use-Related Risk Analysis.

Findings
A Use-Related Risk Analysis (URRA) is essential for identifying potential use-errors (actions or lack of action that may result in harm) and their associated use-related hazards (source of potential harm) when using an sDHT. The analysis must focus on user interactions with the sDHT, including all user groups (end-users, clinicians, carepartners, researchers, and administrators). Critical tasks are defined as those use-errors that may result in serious harm.

Recommendations
Developers should follow these five steps for the Use-Related Risk Analysis:
Describe all user tasks: Identify the sequence of actions a user performs to achieve a goal, which can be derived from sources like a task analysis or formative evaluations.
Describe potential use-errors: Identify and document potential actions or lack of actions that may result in harm for each task, noting that "use-error" is preferable to "user-error".
Describe potential use-related hazards: Determine the source of potential harm resulting from each identified use-error.
Develop a plan to minimize or eliminate known risks: The preferred approach is inherent safety by design (eliminating the error). If not feasible, use protective measures (e.g., warnings) or, as a last resort, provide instructions to users. Identify methods to evaluate the effectiveness of the chosen mitigation strategy.
Keep it up to date: The URRA is a living document requiring ongoing updates throughout the sDHT development and usability validation process.

Regulatory Considerations
The URRA is presented as a fundamental step within the V3+ framework for ensuring device usability and minimizing risk, implicitly setting the groundwork for regulatory compliance related to device safety. The minimization of use-errors, particularly for critical tasks, is a central tenet of device development best practices.

Findings
Cybersecurity threats in healthcare are increasingly frequent and severe, posing risks to device safety and clinical care.
Many vulnerabilities arise from third-party software components and interconnected device ecosystems.
Legacy devices often lack adequate cybersecurity controls, leading to increased patient and organizational risks.
Cybersecurity risk management processes must integrate safety and security assessments throughout the device lifecycle.
Transparency in device cybersecurity is crucial for enabling safe integration and use by healthcare providers and end users.

Recommendations
Implement a Secure Product Development Framework (SPDF) for comprehensive cybersecurity throughout the product lifecycle.
Include a Software Bill of Materials (SBOM) for all premarket submissions to track software dependencies and vulnerabilities.
Perform robust cybersecurity testing, including penetration testing and vulnerability assessments.
Enhance device labeling with clear cybersecurity-related instructions and risks for users.
Develop a coordinated vulnerability disclosure plan for postmarket cybersecurity management.

Regulatory Considerations
Adherence to 21 CFR Part 820 Quality System regulation requirements, including design controls and risk management.
Compliance with Section 524B of the FD&C Act for cybersecurity of cyber devices.
Submission of SBOMs and detailed security risk management reports for premarket applications.
Provision of cybersecurity information as part of device labeling to prevent misbranding under Section 502 of the FD&C Act.
Integration of security testing and validation as part of the FDA review process.

Findings
OTS software introduces unique risks due to its general-purpose design and lack of lifecycle control by medical device manufacturers.
Comprehensive testing and risk management are essential to mitigate safety hazards associated with OTS software in medical devices.
Regular updates and maintenance are critical for managing obsolescence and ensuring long-term safety and effectiveness of OTS components.
Networking and interoperability of OTS software pose additional risks related to data integrity, cybersecurity, and scalability.
Enhanced documentation is required for high-risk devices incorporating OTS software, especially those involving AI or ML functionalities.

Recommendations
Provide comprehensive descriptions of OTS software, including version details and system specifications.
Conduct thorough risk assessments and include mitigation plans in premarket submissions.
Perform rigorous testing, including integration and regression testing, for OTS software components.
Establish mechanisms for continued maintenance, support, and version control of OTS software.
Ensure that device labeling includes warnings and specifications related to OTS software compatibility and restrictions.

Regulatory Considerations
Adherence to 21 CFR Part 820 Quality System regulations, including design controls and purchasing controls for OTS software.
Submission of a risk management file and traceability documentation linking risks, design requirements, and testing outcomes.
Compliance with premarket submission requirements, including 510(k), IDE, and PMA applications, as applicable.
Use of device labeling to communicate hardware and software compatibility and restrictions to users.
Development of beta testing and investigational plans for clinical studies involving OTS software.

Findings
Pre-Submissions (Pre-Subs) allow submitters to obtain FDA feedback on specific questions before submitting formal IDEs, 510(k)s, PMAs, or other applications. Early feedback can improve submission quality and streamline the review process.
Submission Issue Requests (SIRs) provide a mechanism for addressing issues raised in FDA hold letters (e.g., 510(k) deficiencies) to help expedite resolutions.
Study Risk Determinations help sponsors clarify whether clinical studies are significant risk (SR), non-significant risk (NSR), or exempt from IDE regulations.
Informational Meetings are non-feedback sessions aimed at familiarizing FDA staff with new devices or sharing updates on ongoing development.
The program encourages timely submissions, including supplements for ongoing discussions and amendments to update materials.

Recommendations
Clearly define the purpose and goals of the Q-Sub in the submission to facilitate effective FDA review.
Include specific, well-formulated questions that focus on a limited number of topics to ensure actionable feedback.
For Pre-Subs, align planned testing and submissions with FDA guidance and include detailed device descriptions, testing protocols, and relevant background information.
Use SIRs to discuss proposed solutions to deficiencies raised in FDA hold letters, focusing on timely resolution.
Draft and submit meeting minutes promptly (within 15 days of meetings) to ensure accurate documentation of FDA feedback.

Regulatory Considerations
Submitters should adhere to the timelines specified for different Q-Sub types, including 70 days for Pre-Sub feedback or 21 days for SIRs submitted promptly after a hold letter.
Q-Subs should include all relevant regulatory history and references to prior FDA communications to streamline the review process.
FDA feedback through the Q-Sub program is non-binding and based on the information available at the time; subsequent submissions must align with the provided feedback to maintain consistency.
Informational Meeting requests should clearly state that feedback is not expected and may be used to track interactions outside other formal Q-Sub types.
Confidentiality of Q-Subs is maintained in compliance with FDA’s disclosure regulations and the Freedom of Information Act (FOIA).

Findings
There is a notable lack of reports on the use of digital health technology in pediatric patients.
Challenges exist in selecting the design, metrics, and types of sensors best suited for disease evaluation.
False positive detection remains problematic in seizure detection using DHTs.
There is a lack of information on the use of DHTs in infants.
Unique design challenges for pediatric DHTs arise from size, anatomy, physiology, activity levels, and cognitive development.

Recommendations
Further research and evaluation are needed to realize the full potential of remote monitoring in pediatric trials.
Creative approaches, including machine learning, should be employed to identify optimal measurement methods.
Training for caregivers is necessary to ensure DHTs are worn correctly and data are complete.

Regulatory Considerations
Confirming the reliability and clinical relevance of DHT measurements is essential.
Ensuring privacy and confidentiality of patient data must be prioritized.