mA u s FOOD DRUG ADMINISTRATION Developing Software Precertification Program A Working Model v0 2 June 2018 Contents 1 Building the Program with Continuous Public Input 3 2 Release Notes 4 3 Introduction 5 4 5 6 3 1 Program Goal 6 3 2 Software Precertification Program Vision 6 3 3 Scope 8 3 4 Program Overview 10 3 5 Outline 11 Excellence Appraisal and Precertification Component 1 12 4 1 Eligibility 13 4 2 Pre-Cert Application 13 4 3 Appraisal 14 4 4 Examples 16 4 5 Precertification Levels 18 4 6 Maintenance and Monitoring of Pre-Cert Status 19 Review Pathway Determination Component 2 20 5 1 Risk Categorization 20 5 2 Product level elements of a SaMD 21 5 3 Determining SaMD Risk 22 Streamlined Premarket Review Process Component 3 25 6 1 Option for an iterative early engagement Streamlined Review Process for Premarket Notification for Pre-Cert 1 0 26 6 2 Determining elements necessary for assuring safety and effectiveness in premarket review 27 7 Real-World Performance Component 4 29 7 1 Framework for Use of RWPA 30 8 Next Steps and Public Engagement 32 9 Appendix A – Clarification of IMDRF Definition of Software as a Medical Device SaMD 33 10 Appendix B – Proposed Organizational Elements to Demonstrate Excellence Principles 36 11 Appendix C – Possible framework for conducting an iterative early engagement streamlined review for a premarket notification of a software product from a precertified organization 42 12 Appendix D – Real-World Performance Analytics for Product Monitoring 43 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 2 FDA gov 1 Building the Program with Continuous Public Input As part of the Software Precertification Program development process FDA has been providing opportunities to the public to provide input on the program elements by publishing incremental versions of the working model of the program FDA is using this transparent and open approach to provide continuous notice and solicitation of public input by means of an open public docket throughout the program development The public docket is available at https www regulations gov comment D FDA-2017-N-4301-0001 FDA intends to continue reviewing the public docket approximately every two weeks and to refine this program by incorporating as appropriate comments in future versions of the working model We encourage the public to provide feedback early and often I C We request comments on each working model iteration no later than 30 days after the version posting Each iteration of the working model will include release notes detailing changes made since the last version and will highlight specific topic areas where the FDA is seeking public input The challenge questions proposed in the initial working model version 0 1 remain in a separate document 1 and continue to serve as prompts for public consideration when providing feedback on focused areas of development for the Software Precertification Program In this version of the working model FDA has highlighted requests for public input using the large “I” for “input” indicated here Portions of the working model that have been modified in response to comments received in the public docket through May 31 2018 are highlighted using the large “C” for “comment” indicated here FDA continues to evaluate the detailed responses we have received on version 0 1 future iterations of the working model will incorporate this input as appropriate We intend to use this collaborative process to develop version 1 0 of the program by December 2018 with the goal that it will be available for pilot testing within FDA’s current authorities in 2019 We will consider appropriate mechanisms for establishing the program within FDA's current statutory and regulatory authorities Version 0 1 Challenge Questions available at https www fda gov downloads MedicalDevices DigitalHealth DigitalHealthPreCertProgram UCM605686 pdf 1 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 3 FDA gov 2 Release Notes 20180619 v0 2 - Introduction revised to include the vision for the program - Scope of precertification program restated to clarify who is eligible for precertification and which product type review pathways are the focus for development for 2019 - Outline of Program Components includes description of component interdependencies - Component 1 Excellence Appraisal o Further refinement of objectives and goals o New figure describing a conceptual framework for excellence appraisal o New proposed elements and organizational practice domains for the demonstration of excellence for precertification o Updated description of proposed appraisal process o New examples of activities processes and Key Performance Indicators for two example elements o Revisions to descriptions for Levels of Pre-Cert - Component 2 Review Pathway Determination o Further refinement of objectives and goals o Further details for leveraging IMDRF framework o New identification of product level elements - Component 3 Streamlined Premarket Review Process o Clarification of expectations for products entering a streamlined review process o New proposed option for an iterative early engagement review process o New proposal for possible review elements - Component 4 Real-World Performance o New description of benefits for monitoring product-level real-world performance o Refinement of terminology definitions focus on types of analytics rather than data o New elements of real-world performance analytics for post-launch product monitoring 20180426 v0 1 - First version of working model FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 4 FDA gov 3 Introduction The Software Precertification Program is envisioned as a voluntary pathway that embodies a regulatory model more tailored than the current regulatory paradigm to assess the safety and effectiveness of software technologies without inhibiting patient access to these technologies The program goal is to provide more streamlined and efficient regulatory oversight of softwarebased medical devices from manufacturers who have demonstrated a robust culture of quality and organizational excellence CQOE and are committed to monitoring real-world performance Software is increasingly used in healthcare to treat and diagnose disease aid clinical decision making and manage patient care The ability to download these software programs onto ubiquitously connected mobile platforms allows them to be used in the hospital and in the home by clinicians and patients Historically healthcare has been slow to implement technology tools that have transformed other areas of commerce and daily life One factor that has been cited among many is the regulation that accompanies medical products But momentum toward a digital future in healthcare is advancing FDA oversees most mobile apps that are intended to treat diagnose cure mitigate or prevent disease or other conditions as medical devices under federal law These software-based technologies including mobile medical apps are what FDA and other regulators call “Software as a Medical Device” SaMD FDA’s traditional approach for the regulation of hardware-based medical devices is not wellsuited for the faster iterative design and development and type of validation used for software device functions including SaMD Software products offer unique opportunities such as addressing malfunctions quickly and efficiently to reduce adverse events understanding and capturing patient performance outside of the clinical setting and enabling patient engagement Unlike manufacturers of hardware devices who modify their products every few months to years developers of software modify their products in response to real-world performance and user feedback every few weeks to months Evaluating software code alone may not provide a full understanding of the safety and effectiveness of a SaMD product in part because the impact on patients is often indirect As a result the application of FDA’s longstanding regulatory framework to software can impede access to new and improved software-based medical products An agile regulatory paradigm is necessary to accommodate the faster rate of development and potential for innovation in software-based products It is important for public health to address these distinctive aspects of digital health technology -- its clinical promise unique user interface ability to facilitate patient engagement with the developer and compressed commercial cycle of new product introductions – while ensuring that existing standards of safety and effectiveness are met or exceeded To address these challenges in July 2017 FDA announced the Software Precertification Pilot Program to develop a new regulatory paradigm that would focus first on the assessment of organizations that perform high-quality software design testing and monitoring This proposed approach is based on demonstration of a culture of quality and organizational excellence CQOE and a commitment to monitoring product performance Because SaMD products can be adapted to respond to glitches adverse events and other safety concerns quickly FDA is working to establish a regulatory framework that would allow efficient responses to software issues and thus continue to ensure that consumers have access to safe and effective products The Software Precertification Program is envisioned to evaluate a firm’s capability to respond to FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 5 FDA gov real-world performance and FDA intends to work with precertified firms to quickly and effectively address software issues C 3 1 Program Goal The following clarification on applicability of the Software Precertification Program to organizations of all sizes has been made in response to comments received in the public docket The goal of the program is to have tailored pragmatic and least burdensome regulatory oversight that assesses organizations large and small to establish trust that they have a culture of quality and organizational excellence such that they can develop high quality SaMD products leverages transparency of organizational excellence and product performance across the entire lifecycle of SaMD uses a tailored streamlined premarket review and leverages unique postmarket opportunities available in software to verify the continued safety effectiveness and performance of SaMD in the real world The Software Precertification Program is intended to build stakeholder confidence that participating organizations have demonstrated capabilities to build test monitor and proactively maintain and improve the safety efficacy performance and security of their medical device software products so that they meet or exceed existing FDA standards of safety and effectiveness 3 2 Software Precertification Program Vision The program aims to design a new approach for software products a Precertification Program for the assessment of companies that perform high-quality software design and testing Under this program software developers would be assessed by FDA or an accredited third party for the quality of their software design testing clinical practices real-world performance monitoring and other appropriate capabilities to qualify for a more streamlined premarket review while better leveraging postmarket data collection on the device’s safety and effectiveness This new organization-based approach enhances the ability to assure the safety and effectiveness of software products by using the precertification framework in addition to aspects of the Agency’s traditional reliance on individual product-based oversight This program is intended to extend beyond consideration of organizations’ traditional Quality Management Systems and incorporate recognition of excellence in clinical responsibility and cybersecurity practices We believe that this approach would provide patient access to critical evolutions of software technology The software products from precertified companies would continue to meet the same statutory standards as software products that have followed the traditional path to market Precertification is simply one pathway or method to establish that a product provides reasonable assurance of safety and effectiveness This approach is intended to enable more efficient and streamlined oversight without compromising safety and effectiveness of medical device software products C The following clarification on applicability of the Software Precertification Program to organizations of all sizes has been made in response to comments received in the public docket FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 6 FDA gov C There are many innovators that are ready to solve healthcare challenges and are willing to bring unique skills approaches and solutions to solve patient needs We believe that providing a clear framework and expectations would empower these innovators to bring solutions to patients and users This process should enable patients and healthcare providers to have high confidence in precertified companies and the devices they produce because precertified organizations leverage real-world performance to continuously monitor and improve upon the safety and effectiveness of marketed SaMD products The vision for the program is to be available for any size organization who is currently developing products in healthcare including medical devices or has the potential to deliver products that are medical devices By establishing clear organizational excellence expectations and clear regulatory expectations and leveraging real-world performance this program intends to create a regulatory environment that would enable patients and healthcare providers to have timely access to technologies that are built by excellent organizations The following reiteration of FDA’s vision for flexibility within the Software Precertification Program has been made in response to comments received in the public docket FDA recognizes that the underlying principles of the excellence appraisal need to be consistently interpreted and applied across industry However we currently believe that there should be flexibility in the specific mechanisms by which excellence can be demonstrated An organization has flexibility to show how its processes systems and measures of performance track to program’s specified elements performance measures and ultimately the excellence principles FDA anticipates many benefits for various stakeholders We envision that this program would align the device review process with the software development process enabling faster patient access to technologies The program would also establish a consistent appraisal process so that manufacturers know what to expect for software evaluation FDA also recognizes the need for transparency so that end users of these products from precertified companies can understand the premarket review and postmarket monitoring conducted for these products We anticipate that the evidence and insights gleaned from the precertification process including a commitment to robust postmarket oversight would support a streamlined regulatory review paradigm Table 1 below shows anticipated benefits for various stakeholders FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 7 FDA gov Table 1 Example of Anticipated Program Benefits Enhanced confidence in organizations developing SaMD products Improved quality safety proactivity to address known and emerging risks Timely availability of solutions to patients Enhanced regulatory simplicity and experience Business simplicity faster timely market access End user Business FDA Payor Investor Patients Providers Caregivers SaMD Developer Agency Reviewer Insurance Provider Venture Capitalist 3 3 Scope Organizations that are developing or planning to develop a software that could be subject to FDA oversight are included within the scope of the Software Precertification Program Ultimately the product types that may benefit from precertification might include all software that meets the definition of a device in section 201 h 2 of the Federal Food Drug and Cosmetic Act FD C Act including SaMD software in a medical device SiMD and other software that could be considered accessories 3 to hardware medical devices However in developing Version 1 0 of the program the current focus is to establish processes for SaMD technologies which may include software functions that use artificial intelligence and machine learning algorithms C The following clarification on the relationship of the Software Precertification Program and software functions that do not meet the device definition in 201 h of the FD C Act has been made in response to comments received in the public docket The term device does not include software functions excluded pursuant to section 520 o of the FD C Act as amended by the 21st Century Cures Act 3 An accessory is a finished device that is intended to support supplement and or augment the performance of one or more parent devices See Medical Device Accessories – Describing Accessories and Classification Pathways Guidance for Industry and FDA Staff available at https www fda gov downloads MedicalDevices DeviceRegulationandGuidance GuidanceDocuments UCM42967 2 pdf 2 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 8 FDA gov Non-device software functions are not subject to regulation and are not within the scope of the Software Precertification Program In particular software functions intended 1 for administrative support of a health care facility 2 for maintaining or encouraging a healthy lifestyle 3 to serve as electronic patient records 4 for transferring storing converting formats or displaying data or 5 to provide certain limited clinical decision support are not medical devices 4 and are not subject to FDA regulation Current policies for the review of software device functions continue to apply C The following clarification on SaMD has been added in response to comments received in the public docket See boxed definition statement and Figure 1 for a description of SaMD See Appendix A for further clarification on what is considered SaMD The program scope has been limited to SaMD for Version 1 0 in order to allow FDA to gain experience in the precertification process As FDA leverages insights from implementation of Version 1 0 we hope to expand the program to be able to leverage a software manufacturer’s precertification status to the review of all medical device software products During the pilot testing in 2019 if a pilot participant wishes to market a medical device software product that is currently beyond the scope of Version 1 0 of the program FDA might work within the pilot to consider as a test case whether the software can be reviewed by leveraging precertification in existing review pathways as appropriate “Software as a Medical Device” SaMD is defined as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device 5 Figure 1 Description of SaMD including possible data sources from which inputs are derived and that may be used for one or more medical purposes Section 520 o 1 of the FD C Act 21 U S C 360j o 1 A - D as added by Section 3060 a of the 21st Century Cures Act December 13 2016 5 http www imdrf org docs imdrf final technical imdrf-tech-131209-samd-key-definitions-140901 docx 4 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 9 FDA gov 3 4 Program Overview The program concept is based upon precertification of software manufacturers who have demonstrated a culture of quality and organizational excellence and would leverage data from all appropriate sources FDA would evaluate organizational excellence based on five culture of quality and organizational excellence CQOE principles hereafter referred to as “excellence principles” Product Quality – Demonstration of excellence in the development testing and maintenance necessary to deliver SaMD products at the highest level of quality Patient Safety – Demonstration of excellence in providing a safe patient experience and emphasizing patient safety as a critical factor in all decisionmaking processes Clinical Responsibility – Demonstration of excellence in responsibly conducting clinical evaluation and ensuring that patient-centric issues including labeling and human factors are appropriately addressed Cybersecurity Responsibility – Demonstration of excellence in protecting cybersecurity and proactively addressing cybersecurity issues through active engagement with stakeholders and peers Proactive Culture – Demonstration of excellence in a proactive approach to surveillance assessment of user needs and continuous learning Leveraging the data gleaned from the precertification process FDA would seek to adopt a riskbased streamlined regulatory approach to SaMD review to either replace the need for a premarket submission or for higher risk products to allow for streamlined premarket review that maximizes efficiency and engagement The premarket review determination would apply the least burdensome principles of premarket-postmarket balance by leveraging real-world performance data Similar to FDA’s current regulatory system under which not all devices require premarket review e g 510 k -exempt devices this program envisions exemptions from premarket review for lower risk SaMD products or faster review of higher risk SaMD products that are developed delivered and maintained by precertified organizations In addition to demonstrating excellence as established through the five excellence principles precertified organizations would also have a robust mechanism to collect monitor and analyze real-world performance of their organization and the products they deliver FDA also intends to bolster postmarket monitoring by more effectively leveraging real-world data from device registries and other electronic health information sources The collection of real-world performance data on precertified organizations’ SaMD products is anticipated to enable improvements of the Software Precertification Program itself FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 10 FDA gov 3 5 Outline To deliver the goals of the program as described above we have divided the program into four key program components depicted below in Figure 2 • • • • Excellence Appraisal and Precertification Component 1 Review Pathway Determination Component 2 Streamlined Premarket Review Process Component 3 and Real-World Performance Component 4 Figure 2 Software Precertification Program Components The four components of the Software Precertification Program are interdependent and are intended to be part of a comprehensive Precertification Program Excellence appraisal leverages an organization’s demonstration of their commitment to and ability for developing testing and managing software throughout a product’s lifecycle which provides confidence in the products made and marketed by the organization When fully developed this program would provide the logic and parameters to allow precertified organizations to assess if their products may be eligible for streamlined review or if their precertification status may replace the need for a premarket submission Review determination depends not only on the risk of the medical device software product but also on the results of the excellence appraisal for an organization The review of a precertified organization’s software product would rely on the excellence appraisal and a commitment to real-world performance monitoring to proactively manage and continually assure product safety and effectiveness FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 11 FDA gov 4 Excellence Appraisal and Precertification Component 1 Organizations that produce safer and more effective device software products not only “do the right things ” but they also “do the things right” based on evidence that informs better decisionmaking While governance and processes may differ these organizations tend to follow a common set of goals objectives and approaches across the product lifecycle The FDA believes organizations committed to these principles can show through their existing processes and activities a demonstration of the excellence principles for the purposes of precertification C The following clarifications on applicability of the Software Precertifcation Program to organizations of all sizes affirmation of applying least burdensome approaches and leveraging of existing standards have been made in response to comments received in the public docket The excellence appraisal has identified the following development principles Designed for organizations of all sizes Allows organizations to demonstrate excellence based on outcomes achieved by their unique processes operations and capabilities Applies least burdensome approach by observing organizations current processes Recognizes organizations following existing standards e g QSR ISO 13485 ISO 12207 ISO 62304 ISO 14971 ISO 9001 and outcomes achieved by following those processes • • • • The principal objective of the excellence appraisal and precertification component is to develop the process of precertification and the elements necessary for the excellence appraisal process including • Eligibility identifying characteristics of an organization to participate in precertification • Pre-Cert Application identifying the elements necessary and the process of requesting appraisal for precertification • Appraisal identifying reference “domains” and “elements” necessary for the process of collecting observing an organizations’ information for Pre-Cert determination • Pre-Cert Status Determination identifying the method and process of aggregating and analyzing appraisal results to excellence principles to determine Pre-Cert level • Maintenance and Monitoring of Pre-Cert Status identifying the processes and mechanisms for an organization to monitor and maintain Pre-Cert status be transparent with all stakeholders and engage with FDA Figure 3 shows a conceptual framework for excellence appraisal that begins to identify key elements necessary for the appraisal processes and the Pre-Cert determination FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 12 FDA gov Figure 3 Conceptual Framework for Excellence Appraisal 4 1 Eligibility Any organization that intends to develop or market regulated software in the United States would be considered in-scope for the Software Precertification Program This could include organizations that are developing SaMD and organizations that are planning to develop SaMD FDA recognizes the potential for significant variability in the culture and internal processes of different business units within a single organization particularly for large organizations that are multinational or include multiple business units C The following flexibility for the eligibility of the business unit certified under the Software Precertification Program has been added in response to comments received in the public docket As part of determining eligibility the proposed program would allow companies to identify the boundaries of the organization themselves to determine the business unit or center of excellence that should be considered for precertification The boundaries of a “business unit” should be clearly determined by the company itself prior to participating in the precertification process 4 2 Pre-Cert Application The FDA anticipates the Pre-Cert application would describe the business unit or center of excellence boundaries as well as the organization’s portfolio of software products FDA is continuing to develop the details of other information that should be included in such application for an appraisal and subsequent precertification FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 13 FDA gov 4 3 Appraisal C C The following incorporation of the International Medical Device Regulators Forum IMDRF N23 document titled Software as a Medical Device SaMD Application of Quality Management System has been made in response to comments received in the public docket The appraisal process seeks to understand how the organization’s processes and measures are used effectively in turn to demonstrate the organization’s excellence in the five principles Product Quality Patient Safety Clinical Responsibility Cybersecurity Responsibility Proactive Culture Software development methodologies processes and practices differ between and within organizations as well as organizational structure In recognition of these differences the Pre-Cert Program outlines discrete elements that have been identified as leading to safer and more effective SaMD The elements are grouped into domains taken from the IMDRF N23 Software as a Medical Device SaMD Application of Quality Management System The appraisal process is envisioned to assess whether an organization has demonstrated excellence in product development that can be leveraged to provide reasonable assurance of safety and effectiveness of the organization’s SaMD products The elements and domains ultimately map to the Excellence Principles however during the appraisal process an organization seeking precertification is expected to demonstrate how its specific processes are aligned and objectively managed for the identified elements A full list of elements and domains mapped to Excellence Principles they support can be found in Appendix B The incorporation of the following tenets including transparency as elements of the Excellence Principles has been made in response to comments received in the public docket 1 Leadership and Organizational Support – Elements related to the organization’s leadership establishing the strategic direction responsibility authority and communication to assure the safe and effective performance of the SaMD 2 Transparency – Elements related to the organization’s open sharing of relevant information with all stakeholders to build confidence in the organization and its products 3 People – Elements related to providing appropriate resources as needed for ensuring the effectiveness across all lifecycle processes and activities in meeting user requirements 4 Infrastructure and Work Environment – Elements related to the availability of infrastructure such as equipment information communication networks tools and the physical facility etc throughout SaMD lifecycle processes 5 Risk Management A Patient Safety Focus – Elements related to monitoring and managing risks along multiple dimensions such as user based application based device-based use environment based and security based across all lifecycle processes 6 Configuration Management and Change Control – Elements related to identifying and defining the software configuration and controlling the release and change of the software throughout all lifecycle processes FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 14 FDA gov 7 Measurement Analysis and Continuous Improvement of Processes and Products – Elements related to managing and improving product realization and use through realworld performance monitoring 8 Managing Outsourced Processes Activities and Products – Elements related to understanding maintaining control and managing the effect of outsourced activities processes or products 9 Requirements Management – Elements related to clear and often repeated user interaction to understand and clearly articulate user needs throughout all lifecycle processes 10 Design and Development – Elements related to ensuring safe effective and secure SaMD based on user and other performance requirements during all lifecycle phases at key milestones and good development practices incorporating appropriate review activities such as code review peer review and self-review 11 Verification and Validation – Elements related to understanding the criticality and impact to patient safety by providing assurance of conformity to requirements and reasonable confidence that the software meets its intended use user needs and operational requirements I 12 Deployment and Maintenance – Elements related to activities such as delivery installation setup and configuration of software including documentation and user training materials that identify any limitation of the algorithm provenance of data used assumptions made etc that should be considered during deployment Additionally modification of previously deployed software while preserving the integrity of the software by not introducing new safety effectiveness performance and security hazards The elements and domains are proposed in this version of the working model for public review and input including whether the proposed elements appear to reflect the stated intent of the program and the five excellence principles to look holistically at and beyond software design and development and whether any elements should be considered for addition We further request public input on which elements are likely the most impactful to provide confidence that an organization makes high quality products for example you might provide your views on the top ten elements that are most impactful Additionally we seek comment on how to further clarify these elements and the associated domains to provide a least burdensome approach for software organizations to identify their processes activities and outcomes We also seek comments on elements or domains critical to evaluating the development of software functions using artificial intelligence and machine learning algorithms Appraisal Process While the appraisal method is not fully developed we generally intend to evaluate organizational elements based on objective observable evidence Each organization would determine which processes activities and Key Performance Indicators KPIs best meet these elements We recognize that means of demonstrating excellence in product development elements can vary across types and sizes of organizations In addition to having KPIs in place excellent organizations will typically assign action threshold levels and target time frames to help measure performance along the way FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 15 FDA gov The FDA’s current thinking is that organizations would provide this evidence as a part of the application or appraisal process which may be followed up by site visits interviews or other methods The program envisions that ultimately using automation or tapping into an organization’s metric systems could reduce appraisal burden increase confidence in organizations and enhance the capability to respond quickly and improve products without reducing confidence in or the reliability of the program As a part of the appraisal process organizations would describe how their practices and activities fulfill each element and how they measure their output as well as provide their measurements or KPIs targets thresholds and trends Development and tracking of KPIs can help an organization monitor improve and demonstrate performance as well as inform key organizational decisions such as product release readiness KPIs can be at an organizational level group level and or project or product level As part of the objectives of the excellence appraisal or re-appraisal for precertification it is important to assess organizational KPIs as well as post-market product performance KPIs identified as part of the premarket excellence appraisal of the organization would drive the postmarket product-specific evaluation based on real-world performance data 4 4 Examples The following table includes examples of activities processes and KPIs related to two elements Table 2 Examples of activities processes and KPIs and metrics related to two elements Domain Measurement Analysis and Improvement of Processes and Products Element Analyzing and providing the learning collected from realworld data back to development teams throughout all lifecycle processes Example activities processes KPIs Metrics The organization starts the learning and The organization uses two indicators to improvement process throughout the measure performance of integrating the development lifecycle through established learning collected from real-world data retrospectives and post-mortems This learning is back to development teams throughout all captured in a centralized system and is available lifecycle processes and searchable by design and development − Time-to-market of changes to existing teams for future products and feature releases software and − Ratio of positive vs negative sentiment after new feature s are introduced Analysis is performed of the customer data and feature suggestions The firm also performs competitive benchmarking research and usability studies The data is analyzed trended and prioritized and made available through product analysis dashboards that provide fast visualization to the development teams This may also include defects and product issues identified and prioritized for addressing These KPIs align with the business objective to enhance adoption of the software The organization assesses performance and tracks progress of the KPI by analyzing − Number of critical defects − Number of complaints − Reduced rate of customer support FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 16 FDA gov − − − Domain Element Example activities processes contacts User engagement metrics User retention metrics Customer survey results The KPIs are reviewed quarterly and are expected to sustain or show positive trends Drops in the KPIs are analyzed for potential improvements feature releases or new product introductions Configuration Management and Change Control Source control by establishing mechanisms for initiating evaluating and controlling changes to software during all lifecycle processes and after deployment KPIs Metrics The organization has a well-defined system architecture showing system and sub-system configurations There is an established process for assessing the impact to other software subsystems for configuration changes in any part of the system The firm shows traceability from configuration items or components to development and has a process established for “tagging” each component of a software system to identify it throughout the product life-cycle Changes in the software are reviewed by a cross functional team consisting of clinical regulatory engineering marketing and production staff A clinical review of the software change is performed for impact to clinical functionality or performance and the clinical data supporting the change is re-evaluated The product risk analysis is continuously reviewed and updated throughout each change The organization measures the number of returning bugs associated with configuration management issues as an indicator of the quality of the processes for updating the software This KPI aligns with the business quality objectives for product release The organization assesses performance and tracks progress of the KPI by analyzing − Tag coverage i e components are all sync’d or traceable to a master release number − Traceability audit results i e # of unlinked PRs or user stories − # P F regression tests post-release − # risks − Risk analysis update frequency − # design reviews − # of critical bug reports submitted by customers The KPIs are reviewed during release cycles and are expected to trend down No changes or increases in the numbers are indicators that errors are being introduced in updates or their release system and process may need review for improvement FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 17 FDA gov I As part of the program the agency hopes to publish a library of example activities processes and of KPIs that satisfy the elements These are not intended to be prescriptive but are intended to help guide organizations with regards to what works and what others are leveraging This will start to provide alignment and help organizations to review if they have similar processes or KPIs to leverage Your feedback will be used to start building this library We request consideration and input on measures and KPIs For example how the elements might or might not apply in your environment how processes and activities are measured and how you select KPIs and determine whether your choices of KPIs are correct whether you use multiple KPIs to show concordance around a particular element Starting in 2019 during the testing of the Software Precertification Program Version 1 0 the FDA anticipates collecting real-world information on the effectiveness of and ease of appraisal Through development of tools techniques and processes we anticipate the appraisal elements would be further refined with the goal of providing increased precision accuracy and confidence in the appraisal methods and demonstration of excellence in product development By aggregating and analyzing collected information we can understand how organizations build safe and effective SaMD how they know the devices are safe and effective in the real world and how they improve safety and effectiveness as well as efficiency and time to market As the program continues forward our goal is to develop a library of activities processes and KPIs that high performing organizations use In addition we anticipate gaining greater insight into measures that can indicate higher performing organizations FDA believes that driving a focus on performance would encourage the industry to strive for excellence in the manufacture of software device products 4 5 Precertification Levels The goal of establishing levels of certification is to maintain the same standards of safety and effectiveness of products marketed today The levels of precertification are intended to provide to both FDA and the users confidence in an organization’s ability in developing maintaining and marketing safe and effective SaMD Organizations seeking precertification will have different levels of maturity Some organizations may have no or limited experience in delivering medical devices but they have the culture processes and systems to produce high quality products and the capacity to identify and fill gaps and other demonstrable characteristics that support the potential to create safe and effective SaMD C I The following revision to the levels of precertification has been made in response to comments received in the public docket Among organizations that have objectively demonstrated excellence in all five excellence principles the working model distinguishes between those that have successfully marketed and maintained products and those that have not This is a change from the April working model v 0 1 where it was proposed that organizational assessment be based on their successful marketing and maintenance of medical devices We made this proposed change so we can consider whether this more inclusive threshold makes sense when working through scenarios and examples We would appreciate additional public input on this as well FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 18 FDA gov Our current thinking reflects the belief that an organization of any size without a medical device or SaMD currently on the market should have the opportunity to deliver products for medical purposes as a precertified organization We believe organizations that have objectively demonstrated excellence in product development elements in all five excellence principles and have successfully marketed and maintained products can achieve Level 2 Pre-Cert and provide reasonable assurance that it can • • • Understand the clinical use and patient environment disease or condition complexities Identify and rapidly address unanticipated postmarket issues in the SaMD and Apply postmarket lessons to iteratively improve the SaMD throughout the lifecycle processes Level 1 Pre-Cert – This level of certification is designed to allow organizations to develop and market certain lower risk software without review while requiring a streamlined review for other types of software The FDA envisions this level would be awarded to an organization that has objectively demonstrated excellence in product development in all five Excellence Principles with a limited track record in developing delivering and maintaining products in the healthcare space This level of certification may benefit an organization with limited or no experience in delivering SaMD but with established organizational elements and strategies in place that indicate they have or can acquire the capability to deliver and maintain high quality software products that are safe and effective Level 2 Pre-Cert – This level of certification is designed to allow organizations to develop and market certain lower and moderate risk software without review while requiring a streamlined review for other types of software The FDA envisions this level would be awarded to an organization that has objectively demonstrated excellence in product development in all five Excellence Principles with a track record in successfully marketing and maintaining products to suggest a level of assurance in the development of safe and effective software Specific types of SaMD that would require streamlined review or not for the two Pre-Cert levels are described below under “Component 2 Review Pathway Determination ” 4 6 Maintenance and Monitoring of Pre-Cert Status As currently envisioned for the finalized state maintaining Pre-Cert status would be automated Organizational leadership would track and monitor its adherence to the excellence principles and ensure safe and effective operation of their devices by responding appropriately to postmarket indicators including adverse events These details will be developed in a future version of the Software Precertification Program and made available for public comment Determining the objective evidence availability and necessary solutions to support this vision are part of future development activities and learning FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 19 FDA gov 5 Review Pathway Determination Component 2 The principal objective for establishing the review pathway determination component of the Software Precertification Program is to develop a risk-based framework so precertified organizations developing SaMD can determine the premarket review pathway for their products This process will include • • Identifying elements methods and process for precertified organizations to use in determining review pathway based on risk of the product e g by a flow chart or a decision tree Developing a structured method for precertified organizations to inform the public end users and FDA about key elements of the SaMD including a robust description 5 1 Risk Categorization C The premarket review for a precertified organization’s SaMD product would be informed by the organization’s precertification status precertification level and the SaMD’s risk-category The FDA envisions leveraging the risk-category framework for SaMD developed by the International Medical Device Regulatory Forum IMDRF 6 to inform the risk category See Table 2 below The following clarification on the applicability of the IMDRF framework to the definition of a device in 201 h of the FD C Act has been added in response to comments received in the public docket The unmodified IMDRF framework describes the spectrum of SaMD functions some of which may not meet the definition of a device in 201 h of the FD C Act and others that may meet the definition of a device but for which FDA has expressed its intent not to enforce compliance For purposes of the Software Precertification Program the application of the risk-category framework would remain consistent with current definition of device under section 201 h of the FD C Act and FDA’s current enforcement policies The IMDRF framework establishes types and subtypes of SaMD products based on the state of the healthcare condition and the significance of the information provided by the products Table 2 7 6 https www fda gov downloads MedicalDevices DeviceRegulationandGuidance GuidanceDocuments UCM52490 4 pdf 7 The table was first introduced by IMDRF in http www imdrf org docs imdrf final technical imdrf-tech-140918samd-framework-risk-categorization-141013 pdf FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 20 FDA gov Table 3 IMDRF type I to IV and subtype 1 to 9 of SaMD products by state of healthcare condition and significance of information provided by the products to healthcare decision 8 State of Healthcare situation or condition Critical Serious Non-serious Significance of information provided by SaMD to healthcare decision Treat or Drive clinical Inform clinical diagnose management management IV 9 III 7 II 4 III 8 II 6 I 2 II 5 I 3 I 1 Software manufacturers would be able to use the IMDRF “SaMD definition statement” as defined in Software as a Medical Device Possible Framework for Risk Categorization and Corresponding Considerations IMDRF N12 document 9 as a guide to determine where a SaMD falls in the IMDRF risk-categorization table 5 2 Product level elements of a SaMD C FDA proposes to leverage the IMDRF proposed definition statement to develop a structured format for program participants to identify the IMDRF type and subtype based on the significance of information provided by the SaMD to the healthcare decision the state of the healthcare situation or condition and the core functionality of the SaMD Because transparency is one of the key goals of the program we expect all program participants to be transparent in providing information on their SaMD s FDA proposes the following categories of necessary elements which would be described by the SaMD developer The following list of key elements necessary to determine the risk of the SaMD incorporates feedback from comments received in the public docket Key elements necessary to determine the risk of the SaMD and therefore the review pathway for the SaMD needed for initial product introduction to the market 1 Significance of the information provided by the SaMD to the healthcare decision 2 State of the healthcare situation or condition Key elements necessary for identifying modifications that require action in the program 3 Core functionality of the SaMD 4 Device description including key technological characteristics Some functions in Table 2 may not meet the definition of a device or may meet the definition of a device but are functions for which FDA does not intend to enforce compliance with applicable requirements of the FD C Act 9 http www imdrf org docs imdrf final technical imdrf-tech-140918-samd-framework-risk-categorization141013 pdf p 12 This framework was adopted in FDA guidance IMDRF N41 document Software as a Medical Device SaMD Clinical Evaluation 8 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 21 FDA gov Other elements necessary for public end users and FDA to have confidence in SaMD and the organization developing SaMD I 5 Organization’s Precertification Level and other relevant information related to organizational excellence 6 Real-world performance information about the SaMD that complies with all applicable privacy and disclosure laws including user privacy and manufacturer intellectual property rights FDA requests feedback from all stakeholders considering least burdensome principles regarding these proposed elements—for example whether they are appropriate if we are missing any important elements or if there are suggestions for specific details within these elements 5 3 Determining SaMD Risk As described by IMDRF to understand the risk of a SaMD the SaMD definition statement should include a clear and strong statement about the intended medical purpose of the SaMD including the following The “significance of the information provided by the SaMD to the healthcare decision ” which identifies the intended medical purpose of the SaMD The statement should explain how the SaMD meets the definition of a medical device in 201 h of the FD C Act 10 The significance of information and other information on the SaMD product need not be provided for functions that do not meet the definition of a device or that may meet the definition of a device but for which FDA does not intend to enforce compliance with applicable requirements of the FD C Act This statement could be structured in the following terms as defined in section 5 1 of the IMDRF N12 Framework document • • • Significance of Information To treat or to diagnose - To provide therapy to a human body - To diagnose screen detect a disease or condition To drive clinical management - To aid in treatment by providing enhanced support to safe and effective use of medicinal products or a medical device - To aid in making a definitive diagnosis - To triage or identify early signs of a disease or conditions To inform clinical management - To inform of options - To provide clinical information by aggregating relevant information Section 201 h of the FD C Act defines the term “device ” which is distinct from the IMDRF key definitions Final document “medical purposes” definition 10 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 22 FDA gov I FDA requests feedback on the statement for the significance of information provided by the SaMD for example is it sufficiently specific in order to determine the risk of the SaMD See Challenge Question 2 2 The IMDRF risk framework identifies “state of the healthcare situation or condition” that the SaMD is intended for and includes the intended user intended disease or condition and intended population The framework allows for a systematic way to identify the context of the intended medical purpose of the SaMD Information on the SaMD product need not be provided for functions that do not meet the definition of a device or that may meet the definition of a device but for which FDA does not intend to enforce compliance with applicable requirements of the FD C Act This statement would be structured in the following terms as defined in section 5 2 of the IMDRF N12 Framework document The type of disease or condition is Intended target population is Intended to be used by I Critical o Life-threatening state of health including incurable states o Requires major therapeutic interventions o Sometimes time critical depending on the progression of the disease or condition that could affect the user’s ability to reflect on the output information Serious o Moderate in progression often curable o Does not require major therapeutic interventions o Intervention is normally not expected to be time critical in order to avoid death long-term disability or other serious deterioration of health whereby providing the user an ability to detect erroneous recommendations fragile with respect to the disease or condition e g pediatrics high risk population etc specialized trained users NOT fragile with respect to the disease or condition Non-Serious o Slow with predictable progression of disease state may include minor chronic illnesses or states o May not be curable can be managed effectively o Requires only minor therapeutic interventions and o Interventions are normally non-invasive in nature providing the user the ability to detect erroneous recommendations individuals who may not always be patients either specialized trained users or lay users either specialized trained users or lay users FDA requests feedback related to the statement for the criticality of context of the SaMD including for example in order to determine the risk of the SaMD See Challenge Question 2 2 Description of the SaMD’s core functionality11 which identifies the critical features functions of the SaMD that are essential to the intended significance of the information provided by the SaMD to the healthcare decision in the intended healthcare situation or condition This description should include only the critical features These could include specific functionality that is critical to maintain performance and safety profile attributes identified by risk management process undertaken by the manufacturer of SaMD 11 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 23 FDA gov The table below lays out an initial model for determining premarket review pathway for SaMD from precertified companies depending on 1 the IMDRF risk category of the SaMD 2 the level of precertification of the organization and 3 whether the SaMD is a new device or an iteration of an existing device This table describes a proposal for when the precertification of organizations and commitment to leverage real-world performance might allow for no premarket review “no review” in Table 4 below or streamlined premarket review “SR” in Table 4 below according to the IMDRF type subtype of the SaMD and the Pre-Cert Level of the organization L1 Level 1 L2 Level 2 Table 4 Level of Review for Level 1 and Level 2 Precertified Organizations’ SaMD IMDRF Risk Categorization Sub Description type Type IV 9 Critical x diagnose treat Type I Type III 8 Critical x drive Type III 7 Serious x diagnose treat Type II 6 Serious x drive Type II Type II 5 Non-serious x diagnose treat 4 Critical x inform Type I 3 Non-serious x drive Type I 2 Serious x inform Type I 1 Non-serious x inform Level of Review for Level 1 and Level 2 Precertified Organizations’ SaMD Initial product Major changes Minor changes SR SR L1 – SR L2 – No Review L1 – SR L2 – No Review No Review No Review No Review Table 4 above describes when a SaMD from a precertified organization would be reviewed Using this table provide input on how FDA should refine the definition of Level 1 and Level 2 and factors used to determine those levels in the excellence appraisal component FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 24 FDA gov 6 Streamlined Premarket Review Process Component 3 It is expected that software products that are considered for streamlined review are from organizations that have successfully gone through excellence appraisal and as a result these organizations have demonstrated excellence in developing testing maintaining and improving software products The streamlined premarket review process applies to 1 precertified organizations who have made available to the public end users and FDA key elements of their SaMD including a robust description of the SaMD and who have also demonstrated excellence in and showing a capacity for real-world performance analytics Using this information and the processes outlined under the review determination component of this working model see component 2 precertified organizations that determine their SaMD product meets the requirement for being reviewed by FDA would begin a streamlined premarket review process The principal objectives of establishing the streamlined premarket review process component of the Software Precertification Program is to identify the elements necessary for a premarket review and to develop a premarket review process that provides reasonable assurance of safety and effectiveness of a software product from a precertified organization This includes what information would be reviewed how modifications affect marketing authorization and how to leverage existing SaMD standards C The following reference to the FDA guidance IMDRF N41 document titled Software as a Medical Device SaMD Clinical Evaluation has been made in response to comments received in the public docket The FDA envisions reviewing the SaMD’s clinical evaluation results per the FDA guidance IMDRF N41 document Software as a Medical Device SaMD Clinical Evaluation and risk management for the device’s intended use as appropriate The FDA intends to conduct an interactive review supported by automated analysis where appropriate and aspires to provide a decision on the marketing of the precertified organization’s SaMD product within a shorter timeline than other premarket review processes Starting in 2019 during the anticipated testing of proposed Software Precertification Program Version 1 0 if FDA does not authorize the marketing of the product the organization and FDA would complete an after-action review to determine gaps in the evidence supporting the submission and determine a plan for future submission The FDA expects to implement a process where repeated unsuccessful streamlined reviews of a precertified organization’s SaMD trigger a reassessment of the organization’s precertification determination FDA would review the basis of the precertification to address any systemic issues within both the organization and the precertification program At a high level we envision the streamlined review process would work as follows 1 Understanding the product FDA would use the information received during review determination to facilitate a better understanding of the product FDA would work interactively with the program participant to understand the details of the software functions FDA is considering options for how the organization could describe the SaMD and its FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 25 FDA gov intended use such as an interactive demonstration or submission of a wireframe of the SaMD 2 Premarket review FDA envisions interactive review of supporting information which could include evaluating software’s analytical performance clinical performance and appropriate safety measures FDA is considering various options for conducting the review of the supporting information for example through screen sharing access to the development environment and testing logs—using freeform audit of test results 3 Marketing authorization FDA would make a premarket decision document a decision summary and communicate the decision to the organization I For those SaMD that would require premarket notification the current review standard requires that a new device be substantially equivalent to a legally marketed predicate device FDA intends to consider innovative ways to streamline the substantial equivalence review and determination for SaMD products from precertified companies We are describing below a concept for an “iterative early engagement” review process as an option to test using the existing authorities FDA requests feedback on this proposed option including any suggested modifications to this option or alternative solutions 6 1 Option for an iterative early engagement Streamlined Review Process for Premarket Notification for Pre-Cert 1 0 C Components of this option for streamlined review including interactive review review by demo and consideration of reduced submission documentation compared to traditional 510 k s have been included in response to comments received in the public docket In this concept of an “iterative early engagement” option as shown in Figure 4 below see full page image in Appendix C the review portion of streamlined review would include reduced predetermined administrative entry requirements and can be customized to the sponsor’s SaMD device type This conceptual framework makes use of existing regulatory pathways to allow us to test and iterate on the proposed streamlined review process while continuing to provide a viable regulatory pathway for “Pre-Cert” devices As one possible approach FDA is considering a more interactive process similar to the Q-submission process 12 Precertified organizations could opt to participate in this iterative early engagement process if they desire FDA’s input during the development of their SaMD products We would anticipate that as sponsors design their device or develop further supportive information and data new supplements can be reviewed at an agreed upon cadence with documented feedback provided from FDA At the end of the iterative early engagement option a document could be generated which Requests for Feedback on Medical Device Submissions The Pre-Submission Program and Meetings with Food and Drug Administration Staff - Guidance for Industry and Food and Drug Administration Staff available at https www fda gov MedicalDevices DeviceRegulationandGuidance GuidanceDocuments UCM311176 12 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 26 FDA gov substantiates the elements that have been reviewed This document along with pre-determined essential 510 k documentation is envisioned to take the place of other substantive documentation resulting in an expedited review of the product The main objective of this option is to provide marketing clearance as a device is finalized and ready to be marketed This option is intended to demonstrate a method by which a precertified organization can submit product-specific regulatory information in an iterative manner during the development of the product This option has the potential to reduce the time necessary for regulatory clearance once the product development has been completed The intention is to build in the flexibility to allow precertified companies to interact with FDA if desired at the appropriate time in their product development cycle to obtain the right regulatory information at the right time As such this iterative early engagement model aims to harmonize review with the sponsor’s design process so that marketing authorization is received at the time that a device is ready to enter the marketplace Figure 4 Possible conceptual framework for conducting an iterative early engagement streamlined review for a premarket notification of a software product from a pre-recertified organization 6 2 Determining elements necessary for assuring safety and effectiveness in premarket review As part of the process for streamlining the review of SaMDs FDA is proposing to employ least burdensome principles to calibrate the amount and type of information needed to support a determination of substantial equivalence To begin a discussion of this concept in Table 5 below we have identified some of the main product-specific elements that the Agency typically FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 27 FDA gov receives in a premarket notification submission Some of these elements would be required for all submissions such as intended use indications for use and instructions But perhaps not all of the items listed would be needed for every product Rather we have compiled a list of items that could potentially be useful in evaluating a product I FDA is requesting feedback from the public and SaMD manufacturers on this proposal for example what elements are necessary for assuring safety and effectiveness of a SaMD from a precertified organization what elements are missing from our proposed concept and how should necessary elements be determined for various product types Please provide feedback on whether and how the excellence appraisal and real-world performance processes might be used to provide assurance for each of the elements listed in Table 4 below Table 5 –Software-related Review Elements in a 510 k Submission Element 3rd party software Anomalies Bugs Clinical algorithm Clinical performance Configuration management Cover letter Cybersecurity Declarations of conformity and summary reports Development environment Developmental testing e g code review unit test reports integration testing Device description Device summary Executive summary Indications for use Claims Instructions for use Intended use population Labeling review Life cycle Modifications made since clearance Requirements Revision history Risk analysis SaMD product demo Server performance Load testing Software architecture Software bill of materials Substantial equivalence comparison Traceability User interface Story Workflow Verification and validation FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 28 FDA gov       7 Real-World Performance Component 4 The principal objectives for the real-world performance RWP component of the Software Precertification Program are to develop real-world performance data domains and analytic methodologies needed for Pre-Cert Program activities The scope of this program component is to identify and address expectations for use of real-world performance analytics RWPA by both precertified organizations and FDA in the Pre-Cert Program FDA intends to focus its postlaunch product monitoring efforts on product-specific performance analytics such as trends rather than on raw data C The following clarifications concerning an organization’s ability to tailor real-world performance elements to their organization and products has been made in response to comments received in the public docket We believe that excellent organizations not only focus on quality while developing SaMD products but also grow and evolve based on lessons learned from real-world usage of their products after they launch While specific RWP data elements and analytic methodologies may differ across organizations and product categories excellent organizations consistently collect and analyze post-launch data from diverse sources to inform their operations and decision making from quality control to product development for new market segments We envision that precertified organizations would select specific data elements within the proposed framework based on the intended use functionality and risk classification of the SaMD product FDA believes organizations can show excellence per the Pre-Cert Excellence Principles by taking user-centric steps toward continuous improvement through proactive monitoring of RWP data related to their SaMD products During the excellence appraisal all precertified organizations would demonstrate a robust program for monitoring real-world performance data related to their SaMD devices similar to what is done with regards to preventive actions and for sharing analyses of such data with FDA The Agency anticipates that the benefits of monitoring product-level RWPA include  Increased ability of precertified organizations to use collected RWP data to support product claim modifications changes in intended use or expansions of product functionality  Increased public confidence in the PreCert Program and in SaMD products marketed by precertified organizations based on the organizations’ demonstrated ability to leverage RWPA in the continuous improvement of their products and processes  Increased ability of FDA to support industry in taking proactive actions to address emerging safety or cybersecurity issues  Ongoing refinement of all other components of the Pre-Cert Program  Evidence that precertified organizations are tracking product-level RWPA and responding to emerging issues FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 29   FDA gov For the purposes of this document RWPA are defined as systematic computational analyses of all data relevant to the safety effectiveness and performance of a marketed SaMD product from a precertified organization FDA anticipates that such data may be generated collected and analyzed efficiently by leveraging not only data collected from appropriately instrumented SaMD products but also real-world data from device registries and other electronic health information sources including the National Evaluation System for health Technology NEST currently under development FDA considers RWPA to encompass at least three types of analyses as defined below • Real-world health analytics RWHA are analyses of outputs and outcomes related to the SaMD Definition Statement RWHA can inform changes to the intended use of a SaMD product support expanded functionalities and use in broader target populations and monitor the continued safety and effectiveness of a marketed SaMD product • User experience analytics UXA are analyses of user experience outputs related to the real-world use of a SaMD product UXA facilitate timely identification and correction of user issues and improve the utilization and effectiveness of the software • Product performance analytics PPA are analyses of outputs and outcomes demonstrating the accuracy reliability and security of a SaMD product PPA monitoring allows for timely patches and updates to correct software bugs and security vulnerabilities 7 1 Framework for Use of RWPA FDA intends to use RWPA to evaluate performance at product organizational and Program levels Key functions for RWPA would include 1 Monitoring safety effectiveness and performance of marketed SaMD products All precertified organizations should collect and analyze post-launch RWPA and should have demonstrated the capability to instrument their SaMD products to meet such framework FDA anticipates applying a uniform RWPA framework to all SaMD products introduced to market by precertified organizations across organizational precertification tiers product risk classifications and product premarket review requirements FDA intends to adjust the RWPA framework based on experience accumulated in 2019 This RWPA framework as defined by the Software Precertification Program would not apply to SaMD products cleared by FDA prior to precertification of the organization The RWPA framework for post-launch product monitoring is intended to provide robust evidence of SaMD safety and effectiveness while retaining sufficient flexibility to accommodate the full range of organizations capable of demonstrating excellence in digital health To that end FDA envisions engaging in an iterative process with precertified organizations to refine the types of data elements most relevant to SaMD real-world performance monitoring to streamline the monitoring process following the least-burdensome approach and to increasingly automate the process of developing a product-specific RWPA plan In 2019 FDA anticipates that precertified organizations would discuss with FDA elements of RWPA that could include FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 30 FDA gov • • • I Proposed RWP data elements to be collected Intended frequency of data collection and Commit and stretch goals for each proposed data element Proposed RWPA should demonstrate safety effectiveness and performance of the SaMD product and should be representative of the data domains outlined below Figure 5 Please see Appendix D – Real-World Performance Data Elements for PostLaunch Product Monitoring for additional detail We envision that precertified organizations would select specific data elements based on the intended use functionality and risk classification of the SaMD product We request input on the data and analytic domains proposed in this version of the working model Please provide feedback on the comprehensiveness of the productlevel RWPA framework identify any data or analytic domains that you believe are missing and consider strategies to ensure that data access follows least-burdensome principles For the 2019 testing phase for purposes of learning and optimizing the RWPA domains FDA expects precertified organizations participating in the pilot program to provide FDA with access to RWPA on a regular basis e g quarterly or at the request of the Agency In these early phases of the Software Precertification Program FDA would review submitted RWPA and would work with precertified organizations individually to refine needed data domains Following an interactive FDA review and mutual agreement that the proposed RWPA plan is adequate for monitoring safety and effectiveness of the SaMD product precertified organizations should collect and analyze RWP data elements through processes and methodologies defined by the organizations As the Software Precertification Program matures FDA anticipates continuing to refine the process of reviewing and accessing RWPA Figure 5 RWPA Post-Launch Product Monitoring Domains FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 31 FDA gov 2 Supporting modifications of clinical and performance claims for safety and effectiveness FDA anticipates that use of RWPA for this purpose would involve defining the following elements • Methodology and processes to evaluate RWPA used to support an initial SaMD product claim and • Methodology and processes to evaluate RWPA used to support a design change labeling change or change in intended use such changes may reflect either increased or decreased functionality of the SaMD product in real world performance as compared to pre-launch expectations 3 Providing input to initial precertification and changes to precertification status FDA anticipates that use of RWPA for this purpose would involve defining the following elements • Methodologies and processes for using RWPA as inputs into the initial precertification appraisal and • RWPA-based thresholds that would trigger a need to review and modify the precertification status of a SaMD organization I 4 Providing feedback to FDA to further refine the Software Precertification Program FDA anticipates that use of RWPA for this purpose would involve informing the following elements • Framework for using aggregate RWPA of precertified organizations to inform refinement of the precertification appraisal model • Framework for using aggregate RWPA of precertified organizations to inform refinement of the precertification streamlined review process and • Framework for using aggregate RWPA of precertified organizations to inform refinement of the review determination framework We request input on framework elements 2 3 and 4 proposed in this version of the working model Please provide feedback on the comprehensiveness of the elements and identify any elements that you believe are missing 8 Next Steps and Public Engagement FDA is publishing this version 0 2 of the working model of the Software Precertification Program to gather public input as we continue to develop this program FDA will continue to consider and evaluate comments received incorporate comments into the model as appropriate and regularly seek additional public input throughout the development of this program FDA intends to consider stakeholder comments by reviewing the public docket approximately every two weeks and to incorporate comments as appropriate in future versions of the working model We encourage the public to provide feedback early and often FDA is seeking public feedback on this version of the working model by July 19 2018 at https www regulations gov comment D FDA-2017-N-4301-0001 This feedback will be FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 32 FDA gov incorporated into future versions of the program model which will also be disseminated for public input 9 Appendix A – Clarification of IMDRF Definition of Software as a Medical Device SaMD C This clarification of what is considered a SaMD has been added in response to comments received in the public docket The International Medical Device Regulators Forum’s IMDRF defines software as a medical device SaMD as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device This definition is further clarified through the following notes that accompanied the definition of SaMD In order to further explain these notes we are providing the following clarifications • SaMD is a medical device and includes in-vitro diagnostic IVD medical device o For purposes of the Software Precertification Program the application of the riskcategory framework would remain consistent with current definition of device under section 201 h of the FD C Act and FDA’s current enforcement policies The unmodified IMDRF framework describes the spectrum of SaMD functions some of which may not meet the definition of a device in 201 h of the FD C Act and others that may meet the definition of a device but for which FDA has expressed its intent not to enforce compliance o Non-device software functions are not subject to regulation and are not within the scope of the Software Precertification Program including software functions intended 1 for administrative support of a health care facility 2 for maintaining or encouraging a healthy lifestyle 3 to serve as electronic patient records 4 for transferring storing converting formats or displaying data or 5 to provide limited clinical decision support are not medical devices 13 and are not subject to FDA regulation Software functions described in final guidance documents that may meet the definition of a device but for which FDA does not intend to enforce compliance with applicable requirements of the FD C Act are not within the scope of the Software Precertification Program • SaMD is capable of running on general purpose non-medical purpose computing platforms o This means that a SaMD is capable of running on any computing platform that has a microprocessor or microcontroller which can live in many types of products i e a computing platform is location-agnostic A myriad of smart electronic products including glucose meters smart phones MRI machines Section 520 o 1 of the FD C Act 21 U S C 360j o 1 A - D as added by Section 3060 a of the 21st Century Cures Act December 13 2016 13 FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 33 FDA gov     o     laptops infusion pumps smart watches and ECG machines all have a computing platform that executes software A computing platform that may or may not be part of a medical device typically includes microprocessor or microcontroller with peripherals intended solely for executing instruction and software logic or calculations described through a programming language Peripherals provide input to the microprocessor and deliver output from the microprocessor Example inputs may include a mouse keys touchscreen gyroscope accelerometer and GPS whereas example outputs may include display speaker light and vibration actuators “Without being part of” means software not necessary for a hardware medical device to achieve its intended medical purpose o This statement refers to software typically known as “embedded software” that is included in a hardware medical device or an IVD test instrument Such software is primarily relied upon necessary for achieving the “intended use” of the hardware medical device or the IVD instrument Such software is not considered a SaMD Alternatively such software when executed on another general-purpose computing platform would not achieve the same intended use Software does not meet the definition of SaMD if its intended purpose is to drive a hardware medical device o This statement refers to software whose function is to control or drive another medical device Autonomous closed-loop medical devices i e those that do not require clinicians to make an intrepretation from signal-gathering to decisionstate are not considered SaMD  SaMD may be used in combination e g as a module with other products including medical devices o This statement indicates that a SaMD may be used as a module in a larger system interconnected or bundled with other software modules that may or may not be medical devices  SaMD may be interfaced with other medical devices including hardware medical devices and other SaMD software as well as general purpose software o In this statement “interfaced” refers to the notion that input to a SaMD can come from many sources o SaMD is software that acts on data for a medical purpose and that data that SaMD may use as input can come from a variety of medical and non-medical products Medical devices such as ECG machines MRI and in vitro diagnostics collect many types of data that may be used as input into a SaMD Non-medical products such as general wellness devices and general-purpose sensors may also collect data that could be used as input into a SaMD The SaMD algorithm acts on that data for a specific medical purpose which may be to inform drive diagnose or treat a healthcare situation or condition Further clarification on the data quality and collection principles for non-medical sensors used as inputs into SaMD is under development  Mobile apps that meet the definition above are considered SaMD FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 34   FDA gov o The intent of this statement is that SaMD can also be mobile apps as FDA defined such mobile apps as “mobile medical apps ” This could include mobile apps that perform patient-specific analysis provide patient-specific diagnosis or recommend treatments for example FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 35 FDA gov 10 Appendix B – Proposed Organizational Elements to Demonstrate Excellence Principles FDA intends to evaluate organizational elements based on objective and observable evidence Although the appraisal method is under development we expect organizations would provide this evidence as part of the appraisal process which may include site visits interviews or other methods FDA hopes to implement automation for the acceptance and review of an organization’s demonstration of their elements in future iterations to reduce appraisal burden increase transparency and enhance the capability to respond quickly and improve products without reducing public confidence in the program I The elements and domains are proposed in this version of the working model for public input We request consideration and input regarding these elements including as to whether the elements proposed here reflect the intent of the program and the five excellence principles to look holistically beyond software design and development Additionally we seek comment on how to further clarify these elements and the associated domains to reduce the burden on a software organization to identify their processes activities and outcomes Organizational Domains Leadership and Organizational Support Transparency Excellence Principle s Elements Providing clear accountability and responsibility to address product issues user issues constraints and conflicting priorities throughout the product lifecycle Empowering staff to act regarding the decisions or issues impacting users products or patient safety Providing the resources and focus to assure important infrastructure and processes to assure patient safety are sustained and improved Developing and maintaining systems or dashboards where all levels of the organization can rapidly see and understand how they are performing among metrics relevant to the organization Making defects deviations safety issues transparent to internal and external stakeholders as appropriate Security and quality issues are communicated with internal and external stakeholders sufficiently to catalyze corrective and preventive action PS PQ ClinR CybR X X X X X X X X X X X X X X X X X X X X X X X PC X X X X Organizational Domains People Excellence Principle s Elements PS Buyers and users understand design assumptions about expected operational conditions environment to use devices safely securely and effectively Buyers and users patients physicians understand expected or minimum support lifetimes and levels Developing and maintaining access to highly skilled employees with relevant applicable clinical knowledge Involving appropriate cross functional subject matter expertise including engineering clinical expertise and user advocates with frequent engagement and communication in product decisions and potential safety events Continuous development of employees through robust knowledge management employee development options coaching training and succession planning This includes keeping updated with the latest clinical developments and patient safety priorities Developing and maintaining clear and objective employee performance metrics rewards and recognitions aligning behaviors to the business goals values and rapidly responding to patient safety issues Infrastructure and Customer engagement and providing multiple avenues for outreach Work Environment feedback and learning Implementing the tools automation and test environments in development that establishes a centralized and visible process Develop and maintain a robust notification and communication framework Communicate and preserve the relevant results of the activities processes and expectations related to the SaMD lifecycle processes PQ ClinR CybR PC X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 37 FDA gov Organizational Domains Excellence Principle s Elements Developing and maintaining processes and mechanisms for rapid learning from successes failures and near-misses Risk Management Regularly questioning how software works by understanding identifying and proactively anticipating potential issues and factors that can influence A Patient Safety what can go wrong with the software Focused Process Favor rigorously tested software components i e well-vetted cryptographic libraries vs roll your own or identify risks and mitigations Identifying receiving and handling vulnerability reports from third parties directly coordinated vulnerability disclosure or from public sources such as vulnerability databases Accounting for support lifecycles of hardware and software components and dependencies i e If the SaMD is expected to be used longer than the operating system is supported how will you continue to address things like security vulnerabilities Configuration Source control by establishing mechanisms for initiating evaluating and Management and controlling changes to software during all lifecycle processes and after deployment Change Control Good release management with a secure update process The ability to rollback software in the event of an emergency Measurement Analysis and Improvement of Processes and Products PS PQ X X X X X X X X X X X X X X CybR PC X X X X X X X X X X Responsive issue escalation resolution X Actively monitoring analyzing rapidly addressing and implementing resulting process improvements from user feedback and product issues including safety cyber or data issues Analyzing and providing the learning collected from real world data back to development teams throughout all lifecycle processes ClinR X X X X X X FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 38 FDA gov Organizational Domains Excellence Principle s Elements PS Developing and maintaining process performance metrics that are clear simple and actionable across all staff and organizational levels with integrated improvement activities Supporting rigorous interrogation into sources of failure error and tampering including tamper resistant forensically sound evidence capture and publicly known mechanisms to perform or trigger investigations Managing Outsourced Processes Activities and Products Requirements Management Design and Development Comprehensive risk management of third-party and open source software throughout all lifecycle process and activities Avoid third-party software components with known vulnerabilities when less vulnerable alternatives are available Maintain and provide traceability and assurance of third-party and open source software throughout the effective lifetime of the software Understanding the clinical association between the SaMD output and a clinical condition i e clinical performance and understanding and updating the priorities concerns and value to intended user based on user feedback throughout all lifecycle phases Buyers and operators understand the impact of operational isolation e g which features are fully available in standalone no network mode Carefully manage and gate remote access to all device components and dependencies e g Avoid hardcoded default credentials within the device and enforce secure identity and access management for any provideroperated components like software update distribution servers Secure software development lifecycle including adversarial resilience analysis and testing reducing elective attack surface complexity and minimizing elective exposure throughout the software lifecycle PQ ClinR CybR X PC X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 39 FDA gov Organizational Domains Excellence Principle s Elements PS Designing software based on good quality clinical evidence from research and can reference published peer-reviewed studies that show claimed results Incorporating resilience containment and isolation into the design solution so that product fails safely and visibly continue to perform as intended when there are failures in the operating environment and assures the integrity of data input and storage Incorporating anticipated safety risks and mitigations throughout all lifecycle phases and actions taken to prevent recurrence of any unanticipated hazards Reliably identifying and removing code errors at source Integrating user experience Human Factors and Good Clinical Practices Human Subject Protection into development in partnership with patients and caregivers Secure prompt and agile update mechanism and process with high rates of prompt update adoption and clear notification and communication to stakeholders Leadership peer expert review throughout lifecycle phases and at key milestones Verification and Validation Deployment and Maintenance Staged release with active user testing Demonstrating software works for intended use indications for use Measuring quality of the output of the software on the clinical target intended use indication of use Proactive patient and clinical outreach and education including limitations of software and FAQs addressing potential patient safety questions developed as part of release PQ ClinR CybR PC X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 40 FDA gov Organizational Domains Excellence Principle s Elements Active control mechanisms to force push patient safety and security updates Support dependency updates such as routine operating system upgrades PS PQ X X X X ClinR CybR PC X X X FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 41 FDA gov 11 Appendix C – Possible framework for conducting an iterative early engagement streamlined review for a premarket notification of a software product from a precertified organization FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 42 FDA gov 12 Appendix D – Real-World Performance Analytics for Product Monitoring I Proposed RWPA should demonstrate safety effectiveness and performance of the SaMD product and should be representative of the data domains We envision that precertified organizations would select specific data elements based on the intended use functionality and risk classification of the SaMD product We request input on the data and analytic domains proposed in this version of the working model Please provide feedback or specific examples of real world performance analytics that would demonstrate safety effectiveness and performance of the SaMD product Analytic Type Real World Health Analytics Domain Value Excellence Principle s PQ PS ClinR CybR PC Human Factors and Usability Engineering • Pre-Cert Organization Support product claims by understanding user ability to comprehend and correctly navigate user interface • All stakeholders Demonstrate continuous improvement in usability engineering to drive health benefits and safety X Clinical Safety • Pre-Cert Organization Benefit from early safety signal detection across Pre-Cert organizations • All stakeholders Provide assurance that safety risks are managed and mitigated in a timely way X Health Benefits • Pre-Cert Organization Support product claims and future claim modifications by understanding clinical benefits X X X X X X X X X Example KPIs User error rate Anticipated adverse event rate severity Time to resolve anticipated adverse event Unanticipated adverse event rate severity Time to resolve unanticipated adverse event Rate of change in targeted health outcome by user demographic FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 43 FDA gov Analytic Type User Experience Analytics Domain Value • All stakeholders Demonstrate positive impact on health outcomes User Satisfaction • Pre-Cert Organization Provide insight into brand reputation and product performance • All stakeholders Demonstrate excellence in product quality organizational proactivity and product effectiveness Issue Resolution • Pre-Cert Organization Build consumer confidence in organization and SaMD product • All stakeholders Demonstrate excellence in safety and product quality User Feedback Channels User Engagement Excellence Principle s PQ PS ClinR CybR PC X X • Pre-Cert Organization Identify and resolve important user issues early and timely • All stakeholders Demonstrate clinical responsibility and excellence in product quality by ensuring that user feedback is representative of the full user population • Pre-Cert Organization Optimize user experience and meet business targets for user engagement X X X X X X X X X X X Example KPIs Average user ratings over time Complaint rates Customer survey responses Time to resolution by clinical cybersecurity risk category Number of open complaints Time to root cause analysis Number of repeat issues complaints Customer rating of issue resolution X Response rates by demographic Response rates by feedback channel X Unique users User retention Time in app FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 44 FDA gov Analytic Type Domain Value Excellence Principle s PQ PS ClinR CybR PC • All stakeholders Demonstrate product quality clinical responsibility and proactivity by understanding and continuously improving user experience • Pre-Cert Organization Build consumer confidence in organization and SaMD product Cybersecurity • All stakeholders Protect user privacy ensure product integrity and maintain Product system availability Performance Analytics • Pre-Cert Organization Support product claims and future claim modifications Product • All stakeholders Demonstrate sustained Performance analytical validity and excellence in continuous improvement in product quality X X X X X X X X X Example KPIs Number of breaches resulting in loss of user data Number of remediated vulnerabilities vulnerabilities identified System downtime False positive false negative rates Bug defect rates Version failure rates FDA will continue to build and refine this working model by considering public comments incorporating comments received as appropriate and regularly seeking additional public input throughout the development of this program Software Precertification Program Working Model – Version 0 2 – June 2018 45 FDA gov