Guidance for Industry and Food and Drug Administration Staff – Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data

Document issued on: July 3, 2012

The draft of this document was issued on October 21, 2009.

For questions regarding this guidance document, contact Nicholas Petrick (OSEL) at 301-796-2563, or by e-mail at [email protected]; or Mary Pastel (OIVD) at 301-796-6887 or by e-mail at [email protected].

U.S. Department of Health and Human Services
Food and Drug Administration
Center for Devices and Radiological Health
Division of Imaging and Applied Mathematics
Office of Science and Engineering Laboratories
Division of Radiological Devices
Office of In Vitro Diagnostic Device Evaluation and Safety

Preface

Public Comment

You may submit written comments and suggestions at any time for Agency consideration to the Division of Dockets Management, Food and Drug Administration, 5630 Fishers Lane, rm. 1061, (HFA-305), Rockville, MD 20852. Identify all comments with the docket number listed in the notice of availability that publishes in the Federal Register. Comments may not be acted upon by the Agency until the document is next revised or updated.

Additional Copies

Additional copies are available from the Internet . You may also send an e-mail request to [email protected] to receive an electronic copy of the guidance or send a fax request to 301-847-8149 to receive a hard copy. Please use the document number (1697) to identify the guidance you are requesting.

Table of Contents

1. INTRODUCTION
2. BACKGROUND
3. SCOPE
4. DESCRIBING THE DEVICE IN A 510(K) PREMARKET NOTIFICATION
   • 4.1. General Information
   • 4.2. Algorithm Design and Function
   • 4.3. Processing
   • 4.4. Features
   • 4.5. Models and Classifiers
   • 4.6. Algorithm Training
   • 4.7. Databases
   • 4.8. Reference Standard
   • 4.9. Scoring
   • 4.10. Other Information
5. STANDALONE PERFORMANCE ASSESSMENT
   • 5.1. Study Population
   • 5.2. Test Data Reuse
   • 5.3. Detection Accuracy
   • 5.4. Generalizability Testing
   • 5.5. Comparison with Predicate Device
   • 5.6. Electronic Data
6. CLINICAL PERFORMANCE ASSESSMENT
   • 6.1. Comparison with Predicate Device
   • 6.2. Electronic Data
7. USER TRAINING
8. LABELING
   • 8.1. Indications for use
   • 8.2. Directions for use
   • 8.3. Warnings
   • 8.4. Precautions
   • 8.5. Device Description
   • 8.6. Clinical Performance Assessment
   • 8.7. Standalone Performance Assessment

Guidance for Industry and FDA Staff

Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Notification [510(k)] Submissions

This guidance represents the Food and Drug Administration’s (FDA’s) current thinking on this topic. It does not create or confer any rights for or on any person and does not operate to bind FDA or the public. You can use an alternative approach if the approach satisfies the requirements of the applicable statutes and regulations. If you want to discuss an alternative approach, contact the FDA staff responsible for implementing this guidance. If you cannot identify the appropriate FDA staff, call the appropriate number listed on the title page of this guidance.

¹) devices applied to radiology images and radiology device data (often referred to as “radiological data” in this document). CADe devices are computerized systems that incorporate pattern recognition and data analysis capabilities (i.e., combine values, measurements, or features extracted from the patient radiological data) and are intended to identify, mark, highlight, or in any other manner direct attention to portions of an image, or aspects of radiology device data, that may reveal abnormalities during interpretation of patient radiology images or patient radiology device data by the intended user (i.e., a physician or other health care professional), referred to as the “clinician” in this document. We have considered the recommendations on documentation and performance testing for CADe devices made during the Radiology Devices Panel meetings on March 4-5, 2008² and November 17-18, 2009.³ We have also considered the public comments received on the draft guidance announced in the Federal Register on October 21, 2009 (74 FR 54053).

FDA’s guidance documents, including this guidance, do not establish legally enforceable responsibilities. Instead, guidance documents describe the Agency’s current thinking on a topic and should be viewed only as recommendations, unless specific regulatory or statutory requirements are cited. The use of the word should in Agency guidance documents means that something is suggested or recommended, but not required.

⁴

⁵

Common scoring criteria that have been used to determine the nature of each CADe detection include:

• centroid of the CADe detection area or volume falling in the reference standard area or volume;
• distance between centroids of the CADe detection and the reference standard;
• ratio of the distance between centroids of the CADe detection and the reference standard, relative to the maximum width of the reference standard region;
• ratio of the area (A) or volume (V) intersection between the CADe detection and the reference standard, with the total area or volume of the CADe detection, defined as follows:
  ( A(CAD) ∩ A(Ref) ) / A(CAD)
  or
  ( V(CAD) ∩ V(Ref) ) / V(CAD)
• ratio of the area (A) or volume (V) intersection between the CADe detection and the reference standard with the total area or volume union of the reference standard and the CADe detection, defined as follows:
  ( A(CAD) ∩ A(Ref) ) / ( A(CAD) ∪ A(Ref) )
  or
  ( V(CAD) ∩ V(Ref) ) / ( V(CAD) ∪ V(Ref) )

The scoring process should be consistent with the abnormalities being marked by the CADe and the intended use of your device and the performance claims being made. You should provide a scientific justification for this process and it should be fixed prior to initiating your evaluation. In your description of the scoring process, we recommend you indicate whether the scoring is based on:

• electronic or non-electronic means;
• physical overlap of the boundary, area, or volume of the mark in relation to the boundary, area, or volume of the reference standard;
• relationship of the centroid of the mark to the boundary or spatial location of the reference standard;
• relationship of the centroid of the reference standard to the boundary or spatial location of the mark;
• interpretation by reviewing readers; or
• other methods.

For scoring that relies on interpretations by reviewing readers, we recommend you provide the number of readers involved, their qualifications, their level of experience and expertise, the specific instructions conveyed to them prior to participating in the scoring process, and any specific criteria used as part of the scoring process. When multiple readers are involved in scoring, you should describe the process by which their interpretations are combined to make an overall scoring determination or how their interpretations are incorporated in the performance evaluation, including how any inconsistencies are addressed.

4.10. Other Information

We recommend that you include software documentation following the Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices⁶ and in Guidance for Off-the-Shelf Software Use in Medical Devices.⁷ The kind of information we recommend submitting is determined by the “level of concern,” which is related to the risks associated with a software failure. The level of concern for a device may be minor, moderate, or major. Based on prior CADe device submissions, the level of concern for a CADe system is generally moderate.

If the CADe system is an add-on software to be installed within a third party image review platform, we recommend you also provide the names, version/model numbers, and characteristics of these third party platforms as well as a description of the file format of the CADe output that is generated by your device. If applicable, we recommend you refer to Guidance for the Submission of Premarket Notifications for Medical Image Management Devices.⁸

9

To minimize the risk of tuning to the test data and to maintain data integrity, we recommend you develop an audit trail and implement the following controls when you contemplate the reuse of any test data:

• you randomly select the data from a larger database that grows over time;
• you retire data acquired with outdated image acquisition protocols or equipment that no longer represents current practice;
• you place a small fixed limit on the number of times a case can be used for assessment;
• you maintain a firewall such that data access is tightly controlled to ensure that nobody outside of the regulatory assessment team, especially anyone associated with algorithm development, has access to the data (i.e., only summary performance results are reported outside of the assessment team);
• you maintain a data access log to track each time the data is accessed including a record of who accessed the data, the test conditions and the summary performance results.

The purposes of the audit trail include: (1) establishing that you defined the cases in the training and test sets appropriately such that data leakage between training and test sets did not occur; (2) ensuring that you fixed the new CADe algorithm in advance (i.e., before application to the test set); and (3) providing information concerning the extent to which you used the same test set or a subset thereof for testing other CADe algorithms or designs, including results reported to the Agency as well as non-reported results. The controls we are recommending are intended to substantially reduce the chance that you evaluate a new CADe algorithm in a subsequent study using the same test data that you used for a prior CADe algorithm.

If you reuse test data, you should report the test performance for relevant subsets in addition to the overall performance. These subsets include: (1) the portion of the test set for the new CADe algorithm that overlaps with previously used test sets, and (2) the portion of the test set that you have never used before. Since these subsets will be smaller in size compared to the overall test set for the new CADe algorithm, confidence intervals for the subsets will be wider. However, the trends for the mean performances would be helpful to indicate whether you have tuned the CAD system to previously used portions of the test set.

5.3. Detection Accuracy

We recommend you estimate and report the CADe standalone performance following the scoring process (see Subsection 4.9. Scoring). The definition of a true positive, true negative, false positive, and false negative CADe mark should be consistent with the intended use of the device. For example, if the device is intended to detect all abnormalities (e.g., benign and malignant), then a true positive CADe mark should be defined as “marking” any abnormalities. On the other hand, if a device is intended to detect only a subset of abnormalities (e.g., only those lesions with certain imaging features), then you should define a true or false CADe mark accordingly.

For a reference standard (e.g., disease type, location, and extent) that relies on the interpretation by reviewing readers, we recommend that you account for reader variability in the truthing process and for various consensus or agreement rules between expert readers, in the CADe standalone performance estimates. One method of accounting for variability in the reference standard is to resample the expert truthing panel. See Miller et al.¹⁰ for details on one approach.

We recommend you report the overall lesion-based, patient-based or other relevant measure of standalone sensitivity (e.g., probability the CADe algorithm correctly marks the location of an abnormality) at each operating point. You should report a relevant measure of the standalone false positive rate¹¹ (e.g., the average number of false positive marks per case [FPs/case]) at each device operating point. You should report stratified analysis per relevant confounder or effect modifier as appropriate (e.g., lesion size, lesion type, lesion location, disease stage, imaging or scanning protocols, imaging or data characteristics). The standalone false positive rate should be derived from normal and abnormal patient data separately. If your device allows the clinician to select or manipulate the device operating point, we recommend you provide the device performance for each selectable operating point or for the range of possible operating points. In addition, detection accuracy will likely depend upon the scoring criteria and scoring threshold used to determine the nature and extent of each CADe detection (see Subsection 4.9. Scoring). Therefore, we also recommend you estimate and report the performance of your device using various values of the distance and ratio scoring thresholds when applicable. We recommend that you include plots showing the performance change as a function of overlap criteria when appropriate. You should determine and fix the detection accuracy assessment methodology, including the selection of primary and secondary performance endpoints, prior to initiating your evaluation.

All performance measures should be reported with associated confidence intervals (CIs). We recommend you provide a description of your methodology for estimating these CIs and the clinical significance associated with these CIs.

We also recommend you provide graphs of the standalone free-response receiver operating characteristic (FROC) curves (i.e., a plot of patient-based standalone sensitivity vs. average number of FPs/case as a function of operating point) when reporting detection accuracy and the clinical interpretation of this analysis. You should report associated standalone FROC CIs when appropriate. Resampling techniques, such as the bootstrap,¹² are potential methodologies for estimating these CIs.

Finally, we recommend that your testing accounts for differences in both the location and extent of the abnormality (e.g., ratio of the intersection area of CADe mark and reference standard to the union of these areas) when your marker includes information about both of these factors (e.g., CADe marker is outline of the segmented structure).

5.4. Generalizability Testing

We recommend you justify the extent to which your studies generalize to the range of acquisition technologies defined in the device labeling. This may include studying the impact of various acquisition technologies and acquisition parameters on CADe performance to provide insight into the stability of the CADe algorithm. You should conduct this additional testing in a way that is appropriate and that does not instill additional risk (e.g., increase radiation or contrast exposure) to the patient while still examining the impact of various image acquisition confounders, such as:

• imaging hardware,
• imaging or scanning protocol, and
• image or data characteristics (e.g., characteristics associated with differences in digitization architectures for a CADe using scanned films).

Examples of generalizability testing for CADe device include performing:

• a stratified analysis of parameters expected to introduce variability in the results (e.g., scanning characteristics, make and model of the imaging devices, acquisition protocol parameters such as contrast agent or probe positioning) when the data used in the development and testing reflect the imaging requirements;
• a sound technical argument or bench-top data to support compatibility with your CADe device for various imaging systems that use the same or similar underlying technology;
• a performance comparison for a set of images digitized multiple times on different scanners when the CADe input is digitized image data;
• a performance assessment of multiple image acquisitions from the same patient with the same modalities (e.g., MRI or ultrasound). Again any additional testing should be conducted in a way that is appropriate and that does not instill additional risk to the patient; and
• a performance assessment of simulated data in lieu of actual acquisitions (e.g., parametric noise simulation studies, simulated reconstructions, resolution reduction) when the validity of the simulated data has been established.

We recommend you provide the following in your generalizability evaluation:

• an evaluation of parameters expected to introduce variability in the results (e.g., scanning characteristics, make and model of the imaging devices, acquisition protocol parameters such as contrast agent or probe positioning) and their effect on CADe performance;
• a description of the studies conducted; and
• the results and statistical analysis for your evaluation.

We also recommend that you identify the specific devices or device technologies and acquisition techniques that you use in validation testing and include the specific devices or device technologies, and acquisition techniques that are compatible with your CADe device in the product labeling.

5.5. Comparison with Predicate Device

A direct comparison of the standalone performances of your device and that of the predicate device (i.e., comparing performance using the same evaluation process and test data set) is recommended, if such a comparison is feasible. A direct comparison to a predicate device from another manufacturer may not always be possible. However, it is expected that, if you submit a 510(k) for a modified device based on your own device, you should have access to previously cleared versions of the device, and should therefore be able to perform this comparison. We recommend that you describe your comparison analysis, hypothesis to be tested, sample size estimation, and endpoints, and provide comparison results including CADe standalone performance measure such as:

• difference in area under the standalone FROC curves with associated statistical analysis (e.g., see Samuelson et al.¹³), or
• difference in detection standalone sensitivity and the number of FPs/case at the device operating points, and
• a comparison of the true positive and false positive CADe marks in both location (and extent when applicable) between the new or changed CADe device and the predicate (e.g., average percent of overlapping true positive marks prompted by both devices, average percent of overlapping false positive marks prompted by both devices, and an analysis of the size for the CADe marks produced by each device) when a direct comparison between devices is possible.

The goal of the latter comparison is to determine whether both the true positive and false positive CADe marks are consistent with the corresponding marks produced by the predicate device. We recommend that you evaluate this important additional factor because two CADe devices with similar standalone sensitivity and false positive rates may mark different structures within an image and this difference may result in a meaningful difference in clinical performance.

Again, the standalone data set should be large enough such that the study has adequate power to detect with statistical significance your proposed performance claims. Non-inferiority studies may be appropriate in some situations to facilitate comparisons. As mentioned above, the study should contain a sufficient number of cases from important cohorts (e.g., subsets defined by clinically relevant confounders, effect modifiers, and concomitant diseases) such that you can obtain standalone performance estimates and confidence intervals for these individual subsets (e.g., performance estimates for different nodule size categories when evaluating a lung CADe device). Powering these subsets for statistical significance should not be necessary unless you are making specific subset performance claims.

Reporting of standalone performance results may be guided by the FDA Guidance entitled Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.¹⁴

5.6. Electronic Data

We recommend submitting the data used in any statistical analysis in your study including patient information, disease or normal status, lesion size, lesion type, imaging and scanning setting, and imaging and data characteristics electronically (e.g., on a CD-ROM) when possible to streamline device review. For more information on submitting data electronically, please see the FDA white paper entitled Clinical Data for Premarket Submissions.¹⁵

16

Examples of changes to an already cleared CADe device for which we recommend submitting a clinical performance assessment include:

• the characteristics or makeup of the database used to assess standalone performance (see Section 5. Standalone Performance Assessment) cannot be demonstrated to be comparable to the characteristics or makeup of the database used in assessing the predicate device and these differences raise clinical concerns (i.e., could significantly affect safety or effectiveness);
• the results of the standalone performance assessment (see Section 5. Standalone Performance Assessment) are inferior to those of the predicate device;
• the location or extent of the CADe marks are substantially different from those of the predicate device, and these differences raise clinical concerns (i.e., could significantly affect safety or effectiveness);
• the reference standard definition, scoring process, analysis methodology, or performance endpoints are different from those of the predicate device, and the significance and effect on the clinician or patient of these differences are not well-known or well-described in the literature;
• the algorithm design is different from that of the predicate device and this difference raises clinical concerns (i.e., could significantly affect safety or effectiveness);
• a new acquisition technology or protocol is employed, changing the nature of the inputs to the CADe (e.g., the current CADe device is applied to digital radiographs whereas the predicate device was applied to film-based radiographs) and these differences raise clinical concerns (i.e., could significantly affect safety or effectiveness).

We recommend that a standalone performance assessment be included in your submission regardless of whether or not a clinical performance assessment is included.

6.1. Comparison with Predicate Device

We recommend you compare your clinical performance assessment results to those of the predicate device to which you are claiming substantial equivalence (e.g., a previously released version of the device) when a clinical assessment is necessary. A direct comparison to a predicate device may not always be possible. In this case, an alternative study could be a comparison to a control arm that establishes that the use of your device for its intended use, when used by the intended user and in accordance with its proposed labeling and instructions, leads to a statistically significant improvement in performance over that in the control arm (e.g., with CADe reading is statistical superior to without CADe reading). The performance results would then be compared to the corresponding results for a predicate that underwent a similar clinical assessment and showed a statistically significant improvement in performance over that of the control. This type of alternative clinical performance assessment study can be used when a direct comparison of clinical performance between your device and the predicate is not possible.

Reporting of clinical performance results may be guided by the FDA Guidance entitled Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.¹⁷

6.2. Electronic Data

We recommend submitting the data used in any statistical analysis in your clinical assessment study including patient information, disease or normal status, lesion size, lesion type, imaging and scanning setting, and imaging and data characteristics electronically¹⁸ (e.g., on a CD-ROM) when possible to streamline device review.

19

Your user manual should include the information described below.

8.1. Indications for use

We recommend that the indications for use (IFU) address how the device will be used, for example:

The device is intended to assist [target users] in their review of [patient/data characteristics] in the detection of [target disease/condition/abnormality] using [image type/technique and conditions of imaging].

8.2. Directions for use

There must be adequate directions for use as described in 21 CFR 801.5; the requirements applicable to prescription devices are described in 21 CFR 801.109. You should submit clear and concise instructions that delineate the technological features of the specific device and how the device is to be used on patient images/data. Instructions should encourage local/institutional training programs designed to familiarize clinicians with the features of the device and how to use it in a safe and effective manner. The direction should also clearly define the intended user of the device.

8.3. Warnings

The warnings should address limitations of the device. For example:

[target user] should not rely solely on the output identified by [device trade name], but should perform a full systematic review and interpretation of the entire patient dataset.

Another example may be:

This CADe device has been found to be ineffective for patients with [disease/condition/ abnormality]. This CADe should not be utilized with patients presenting with this [disease/condition/abnormality].

8.4. Precautions

The precautions should discuss the potential for adverse events associated with the use of the device and recommend mitigation measures. The adverse event discussion should at least include a discussion of potential adverse events associated with an increased workup rate (i.e., events from false-positives) and missed disease/condition/abnormality.

8.5. Device Description

We recommend you include the following in your device description:

• an overview of the algorithm design and features,
• an overview of the training paradigm and the training or development database,
• a description of the reference standard used for patient data utilized in the development and adjustment of the algorithm,
• specific devices or device technologies, and acquisition techniques compatible with your CADe device, and
• the requirements for appropriately displaying your CADe marks for devices designed as add-on software for general image review workstations and PACS.

8.6. Clinical Performance Assessment

When appropriate, we recommend you include a summary of the clinical performance assessment including:

• study objectives,
• study design,
• patient population, e.g., age, ethnicity, race,
• number of clinicians and their qualification,
• description of the methodology used in gathering clinical information,
• description of the statistical methods used to analyze the data, and
• study results.

Additional information on reporting clinical performance results can be found in the guidance entitled Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions.²⁰

8.7. Standalone Performance Assessment

We recommend you provide a summary of the device standalone performance and generalizability testing including:

• the scoring criteria used to determine the nature of each region marked by your CADe device;
• the overall standalone sensitivity and standalone false positive rate metrics at each available device operating point;
• stratified analysis (e.g., per lesion size, per lesion type, per imaging or scanning protocols, per imaging or data characteristics), as appropriate;
• the confidence intervals (CIs) on each measure; and
• the standalone FROC performance, as appropriate.

---

¹ The use of the acronym CADe for computer-assisted detection may not be a generally recognized acronym in the community at large. It is used here to identify the specific type of devices discussed in this document.

2Radiological Devices Panel – March 4 and 5, 2008

^{Radiological Devices Panel – November 17 and 18, 2009}

^{Guidance for Industry and FDA Staff: Format for Traditional and Abbreviated 510(k)s}

^{Guidance for Industry and FDA Staff – Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions}

^{Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices}

^{Guidance for Off-the-Shelf Software Use in Medical Devices}

^{Guidance for the Submission of Premarket Notifications for Medical Image Management Devices}

^{Establishment and Operation of Clinical Trial Data Monitoring Committees for Clinical Trial Sponsors}

^{Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests}

^{Clinical Data for Premarket Submissions}

^{Guidance for Industry and FDA Staff – Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions}

^{Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests}

¹⁸ See footnote 15.

^{Guidance for Industry and FDA Staff – Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions}