State-of-the-Art Review

Download PDF (509.85 KB)

TOMOGRAPHY, June 2020, Volume 6, Issue 2:60-64
DOI: 10.18383/j.tom.2019.00022

Clinical Trial Design and Development Work Group Within the Quantitative Imaging Network

Ella F. Jones1, John M. Buatti2, Hui-Kuo Shu3, Richard L. Wahl4, Brenda F. Kurland5, Hannah M. Linden6, David A. Mankoff7, Daniel L. Rubin8, Darrell Tata9, Robert J. Nordstrom9, Lubomir Hadjiyski10, Matthias Holdhoff11, Lawrence H. Schwartz12

1School of Medicine, University of California San Francisco, San Francisco, CA;2Carver College of Medicine, The University of Iowa, Iowa City, IA;3Winship Cancer Institute, Emory University, Atlanta, GA;4School of Medicine, Washington University, St. Louis, MO;5Department of Biostatistics, University of Pittsburgh, Pittsburgh, PA;6School of Medicine, University of Washington, Seattle, WA;7Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA;8School of Medicine, Stanford University, Stanford, CA;9Cancer Imaging Program, National Cancer Institute, Bethesda, MD;10Department of Radiology, University of Michigan, Ann Arbor, MI;11Sidney Kimmel Comprehensive Cancer Center, John Hopkins University, Baltimore, MD; and12Irving Medical Center, Columbia University, New York Presbyterian Hospital, New York, NY


The Clinical Trial Design and Development Working Group within the Quantitative Imaging Network focuses on providing support for the development, validation, and harmonization of quantitative imaging (QI) methods and tools for use in cancer clinical trials. In the past 10 years, the Group has been working in several areas to identify challenges and opportunities in clinical trials involving QI and radiation oncology. The Group has been working with Quantitative Imaging Network members and the Quantitative Imaging Biomarkers Alliance leadership to develop guidelines for standardizing the reporting of quantitative imaging. As a validation platform, the Group led a multireader study to test a semi-automated positron emission tomography quantification software. Clinical translation of QI tools cannot be possible without a continuing dialogue with clinical users. This article also highlights the outreach activities extended to cooperative groups and other organizations that promote the use of QI tools to support clinical decisions.


The Quantitative Imaging Network (QIN) program under the National Cancer Institute (NCI) is chartered to improve clinical decision-making by developing and validating quantitative imaging (QI) methods including data acquisition and analysis methods to support the measure and prediction of tumor response to therapy (1). The clinical focus of the QIN requires both the promotion and the broad dissemination of the aforementioned tools within the imaging and clinical oncology communities (2). To accomplish these goals, the QIN brought together multidisciplinary teams under the governance of the Executive Committee comprising QIN principal investigators and the Coordinating Committee with leaders from three working groups (WGs) (Figure 1). These WGs are tasked with addressing problems encountered in Clinical Trial Design and Development (CTDD), Bioinformatics and Data Sharing (BIDS), and Image Analysis and Performance Metrics (IAPM), with subgroups focusing on positron emission tomography computed tomography (PET-CT) and magnetic resonance imaging (MRI). In this article, we summarize the activities and challenges of the CTDD working group (CTDDWG) in the past 10 years.

Figure 1.

Quantitative Imaging Network (QIN) governance structure. The Executive Committee oversees the entire technical network, while the Coordinating Committee deals with working group (WG) issues. Each QIN technical team provides membership to all of the WGs, namely, Clinical Trial Design and Development and Data Acquisition, Bioinformatics and Data Sharing, and Image Analysis and Performance Metrics, with subgroups focusing on positron emission tomography-computed tomography (PET-CT) and magnetic resonance imaging (MRI).


The mission of the CTDDWG is to develop, validate, and harmonize methods and tools of quantitative imaging for use in cancer clinical trials to predict outcome and tumor response to therapy. This group specifically aims to:

  1. Identify challenges and opportunities in CTDD relevant to trials using QI.

  2. Identify best practices for QI-based clinical trial design, analysis, and reporting.

  3. Facilitate and introduce QIN-developed methods into cancer imaging trials through collaboration with other QIN WGs.

  4. Disseminate the best clinical trial design and development method through publications and guidelines.

  5. Facilitate outreach to cooperative groups and organizations to apply QIN methods in multicenter trials through cross-membership, education, and presentations.

  6. Translate relevant and mature QIN methods into clinical practice settings as appropriate.

Initial activities of the CTDDWG began in 2008 that involved a fledgling group of 5 institutions. Initial efforts sought to identify constraints of using QI in oncology trials that included a review published by Kurland et al. (3) to examine the potential constraints on clinical trials involving QI and strategies to overcome them. The group additionally explored barriers to effectively accrue patients to single- and multi-institutional trials.

The CTDDWG was active in presenting at national meetings such as American Society for Therapeutic Radiology and Oncology (ASTRO), American Association of Cancer Research (AACR), and the Radiological Society of North America (RSNA) to publicize the work of the QIN and the potential for these tools to impact clinical trials. A white paper was published in Clinical Cancer Research (4), and a letter to Cancer Center Directors was published in the Journal of Clinical Oncology (5).

Identify Challenges and Opportunities in Clinical Trial Design and Development

Patient accrual is essential for the success of clinical trials. With additional procedures and administration of imaging agents, trials involving the development of new QI methods/tools may face more challenges in patient recruitment. The CTDDWG surveyed 25 prospective clinical studies from 12 sites to evaluate the accrual performance, barriers, and solutions that are specific to QI-based trials (6). Our findings showed that the median percent annual accrual attained was 94.5% (range, 3%–350%). The most common barrier to recruitment was that “patients decline participation” (44%), followed by “too few eligible patients” (40%). Response from our survey indicated that interactions between institutions, physicians, and patients are essential factors for accrual success, with attention paid to ensuring and communicating potential trial benefits to enrolled and future patients.

Recent advances in radiation oncology have shown an increasing reliance on both high-resolution anatomic-based imaging (CT and MRI) and functional imaging (PET, diffusion-weighted imaging (DWI), magnetic resonance spectroscopic (MRS) imaging, etc.) for accurate radiation therapy delivery. The CTDDWG reviewed the utility of QI by CT, MRI, and PET/single-photon emission computed tomography (SPECT) with different tracers for radiation treatment planning and patient management (7). It was concluded that QI provides quantitative measures that complement the standard qualitative assessment methods. Further integration of QI into the radiation oncology workflow for treatment planning, risk stratification, guidance of dose escalation, and characterization of post-treatment effects is expected.

Identify Best Practices for Clinical Trial Design, Analysis, and Reporting

As QI methods are being increasingly used in clinical trials as primary, secondary, or correlative endpoints, standard criteria for QI reporting should be defined to ensure that quantitative data extracted from images are reported in a meaningful, consistent, and repeatable manner. The group led the effort of assembling a guideline of what is viewed as “best practices” for reporting (and potentially assessing) studies in which QI has been performed and reported. This guideline of standardizing the reporting of quantitative imaging (STIRQI) adapts the STAndards for Reporting of Diagnostic Accuracy (STARD) criteria first reported in 2003 (8, 9) and subsequently updated in 2015 (1012). Information regarding the purpose/background of the study, patient characteristics, image acquisition methodology, quantification methods, and statistical analysis should be described in detail. The outcome of the STIRQI is to ensure rigorous approaches are applied in QI studies so that results are reproducible and suitable for generalization to larger populations. Recommendation gathered from members of the CTDDWG and the leadership of the Quantitative Imaging Biomarkers Alliance (QIBA) group was drafted in the form of a white paper that will soon be published.

Translate Relevant and Mature QIN Methods into Clinical Practice Settings

PET Response Criteria in Solid Tumors (PERCIST) was initially proposed by Wahl et al. (13) as an approach to standardize interpretation of FDG-PET results using a consistent PET protocol. This group examined the variability for interpretation of FDG-PET results in 22 readers across 15 institutions on 30 test cases with scans before and after therapy using each institution's preferred software for this analysis. This study (14) showed a high correlation across readers and institutions but also identified some level of variability. Based on the PERCIST criteria, Wahl's team developed a semiautomated QI quantification software (AutoPERCIST™) to identify/measure reference tissue (liver), to set disease threshold values, and to calculate SUVs (standardized uptake values) (peak, max, mean, volume, and total lesion glycolysis). Using the CTDDWG as a testbed for tool validation, a multireader study was initiated. The main goal was to determine whether variance among readers and institutions was further reduced when AutoPERCIST™ was used. The latest version of this software was installed by 11 participating institutions (from United States, Asia, and Europe) through a material transfer agreement (MTA), and the 30 paired cases of anonymized FDG PET-CT images were downloaded for evaluation. Instructions for this study were given to each participating institution, with up to 5 tumor lesions from each PET image to be evaluated. All selections were recorded and sent to the central database of the Image Response Assessment Team at Johns Hopkins for quality control. Initial results show that consistent SUVpeak values were obtained, and the AutoPERCIST™ analysis had less variability than was seen when different institutional software was used. The initial results were presented at the Society of Nuclear Medicine and Molecular Imaging annual meeting (15), and a full manuscript was drafted and will soon be submitted for publication.

Outreach to Cooperative Groups and Organizations

The CTDDWG plays a key role in QIN outreach activities with the goal to raise awareness of the potential utility of advanced quantitative imaging for both research and general clinical practice. Several barriers (shown below) have limited the wider adoption of advanced quantitative imaging techniques in the clinic. The CTDDWG designed various panel discussions with oncology groups to educate trialists about the value of quantitative imaging and further promote the development and validation of new tools and methods needed to bring quantitative imaging into the clinical setting.

Learning Objectives

  1. Differentiate conventional qualitative from advanced quantitative imaging techniques and describe the added value of the quantitative methodologies to clinical management.

  2. Demonstrate knowledge of specific tools being developed by featured QIN research groups that will allow various advanced quantitative imaging techniques to be utilized on oncology clinical trials.

  3. Demonstrate an understanding of computing architecture/pipeline and informatic needs to fully exploit development of quantitative imaging methods/tools for clinical use.

Members from the CTDDWG took the lead to promote the collaborations with the National Clinical Trials Network (NCTN) cooperative groups, testing QI tools in the clinical trial settings in the Alliance, Children's Oncology Group (COG), Eastern Cooperative Oncology Group, and the American College of Radiology Imaging Network (ECOG-ACRIN), NRG Oncology, and Southwest Oncology Group (SWOG). In addition, special panels highlighting QIN tools and their potential use in clinical trials were organized and presented at the American Association for Physicists in Medicine (AAPM) and the American Society for Therapeutic Radiology and Oncology (ASTRO) annual meetings.

Pathways for Quantitative Imaging Tools to Clinical Translation

Bringing imaging tools and methods into clinical utility is not a simple task. Tools need be tested–retested, optimized, and validated to be clinically ready. In addition, open dialogues between tool developers and clinicians are needed to crystalize tool capabilities that should not only reliably serve specific clinical needs but also be easily incorporated into the clinical workflow and be compatible with existing data systems.

To date, there are 67 QIN imaging tools at various stages of development. Using a benchmark process for clinical-readiness: 1) prebenchmark; 2) basic benchmark; 3) technical test; 4) clinical trial and 5) clinical use, Farahani et al. identified approximately 12 tools to be qualified at levels 4 and 5 (16). These tools include software packages for tumor segmentation, volumetric measurement, metric calculation, and bioinformatics interface that are all important for disease assessment, management, and intervention. A representative of QIN tools being used in clinical trials is listed in Table 1.

Table 1.

Representation of QIN Tools Used in Clinical Trials

Trial Description Modality Measurement QIN Tool Application
NCT01015833 Phase III randomized study of sorafenib + doxorubicin vs sorafenib alone in patients with advanced hepatocellular carcinoma (HCC) MRI/CT Tumor volume/contour Tumor segmentation Treatment monitoring
NCT03137888 Spectroscopic MRI-guided radiation therapy planning in glioblastoma MRI Spectroscopic MR signal Web-based clinical interface for spectral MRI analysis, visualization, and integration into patient management Treatment planning
NCT01042379 I-SPY 2 TRIAL: Neoadjuvant and personalized adaptive novel agents to treat breast cancer MRI Functional tumor volume Tool kit for image processing, tissue segmentation, and tumor volume calculation Treatment monitoring
NCT02398733 [18F]Fluoroestradiol (FES) PET as a predictive measure for endocrine therapy in patients with newly diagnosed metastatic breast cancer PET Tracer uptake value Phantom and companion software for PET scanner calibration Imaging quality assurance and control (QA/QC)

To translate these sophisticated tools to the clinic, the QIN formed a team of “ambassadors,” with members from the CTDDWG reaching out to the NCTN cooperative groups to highlight the use of QI tools in treatment trials. In particular, a new membership was extended to ECOG-ACRIN with the intent of developing additional resources to support retrospective studies for tool testing and prospective studies for tool validations. Through the collaboration between the QIN members and the National Institute of Standards and Technology (NIST), imaging phantoms have been developed and commercialized to support image quality assurance and control (QA/QC) in clinical trials. All these collaborations and outreach activities have proven to be valuable as part of our effort in translating QI to clinical utility.

Future Directions

Looking forward, the CTDDWG members are working together to develop a clear roadmap and imaging workshops to provide education on how to include QI as part of the clinical trial. Other future directions also include a new collaboration with the Bioinformatics and Data Sharing Working Group (BIDSWG) to package QIN tools into open-source Docker containers ( in an automated pipeline infrastructure (17) to facilitate the connection, execution, and deployment of QIN tools into the clinical workflow. The BIDSWG is planning to work with the Imaging and Radiation Oncology Core (IROC) to test and validate the resulting pipeline. Through these activities, the CTDDWG will continue to facilitate the clinical translation of QI tools.


[1] Abbreviations:


quantitative imaging


Quantitative Imaging Network


working group


Clinical Trial Design and Development


positron emission tomography


computed tomography


magnetic resonance imaging


This work was supported in part by NIH/NCI U01CA151235 and U01CA225427 for EFJ; U01CA140206 for JMB; U01CA172027 for HKS; U01CA140204 for RLW; U01CA148131 and U01CA140230 for BFK; U01CA148131 for HML; U01CA142555, 1U01CA190214, 1U01CA187947, and U01CA242879 for DLR; U01CA179106 and U01CA232931 for LH; P30CA006973 for MH; and U01CA140207 for LHS.


    Yankeelov TE. The quantitative imaging network: a decade of achievement. Tomography. 2019;5:A8.
    Clarke LP, Nordstrom RJ, Zhang H, Tandon P, Zhang Y, Redmond G, Farahani K, Kelloff G, Henderson L, Shankar L, Deye J, Capala J, Jacobs P. The quantitative imaging network: NCI's historical perspective and planned goals. Transl Oncol. 2014;7:1–4.
    Kurland BF, Gerstner E, Mountz JM, Schwartz LH, Ryan CW, Graham MM, Buatti JM, Fennessy FM, Eikman EA, Kumar V, Forster KM, Wahl RL, Lieberman FS. Promise and pitfalls of quantitative imaging in oncology clinical trials. Magn Reson Imaging. 2012;30:1301–1312.
    Yankeelov TE, Mankoff DA, Schwartz LH, Lieberman FS, Buatti JM, Mountz JM, Erickson BJ, Fennessy FM, Huang W, Kalpathy-Cramer J, Wahl RL, Linden HM, Kinahan PE, Zhao B, Hylton NM, Gillies RJ, Clarke L, Nordstrom R, Rubin DL. Quantitative imaging in cancer clinical trials. Clin Cancer Res. 2016;22:284–290.
    Mountz JM, Yankeelov TE, Rubin DL, Buatti JM, Erikson BJ, Fennessy FM, Gillies RJ, Huang W, Jacobs MA, Kinahan PE, Laymon CM, Linden HM, Mankoff DA, Schwartz LH, Shim H, Wahl RL. Letter to cancer center directors: progress in quantitative imaging as a means to predict and/or measure tumor response in cancer therapy trials. J Clin Oncol. 2014;32:2115–2116.
    Kurland BF, Aggarwal S, Yankeelov TE, Gerstner ER, Mountz JM, Linden HM, Jones EF, Bodeker KL, Buatti JM. Accrual patterns for clinical studies involving quantitative imaging: results of an NCI Quantitative Imaging Network (QIN) survey. Tomography. 2016;2:276–282.
    Press RH, Shu HG, Shim H, Mountz JM, Kurland BF, Wahl RL, Jones EF, Hylton NM, Gerstner ER, Nordstrom RJ, Henderson L, Kurdziel KA, Vikram B, Jacobs MA, Holdhoff M, Taylor E, Jaffray DA, Schwartz LH, Mankoff DA, Kinahan PE, Linden HM, Lambin P, Dilling TJ, Rubin DL, Hadjiiski L, Buatti JM. The use of quantitative imaging in radiation oncology: a Quantitative Imaging Network (QIN) perspective. Int J Radiat Oncol Biol Phys. 2018;102:1219–1235.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HC; Standards for Reporting of Diagnostic Accuracy. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Radiology. 2003;226:24–28.
    Bossuyt PM, Reitsma JB; Standards for Reporting of Diagnostic Accuracy. The STARD initiative. Lancet. 2003;361:71.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HCW, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HCW, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology. 2015;277:826–832.
    Korevaar DA, Wang J, van Enst WA, Leeflang MM, Hooft L, Smidt N, Bossuyt PM. Reporting diagnostic accuracy studies: some improvements after 10 years of STARD. Radiology. 2015;274:781–789.
    Wahl RL, Jacene H, Kasamon Y, Lodge MA. From RECIST to PERCIST: evolving considerations for PET response criteria in solid tumors. J Nucl Med. 2009;50:122S–150S.
    Lodge MA, Holdhoff M, Leal JP, Bag AK, Nabors LB, Mintz A, Lesser GJ, Mankoff DA, Desai AS, Mountz JM, Lieberman FS, Fisher JD, Desideri S, Ye X, Grossman SA, Schiff D, Wahl RL. Repeatability of (18)F-FLT PET in a multicenter study of patients with high-grade glioma. J Nucl Med. 2017;58:393–398.
    O JH, Leal J, Wahl R. Quantitation of tumor SUL peak on 18F-FDG PET/CT: multicenter assessment using Auto-PERCIST. J Nucl Med. 2018;59:294.
    Farahani K, Tata D, Nordstrom RJ. QIN benchmarks for clinical translation of quantitative imaging tools. Tomography. 2019;5:1–6.
    John S, Rubin D, Gude D, Echegaray S, Bakr S, Mattonen S, Napel S. QIFP: a web application supporting quantitative medical imaging pipelines 2018 [cited 2020 January 09]. Available from:


Download the article PDF (509.85 KB)

Download the full issue PDF (12.51 MB)

Mobile-ready Flipbook

View the full issue as a flipbook (Desktop and Mobile-ready)