Diagnostic Performance of MagneticdntaPhilippe Maeder, MD,3 Paolo Galluzzi, MD,4 Herv J. Brisse, MD, PhD,5 Annette C. Moll, MD, PhD,6

Jonas A. Castelijns, MD, PhD,1 on behalf of the European Retinoblastoma Imaging Collaboration (ERIC)*

Purpose: To determine and compare the diagnostic performance of magnetic resonance imaging (MRI) andcomputed tomography (CT) for the diagnosis of tumor extent in advanced retinoblastoma, using histopathologicanalysis as the reference standard.

Design: Systematic review and meta-analysis.Participants: Patients with advanced retinoblastoma who underwent MRI, CT, or both for the detection of

tumor extent from published diagnostic accuracy studies.Methods: Medline and Embase were searched for literature published through April 2013 assessing the

diagnostic performance of MRI, CT, or both in detecting intraorbital and extraorbital tumor extension of retino-blastoma. Diagnostic accuracy data were extracted from included studies. Summary estimates were based on arandom effects model. Intrastudy and interstudy heterogeneity were analyzed.

Main Outcome Measures: Sensitivity and specicity of MRI and CT in detecting tumor extent.Results: Data of the following tumor-extent parameters were extracted: anterior eye segment involvement

and ciliary body, optic nerve, choroidal, and (extra)scleral invasion. Articles on MRI reported results of 591 eyesfrom 14 studies, and articles on CT yielded 257 eyes from 4 studies. The summary estimates with their 95%condence intervals (CIs) of the diagnostic accuracy of conventional MRI at detecting postlaminar optic nerve,choroidal, and scleral invasion showed sensitivities of 59% (95% CI, 37%e78%), 74% (95% CI, 52%e88%), and88% (95% CI, 20%e100%), respectively, and specicities of 94% (95% CI, 84%e98%), 72% (95% CI, 31%e94%), and 99% (95% CI, 86%e100%), respectively. Magnetic resonance imaging with a high (versus a low)image quality showed higher diagnostic accuracies for detection of prelaminar optic nerve and choroidal invasion,but these differences were not statistically signicant. Studies reporting the diagnostic accuracy of CT did notprovide enough data to perform any meta-analyses.

Conclusions: Magnetic resonance imaging is an important diagnostic tool for the detection of local tumor extentin advanced retinoblastoma, although its diagnostic accuracy shows room for improvement, especially with regard tosensitivity. With only a fewdmostly olddstudies, there is very little evidence on the diagnostic accuracy of CT, andgenerally these studies show low diagnostic accuracy. Future studies assessing the role of MRI in clinical decisionmaking in terms of prognostic value for advanced retinoblastoma are needed. Ophthalmology 2014;121:1109-1118 2014 by the American Academy of Ophthalmology.

*Supplemental material is available at www.aaojournal.org.

Retinoblastoma is the most frequent malignant ocular tumorin children, typically presenting in the rst years of life. Itrepresents approximately 3% of all pediatric malignancies,with an incidence of 1:17 000.1 Retinoblastoma can bediagnosed accurately by funduscopy and ultrasound, whichtypically demonstrates an intraocular vascularized andcalcied mass. Cross-sectional imaging is used primarilyfor local tumor staging (related to metastatic risk) anddepiction of associated intracranial primitive neuroectodermal

tumors (mostly pineoblastoma).2,3 Magnetic resonance im-aging (MRI) also can aid retinoblastoma diagnosis in case ofan unclear ocular medium. Treatment strategies are focusedon survival, preserving vision, and nally on avoidingenucleation. Although the gold standard for diagnosis andlocal staging relies on pathologic analysis, the treatmentstrategy frequently is based on clinical and radiologic nd-ings only, notably in eyes treated conservatively. In patientstreated by primary enucleation, extraocular extension rst

1109 2014 by the American Academy of Ophthalmology ISSN 0161-6420/14/$ - see front matterPublished by Elsevier Inc. http://dx.doi.org/10.1016/j.ophtha.2013.11.021Marcus C. de Jong, MD, MSc,1 Pim de Graaf, MD, PhD,1 Daniel P. Noij, BSc,1 Sophia Gricke, MD,2Resonance Imaging anTomography for AdvaA Systematic Review and MeComputedced Retinoblastoma-analysis

must be ruled out by imaging to avoid leaving behind tumor resolved by consensus. Studies were included if they met all of the

Ophthalmology Volume 121, Number 5, May 2014tissue after resection. Therefore, accurate local assessmentbased on imaging is critical at diagnosis. As recommendedcurrently, we perform MRI in all newly diagnosed retino-blastoma patients, but in the past, MRI often was performedas a second diagnostic step; we are not sure about the policiesin other institutions.4,5

The most important role of MRI is to aid in the decisioneither to treat an eye conservatively or to enucleate the eye.The prediction of high-risk features of retinoblastoma basedon clinical and radiologic features is pivotal in selecting thebest treatment option.6 Important risk factors for localrecurrence and metastasis are massive choroidal invasion,scleral invasion, optic nerve invasion (ONI) posterior tothe lamina cribrosa (especially if the surgical resectionmargin of the optic nerve is invaded), and involvement ofthe anterior eye segment (AES).6e12

Several studies of the diagnostic accuracy of MRI andonly a few of computed tomography (CT) for varioustumor-extent parameters have been published. Computedtomography has been considered important in detectingcalcications, but it recently was demonstrated that thecombination of ultrasound and MRI has the same sensitivityand specicity compared with CT in detecting intratumoralcalcications in retinoblastoma.13 Moreover, wheneverpossible CT should be avoided in young children becauseit poses a signicant radiation risk,14e17 and even more soin patients with hereditary retinoblastoma.4,13 However,because the incidence of retinoblastoma is low, study pop-ulations generally are small. There is also considerableheterogeneity (e.g., in terms of image quality) among thestudies. To overcome these problems, a critical systematicreview of the different studies is required.

The purpose of this study was to provide a completeoverview of available evidence and to determine andcompare the diagnostic performance of MRI and CT for thedetection of tumor extent in advanced retinoblastoma pa-tients, with histopathologic analysis as the referencestandard.

Methods

We performed this study according to the Preferred ReportingItems for Systematic Reviews and Meta-Analyses (PRISMA)statement.18,19

Search Strategy

We searched Medline (PubMed) and Embase for English, Dutch,German, and Spanish literature published through April 2013evaluating tumor extent of retinoblastoma by MRI or CT. We alsoincluded studies found in alternative ways (e.g., through checkingreferences in included studies). When necessary, we contactedauthors for additional data. The search included keywords corre-sponding to the index test MRI and CT, the target condition(retinoblastoma), and diagnostic performance (Appendix A,available at www.aaojournal.org).

Study Selection

Article titles and abstracts were reviewed independently for eligi-bility by 2 authors (M.C.J. and D.P.N.), and discrepancies were

1110following criteria: (1) the study population consisted of retino-blastoma patients; (2) the study assessed diagnostic performance ofMRI, CT, or both as a diagnostic test for tumor invasion into theocular wall, ONI, or AES involvement; (3) histopathologic analysiswas used as the reference standard test; and (4) if at least one pairof the absolute numbers of true-positive results and false-negativeresults, or true-negative results and false-positive results wereavailable or could be derived adequately. To include true-positiveresults, false-positive results, true-negative results, and false-negative results in a meta-analysis, all 4 should be available.Studies were excluded if they met one of the following criteria: (1)the article was a review or meta-analysis, and (2) (potentially)overlapping study populations were reported for the same outcome.

Diagnostic accuracy of tumor extension into the optic nerve canbe assessed in different ways. To avoid unclear denitions, in thisstudy we considered 3 categories: (1) if the optic nerve disc hasbeen invaded by tumor tissue (henceforth referred to as prelaminarONI), (2) invasion exactly into the lamina cribrosa (henceforthreferred to as intralaminar ONI), and (3) invasion posterior to thelamina cribrosa (henceforth referred to as postlaminar ONI).

Data Extraction

Two authors (M.C.J. and D.P.N.) independently extracted studydata. Discrepancies were resolved by consensus. When studiesreported multiple sets of sensitivities and specicities from multi-ple readers separately, the set with the highest diagnostic odds ratio(DOR) was used for the gures and meta-analysis. We did this toprevent data from the same patient population from being usedtwice. This could have resulted in an overestimation of the sum-mary estimates; therefore, we also reported the overall summaryestimates including sensitivity and specicity from the reader withthe lowest DOR.

Risk of Bias Assessment

Ideally, studies included only eyes that were primarily enucleated(i.e., retinoblastoma that was not previously treated) because reti-noblastoma treatment can inuence the appearance of tumor-extentparameters on MRI scans. However, some authors included bothsecondary and primary enucleations or did not mention this at all intheir article. We performed sensitivity analyses by assessing theeffect of analyzing the diagnostic performance of studies thatexplicitly state that they included only eyes that were primaryenucleations.

In an empirical study of bias in diagnostic tests, Lijmer et al20

demonstrated that study design can be very important to the resultsof diagnostic tests; they showed that a case-control design causedan overestimation of the DOR by 3 times, the use of differentreference tests overestimated the DOR twice, and not blinding thetest interpretation overestimated the DOR by 30%. However,verication bias, a nonconsecutive or random patient selection, anda retrospective study design did not seem to affect the DOR.

We used the Quality Assessment of Studies of Diagnostic Ac-curacy Included in Systematic Reviews (QUADAS-2) checklist toassess the study quality in terms of the risk of bias and the applica-bility of included studies.21,22 Two authors (M.C.J. and D.P.N.)independently assessed the study quality of the included articles.

Two authors (P.d.G. and H.J.B.) with 11 and 16 years expe-rience, respectively, in ocular MRI independently assessed theimage quality (low, intermediate, or high) of MRI and CT scansprovided in the articles, resulting in semiquantitative quality scores.The guidelines for imaging retinoblastoma by De Graaf et al4

served as a checklist for the quality assessment. Theyrecommended imaging of the eyes with a slice thickness of at

most 2 mm and a pixel size that is no larger than 0.50.5 mm. Toachieve this, it is advisable to use either a multichannel head coil(3.0-T system) or surface coils (1.5-T system). If information onslice thickness, pixel size, or both was not mentioned in the pub-lished articles, the included images served as a qualitative measurefor image quality. Proper sedation also is important to ensure ahigh image quality in the usually very young retinoblastoma pa-tient group. Discrepancies between image quality scores wereresolved by consensus.

Data Synthesis and Statistical Analysis

If tumor-extent parameters were analyzed in enough studies toallow statistical analysis, we analyzed data using a bivariaterandom effects regression model and provided summary esti-mates.23 This model assumes a binomial distribution of the within-study variability (i.e., the variability between sensitivity andspecicity within a study). The model assumes correlated normallydistributed random effects between studies. The inverse relationbetween sensitivity and specicitydwhen the positivity criterion isvarieddcorresponds to the degree of correlation between the logitsensitivity and logit specicity. We performed metaregression toexplore the effect of image quality on the diagnostic accuracy oftumor-extent parameters if sample sizes allowed this.

We summarized the data of each study and overall estimates inforest plots with a 95% condence interval (CI) of sensitivity and

specicity for each tumor-extent parameter. We also plotted thesenumbers in receiver operator characteristic (ROC) spaces showingthe summary estimates with a 95% condence region. Potentialoutliers have been identied in these ROC plots on the basis of theposition of individual studies relative to the other studies. Weperformed sensitivity analyses by excluding these potential out-liers, but these results should be interpreted with care.

We calculated the DOR along with sensitivity and specicity asan overall measure of diagnostic performance. The advantage ofDOR is its independence from disease prevalence and theapproximately normal distribution of the natural logarithm ofDOR.24 As a general rule, diagnostic tests with a DOR of morethan 25 are considered moderately accurate, and tests with aDOR of more than 100 are considered highly accurate.25,26

For the meta-analysis, we used the statistical software packageSAS version 9.3 (Proc NLMIXED; SAS Inc, Cary, NC). Wecreated the forest plots with Photoshop CS6 (Adobe, San Jose,CA). We used Cochranes Review Manager version 5.2 (Copen-hagen, Denmark) to create the ROC plots.

Results

Medline and Embase searches yielded 426 unique studies. Weexcluded 376 articles based on title and abstract (Fig 1). Weexcluded 32 studies based on the full text; see Figure 1 for

tomve re

De Jong et al Diagnostic Performance of Retinoblastoma ImagingFigure 1. Flowchart showing systematic literature search. CT computedmagnetic resonance imaging; TN true-negative results; TP true-positiography; FN false-negative results; FP false-positive results; MRI sults.

1111

reasons for exclusion. The study characteristics show considerable studies for scleral invasion explicitly reported data based on pri-mary enucleations only. Results based on this subset gave lower

ONI, and 2 of 10 studies assessing postlaminar ONI reported 2 setsof sensitivity and specicity from 2 readers. When data from the

reported using contrast enhancement in only 3 of 11 patients.

Ophthalmology Volume 121, Number 5, May 2014differences between the included studies (Table 1, available atwww.aaojournal.org). Eighteen studies met the inclusion criteriafor qualitative synthesis, and 13 studies were included in themeta-analysis. The study population described by Schueleret al27 overlapped with the more recent study by Lemke et al28;therefore, we extracted data only from the study by Schueleret al for tumor-extent parameters not included in the study byLemke et al (i.e., scleral invasion). All analyses were performed ona per-eye basis and not a per-patient basis because some studiesincluded 2 eyes from 1 patient (Table 1, available at www.aaojournal.org). Articles on MRI reported results of 591 eyesfrom 14 studies (excluding Schueler et al), and articles on CTyielded results of 257 eyes from 4 studies. The articles reportedages at diagnosis of retinoblastoma that ranged from a mean of11 months to a mean of 32 months. To avoid very large tables,we split the tumor-extent parameters into 3 groups: ONI, ocularwall invasion, and AES involvement and ciliary body invasion(Tables 2e4, available at www.aaojournal.org). Data on thediagnostic accuracy of MRI at detecting ONI, choroidal invasion,and scleral invasion proved to be sufcient for meta-analysis.

The sensitivities and specicities of tumor-extent parameters arepresented in Tables 2 through 4 (available at www.aaojournal.org).Forest plots show the sensitivities and specicities of each tumor-extent parameter, with their 95% CIs as horizontal lines sortedby sensitivity (Fig 2, available at www.aaojournal.org).

Summary Estimates

Compared with histopathologic analysis, the meta-analysis of MRIfor the different tumor-extent parameters gave us a sensitivity of73% (95% CI, 45%e90%), a specicity of 96% (95% CI, 64%e100%), and a DOR of 59.7 (95% CI, 4.36e818) for prelaminarONI from 8 studies (Table 5; Fig 2, available at www.aaojournal.org). For intralaminar ONI, MRI showed a sensitivityof 38% (95% CI, 0%e100%), a specicity of 89% (95% CI,47%e99%), and a DOR of 4.84 (95% CI, 0.00e3.14104) from4 studies. For postlaminar ONI, MRI demonstrated a sensitivityof 59% (95% CI, 37%e78%), a specicity of 94% (95% CI,84%e98%), and a DOR of 20.9 (95% CI, 4.75e92.2) from 10studies. For choroidal invasion, MRI showed a sensitivity of74% (95% CI, 52%e88%), a specicity of 72% (95% CI, 31%e94%), and a DOR of 7.38 (95% CI, 1.21e45.0) from 6 studies.For scleral invasion, MRI demonstrated a sensitivity of 88%(95% CI, 20%e100%), a specicity of 99% (95% CI, 86%e100%), and a DOR of 503 (95% CI, 24.9e1.02104) from 6studies. The ROC plots show the summary estimates ofspecicity on the x-axis and sensitivity on the y-axis for these 5tumor-extent parameters with their respective 95% condenceareas (Fig 3).

Sensitivity Analysis and Metaregression

Based on the ROC plots, we identied 2 studies as potential out-liers: the study by Wilson et al29 for postlaminar ONI and the studyby Khurana et al30 for choroidal invasion (Fig 3). After exclusionof these potential outliers, the diagnostic accuracy of MRI forpostlaminar ONI showed a sensitivity of 61% (95% CI, 32%e84%), a specicity of 96% (95% CI, 80%e99%), and a DOR of34.6 (95% CI, 5.80e206). For choroidal invasion, MRI showeda sensitivity of 77% (95% CI, 41%e94%), a specicity of 78%(95% CI, 46%e94%), and a DOR of 11.8 (95% CI, 2.14e65.4).

We performed a sensitivity analysis of including data fromcertain primary enucleations only. Four of 8 studies for prelaminarONI, 3 of 4 studies for intralaminar ONI, 7 of 10 studies forpostlaminar ONI, 4 of 6 studies for choroidal invasion, and 2 of 6

1112Ainbinder et al34 did not describe the reference test in theirstudy, leading to an unclear concern of the applicability (Fig 5).

Only 9 studies explicitly reported that patients were treated withprimary enucleation (i.e., without other retinoblastoma treatmentbefore enucleation), one study mentioned the use of preoperativechemotherapy in some patients but did not stratify the results, 2studies were unclear about this issue, and 6 studies did not reporton this at all (Table 1, available at www.aaojournal.org).

Discussion

We were able to perform a meta-analysis of the diagnosticaccuracy of MRI for detection of optic nerve, choroidal, andscleral invasion in retinoblastoma. Study data of extrascleralinvasion, AES involvement, and ciliary body invasionproved insufcient for meta-analysis. Studies reporting theworst reader (i.e., with the lowest DOR) were included in theanalysis, the results did not change much (Table 5).

For prelaminar ONI and choroidal invasion, the results frommetaregression analysis showed higher DORs for studies with ahigher image quality. The results for postlaminar ONI showed ahigher DOR only after removal of a potential outlier. However,none of these differences were statistically signicant (Table 6;Fig 2, available at www.aaojournal.org).

Risk of Bias Assessment

See Figures 4 and 5 for the QUADAS summary scores regardingrisk of bias and concern of applicability of the included studies.The entire list of QUADAS scores for each study is available inthe electronic supplement (Appendix B, available at www.aaojournal.org).

None of the included studies had a case-control design. Allstudies had the same reference standard because histopathologicanalysis was one of the inclusion criteria of this systematic review.The MRI interpretation was blinded in 10 studies, but this wasunclear in the other 8 studies. Histopathologic interpretation wasblinded in 8 studies, whereas this was unclear in 10 studies. Wescored 7 of 18 studies as having an appropriate interval betweenthe MRI and enucleation, and we scored the other 11 studies asunclear because they either did not report an interval at all orbecause they reported relatively wide ranges (Table 1 andAppendix B, available at www.aaojournal.org).

Only the most severe retinoblastoma cases were includedbecause enucleation is necessary for histopathologic assessment,but we do not know how this affects the applicability of the resultsfor patients with less severe retinoblastoma. Three studies raisedapplicability concerns for the index test: Lee et al31 report using 4different MRI scanners, Brisse et al32 reported using variousdifferent MRI and CT scanners, and John-Mikolajewski et al33DORs for most tumor-extent paramaters; only the DOR ofchoroidal invasion did not show much of a difference, and forscleral invasion, we could not calculate summary estimates(Table 5).

We also assessed the effect of including data from the readers ofthe MRI scans with the lowest DORdfrom studies that reportmultiple readersdon the summary estimates. Three of 8 studiesassessing prelaminar ONI, 1 of 4 studies assessing intralaminar