Real-Time Ultrawide Field ImageEvaluation of Retinopathy in aDiabetes Telemedicine ProgramDiabetes Care 2015381643ndash1649 | DOI 102337dc15-0161

OBJECTIVE

To evaluate the ability of trained nonphysician retinal imagers to perform diabeticretinopathy (DR) evaluation at the time of ultrawide field retinal (UWF) imagingin a teleophthalmology program

RESEARCH DESIGN AND METHODS

Clinic patients with diabetes received Joslin Vision Network protocol retinal imagingas part of their standard medical care Retinal imagers evaluated UWF images forreferable DR at the time of image capture Training of the imagers included 4 h ofstandardized didactic lectures and 12 h of guided image review Real-time evaluationswere compared with standard masked gradings performed at a centralized readingcenter

RESULTS

A total of 3978 eyes of 1989 consecutive patients were imaged and evaluated Byreading center evaluation 3769 eyes (947)were gradable for DR 1376 (365) hadDR and 580 (153) had referable DR Compared with the reading center real-timeimage evaluation had a sensitivity and specificity for identifying more than minimalDR of 095 (95 CI 094ndash097) and 084 (082ndash085) respectively and 099 (097ndash100)and 076 (075ndash078) respectively for detecting referableDROnly three patientswithreferable DR were not identified by imager evaluation

CONCLUSIONS

Point-of-care evaluation of UWF images by nonphysician imagers following standard-ized acquisition and evaluation protocols within an established teleophthalmologyprogram had good sensitivity and specificity for detection of DR and for identificationof referable retinal diseaseWith immediate image evaluationlt01of patientswithreferable DR would be missed reading center image grading burden would be re-duced by 60 and patient feedback would be expedited

Patients with diabetes require lifelong ophthalmic care that generally includes anannual retinal evaluation (1) Given the rapidly growing population affected bydiabetes a 20-year estimate of 27 million eyes worldwide will need to be eval-uated each day just to fulfill these needs (2) This enormous task is unlikely to beaccomplished by the current approaches of diabetes eye care programs Despitemore than one decade of research no real-time fully automated retinal imageanalysis system is currently in active clinical use (2) Until such capability existsother approaches to speed efficiency of current programs without sacrificing accu-racy are urgently needed

1Beetham Eye Institute Joslin Diabetes CenterBoston MA2Department of Ophthalmology Harvard Medi-cal School Boston MA3Teleophthalmology and Image Reading CenterPhilippine Eye Research Institute National Insti-tutes of Health University of the PhilippinesMa-nila Philippines4New England College of Optometry BostonMA

Corresponding author Paolo S Silva paoloantoniosilvajoslinharvardedu

Received 22 January 2015 and accepted 12 May2015

This article contains Supplementary Data onlineat httpcarediabetesjournalsorglookupsuppldoi102337dc15-0161-DC1

copy 2015 by the American Diabetes AssociationReaders may use this article as long as the workis properly cited the use is educational and notfor profit and the work is not altered

Paolo S Silva123 Jerry D Cavallerano12

Ann M Tolson1 Jessica Rodriguez1

Sashida Rodriguez1 Radwan Ajlan1

Dorothy Tolls1 Bina Patel14

Mina Sehizadeh1 Komal Thakore1

Jennifer K Sun12 and Lloyd Paul Aiello12

Diabetes Care Volume 38 September 2015 1643

CLIN

CAREED

UCATIO

NN

UTR

ITIONPSYC

HOSO

CIAL

The primary care or endocrinologyclinic is perhaps the ideal environmentfor the retinal imaging of patients withdiabetes Previous studies have estab-lished that people with diabetes willroutinely present to their primary carephysician or endocrinologist but thatonly 60 will adhere to the minimumrecommended annual eye care evalua-tion guidelines (3) Studies suggest thatophthalmic counseling during endocri-nology visits may improve diabetescontrol (4) Furthermore at presentcommercially available retinal imagingdevices require trained personnel for re-cording patient information and acquir-ing retinal imagesStudies have shown that accurate as-

sessment of vision-threatening diabeticretinopathy (DR) can be determined atthe time of nonmydriatic 308 and 458retinal photography by trained and cer-tified retinal imagers in a DR telemedi-cine program (5) At the Joslin DiabetesCenter all telemedicine retinal photog-raphy transitioned to ultrawide field im-aging in 2012 Ultrawide field retinalimaging (UWFI) uses scanning laserophthalmoscopy and an ellipsoidal mir-ror to allow the nonmydriatic acquisi-tion of high-resolution retinal imagesthat encompass more than double thetotal retinal surface area captured withmydriatic standard 308 7-field EarlyTreatment Diabetic Retinopathy Study(ETDRS) photography The image acqui-sition time with UWFI is less than one-half that of ETDRS photography evenwhen the time for dilation is excluded(6) Independent groups have demon-strated that UWFI compares favorablywith dilated retinal examination by aretina specialist ETDRS photographyand various retinal imaging protocolsfor determining DR severity (67) Imple-mentation of UWFI reduces the ungrad-able rate by 71 for DR and 56 fordiabetic macular edema (DME) to 3and 4 respectively (8) Comparedwith traditional nonmydriatic fundusphotography image evaluation timehas been reduced by 28 (8) Further-more UWFI has identified additional pe-ripheral retinal lesions suggesting amore severe level of DR in approxi-mately 10 of eyes (89) Given thesepotential efficiency benefits we pro-spectively evaluated the ability oftrained imagers to perform ultrawidefield point-of-care DR evaluations to

determine the presence or absenceof DR or referable DR at the time ofimaging

RESEARCH DESIGN AND METHODS

The Joslin Vision Network (JVN) is an ocu-lar telehealth program for DR developedat the Joslin Diabetes Center and has beenin continuous clinical operation since 1998(1011) The JVN follows a strict protocolfor acquiring nonmydriatic retinal imagesand for grading and reporting the level ofDR Patients with diabetes have non-mydriatic JVN imaging as part of their rou-tine physical examinations at the AdultDiabetes Clinic of the Joslin Diabetes Cen-ter Images are acquiredusingnonmydriaticultrawide field scanning laser retinal im-aging (Optos P200MA and P200C Optosplc Dunfermline Fife UK) The protocolfor retinal imaging has been previouslydescribed (68) and involves the acquisi-tion of nonsimultaneous stereoscopic1008 and 2008 ultrawide field image pairscentered on the macula of each eye Allimages are acquired through undilatedpupils (see Supplementary Fig 1 for thestudy protocol flowchart)

At the time of imaging JVN imagersprospectively performed two levels ofDR assessment of the ultrawide field im-ages 1) American Telemedicine Associ-ation (ATA) category 1 assessment toidentify patients with no or minimalDR (ETDRS level 20) versus thosewith DR more severe than ETDRS level20 and 2) ATA category 2 assessment toidentify patients with referable DR(moderate or worse levels of nonproli-ferative DR [NPDR] [ETDRS level 43]proliferative DR [PDR] [ETDRS level61] or any level of DME) (12) Imagerswere permitted to manipulate the colorbrightness contrast gamma or magnifi-cation of the images but did not view im-ages stereoscopically To suspect retinalthickening without stereoscopic viewingimagers relied on identifying hard exu-dates or microaneurysms within 3000mm from the center of the macula as sur-rogate markers for DME Reading centerevaluationofDMEwasperformed stereo-scopically on all images

One of the trained JVN imagers wasa college graduate with a bachelor ofarts degree in psychology and the sec-ond was a certified medical assistantNeither imager had prior ophthalmol-ogy or eye care experience other thannonmydriatic retinal imaging Both

imagers had been acquiring ultrawidefield images since 1 April 2012 but nei-ther had evaluated retinal images at thereading center The study was con-ducted from 1 October 2013 to 30 Sep-tember 2014 One imager had 35 yearsand the other 6 months of prior retinalimaging experience using a TopconTRC-NW6S (Topcon Medical Systems IncOakland NJ) nonmydriatic fundus cameraBoth imagers had 18 months of UWFI ex-perience using the Optos P200MA andP200C devices Each retinal imagerreceived a validated standardized methodof certification and training that included4 h of didactic lectures and 12 h of guidedreview of DR images Before the initiationof this study imagers underwent a1-month provisional period during whichquestionable assessments were reviewedby the supervising retina specialist witheach imager Monthly 1-h meetings wereconvened to discuss questionable retinalimages and findings under the supervisionof a retina specialist or the senior retinalspecializing optometrist For the purposesof this study the assessment of the imagerwas not changed after discussions with theretina specialists

All images were subsequently gradedindependently by trained optometristscertified in JVN grading at a centralizedreading center and masked to the im-agersrsquo evaluation All reading center eval-uations were overread by a supervisingretinal physician who was also maskedto imager evaluations Identical dual-monitor workstations were used at boththe imaging and the reading stations Thespecifications of the reading center dis-plays have been reported previously andare calibrated biannually (8) All findingswere recorded on specifically designedtemplates The detailed protocol for eval-uating ultrawide field images has beenpreviously described and has shown sub-stantial agreement with the grading ofstandard dilated 7-field ETDRS photogra-phy (6) For both imagers and reading cen-ter graders an ungradable image for DRwas defined as inadequate photographicquality or media opacity preventing thedetermination of the presence or absenceof DR If at least one or more disc areas ofretina were visible in each ultrawide fieldndashequivalent ETDRS-defined photographicfield and that area was free of DR lesions(hemorrhages andor microaneurysms[HMa] venous beading intraretinal mi-crovascular abnormalities [IRMA] new

1644 Real-Time Evaluation of Diabetic Retinopathy Diabetes Care Volume 38 September 2015

vessels on the disc new vessels elsewherein the retina) the grading was deemedDR absent If DR lesions were present inthe unobscured part of the field DR wasrecorded as present regardless of the ex-tent of the field obscured and that sever-ity of DR lesions was assumed to existunder the obscured portions of the fieldBoth the imagers and the reading centergraders were able to magnify and adjustthe image color contrast brightness andgamma correction for each imageThe study design was consistent with

the tenets of the Declaration of Helsinkiand the Committee on Human Studies ofthe Joslin Diabetes Center approved theresearch protocol The conduct of thestudy complied with the Health InsurancePortability and Accountability Act

Statistical AnalysisNonparametric analyses (Wilcoxon ranksums) and Pearson correlations wereused to compare distributions of contin-uous variables between groups The x2

test was used to compare frequencies ofcategorical variables When DR severitywas evaluated per patient rather thanper eye the more severe level of DRand DME present in either eye was usedas the severity level present in the pa-tient If one eye was ungradable the levelof DR and DME present in the gradableeye was considered the severity level ofDR and DME present in the patient Thepresence or absence of DR (ATA category1 evaluation) and the presence or ab-sence of referable DR were tested usingsensitivity and specificity percentagesand positive and negative predictive val-ues Eyes classified as ungradable wereexcluded from the analysis All analyseswere performed using SAS version 93software (SAS Institute Inc Carey NC)

RESULTS

During the study period 3978 eyes of1989 consecutive patients were evalu-ated The mean age was 516 6 174(range 18ndash95) years mean duration ofdiabetes was 137 6 109 (0ndash64) yearsand 451 were female and 781white The distribution of DR severitybased on the reading center evaluationis presented per eye and per patient inTable 1

Identification of Ungradable ImagesThe identification of ungradable images isintegrated into the protocol of image ac-quisition because suboptimal images are

retaken immediately The rate of ungrad-able images for identifying the presenceof DR as determined by the point-of-careimagers was 21 per eye and 09 perpatient Images deemed ungradable forDR severity at the reading center were53 per eye and 29 per patient

Identification of RetinopathyBased on masked standardized readingcenter evaluation more than minimalDRwas present in 885 eyes (235) Basedon imager evaluation the sensitivity fordetermining the presence of more thanminimal DR was 095 (95 CI 094ndash097)and the specificity was 084 (082ndash085)Imager evaluation for DR resulted in afalse-negative determination for 43 eyes(11) with a false-negative rate [falsenegative (true positive + false negative)]of 016 Of these 31 (721) had mildNPDR 5 (116) had moderate NPDR 2(47) had macular edema and 5 (116)had DR but the severity could not be de-termined In terms of DR on a per-patientlevel imager grading had a sensitivity of096 (094ndash098) and specificity of 077(075ndash080) Imager evaluation resultedin a false-negative determination for 21(11) patients with a false-negativerate of 005 Of these 13 (619) hadmild NPDR 1 (48) had moderateNPDR 2 (95) had mild NPDR and mac-ular edema and 5 (238) hadDR but theseverity could not be determined Asummary of sensitivity specificity true-positive false-positive true-negative andfalse-negative results at both the eye andthe patient level is presented in Table 2

Individual ultrawidefield images of all eyeswith false-negative results with annotationof specificDR lesions arepresented in Fig 1The lesions resulting in false-negative re-ferable DR findings were primarily subtleearly IRMA less than ETDRS standard pho-tograph 8A (five eyes) and hard exudatesless than ETDRS standard photograph 3(three eyes)

Identification of ReferableRetinopathyBased on masked standardized readingcenter evaluation referable DR was pres-ent in 372 eyes (159) Based on imagerevaluation the sensitivity for determiningthepresenceof referableDRwas099 (95CI 097ndash100) and specificity 076 (075ndash078) Seven (19) eyes were identifiedas having false-negative results by imagerevaluation for referable DR ModerateNPDR was present in five eyes and twoeyes had macular edema Considering thepresence of referable DR on a per-patientlevel sensitivity was 099 (097ndash100) andspecificity 069 (066ndash071) Three (16)patients were identified as having false-negative results by imager evaluation forreferable DR one with moderate NPDRand two with macular edema A summaryof these results at both the eye and thepatient levels is presented in Table 2

Comparison With Existing AutomatedAlgorithmsTable 3 relates the results of point-of-careimager grading for the identification ofDR (1314) and referable DR (15) to priorpublications using commercially available

Table 1mdashGradable rates and distribution of DR severity

Per eye (n = 3978) Per patient (n = 1989)

Gradable for severity of DR at reading center 3769 (947) 1931 (971)

Gradable for presence of DME at reading center 3704 (931) 1912 (961)

Presence of referable DR at reading center 580 (153) 344 (178)

DR severityNo DR 2393 (635) 1109 (574)Very mild NPDR 491 (130) 304 (157)Mild NPDR 362 (96) 217 (112)Moderate NPDR 230 (61) 131 (68)Severe NPDR 63 (17) 37 (19)Very severe NPDR 4 (01) 2 (01)PDR 214 (57) 124 (64)PDR with high-risk characteristics 9 (02) 7 (04)

DME present 262 (71) 178 (93)

Data are n () Referable DR is defined as moderate NPDR or worse PDR or presence of DMEWhen DR severity was evaluated per patient rather than per eye the more severe level of DRand DME present in either eye was used as the severity present in the patient If one eye wasungradable the level of DR and DME present in the gradable eye was considered the level of DRand DME present in the patient

carediabetesjournalsorg Silva and Associates 1645

automated algorithms The imager point-of-care evaluation compared favorably inall respects even though the currentstudy population of 3978 eyes wassmaller than the number tested by theother modalities Sensitivity specificityand negative and positive predictive val-ues were similar to or better than thosereported for the automated algorithms(13ndash15) However different methods ofmanual and automated grading as wellas different image sets were used bythe various studies so these compari-sons are presented as a reference andgenerally do not represent comparativeperformance

Image Evaluation Time at the Time ofImagingThe research study was conducted in anactive telemedicine program that evalu-ates nearly 4000 eyes each year (8) andwhere accurate measurement of imageevaluation time by the imagers is not rou-tinely feasible However in a subset of118 consecutive patients imager evalua-tion was specifically timed without anyclinical interruptions Imagerswere awarethat they were being timed and were in-structed to complete the image evalua-tion without interruption immediatelyfollowing the conclusion of the patientencounter Image evaluation time was re-corded electronically for the followingsteps image retrieval image display

image evaluation and recording of find-ings on the specifically designed templateAt least eight images from each patientcomprising 1008 and 2008 image pairs ofeach eye were evaluated (6816) Themean point-of-care image evaluationtime by retinal imagers was 7776 396 sOf note the imagers evaluated imagesonly for the presence or absence of DRindividual DR lesions and the ETDRS se-verity of DR were neither assessed norrecorded by the imagers Furthermorethe environment where the imageswere evaluated was optimized for read-ing ultrawide field images as described inRESEARCH DESIGN AND METHODS

CONCLUSIONS

Point-of-care DR evaluation at the timeof UWFI by trained and certified non-physician retinal imagers can reliablydifferentiate no to mild disease frommore severe disease and can reliablyidentify referable DR This approachmay permit effective triage of patientswith minimal retinopathy to less urgentfollow-up and of those with referabledisease to prompt ophthalmic careEyes with no or minimal DR may not re-quire further evaluation by a centralizedreading center thereby potentially re-ducing reading center load by 60In addition at least for patients withno or minimal DR results might be pro-vided at the conclusion of the imaging

session speeding patient feedbackSuch a process might substantially in-crease efficiency of DR telemedicineprograms to meet the huge ongoingmedical need until real-time automatedimage grading algorithms are effectivelyincorporated into DR care

It is important that telemedicine pro-grams for DR conform to the recommen-ded minimum 80 sensitivity and 95specificity for detecting mild retinopathyThe methods described here do notreplace a reading center evaluation butrather optimize the current workflow byreducing the overall number of imagesneeding to be evaluated by the readingcenter by using imagers to provide a real-time initial assessment on the presenceorabsence ofDR In thismodel it is essentialfor patient safety that sensitivity ap-proach 100 to ensure that all patientswith disease are accurately identified Inmost diabetes populations 50 of pa-tients will have no or minimal disease(ETDRS level 20) Imager grading atthe point of care can potentially identifythese cases with a sensitivity that ap-proaches 100 Eyes identified by the im-agers as having no or minimal DR wouldnot require further reading center evalu-ation Eyes identified by the imagers withETDRS level$20 would be further evalu-ated This model optimizes the workflowby reducing the number of images evalu-ated maintaining a very high sensitivity

Table 2mdashSummary statistics per eye and per patient for point-of-care image evaluation for identification of DR and referableDR

ATA category 1 evaluation (no or minimal DR vs more than minimal DR) Referable DR

Per eye(n = 3978)

Per patientdagger(n = 1989)

Per eye(n = 3978)

Per patientdagger(n = 1989)

Total gradable imagesDagger 3799 (955) 1944 (977)sect 3758 (945) 1926 (964)kSensitivity 095 (094ndash097) 096 (094ndash098) 099 (097ndash099) 099 (097ndash100)

Specificity 084 (082ndash085) 077 (075ndash080) 076 (075ndash078) 069 (066ndash071)

PPV 066 (063ndash068) 063 (060ndash066) 043 (033ndash037) 040 (037ndash044)

NPV 098 (098ndash099) 098 (097ndash099) 100 (099ndash100) 100 (099ndash100)

FDR 034 (032ndash037) 037 (034ndash040) 057 (054ndash060) 059 (055ndash063)

FOR 002 (001ndash002) 002 (001ndash003) 0002 (000ndash001) 0002 (000ndash001)

FPR 016 (015ndash018) 022 (020ndash024) 024 (022ndash025) 031 (029ndash033)

FNR 005 (004ndash007) 004 (003ndash006) 001 (001ndash003) 001 (000ndash003)

Data are n () or n (95 CI) FDR false-discovery rate FNR false-negative rate FOR false-omission rate FPR false-positive rate NPV negativepredictive value PPV positive predictive value Referable DR is defined as moderate NPDR or worse PDR or presence of DME daggerWhen DR severitywas evaluated per patient rather than per eye the more severe level of DR and DME present in either eye was used as the severity present in thepatient If one eye was ungradable the level of DR and DME present in the gradable eye was considered the level of DR and DME present in thepatient DaggerImages gradable at both the point-of-care evaluation by retina imagers and the reading center sectIn 41 eyes the presence of DR wasgradable but the severity of DR could not be determined (eg definite signs of DR such as HMA but the disc andor macula were obscured or theimage quality in one or more quadrants did not allow for assessment of retinal lesions) kIn eight patients the presence of DR was gradable but theseverity of DR could not be determined (eg definite signs of DR such as HMA but the disc andor macula were obscured or image quality in one ormore quadrants did not allow for assessment of retinal lesions)

1646 Real-Time Evaluation of Diabetic Retinopathy Diabetes Care Volume 38 September 2015

and allowing the reading center to fullyevaluate all cases identified as DR Thegrading center sensitivity and specificityfor grading nonmydriatic ultrawide fieldimages compared with mydriatic 7-fieldETDRS photography for DR have been re-ported and exceed the current recommen-dations (ETDRS level 20 sensitivity 099 and

specificity 100 ETDRS level$43 sensitivity095 and specificity 094) (7) The agree-ment of nonmydriatic UWFI and gold stan-dard mydriatic 7-field ETDRS photographyhas been published by independent groups(67) There is near-perfect agreement indetermining DR severity (weighted k 085)and good agreement in DME severity

(weighted k 066) There is comparativelyless agreement of UWFI in evaluation ofDME severity but the rate meets or ex-ceedspublishedagreement ratesusingmul-tiple imaging devices and methodologies

Imagers did not evaluate for specificDR severity or each individual DR lesionbut rather provided an assessment onlyof the presence or absence of DR whichmay reflect the less critical assessmentfor gradability by the imagers The un-gradable rate per patient (29 vs 29)and per eye (54 vs 53) remainedconstant in this study that evaluatedall imaging encounters from 1 October2013 to 30 September 2014 comparedwith a previous study that evaluated allimaging encounters from 1 April 2012 to1 November 2012 (6)

In the current study imagers did notassess the severity of macular edema orthe presence or severity of individualDR lesions Given that the speed of assess-ment is important they only assessedwhether images were potentially grad-able whether DR was present andwhether referable DR was present How-ever we have previously reported on theindividual lesion sensitivity and specificitycomparing nonmydriatic ultrawide fieldand ETDRS photography finding that sen-sitivity is least for small IRMA and newvessels (7)

Key issues for DR telemedicine pro-grams include lowering the rates offalse-positive and false-negative imageevaluations Lowering the false-positiverate reduces burden onmore formal read-ing resources and specialist care if theseverity warrants a referral Loweringfalse-negative rates prevents the pro-gram from not appropriately referring apatient for care when it is truly neededOf the two the false-negative rate ismore important from the patient carestandpoint In this study the false-negativerate of point-of-care ultrawide fieldimage evaluation was 001 for boththe presence of referable DR per eyeand the presence of referable DR perpatient These low rates suggest thatif point-of-care grading was per-formed and used for referral for every500 eyes evaluated in this manner 1eye with referable disease would not beidentified and triaged to prompt care Nocases of severe NPDR PDR or clinicallysignificant macular edema or center-involved DME were missed by this man-ner of evaluation at the point of care

Figure 1mdashImages from all seven eyes with false-negative point-of-care results for referable DR(defined as reading center evidence of moderate NPDR or worse or any level of DME) comparedwith standard masked reading center evaluation Each ultrawide field image is inlaid witha corresponding magnified section of the boxed area showing the missed lesion Solid arrow-heads indicate IRMA and open arrowheads indicate hard exudates within 1500mmof the fovealcenter AndashE Eyes with moderate NPDR due to IRMA less than ETDRS standard photograph 8AF and G Eyes with hard exudates less than ETDRS standard photograph 3

carediabetesjournalsorg Silva and Associates 1647

and allowing the reading center to fullyevaluate all cases identified as DR Thegrading center sensitivity and specificityfor grading nonmydriatic ultrawide fieldimages compared with mydriatic 7-fieldETDRS photography for DR have been re-ported and exceed the current recommen-dations (ETDRS level 20 sensitivity 099 and

specificity 100 ETDRS level$43 sensitivity095 and specificity 094) (7) The agree-ment of nonmydriatic UWFI and gold stan-dard mydriatic 7-field ETDRS photographyhas been published by independent groups(67) There is near-perfect agreement indetermining DR severity (weighted k 085)and good agreement in DME severity

(weighted k 066) There is comparativelyless agreement of UWFI in evaluation ofDME severity but the rate meets or ex-ceedspublishedagreement ratesusingmul-tiple imaging devices and methodologies

Imagers did not evaluate for specificDR severity or each individual DR lesionbut rather provided an assessment onlyof the presence or absence of DR whichmay reflect the less critical assessmentfor gradability by the imagers The un-gradable rate per patient (29 vs 29)and per eye (54 vs 53) remainedconstant in this study that evaluatedall imaging encounters from 1 October2013 to 30 September 2014 comparedwith a previous study that evaluated allimaging encounters from 1 April 2012 to1 November 2012 (6)

In the current study imagers did notassess the severity of macular edema orthe presence or severity of individualDR lesions Given that the speed of assess-ment is important they only assessedwhether images were potentially grad-able whether DR was present andwhether referable DR was present How-ever we have previously reported on theindividual lesion sensitivity and specificitycomparing nonmydriatic ultrawide fieldand ETDRS photography finding that sen-sitivity is least for small IRMA and newvessels (7)

Key issues for DR telemedicine pro-grams include lowering the rates offalse-positive and false-negative imageevaluations Lowering the false-positiverate reduces burden onmore formal read-ing resources and specialist care if theseverity warrants a referral Loweringfalse-negative rates prevents the pro-gram from not appropriately referring apatient for care when it is truly neededOf the two the false-negative rate ismore important from the patient carestandpoint In this study the false-negativerate of point-of-care ultrawide fieldimage evaluation was 001 for boththe presence of referable DR per eyeand the presence of referable DR perpatient These low rates suggest thatif point-of-care grading was per-formed and used for referral for every500 eyes evaluated in this manner 1eye with referable disease would not beidentified and triaged to prompt care Nocases of severe NPDR PDR or clinicallysignificant macular edema or center-involved DME were missed by this man-ner of evaluation at the point of care

Figure 1mdashImages from all seven eyes with false-negative point-of-care results for referable DR(defined as reading center evidence of moderate NPDR or worse or any level of DME) comparedwith standard masked reading center evaluation Each ultrawide field image is inlaid witha corresponding magnified section of the boxed area showing the missed lesion Solid arrow-heads indicate IRMA and open arrowheads indicate hard exudates within 1500mmof the fovealcenter AndashE Eyes with moderate NPDR due to IRMA less than ETDRS standard photograph 8AF and G Eyes with hard exudates less than ETDRS standard photograph 3

carediabetesjournalsorg Silva and Associates 1647

Although the assessment of the retinalimagers had lower specificity and lowerpositive predictive value than formalgrading this finding was incorporatedinto the design of the methodology toensure that sensitivity values would ap-proach 100 and thus ensure that no pa-tient with retinopathy was missed Allpositively identified eyes were subse-quently fully evaluated by the readingcenter Despite the relatively lower spec-ificity and positive predictive value in thiscohort 576 of patients and 636 ofeyes had no DR whereas 153 of pa-tients and 178 of eyes did not have re-ferable DR As a result reading centerburden could be reduced by 60 and15 if images with no DR or no referableDR respectively are excluded from theoverall reading queue Extrapolated tothe entire US diabetic population thiswould reduce the reading center gradingburden by78ndash31million eyes annuallyThis study was not designed for cost-

effectiveness analysis however the maincost differential between UWFI and stan-dard retinal photography is the cost of theimaging device In large populations theefficiency operational and patient carebenefits may outweigh the increasedcosts of the UWFI device Furthermorethe costs of imaging devices are likely todecrease over time with further techno-logical innovations and market competi-tion Standard nonmydriatic fundusphotography requires trained retinal pho-tographers to ensure high-quality retinalimages adequate for retinal evaluationHowever even with dedicated retinalphotographers a comparatively higherungradable ratewhichhas been reportedto bebetween 13and 54 continues torepresent a significant barrier to the ap-propriate identification of disease In pri-mary care and nonophthalmic settingsthe relative ease of UWFI compared

with nonmydriatic fundus photographyprovides a significant advantage At theJoslin Diabetes Center 20000 patientsare imaged over a 5-year period A com-parison of the $100000 cost of the ul-trawide field device with a $25000traditional fundus camera spread overthe 5 years results in an additional costper patient of $375 Given the70 re-duction in the ungradable rate20 re-duction in grading time improved ease ofuse and increased DR identification thisincreased cost per patient may be out-weighed by the significant benefits pro-vided by UWFI

Multiple benefits may result from im-mediate point-of-care imager gradingWe previously demonstrated that un-gradable images can be readily identi-fied by the imager allowing immediaterecapture of images to improve qualitybefore the patient telemedicine en-counter ends (5) The current studydemonstrates that the presence ofmore than minimal DR as well as re-ferable DR can be comprehensivelyidentified Thus urgent referrals toappropriate care could be initiated rap-idly Patients without DR are also identi-fied and a report could be generated atthe end of the telemedicine sessioneliminating the need for further imageevaluation of patients without diseaseThe 2011 introduction of telemedicineCurrent Procedure Terminology codes(92227 remote diagnostic retinal imag-ing) that are now standard in the 2013Current Procedure Terminology FourthEdition (wwwcmsgovappsphysician-fee-schedulesearchsearch-resultsaspxY=0ampT=0ampHT=0ampCT=3ampH1=92227ampM=5) by the Centers for Medicare ampMedicaid Services allows a patient re-port to be generated under physician su-pervision only if no disease is presentthus potentially supporting this approach

(17) In addition a preliminary report re-garding the presence or absence of DRcould be provided to the patients at theconclusion of the point-of-care imagingsession This approach would allow forimmediate highly structured patient-specific education potentially enhanc-ing the value of the patient encounterfrom both an ophthalmic and a medicalperspective

There are limitations to this study re-garding generalizability of the resultsRetinal imagers were under the directsupervision of a retina specialist andreceived a standardized method of cer-tification and training Whether addi-tional training might have improvedoutcomes or how differences in thetraining approach would alter results isunknown Furthermore the investmentin training and ongoing oversight mustbe considered In this study imagerpoint-of-care ultrawide field evaluationcompared favorably in all respects topublished outcomes of automatedgrading algorithms However differentmethods of manual and automatedgrading as well as different image setswere used by the various studiesWhether the small performance differ-ences between the methodologies maybe attributed to the accuracy of eachmethod or related to the different im-age sets or approaches studied is un-known Although implementation ofhighly sensitive and specific auto-mated grading algorithms would beideal to address the burden of DR eval-uation until such ideals are realizedthe approach detailed in the currentstudy represents an accurate and viablealternative

In summary this study demonstratesthat appropriately trained and certifiedimagers following a defined imagingand grading protocol can accurately

Table 3mdashResults of point-of-care imager grading for the identification of DR and referable DR to prior publications ofcommercially available automated algorithms

Point-of-care imager evaluation iGradingM IDx-DRdagger RetmarkerDRDagger

Sensitivity 099 (097ndash099) 098 (097ndash099) 097 (094ndash099) 096 (094ndash098)

Specificity 076 (075ndash078) 041 059 (557ndash630) 052 (050ndash053)

PPV 043 (033ndash037) d 049 (035ndash044) d

NPV 100 (099ndash100) d 099 (097ndash100) d

No eyes evaluated 3978 33535 16670 5386

Each DR grading program referenced was evaluated using different image set and grading methods The differences in the performance measuresmay be due to differences in the image set grading methods algorithm threshold or a combination of various factors These are presented as areference and generally do not represent comparative performance NPV negative predictive value PPV positive predictive value Fleming et al (13)daggerAbramoff et al (15) DaggerOliveira et al (14)

1648 Real-Time Evaluation of Diabetic Retinopathy Diabetes Care Volume 38 September 2015

evaluate images for the presence of ei-ther DR or referable DR at the time ofUWFI With a sensitivity and negativepredictive value that approaches 100this methodology could result in a sub-stantial reduction of centralized readingcenter burden and speed delivery of in-formation and education to the patientFurthermore the accurate identifica-tion of referable DR allows prompt eyecare referral reducing the burdencaused by false-positive results The costof current UWFI devices remains restric-tive in resource-poor settings or in pro-grams that care for only a limited numberof patients with diabetes However if thefindings of the current study are repli-cated across large and diverse popula-tions and if innovations such as thosediscussed here are rigorously validatedand adopted the impact on delivery ofdiabetes eye care could be substantial

Funding The study was supported by grantfunding from the Center of Integration ofMedicine and Innovative Technology to PSSJVN technology was developed at the JoslinDiabetes Center All the authors were employeesof the Joslin Diabetes Center at the time the studywas conductedDuality of Interest One of the two Optos P200instruments used in this study was provided byOptos plc (Dunfermline Fife UK) to the JoslinDiabetes Center on temporary loan No otherpotential conflicts of interest relevant to thisarticle were reportedAuthor Contributions PSS and JDC re-searched data and wrote the manuscriptAMT JR SR RA DT BP MS and KTresearched data and reviewed and edited themanuscript JKS researched data contributedto the discussion and reviewed and edited the

manuscript LPA contributed to the discussionand reviewed and edited the manuscript PSSand JDC are the guarantors of this work andas such had full access to all the data in the studyand take responsibility for the integrity of thedata and the accuracy of the data analysisPrior Presentation Parts of this study werepresented in abstract form at ATA 2014 Balti-more MD 17ndash19 May 2014 and the Associa-tion for Research in Vision and OphthalmologyAnnual Meeting Denver CO 3ndash7 May 2015

References1 American Diabetes Association 9 Microvas-cular complications and foot care In Standardsof Medical Care in Diabetesd2015 DiabetesCare 201538(Suppl 1)S58ndashS662 Silva PS Cavallerano JD Aiello LM Aiello LPTelemedicine and diabetic retinopathy movingbeyond retinal screening Arch Ophthalmol2011129236ndash2423 Schoenfeld ER Greene JMWu SY LeskeMCPatterns of adherence to diabetes vision careguidelines baseline findings from the DiabeticRetinopathy Awareness Program Ophthalmol-ogy 2001108563ndash5714 Salti H Cavallerano JD Salti N et al Nonmydri-atic retinal image review at time of endocrinologyvisit results in short-term HbA1c reduction inpoorly controlled patients with diabetic retinopa-thy Telemed J E Health 201117415ndash4195 Cavallerano JD Silva PS Tolson AM et alImager evaluation of diabetic retinopathy atthe time of imaging in a telemedicine programDiabetes Care 201235482ndash4846 Silva PS Cavallerano JD Sun JK Noble JAiello LM Aiello LP Nonmydriatic ultrawidefield retinal imaging compared with dilatedstandard 7-field 35-mm photography and reti-nal specialist examination for evaluation of di-abetic retinopathy Am J Ophthalmol 2012154549ndash5597 Kernt M Hadi I Pinter F et al Assessment ofdiabetic retinopathy using nonmydriatic ultra-widefield scanning laser ophthalmoscopy (Op-tomap) compared with ETDRS 7-field stereophotography Diabetes Care 2012352459ndash2463

8 Silva PS Cavallerano JD Tolls D et al Poten-tial efficiency benefits of nonmydriatic ultrawidefield retinal imaging in an ocular telehealth diabeticretinopathy program Diabetes Care 20143750ndash559 Wessel MM Aaker GD Parlitsis G Cho MDrsquoAmico DJ Kiss S Ultra-wide-field angiogra-phy improves the detection and classificationof diabetic retinopathy Retina 201232785ndash79110 Aiello LM Bursell SE Cavallerano J GardnerWK Strong J Joslin Vision Network ValidationStudy pilot image stabilization phase J Am Op-tom Assoc 199869699ndash71011 Sanchez CR Silva PS Cavallerano JD AielloLP Aiello LM Ocular telemedicine for diabeticretinopathy and the Joslin Vision NetworkSemin Ophthalmol 201025218ndash22412 Li HK HortonM Bursell SE et al Telehealthpractice recommendations for diabetic retinop-athy second edition Telemed J E Health 201117814ndash83713 Fleming AD Goatman KA Philip S PrescottGJ Sharp PF Olson JA Automated grading fordiabetic retinopathy a large-scale audit usingarbitration by clinical experts Br J Ophthalmol2010941606ndash161014 Oliveira CM Cristovatildeo LM Ribeiro MLAbreu JR Improved automated screening of di-abetic retinopathy Ophthalmologica 2011226191ndash19715 Abramoff MD Folk JC Han DP et al Auto-mated analysis of retinal images for detection ofreferable diabetic retinopathy JAMA Ophthal-mol 2013131351ndash33516 Silva PS Cavallerano JD Sun JK Soliman AZAiello LM Aiello LP Peripheral lesions identifiedby mydriatic ultrawide field imaging distribu-tion and potential impact on diabetic retinopa-thy severity Ophthalmology 20131202587ndash259517 Centers for Medicare amp Medicaid ServicesMedicare program payment policies under thephysician fee schedule and other revisions to partB for CY 2011 [article online] 2011 Available fromhttpwwwregulationsgovdocumentDetailD=CMS-2010-0205-1982 Accessed 21 January2015

carediabetesjournalsorg Silva and Associates 1649