Research Article Multivendor Spectral-Domain Optical Coherence …downloads.hindawi.com/journals/joph/2016/3898750.pdf · 2019-07-30 · Research Article Multivendor Spectral-Domain

Research ArticleMultivendor Spectral-Domain Optical CoherenceTomography Dataset Observer Annotation PerformanceEvaluation and Standardized Evaluation Framework forIntraretinal Cystoid Fluid Segmentation

Jing Wu1 Ana-Maria Philip1 Dominika Podkowinski1 Bianca S Gerendas1 Georg Langs2

Christian Simader1 Sebastian M Waldstein1 and Ursula M Schmidt-Erfurth1

1Christian Doppler Laboratory for Ophthalmic Image Analysis Department of Ophthalmology and OptometryMedical University of Vienna Vienna Austria2Christian Doppler Laboratory for Ophthalmic Image Analysis Computational Imaging Research LabDepartment of Biomedical Imaging and Image-Guided Therapy Medical University of Vienna Vienna Austria

Correspondence should be addressed to Sebastian M Waldstein sebastianwaldsteinmeduniwienacat

Received 23 December 2015 Accepted 29 June 2016

Academic Editor Majid M Moshirfar

Copyright copy 2016 Jing Wu et al This is an open access article distributed under the Creative Commons Attribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Development of image analysis and machine learning methods for segmentation of clinically significant pathology in retinalspectral-domain optical coherence tomography (SD-OCT) used in disease detection and prediction is limited due to theavailability of expertly annotated reference data Retinal segmentation methods use datasets that either are not publicly availablecome from only one device or use different evaluation methodologies making them difficult to compare Thus we present andevaluate a multiple expert annotated reference dataset for the problem of intraretinal cystoid fluid (IRF) segmentation a keyindicator in exudative macular disease In addition a standardized framework for segmentation accuracy evaluation applicableto other pathological structures is presented Integral to this work is the dataset used which must be fit for purpose for IRFsegmentation algorithm training and testing We describe here a multivendor dataset comprised of 30 scans Each OCT scan forsystem training has been annotated bymultiple graders using a proprietary system Evaluation of the intergrader annotations showsa good correlation thus making the reproducibly annotated scans suitable for the training and validation of image processing andmachine learning based segmentation methods The dataset will be made publicly available in the form of a segmentation GrandChallenge

1 Introduction

Spectral-domain optical coherence tomography (SD-OCT) isthe most important ancillary test for the diagnosis of sightdegrading diseases such as retinal vein occlusion (RVO) age-related macular degeneration (AMD) and glaucoma [1] SD-OCT is a noninvasive modality for acquiring high resolution3D cross-sectional volumetric images of the retina and thesubretinal layers in addition to retinal pathology such asintraretinal fluid subretinal fluid and pigment epithelialdetachment [2 3] Detection and segmentation of suchpathologies are an important step in the diagnosis of disease

severity and treatment success as well as an early stagetowards the accurate prediction of both [4 5] The detectionof intraretinal cystoid fluid (IRF) is a particularly importantindicator of disease severity and change in exudative maculardisease as increased retinal thickness has shown to correlatewith poor visual acuity [6] thus automated detection andsegmentation methods are required to employ ldquobig datardquo invisual acuity and treatment progression predictionThus IRFshave been chosen as the basis for this multivendor referencedataset and grader performance assessment [7 8]

At the time of writing there is no publically availabledataset of SD-OCT scans acquired from multiple SD-OCT

Hindawi Publishing CorporationJournal of OphthalmologyVolume 2016 Article ID 3898750 8 pageshttpdxdoiorg10115520163898750

2 Journal of Ophthalmology

Table 1 Dataset composition showing total scans of each scanner vendor within each dataset

Set Spectralis scans Cirrus scans Topcon scans Nidek scans Total scansTraining 4 4 4 3 15Testing 4 4 4 3 15

devices and featuring a wide variety of IRF appearances withan accompanying expertly annotated ground truth that is tosay manually annotated IRF regions by trained individualsSuch a dataset is important for the development of novelsegmentation algorithms as it allows for the training andtesting of new systems with a general reference In the currentliterature methods of IRF segmentation are limited usingtraining and validation datasets that are not always publicallyavailable [9ndash12] This results in difficulty in reproducingresults for comparative purposes which in addition do notalways use the same evaluation measures Equally importantis the reproducibility of IRF annotations used to constructthe reference standard High interobserver agreement isnecessary however this is difficult due to the challengingnature of manual IRF delineation The combination of areproducibly annotated dataset and evaluation frameworkwill facilitate the consistent and uniform comparison ofnewly developed and current methods through standardizedmeasures of segmentation accuracy [13] Furthermore thiswould allowmethods to be assessed as part of a segmentationchallenge [14 15] an important and effective means by whichnovel methods are developed in not only medical imagingresearch but also many other fields in computer vision Thismay facilitate a better understanding of the positive andnegative aspects of each developed method in an effort toimprove performance as well as opening avenues for furtherdevelopment or collaboration

Thus the purpose of this work is to create a multivendorSD-OCT dataset comprised of clinically representative scanswith IRFs annotated by multiple expert graders This workwill show the reproducibility of the annotations suitablefor use as a reference standard to both train and validateIRF segmentation methods Furthermore a standardizedevaluation framework for IRF segmentation is presented

2 Materials and Methods

21 Dataset The dataset constructed here is comprised of 30distinct SD-OCT scans from fourmajor OCT devices used inophthalmology (Zeiss Cirrus Heidelberg Spectralis Topcon3D 2000 and Nidek RS3000) in the proportions describedin Table 1 The image datasets were selected from the imagedatabase of Vienna Reading Center (VRC) featuring largedatasets from several international phase II and III pharma-ceutical trials in retinal disease The individual images werechosen by medical experts in order to reflect a representativedistribution of OCT scanners acquisition settings diseasestages and image quality

This study was conducted in compliance with the tenetsset forth in the declaration of Helsinki The trials from whichthe scans were taken were approved by the institutionalreview board of theMedicalUniversity ofVienna All patients

gave written consent for participation in the respective trialand all data was appropriately anonymized

The dataset is further divided into 15 training scans and15 testing scans chosen to be representative of the widevariety of scans seen in the clinical environment in additionto the wide variety of IRF appearances and distributionsBoth the training and testing subsets comprised 4 scansper vendor aside from Nidek with 3 Each scan within thisdataset has been explicitly chosen to contain a wide varietyof IRF sizes shapes and appearances This is particularlyimportant for algorithm training (such as that of machinelearning techniques) asmethods will need to learn the varietyof possible cyst appearances across different devices whilefactoring in the noise pattern and signal response variationacross different devices All 15 training scans have beenannotated on each individual slice comprising the OCTvolume (henceforth known as a B-scan) by two distinctexpert graders at the Christian Doppler Laboratory forOphthalmic Image Analysis (OPTIMA) Medical Universityof Vienna who have been trained to identify IRFs using acriteria explained in the following section

The testing set is intended for validation of IRF segmen-tation systems and thus also contains the same spectrum ofIRF appearances sizes and shapes as seen in the trainingsubset in addition to normal cases to act as control imagesFigure 1 presents exemplar B-scans from each of the 4devices exemplifying the varying signal and noise and IRFappearance variations (indicated by the white arrows)

Each retinal OCT volume is approximately 6 times 6 times2mm3 and centered on the macula The coordinate systemused to represent the retinal volume is shown in Figure 2[16] Figure 2(a) demonstrates the location of the B-scans inrelation to the anatomical eye and the respective 119883 119884 and119885 image planes in red green and blue In Figure 2(b) theprimary (119887

119901) and secondary (119887

119904) scan directions are depicted

in addition to their relationships with themajor image planesUsing the same color coding system as described previouslythe red B-scan can be seen in addition to the perpendiculargreen A-scan The windows defined by 119908

119901and 119908

119904are not

utilized here Furthermore Figure 2(a) shows the raster scanpattern (blue arrow) utilized by the OCT devices used toacquire the scans for this dataset Dependent on device thephysical dimensions equate to 200 times 200 times 1024 256 times 256 times885 512 times 128 times 885 512 times 128 times 1024 or 512 times 49 times 496 pixels

22 IRF Annotation Annotation was performed using aproprietary system developed at the OPTIMA Lab withfunctionality to perform manual pixel level annotations ofretinal SD-OCT scans Annotation is performed in the B-scan plane examples of which are shown in Figure 3 for eachdevice where the annotated IRF outline is shown in greenNot only do the examples in Figure 3 exemplify the varying

Journal of Ophthalmology 3

Cirrus Topcon

Nidek Spectralis

Figure 1 Exemplar retinal B-scans from 4 SD-OCT devices showing variations in noise and appearance White arrows indicate exemplarIRFs

(a)

Z

X

Y

ws

wp

bp

bs

Secondary

Primary

A-scan

(b)

Figure 2 (a) Retinal OCT scan coordinate space in relation to anatomical eye (b) OCT scan pattern representing the red green and bluecolored planes shown in (a) [16]

Cirrus Topcon

Nidek Spectralis

Figure 3 Exemplar annotated B-scans showing annotated cysts in green


size and appearance of IRFs in SD-OCT but also it can be seenthat scans from different vendors vary sometimes greatlywith respect to image quality signal-to-noise ratio (SNR) andcontrast thus making the task of manual IRF segmentationvery difficult and time-consuming

Each grader was tasked with manually delineating theIRF structures that were visible to them in each B-scan of avolume using free-hand drawing with a stylus and tablet

The criteria the graders used to analyze IRFs were asfollows

(i) Shapeintensity the IRF shape spectrum is broadranging from circularoval to an amorphous blobHowever IRF intensity is generally low due to theattenuation of light as the medium is primarily liquid

(ii) Distinction IRFs usually have distinct borders sep-arating their interior with the surrounding tissueHowever this is dependent on the scan image qualityand the presence of noise

(iii) Continuity IRFs are three-dimensional objects and assuch may be present across multiple contiguous B-scans However this is dependent on IRF size andthe B-scan slice thickness used at acquisition by thedevice and study protocol

(iv) Position IRFs which significantly affect visual acuityare generally located in and around the fovea ofmacula centered retinal OCT scans which is thefunctional center of vision

This annotation process stores the manually delineatedregions on each B-scan 119878Bscan(119885119883) within a separatevolume containing the positions of the annotated cysts119881(119885119883 119884) extractable using various computational meansFor usage purposes the annotated IRFs are extracted usingMATLAB (The Mathworks Inc) and stored using thestandardized XML format [17] and the coordinate systemdescribed previously and in Figure 2

Figure 3 shows exemplar annotated IRFs from each ofthe four devices outlining in green the annotated IRF struc-ture(s) As can be seen IRFs range in size and appearance aswell as location In addition Figure 3 demonstrates the chal-lenging nature of manual human expert annotation of suchobjects given that cysts may be extremely small in size withdifficult-to-delineate boundaries requiring approximately150 hrs in total to annotate the 30 scans The time intensivenature of manual IRF segmentation further illustrates therequirement for accurate and reproducible automated meth-ods for IRF segmentation as it is not feasible nor possible forhuman graders to perform this task accurately for such longperiods or for large datasets

23 Standardized Evaluation Framework IRF segmentationalgorithm results must be evaluated in a standardized wayso that results from different methods are comparable Inaddition as IRFs are delineated by their boundaries arelevant measure of accuracy is required to gauge systemperformance Thus we propose the use of three initialmeasures firstly area overlap with reference IRF positionssecondly distance from reference IRF boundaries and thirdly

the intersection-over-union which is also widely used inevaluating image segmentation The first measure examinesthe overlap between system segmented IRF area resultsand reference standard based on the Soslashrensen-Dice index(DSC) [18] The second measure is based on the Hausdorffdistance [19]which examines the distance between the systemsegmented IRF regions and ground truth The third measureexamines the overlap between system and reference IRF areasby computing the intersection divided by the union The setof IRF coordinate points of all segmented IRFs on a given B-scan is defined as 119878Bscan(119885119883) and the reference IRF pointsfor a given B-scan are defined as 119877Bscan(119885119883) where 119885 is theposition on the vertical axis of the B-scan 119883 is the positionon the horizontal axis of the B-scan and 119884 is the B-scan inthe volume (Figure 2)

119874Bscan =21003816100381610038161003816119878Bscan cap 119877Bscan

1003816100381610038161003816

1003816100381610038161003816119878Bscan1003816100381610038161003816+1003816100381610038161003816119877Bscan1003816100381610038161003816

(1)

where 119874Bscan is the overlap for a specific B-scan

119867(119878Bscan 119877Bscan) = max (ℎ (119878Bscan 119877Bscan) ℎ (119877Bscan 119878Bscan)) (2)

where 119867 is the Hausdorff distance between sets 119878Bscan and119877Bscan

119868Bscan =119886119903119890119886 (119878Bscan cap 119877Bscan)

119886119903119890119886 (119878Bscan cup 119877Bscan) (3)

where 119868Bscan is the intersection-over-union overlap for aspecific B-scan

Thus we compute the overlap between the referenceannotation and system segmentation for a given B-scan using(1) resulting in a value within 0 sdot sdot sdot 1 where being closerto 0 represents poor overlap and being closer to 1 a highoverlap taking themean over all B-scanswith cysts to give theoverlap for the entire volume (119874Volume)We use theHausdorffdistance between point sets 119878Bscan and 119877Bscan as described by(2) to compute the distance between the ground truth andsegmented IRFs for a given B-scan resulting in a pixel value(119867Bscan)We compute themean distance over all B-scans withIRFs to give the overall distance for the volume (119867Volume)The intersection-over-union overlap between reference andsystem segmentation for a given B-scan is computed using(3) resulting in a value within 0 sdot sdot sdot 1 where being closer to0 represents poor overlap and being closer to 1 a high overlapAgain the mean over all B-scans with cysts is computed togive the overlap for the entire volume (119868Volume)

In addition to the overall score resulting from the threequantitative measures mentioned here system performanceis further evaluated using two further criteria clinical sig-nificance of the IRF and IRF size Due to their compositionand position some IRFs may be more clinically significant todisease than others These IRFs tend to be larger and locatedbelow and around the fovea which is the functional center ofvision [20]Thus their size and position are used as classifierswith the central 3mm circular region used as a mask 119898applied to the enface OCT image [21] This is demonstratedin Figure 4 where the red circle in Figure 4(a) denotes the


(a) (b)

Figure 4 (a) Exemplar retinal OCT volume depicting the circular ROI in red (b) Exemplar B-scan taken from the location represented inblue in (a)

Table 2 Vendor specific small cyst size dimensions in micrometers (width times height)

Spectralis Topcon Cirrus NidekSize 120583m (width times height) 5808 times 1936 3900 times 1300 2933 times 9775 6323 times 2108

masked region displayed on a volume render of the retinalOCT The B-scan seen in Figure 4(b) demonstrates postmasking where the blue line in Figure 4(a) denotes where theB-scan is located Thus119874Mask and119867Mask respectively denotethe DSC overlap and Hausdorff distance for the maskedregion

However larger IRFs are generally much more visiblethus smaller IRFs are harder to delineate due to poor SNRand poor boundary distinction Thus the second additionalmeasure assigns a label to small cysts such that their seg-mentation accuracy is evaluated separately For the purposesof this evaluation framework a small IRF is assigned aphysical minimum size (120583m) computed from the minimumIRF size as annotated by expert graders at the OPTIMA LabThus 119874119878Volume and 119867119878Volume denote the DSC and Hausdorffdistance of small IRFs per volume and 119874119878Mask and 119867119878Maskdenote the DSC andHausdorff distance for small IRFs withinthemasked region Small IRF size is defined by theminimumIRF size for each vendor in Table 2 It should be noted that aseparate minimum IRF size has been identified per devicethis is due to the interdevice image acquisition differences

In summary the ten measures defined to evaluate seg-mentation performance are as follows

(1) Overall overlap using DSC 119874Volume(2) Mean Hausdorff distance between IRF boundaries119867Volume

(3) Intersection-over-union 119868Volume(4) Measures 1 2 and 3 within the central 3mm masked

region (Figure 4(a)) 119874Mask and 119867Mask(5) Measures 1 2 3 and 4 for small IRFs 119874119878Volume119867119878Volume 119868119878Volume 119874119878Mask and 119867119878Mask

3 Results

Fifteen scans comprising the training dataset were annotatedby two separate graders (G1 and G2) Table 3 shows thenumber of IRFs annotated by each grader resulting in a totalof 9457 annotated IRFs Grader 1 annotated a mean plusmn SDof 3026 plusmn 3491 and Grader 2 annotated a mean plusmn SD of

3279 plusmn 3681 IRF regions The agreement of the manual IRFannotation between Graders 1 and 2 was good with a meandifference of 253 IRF regions as shown in Figure 5(a) inaddition to Pearsonrsquos 119903 = 098 (119875 lt 00001) Furthermorethere was 120581 = 076 between the two graders based on totalIRF annotation

This is expanded upon in Table 4 in which the differencein total annotated IRFs is presented between the two gradersThe total difference in annotated scans between the twograders was 629 IRF regions with a mean plusmn SD of 419 plusmn 452IRF regions

A challenging aspect of IRF annotation is poor distinc-tion between IRF regions This may result in one observerannotating one large IRF and another observer annotatingmultiple smaller IRFs Thus we analyze the pixel wise area ofthe annotated IRF regions presented in Table 5

Between the two graders the total annotated IRF areawas 3833289 pixels Grader 1 annotated IRFs comprised1900960 pixels and Grader 2 annotated IRFs comprised1932329 pixels with an intersecting area of 1447480 pixelsAs shown in Figure 5(b) agreement between the two graderswas again good based on IRF area with a mean difference of20913 pixels in addition to Pearsonrsquos 119903 = 099 (119875 lt 00001)Furthermore there was 120581 = 086 between the two gradersbased on annotated IRF pixel wise area

Grader reproducibility is further assessed using Haus-dorff distance [19] computation between annotated IRF pointsets shown in Table 6 The mean Hausdorff distance plusmn SDbetween the two graders was 3471 plusmn 3098 pixels

4 Discussion

The resulting manual IRF annotations obtained in this studymust be fit for purpose as a reference standard for bothIRF segmentation training and validation That is to saynot only is it necessary for annotations to be accurate tothe position and delineation of the objects in question butalso in the case of the training dataset where annotationwas performed by two graders the annotations must besimilar The first major contribution of this work is a datasetcomprised of multidevice SD-OCT scans representative of


Bland and Altman plot of annotated IRFs Grader 1 and Grader 2

Mean of annotated IRFs(Grader 1 + Grader 2)2

0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253

minus196 SDminus1588

minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)

Bland and Altman plot Annotated IRFs area (pixels) Grader 1 and Grader 2

Mean of annotated IRF area (pixels)(Grader 1 + Grader 2)2

Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)

Figure 5 Bland Altman plots of annotated IRFs by the two graders (a) Agreement of manual annotation between Graders 1 and 2 was goodwith a mean difference of 253 IRFs (b) Agreement between Graders 1 and 2 based on area of annotated IRFs in pixels was also good withmean difference of 20913 pixels

Table 3 Annotated IRFs by Grader 1 (G1) and Grader 2 (G2) training scans 1 to 4 for each vendor

Set Spectralis Cirrus Topcon Nidek Mean plusmn SD total IRFsG1 G2 G1 G2 G1 G2 G1 G2

Training 1 128 129 39 46 399 547 299 323 2388 plusmn 1824Training 2 16 19 69 77 1170 1276 258 353 4048 plusmn 5195Training 3 136 115 995 928 455 409 370 523 4914 plusmn 3243Training 4 55 47 18 27 132 99 na na 63 plusmn 4404Mean plusmn SD total IRFs 8063 plusmn 5154 2749 plusmn 4246 5609 plusmn 4376 3543 plusmn 9167

Table 4 Difference in number of annotated IRFs between Grader 1 and Grader 2 in the training set scans 1 to 4 for each vendor

Set Spectralis Cirrus Topcon Nidek Mean diff plusmn SD (IRFs)Training 1 1 7 48 24 20 plusmn 2106Training 2 3 8 106 95 53 plusmn 5507Training 3 21 67 46 153 7175 plusmn 5734Training 4 8 9 33 na 1667 plusmn 1415Mean diff plusmn SD (IRFs) 825 plusmn 899 2275 plusmn 2951 5825 plusmn 3252 9067 plusmn 6461

Table 5 Total IRF area in pixels annotated by each grader in the training (Trn) set including total number of pixels intersecting (cap)

Set Spectralis (area) Cirrus (area) Topcon (area) Nidek (area) Mean diff plusmn SD (area)G1 G2 G1 G2 G1 G2 G1 G2

Trn 1 43986 51895 24549 30017 121654 128716 161714 149601 8138 plusmn 2837cap = 38197 cap = 18840 cap = 85268 cap = 126594



Trn 4 7576 8361 3734 4623 70152 74288 na na 1937 plusmn 1905cap = 6619 cap = 2716 cap = 58692

Mean diff plusmn SD (area) 3203 plusmn 3492 8049 plusmn 11817 19236 plusmn 26289 8213 plusmn 3647


Table 6 Hausdorff distance between grader annotations in pixels

Set Spectralis (pixels) Cirrus (pixels) Topcon (pixels) Nidek (pixels) Mean dist plusmn SD (pixels)Training 1 3742 1892 4851 1231 5699 plusmn 4575Training 2 3162 1479 5243 8 1956 plusmn 2240Training 3 1828 6070 4415 5025 4334 plusmn 1806Training 4 4123 1603 2083 na 5512 plusmn 7825Mean dist plusmn SD (pixels) 1574 plusmn 1602 2761 plusmn 2213 4148 plusmn 1418 6046 plusmn 5824

IRF compositions seen in exudative macular disease Thisdataset was annotated by trained graders at the OPTIMA Labusing a predefined annotation criteria based on observed IRFcharacteristics described in Section 22 A standardized eval-uation framework comprised of 4 key measures (Section 23)was created to evaluate IRF segmentation algorithms trainedusing the aforementioned dataset

Manual delineation of intraretinal cystoid fluid is anextremely time-consuming and difficult task However accu-rately and reproducibly segmented IRFs are necessary asthey provide clinically significant information regarding thedevelopment progression and treatment success of patientswith exudative macular disease As shown in [4] IRFs are animportant spatiotemporal feature for longitudinal and cross-patient disease analysis in diseases such as RVO and neovas-cular AMD For such purposes larger datasets are requiredfrom which such features are extracted thus in ldquobig datardquosituations there is a need for automated methods of featureextraction (such as IRFs) Furthermore accurate delineationof features allows the implementation of semisupervised andweakly supervised learning techniques [22] to be applied toldquobig datardquo

Our findings show that given the criteria of shapeinten-sity distinction continuity and position describing IRFsit is possible to annotate these regions reproducibly bytwo trained graders who are masked to each other Thisis exemplified by the high degree of intersection betweenthe two graders with respect to IRF annotation area (gt75intersection pixels) and correlation coefficients 098 and099 for IRF region and IRF area respectively Furthermoregrader agreement was good exemplified by high 120581

The difference in total annotated IRFs between Graders 1and 2 is shown in Table 4 (calculated from the total annotatedIRFs by each grader shown in Table 3) ranging from 1 toover 150 objects This large range is possibly a result ofthe subjective nature of human observer annotation despitethe presence of guidelines For example one grader mayjudge an object as 1 large IRF whereas another grader maydelineate it as a series of smaller IRFs with aminimal distancebetween region boundaries Another possible explanation forIRF region variability is related to the device Of note is themean plusmn SD difference in annotated IRFs by vendor showingthat in the case of the Heidelberg Spectralis scans where thepresence of noise is lower due to the averaging of multipleB-scan acquisitions and motion correcting eye tracker islowest This value increases for Zeiss Cirrus scans andcontinues to do so for Topcon 3D 2000 and Nidek RS3000respectively This trend correlates with the observed changein image quality in combinationwith increasing speckle noise

(Figure 1) increasing the difficulty for human observers toaccurately and reproducibly annotate IRFs

Thus the number of annotated IRFs is not a representativemeasure of actual IRF composition and is less suitable forcalculating intergrader reproducibility A more accurate andprecise measure is the total object area in pixels annotatedby each grader This can be seen in Table 5 in addition tothe total intersection area for each scan representing thevoxels annotated by both graders As can be seen in 10 of15 cases the difference between graders total IRF areas wasless than 10 of the respective total IRF area for a givenscan In addition 4 cases were calculated with a differencebetween grader IRF areas below 5 of the respective totalIRF area This figure rises to 14 out of 15 cases when thethreshold is raised to 20 of total annotated IRF area byeach grader Furthermore examination of the multigraderannotated training dataset Hausdorff distance (Table 6)examining if an annotated voxel from Grader 1 is close toan annotated point from Grader 2 results in a mean plusmn SDHausdorff distance of 3422 plusmn 3098 pixels Again this isnoticeably lower for Spectralis scans (1574 plusmn 1602 pixels)where image quality is better which is to be expected asgrader delineation difficulty is lower compared to Cirrus(2761 plusmn 2213 pixels) Topcon (4148 plusmn 1418 pixels) andNidek (6046 plusmn 5824 pixels) correlating with their respectivelevels of noise andpoorer image qualityThis is the same trendseen in the analysis of total IRF objects annotated from eachdevice Despite this the mean Hausdorff distance is still lowindicating a good correlation between graders

To the best of the authorsrsquo knowledge the dataset pre-sented here is the only publically available dataset comprisedof expertly manually annotated intraretinal fluid in SD-OCTscans frommultiple vendor devicesThe high reproducibilitywe have shown between grader annotations for each scan inthe training dataset is a major advantage of a training datasetannotated by multiple graders as this demonstrates goodaccuracy and precision Furthermore this makes this datasetsuitable and fit for use as accurate and reproducible referencestandard for the development of retinal IRF segmentationalgorithms In addition this has also shown that annotationby a single grader examined here is sufficient for use inalgorithm testing based on the inclusion criteria describingthe IRFs As such the testing dataset described in Table 1has been annotated by a single expert grader per scan andas mentioned previously is intended for testing of developedmethods

Competing Interests

The authors declare that they have no competing interests


Acknowledgments

The financial support of the Austrian Federal Ministry ofEconomy Family and Youth and the National Foundationfor Research Technology and Development is gratefullyacknowledged The authors would also like to acknowledgethe tireless work of their graders in performing the vastamounts of annotations required for creating the referencedataset and the software development team for providing andsupporting the tools integral to obtaining them

References

[1] WGeitzenauer C KHitzenberger andUM Schmidt-ErfurthldquoRetinal optical coherence tomography past present and futureperspectivesrdquo British Journal of Ophthalmology vol 95 no 2pp 171ndash177 2011

[2] G J Jaffe and J Caprioli ldquoOptical coherence tomography todetect and manage retinal disease and glaucomardquo AmericanJournal of Ophthalmology vol 137 no 1 pp 156ndash169 2004

[3] SMMeuer C EMyers B E K Klein et al ldquoThe epidemiologyof vitreoretinal interface abnormalities as detected by spectral-domain optical coherence tomography The Beaver Dam EyeStudyrdquo Ophthalmology vol 122 no 4 pp 787ndash795 2015

[4] W Vogl SMWaldstein B S Gerendas et al ldquoSpatio-temporalsignatures to predict retinal disease recurrencerdquo in Proceedingsof the 24th International Conference Information Processing inMedical Imaging (IPMI rsquo15) June-July 2015

[5] H Bogunovic M D Abramoff L Zhang and M SonkaldquoPrediction of treatment response from retinal OCT in patientswith exudative age-related macular degenerationrdquo in Proceed-ings of the Ophthalmic Medical Image Analysis Workshop inConjunection with MICCAI 2014 pp 129ndash136 Boston MassUSA September 2014

[6] D J Browning A R Glassman L P Aiello et al ldquoRelation-ship between optical coherence tomography-measured centralretinal thickness and visual acuity in diabetic macular edemardquoOphthalmology vol 114 no 3 pp 525ndash536 2007

[7] J J Fuller and J O Mason III Retinal Vein OcclusionsUpdate on Diagnostic and Therapeutic Advances Focal PointsClinical Modules for Ophthalmologists American Academy ofOphthalmology San Francisco Calif USA 2007

[8] U Schmidt-Erfurth V Chong A Loewenstein et al ldquoGuide-lines for the management of neovascular age-related maculardegeneration by the European Society of Retina Specialists(EURETINA)rdquo British Journal of Ophthalmology vol 98 no 9pp 1144ndash1167 2014

[9] E K Swingle A Lang A Carass et al ldquoSegmentation ofmicrocystic macular edema in Cirrus OCT scans with anexploratory longitudinal studyrdquo Proceedings of SPIEndashthe Inter-national Society for Optical Engineering vol 9417 2015

[10] E K Swingle A Lang A Carass H S Ying P A Calabresiand J L Prince ldquoMicrocystic macular edema detection inretina OCT imagesrdquo in Proceedings of the Medical Imaging 2014Biomedical Applications inMolecular Structural and FunctionalImaging vol 9038 of Proceedings of SPIE February 2014

[11] X Chen M Niemeijer L Zhang K Lee M D Abramoffand M Sonka ldquoThree-dimensional segmentation of fluid-associated abnormalities in retinal OCT probability con-strained graph-search-graph-cutrdquo IEEE Transactions on Medi-cal Imaging vol 31 no 8 pp 1521ndash1531 2012

[12] G R Wilkins O M Houghton and A L Oldenburg ldquoAuto-mated segmentation of intraretinal cystoid fluid in opticalcoherence tomographyrdquo IEEE Transactions on Biomedical Engi-neering vol 59 no 4 pp 1109ndash1114 2012

[13] C T Metz M Schaap T van Walsum et al ldquo3D segmentationin the clinic a grand challenge IImdashcoronary artery trackingrdquoInsight Journal vol 1 no 5 pp 1ndash6 2008

[14] M Schaap C T Metz T van Walsum et al ldquoStandardizedevaluation methodology and reference database for evaluat-ing coronary artery centerline extraction algorithmsrdquo MedicalImage Analysis vol 13 no 5 pp 701ndash714 2009

[15] ldquoVisual Concept Extraction Challenge in Radiologyrdquo 2015httpwwwvisceraleu

[16] A Montuoro S M Waldstein B S Gerendas G Langs CSimader and U Schmidt-Erfurth Motion Artefact Correctionin Retinal Optical Coherence TomographyUsing Local SymmetryMICCAI Boston Mass USA 2014

[17] Extensible Markup Language (XML) 11 2nd edition 2014httpswwww3org

[18] T Sorensen ldquoA method of establishing groups of equal ampli-tude in plant sociology based on similarity of species and itsapplication to analyses of the vegetation on Danish commonsrdquoKongelige Danske Videnskabernes Selskab vol 5 pp 1ndash34 1948

[19] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[20] J M Provis A M Dubis T Maddess and J Carroll ldquoAdapta-tion of the central retina for high acuity vision cones the foveaand the a vascular zonerdquo Progress in Retinal and Eye Researchvol 35 pp 63ndash81 2013

[21] J Wu S Waldstein B Gerendas G Langs C Simader and USchmidt-Erfurth ldquoAutomated retinal fovea type distinction inspectral-domain optical coherence tomography of retinal veinocclusionrdquo inMedical Imaging 2015 Image Processing vol 9413of Proceedings of SPIE Orlando Fla USA March 2015

[22] T Schlegl SWaldsteinW-D Vogl U Schmidt-Erfurth andGLangs ldquoPredicting semantic description from medical imageswith convolutional neural networksrdquo in Information Processingin Medical Imaging 24th International Conference IPMI 2015Sabhal Mor Ostaig Isle of Skye UK June 28ndashJuly 3 2015Proceedings vol 9123 of Lecture Notes in Computer Science pp437ndash448 Springer Berlin Germany 2015

Submit your manuscripts athttpwwwhindawicom

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


MEDIATORSINFLAMMATION

of


Behavioural Neurology

EndocrinologyInternational Journal of



Disease Markers


BioMed Research International

OncologyJournal of



Oxidative Medicine and Cellular Longevity


PPAR Research

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Immunology ResearchHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of

ObesityJournal of



Computational and Mathematical Methods in Medicine

OphthalmologyJournal of


Diabetes ResearchJournal of



Research and TreatmentAIDS


Gastroenterology Research and Practice


Parkinsonrsquos Disease

Evidence-Based Complementary and Alternative Medicine

Volume 2014Hindawi Publishing Corporationhttpwwwhindawicom


Table 1 Dataset composition showing total scans of each scanner vendor within each dataset

Set Spectralis scans Cirrus scans Topcon scans Nidek scans Total scansTraining 4 4 4 3 15Testing 4 4 4 3 15

devices and featuring a wide variety of IRF appearances withan accompanying expertly annotated ground truth that is tosay manually annotated IRF regions by trained individualsSuch a dataset is important for the development of novelsegmentation algorithms as it allows for the training andtesting of new systems with a general reference In the currentliterature methods of IRF segmentation are limited usingtraining and validation datasets that are not always publicallyavailable [9ndash12] This results in difficulty in reproducingresults for comparative purposes which in addition do notalways use the same evaluation measures Equally importantis the reproducibility of IRF annotations used to constructthe reference standard High interobserver agreement isnecessary however this is difficult due to the challengingnature of manual IRF delineation The combination of areproducibly annotated dataset and evaluation frameworkwill facilitate the consistent and uniform comparison ofnewly developed and current methods through standardizedmeasures of segmentation accuracy [13] Furthermore thiswould allowmethods to be assessed as part of a segmentationchallenge [14 15] an important and effective means by whichnovel methods are developed in not only medical imagingresearch but also many other fields in computer vision Thismay facilitate a better understanding of the positive andnegative aspects of each developed method in an effort toimprove performance as well as opening avenues for furtherdevelopment or collaboration

Thus the purpose of this work is to create a multivendorSD-OCT dataset comprised of clinically representative scanswith IRFs annotated by multiple expert graders This workwill show the reproducibility of the annotations suitablefor use as a reference standard to both train and validateIRF segmentation methods Furthermore a standardizedevaluation framework for IRF segmentation is presented

2 Materials and Methods

21 Dataset The dataset constructed here is comprised of 30distinct SD-OCT scans from fourmajor OCT devices used inophthalmology (Zeiss Cirrus Heidelberg Spectralis Topcon3D 2000 and Nidek RS3000) in the proportions describedin Table 1 The image datasets were selected from the imagedatabase of Vienna Reading Center (VRC) featuring largedatasets from several international phase II and III pharma-ceutical trials in retinal disease The individual images werechosen by medical experts in order to reflect a representativedistribution of OCT scanners acquisition settings diseasestages and image quality

This study was conducted in compliance with the tenetsset forth in the declaration of Helsinki The trials from whichthe scans were taken were approved by the institutionalreview board of theMedicalUniversity ofVienna All patients

gave written consent for participation in the respective trialand all data was appropriately anonymized

The dataset is further divided into 15 training scans and15 testing scans chosen to be representative of the widevariety of scans seen in the clinical environment in additionto the wide variety of IRF appearances and distributionsBoth the training and testing subsets comprised 4 scansper vendor aside from Nidek with 3 Each scan within thisdataset has been explicitly chosen to contain a wide varietyof IRF sizes shapes and appearances This is particularlyimportant for algorithm training (such as that of machinelearning techniques) asmethods will need to learn the varietyof possible cyst appearances across different devices whilefactoring in the noise pattern and signal response variationacross different devices All 15 training scans have beenannotated on each individual slice comprising the OCTvolume (henceforth known as a B-scan) by two distinctexpert graders at the Christian Doppler Laboratory forOphthalmic Image Analysis (OPTIMA) Medical Universityof Vienna who have been trained to identify IRFs using acriteria explained in the following section

The testing set is intended for validation of IRF segmen-tation systems and thus also contains the same spectrum ofIRF appearances sizes and shapes as seen in the trainingsubset in addition to normal cases to act as control imagesFigure 1 presents exemplar B-scans from each of the 4devices exemplifying the varying signal and noise and IRFappearance variations (indicated by the white arrows)

Each retinal OCT volume is approximately 6 times 6 times2mm3 and centered on the macula The coordinate systemused to represent the retinal volume is shown in Figure 2[16] Figure 2(a) demonstrates the location of the B-scans inrelation to the anatomical eye and the respective 119883 119884 and119885 image planes in red green and blue In Figure 2(b) theprimary (119887

119901) and secondary (119887

119904) scan directions are depicted

in addition to their relationships with themajor image planesUsing the same color coding system as described previouslythe red B-scan can be seen in addition to the perpendiculargreen A-scan The windows defined by 119908

119901and 119908

119904are not

utilized here Furthermore Figure 2(a) shows the raster scanpattern (blue arrow) utilized by the OCT devices used toacquire the scans for this dataset Dependent on device thephysical dimensions equate to 200 times 200 times 1024 256 times 256 times885 512 times 128 times 885 512 times 128 times 1024 or 512 times 49 times 496 pixels

22 IRF Annotation Annotation was performed using aproprietary system developed at the OPTIMA Lab withfunctionality to perform manual pixel level annotations ofretinal SD-OCT scans Annotation is performed in the B-scan plane examples of which are shown in Figure 3 for eachdevice where the annotated IRF outline is shown in greenNot only do the examples in Figure 3 exemplify the varying


Cirrus Topcon

Nidek Spectralis


(a)

Z

X

Y

ws

wp

bp

bs

Secondary

Primary

A-scan

(b)


Cirrus Topcon

Nidek Spectralis















1003816100381610038161003816

1003816100381610038161003816119878Bscan1003816100381610038161003816+1003816100381610038161003816119877Bscan1003816100381610038161003816

(1)





119886119903119890119886 (119878Bscan cup 119877Bscan) (3)





(a) (b)










3 Results







4 Discussion





0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253


minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)



Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)
























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of

















Cirrus Topcon

Nidek Spectralis


(a)

Z

X

Y

ws

wp

bp

bs

Secondary

Primary

A-scan

(b)


Cirrus Topcon

Nidek Spectralis















1003816100381610038161003816

1003816100381610038161003816119878Bscan1003816100381610038161003816+1003816100381610038161003816119877Bscan1003816100381610038161003816

(1)





119886119903119890119886 (119878Bscan cup 119877Bscan) (3)





(a) (b)










3 Results







4 Discussion





0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253


minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)



Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)
























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of





























1003816100381610038161003816

1003816100381610038161003816119878Bscan1003816100381610038161003816+1003816100381610038161003816119877Bscan1003816100381610038161003816

(1)





119886119903119890119886 (119878Bscan cup 119877Bscan) (3)





(a) (b)










3 Results







4 Discussion





0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253


minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)



Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)
























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of

















(a) (b)










3 Results







4 Discussion





0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253


minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)



Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)
























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of



















0 200 400 600 800 1000 1200 1400

Diff

eren

ce o

f ann

otat

ed IR

Fs

0

50

100

150

1083

Mean

+196 SD

minus253


minus50

minus100

minus150

minus200

Gra

der1

and

Gra

der 2

(a)



Diff

eren

ce o

f ann

otat

ed IR

F ar

ea (p

ixel

s)

0

0 200000 400000

20000

40000

Mean

329404+196 SD

minus196 SD

minus20913

minus371229

minus20000

minus40000

minus60000

Gra

der1

and

Gra

der 2

(b)
























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of


























Competing Interests



Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of

















Acknowledgments


References




























of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of





















of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of
















Documents

Research Article Multivendor Spectral-Domain Optical Coherence …downloads.hindawi.com/journals/joph/2016/3898750.pdf · 2019-07-30 · Research Article Multivendor Spectral-Domain