Comparison of the Effectiveness of Retinal Cameras for
Malarial Retinopathy Detection in Malawi
Joshi, Vinayak S.1; Nemeth, Sheila C.1; Barriga, E. Simon1; Soliz, Peter 1; Harding, Simon P.2; Lewallen, Susan4; Taylor, Terrie E.5,6 ; MacCormick, Ian J.2,3
1. VisionQuest i-Rx Corp., Albuquerque, NM, United States. 2. Eye and Vision Science, Institute of Ageing and Chronic Disease, University of Liverpool, Liverpool, United Kingdom. 3. Malawi-Liverpool-Wellcome Trust Clinical Research Programme, Blantyre, Malawi. 4. Kilimanjaro Centre for Community Ophthalmology, University Cape Town Groot Schuur Hospital, Cape Town, South Africa. 5. Osteopathic Medical Specialties, Michigan State University, East Lansing, MI, United States. 6. Blantyre Malaria Project, Blantyre, MI, United States.
November 4, 2015
Final Report
Peter Soliz / CEO
505-508-1994 / [email protected]
VisionQuest i-Rx, Corp. ©2015 Page 2
Background. Cerebral malaria (CM) is a life-threatening clinical syndrome associated with malarial
infection. The 2014 World Malaria report estimates that malaria affected over 198 million people in
2013, and claimed the lives of 584,000 people worldwide, about 75% of whom were African children
under 5 years of age1. Many of these mortalities were thought to be due to CM2. This observation points
to the need for a highly specific diagnostic to ensure detection of CM.
Making an accurate diagnosis of CM remains a clinical challenge in many parts of Africa. In areas where
malaria is endemic, the presence of the malaria parasite is common, even among children without
symptoms. Thus, when a child presents with symptoms of cerebral malaria and is in a comatose state
from any cause, simply detecting malaria parasites in the blood is not sufficient for a specific diagnosis of
CM. In fact, a large autopsy study revealed that 23% of children with clinically defined CM were found
to have non-malaria causes2. This study also showed that the presence of highly specific lesions in the
retina, known as malarial retinopathy (MR), which was detected in the comatose children before death,
was the best indicator of CM as verified by autopsy. MR occurs in a spectrum of severity. Among
children with clinically defined CM, the acuteness of MR correlates well with severity of various signs of
CM. In the past two decades it has been observed that approximately 60% of children in the Blantyre
(Malawi, Africa research project) diagnosed with clinical CM, were positive for MR, suggesting 40% likely
had the symptoms from something other than CM.
Dilated Binocular Indirect Ophthalmoscopy (BIO) examination by an ophthalmologist is the current
reference standard for detection of MR. This form of examination depends upon the availability of
qualified and highly-skilled ophthalmic expertise. Even among experienced ophthalmologists there will
be disagreements on findings. Most non-ophthalmologists do not practice enough ophthalmoscopy to
be proficient in the BIO examinations. In the regions of the African continent where CM is highly
prevalent there is a scarcity of ophthalmic expertise. Thus, in most settings where CM is treated in
Africa, looking for signs of MR is not considered and thus a misdiagnosis can result. Additionally, in this
resource limited region of the world, the solution to making this medical diagnosis must be accessible
and affordable to the affected population.
Digital retinal imaging technology has improved recently, and the retinal imaging process is now possible
at sites other than the malaria care clinics, thus making possible the diagnoses of CM using image
visualization by ophthalmic experts located remotely, through tele-retinal screening services. However,
a routine use of retinal fundus photography is still problematic. One issue is that table top retinal
cameras are cumbersome and expensive and are not suitable for clinical environment in Africa. New
technologies using handheld retinal cameras and smart phone cameras may allow retinal fundus
imaging to be done simply and inexpensively. Retinal imaging with hand held devices opens for the first
time the possibility of providing clinicians who treat children in malaria areas, with high quality fundus
photos of children with clinically defined CM. BIO and retinal photography technologies have been
tested separately, thus no direct comparisons in a clinical setting. In this study we compared BIO exam
findings with the clinical performance of three retinal cameras in capturing signs of malarial retinopathy.
1 World Malaria Report, World Health Organization Global Malaria program, Dec 2014. 2 Taylor TE, et al. “Differentiating the pathologies of cerebral malaria by postmortem parasite counts.” Nat Med. 2004; 10:143-145.
VisionQuest i-Rx, Corp. ©2015 Page 3
Figure 1. Retinal cameras utilized in the study: Left: Peek. Middle: Pictor-plus. Right: Topcon
Study objective. The objective of this study was to test and compare three retinal cameras: the Pictor-
Plus (Volk Optical), the Topcon 50EX (Topcon Corporation) and the Peek camera (Portable eye
examination kit). We quantified the performance of these cameras for MR detection by comparing the
interpretation of images from each against the clinically accepted reference standard, BIO, in terms of
sensitivity, specificity, and Kappa agreement. The current clinical requirement for the verification of CM
diagnosis is MR detection with high specificity. Examination of a patient for MR will identify the non-CM
patients who may be mistakenly diagnosed for CM. Therefore, one of the criteria for analyzing the
camera performance included the specificity for MR detection, which is the first step toward
accomplishing the ultimate goal of reducing the false positive rate for CM. The results of this study can
be used by clinicians implementing MR screening programs to select the most appropriate camera for
their intended application and clinical setting. Figure 1 shows the three retinal cameras used in this
study.
The aim of this study was achieved through primary and secondary objectives. The primary objective
was:
1. To determine the inter-device agreement as judged by a single trained reader between, 1)
handheld cameras (Pictor-Plus and Peek) and a table top camera (Topcon), 2) handheld cameras
(Pictor-Plus and Peek) and BIO, and 3) Topcon and BIO, in classifying a clinically diagnosed CM
patient into MR positive or MR negative (as determined by the BIO exam); when BIO and camera
imaging are performed by an experienced ophthalmologist.
The secondary objectives were:
2. To determine the inter-device agreement as judged by the same observer between, 1) handheld
cameras (Pictor-Plus and Peek) and table top camera (Topcon), 2) handheld cameras (Pictor-Plus
and Peek) and BIO, 3) Topcon and BIO; (both performed by an experienced ophthalmologist) in
identifying and grading the various features of malarial retinopathy: (a) hemorrhages in macula (b)
peripheral hemorrhages (c) macular whitening, (d) peripheral whitening, (e) abnormal vessels and
(f) papilledema.
3. To determine the inter-camera variation in image collection quality in relation to the classification
of a clinically diagnosed CM patient into MR positive or MR negative, presence of papilledema, and
presence of the 3 individual signs of MR (hemorrhage, retinal whitening and vessel abnormalities)
between photos taken by Peek, Pictor-Plus, and Topcon; when all the three image sets are graded
by an independent retinal grader.
VisionQuest i-Rx, Corp. ©2015 Page 4
Figure 2. Signs of malarial retinopathy in retinal images. Left: vessel discoloration. Middle: white-centered retinal hemorrhages. Right: retinal whitening.
Table 1: Camera specifications
Camera FOV Pixel format Mode
Topcon 50 96 DPI, 8 bits Single frame
Pictor-Plus 45 72 DPI, 8 bits Single frame
Peek 20 72 DPI, 8 bits Video (30 frames/sec)
Camera Specifications. Two of the factors affecting the results of the inter camera comparison are,
naturally, the resolution and field of view (FOV) of each (See Table 1). While the optimum set of images
would include the posterior pole as well as the periphery of the retina, the limits of the cameras’ FOV
required the ophthalmologist to attempt to acquire multiple images to maximize the area covered by
each camera. The Peek had a different
implementation of the image capture
process. Peek acquires a video rather
than a single image. This allows the user
to scan the retina and collecting multiple
frames of the area.
Study Design. The study consisted of prospective data collection in the context of an ongoing descriptive
study of pediatric CM at the Queen Elizabeth’s Hospital in Blantyre, Malawi. A single ophthalmologist
(IM) performed MR assessment through BIO as well as mydriatic retinal imaging using the three retinal
cameras (Peek, Pictor-Plus, and Topcon) on 23 comatose children (aged 6 months - 12 years) admitted in
Hospital, with a Blantyre coma score of ≤2 and peripheral P. falciparum. Parents of the children provided
consent according to international research standards and with IRB approvals from the College of
Medicine Research and Ethics Committee, University of Malawi.
The retinal images were graded by a certified retinal grader for the detection of specific signs of MR. A
statistical agreement coefficient (Kappa) and sensitivity/specificity metrics were obtained by comparing
the retinal image assessments and indirect ophthalmoscopy examination findings to evaluate the
performance of each camera and its usability in identifying MR. The ophthalmologist also evaluated
each camera in terms of factors such as image quality, usability, ergonomics, portability, cost, etc.
Data. Of the N=23 children enrolled in the study, N=8 were controls; that is, they did not present with
signs of MR, and N=15 were clinically diagnosed with CM and presented signs of MR. Using an
internationally recognized grading system developed by the MR grading consensus group at the
University of Liverpool, the retinal images were graded by a certified retinal grader for specific signs of
MR. Figure 2 presents examples of the three lesions: vessel discoloration (left), white-centered retinal
VisionQuest i-Rx, Corp. ©2015 Page 5
hemorrhages (middle), and retinal whitening (right). The results of the reader’s evaluation of the images
were compared to the BIO exam results, and sensitivity, specificity, and Kappa for the presence of MR
lesions were calculated.
The BIO exams found 12 cases with whitening, 8 cases with hemorrhages, and 4 cases with vessel
discoloration. Several cases had multiple MR lesions. Papilledema was present in 3 cases with 2 of them
presenting signs of MR. The reader evaluated the images from each of the cameras tested. The Peek
camera, which is cell phone-based, captured an average of four videos per subject (two per eye); the
Pictor-plus, a handheld retinal camera, eight images per subject (four per eye); and finally, six images
per subject were captured with the Topcon, a desktop camera. The reader’s grading was masked to
previous reads from other cameras as well as the BIO results.
Figure 3 shows sample images for the same subject. This patient had retinal hemorrhages, the majority
of which were white centered, and retinal whitening. A large hemorrhage is centered over the
fovea/macula. For the Peek, a sample of three still frames from the video captured is presented in Figure
3. However, for the reading of the cases using Peek, the entire set of four 20-30 second videos is used.
Patches of retinal whitening are visible temporally from the macula in both the Topcon and Pictor-plus
views. The Topcon has a 50○ field of view (FOV) whereas the Pictor has 45○ FOV. As seen see in the
images, the Pictor captures the complete hemorrhages in the far temporal region of the retina, while
the Topcon has cut them in half due to its smaller horizontal FOV. The large central macula hemorrhage
and several smaller ones can be seen in the Peek video. Peek also captures signs of whitening, but is
frequently masked by glare or reflection artifacts, which makes the whitening detection difficult and
Figure 3. Images captured for same subject: (Top) Sample frames from video captured with Peek.
(Bottom) Left: Pictor-plus, Right: Topcon
VisionQuest i-Rx, Corp. ©2015 Page 6
Figure 4. Images showing retinal whitening: (Top) Sample frames from Peek video. (Bottom) Left: Pictor-plus, Right: Topcon. Whitening was not evident in the Peek image which led to incorrect evaluation of the subject.
unreliable. In addition to masking true whitening, these artifacts can be mistakenly misidentified as
whitening. Retinal reflexes are natural occurrences due to the nature of the retina and can be
differentiated from the glare artifacts mentioned previously. The retinal reflexes were more common
with Peek and were at minimum a distracting, but often masked the presence of one of the MR
pathologies. The artifacts in Peek can be seen in Figure 3 and Figure 4. The Peek has a limited FOV of
approximately 20○, and the views captured will depend on examiner’s skill. Furthermore, the design of
Peek’s camera attachment to a cell phone makes it difficult to bring the camera-optics close to the
patient’s cornea, which affects the image quality. The flat, rectangular shape of the Peek makes it
difficult to align and requires an unnatural handling of the device. Its small FOV and its flat rectangular
shape make the Peek a challenge to align in order to capture the intended region of the retina.
Figure 4 shows an example of a patient with retinal whitening in the temporal region of the retina (near
the macula), which was clearly seen on both the Pictor-plus and Topcon images. However, using the
Peek video, the physician could only image the area surrounding the optic disc, but could not image the
temporal region where the retinal whitening was present. Additionally, when the macula was captured,
there was significant reflection from cornea of the camera illumination source. Focusing the Peek was
problematic. Being able to both maintain the correct focus on the retina while at the same time moving
the camera in the direction needed to image all retinal regions is one aspect that adds difficulty when
trying to image all the areas of interest in the retina. The Peek camera may perform adequately in the
hands of an ophthalmologist experienced with this device and when used for immediate diagnosis, but it
is more challenging to master than the Pictor-plus or Topcon for a non-ophthalmic technician.
VisionQuest i-Rx, Corp. ©2015 Page 7
Results. This section describes the results of the comparison of each camera against the clinical
reference standard (BIO examination) and clinical imaging standard (Topcon), as well as an inter-camera
comparison in terms of grading for overall signs of MR and then by lesion types: hemorrhages,
whitening, and vessel discoloration. We also include the detection performance for Papilledema (PE) as
it represents one of the life-threatening signs associated with CM and observed in as many as 20% of CM
patients. We determined sensitivity/specificity metrics for overall MR and lesion detection, as well as
the Kappa agreement in defining the grade of each lesion. Due to the small sample of cases, the 95%
confidence intervals for sensitivity, specificity, and Kappa values overlapped between the three cameras.
Camera performance against BIO. Figure 5 shows sensitivity/specificity of each camera against the BIO
in detecting MR and individual lesions. From figure 5 we see that the Pictor-plus produced images with
quality and resolution that allows the best overall performance, sensitivity of 100% in detecting MR at a
specificity of 87%. The Pictor-plus performance was comparable or superior to the other two cameras in
terms of sensitivity and specificity of detecting MR lesions (Figure 5). Although, sensitivity for Topcon
images was as high as Pictor-plus, Topcon showed lower specificity as the retinal reader overcalled the
MR lesions in Topcon images. The reason for lower specificity when interpreting the Topcon images was
that contrast between the retinal background and MR lesions (e.g. whitening) was poorer in Topcon
images; whereas the contrast was relatively higher in the Pictor-plus images.
As described previously, the specificity for MR detection is of clinical importance for reducing the over-
diagnosis of CM. In this study, Pictor-plus provided the highest specificity among the three cameras.
However, none of the cameras imaged the vessel discoloration seen in the BIO examination by the
Figure 5. Sensitivity and specificity of each camera against BIO in detecting MR and its lesions. BIO exams revealed 8 cases with hemorrhages, 12 cases with whitening and 4 cases with vessel discoloration. Several cases had multiple MR lesions. Papilledema was present in 3 cases with 2 of them presenting signs of MR.
0%
20%
40%
60%
80%
100%
Malaria Retinopathy Hemorrhages Whitening Papilledema
Sensitivity to BIO Peek Pictor Topcon
0%
20%
40%
60%
80%
100%
Malaria Retinopathy Hemorrhages Whitening Papilledema
Specificity to BIO
VisionQuest i-Rx, Corp. ©2015 Page 8
ophthalmologist. This is a result of the vessel discoloration being in the far-periphery beyond the field of
view (FOV) of a single image acquisition by these cameras. This can be overcome by the photographer
realigning the camera to the periphery or using a lens with a wider FOV. For the small sample of cases,
the Pictor-plus performance was consistently comparable or superior to the other two cameras (See
Figure 5.
The reader assigned levels between 1 and 4 of the severity of each MR lesion based on criteria described
in Table 2. Hemorrhages and whitening were graded separately in each of the clinically significant retinal
regions such as macular, foveal, out of macula, and overall retinal region. In case of hemorrhages, a
proportion of white-centered hemorrhages among the total was also graded. Papilledema was detected
for its presence/absence only and not graded for its severity level.
Figure 6. Kappa agreement of each camera with BIO in determining the grades of hemorrhages and whitening.
0
0.2
0.4
0.6
0.8
1
Macular Out of macula White-centered Overall
Kappa for grade of hemorrhagesPeek and BIO Pictor and BIO Topcon and BIO
0
0.2
0.4
0.6
0.8
1
Macular Foveal Out of macula Overall
Kappa for grade of whiteningPeek and BIO Pictor and BIO Topcon and BIO
Table 2: MR lesion grading criteria
MR lesion Grading criteria Grade
1 2 3 4
Hemorrhage Number of hemorrhages 0 1-5 5-20 20-50
Whitening Area occupied compared
to the optic disc (OD) area 0
< 1/3 OD
1/3-1 OD
> 1 OD
Vessel discoloration
Color of discolored vessels N OR W NA
N: No color change, OR: Orange, W: White, NA: Not applicable
VisionQuest i-Rx, Corp. ©2015 Page 9
A Cohen’s Kappa agreement was calculated between each camera and the BIO for hemorrhages and
retinal whitening (Cohen’s Kappa was linearly weighted), as shown in charts (Figure 6). The 95%
confidence intervals (CI) were large and overlapping due to the limited amount of cases. Nonetheless,
the Pictor-plus camera performed consistently comparable or better than Peek and Topcon. The Kappa
agreement for vessel discoloration was nearly zero as none of the cameras could capture far-peripheral
views of retina where the discoloration is commonly observed; which was also evident from the
sensitivity, specificity results as before.
Camera performance against Topcon. A similar set of statistics as above was also calculated for the
sensitivity, specificity of Peek and Pictor-plus cameras against the Topcon camera (clinical standard for
retinal imaging), as shown in Figure 7. Again, Pictor-plus presented a higher overall sensitivity and
specificity compared to Peek. Similar to the previous results, the confidence intervals for sensitivity,
specificity values were large and the values between cameras were not statistically significantly
different. Vessel discoloration results were not included here as none of the cameras including Topcon
imaged regions of the retina where vessel discoloration was seen in the BIO exam. Figure 8 shows the
Cohen’s Kappa agreement of each of the cameras (Peek and Pictor-plus) with Topcon in determining the
grades of hemorrhages and whitening. The Pictor-plus camera performed consistently better than Peek.
The Kappa for vessel discoloration between Peek/Pictor and Topcon could not be determined as none of
the cameras including Topcon imaged the regions of vessel discoloration.
Figure 7. Sensitivity and specificity of each camera against Topcon in detecting MR and its lesions.
0%
20%
40%
60%
80%
100%
Malaria Retinopathy Hemorrhages Whitening Papilledema
Sensitivity to Topcon Peek Pictor
0%
20%
40%
60%
80%
100%
Malaria Retinopathy Hemorrhages Whitening Papilledema
Specificity to Topcon
VisionQuest i-Rx, Corp. ©2015 Page 10
Table 3. Image quality assessment
Image quality Peek Pictor Topcon
Macular 72% 83% 87%
Out of macula (Peripheral)
3% 47% 69%
Inter-camera agreement. Apart from calculating the
performance of each camera in detecting MR lesions, the
cameras were also evaluated for image quality by
quantifying the fraction of the total retinal area that was
available for grading (captured by the camera) and
gradable (adequate image quality as determined by the
retinal grader). The results are shown in table 3. As presented in Table 3, all the three cameras could
capture only a partial view of the peripheral retina.
Discussion. Although BIO has been the reference standard for identifying MR, it is important to
recognize that it may not necessarily be optimum for detecting MR. Retinal fundus photography has the
advantage of allowing the grader to study an area more closely and can provide higher magnification
than BIO. Desktop cameras such as the Topcon pose a significant challenge to the imager when trying to
maneuver the camera and perform alignment when the subject is comatose and in a supine position.
This was the major disadvantage of the Topcon.
The handheld retinal imaging systems such as the Pictor-Plus and the Peek provide varying quality color
fundus images or videos. Pictor-plus produces images with good quality and resolution to allow for the
best performance in terms of overall MR and lesion detection and presents a substantial agreement
(Kappa > 0.6) with BIO. These results for the Pictor-plus were comparable or superior to the Peek and
Topcon. Hemorrhages and whitening were detected with high accuracy in images from Pictor-plus and
Figure 8. Kappa agreement of each camera with Topcon in determining the grades of hemorrhages and whitening.
0
0.2
0.4
0.6
0.8
1
Macular Out of macula White-centered Overall
Kappa for grade of hemorrhagesPeek and Topcon Pictor and Topcon
0
0.2
0.4
0.6
0.8
1
Macular Foveal Out of macula Overall
Kappa for grade of whiteningPeek and Topcon Pictor and Topcon
VisionQuest i-Rx, Corp. ©2015 Page 11
Table 4: Factors evaluating the camera usability
Factor PEEK Pictor-Plus Topcon
Set-up Cellphone Handheld Desktop
Field of view 20○ 45○ 50○
Ergonomic Handy Handy Bulky
Size, weight Small, light Portable Cumbersome
Imaging stability Highly unstable Stable Stable
Image quality Poor, artifacts Adequate Adequate
Cost $500 $10,000 >$25,000
Figure 9. Snapshots of the Imaging process: Left: Peek, Middle: Pictor-plus, Right: Topcon
Topcon, and with lower accuracy using the Peek images. Because of the aforementioned difficulty in
aligning the Peek, peripheral regions were almost absent in Peek’s field of view (Table 3).
Apart from the accuracy of each
camera in detecting MR lesions,
the cameras were also evaluated
by the user (ophthalmologist) in
terms of usability, ergonomics and
relevant factors, as described in
table 4. Figure 9 shows the
imaging procedure for each
camera. The handheld Pictor-plus
camera was found by the
ophthalmologist to be more
stable, user-friendly, handy, and
ergonomically suitable for imaging
comatose children, unlike the Topcon. Topcon camera, although provides high accuracy for MR
detection, its bulky and difficult to handle especially when imaging The Peek camera is inexpensive,
small and lightweight, but its structure makes it hard to position the camera optics close enough to the
patient’s cornea to get the maximum field of view while holding the eyelids open. Peek was found to be
unstable for imaging due to patient’s non-cooperation and involuntary eye movements (quite common
among comatose children) which can result in poor imaging performance and image quality.
Peek imaging also resulted in number of artifacts which can make the images challenging for manual
grading as well as for automated image analysis. Peek imaging process necessitates the presence of a
trained ophthalmologist making an immediate diagnosis, but can be more challenging for a non-
ophthalmic technician or for a remote grading of images. Furthermore, because of the alignment
difficulty, the photographer could only manage to capture a limited region of the retina thereby missing
the vessel discoloration in the periphery. Considering the factors discussed above, the Pictor-plus offers
the best overall performance compared to other cameras in terms of image quality, MR and lesion
detection accuracy, ergonomic, size/weight, and costs.
VisionQuest i-Rx, Corp. ©2015 Page 12
Nonetheless, there are design changes for each of the three cameras that could improve their
performance in this specific application. Any table top retinal camera, including the Topcon, is
cumbersome and difficult to use given the comatose state and position of the malaria patient. Reading
of the images will likely be done offsite from the hospital bed, therefore the best quality image must be
forwarded for reading. None of the three cameras that were tested offer real-time image quality
feedback to the user. In these resource limited application in Africa, the imager will likely be a minimally
trained healthcare worker who may not recognize an inadequate image prior to transmitting the image
to the specialist who will make the diagnosis. In a rapidly progressing disease like cerebral malaria, the
time taken to re-image the patient could significantly impact the outcome of the patient’s recovery.
A single shot wide field of view ( 120○) image would improve significantly the utility of the retinal
imaging device, given that some MR lesions occur in the periphery where multiple images are otherwise
required to capture the full area of the posterior pole. Finally, a device that can include or link to an
automatic MR detection system is needed in these regions where immediate care is required, but
accurate diagnosis by a specialist is not easily available.
Future Work. A hand-held, low-cost, smart camera is needed that encompasses many of the attributes
of the Pictor-plus, but has wider field of view, integrated software for real-time image quality
assessment, and on-site, automatic MR/no MR determination.
Acknowledgments. We would like to thank Dr. Terrie Taylor, Dr. Ian MacCormick, Dr. Susan Lewallen,
and Dr. Simon Harding for their valuable contribution to the management of the study and relevant
clinical procedures. We also thank Michigan State University, MI, USA, and Blantyre Malaria Project,
University of Malawi College of Medicine, Malawi, Africa; for their valuable assistance in providing the
clinical site and staff required for conducting this study. Furthermore, we appreciate the assistance from
Dr. Andrew Bastawrous and Dr. Matthew Burton from the London School of Hygiene and Tropical
Medicine, UK, in providing the Peek camera equipment. We appreciate the loan of the Pictor Plus
camera from Volk Optical, Inc. This study was funded using VisionQuest i-Rx’s internal research and
development funds. None of the authors have any financial interest in any of the three cameras.