Schepens Eye Research Institute
Development of an Objective, Language-Based Measure of
Perceived Video QualityDaniel R. Saunders,
Peter J. Bex,and Russell L. Woods
Schepens Eye Research Institute and Harvard Medical SchoolBoston, Massachusetts
Schepens Eye Research Institute
• Dedicated to fighting blindness and visual impairment• Treatments and assistive technology for eye disorders
• Example: Age-related Macular Degeneration– Affects at least 1.75 million in the U.S.– Central vision loss, leading to reduced
contrast sensitivity and visual acuity– No cure
The Schepens Eye Research Institute
Daniel Saunders
Schepens Eye Research Institute
Digital video enhancement
Daniel Saunders
Contrast enhancement (Peli and Peli, 1984; DigiVision CE-3000)
QuickTime™ and aH.264 decompressor
are needed to see this picture.
Schepens Eye Research Institute
Digital video enhancement
Daniel Saunders
Digital magnification
QuickTime™ and aVideo decompressor
are needed to see this picture.
Schepens Eye Research Institute
How to evaluate enhancements?• Easy to measure preference:
– Dual monitor– Rating
Adaptive contrast enhancement has been shown to be preferred in many cases. (Kim et al., 2004; Fullerton et al., 2008)
But subjective.
An objective measure would derive from how much it helps people perform a task. Reading, driving, watching TV.
Daniel Saunders
Schepens Eye Research Institute
Information transfer• Principle: The better the perceived quality of a video
clip, the more (and more accurate) information the viewer should be able to report about it.
• Impaired perception should result in reduced amount of accurate information.
• Use of multiple choice quiz questions has problems– Creation of questions is subjective– Labour intensive– Also may have low sensitivity (Peli, 2005)
Daniel Saunders
Schepens Eye Research Institute
Using free recall to measure information transfer
Daniel Saunders
“Describe this video as if to someone who hasn't seen it.”
Standard set of 200 video clips, 30 seconds each, from Hollywood films and nature documentaries.
Schepens Eye Research Institute
Collecting “normal” responses• 4000 responses from Amazon.com Mechanical Turk
workers (20 per clip) working from home• 2400 responses from normally-sighted in a lab
setting, stratified by age and gender (in progress)
Daniel Saunders
A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead appears from behind the
spotlight wearing a black negligee and robe. She flirtatiously walks over to the caped crusader and they
converse.
Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman
must thwart and she seems helpful by her look.
Batman flies onto a roof top and starts talking to a blonde woman.
The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A woman with long hair comes
out and speaks with Batman.
It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole
Kidman to greet him there.
It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's a very dark scene,
when Batman arrives. He is then approached by a young woman with long wavy red hair in a black robe and slinky
nightgown. There is a slight breeze blowing her hair.
A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the spotlight. A
woman wearing a sexy dress walks into the room and talks with Batman.
Examples:
Schepens Eye Research Institute
Evaluate responses in terms of their discrepancy
Daniel Saunders
A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead
appears from behind the spotlight wearing a black negligee and robe. She flirtatiously walks over to
the caped crusader and they converse.
Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman must thwart and she seems
helpful by her look.
Batman flies onto a roof top and starts talking to a blonde woman.
The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A
woman with long hair comes out and speaks with Batman.
It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole Kidman to
greet him there.
It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's
a very dark scene, when Batman arrives. He is then approached by a young woman with long
wavy red hair in a black robe and slinky nightgown. There is a slight breeze blowing her
hair.
A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the
spotlight. A woman wearing a sexy dress walks into the room and talks with Batman.
• Score for an individual is average distance of their responses from the normally-sighted responses.
• A high score indicates poor vision/perceived video quality.
Schepens Eye Research Institute
Text passage similarity metrics• Baseline: shared word count (common word
removed)– Performance (Mechanical Turk): Mean shared words
within a clip = 6.8, outside a clip = 1.7. Accuracy at matching responses to clips = 95%.
• Latent Semantic Analysis could deal with synonyms etc. Used in essay grading and finding articles by matching abstracts. – Not our wheelhouse!
Daniel Saunders
Schepens Eye Research Institute
Assessing validity of the measurePerceived video quality = Perception + video quality.
Measure can be used to assess video, with applications in e.g. video compression algorithms.
Future studies:• Normally-sighted participants watch blurred video.
– Greater blur should cause larger discrepancy scores– Within-subject design, so compensates for variability
• Visually-impaired participants watch enhanced video.
Daniel Saunders
Schepens Eye Research Institute Daniel Saunders
What we have: • An application (evaluating the benefit of video
enhancement to the visually impaired).• A dataset (free descriptions of 200 video clips).
What we need: A metric of text passage discrepancy. (LSA?)
Thanks!
Schepens Eye Research Institute
Mechanical Turk for data collection
Daniel Saunders 13 / 31
• 3000 responses in 30 days
Schepens Eye Research Institute
No objective measure of perceived video quality
• Most are based on ratings or side-by-side comparisons
• Relevant to perception as well as to videos.
Daniel Saunders 14 / 31
Schepens Eye Research Institute
Age-related macular degeneration• Affects more than 1.75 million people in the United
States(Friedman et al., 2004), 3 million by 2020.• Causes 50% of all cases of blindness in the United
States (WHO, 2002).• No cure, only aids.• Which aids really help? (the same question as
perceived visual quality)
Daniel Saunders 15 / 31
Schepens Eye Research Institute
TV is an important activity• How can we help people watch TV?• How can we evaluate whether we’ve helped?
Daniel Saunders 16 / 31
20% of seniors who went in for a visual impairment exam said that one of their main motivations was watching TV.
69% of people with central field loss reported at least “some” difficulty with watching TV