Embed Size (px)
Citation preview
Event Understanding in Endoscopic Surgery
VideosMario Guggenberger*, Michael Riegler°, Mathias Lux*, Pål Halvorsen°
Workshop on Human Centered Event Understanding from Multimedia (HuEvent14)
*Institute of Information TechnologyAlpen-Adria-Universität Klagenfurt
Austria
°Media Performance GroupSimula Research Laboratory AS
Norway
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
2
The problem: Analysis of endoscopic surgeries
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
3
What we did
• Development of an annotation tool for endoscopic surgery videos• Evaluation with a renowned surgeon• Thinking aloud test & interview
• Evaluation of the evaluation• Conceptual findings (event model)• Technical findings (annotation tool requirements)
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
4
Take home message
• Surgeries can be seen as events• Which contain subevents
• Hierarchical model of endoscopic surgery events
• Granularity of the events is connected to the type of annotation
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
5
Annotation tool
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
6
Thinking aloud test
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
7
Conceptual findings 1/2
• Surgery can be segmented into events and subevents.• Granularity of event is directly connected to annotation type.
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
8
The event model
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
9
Technical findings 1/3
• Prepopulate timeline• With automatically detected intervals (preprocessing step).• With bookmarks marked as key events
during the surgery (planned).
• No interest in classical image processing methods• E.g. image sharpness, dominant colors, movement intensity.• Except for out-of-patient detection!
• Drawn annotations on moving video are pointless• Except maybe with object tracking study with Rikshospital.
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
10
Technical findings 2/3
• Tablet size (7‘‘ too small, min 10‘‘)• Full screen video, control overlay
• Pen! • Frame-exact seeking
• Surgeon noticed even single-frame misplacements
• Non computer savvy surgeons do not know UI paradigms• Action feedback needs greater visual or tactile impact
• Zoomed timeline• Separation of voice and drawn annotations anticipated
• But brings up many problems
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
11
Technical findings 3/3
• Identified stroke usage types:• Marking borders with solid stroke• Marking areas with dots or hatched lines (optional solid stroke border)• Arrows to indicate directional actions• thin pen, thick felt pen, semitransparent marker
• Zoom-in• Export annotated stills• FF unimportant segments, SlowMo important segments• Render standalone video• Multimedia integration (e.g. linking x-ray images)• Measurements (size, area, structure, color)
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
12
A little bit of advertisement…
• ITEC MediaPlayer Library for Android• GPL licensed, sources on GitHub
• http://git.io/SoH2ig• API-compatible • Frame-exact seeking• Playback speed adjustment• GLES support
• Shader effects, Gesture Zoom & Pan• Frame extraction• Supports local and network sources
• incl. MPEG-DASH• Demo-App on the Play Store
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
13
Conclusion
• Deduced a general event model of a surgery• direct mapping between event granularity and annotation type
• Evaluation of an annotation tool for endoscopic surgery videos• Evaluated with a renowned surgeon• On his own recordings
• Provided many insights, ideas and a better understanding for developing appropriate tools and techniques to understand surgeries better
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula) / HuEvent14
14
Thank You!Questions?
