Tutorial on Medical Image RetrievalMedical Informatics Europe 2005

28.08.2005 Henning Müller, Thomas Deselaers

Service of Medical InformaticsGeneva University & Hospitals, SwitzerlandChair for Computer Science VIRWTH Aachen University, Germany

2©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Overview

•• Who are we?Who are we?•• Materials and MethodsMaterials and Methods•• SubjectsSubjects

•• IntroductionIntroduction•• Content-based image retrievalContent-based image retrieval•• Image processing and pre-processingImage processing and pre-processing•• Visual features (descriptors), matching & classificationVisual features (descriptors), matching & classification•• Medical applicationsMedical applications•• Demonstration, system aspectsDemonstration, system aspects•• User interactionUser interaction•• Evaluation of systems (Evaluation of systems (imageCLEFimageCLEF))•• Examples (Examples (medGIFTmedGIFT, IRMA), IRMA)

3©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Thomas Deselaers deselaers@cs.rwth-aachen.de

•• DiplDipl. (Master) Computer Science, RWTH . (Master) Computer Science, RWTH AachenAachen•• Thesis: Features for Image Retrieval in 2003Thesis: Features for Image Retrieval in 2003

•• RWTH RWTH Aachen Aachen University, GermanyUniversity, Germany•• Computer Science faculty & PhD student since 2004Computer Science faculty & PhD student since 2004

•• Member of the Member of the IRMAIRMA project project•• IRMA = Image Retrieval in Medical ApplicationsIRMA = Image Retrieval in Medical Applications

•• Research interestsResearch interests•• Image retrieval & medical imagingImage retrieval & medical imaging•• Pattern recognition & statistical modelingPattern recognition & statistical modeling•• Image object recognition & computer visionImage object recognition & computer vision

4©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Henning Müller

•• Master in Medical Informatics of the University ofMaster in Medical Informatics of the University ofHeidelberg (1997)Heidelberg (1997)•• DICOM Implementations in DICOM Implementations in teleradiology teleradiology at the Germanat the German

Cancer Research CentreCancer Research Centre

•• PhD on image retrieval at the University of GenevaPhD on image retrieval at the University of Geneva(2002)(2002)•• Viper project, outcome is the GNU Image Finding ToolViper project, outcome is the GNU Image Finding Tool•• Main areas are user interaction and performanceMain areas are user interaction and performance

evaluationevaluation

•• medGIFTmedGIFT project at the University and University project at the University and UniversityHospitals of Geneva since 2002Hospitals of Geneva since 2002•• imageCLEFimageCLEF medical taskmedical task

5©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Materials

•• DVDDVD containing some material on image retrieval containing some material on image retrieval•• Casimage Casimage databasedatabase

•• 8751 images with case descriptions8751 images with case descriptions•• Used for the Used for the imageCLEF imageCLEF competitioncompetition

•• Source codesSource codes•• GiftGift•• Interface in JavaInterface in Java•• RWTH-i6 (IRMA) retrieval engine, FireRWTH-i6 (IRMA) retrieval engine, Fire

•• ArticlesArticles•• LinksLinks

•• Printed Printed articlesarticles•• Limited due to copyright problemsLimited due to copyright problems

6©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

The need for content-based visual IR

•• Rising amount of visual dataRising amount of visual data is produced digitally is produced digitally•• Digital cameras at consumer pricesDigital cameras at consumer prices•• Publications on the InternetPublications on the Internet

•• Billions of imagesBillions of images

•• Journalists (Millions of images produced every day)Journalists (Millions of images produced every day)•• Trademarks (>100.000 visual marks in Switzerland alone)Trademarks (>100.000 visual marks in Switzerland alone)•• HospitalsHospitals (Geneva radiology: >30,000 images per day) (Geneva radiology: >30,000 images per day)

•• Only small part of the images is Only small part of the images is annotatedannotated•• Annotation is expensive, subjective, task dependentAnnotation is expensive, subjective, task dependent•• Not everything can be described by textNot everything can be described by text

7©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

The problems

•• How to How to formulateformulate a query visually? a query visually?•• SketchSketch•• Colored regionsColored regions•• Example imageExample image

•• But how to get a good example?But how to get a good example?

•• Regions in example imagesRegions in example images•• Pre-segmentedPre-segmented•• Marked with a pencilMarked with a pencil

•• How to How to representrepresent an image with an image with ««featuresfeatures»»•• Without knowing what someone is looking forWithout knowing what someone is looking for•• Features need to be extracted automaticallyFeatures need to be extracted automatically

8©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

The solution

9©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Some concepts

•• Query by image Query by image exampleexample(s) - QBE(s) - QBE•• « « page zero problempage zero problem » »

•• Relevance feedbackRelevance feedback•• Positive and negative feedbackPositive and negative feedback

•• Semantic gapSemantic gap•• Gap between the extracted visual features and theGap between the extracted visual features and the

semantic concepts a user searches forsemantic concepts a user searches for

•• Sensory gapSensory gap•• Gap between reality and the imagesGap between reality and the images

•• Through limited resolution, projection, Through limited resolution, projection, ……

10©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Other multimedia data for retrieval

•• Text (web pages mixed with other media)Text (web pages mixed with other media)•• Images, graphics, Images, graphics, ……•• Signals (ECG, EEG)Signals (ECG, EEG)•• SoundSound

•• Music retrievalMusic retrieval

•• VideoVideo•• Mix of media sound and temporal series of imagesMix of media sound and temporal series of images

•• 3D data3D data•• Tomographic Tomographic imagesimages•• ConstructionsConstructions

11©2005 Hôpitaux Universitaires de Genève / RWTH Aachen

Questions, Desires ?

?