Towards Cognitive Vision:Knowledge and Reasoning

for Image Analysis and Understanding

Monique THONNAT

Orion teamINRIA Sophia Antipolis FRANCE

16/08/2004 Bonn 2

Introduction Past Research

Object Categorisation Program Supervision

Cognitive Vision Platform Application

Conclusion

Overview

16/08/2004 Bonn 3

Introduction: the Problem• Problem 1:

•What does it mean to perform image interpretation ?

semantic image interpretation (i.e. object classification)

•What does it mean to associate semantics to a particular image ?

16/08/2004 Bonn 4

Different interpretations of this image are possible:

• A light object on a dark background

• An astronomical object

• NGC4473 galaxy

Introduction: the Problem

Image semantics is not inside the image

Image interpretation depends on a priori knowledge

16/08/2004 Bonn 5

Introduction: the Problem

• Problem 2:

• Is it possible to build fully automatic image interpretation systems?

• Given an image and a set of vision programs: which programs have to be applied on a particular image ?

16/08/2004 Bonn 6

Introduction: the Problem• Is there a unique or an optimal solution ?

In this image:

shall we extract the biggest object

in the center?

shall we detect all the objects

in the image?

• If solution depends on the objective

low level processing related to high level interpretation?

16/08/2004 Bonn 7

Introduction: the Approach

Proposed approach: Knowledge-based Vision

• formalize the a priori knowledge for image interpretation in knowledge bases

• explicit the reasoning (how to use a priori knowledge)

• define sub-problems and build solutions independent of any application (object categorization, program supervision, video understanding)

• test the solutions on difficult real cases

16/08/2004 Bonn 8

Introduction Past Research

Object Categorisation Program Supervision

Cognitive Vision Platform Application

Conclusion

Overview

16/08/2004 Bonn 9

Object Categorisation

• Goal:

object categorization = find the class of an object

(versus object identification = find a particular individual)

• Focus:

complex natural objects with existing taxonomy

• Method:

a priori knowledge of hierarchy of classes

16/08/2004 Bonn 10

Object Categorisation: Method

Image

Parameters

Class +Description

Classification

Object description Image processing

Knowledge-based systemCLASSIC engine

16/08/2004 Bonn 11

Object Categorisation: Knowledge

Knowledge formalization:

• Class:• symbolic description of a class• implemented by frames

• Tree: • predefined hierarchy of classes• implemented by a tree of frames

• Rules:• inference of symbolic terms from numerical parameters • implemented by inference rules

16/08/2004 Bonn 12

Object Categorisation: Reasoning

CLASSIC:

• Categorization engine

Algorithm:

• Depth-first tree traversal from root (general class) to leaves (specific classes)

• 3 recursive phases:• data abstraction (activation of inference rules)• object - class matching• classification refinement (tree traversal)

16/08/2004 Bonn 13

Object Categorisation: Applications

• Galaxy classification:

• Fish sorting

• Zooplankton classification:

•Foraminifers classification:

• Pollen recognition:

16/08/2004 Bonn 14

Object Categorisation: Applications

35 parameters

Class=E5

SYGAL Knowledge based system

Galaxy description

Built with CLASSIC engine

16/08/2004 Bonn 15

Object Categorisation: Applications

Tree of galaxy classes (ref. De Vaucouleurs)

16/08/2004 Bonn 16

Object Categorisation: Conclusion

+ 1 unique engine CLASSIC for several applications

+ 3 PhD thesis (M-H Gandelin, S Liu-Yu, J-C Ossola)

+ 1 European project ASTHMA

- how to perform object segmentation?--> program supervision

- only one single object--> future research on cognitive vision

- mapping between domain objects and vision features

--> future research on ontology and learning

16/08/2004 Bonn 17

Program Supervision• Program supervision definition :

• automate the configuration and execution of programs (versus fixed procedure)• selection, scheduling, execution, and control of results

• Focus:

mainly image processing libraries of programs

• Method:

knowledge-based approach: explicit formalization of expertise on how to use programs

Planning techniques (HTN)

16/08/2004 Bonn 18

Program Supervision: Reasoning

Reasoning: Program selection in a library of programs Selected programs execution Evaluation and adjustment if needed

Program Supervision Engine

Library of programs

Program Utilisation

KBPlanning Execution

EvaluationRepair

resultsplan

(part of)

judgementsActionsto correct

1 2

3

4

56

7

correct

incorrect

Request + data

16/08/2004 Bonn 19

Program Supervision: Knowledge

Operator: a means of achieving a specific goal primitive operator: a particular program composite operator: a particular combination of

programs sequential decomposition (THEN) parallel decomposition (AND) specialization (OR)

Criteria: decision rules attached to an operator choice rules parameter initialization rules evaluation rules repair rules

16/08/2004 Bonn 20

Program Supervision: Applications

These techniques have been used for very different applications:

Road obstacle detection: off-line processing of stereovision programs real-time reconfiguration of programs for Vehicle

Driving Assistance

Medical imaging: segmentation of 3D MRI brain images

Astronomical imagery: morphological description of galaxies

16/08/2004 Bonn 21

Program Supervision : Conclusion+ 4 more or less sophisticated engines:

Ocapi, Planete (real-time) Pegase (repair), MedIA (planning)

+ 7 PhD thesis (V. Clement, J. Van den Elst, J-C Ossola, R. Vincent, M. Crubezy, M. Marcos, J-C Noel)

+ 1 European project PROMETHEUS

- communication between program supervision and image interpretation

--> future research on cognitive vision- program supervision engines handling data flow (videos) and temporal constraints (real-time)

--> future research on video understanding

16/08/2004 Bonn 22

Past Research : Where We Are

+ Formalization of knowledge & reasoning Feasibility proven on difficult cases Reusable engines & knowledge languages

- No general solution

only partial ones for 3 sub-problems object categorization, program supervision, video understanding

Knowledge bases still difficult to build

16/08/2004 Bonn 23

Introduction The Problem The Approach

Past Research Object Categorisation Program Supervision

Cognitive Vision Platform Application

Conclusion

Overview

16/08/2004 Bonn 24

Where we go: No general solution for image interpretation

--> Conception of a cognitive vision platform (Celine Hudelot PhD)

Knowledge bases still difficult to build --> Design of ontology driven knowledge bases

i.e. visual concept ontology, video event ontology (Nicolas Maillot & T. Van Vu PhDs)

--> Design of learning techniques to complement knowledge bases (N. Maillot & V. Martin PhDs)

Cognitive Vision: Introduction

16/08/2004 Bonn 25

Cognitive Vision Platform

Cooperation of three knowledge based systems: Separate the problem of image

interpretation into tractable tasks For each task:

Formalize the different types of knowledge involved in the image interpretation problem

Explicit the reasoning ( use of a priori knowledge)

16/08/2004 Bonn 26

Cognitive Vision Platform

A platform with 3 dedicated tasks Semantic data interpretation

Application expert knowledge (domain taxonomy and terminology)

Ontological engineering to facilitate knowledge acquisition

Data management Matching between numerical image data and

symbols Scene analysis using spatial reasoning

Image processing numerical object description program supervision techniques : to

automate the management of an image processing library

16/08/2004 Bonn 27

Cognitive Vision Platform

Data Management Knowledge Base

of Visual Concepts and Data

Data Management Engine

Interpretation Knowledge Base

of Application Domainand Visual Concepts

Interpretation Engine

Program SupervisionLibrary of

vision programs

Knowledge Base of Program Utilization

Program Supervision Engine

CurrentImage

Interpretation

ObjectHypotheses

Image Processing

Request

Numerical data

Image description

Visual Concept

Ontology

Use of visual concept ontology in a cognitive vision platform

16/08/2004 Bonn 28

Cognitive Vision: Interpretation

Goal: Find the semantic class of physical objects

observed on images

How: Perform the interpretation in the same way

experts do: Use of domain terminology and taxonomy Top down strategy

Physical object hypothesis propagation

16/08/2004 Bonn 29

Cognitive Vision: Interpretation

Knowledge Acquisition: Ontological engineering contribution

Ontology : set of concepts and relations useful to describe a domain

[maillot03]: contribution of a visual concept ontology for the task of object description:

spatio-temporal concepts color concepts texture concepts

Domain concepts are described by visual concepts

16/08/2004 Bonn 30

Cognitive Vision: Interpretation

Texture

Regular Texture Irregular Texture

Smooth

Texture

Weaved

Texture

Oriented

Texture

Periodic

Texture

Granulated

Texture

Marbled

Texture

Veined

Texture

3D

Texture

Visual concept ontology content: some texture concepts

16/08/2004 Bonn 31

Cognitive Vision: Interpretation

Knowledge Formalization Domain concept tree : specialization

relations Sub-part tree linked to domain conceptReflects the domain taxonomy

Class: a domain concept (plant leaf, pollen grain) described by visual concepts (green color and oval shape or pink and circular)

Representation by frames with slots

16/08/2004 Bonn 32

Cognitive Vision: Interpretation

Subpart Tree

Poaceae :

• Circular Shape

• Granulated Texture

• Pink Color

Poaceae

Pore Cytoplasm

Pore:

• Subpart of Poaceae

• Elliptic Shape

• Small Size

Domain knowledge described using visual concept ontology

16/08/2004 Bonn 33

Cognitive Vision: Interpretation

Knowledge FormalizationDomain Conceptname White_Flysub-class of InsectsList of componentsDomain Concept Fly_BodyDomain Concept 2 Fly_Antenna

Domain Conceptname Fly_Bodysub-part of White_FlyList of attributesST_VisualConcept Shape[oval] Elongation [important]Color_VisualConcept Hue [white]

Domain Conceptname Fly_Antennasub-part of White_FlyList of attributesST_VisualConcept Shape [line] Thickness [thin]Color_VisualConcept Hue [white]Spatial_Relation Connected [Fly_Body]Spatial_Relation Right_of [Fly_Body]

Sub-part

16/08/2004 Bonn 34

Cognitive Vision: Interpretation

Reasoning Depth-first domain concept tree traversal

Physical object hypotheses by building symbol grounding requests (visual object instance finding)

Matching between visual object instances and predefined classes

Classification refinement

16/08/2004 Bonn 35

Cognitive Vision: Data Management

Goal: Matching between symbols and sensor data

How: Data management, spatial reasoning, top

down and bottom up strategies Symbol grounding or Anchoring:

Anchoring = « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99]

16/08/2004 Bonn 36

Cognitive Vision: Data Management

Reasoning Image processing request building

according to visual object hypotheses (Object extraction criteria)

Matching between image processing results and symbolic data (Verification and data management criteria)

Instantiation and sending of visual objects to the Interpretation task

Spatial Reasoning: multiple objects (scene

analysis criteria)

16/08/2004 Bonn 37

Cognitive Vision: Data Management

Knowledge Formalization: Declarative knowledge:

Data concepts : primitives (ridge, region, edge), descriptors (compacity, area, perimeter)

Spatial relations : topology (RCC8), distance and orientation

Inferential knowledge : Object extraction criteria : to build image

processing requests Object verification criteria : to diagnose the

image processing results Scene analysis criteria: to manage multi-object

hypotheses

16/08/2004 Bonn 38

Cognitive Vision: Image Processing

Goal : Object extraction and numerical

description How:

Automate the configuration and execution of a library of programs for a given objective

Use of program supervision techniques

16/08/2004 Bonn 39

Cognitive Vision: Image Processing

Knowledge formalization: Declarative knowledge:

Goals: image processing functionality (thresholding, edge extraction,…)

Operators: knowledge to solve a given problem:

primitive: particular program composite: particular combination of programs

Requests: instantiations of goals on particular data, under particular constraints

16/08/2004 Bonn 40

Cognitive Vision: Image Processing

Knowledge FormalizationPrimitive OperatorName Ridge_LinkingFunctionality Ridge Point LinkingInput data in image_sequenceOutput data out imageParameters Thresh_first_dev

Thresh_second_dev nb_sigma

Preconditions in.coding.format =Calling syntax: PointLinking -s in out

Composite OperatorName Ridge_ExtractionFunctionality Extraction of ridge linear featuresInput data in imageOutput data out image

out_data ridge_feature_filParameters sigma

nb_sigmaSub-components MulRidPointDetection, RidgeLinkingDecomposition DO MulRidPointDetection

THEN Ridge_Linking

16/08/2004 Bonn 41

Application on plant disease diagnosis

Why Image Understanding ? Plant disease diagnosis = visual observation

which aims at inferring disease presence by the observation of signs and symptoms

TO BE ABLE TO REASON : signs and symptoms interpretation in terms of diseases

TO BE ABLE TO SEE : Focusing on relevant criteria

Star shape network of white and thin filaments (5-10 μ)Presence of elliptical white blobs in the centre of the networkClimatic Context: High humidity, Temperature : 25 °C

Early powdery mildew infection in propitious conditions

Early diagnosis:Microscopic image (x64) of rose leaf part

16/08/2004 Bonn 42

Application on Plant Disease Diagnosis: Rose Diseases

Powdery mildew :State of infection : earlyVegetal support : red leaf

Powdery mildew :State of infection : very earlyVegetal support : green leaf

Two white flies close to their eggs

Need of domain knowledge

Intelligent management of image processing programs

Complexity and variability of object appearance

Variability of contexts

Scene knowledge and spatial reasoning

Multiple objectsand various object types

16/08/2004 Bonn 43

Leaf

Healthy

Non Healthy

Insects

Virus

Fungi

White fly

Penicillium

Powdery mildew

Germinated tubes

Filamentous

Aphid

Vegetal tissue

Veins

red

green

Subpart

SubclassAcarid

Ungerminated

Pellets

Application on plant disease diagnosis

Domain knowledge base : the concept tree

Mycelium: • Part of : Fungi• network of at least 2 connected Hyphae•nb_hyphae = {unknown}

Hyphae:• Part of : Mycelium•Geometry: line•Thickness: thin, very thin•Straightness:=almost straight•Luminosity=bright•...

16/08/2004 Bonn 44

Input : User Request Fungi infection?

Image + Context

Variety : LeonidasLeaf : youngSeason: summerTemp: 24° CHumidity: 80...

Application : early detection of plant diseases

InterpretationDomain concept tree traversal to build visual object hypotheses

Leaf Scene

VegetalPart

Disease

Insects

Virus

Fungi

White fly

Penicillium

Powdery mildew

Dispersed

Clump

Aphid

Subpart

Subclass

AcaridVery Early

Pellet

1

Data Management Symbolic request to image processing

request

3

Goal: segmentationContraints:Image entity = ridge

Object.width = [1..3]Object.intensity > 150

Input Data: Image : input imageMask : area of interest

Image Processing Request

4

Image Processing:request solving by

program supervision techniques

5Image Data

Ridge 1

numerical descriptors

Ridge 3+ Numerical descriptors

Ridge 2+

NumericalDescriptors

6

InterpretationDomain concept tree traversal to build visual object hypotheses

Visual Object Hypothesis

2

Group of :Geometry: star shape network of { Geometry: line Thickness : thin width [7..10 m] very thin width [5..7 m] Straightness : almost straight Lightness: bright}Spatial Relation: Connected}

1

16/08/2004 Bonn 45

Application : early detection of plant diseases

Image Data

Ridge 1

numerical descriptors

Ridge 3+ Numerical descriptors

Ridge 2+

NumericalDescriptors

Data Management Visual object

hypothesis verification and instantiation

7

Interpretation : classificationMatching between visual object instances and domain concepts

9

Interpretation : diagnosis

Post classification rules activation

11

Visual Object InstanceNetwork of lines

Line 1

Line 2 Line 3

Line 5 Line 4

ECEC

ECEC

Line line1Thickness:=thin (0.8)Straightness:= straight (0.5)Lightness:=bright (0.7)Connected (line2)Connected (line4)+ link to image data

8

Diagnosis

Early powdery mildew infection on young leaf

12

Recognised domain concept

10

Freely dispersed mycelium

16/08/2004 Bonn 46

Conclusion

A generic platform for automatic recognition of natural objects a formalism and an ontology for knowledge base

building 3 dedicated reusable engines

semantic interpretation image/symbol matching and spatial reasoning management of a generic image processing library

Future works machine learning for image/symbol matching

(Nicolas Maillot) and for image segmentation (Vincent Martin)

16/08/2004 Bonn 47

Conclusion

Next Lectures Program Supervision (in more details)

Video Understanding (temporal scenario recognition)

16/08/2004 Bonn 48

Thank you for your attention!