M. Aguilar, J. R. New and E. Hasanbelliu Knowledge Systems Laboratory MCIS Department

11Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Advances in the Use of Advances in the Use of Neurophysiologycally-based Neurophysiologycally-based Fusion for Visualization and Fusion for Visualization and

Pattern Recognition of Medical Pattern Recognition of Medical ImageryImagery

M. Aguilar, J. R. New and E. M. Aguilar, J. R. New and E. HasanbelliuHasanbelliu

Knowledge Systems LaboratoryKnowledge Systems LaboratoryMCIS DepartmentMCIS Department

Jacksonville State UniversityJacksonville State UniversityJacksonville, AL 36265Jacksonville, AL 36265

22Knowledge Systems Knowledge Systems

LaboratoryLaboratory

OutlineOutline Introduce Med-LIFE.Introduce Med-LIFE. Revisit 3D image fusion architecture.Revisit 3D image fusion architecture. Compare 2D and 3D fusion results.Compare 2D and 3D fusion results. Fusion for segmentation and pattern Fusion for segmentation and pattern

recognition.recognition. Contextual zoom tool.Contextual zoom tool. Segmentation results.Segmentation results.

33Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Med-LIFE: Learning, Image Med-LIFE: Learning, Image Fusion, and Exploration SystemFusion, and Exploration System

44Knowledge Systems Knowledge Systems

LaboratoryLaboratory

3D Shunt Equation3D Shunt Equation

ijkS

sijk

ijkc

cijkijkijk

IGxD

IGCxBAxx

]*)[(

]*[)(

2

222

2)(

32),,(4

1

zyx

zyx eG

Shunting Neural Network Equation:

Where:A – decay rateB – maximum activation level (set to 1)D – minimum activation level (set to 1)IC – excitatory inputIS – lateral inhibitory inputC, Gc and Gs are as follows:

3D Shunt Operator Symbol

Grossberg (1968), Elias & Grossberg (1972)

55Knowledge Systems Knowledge Systems

LaboratoryLaboratory

2-Band 3D Fusion 2-Band 3D Fusion Architecture Architecture

66Knowledge Systems Knowledge Systems

LaboratoryLaboratory

4-Band 3D Fusion 4-Band 3D Fusion ArchitectureArchitecture

77Knowledge Systems Knowledge Systems

LaboratoryLaboratory

2D vs. 3D Fusion Results2D vs. 3D Fusion Results

MRI-PD MRI-T1 MRI-T2 SPECT

2D Fusion 3D Fusion

88Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Color Fuse Result

4-Band Hybrid Fusion 4-Band Hybrid Fusion Architecture Architecture

T1Images

T2Images

Q

I

Y

ColorRemap

Noise cleaning &registration if needed

ContrastEnhancement

Between-band Fusionand Decorrelation

SPECTImages

PDImage _+

.

.

.

.

.

.

99Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Hybrid Fusion ResultsHybrid Fusion Results

2D Fusion 3D Fusion

1010Knowledge Systems Knowledge Systems

LaboratoryLaboratory

User-Driven Learning for User-Driven Learning for Segmentation & Pattern Segmentation & Pattern

RecognitionRecognition

1111Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Zoom in place supports:

1. focused attention 2. improved screen

real-estate usage

Contextual Zoom Contextual Zoom VisualizationVisualization

Zoom in place:1. occludes information 2. reduces efficiency by

forcing user to maintain context

1212Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Contextual Zoom Contextual Zoom VisualizationVisualization

1313Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Contextual Zoom Contextual Zoom VisualizationVisualization

• Developed based on COTS software developed by Idelix

• Supports visualization of fused imagery at multiple details levels

• Supports detailed analysis and selection for user-driven pattern learning…

1414Knowledge Systems Knowledge Systems

LaboratoryLaboratory

User-Driven Pattern User-Driven Pattern Learning Learning

Supervised learning where training data Supervised learning where training data is selected by user/expert (Waxman et is selected by user/expert (Waxman et al).al).

Results assessed and corrected by user.Results assessed and corrected by user. Fuzzy ARTMAP neural network for fast Fuzzy ARTMAP neural network for fast

and stable learning.and stable learning. Address order sensitivity by introducing Address order sensitivity by introducing

N voters trained with alternate ordering N voters trained with alternate ordering of the training data.of the training data.

1515Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Pattern Recognition ResultsPattern Recognition Results

1616Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Heterogeneous VotingHeterogeneous Voting Train 3 Fuzzy ARTMAP systems with Train 3 Fuzzy ARTMAP systems with

parameters as before (different data parameters as before (different data orderings)orderings)

Train remaining 2 systems with all Train remaining 2 systems with all parameters as in the 3parameters as in the 3rdrd system system except for except for VigilanceVigilance (which is a (which is a threshold measure that controls the threshold measure that controls the sensitivity of the system).sensitivity of the system).

1717Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Homogeneous vs. Homogeneous vs. Heterogeneous VotersHeterogeneous Voters

5 Homogeneous Voters 5 Heterogeneous Voters

1818Knowledge Systems Knowledge Systems

LaboratoryLaboratory

2D vs. 3D Fusion 2D vs. 3D Fusion Segmentation ResultsSegmentation Results

2D Fusion-based Segmentation

3D Fusion-based Segmentation

1919Knowledge Systems Knowledge Systems

LaboratoryLaboratory

GeneralizationGeneralizationTrainingResults

TestingResults

Slice 11 Slice 10

2020Knowledge Systems Knowledge Systems

LaboratoryLaboratory

ConclusionsConclusions Modified fusion approach combines benefits Modified fusion approach combines benefits

of 2D and 3D fusion.of 2D and 3D fusion. Preliminary learning segmentation results Preliminary learning segmentation results

indicate robustness across slices and cases.indicate robustness across slices and cases. Demonstrated superior performance of 3D Demonstrated superior performance of 3D

fusion for both visualization and pattern fusion for both visualization and pattern recognition.recognition.

Heterogeneous voting scheme improves Heterogeneous voting scheme improves learning performance.learning performance.

2121Knowledge Systems Knowledge Systems

LaboratoryLaboratory

BACK-UPSBACK-UPS

2222Knowledge Systems Knowledge Systems

LaboratoryLaboratory

2D vs. 3D Generalization2D vs. 3D Generalization

TestingResults

Slice 10

2D Fusion 3D Fusion

2323Knowledge Systems Knowledge Systems

LaboratoryLaboratory

Image FusionImage Fusion