Advances in the Use of Neurophysiologycally-based Fusion for Visualization and Pattern Recognition of Medical Imagery M. Aguilar, J. R. New and E. Hasanbelliu

11Knowledge Systems Knowledge Systems LaboratoryLaboratory

Advances in the Use of Advances in the Use of Neurophysiologycally-based Neurophysiologycally-based Fusion for Visualization and Fusion for Visualization and

Pattern Recognition of Medical Pattern Recognition of Medical ImageryImagery

M. Aguilar, J. R. New and E. M. Aguilar, J. R. New and E. HasanbelliuHasanbelliu

Knowledge Systems LaboratoryKnowledge Systems LaboratoryMCIS DepartmentMCIS Department

Jacksonville State UniversityJacksonville State UniversityJacksonville, AL 36265Jacksonville, AL 36265


OutlineOutline Introduce Med-LIFE.Introduce Med-LIFE. Revisit 3D image fusion architecture.Revisit 3D image fusion architecture. Compare 2D and 3D fusion results.Compare 2D and 3D fusion results. Fusion for segmentation and pattern Fusion for segmentation and pattern

recognition.recognition. Contextual zoom tool.Contextual zoom tool. Segmentation results.Segmentation results.


Med-LIFE: Learning, Image Med-LIFE: Learning, Image Fusion, and Exploration SystemFusion, and Exploration System


3D Shunt Equation3D Shunt Equation

ijkS

sijk

ijkc

cijkijkijk

IGxD

IGCxBAxx

]*)[(

]*[)(

2

222

2

)(

32),,(4

1

zyx

zyx eG

Shunting Neural Network Equation:

Where:A – decay rateB – maximum activation level (set to 1)D – minimum activation level (set to 1)IC – excitatory inputIS – lateral inhibitory inputC, Gc and Gs are as follows:

3D Shunt Operator Symbol

Grossberg (1968), Elias & Grossberg (1972)


2-Band 3D Fusion 2-Band 3D Fusion Architecture Architecture


4-Band 3D Fusion 4-Band 3D Fusion ArchitectureArchitecture


2D vs. 3D Fusion Results2D vs. 3D Fusion Results

MRI-PD MRI-T1 MRI-T2 SPECT

2D Fusion 3D Fusion


Color Fuse Result

4-Band Hybrid Fusion 4-Band Hybrid Fusion Architecture Architecture

T1Images

T2Images

Q

I

Y

ColorRemap

Noise cleaning &registration if needed

ContrastEnhancement

Between-band Fusionand Decorrelation

SPECTImages

PDImage _+

.

.

.

.

.

.


Hybrid Fusion ResultsHybrid Fusion Results

2D Fusion 3D Fusion


User-Driven Learning for User-Driven Learning for Segmentation & Pattern Segmentation & Pattern

RecognitionRecognition


Zoom in place supports:

1. focused attention 2. improved screen

real-estate usage

Contextual Zoom Contextual Zoom VisualizationVisualization

Zoom in place:1. occludes information 2. reduces efficiency by

forcing user to maintain context





• Developed based on COTS software developed by Idelix

• Supports visualization of fused imagery at multiple details levels

• Supports detailed analysis and selection for user-driven pattern learning…


User-Driven Pattern User-Driven Pattern Learning Learning

Supervised learning where training data Supervised learning where training data is selected by user/expert (Waxman et is selected by user/expert (Waxman et al).al).

Results assessed and corrected by user.Results assessed and corrected by user. Fuzzy ARTMAP neural network for fast Fuzzy ARTMAP neural network for fast

and stable learning.and stable learning. Address order sensitivity by introducing Address order sensitivity by introducing

N voters trained with alternate ordering N voters trained with alternate ordering of the training data.of the training data.


Pattern Recognition ResultsPattern Recognition Results


Heterogeneous VotingHeterogeneous Voting

Train 3 Fuzzy ARTMAP systems with Train 3 Fuzzy ARTMAP systems with parameters as before (different data parameters as before (different data orderings)orderings)

Train remaining 2 systems with all Train remaining 2 systems with all parameters as in the 3parameters as in the 3rdrd system system except for except for VigilanceVigilance (which is a (which is a threshold measure that controls the threshold measure that controls the sensitivity of the system).sensitivity of the system).


Homogeneous vs. Homogeneous vs. Heterogeneous VotersHeterogeneous Voters

5 Homogeneous Voters 5 Heterogeneous Voters


2D vs. 3D Fusion 2D vs. 3D Fusion Segmentation ResultsSegmentation Results

2D Fusion-based Segmentation

3D Fusion-based Segmentation


GeneralizationGeneralization

TrainingResults

TestingResults

Slice 11 Slice 10


ConclusionsConclusions

Modified fusion approach combines Modified fusion approach combines benefits of 2D and 3D fusion.benefits of 2D and 3D fusion.

Preliminary learning segmentation results Preliminary learning segmentation results indicate robustness across slices and indicate robustness across slices and cases.cases.

Demonstrated superior performance of 3D Demonstrated superior performance of 3D fusion for both visualization and pattern fusion for both visualization and pattern recognition.recognition.

Heterogeneous voting scheme improves Heterogeneous voting scheme improves learning performance.learning performance.


BACK-UPSBACK-UPS


2D vs. 3D Generalization2D vs. 3D Generalization

TestingResults

Slice 10

2D Fusion 3D Fusion


Image FusionImage Fusion

Documents

Advances in the Use of Neurophysiologycally-based Fusion for Visualization and Pattern Recognition of Medical Imagery M. Aguilar, J. R. New and E. Hasanbelliu