Medical Image Registrationee.sharif.edu/~miap/Files/Image RegistrationLecture.pdf · • Anatomy...

Medical Image Medical Image RegistrationRegistration

IntroductionIntroduction

Medical Images Applications:Medical Images Applications: Clinical DiagnosisClinical Diagnosis Treatment PlanningTreatment Planning Image Guided Surgery (Intervention)Image Guided Surgery (Intervention) Retrospective Studies (Chemical/Beam Retrospective Studies (Chemical/Beam Therapy)Therapy) PrePre--Surgery Simulation (VR environment)Surgery Simulation (VR environment) Growth StudyGrowth Study TrainingTraining

Medical Imaging ModalitiesMedical Imaging Modalities Visual description (wide variety) of organs Visual description (wide variety) of organs characteristics:characteristics: CT (XCT (X--Ray absorption)Ray absorption) MRI (TMRI (T11, T, T22, and H, and H++ distribution density)distribution density) US (Reflection coefficients, TOF, Doppler)US (Reflection coefficients, TOF, Doppler) Gamma Camera (Nuclear source radiation)Gamma Camera (Nuclear source radiation) SPECT (Nuclear source radiation, Tomography)SPECT (Nuclear source radiation, Tomography) PET (Nuclear source radiation, Tomography)PET (Nuclear source radiation, Tomography) Video (Laparoscopy, Video (Laparoscopy, EndoscopyEndoscopy , and , and LaryngoscopyLaryngoscopy))

Medical ImagesMedical Images

Two major categories:Two major categories: Anatomical: Anatomical: •• AnatomyAnatomy

•• MorphologyMorphology Functional:Functional:•• PhysiologyPhysiology

•• MetabolismMetabolism

Anatomical ModalitiesAnatomical Modalities

ExamplesExamples XX--Ray (DSA)Ray (DSA) CT (CTA)CT (CTA) MRI (MRA)MRI (MRA) US (Doppler)US (Doppler) Video (XVideo (X--ScopyScopy))

Functional ModalitiesFunctional Modalities

Examples:Examples: Gamma CameraGamma Camera SPECTSPECT PETPET Brain Mapping (EEG/MEG)Brain Mapping (EEG/MEG) fMRIfMRI fCTfCT

Medical Image Integration (1)Medical Image Integration (1)

Why:Why: Too many data:Too many data:•• Limitation of conventional Diagnosis (hardcopy)Limitation of conventional Diagnosis (hardcopy) Data Dependency:Data Dependency:•• Multiple information from a single sliceMultiple information from a single slice Physical Requirements:Physical Requirements:•• StereotacticStereotactic surgery (Physical frames)surgery (Physical frames)

Medical Image Integration (2)Medical Image Integration (2)

The goal is data integration:The goal is data integration: RegistrationRegistration•• Bring the modalities involved into spatial Bring the modalities involved into spatial alignmentalignment Fusion:Fusion:•• Integrated display of the data involvedIntegrated display of the data involved Vocabulary:Vocabulary:•• WarpingWarping, Co, Co--Registration, Matching, Alignment, Registration, Matching, Alignment,

Normalization, Morphing.Normalization, Morphing.

Registration DefinitionRegistration Definition General Definition:General Definition: A Transform (Mathematical mapping) which A Transform (Mathematical mapping) which relatesrelatesposition of position of correspondencecorrespondence structures in two images.structures in two images. Correspondence:Correspondence:

•• Exact: Point by point.Exact: Point by point.

•• NonNon--Exact: Structure by structureExact: Structure by structure Two Images:Two Images:

•• SourceSource--DestinationDestination

•• HeadHead--HatHat

•• SourceSource--TargetTarget

Mathematical mappingMathematical mapping

ExamplesExamples

A Mathematical MappingA Mathematical Mapping

Funny ExampleFunny Example

Correspondence (1)Correspondence (1)

Related Point

Related Structure

Landmark

Correspondence (2)Correspondence (2)

Registration ExamplesRegistration Examples

Registration ExampleRegistration ExampleDefinitionDefinition of local of local landmarkslandmarks

DefinitionDefinition of a of a deformationdeformation modelmodelBeforeBefore registrationregistrationAfterAfter deformabledeformable registrationregistration AveragingAveraging of 9 of 9 brainsbrains

Fusion ExampleFusion Example

CT-SPECT Fusion MR-SPECT Fusion

General ApplicationGeneral Application

Uses of image registrationUses of image registration•• Image segmentation/deformable atlasImage segmentation/deformable atlas

•• Characterization of normal vs. abnormal Characterization of normal vs. abnormal Shape/variationShape/variation

•• MultiMulti--modality fusionmodality fusion

•• Functional brain mapping/removing shape variationFunctional brain mapping/removing shape variation

•• Surgical planning and evaluationSurgical planning and evaluation

•• Image guided surgeryImage guided surgery

•• PrePre--surgical Simulationsurgical Simulation

•• Other field (GIS and etc.)Other field (GIS and etc.)

Mathematical DefinitionMathematical Definition

An optimization problem:An optimization problem: Source and Target Images:Source and Target Images:

Seek for a Mapping: Seek for a Mapping:

•• Map features ofMap features of ΣΣ region of region of SS to same region of to same region of DD Goal of optimization:Goal of optimization:

( ) ( ) 1 2 1 2

1 2 1 21, 1 1, 1, ,x Y x YN N N N

S Dx x x xI x x I x x

= = = =

1 2( , ) :x xT T T S D= →

( ) ( ) ( ) ( )( )2

1 21 2

1 1 2 1 1 2 2 1 2,1 1

, arg min , , , ,yx

x x D ST T

T T I x x I T x x T x x= =

= − ∑∑

Classification of MethodsClassification of Methods

ObjectObjectSubjectSubjectModalities Modalities involvedinvolved

Optimization Optimization procedureprocedureInteractionInteractionDomain of Domain of

transformationtransformation

Nature of Nature of transformationtransformation

Nature of Nature of Registration Registration basisbasis

DimensionalityDimensionality

Image are spatial or temporally relatedImage are spatial or temporally related Spatial Dimension:Spatial Dimension:•• 2D2D--2D2D

•• 2D2D--3D3D

•• 3D3D--3D3D Temporal (Time Series) with spatial dimension:Temporal (Time Series) with spatial dimension:•• 2D2D--2D2D

•• 2D2D--3D3D

•• 3D3D--3D3D

Spatial Registration MethodSpatial Registration Method

3D/3D registration of two images3D/3D registration of two images

2D/2D registration:2D/2D registration: Less complex by an order of magnitudeLess complex by an order of magnitude

2D/3D registration2D/3D registration Direct alignment of spatial data to projective Direct alignment of spatial data to projective data.data. Alignment of a single Alignment of a single tomographictomographic slice to slice to spatial data. (Atlas based segmentation)spatial data. (Atlas based segmentation)

Temporal RegistrationTemporal Registration

Time series of images application:Time series of images application: Monitoring of bone growth in children (long Monitoring of bone growth in children (long time interval)time interval) Monitoring of tumor growth (medium interval)Monitoring of tumor growth (medium interval) PostPost--operative monitoring of healing (short operative monitoring of healing (short interval)interval) Observing the passing of an injected bolus Observing the passing of an injected bolus through a vessel tree (ultrathrough a vessel tree (ultra--short interval)short interval)

Two images need to be compared.Two images need to be compared.

Nature of Registration BasisNature of Registration Basis

Image basedImage based ExtrinsicExtrinsic•• Based on foreign objects introduced into the Based on foreign objects introduced into the

imaged space.imaged space. IntrinsicIntrinsic•• Based on the image information as generated by Based on the image information as generated by

the patientthe patient

NonNon--image based (calibrated coordinate image based (calibrated coordinate systems)systems)

Extrinsic registration methodsExtrinsic registration methods

AdvantageAdvantage:: Registration is easy, fast, and can be automated.Registration is easy, fast, and can be automated. No need for complex optimization algorithms.No need for complex optimization algorithms.

DisadvantageDisadvantage Prospective character must be made in the preProspective character must be made in the pre--acquisition phase.acquisition phase. Often invasive character of the marker objects.Often invasive character of the marker objects. Patient Problems (Movement, detachment)Patient Problems (Movement, detachment) NonNon--invasive markers can be used, but less accurate.invasive markers can be used, but less accurate.

InvasiveInvasive StereotacticStereotactic frameframe FiducialsFiducials (screw markers)(screw markers)

NonNon--invasiveinvasive Mould,frame,dentalMould,frame,dental adapter,etcadapter,etc FiducialsFiducials (skin markers)(skin markers)

The registration transformation is often The registration transformation is often restricted to be rigid (translations and restricted to be rigid (translations and rotations only)rotations only)

Rigid transformation constraint, and Rigid transformation constraint, and various practical considerations, use of various practical considerations, use of extrinsic 3D/3D methods are limited to extrinsic 3D/3D methods are limited to brain and orthopedic imagingbrain and orthopedic imaging

Intrinsic registration methodsIntrinsic registration methods

Landmark basedLandmark based

Segmentation basedSegmentation based

VoxelVoxel property basedproperty based

Landmark based registrationLandmark based registration

AnatomicalAnatomical:: Salient and accurately locatable points of the Salient and accurately locatable points of the morphology of the visible anatomy, usually morphology of the visible anatomy, usually identified by the useridentified by the user

GeometricalGeometrical:: Points (structure) at the locus of the optimum Points (structure) at the locus of the optimum of some geometric property, e.g., local of some geometric property, e.g., local curvature curvature extremaextrema, corners, etc, generally , corners, etc, generally localized in an automatic fashion.localized in an automatic fashion.

Landmark Based Landmark Based RedistrationRedistration

T(y) S(x) T(h(x))

h(x)⇒

Landmark based registrationLandmark based registration The set of registration points is sparse:The set of registration points is sparse: Fast optimization proceduresFast optimization procedures Optimize Measures :Optimize Measures : Average distance between each landmark.Average distance between each landmark. Closest counterpart (Procrustean Metric).Closest counterpart (Procrustean Metric). Iterated minimal landmark distances.Iterated minimal landmark distances. AlgorithmAlgorithm Iterative closest point (ICP)Iterative closest point (ICP) Procrustean optimumProcrustean optimum QuasiQuasi--exhaustive searches, graph matching and exhaustive searches, graph matching and dynamic programming approaches.dynamic programming approaches.

Segmentation based RegistrationSegmentation based Registration

Rigid model based:Rigid model based: Anatomically the same structures (mostly Anatomically the same structures (mostly surfaces) are extracted from both images to surfaces) are extracted from both images to be registered, and used as the sole input for be registered, and used as the sole input for the alignment procedure.the alignment procedure.

Deformable model based:Deformable model based: An extracted structure (also mostly surfaces, An extracted structure (also mostly surfaces, and curves) from one image is elastically and curves) from one image is elastically deformed to fit the second image.deformed to fit the second image.

Rigid Model BasedRigid Model Based

““HeadHead--HatHat”” method:method: Rely on the segmentation of the skin surface Rely on the segmentation of the skin surface from CT,MR, and PET images of the head.from CT,MR, and PET images of the head.

Chamfer matching:Chamfer matching: Alignment of binary structures by means of a Alignment of binary structures by means of a distance transformdistance transform

Deformable model basedDeformable model based

Deformable curves:Deformable curves: Snakes, active contours,nets(3D)

Data structure:Data structure: Local functions, i.e., splines

Deformable model approach:Deformable model approach: Template model defined in one image template is deformed to match second image:

• Segmented structure• Un-segmented

VoxelVoxel property based registrationproperty based registration

Operate directly on the image grey valuesOperate directly on the image grey values

Two approaches:Two approaches: Immediately reduce the image gray value Immediately reduce the image gray value content to a representative set of scalars and content to a representative set of scalars and orientationsorientations Use the full image content throughout the Use the full image content throughout the registration process.registration process.

VoxelVoxel Property Based RegistrationProperty Based Registration

T(y) S(x) T(h(x))

h(x)⇒

Principal axes and moments basedPrincipal axes and moments based

Image center of gravity and its principal Image center of gravity and its principal orientations (principal axes) are computed orientations (principal axes) are computed from the image zero and first order from the image zero and first order momentmoment Align the center of gravity and the principal Align the center of gravity and the principal

orientationsorientations Principal axes :Easy implementation, no high Principal axes :Easy implementation, no high accuracyaccuracy Moment based: require preMoment based: require pre--segmentationsegmentation

Full image content basedFull image content based

Use all of the available information Use all of the available information throughout the registration process.throughout the registration process.

Automatic methods presentedAutomatic methods presented

Example of MethodsExample of Methods CrossCross--correlationcorrelation Fourier domain based.Fourier domain based. Minimization of variance of gray values within Minimization of variance of gray values within segmentationsegmentation Minimization of the histogram entropy of Minimization of the histogram entropy of difference images.difference images. Histogram clustering and minimization of Histogram clustering and minimization of histogram dispersionhistogram dispersion Maximization of Maximization of mutual informationmutual information Minimization of the absolute or squared intensity Minimization of the absolute or squared intensity differencesdifferences

NonNon--image based registrationimage based registration

Calibrated coordinate systemCalibrated coordinate system If the imaging coordinate systems of the two If the imaging coordinate systems of the two scanners involved are somehow calibrated to scanners involved are somehow calibrated to each other, which necessitates the scanners each other, which necessitates the scanners to be brought in to he same physical locationto be brought in to he same physical location

Registering the position of surgical tools Registering the position of surgical tools mounted on a robot arm to imagesmounted on a robot arm to images

Nature of TransformationNature of Transformation

Rigid (Translation and Rotation)Rigid (Translation and Rotation)

Affine (Parallel Lines to Parallel lines)Affine (Parallel Lines to Parallel lines)

Projective (Line to Line)Projective (Line to Line)

Curved (Elastic)Curved (Elastic)

Domain of transformationDomain of transformation

GlobalGlobal Apply to entire imageApply to entire image

LocalLocal Subsections have their ownSubsections have their own

ExamplesExamplesRigid

Affine

Projective

Nonlinear

Local Global

TransformationTransformation

Many methods require a preMany methods require a pre--registration registration (initialization) using a rigid or affine (initialization) using a rigid or affine transformationtransformation

Global rigid transformation is used most Global rigid transformation is used most frequently in registration applications frequently in registration applications ((FiducialFiducial markers).markers).

Application: Human headApplication: Human head

InteractionInteraction

InteractiveInteractive

SemiSemi--automaticautomatic

AutomaticAutomatic Present method not act fine.Present method not act fine.

Goal:Goal: Minimal interaction and speed, accuracy, or Minimal interaction and speed, accuracy, or robustnessrobustness

InteractionInteraction

Extrinsic methodsExtrinsic methods AutomatedAutomated SemiSemi--automaticautomatic Intrinsic methodsIntrinsic methods SemiSemi--automaticautomatic

•• Anatomical landmarkAnatomical landmark•• Segmentation basedSegmentation based AutomatedAutomated•• Geometrical landmarkGeometrical landmark•• VoxelVoxel property basedproperty based

Optimization ProcedureOptimization Procedure

Parameters for registration transformationParameters for registration transformation Parameters computed AnalyticallyParameters computed Analytically Parameters searched for reach an criteria.Parameters searched for reach an criteria.

Methods:Methods: Downhill simplex methodDownhill simplex method LevenbergLevenberg--Marquardt optimizationMarquardt optimization Simulated annealingSimulated annealing Genetic methodsGenetic methods QuasiQuasi--exhaustive searchingexhaustive searching

Modalities involvedModalities involved

MonomodalMonomodal

MultimodalMultimodal

Modality to modelModality to model

Patient to modalityPatient to modality

MonomodalMonomodal

One patient, One modalityOne patient, One modality CT, MRI, US, PET, SPECT, and etc.CT, MRI, US, PET, SPECT, and etc. Registration of temporal sequencesRegistration of temporal sequences::•• Temporal deformation of anatomical structures Temporal deformation of anatomical structures

(heart, chest, blood flow)(heart, chest, blood flow)

•• Growth, Pathology followGrowth, Pathology follow--upup

MultimodalMultimodal

One patient, Several modalitiesOne patient, Several modalities CTCT--MR, CTMR, CT--PET, MRPET, MR--PET, PETPET, PET--US, and etc.US, and etc. Correction of Correction of MRI/MRI/fMRIfMRI acquisitionsacquisitions Constraints to reconstruction/restoration algorithmsConstraints to reconstruction/restoration algorithms Interventional Imaging : registration between preInterventional Imaging : registration between pre-- and and intraintra--operative images (e.g. MRI and Ultrasound)operative images (e.g. MRI and Ultrasound)

Modality to ModelModality to Model

Several patients, One Several patients, One modalitmodalityy ModelModel--based (guided) segmentationbased (guided) segmentation Building of digital atlasesBuilding of digital atlases Registration/matching with an anatomical Registration/matching with an anatomical atlasatlas Spatial normalization, study of anatomical Spatial normalization, study of anatomical variabilityvariability

Patient to ModalityPatient to Modality

Several patients, Several Several patients, Several modalitmodalityy Human brain mappingHuman brain mapping AnatomoAnatomo--functional normalization functional normalization ((aaid for the id for the

study of functional variabilitystudy of functional variability))

SubjectSubject

IntrasubjectIntrasubject (data from one patient)(data from one patient)

IntersubjectIntersubject (data from multiple patient)(data from multiple patient)

Atlas (one image from patient and others Atlas (one image from patient and others from Atlas)from Atlas)

ObjectObject

Several organs are studied:Several organs are studied: BrainBrain HeartHeart SpineSpine Others.Others.

Theory of RegistrationTheory of Registration

Two Images:Two Images:

With Geometrical Mapping of:With Geometrical Mapping of: :1D Gray level mapping:1D Gray level mapping :2D Coordination mapping:2D Coordination mapping

( ) ( ), , ,s s s d d dI x y I x y

( ) ( )( )( ), ,d d d s s sI x y B I x y= ϕϕϕϕ( )B ⋅

( ),⋅ ⋅ϕϕϕϕ

( ) ( ) ( ) ( ) ( )( ), , , , , ,d d s s d d d s x s s y s sx y x y I x y I x y x yϕ ϕ= ⇒ =ϕϕϕϕ

Registration approach:Registration approach: Forward:Forward:•• Map Map sourcesource coordination to coordination to destinationdestination and and

consider consider sourcesource gray level for mapped gray level for mapped coordination.coordination. Backward:Backward:

•• Find correspondences of destination pixels in Find correspondences of destination pixels in source image and estimate suitable gray level for source image and estimate suitable gray level for each one.each one.

ForwardForward--Reverse MappingReverse Mapping

Source Image Destination Image

Forward mapping drawbacks:Forward mapping drawbacks: Some pixel in destination has no value!Some pixel in destination has no value! HoleHole OverlapOverlap

Reverse mapping Problems:Reverse mapping Problems: Most pixel in destination has value Most pixel in destination has value HoleHole OverlapOverlap

Need for gray level interpolation:Need for gray level interpolation: Mapping is integer to real, and we have gray Mapping is integer to real, and we have gray level at non integer coordination (level at non integer coordination (ForwardForward) or ) or need gray level at non integer coordination need gray level at non integer coordination ((ReverseReverse).).

Methods:Methods: Nearest NeighborhoodNearest Neighborhood SincSinc weighted windowsweighted windows SplineSpline

InterpolationInterpolation( ) [ ]( ) [ ]( ) ( )

[ ]( ) [ ]( ) ( )[ ]( ) [ ]( ) ( )

[ ]( ) [ ]( ) ( )

( ) ( ) ( ) ( )

, 1 1 ,

1Weight: 1 cos

d d d s s s s s s s

s s s s s s s

d d d d i d i s i ii j

I x y x x y y I x y

x x y y I x y

I x y sinc x x sinc y y I x y

• = − + − + +

− − + + +

− + − + +

− − + +

• = − −

⋅ • ⋅ = + +

∑∑

Geometrical MappingGeometrical Mapping

Rigid Body Transform:Rigid Body Transform: Rotation TranslationRotation Translation 33--6 freedom degree in 2D6 freedom degree in 2D--3D cases.3D cases. Application: Hard organs: Radiology, Spinal Application: Hard organs: Radiology, Spinal cord, Hip, skull, and femur.cord, Hip, skull, and femur.

( ) ( )( ) ( )

cos sin,

sin cos

θ θθ θ

= = −

X RX T

Rigid Body TransformRigid Body Transform

Mathematical formulation:Mathematical formulation: 3 pixels to 33 pixels to 3--pixels (2D):pixels (2D):•• Exact solutionExact solution

•• Problems with landmark.Problems with landmark. N pixels to NN pixels to N--pixels (2D), N>>3:pixels (2D), N>>3:

Data: ,

Model:

: min ,

i ii i

i i D Di

+ − = ∑R T

y Rx + T

Goal Rx T y RR I

Rigid Body TransformRigid Body Transform

Reformulation:Reformulation:

Transform to center:Transform to center:

Need to maximize:Need to maximize:

[ ][ ] 21 2

, , ,min ,

D NN T

D DD NN

= ∈ℜ ⇒ == ∈ℜ R T

X x x xRX + T - Y RR I

Y y y y

……,= − = −X X X Y Y Yɶ ɶ

T T T TTr Tr Tr= + −Y - RX X X Y Y YX R

T TTr YX R

SimplifyT T→

R = WVYX = W V

T Y R X∑∑∑∑

TTTT= - *= - *= - *= - *

Procrustean TransformProcrustean Transform

RotationRotation

TranslationTranslation

Scaling in two axes.Scaling in two axes.

( ) ( )( ) ( )( ) ( )( ) ( )

cos sin,

sin cos

cos sin,

sin cos

θ θθ θ

= = −

X RX T

Ordinary Version (Ordinary Version (KKxx==KKyy)) PhotogrammeteryPhotogrammetery and Medicaland Medical Four parameters and two points in2D spaceFour parameters and two points in2D space

Multiple points formulation:Multiple points formulation:[ ]( )

min , ,

D NTji i

j ji ji ij

diag k ,k , k

= ⇒ = =∂ = − + = ∂ ∑ ∑

RX - Y RR K RKX - Y RR I

R = R*K

YX R X

Iterative Solution:Iterative Solution: Initialization:Initialization: Solve this problem:Solve this problem: Update Update KK(i+1)(i+1) UsingUsing

Check for convergenceCheck for convergence

[ ]1 1,1, ,1=( )K ⋯ 2

min ,T

D×D=R

RKX - Y RR I

2 2 0D NT

ji ij ji ji ij

∂ = − + = ∂ ∑ ∑YX R X

Affine TransformAffine Transform

6/12 freedom degrees (2D/3D)6/12 freedom degrees (2D/3D)

Multiple points formulation:Multiple points formulation:

,x x x

T− =⇒

TR XX XY

RX - YT Y R X= - *= - *= - *= - *

Projective/Perspective TransformProjective/Perspective Transform

NonlinearNonlinear

Observation correction (photo)Observation correction (photo)

Need 4 points.Need 4 points.

Multiple points Multiple points --> Iterative optimization> Iterative optimization

Less application in MedicalLess application in Medical

11 12 10 21 22 20

1 2 1 2

s s s sd d

s s s s

a x a y a a x a y ax y

b x b y b x b y

+ + + += =+ + + +

Bilinear TransformBilinear Transform

NonlinearNonlinear

Quadrilateral to Quadrilateral with curved Quadrilateral to Quadrilateral with curved side.side.

10 10 01 11

d s s s s

x a a x a y a x y

y b b x b y b x y

= + + += + + +

Polynomial TransformPolynomial Transform

Nonlinear (polynomial of order P):Nonlinear (polynomial of order P):

Less application!Less application!

0 0 0 0

,p p i p p i

i i i id ij s s d ij s s

i j i j

x a x y y b x y− −

= = = =

= =∑∑ ∑∑

NonNon--Parametric MethodsParametric Methods

Correlation (Matched Filter)Correlation (Matched Filter)

( )( ) ( )

( )( )( ) ( )( )

( )( ) ( )( )

1 2 1 1

2 22 2

I: Image and

T: Template

u v u v

T u v I u x v xC x x

I u x v x

T u v T I u x v x ICov x x

I u x v x I T u v T

− −=

− − − − − −

= − − − −

∑∑

∑∑∑∑

∑∑ ∑∑

Correlation (Matched Filter):Correlation (Matched Filter): Very Sensitive to NoiseVery Sensitive to Noise

A variation:A variation:

Summation will performed until reach a Summation will performed until reach a threshold.threshold.

( ) ( ) ( )

( ) ( ) ( ) ( )

1 2 1 2

1 2 1 2 1 2

, , , ,

Normalu v

E x x T u v I u x v x

E x x T u v T I u x v x I x x

= − − −

= − − − − +

∑∑∑∑

Frequency Domain:Frequency Domain: Fast MethodsFast Methods Rotation, Translation, Scaling has Rotation, Translation, Scaling has FrerquencyFrerquencydomain correspondence.domain correspondence. NonNon--robust to noise and restricted to rigid.robust to noise and restricted to rigid.

Frequency DomainFrequency Domain

TranslationTranslation

Search for peaks in space domainSearch for peaks in space domain

( ) ( )( ) ( )( ) ( )

( ) ( )1 1 2 2

2 1 2 1 1 1 2 2

*1 1 2 2 1 2

1 1 2 2*1 1 2 2 1 2

, ,j x x F

I x x I x x x x

F Fe x x x x

F Fω ωω ω ω ω

δω ω ω ω

∆ + ∆

= − ∆ − ∆

= ←→ − ∆ − ∆

Frequency DomainFrequency Domain Rotation (Polar Coordination)Rotation (Polar Coordination)

Seek for Translation in angle axisSeek for Translation in angle axis

( ) ( ) ( ) ( ) ( )( )( ) ( )

( ) ( ) ( ) ( )( )( ) ( )

( ) ( ) ( )( ) ( )

1 1 2 2

2 1 2 1 1 2 1 1 2 2

1 1 2 1 2

, cos sin , sin cos

cos sin , sin cos

cos , sin

, ,, ;

I x x I x x x x x x

F r F rH r

F r F r

ϕ ϕ ϕ ϕ

ω ϕ ω ϕ ω ϕ ω ϕ

ω θ ω θ

θ θ ϕθ ϕ

θ θ ϕ

− ∆ + ∆

= + − ∆ − + − ∆

∆ + ∆ − ∆ + ∆

Frequency DomainFrequency Domain

Rotation and Scaling (Polar Coordination)Rotation and Scaling (Polar Coordination)

( ) ( ) ( )( ) ( ) ( )( )( )( ) ( )( )

2 1 2 1 0 1 2 0 1 2 2

2 1 020

, cos sin , sin cos

1, ln ,

I x x I r x x r x x

F r F r rr

ϕ ϕ ϕ ϕ

θ θ ϕ

= + − +

= − + ∆

Radial Basis Function InterpolationRadial Basis Function Interpolation

Registration is an interpolation problemRegistration is an interpolation problem

RBFRBF--based in most used method.based in most used method.

RBF definition:RBF definition: Symmetric around a central pointsSymmetric around a central points For arbitrary For arbitrary the is RBF!the is RBF!

Goal: Find Goal: Find interpolantinterpolant function: function:

( )xφ ( )xφ µ−

( ) : D DT S= Ψ ℜ → ℜ

Radial Basis Function InterpolationRadial Basis Function Interpolation Problem Formulation:Problem Formulation: ΨΨ is RBF (same effect on all equidistance pointsis RBF (same effect on all equidistance points Mapping is error free:Mapping is error free: Landmarks points are more than space dimension Landmarks points are more than space dimension ((N>D+1N>D+1), ), N=D+1N=D+1 is affine transform. is affine transform. ΨΨ has linear components to cope with affine transform. A trade-off between Space warping and landmarks support. For erroneous interpolation, A tradeoff between error and smoothness is necessary.

( ) , 1, 2, ,i iT S i N= = ⋯ΨΨΨΨ

A smoothness criteria:A smoothness criteria:

Optimization Problem:Optimization Problem:

22 2 22

2 2( ) ( ) 2

Bending Energy of a plate!

f f fJ f f x dxdy

x x y yℜ

∂ ∂ ∂= = + + ∂ ∂ ∂ ∂ ∫

( )( )

1Suboptima

min , ( )

min ( ) , (l: ) 1, 2, ,k

i ik k

J x T x

J x T x i N

Ψ = Ψ Ψ = Ψ =

∑⋯

A Unique solution based on previous A Unique solution based on previous criteria is based on:criteria is based on:

For erroneous problem:For erroneous problem:

( )2 2logr r

min ( ) ( )D N

i ikk i

J x T xΨ = =

Ψ + − Ψ ∑ ∑ iiiiλλλλ

Region support:Region support: GlobalGlobal LocalLocal

Limited Region Support:Limited Region Support: Specific RBFSpecific RBF Multiply RBF by a LimiterMultiply RBF by a Limiter Use Compact Support RBFUse Compact Support RBF

( )2 2exp 2r σ−

( ) ( ),f r g r σ

( )21 0

σ σσ

− ≤ ≤>

Features of RBFFeatures of RBF--based interpolation:based interpolation: No need to grid data (scattered data is No need to grid data (scattered data is possible)possible) No critical condition for data.No critical condition for data. Smoothness control (Bending Energy).Smoothness control (Bending Energy).

Mathematical Formulation:Mathematical Formulation:

( ) ( )( )

11 12 1 1 1

21 22 2 2 2

Data: , and a RBF Kernel: :

Interpolant Function:

, 1, 2, ,

N Di i D

N N NN N N

R x g x x

y R x i N

g g g y

g g g yg

g g g y

∈ℜ ℜ → ℜ∈ℜ

= = = ∑⋯⋯⋯⋮ ⋮ ⋮ ⋮ ⋮ ⋮⋯ ( ) , , 1, 2, , .i jg x x i j N= − = ⋯

Matrix Formulation:Matrix Formulation:

Crucial Problem: (Inverse of G)Crucial Problem: (Inverse of G) ExistenceExistence UniquenessUniqueness

Condition on g:Condition on g: Positive Definite FunctionPositive Definite Function•• Positive Definite GPositive Definite G

GGGGΩ= YΩ = YΩ = YΩ = Y

( , ) 0N N

i ji ji j

g x xω ω= =

≥=∑∑TΩ GΩ

Examples:Examples:

( )( ) ( )

1 log : 0, 22

: 0, 0

log : 0

Dr r m D

+ ∈ − ≥

− = − > ∈

∈ℜ − >

ℝ ℕ

Need for additional terms for Need for additional terms for pdmpdm (Not (Not spdmspdm):):

Number of Unknown Parameters:Number of Unknown Parameters:

2 22 00 10 1 01 2 11 1 2 20 1 02 2

( ) ( ), ( )

ii m m mi

R x g x x p x p x

P x b b x b x b x x b x b x

ω π=

= − + ∈

= + + + + +

m DM N M

− + + =

N Conditions:N Conditions: Exact interpolation:Exact interpolation:

( ) ( )1

11 12 13 14

21 22 23 24

, 1, 2, ,

Matrix Form:

Sample P: (N=4, m=2, D=2)

1 1 1 1

k i ii k mk

y g x x p x i N

x x x x

= − + =

∑TY = GΩ+ P B

It can be shown (Not easily)It can be shown (Not easily)

( ) ( )1

k i ik m ik

g x x p x y

− + =

= ≤ −

∑∑

M×1M×M

YΩG P=

Gaussian Minimization:Gaussian Minimization:

Thin Plate Thin Plate SplineSpline Minimization:Minimization:

22( )E F e dλλ λ= ∫ℝ

min ( ) ( )D N

i ikk i

J x T xλ

Ψ = =

Ψ + − Ψ

∑ ∑T

M×1M×M

YΩG + I P=

Thin Plate Thin Plate SplineSpline in 2Din 2D

Formulation:Formulation:

( ) ( ) ( ) ( )

( )( )

22 22 2 2

00 01 02 01

10 11 12 11

log , 0,0

i i ii

g r r r g x yx y

x a a x a y w g x x y y

y a a x a y w g x x y y

∂ ∂= − ⇒ ∆ = + ≈ ∂ ∂ ′ = + + + − + −

′ = + + + − + −

∑∑

Formulation for exact matching:Formulation for exact matching:

( )( )

00 01 02 01

10 11 12 11

0 11 1

0 01 1

1 11 1

k k k i k i k ii

k kk k

k k k kk k

x a a x a y w g x x y y

y a a x a y w g x x y y

x w y w

′ = + + + − + −

∑∑

∑ ∑∑ ∑∑ ∑

Matrix FormulationMatrix Formulation

[ ] ( )

01 02 0 00 01 02

11 11 1 10 11 12

k k kN ki k i k i

k k kN

x w w w a a a G G g x x y y

y w w w a a a G

′ = = − + − ′ =

⋮⋯⋮⋯

Matrix FormMatrix Form1 2 01 02 0 00 01 02

1 2 11 12 1 10 11 12

12 1 1 1

21 2 2 2

0 0 0,

1 1 1 0 0 0

N N N N

x x x a a a

y y y a a a

G G x y

ω ω ωω ω ω

′ ′ ′ = = ′ ′ ′ = =

⋯ ⋯⋯ ⋯⋯ ⋯⋯ ⋯⋮ ⋮ ⋱ ⋯ ⋮ ⋮ ⋮ ⋮⋮ ⋮ ⋮ ⋱ ⋮ ⋮ ⋮ ⋮⋯ ⋯⋯ ⋯⋯ ⋯⋯ ⋯Y

ΩΩΩΩ

⇒ T T

PY = LΩ

Compact Support RBFCompact Support RBF

We need Positive Definite Function with We need Positive Definite Function with Compact Support Property.Compact Support Property.

How to generate:How to generate: Select one and recursive convolution!Select one and recursive convolution!

Practical problems:Practical problems:•• Convolution is hard or nonConvolution is hard or non--closed form.closed form.

( ) ( ) ( )2 2 2Dy

x y x y d xϕ ψ ψ∈ℜ

= −∫

A general class is derived:A general class is derived:

( ) ( )

( )( ) ( )( ) ( )

1 0 11

(1 ) 8 5 1

1 35 18 3

− ≤ ≤− >

− ≤ ≤− = >

− + +

CSRBF:CSRBF: No need to additional affine terms.No need to additional affine terms. Defined over [0,1]Defined over [0,1]•• Solve with scaling:Solve with scaling:

•• Unsolved problem: How to choose Unsolved problem: How to choose δδ

i ki i

x xR x ω ϕ

− = ∑

Principal Axes RegistrationPrincipal Axes Registration

A Primary phase of registrationA Primary phase of registration( ) ( )

( )( )

1 , , Desired volume, ,

0 O.W.

x y zB x y z

xB x y z

yB x y z

zB x y z

∑∑∑∑∑∑

Principal Axes RegistrationPrincipal Axes Registration

Rotation AnglesRotation Angles

( ) ( ) ( )( )

( )( ) ( )( )

( ) ( ) ( )( )

xx xy xz

yx yy yz

zx zy zz

xx g gx y z

yy g gx y z

zz g gx y z

xy g gx y z

yz g gx y z

I y y z z B x y z

I x x z z B x y z

I x x y y B x y z

I z z y y B x y z

− − = − − − − = − + − = − + − = − + − = − −

= − −

∑∑∑∑∑

( ) ( )( ), ,

, ,gx y z

x x B x y z−∑

Landmark ExtractionLandmark Extraction

Landmark Definition:Landmark Definition:

A Principal biological DescriptorA Principal biological Descriptor

Examples:Examples: Lips, Eyes, Nasal, and etc.Lips, Eyes, Nasal, and etc.

Landmark is related to:Landmark is related to: Data GeometryData Geometry Mathematical DeformationMathematical Deformation Biology of organsBiology of organs

Landmark ExtractionLandmark Extraction Landmark Landmark vsvs Fourier Descriptor:Fourier Descriptor: Local InformationLocal Information Category:Category: Juxtaposition (Proximity)Juxtaposition (Proximity) Maximum CurvatureMaximum Curvature Marginal:Marginal:•• Limiter (Left, Right, Top, Bottom)Limiter (Left, Right, Top, Bottom)

•• Center of GravityCenter of Gravity

•• Point of maximum distance (Point of maximum distance (CoGCoG))

•• Center of circles and EllipsoidCenter of circles and Ellipsoid

A medical landmark:A medical landmark: Easy detection in all modalitiesEasy detection in all modalities Robustness (noise, interference, and etc.)Robustness (noise, interference, and etc.) NonNon--complex mathematical descriptioncomplex mathematical description Medical significance.Medical significance.

LandmarksLandmarks Internal:Internal:•• Hard to detection and recognition;Hard to detection and recognition;•• Time Consuming;Time Consuming;•• Task Dependent.Task Dependent. External:External:•• Geometrical/Image Processing;Geometrical/Image Processing;•• Robustness Problem against the noise.Robustness Problem against the noise.•• An open and unsolved problem!An open and unsolved problem!

A Simple approaches:A Simple approaches:

Correspondence problem!Correspondence problem!

( )( )

( ) ( )( )

( )( ) ( ) ( )

( ) ( )( )( )( )

, ,, ,

det ,,

x y x y

x y x yx y x y

I x y I x y I x y

x x yx, y

I x y I x y I x y

x yx y

trace x y

′ ′ ′ ′∈Ω ∈Ω

′ ′ ′ ′∈Ω ∈Ω′ ′ =

′ ′ ′ ′ ′ ′ ∂ ∂ ∂ ′ ′ ′∂ ∂ ∂ = ′ ′ ′ ′ ′ ′ ∂ ∂ ∂ ′ ′ ′∂ ∂ ∂

∑ ∑

∑ ∑D

Image 2Image 2Image 1Image 1

SegmentationSegmentationSegmentationSegmentation

Contour ExtractionContour ExtractionContour ExtractionContour Extraction

Automatic Landmark Extraction and CorrespondenceAutomatic Landmark Extraction and Correspondence

Another method:Another method:

Landmark Extraction:Landmark Extraction: Dominant Point Detection;Dominant Point Detection; Contour Estimation with polygon;Contour Estimation with polygon; Clustering.Clustering.

Dominant Point Detection:Dominant Point Detection: Uses local curvature estimation.Uses local curvature estimation.

Contour Estimation with polygon:Contour Estimation with polygon: Principal Curves (Polygon):Principal Curves (Polygon):( ) ( ) ( ) sup : min

D Dτλ λ λ τ= − = −f X X f X f

Contour Estimation with polygon:Contour Estimation with polygon:

Clustering:Clustering: A self organizing Growing structures A self organizing Growing structures

Adaptation:

Growing

Clustering:Clustering: Sample are NOT simple point but edges.Sample are NOT simple point but edges.•• Need to new definition of membership, & etc.Need to new definition of membership, & etc.

Clustering:Clustering: Method:Method:•• Start from an consistence and suitable Start from an consistence and suitable

condition.condition.

•• Train to reach next match of current model.Train to reach next match of current model.

•• Insert/Delete Correspondence edges.Insert/Delete Correspondence edges.

Preprocessing:Preprocessing: SegmentationSegmentation Contour tracingContour tracing Contour correction (Contour correction (splinespline, Interpolation), Interpolation) SemiSemi--automated correspondenceautomated correspondence

Output:Output: Two sets of related contourTwo sets of related contour

Need for new definition:Need for new definition: Distance:Distance:•• Point to polygonPoint to polygon

•• Error Energy (point to side)Error Energy (point to side)

•• Error Energy (point to polygon)Error Energy (point to polygon)

Landmark ExtractionLandmark Extraction Clustering:Clustering: NG and GNGNG and GNG New data arrangement:New data arrangement: 2D to 3D conversion2D to 3D conversion Initial Condition:Initial Condition: Two triangles in two plane Two triangles in two plane •• Randomly But relatedRandomly But related•• Three lines with 120 degree distance (PCA)Three lines with 120 degree distance (PCA) Distance of incoming data and cell center.Distance of incoming data and cell center. Three statesThree states Membership:Membership: New definition:New definition:

( )1 1i

m x dsx v

=−∫

VoronoiVoronoi ImagesImages

Updates:Updates: Vertices and sidesVertices and sides Need for insert new side,Need for insert new side,•• Local Error (sides and vertices)Local Error (sides and vertices)

•• Estimation with a lineEstimation with a line2

0 2 2( )12

DP BP Epi i i

i irmsx P x P x Pi

a x aE x V x V

∈ ∈ ∈

+= + − + −∑ ∑ ∑

Updates:Updates: Need for delete avoidable edge:Need for delete avoidable edge:•• Linearity errorLinearity error

•• Principal anglePrincipal angle

Stop:Stop: AkaikiAkaiki Criteria:Criteria:

( )( ) ( )log

S DG G

E ENρ α

Error and ValidationError and Validation

Actions after registration:Actions after registration: Accuracy of registration?Accuracy of registration? How to increase?How to increase?

The main PROBLEM:The main PROBLEM: No No GOLDGOLD standardstandard

Statistical Measure:Statistical Measure: Inconsistency in different experiments.Inconsistency in different experiments. Image Dependency;Image Dependency; All experiments has special protocol.All experiments has special protocol. Unclear description of Accuracy and Precision.Unclear description of Accuracy and Precision.

Errors:Errors: FiducialFiducial Localization Error (FLE):Localization Error (FLE):•• Physical Landmark Uncertainty.Physical Landmark Uncertainty. FiducialFiducial Registration Error (FRE):Registration Error (FRE):•• Distance between correspondence Distance between correspondence LandmarkLandmark

after registration phase.after registration phase. Target Registration Error (TRE): Distance Target Registration Error (TRE): Distance between correspondence between correspondence pointspoints after after registration phase.registration phase.

Errors:Errors:

Errors:Errors: FLE:FLE:•• Operator and Algorithm.Operator and Algorithm. FRE:FRE:•• Rigid transform (large sets of points)Rigid transform (large sets of points) TRE:TRE:•• Hard to estimation: Training and test sets.Hard to estimation: Training and test sets.

Errors (Rigid Transform):Errors (Rigid Transform): If FLE then FRE If FLE then FRE •• Expected value of FRE:Expected value of FRE: TRE (1998):TRE (1998):

( )20,N σ∝ 2 23 6N Nσ κ −∝

( ) 23 6N Nσ−

, : Matrices of eignvalues of landmarks

TRE FLEi j i i j

N Dσ σ

≅ + Λ + Λ

∑∑

Sensitivity Analysis of landmarks:Sensitivity Analysis of landmarks: Analytical relation.Analytical relation. Some nonSome non--closed form of mathematical closed form of mathematical equations.equations.( )( ) ( ) ( )

( )( ) ( ) ( ),,,,( , ) ( , ), , ( , ) ( , )( , ) ( , ), , ( , ) ( , )

s si id di ix xs si if x y s sx y i i y ys si ix xd di if x y d dx y y yi i d di if x y f x yx yf x y f x yx y f x y f x yx yf x y f x yx yf x y f x y f x y f x yx y x y

∂ ∂ ∂ ∂ ∂ ∂ = = ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂ = = ∂ ∂∂ ∂ ∂ ∂S

Local RegistrationLocal Registration

Local Model:Local Model: Nearest NeighborNearest Neighbor•• Set pixel value to nearest landmark Set pixel value to nearest landmark Weighted AverageWeighted Average•• Set pixel value to weighted average (Distance)Set pixel value to weighted average (Distance) Local Weighted FittingLocal Weighted Fitting•• Fit a curve or surface based on NFit a curve or surface based on N--nearest nearest

landmark. landmark.

Example, Shepard Method:Example, Shepard Method:

( ) ( ) ( )

( ) ( )

( ) ( )( )

Condition: ; 1 , 0 ; 1, 1, 2, ,

; 1, ; 0, , 1, 2, , ,

; 1Shepard: ; , ;

i ii ii

i i j ii i

iii ii iN

f x x x f x

x x x x i N

x x x x i j N j i

x xx x x x

x xx xµ

σα σ

= = ≤ ≤ = = = = ≠= =

Another approach:Another approach:

( ) ( ) ( )

Goal to Minimize: , 1, 2, ,

( ) , , 1, 2, ,

ji ij i ij j i

ii i i ii

i ii i i

E f x y i N

f x x f x f x y i N

f x x P x x P

= − =

= + − =

∑∑

……

Problem of Region Support in RBF:Problem of Region Support in RBF: Fixed Fixed Nearest Neighbor (Large for isolated)Nearest Neighbor (Large for isolated) Global (Optimization)Global (Optimization)

Distance Effect:Distance Effect:

( ) ( ) ( )( ) ( ) ( )( ) ( )

( ) ( )( )1

i ip i

d x a x a x a

d x a d Mx Ma M diag m

d x a d Mx Ma

d x a x a

= − −

= −∑

Our idea:Our idea: VoronoiVoronoi and Tessellation Diagramand Tessellation Diagram

Our idea:Our idea: Distance point Distance point –– structurestructure Weighing function for merging multiple maps.Weighing function for merging multiple maps. Region SupportRegion Support Smoothness Smoothness CriteriCriteri

Structures:Structures: Single PointsSingle Points Single (Smooth) are of contourSingle (Smooth) are of contour Correspondence ContourCorrespondence Contour

Correspondence StructureCorrespondence Structure

Related Landmark ExtractionRelated Landmark Extraction

Split Structure to subSplit Structure to sub--structurestructure

Mapping Determination for each Mapping Determination for each subsub--Structure.Structure.

VoronoiVoronoi ImagesImages

Local Weighted MappingLocal Weighted Mapping

Alternative definition of distance:Alternative definition of distance: MinMin New definitionNew definition2 2

1 2( ) ( )1 ( ) 1( )

( ) ( )c

C p pt tc c

dx t dx td x tm x dt dt

dt dt dtx x t x x t

= = + − −∫ ∫

Region of support definition:Region of support definition: Use of GRBFUse of GRBF Use of EBFUse of EBF Use of QBFUse of QBF

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

, ; , ( , , )

i mi i

E Dj j

xf x p x

x cN x c x c x c diag

N x c x c x c

µω ϕ

σ σ σσ

− = + −

Σ = = − Σ − Σ =

Σ = − Σ −

∑ …

Number of Center:Number of Center: GrowingGrowing--Pruning MethodsPruning Methods Cross ValidationCross Validation

Local Warping:Local Warping:

( ) ( )

( ) 1Point to Curve Membership: ( ) , ( )

( ) ( ) ( )

ln(3) 2, e , tanh ,

Ck CN p

C k kk

m xx m x ds

x cm x

f x x m x x f x x

m m mm Arctg

λσ σ π σ

= =−

= + − = + ∫∑

Medical Image Registrationee.sharif.edu/~miap/Files/Image RegistrationLecture.pdf · • Anatomy...

Documents

Moving Image Archiving and Preservation Program …...Department of Cinema Studies New York University May 2016 !!!!! 2 Acknowledgements To all of my MIAP classmates who have been

1448 Max MIAP Leaflet NGL(Web) CTC

NYU Moving Image Archiving and Preservation (MIAP) Program … · 2019-09-04 · Demonstrate familiarity with basic goals, techniques and limitations of conservation treatment Apply

Medical Image Analysis Medical Imaging Modalities Figures come from the textbook: Medical Image Analysis, by Atam P. Dhawan, IEEE Press, 2003

Combining text/image in WikipediaMM task 2009clef.isti.cnr.it/2009/CLEF2009-workshop-slides/... · Model overview Combining text and image modalities query documents Score matching

MIAP: Manifold-Interface Amplitude Panning in Max/MSP …...MIAP (Manifold-Interface Amplitude Panning) Historical Signiﬁcance: in custom built Max or Processing patches. Whilst

2019 MIAP Conference State Immunization Update Three

MIAP Newsletter · Preserve Paul Sharits Films As the result of a recent grant from the Academy of Motion Picture Arts and Sciences, MIAP students will be given a unique opportunity

Light Therapy ESAT 3640 Therapeutic Modalities ESAT 3640 Therapeutic Modalities

Mechanical Modalities Therapeutic Modalities in Athletic Rehabilitation

Besser-NARA, 3/27/03 Moving Image Preservation: Analog or Digital? Howard Besser, Director MIAP Program NYU’s Tisch School of the Arts

•Image Degradation Model (Linear/Additive)sbisc.sharif.edu/~miap/Files/DIP4MIAP(ForView... · 2011. 2. 20. · ee.sharif.edu/~miap E. Fatemizadeh, Sharif University of Technology,

Image segmentation by EM-based adaptive pulse coupled ...ee.sharif.edu/~miap/paper/89300465.pdf · Image segmentation by EM-based adaptive pulse coupled neural networks in brain magnetic

Scholastic Backwardness In Adolescent Dr.Geeta Khanna M.D.( Pediatrics); MIAP Consultant Peadiatrician

Thermal Modalities ESAT 3640 Therapeutic Modalities

MIAP Introduction

Therapeutic Modalities. Basic Modalities Ice Heat Whirlpool Paraffin Bath

3/3/03Besser--NYU Trustees Moving Image Archiving Preservation Howard Besser, Director MIAP Program NYUs Tisch School of the Arts

Submodular Reranking with Multiple Feature Modalities for ...fyang/papers/accv14.pdf · Submodular Reranking with Multiple Feature Modalities for Image Retrieval Fan Yang y, Zhuolin

Survey: interpolation methods in medical image …ee.sharif.edu/~miap/Files/Survey- Interpolation Methods in Medical... · IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 18, NO. 11, NOVEMBER