Ron Goldberg Yulia Turovski Supervisor: Arie Nakhmani Winter 2011 Date: 07.05.2012 Technion –...

Modeling of visual form and motion of nano-particles drifting in a polymeric fluid

Ron Goldberg

Yulia Turovski

Supervisor: Arie Nakhmani

Winter 2011

Date: 07.05.2012

020

4060

80

020

4060

80-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

x

U(r)

y

inte

nsity

Technion – Israel Institute of TechnologyFaculty of Electrical EngineeringControl & Robotics Laboratory

• Motivation & Goals• Previous work on the subject• System description• Modules• Example• Results• Future work

Outline

• Active and controllable drug transport– Few and isolated damaged cells– Healthy tissue unaffected

• Super paramagnetic nanoplatforms– Control of platforms via magnetic field

• Improve control of nanoplatforms motion– Automatic platforms characteristics and motion

analysis

Motivation

• Automatic analysis of platforms motion and characteristics:– Static noisy background subtraction– Dynamic noise filtering– Platforms detection– Platforms modeling and reconstruction– Motion analysis

• MATLAB Environment• Non real time• Short processing time (~minutes)

Goals

Input movies

• Microscope generated movies• Diffraction patterns• Polymeric fluid• 15 seconds

Example

• Previous solution (Nakhmani et al., 2010)– Static noisy background subtraction– Dynamic noise filtering– Platforms detection– Platforms modeling and reconstruction– Motion analysis

• Background subtraction: classic & advanced• Unique problem

– Collection of issues– Unrelated & uncommon solutions

Previous Works

• Noise cleaning–Static noise (background subtraction)–Dynamic noise (optional)

• Particles modeling

System Description

Block DiagramBackgroun

d subtraction

Marking suspicious sub frames

Gaussian fitting

process

Per frame

Circles detection

Particles reconstruction

Sorting Algorithm

Original movie Cleaned movie

Fitting errors

Sub frames locations

Circles parameters

Sorting results & parameters

Reconstructed movie

• Based on Stauffer & Grimson GMM algorithm• GMM – Gaussian Mixture Model

– Linear superposition– Different expectations, variances and weights

Module:Background Subtraction

• Stauffer & Grimson– Threshold operation– Pixel wise analysis– Mixture of Gaussians PDF– Multiple background objects– Continuously updating model’s parameters

Module:Background Subtraction

• Improved Implementation– External source– Dynamic number of Gaussians– Results & run time improved

Module:Background Subtraction

• Improvement– Merging regular and reversed movies– Learning process– Later frames better cleaned– Linear weight:

Module:Background Subtraction

1 1

1

1n

forward backwardn n n N n N n

n

N

I I I

Background Subtraction:Example

Original frame Cleaned frameOriginal frame

• Particles’ diffraction patterns– Theoretically: Bessel functions– Practically: Bessel functions & Gaussians

• Initial detection– Sub frames – Gaussian fitting– Revaluation error

Module:Particles Detection I

• Gaussian fitting– Least squares

• Linearization of Gaussian model• Pseudo Inverse

– Mean Square Error• Normalized to revaluated amplitude

Module:Particles Detection I

02

46

810

0

5

100

0.2

0.4

0.6

0.8

1

x

noised gaussian - distribution

y

inte

nsity

02

46

810

0

5

100

0.2

0.4

0.6

0.8

1

x

revaluated gaussian (with noise) - distribution

y

inte

nsity

• Improved disadvantages– Sensitivity to zeros & low intensities– Saturation– Pseudo inverse

• Perfect revaluation for ideal Gaussians• Impressive revaluation & detection capabilities• Excellent reliability

– Thousands sub frames per frame– Numbered error messages

Module:Particles Detection I

Particles Detection I:Example (I)

0

5

10

15

0

5

10

150

0.2

0.4

0.6

0.8

1

x

noised gaussian - distribution

y

inte

nsity

noised gaussian - image

0

5

10

15

0

5

10

150

0.2

0.4

0.6

0.8

1

x

revaluated gaussian (with noise) - distribution

y

inte

nsity

revaluated gaussian (with noise) - image

Particles Detection I:Example (II)

05

1015

20

0

5

10

15

200

0.2

0.4

0.6

0.8

1

x

original particle - distribution

y

inte

nsity

05

1015

20

0

5

10

15

200

0.2

0.4

0.6

0.8

1

x

revaluated particle - distribution

y

inte

nsity

original particle - image

revaluated particle - image

• Particles detection in frames– Uniform sub frames ( )– Overlap (50% in each axis)– Filtering out hopeless sub frames– Negative revaluation error image

Module:Particles Detection I

10 10

Particles Detection I:Example (III)

frame1 frame2

fitting error - frame 1 fitting error - frame 2

Frame 1 Frame 2

Fitting error - frame 1 Fitting error - frame 2

• Sub frames matching– For circle detection– Sub frames depend on suspicious areas

• Revaluation error based algorithm– Clear distinction– Suspicious areas– Size of sub frames

Module:Particles Detection II

• Chosen method– Lower threshold– Square sub frame– Exponential formula for area:

Module:Particles Detection II

min min

min

1max min min max1

1

1z T T

TS S S e S S e

e

– : Revaluation error– : Lower threshold– : Sub frame’s maximum area– : Sub frame’s minimum area– : Curvature of exponential function

• , descending function• Spans sub frames sizes

Module:Particles Detection II

min min

min

1max min min max1

1

1z T T

TS S S e S S e

e

0

z

minT

minSmaxS

Particles Detection II:Example

frame1 frame2Frame 1 Frame 2

Calculated frames of frame 1 Calculated frames of frame 2calculated frames of frame 1 calculated frames of frame 2

• Good compatibility with particles– Size– Location

• Multiple sub frames dealt by sorting algorithm

Module:Particles Detection II

• Centers and radii• Basis for particles modeling• Popular problem

– Many circles detection algorithms exist– Chosen solution from external source

• Chosen algorithm– Gray scale input images– Based on circular Hough transform

Module:Circles Detection

• Circular Hough transform– Method for detecting shapes in images– Basic transform detects straight lines

• Generalization to circles & ellipses• Further Generalization to any parametric shape

– Shapes detected in parameter space

• Chosen algorithm enables control of:– Allowed asymmetry– Sensitivity to concentric circles

Module:Circles Detection

• Suitable solution– Revaluation error based detection– Sub frames matching for suspicious areas– On each sub frame

• Chosen algorithm is performed• Uniform parameters set

– Circles data is accumulated

Module:Circles Detection

Circles Detection:Example

Frame 1 Frame 2

• Overlap causes need to cross data from different structures

Module:Sorting Algorithm

Original frameReconstructed frame

• Sorts to Gaussians and Besselians• Considers all circles detected

I. Handles structures separately• Each structure can contain several particles• Initial & temporary sorting

II. Crosses data from different structures• Filtering out resembling circles• Final sorting

Module:Sorting Algorithm

• Determines equivalent centers for Besselians– Based on two largest radii– Linear weight– Bigger weight for larger circle

Module:Sorting Algorithm

Sorting Algorithm:Example (I)

Circles frame Sorted circles frame

Circles frame Sorted circles frame

Sorting Algorithm:Example (II)

Sorting Algorithm:Example (II)

Original frameReconstructed frame

Sorting Algorithm:Example (III)

Frame 1 Frame 2

• Based on sorted circles• Gaussian particles

– Least squares Gaussian fitting– Same algorithm used for particles detection– Selected sub frames– Sub frames’ sizes determined by circles data

Module:Particles Reconstruction

• Besselian particles– Sub frames’ sizes determined by circles data– Besselian formula:

– Needed parameters: &

Module:Particles Reconstruction

2

1 0

0

/

/

J r rI r A

r r

0r A

• :– Zeros of Besselian known– Detected circles are zero contours– computed using smallest circle’s radius

• :– Common Besselians:

• Truncated main lobe • Just one ring

Module:Particles Reconstruction

0r

0r

A

Normalized Particle - first ring

• :– Reconstruction based on first ring:

• Analytic function’s mean known• First ring’s mean computed• Comparison of both gives

Module:Particles Reconstruction

A

A

Normalized ParticleAnalytic Function First ring

Particles Reconstruction:Example (I)Original particleOriginal particle with its detected circles

Original particle - first ringReconstructed particle

Original particle Detected circles

Original particle’s first ring Reconstructed particle

Original frameReconstructed frame

Particles Reconstruction:Example (II)

Example

• System’s products– Reconstructed movie– Circles’ data

• Limited quantitative analysis• Qualitative analysis

– Satisfactory results– Unsatisfactory results

Results

frame 1 frame 2

Quantitative analysis

Frame 2Frame 1

frame1 reconstructed with original particles areas marked frame2 reconstructed with original particles areas marked

Quantitative analysisFrame 2Frame 1

frame1 with manually marked particles frame2 with manually marked particles

Frame 1 reconstructed Frame 2 reconstructed

Quantitative analysis

• Frame 1:– 16 particles– 9 correct detections – 7 misses– 6 false detections– Mean distance: 1.32– Distance standard

deviation: 0.96

• Frame 2:– 13 particles– 12 correct detections – 1 miss– 1 false detection– Mean distance: 2.75– Distance standard

deviation: 2.16

• Centers of mass• Manual radii calculation

• Satisfactory results

Qualitative analysisexample 1 - original

example 1 - reconstructedexample 1 - background subtracted

Original Frame

Cleaned Frame Reconstructed Frame

• Impressive reconstruction• Conspicuous & small particles• Inconspicuous & weak particles• Asymmetric & imperfect particles• Particles in noisy environment• Reconstruction algorithm corrects detection

algorithm’s faults.

Qualitative analysis

• Unsatisfactory results

Qualitative analysisOriginal Frame

Cleaned Frame Reconstructed Frame

example 4 - original

example 4 - background subtracted example 4 - reconstructed

• False detections– Prominent in final movies– Reconstruction of large & bright particles

• Multiple detections per particle– Result of sub frames matching

• Extremely bright particles• False detections rejection capabilities

– Deficient for Besselians

Qualitative analysis

• Big blurry particles• Difficulty detecting Besselian particles• Noise

– Damages detection & reconstruction– Increases false detections

• Independent frames– Various results in adjacent frames

Qualitative analysis

• Particles reconstruction: Impressive & unique results

• Complementary modules improve results• Limited theoretical model• Significant disadvantage: false detections

– Flickering– Exceptionally large particles

• Independent frame analysis– Various results in adjacent frames– Incapability of handling flickering

Conclusions

• Motion analysis– Reduced false detections and miss rates– Consistent reconstructed movie

• Additional system products– Types of particles– Particles’ characteristics

• Extension of the theoretical model• Re-examination of dynamic noise reduction• Further exploration of edge detection

Future Work

References• [1] Q.Wu, F.A.Merchant, K.R.Castelman, ”Microscope Image Processing,” Academic Press,

2008.• [2] A. Nakhmani, L. Etgar, A. Tannenbaum, E. Lifshitz, R. Tannenbaum, "Visual Motion

Analysis of Nanoplatforms Flow under an External Magnetic Field",NSTI – Nanotech 2010, Vol 2, chapter 8, Pp.504-507.

• [3] A. Nakhmani, L. Etgar, A. Tannenbaum, E. Lifshitz, R. Tannenbaum, "Trajectory control of nanoplatforms under viscous flow and an external magnetic field", 2010.

• [4] M. Piccardi, "Background subtraction techniques: a review".• [5] Z. Zivkovic. Improved adaptive Gaussian mixture model for background subtraction.

International Conference Pattern Recognition, Vol. 2, 2004, Pp.28-31.• [6] Z. Zivkovic, "Efficient adaptive density estimation per image pixel for the task of

background subtraction", Pattern Recognition Letters 27, 7/2006, Pp.773–780.• [7] Kenneth R. Castelman, "Digital Image Processing",Prentice Hall, 1979, Chap. 19, Sec. 5.• [8] E. Trucco, A. Verri, "Introductory Techniques For 3-D Computer Vision", Prentice Hall,

1998, Pp. 86-87.• [9] J.W. Goodman, "Introduction to Fourier Optics", Third Edition, Roberts and Company,

2005.• [10] C.A. Balanis, "Antenna Theory: Analysis and Design". 3rd Ed. Wiley, 2005.

The End…