Automatic in vivo Microscopy Video Mining for Leukocytes

* Chengcui Zhang, Wei-Bang Chen, Lin Yang, Xin Chen, John K. Johnstone

Background Information What is in vivo microscopy?

Images of the cellular and molecular processes in a living organism

Why video-mine leukocytes? To Predict Inflammatory response

• Rolling velocity and magnitude of adhesion of leukocytes are the main predictors

Currently analyzed manually• Time consuming / Expensive• Subjective

Objectives

Given a sequence of in vivo images, Track the moving leukocytes Calculate their average velocity Find the magnitude of adherent leukocytes

Challenges

Server Noise Background movement

Due to movement of the living organism Deformation of leukocytes Change of contrast in different frames

Previous Work [Eden et al.] use local features (e.g. color)

for a tracking system Assume that leukocytes roll along the vessel

centerline

[Acton et al.] Background removal + morphological filter Assumes the shape/size leukocytes does not

change

Suggested Approach

Three main steps:

1. Frame Alignment• To correct the camera/subject movement

2. Detect Moving Leukocytes

3. Detect Adherent Leukocytes• After moving leukocytes are removed

Step 1- Frame Alignment 1.1- Detect Camera/Subject Movement

Define a (dis)similarity measure between consecutive frames

• This allows for some tolerance within radius r

If S(ft-1, ft) is larger than a threshold, then ft requires frame alignment

1.2- Frame Matching Generate a number of high dimensional, local

scale-invariant features [SIFT] for the frame and its predecessor

Use nearest-neighbor to find a match for each feature point

Calculate the transformation matrix H, such that

Step 1- Frame Alignment

• For every matched point x and x’

Step 1- Frame Alignment

Use Random Sample Consensus (RANSAC) to correct the mismatches

Step 2 - Detecting Moving Leukocytes Approach 1 - Probabilistic Learning

For pixel j in the image, let x1j, x2j, ..., xNj be the intensity of the pixel over N frames

Assume that P(xtj) has a normal distribution over time with mean xtj

If P(xtj) is smaller than a threshold, then it is a foreground pixel

Problem: Difficult to find a threshold

Step 2 - Detecting Moving Leukocytes Approach 1 - Probabilistic Learning

Problem: Difficult to find the threshold value

Solution: Use One-Class SVM to classify background and foreground pixels

Step 2 - Detecting Moving Leukocytes Approach 2 - Neural Network

Train a neural net to learn the predictable pattern of the background pixels

Input: [x(t-m), x(t-m+1),... , x(t-1)] • A sliding window of the intensity sequence

Output: x(t) • Prediction for the intensity of the pixel at the next frame

If the neural-net prediction and the real pixel intensity are very different, the pixel in the current frame is in foreground

Step 2 - Detecting Moving Leukocytes Approach 2 - Neural Network

Step 2 - Detecting Moving Leukocytes

Calculating the leukocytes velocity Find the centroid of each group of connected

foreground pixels For each centroid, find the closest centroid in

the previous frame If their distance is smaller than a threshold,

they are a match Compute the mean velocity

Step 3- Detecting Adherent Leukocytes

First, remove the moving leukocytes Three main types of regions left

Tissues Vessels Adherent Leukocytes

These three have different intensity values

Step 3- Detecting Adherent Leukocytes

Step 3- Detecting Adherent Leukocytes

Finding the threshold values Fit an 8th degree polynomial to the histogram

curve The real part of the second largest root is the

ideal threshold• Justification?

Problem with false positives and false negatives

Experimental Results Test video of 148 frames Detecting moving leukocytes:

1% false positive for probabilistic learning(?) 49% false positive for neural-net approach 50% recall

Detecting Adherent leukocytes 2% false positive 95% recall

Final Remarks Paper is mainly related to Vision

The algorithms require many “magic parameters” that need hand tuning Would the current parameters work as well for a new

video sequence from a new equipment?

Do we want to pursue more video-mining papers?