Centre for Mathematical Sciences, Mathematics, LTH, November 2014

Medical Image Analysis, Assignment 1

1 Introduction

The purpose of this assignment is to perform landmark-based image registration.The assignment has two parts. In the first part we consider registration using Eu-clidean (rigid) transformation and in the second part, similarity transformations.The landmarks will be generated automatically using SIFT and RANSAC, and aretherefore to be considered as mathematical landmarks. Landmark-based methodsare especially interesting when dealing with image pairs having different modalities.This is illustrated clearly in the second part of the assignment.

2 The Rules

The assignment is handed out at the lecture Monday 10 November. The de-adline is on Monday 24 November (23:59 CET). Each student should hand inhis or her own individual solution and should, upon request, be able to present thedetails in all the steps of the used algorithm. You are, however, allowed to discussthe assignmet-problem and share your experiences with people on the course andothers. You may even ask your teachers for advice, if needed.

The report. Present your work in a report of approximately four A4-pages writtenin English, Swedish or Danish. The report should contain a short introduction, adescription of the methods used, a section on the results obtained and a discussion.The MATLAB-code (m-files) should be appended to the report. You should alsoemail the actual m-files, see below. Make sure that your matlab scripts are wellcommented and can be executed directly (that is, without loading any data, settingparameters etc. Such things should be done in the script).

Submission. Email the report with the appended code to fman30lth@gmail.com.Attach the m-files and write your name and the assignment number in thesubject line. If you prefer to submit a hard copy of the (hand-written) report,please put them in the locked shelf on the 3rd floor in the slot marked “bildanalys”.You should still email the m-files.

3 The data

Two data sets (collection of image pairs) are provided for the assignment. They canbe downloaded from the course webpage.

Collection 1. This data consists of 29 pairs of microscopy (20×) images of histopa-tological material obtained from radical prostatectomy (surgically removed prosta-tes). There are two folders, each containing one image from each pair. The images

1

were created in the following manner. Cylindrical cores of 1mm in diameter werepunched out form the specimens, arranged in tissue microarrays (TMAs), and em-bedded in paraffine. Each image pair consists of two consecutive sections of thickness4µm from these cores. One slice was stained with hematoxolin-eosin (H&E), whichlooks red or pink, and the other slice with p63AMACR, which appears blue orpurple. The tissue samples in the image pair will therefore appear to have similar,but not identical, shapes. Since the images were acquired with the same magnifica-tion our assumption is that they may be aligned using a Euclidean transformation(translation+rotation).

Figur 1: An image pair from Collection, prostate tissue cores. The two tissue samplesare adjacent slices from the same core. The sample to the left is stained with H&E,the one to the right with p63AMACR.

For the report: Perform rigid registration of the image pairs. Record the magnitudeof the translation vector and the rotation angels. Perform visual assessment of theperformance of the method. Describe the required theory adequately.

Collection 2. Here 12 pairs of images are provided. Each pair consists of an image(10x) stained with H& E and another (40x) stained with a flourescent marker andacquired using time-resolved flourescence microscopy (TRF), which looks a faintnebulous signal on a dark background. Both images comes from the same slice,where the flourescent staining was applied and the TRF image acquired first. Thenthe flourescent staining was washed off, the H&E staning was applied subsequentlyand the corresponding image acquired. Due to the difference in magnification, theTRF image shows only a detail of the H&E image. The alignment should thereforeuse a similarity transformation (translation+rotation+scale).

More information about the images can be found in Lippolis et al. [1].

For the report: Perform image registration using similarity transformations. Recordthe magnitude of the translations, scale factors and the rotation angels. Assess per-formance visually. Describe the required theory adequately.

4 VLFeat

Download the open source computer vision toolbox VLFeat (release 0.9.19) forMATLAB from http://www.vlfeat.org/. We will use the MATLAB-command

2

Figur 2: An image pair form Collection 2. The tissue is from a prostate biopsy. Theimages are obtained from the same slice. The left one with a H&E-staining and 10×magnification, the right one with an flourescent staininf and 40× magnification.

[F,D] = VL_SIFT(I), which computes SIFT-keypoints (F) and SIFT-descriptors(D). A description of the features of the toolbox can be found in [3].

5 Landmark Generation

In both parts of the assignment the given images should be converted to gray scaleimages. In the case of the TRF images, it may be an advantage to make an histogramequalization as well. It may even be advisable to downscale the images to half thesize (a quarter of the number of pixels) to reduce runtimes.

5.1 Keypoints and descriptors – SIFT

The first step in the landmark-generation is to find salient points in the images(keypoints) which are tagged with “fingerprints” (descriptors) which will allow us tofind corresponding keypoints in pairs of images showing the same scene. To do this weuse D. Lowe’s SIFT-algorithm [2], in particular the command [F,D] = VL_SIFT(I),referred to above.

5.2 Generating landmarks from the keypoints – Lowe’s cri-terion and RANSAC

We first generate a set of preliminary matchings using Lowe’s criterion. We givea brief explanation here. Let (xi,f i)

Ni=1 denote the N pairs of keypoints xi ∈ R2

and its corrsponding descriptors f i ∈ R128 found by SIFT in the first image. Letsimilarly (x′j,f

′j)

M

j=1denote keypoint-descriptor pairs detected in the second image.

The we first compute the N ×M -matrix D = [dij] where the entries dij = ‖xi−x′j‖are the euclidean distances between all pairs of descriptors, on from each image.Next, for each index i we find the index j∗ such that

dij∗ = min1≤j≤M

dij.

3

The keypoint x′j∗ in image two is preliminarily matched to the keypoint xi in imageone if Lowe’s criterion is fulfilled:

minj dijminj:j 6=j∗ dij

< 0.77.

This criterion states that (xi,x′j∗) is a preliminary match only if the corresponding

descriptors f i and f j∗ have a mutual distance dij∗ which is significantly shorter thanthat from f i to any other descriptor f ′j from the second image. The number 0.77is a heuristically chosen. The parameter may be chosen to be any number betweenzero and one.

Among these matches we find real pairs of matching keypoints using RANSAC [4].When RANSAC is used, it is necessary to compute the registration models (rigid orsimilarity referred to below) on minimal or near-minimal sets of preliminary keypointcorrespondances. The result of this step is to obtain a maximal set of inliers, i.e. amaximal set of keypoint-pairs which are consistent with the registration model used.This set of inliers will be our landmarks. The final registration is estimated fromthese landmarks.

5.3 Rigid alignment

The landmarks should be aligned using a transformation T : R2 → R2 of thefollowing type,

T (x) = t+Qx (1)

where t ∈ R2 is a translation vector and Q is a 2 × 2 orthogonal matrix, i.e. sa-tisfying QTQ = I. An algorithm for computing t and Q, based on singular valuedecomposition of a correllation matrix for the data, is presented in the lectures. Thisalgorithm is formulated for the three-dimensional case, but is easily modified to suitour purpose.

5.4 Alignment using similarity transformation.

The landmarks should be aligned using a transformation T : R2 → R2 of thefollowing type,

T (x) = t+ sQx (2)

where t ∈ R2 is a translation vector and Q is a 2×2 orthogonal matrix, and s > 0 isa scale factor. Again, t, Q and s may be computed using an algorithm presented inthe lectures. Alternatively one may use the method employded in Lippolis et al. [1].

5.5 Evaluation of results.

Discuss how the registration results should be presented and the performance of theproposed algorithm can be adequately evaluated. In your report, only the simpleskind of evaluation is required. This could be simply a subjective visual assessment ofthe obtained results. However, this will not be enough in a scientific paper, and wetherefore need to be aware of how we could proceed in order to get more objectiveassessments.

4

Extra assignment for higher grade (4–5): Make an objective assessment of the per-formance of the two algorithms used above. Use manually recorded landmarks (5 to10 correspondances for each image pair) and follow the evaluation method outlinedin the paper Giuseppe et al. [1].

Referenser

[1] G. Lippolis, A. Edsjö, L. Helczynski and N. Chr Overgaard: Automatic regi-stration of multi-modal microscopy images for integrative analysis of prostatetissue sections, BMC Cancer, 13:408, 2013 (Open Access).

[2] D. G. Lowe: Distinctive Image Features form Scale-Invariant Keypoints, Inter-national J. Comp. Vis. 60:2, 91–110, 2004.

[3] A. Vidaldi and B. Fulkerson: VLFeat - An open portable library of computervision algorithms, MM’10, October 25–29, 2010, Firenze, Italy.

[4] M.A. Fischler and R.C. Bolles: Random sample consensus - a paradigm formodel-fitting with applications to image-analysis and automated cartography,Comm. of the AMC 24(6), 381–395, 1981.

5