Synchronized slice viewing of similar image series

Sharib Aliac, Antonio Foncubiertaa, Adrien Depeursingea, Fabrice Meriaudeauc,Osman Ratibd, Henning Mullerab

aUniversity of Applied Sciences Western Switzerland (HES–SO), Sierre, Switzerland;bMedical Informatics, University Hospitals & University of Geneva, Switzerland;

cUniversity of Burgundy, Le Creusot, France;dNuclear Medicine, University Hospitals of Geneva, Switzerland

ABSTRACT

Comparing several the series of images is not always easy as the corresponding slices often need to be selectedmanually and in times where series contain an ever–increasing number of slices this can mean manual work whenmoving several series to the corresponding slice. Particularly two situations were identified in this context: (1)patients with a large number of image series over time (such as cancer patients) frequently need to comparethe series, for example to monitor tumor growth over time. Manually adapting two series is possible but withfour or more series this can mean loosing time. Having automatically the closest slice by comparing visualsimilarity also in older series with differing slice thickness and inter slice distance can save time and synchronizethe viewing instantly. (2) analyzing visually similar image series of several patients can profit from being viewedin a synchronized way to compare the cases, so when sliding through the slices in one volume the correspondingslices in the other volumes are shown. This application could be employed after content–based 3D image retrievalhas found similar series. Synchronized viewing can help finding the most relevant cases quickly.

To allow for synchronized viewing of several image volumes, the target image series are registered and theexisting slices are located in the volumes. Subsequently, the user can move inside the slides of one volume andselect target volumes to be synchronized with. These volumes are then synchronized in that the slice closest to theoriginal volume is shown even when the slice thickness or inter slice distances differ, and this fully automaticallyby comparing the visual image content. The system uses ITK (Insight toolkit) and VTK (visualization toolkit)for the image processing and analysis parts. A simple interface using Qt has been developed and an integrationinto the image viewer OsiriX has been started to make tbe tool available to a larger public.

A first evaluation with image volumes of the same slice thickness and inter-slice distance for lung CTs andbrain MRIs shows a very good performance. Goal is now to increase the complexity of the task progressively tovarying scanners and protocols and then to partial volumes and potentially to localize single slices in volumes.To allow for synchronized viewing, the target series are registered to the reference series and the slices are locatedin the volumes. Currently, the system uses 3D rigid registration and non–rigid methods are under investigation.Subsequently, the user can move inside the slices of one volume and all others are synchronized automaticallyalso when slice thickness or inter slice distance vary. The system uses ITK and VTK for the image processingand visualization parts. A simple interface using Qt was developed and integration into OsiriX was started tomake the tool available to a larger public and with a better user interface. A first evaluation with lung CTs andbrain MRIs shows a good performance for series of basically the same anatomic area. The goal is to increasethe complexity of the task progressively to varying protocols and partial volumes and localize single slices involumes.

Keywords: medical image retrieval, information retrieval, synchronized viewing, 3D data analysis

Further author information: (Send correspondence to Henning Muller, Email: henning.mueller@hevs.ch)

1. DESCRIPTION OF PURPOSE

Medical image retrieval has been an extremely active domain in the medical imaging field over the past years.?,?

Image retrieval can supply similar images or cases with similar images to clinicians in case of doubt that canthen be used to aid diagnosis as the past cases often have a detailed description including a confirmed diagnosis.When data sets include 3D data such as tomographic series,? it is often not trivial to quickly choose the mostrelevant cases and to compare the images, something that is usually fast in 2D. To synchronize the viewing ofseveral 3D series mainly in axial views automatically, corresponding structures in the series have to be matchedand then all slices in the series need to be located in the volumes. Information on slice thickness and inter slicedistance can also be found in the DICOM header. For the synchronization, simple registration algorithms canbe used assuming that the reference and target volumes have several structures in common and are indeed of thesame area. Tools for registration such as ITK (Insight Toolkit) and MATLAB exist for these tasks. Applicationsof such synchronized viewing can be manifold as they can be used in any situation where several tomographicvolumes need to be compared. One situation could be the case of patients with many follow–up image series,where the visual content in these series needs to be compared over time. In this case a clinician can scroll throughthe slices and then compare all target volumes, always having the multiple views synchronized automatically.For content–based 3D image retrieval this can also help to better judge the results, particularly when the textureis the distinguishing element and not the shape of objects, which could be shown in 3D–rendered views aftersegmentation. Manually adapting the slices can be an option when there are few series and few images per seriesbut with modern scanners producing hundreds of images even for small areas, new tools become essential to dealwith the increasing amount of data and give clinicians the possibility to analyze data efficiently.

2. METHODS

Several existing tools were adapted for the task described in this paper. The Insight toolkit (ITK∗) containsmany tools for image processing and image registration. In this article ITK is used for the image processingalgorithms such as the registration. VTK (Visualization Toolkit†) on the other hand allows visualization of datasets such as 3D volumes. For the quick development of the user interface the Qt‡ framework was employed andthe ITK/VTK code was integrated.

Two sets of images have been used in a first step. A database of 100 patients with interstitial lung diseasescontaining high resolution CT of the lung, mostly with 1mm slice thickness and 10mm inter slice distance wasused.? A database of brain MRI images§ is also used to show the tools work for a variety of modalities.

3. RESULTS

3.1. Synchronizing

To synchronize several image volumes (mainly axial views) with a reference volume that is selected, all targetvolumes are automatically registered with the reference volume using 3D rigid registration of ITK. After theregistration, the slices can be situated in the volumes and thus the closest slices to any given slice in the referencevolume can in any situation be calculated and shown on screen, based on the registration and also based on theinformation of slice thickness and inter slice distance in the DICOM header. Visual similarity of the slices canalso be used to synchronize the slices in an optimal manner. The current 3D rigid registration consists of rigidtransformations that are a combination of rotations R and translations t:

I2 = RI1 + t, (1)

∗http://www.itk.org/†http://www.vtk.org/‡http://qt.nokia.com/§The MR brain data sets and their manual segmentations were provided by the Center for Morphometric Analysis at

Massachusetts General Hospital and are available at http://www.cma.mgh.harvard.edu/ibsr/.

Figure 1. SSD values for various registration tasks. The registration of the lung CT scans did not converge.

Figure 2. A first screenshot of the test interface showing four image series at the bottom of the interface synchronizedwith a source series that is shown in an axial view and also in a simple 3D reconstruction.

where I1 and I2 are the intensity values of the voxels of the reference and registered images respectively. Thetransformations are applied to minimize the sum of squared differences (SSD) between intensity values:?

SSD =1

NxNyNz

Nx∑i=1

Ny∑j=1

Nz∑k=1

ÅI1(i, j, k) − I2(x + i, y + j, z + k)

ã2. (2)

Rigid transforms based on (1) are iteratively applied until the difference of SSD between two iterations isminimized to a threshold q = 10−4. The evolutions of SSD of consecutive iterations of the rigid registrationare shown in Figure 1 for the database of lung CT images and brain MRIs. The synchronization of the series isrejected if SSD cannot achieve the quality threshold q, leading to a non–convergence of the rigid registration.Other similarity measures that are invariant to greyscale value ranges are currently under investigation, such asthe risk of non–convergence proposed by Skerl et al.?

3D rigid registration delivered satisfying qualitative results for series of basically the same anatomic areaand having sufficient resolution in the z axis. If there is only a partial overlap between the reference and targetvolumes other techniques are required that are currently being implemented and tested.

The current system uses a desktop PC with 16 GB of RAM and a 4–core processor. Registering the seriescan be fast but may take quite long if rotations and translations are necessary. Depending on the number ofiterations required to obtain convergence, the registration in the tests took between 3 seconds for series withbasically no translation and rotation to almost 10 minutes for a series with very strong shift and rotation. So far,computation efficiency was not a major goal. Synchronization is then fast, taking around 1 second per series inaverage. So far, content–based retrieval of similar image series has only been tested with lung CT images usingthe methods described in.?

3.2. User interface

A simple test interface was developed using the Qt cross–platform application framework and then integratingthe image processing part for the corresponding volumes. Figure 2 shows the interface with the view of thereference series including a 3D view and the target volumes that are synchronized based on a user manipulatingthe source series. When the user slides through the source series, the target series are all synchronized. Thesame tools can be used in a content–based retrieval setting where several results series need to be synchronizedbut also for patients where several image series need to be compared over time. Such synchronized viewing cansave time when handling several image series.

Besides the shown test interface, the development of a plugin to OsiriX¶ has started as this will make thetools available to a large number of radiologists with a better interface, thus potentially increasing the impactof the tool and obtaining more feedback on adapting the system.

4. NEW OR BREAKTHROUGH RESULTS

To the best of our knowledge this is the first article describing a system that uses purely visual image analysismethods and registration to allow for synchronized viewing of several tomographic volumes of imaging data ina fully automatic way. Such automatic synchronization of several image series adapts the viewed slice of all

¶http://www.osirix-viewer.com/

volumes to the same place inside the volume without manual intervention making such viewing much faster andeasier to use. This synchronized viewing can help working with patients containing many volumes of the samearea that needs to be compared manually. For 3D content–based medical image retrieval this can also helpcomparing several results series in a synchronized way, making it much faster to find relevant data. Currently,synchronization requires to at least align one slice between several volumes, as then the slice thickness and interslice distance of the DICOM headers can be used for making the correspondence.

5. CONCLUSIONS

The system developed in the project described in this paper has several practical applications and can makeimage viewing and image comparisons of 3D data sets much faster and more effective by synchronizing severalvolumes. The tool is based on open source components and thus allows to be reproduced fairly easily and withoutmajor cost. An integration into OsiriX will give access to the tools for a larger number of radiologists and willhelp a large–scale validation of the tool. Currently, the validation has only been performed on data sets takenunder fairly similar conditions, so with almost the exact same location of the volumes. Besides the advantagesof the described tools we have also identified several shortcomings of the current system that are currently beingworked on. It would be optimal to have standard models for all anatomic regions that all other volumes oreven single slices can be mapped to. Single slices would be hard to register as only very limited information isavailable and finding the visually most similar slice does not necessarily mean to be in the same region. Thiscould help extracting information from many information sources such as the medical literature (JPG images oflimited resolution) and link these with certain regions and even local abnormalities in clinical cases in DICOMformat, potentially connecting a clinical record with related articles in the medical literature.

6. SUBMITTED ELSEWHERE

The content of this paper has never been submitted elsewhere and is not planned to be submitted elsewhere.

7. ACKNOWLEDGEMENTS

We would like to thank the EU for their funding in the context of the Khresmoi project (grant 257528) and theSwiss National Science foundation (205321–130046).