MONITORING MESENCHYMAL STEM CELL CULTURES …eprints.qut.edu.au/43623/1/Christopher_Bradhurst_Thesis.pdf · MONITORING MESENCHYMAL STEM CELL CULTURES USING IMAGE PROCESSING AND PATTERN

MONITORING MESENCHYMAL STEM CELL CULTURES USING

IMAGE PROCESSING AND PATTERN RECOGNITION

TECHNIQUES

Christopher J. Bradhurst

BEng (Hons)

SUBMITTED AS A REQUIREMENT OF

THE DEGREE OF

MASTERS BY RESEARCH

AT

QUEENSLAND UNIVERSITY OF TECHNOLOGY

BRISBANE, QUEENSLAND

30 JULY 2010

i

Keywords

Image processing, pattern recognition, mesenchymal stem cell, bone marrow stromal cell,

quality assessment

iii

Abstract

Stem cells have attracted tremendous interest in recent times due to their promise in

providing innovative new treatments for a great range of currently debilitating diseases.

This is due to their potential ability to regenerate and repair damaged tissue, and hence

restore lost body function, in a manner beyond the body's usual healing process. Bone

marrow-derived mesenchymal stem cells or bone marrow stromal cells are one type of

adult stem cells that are of particular interest. Since they are derived from a living human

adult donor, they do not have the ethical issues associated with the use of human

embryonic stem cells. They are also able to be taken from a patient or other donors with

relative ease and then grown readily in the laboratory for clinical application.

Despite the attractive properties of bone marrow stromal cells, there is presently no quick

and easy way to determine the quality of a sample of such cells. Presently, a sample must

be grown for weeks and subject to various time-consuming assays, under the direction of

an expert cell biologist, to determine whether it will be useful. Hence there is a great need

for innovative new ways to assess the quality of cell cultures for research and potential

clinical application.

The research presented in this thesis investigates the use of computerised image

processing and pattern recognition techniques to provide a quicker and simpler method for

the quality assessment of bone marrow stromal cell cultures. In particular, aim of this

work is to find out whether it is possible, through the use of image processing and pattern

recognition techniques, to predict the growth potential of a culture of human bone

marrow stromal cells at early stages, before it is readily apparent to a human observer.

With the above aim in mind, a computerised system was developed to classify the quality

of bone marrow stromal cell cultures based on phase contrast microscopy images. Our

system was trained and tested on mixed images of both healthy and unhealthy bone

marrow stromal cell samples taken from three different patients. This system, when

presented with 44 previously unseen bone marrow stromal cell culture images,

outperformed human experts in the ability to correctly classify healthy and unhealthy

cultures. The system correctly classified the health status of an image 88% of the time

compared to an average of 72% of the time for human experts. Extensive training and

testing of the system on a set of 139 normal sized images and 567 smaller image tiles

showed an average performance of 86% and 85% correct classifications, respectively.

The contributions of this thesis include demonstrating the applicability and potential of

computerised image processing and pattern recognition techniques to the task of quality

assessment of bone marrow stromal cell cultures. As part of this system, an image

normalisation method has been suggested and a new segmentation algorithm has been

developed for locating cell regions of irregularly shaped cells in phase contrast images.

Importantly, we have validated the efficacy of both the normalisation and segmentation

method, by demonstrating that both methods quantitatively improve the classification

performance of subsequent pattern recognition algorithms, in discriminating between cell

cultures of differing health status. We have shown that the quality of a cell culture of bone

marrow stromal cells may be assessed without the need to either segment individual cells

or to use time-lapse imaging. Finally, we have proposed a set of features, that when

extracted from the cell regions of segmented input images, can be used to train current

state of the art pattern recognition systems to predict the quality of bone marrow stromal

cell cultures earlier and more consistently than human experts.

v

Contents

Keywords .................................................................................................................................. i

Abstract .................................................................................................................................. iii

Contents .................................................................................................................................. v

List of Tables ........................................................................................................................... xi

List of Figures ........................................................................................................................ xiii

List of Abbreviations .............................................................................................................. xv

Authorship ............................................................................................................................ xvii

Acknowledgments ................................................................................................................. xix

Publications ........................................................................................................................... xxi

Chapter 1 Introduction ......................................................................................................... 1

1.1 Motivation and Overview ........................................................................................ 1

1.2 Aims and Research Questions ................................................................................. 2

1.3 Research Scope ........................................................................................................ 4

1.4 Thesis Organisation ................................................................................................. 5

1.4.1 Culturing and Imaging of Stem Cells ................................................................ 5

1.4.2 Segmentation of Phase Contrast Images ......................................................... 6

1.4.3 Feature Extraction and Pattern Recognition ................................................... 6

1.4.4 Software System ..............................................................................................6

1.4.5 Results and Discussion .....................................................................................7

1.4.6 Conclusions and Future Work ..........................................................................7

1.5 Thesis Contributions ................................................................................................7

1.5.1 Publications ......................................................................................................8

1.6 Human Experts .........................................................................................................8

Chapter 2 Culturing and Imaging of Stem Cells ................................................................. 11

2.1 Stem Cells .............................................................................................................. 11

2.1.1 Bone Marrow Stromal Cells .......................................................................... 13

2.2 Computerised Quality Assessment of Stem Cells ................................................. 17

1.1.1. Culture Based Assessment .................................................................................. 18

2.2.1 Single Cell Based Assessment ....................................................................... 19

2.2.2 Cell Tracking Based Assessment ................................................................... 20

2.3 Data Collection ...................................................................................................... 20

Overview of Proposed System ...................................................................................... 21

2.3.1 Equipment ..................................................................................................... 22

2.3.2 Pilot Experiments .......................................................................................... 24

2.3.3 Method - Main Experiment ........................................................................... 27

2.4 Example Images .................................................................................................... 32

2.4.1 Cells from different patients ......................................................................... 32

2.4.2 Cells at day 1 - 10% FCS vs 0.2% FCS ............................................................. 33

2.4.3 Sparse versus dense ...................................................................................... 34

2.4.4 Progression over 6 days ................................................................................ 34

2.5 Summary ............................................................................................................... 34

Chapter 3 Segmentation of Phase Contrast Images of Bone Marrow Stromal Cells ......... 39

vii

3.1 System Overview ................................................................................................... 39

3.2 Preprocessing ........................................................................................................ 41

3.2.1 Image Normalisation ..................................................................................... 42

3.3 Segmentation ........................................................................................................ 43

3.3.1 Rough Segmentation ..................................................................................... 45

3.3.2 Refined Segmentation ................................................................................... 47

3.4 Performance Evaluation ........................................................................................ 49

3.4.1 Method .......................................................................................................... 52

3.5 Results and Discussion ........................................................................................... 54

3.5.1 Effect of Normalisation .................................................................................. 55

3.5.2 Effect of Segmentation .................................................................................. 56

3.5.3 Performance on Test Data ............................................................................. 56

3.5.4 Limitations ..................................................................................................... 57

3.6 Final Comments ..................................................................................................... 58

Chapter 4 Feature Extraction and Pattern Recognition for BMSC Quality Assessment .... 59

4.1 Introduction ........................................................................................................... 59

4.2 Feature Extraction ................................................................................................. 60

4.2.1 Morphometric Features ................................................................................ 60

4.2.2 Densiometric/Intensity-based Features ........................................................ 61

4.2.3 Textural Features ........................................................................................... 62

4.2.4 Structural/Contextual Features ..................................................................... 62

4.2.5 Features for this project ................................................................................ 62

4.3 Pattern Recognition ............................................................................................... 65

4.3.1 Multi-Layer Perceptron ................................................................................. 67

4.3.2 Support Vector Machine ............................................................................... 68

4.4 Evaluating Performance ........................................................................................ 69

4.5 Summary ............................................................................................................... 71

Chapter 5 Software System ............................................................................................... 73

5.1 Overview ............................................................................................................... 73

5.2 CellScan ................................................................................................................. 75

5.3 FeatureDB ............................................................................................................. 76

5.4 CellAPR .................................................................................................................. 79

5.5 Expert Performance Evaluation Tool (EPET) ......................................................... 81

5.6 Summary ............................................................................................................... 82

Chapter 6 Results and Analysis .......................................................................................... 83

6.1 Introduction .......................................................................................................... 83

6.2 System Performance - Day 1 Images .................................................................... 84

6.2.1 Whole Images ............................................................................................... 84

6.2.2 Tiled Images .................................................................................................. 86

6.2.3 Random Windows ......................................................................................... 88

6.2.4 MLP versus SVM ............................................................................................ 90

6.2.5 Effect of Including Incorrectly Segmented Images ....................................... 91

6.3 Comparison with Human Experts ......................................................................... 93

6.4 System Performance: Days 1 to 6 ........................................................................ 97

6.5 Performance by Feature Groups ........................................................................... 99

6.5.1 Total Cell Area Features Only ...................................................................... 100

6.5.2 Inner Cell Area Features Only ..................................................................... 101

6.5.3 Outer Cell Area Features Only .................................................................... 102

6.5.4 Mixture Modelling Threshold Feature Only ................................................ 102

6.5.5 Outer Cell Area Features and Mixture Modelling Threshold Only ............. 103

ix

6.6 Summary .............................................................................................................. 105

Chapter 7 Conclusions and Future Work ......................................................................... 107

7.1 Conclusions .......................................................................................................... 107

7.1.1 How should cell culture images be captured that are likely to contain useful

predictive information? ............................................................................................... 108

7.1.2 What useful measurable attributes can be extracted from the captured

images? ..................................................................................................................... 109

7.1.3 How can sets of attributes extracted from captured images be classified in

order to predict the outcome of a cell culture? .......................................................... 110

7.2 Thesis Contribution ............................................................................................. 111

7.3 Future Work ......................................................................................................... 112

7.3.1 Automatic Removal of Segmentation Failures ............................................ 112

7.3.2 Expansion of Cell Image Data ...................................................................... 113

7.3.3 Further Expert Evaluation ............................................................................ 113

7.3.4 Further Biological Evaluation ....................................................................... 113

7.3.5 Selection and Biological Interpretation of Features .................................... 113

7.3.6 Classification Confidence ............................................................................. 114

7.4 Final Remarks ...................................................................................................... 115

Appendix A Definition of Feature Statistics .................................................................... 117

Mean ............................................................................................................................ 117

Mode ........................................................................................................................... 117

Median ......................................................................................................................... 117

Standard Deviation ...................................................................................................... 118

Skew ............................................................................................................................. 118

Kurtosis ........................................................................................................................ 118

Entropy ........................................................................................................................ 119

Uniformity ................................................................................................................... 119

Appendix B Additional Performance Data ..................................................................... 121

Performance on Day 1 .................................................................................................... 122






Appendix C Software Source Code ................................................................................. 129

CellScan Source Code ...................................................................................................... 130

Dependencies .............................................................................................................. 130

Notes ........................................................................................................................... 130

Source Code Listing ..................................................................................................... 130

CellAPR Source Code ....................................................................................................... 162

Dependencies .............................................................................................................. 162

Notes ........................................................................................................................... 162

Source Code Listing ..................................................................................................... 163

Bibliography ........................................................................................................................ 177

xi

List of Tables

Table 1 - Effect of normalisation method vs classifier performance .................................... 55

Table 2 - Effect of segmentation on classifier performance ................................................. 56

Table 3 - Performance on previously unseen test data ......................................................... 57

Table 4 - Total feature set used in our quality assessment system ...................................... 65

Table 5 - Classifiers used in recent cell or tissue related research ........................................ 66

Table 6 - FeatureDB feature data tables ............................................................................... 77

Table 7 - MLP performance on whole images - Day 1 ........................................................... 85

Table 8 - SVM performance on whole images - Day 1 .......................................................... 85

Table 9 - MLP performance on non-overlapping tiles - Day 1 ............................................... 87

Table 10 - SVM performance on non-overlapping tiles - Day 1 ............................................ 87

Table 11 - MLP performance on random windows - Day 1 ................................................... 89

Table 12 - SVM performance on random windows - Day 1 .................................................. 89

Table 13 - MLP performance on whole images - Day 1 - Segmentation failures included ... 92

Table 14 - SVM performance on whole images - Day 1 - Segmentation failures included ... 92

Table 15 - Human Expert Performance vs System ................................................................ 94

Table 16 - Confusion Matrix - Human Expert 1 ..................................................................... 95

Table 17 - Confusion Matrix - Human Expert 2 ..................................................................... 95

Table 18 - Confusion Matrix - MLP ........................................................................................ 95

Table 19 - Confusion Matrix - SVM ........................................................................................ 95

Table 20 - Test subject misclassifications by patient sample ............................................... 96

Table 21 - Test subject misclassifications by actual classification, sample and well ............ 97

Table 22 - MLP and SVM Test set performance on whole images - Days 1 to 6 .................. 98

Table 23 - MLP performance – Total cell area features only .............................................. 100

Table 24 - SVM performance – Total cell area features only.............................................. 100

Table 25 - MLP performance – Inner cell area features only.............................................. 101

Table 26 - SVM performance – Inner cell area features + MM Threshold only ................. 101

Table 27 - MLP performance – Outer cell area features only ............................................. 102

Table 28 - SVM performance – Outer cell area features only ............................................ 102

Table 29 - MLP performance – Mixture Modelling Threshold feature only ....................... 103

Table 30 - SVM performance – Mixture Modelling Threshold feature only ....................... 103

Table 31 - MLP performance – Outer cell area features + MM Threshold only ................. 104

Table 32 - SVM performance – Outer cell area features + MM Threshold only ................. 104

Table 33 - MLP and SVM Test set performance - By feature groups .................................. 105

Table 34 - MLP performance on whole images - Day 1 ...................................................... 122

Table 35 - SVM performance on whole images - Day 1 ...................................................... 122











xiii

List of Figures

Figure 1 - Embryonic stem cell colony. .................................................................................. 12

Figure 2 - Human bone marrow stromal cells. ...................................................................... 13

Figure 3 - Bone marrow extraction........................................................................................ 15

Figure 4 - Proposed quality assessment system .................................................................... 21

Figure 5 - Inverted microscope.and controlling computer.................................................... 22

Figure 6 - PixeLINK digital camera and PixeLINK Capture SE controlling software. .............. 24

Figure 7 - Field of view by objective lens magnification. ...................................................... 25

Figure 8 - Same culture dish location imaged at 24hr intervals over 3 days......................... 26

Figure 9 - General order and location of image capture for each culture well ..................... 28

Figure 10 - Six well plate showing condensation................................................................... 30

Figure 11 - Laminar flow hood ............................................................................................... 30

Figure 12 - Cell culture growth rates for cultures from patients A, B and C ......................... 31

Figure 13 - Example of different cell morphologies from three different patients .............. 32

Figure 14 - Comparison of cell images for 0.2% FCS and 10% FCS ........................................ 33

Figure 15 - Variation in cells numbers present in two concurrent images ........................... 34

Figure 16 - Cell culture growth over six days for 0.2% FCS and 10% FCS .............................. 36

Figure 17 - Overview of proposed quality assessment system ............................................. 40

Figure 18 - Expanded overview of quality assessment system ............................................. 41

Figure 19 - Phase contrast images of BMSCs taken at 10x magnification ............................. 42

Figure 20 - Challenges associated with phase contrast imaging ........................................... 44

Figure 21 - Rough segmentation ........................................................................................... 46

Figure 22 - Filter mask sizes .................................................................................................. 46

Figure 23 - Pixel intensity profile .......................................................................................... 48

Figure 24 - Refined segmentation ......................................................................................... 49

Figure 25 - Examples of touching and partially visible cells .................................................. 53

Figure 26 - Classifier performance vs % saturated pixels (contrast stretching) ................... 56

Figure 27 - Incorrect segmentation on highly confluent culture .......................................... 57

Figure 28 - Texture examples ................................................................................................ 61

Figure 29 - Cell area masks. .................................................................................................. 63

Figure 30 - Effect of varying the erosion radius on cell region size ...................................... 64

Figure 31 - Circular structuring element used for erosion operation. .................................. 64

Figure 32 - Software system components ............................................................................ 74

Figure 33 - Activating the CellScan plug-in from ImageJ....................................................... 75

Figure 34 - The CellScan graphical user interface ................................................................. 76

Figure 35 - Form for visual inspection of segmentation results ........................................... 79

Figure 36 - Tabular output produced by CellAPR .................................................................. 80

Figure 37 - Expert Performance Evaluation Tool .................................................................. 81

Figure 38 - Performance over Days 1 to 6 ............................................................................ 98

Figure 39 - Existing system modified to include extra segmentation validation stage ...... 112

xv

List of Abbreviations

BMSC Bone Marrow Stromal Cell

DMEM Dulbecco’s Modified Eagle's Medium

GUI Graphical User Interface

FCS Foetal Calf Serum

FOV Field of View

ML Machine Learning

MLP Multi-Layer Perceptron

MSC Mesenchymal Stem Cell

PBS Phosphate Buffered Saline

PC Personal Computer

PR Pattern Recognition

RBF Radial Basis Function

SVM Support Vector Machine

SQL Structured Query Language

VBA Visual Basic for Applications

WBC White Blood Cell

xvii

Authorship

The work contained in this thesis has not been previously submitted for a degree or

diploma at any other higher educational institution. To the best of my knowledge and

belief, the thesis contains no material previously published or written by another person

except where due reference is made.

Signed: _________________________

Date: _________________________

xix

Acknowledgments

I would first like to express my appreciation for Wageeh Boles, my supervisor, for his time,

advice, encouragement and patience over the duration of my research. Many thanks also

to my associate supervisor Yin Xiao, for helping me to navigate the unfamiliar territory of

the biological sciences. Along the same lines, I would like to make special mention of Thor

Friis, for his frequent and much needed assistance in the cell culture lab.

Thanks also to the many others at IHBI who I've had the pleasure of associating with, and

who have provided help in many different ways, especially Shobha Mareddy, Sanjleena

Singh, Wei Fan, Navdeep Kaur and Indira Prasadam.

I also had the good fortune to work with Arnold Wiliem, and have very much appreciated

his thoughtful words and friendship during my time at QUT. Likewise, thanks to Elfrith

Footit, for his various tips and advice during the earlier part of my research. I'd also like to

acknowledge the friendly support of the staff in the Research Portfolio office at QUT,

including Elaine Reyes, Kathryn Pinel and Diane Kolomeitz.

A research project and the writing up of a thesis can be a long and winding road. Special

thanks to those whose support and kind words helped me along the journey, including

David and Jackie Russell, Helen Munro and Violette Lewis.

Finally, and most especially, I would like to thank my mother Jill Bradhurst, and my wife

Rachael for their enduring understanding, support and encouragement, that have helped

turn my ideas into reality.

Christopher Bradhurst

Queensland University of Technology

July 2010

xxi

Publications

This research has resulted in the following fully peer-reviewed publication:

C. J. Bradhurst, W. Boles, and Y. Xiao, "Segmentation of bone marrow stromal cells in phase

contrast microscopy images," in 23rd International Conference on Image and Vision

Computing, New Zealand, Christchurch, 2008.

1

Chapter 1

Introduction

1.1 Motivation and Overview

Bone marrow stromal cells (BMSCs), also called bone marrow-derived mesenchymal stem

cells (MSCs) [1, 2] have generated considerable excitement in recent times because of their

ability to change (differentiate) into a considerable variety of cell types. Because of this,

they hold significant promise for use in regenerative therapies whereby damaged tissue

may be re-grown, offering hope for the treatment of many debilitating and otherwise

untreatable diseases.

An attractive feature of these types of stem cells is that they can be extracted from a living

patient, without the ethical and moral issues associated with embryonic stem cells, which

require the destruction of a human embryo.

Once cells are extracted from a patient, it is necessary to culture the cells outside the body

(in vitro) so that they may grow (proliferate) in sufficient numbers for use in research or

possible treatments. Once sufficient numbers of cells have been grown in this way, they

are then induced to differentiate into desired cell types appropriate to their intended

application. For example, in attempting to repair damaged cartilage, the cells can be

differentiated to chondrocytes, the cells making up cartilage tissue.

However, not all samples initially taken from a patient will result in a useful batch of stem

cells. In fact, even different samples taken from the same patient at the same time can

vary substantially in their growth potential [3]. In some cases, the cells may spontaneously

differentiate into another cell type and hence lose their 'stemness'. In other cases, the

2 Chapter 1. Introduction

cells may just not grow in sufficient quantity or lose their ability for self-renewal.

Additionally, cells may also become limited into what particular cell types they can

become.

Since the culturing process is time-consuming, it is desirable to be able to assess as early as

possible the quality of a stem cell culture. In this way, a culture deemed to be of poor

quality can be replaced with a new sample. The problem is that there is presently no quick

and easy method of assessing the quality of a culture of BMSCs. Laboratories must rely on

time consuming lab methods to identify the quality of stem cells carried out by an

experienced cell biologist using extensive assays and tests. This current difficulty in

monitoring stem cell selection and culture has hampered the potential application of stem

cell based therapy. Therefore there is an urgent need for new, more efficient, methods in

assessing and monitoring stem cell cultures.

This project proposes to address the problems of the current stem cell quality assessment,

as mentioned above, by using computer image processing and pattern recognition

techniques. The basic idea is to create a computerised system such that a cell researcher

can take digital images of a cell culture, input them to the (appropriately trained) system

and have the system give back an indication of the quality of the cell culture.

1.2 Aims and Research Questions

The general aim of the work presented here is to find out whether it is possible through

the use of image processing and pattern recognition techniques to predict the growth

potential of a culture of human mesenchymal stem cells at early stages, before it is readily

apparent to a human observer.

The following main questions to be considered are:

How should cell culture images be captured that are likely to contain useful predictive

information?

Key sub-questions to be investigated here include:

• What microscope magnification (i.e. objective lens) or combination of

magnifications will provide the best trade-off between the number of cells visible

Chapter 1. Introduction 3

(i.e. field of view) and the detail of individual cells, given the resolution of the

camera equipment used?

• How will a reasonable consistency of view from day to day be managed? For

example, should the culture dishes be marked with a reference indicator so that

images can be taken from the same place each day, or can a portion of the cell

culture that is considered to be representative of the whole be taken?

• At what frequency throughout the early culture process should images be taken

(e.g. daily)?

What useful measurable attributes can be extracted from the captured images?

In order to form predications of culture viability, a consistent set of measurable attributes

need to be extracted from the images. However, to extract such measurements, it will

likely be necessary to segment each image into the parts containing cells and the parts

containing the background. Also, although there are a great many possible attributes that

may be extracted from an image, we need to select those attributes that have some

predictive potential.

Some sub-questions to be considered include the following:

• What algorithms will be most effective in dealing with the relatively low contrast

nature of living cell images taken under phase contrast microscopy, and with the

irregular morphology associated with BMSCs?

• Will it even be possible to segment individual cells, especially as the culture grows

and the cells become more densely packed? Should we instead treat the entire

culture as one large texture, or patchwork of textures?

• Of the extracted features, how will an optimal subset be selected that is most

indicative of a viable or unviable culture?

How can sets of attributes extracted from captured images be classified in order to

predict the outcome of a cell culture?

Particular sub-questions to be examined include:


• What classes should be used for classifying individual cells (or areas of similar

texture in an entire image)?

• What classification algorithm or algorithms should be employed for the

determination of cell types or image texture area types?

• What classification algorithm or algorithms should be employed for the prediction

of overall cell culture viability?

• How will the actual state of a cell culture (in terms of viability) be quantified, so

that images from it may serve as a ground truth during the training and testing of

our system? What set of measured characteristics (obtained from various

laboratory assays, cell counts, flow cytometry, etc) will denote a viable culture

versus an unviable culture?

• Will it be possible to predict the outcome from just one image of the cell culture,

rather than a time-series of images? If so, what day of culture development

seems the most indicative of culture outcome?

The research questions above help guide development of the key aspects of this project.

These are the capture of stem cell culture images, extraction of data from these images,

and the use of this data for computerised classification of cell culture quality. Some limits

on this research, mostly relating to the nature of the cells available for study, are

considered in the section below.

1.3 Research Scope

• The cells studied in this thesis are bone marrow stromal cells taken from patients

with osteoarthritis. These cells are deemed acceptable, as it has been shown that

bone marrow stromal cells taken from such patients can still have the potential to

proliferate and differentiate into desired cell types, such as chondrocytes for

cartilage repair [4].

• The cells used in this research have been grown from frozen stocks that were

collected prior to this research project. The freezing process has been shown not


to substantially interfere with the later growth and differentiation potential of

bone marrow stromal cells [5].

• Cells to be used as examples of unhealthy cultures were artificially induced to be of

poor quality by adjusting the amount of available nutrients. This was necessary due

to the lack of available cell stocks for which quantified cell quality information was

available. It was also to ensure there were clear examples of both normally

growing and poorly growing cells for use in training and testing the pattern

recognition system.

• For the purposes of this study, the goal is not to find the definitively best pattern

recognition scheme (e.g. multi-layer perceptron versus support vector machine,

etc) or the highest possible accuracy, but rather what can be achieved with readily

available state-of-the-art systems. As described at the beginning of the previous

section, our aim is to determine whether it is possible for a computerised system

to predict the growth potential of stem cell cultures. Thus the main concern is to

evaluate the feasibility and potential of using such a system for this task,

particularly in comparison to the efficacy of human experts.

1.4 Thesis Organisation

The following is a brief synopsis of the chapters to follow:

1.4.1 Chapter 2 – Culturing and Imaging of Stem Cells

This chapter provides a general introduction to the subject of stem cells, and in particular,

the culturing and imaging of bone marrow stromal cells (the type studied in this thesis).

Following this general background, we review the current literature pertaining to

computerised quality assessment of stem cell cultures. We then look at the more specific

requirements for this project, namely the gathering of a suitable set of stem cell culture

images. In the context of this research, this means a set of images containing examples of

both quantifiably healthy and unhealthy cultures. The equipment and methodology used is

described including the various practicalities of cell image capture, such as dealing with

condensation, camera and microscope settings. The achieved growth profiles of our

healthy and unhealthy cells, over six days in culture is presented to validate their health


status. The chapter concludes with a qualitative examination of typical cell images

acquired. For example, images are compared in a variety of ways, such as the variation by

patient, cell health status, culture time and so on. This aims to give the reader an intuitive

feel for the challenge faced by our computerised system, and indeed the human expert, in

telling the difference between healthy and unhealthy cells in the initial stage of cell culture.

1.4.2 Chapter 3 - Segmentation of Phase Contrast Images of Living Bone

Marrow Stromal Cells

Having acquired a set of suitable cell culture images based on the work outlined in the

latter part of chapter 2, this chapter covers the image processing required to locate the

cells within each image. The purpose is to allow cell feature measurements to be extracted

for use by the pattern recognition algorithms described in chapter 4. After a brief review

of relevant literature pertaining to the segmentation of living cell images, the

normalisation of images is examined. Normalisation is required in order to deal with

variations in image intensity between different images that is not related to the cells

themselves. Following this, a new algorithm to locate bone marrow stromal cells within

phase contrast images is presented. The effectiveness of both the normalization and the

segmentation algorithm is evaluated by testing their effect on the recognition accuracy of a

subsequent pattern recognition phase.

1.4.3 Chapter 4 - Feature Extraction and Pattern Recognition for BMSC Quality

Assessment

Chapter 4 looks at the extraction of a set of statistical features from the segmented images

created according to the algorithms described in chapter 3. The definition of the features

used is provided. Two different pattern recognition schemes are described, including a

basic description of the theory involved for each. The chapter concludes with a description

of the techniques employed to evaluate performance of our classifiers for cell quality

assessment, including the division of our image data into training and testing sets.

1.4.4 Chapter 5 - Software System

Here we discuss the software tools developed as part of this research project. The

software includes tools that automate both the image processing and pattern recognition


functions, that are described in chapters 3 and 4, respectively. Additionally, a database

was created for the storage and organisation of extracted feature data, and its division into

multiple training and testing sets. Also described is a tool created for evaluating the ability

of human experts to differentiate between healthy and unhealthy cell cultures.

1.4.5 Chapter 6 - Results and Discussion

This chapter evaluates our quality assessment system as a whole, by training it and testing

its performance using the stem cell images collected as part of the work described in

Chapter 2. The complete system combines the normalisation and segmentation scheme

presented in Chapter 3 with the pattern recognition schemes and performance evaluation

methodology described in Chapter 4. The accuracy of this resulting computerised system is

reported and discussed. In particular, the effect on accuracy of such things as the choice of

recognition algorithm, image tile size, age of culture and feature subsets is described and

discussed. Where possible, likely explanations are offered for observed effects, along with

the implications for cell quality assessment. Importantly, the performance achieved by

human experts is presented and compared with that achieved by our computerised

system.

1.4.6 Chapter 7 - Conclusions and Future Work

This final chapter provides a summary and draws overall conclusions regarding the

methods presented for the task of assessing the quality of bone marrow stromal cell

cultures. Our research questions are revisited, and answers given in the light of our

research results. Finally, suggestions are also made for how this research could be further

extended.

1.5 Thesis Contributions

• Have demonstrated the applicability and potential of computerised image

processing and pattern recognition techniques to the task of bone marrow stromal

cell culture quality assessment.

• A new segmentation algorithm has been developed for locating cell regions of

irregularly shaped cells in phase contrast images. This is useful, not only as a


precursor to our quality assessment application, but also potentially as a

measurement aid to help cell researchers make more objective measurements of

cell area, perimeter and so on.

• Importantly, we have validated the efficacy of our choice of normalisation and

segmentation method, by demonstrating that both methods quantitatively

improve the classification performance of subsequent pattern recognition

algorithms, in discriminating between cell cultures of differing health status.

• Have established that it is not necessary to segment individual cells in an image of

a bone marrow stromal cell culture, in order for a pattern recognition system to

classify it on the basis of its health state. Segmenting cell regions with the cell

image was found to be sufficient.

• Have proposed a set of features that when extracted from segmented input images

can be used to train current state of the art pattern recognition systems to predict

the quality of bone marrow stromal cell cultures earlier and more consistently than

human experts.

1.5.1 Publications

C. J. Bradhurst, W. Boles, and Y. Xiao, "Segmentation of bone marrow stromal cells

in phase contrast microscopy images," in 23rd International Conference on Image

and Vision Computing, New Zealand, Christchurch, 2008.

1.6 Human Experts

Due to the multi-disciplinary nature of this research, the term 'human experts' is used in a

number of places throughout this thesis. This term, unless indicated otherwise, refers to

those with considerable knowledge and experience in the field of cell biology, and in

particular, bone marrow stromal cells. The assistance of such experts was required both in


providing information, advice and direction on matters pertaining to bone marrow stromal

cells, as well as for comparative purposes when evaluating the classification accuracy of

our computerised system.

The experts referred to in this research project are all full-time researchers at the Institute

of Health and Biomedical Innovation (http://www.ihbi.qut.edu.au) based at the

Queensland University of Technology in Brisbane, Australia. They include Associate

Professor Yin Xiao, Thor Friis and Shobha Mareddy.


[This page is intentionally left blank]

11

Chapter 2

Culturing and Imaging of Stem

Cells

In this chapter we examine the current literature relating to the culturing and imaging of

stem cells then later, and more specifically, we look at how the images and other data

required for this project were gathered. We also preview and discuss representative

examples of these images. We begin now with a basic overview of stem cells in general,

before progressing to bone marrow stromal cells, the type dealt with in this project.

2.1 Stem Cells

Stem cells are unspecialised cells which are able reproduce themselves as well as give rise

to the specialised cell types that are necessary to the normal development and operation

of the body. The range of specialised cell types that a given stem cell can produce depends

upon its type. There are three broad categories of stem cells: embryonic, adult and induced

pluripotent stem cells (IPSCs). These are now briefly described below, however, for a more

complete introduction to the subject the interested reader is referred to [1].

Embryonic stem cells are derived from embryos developed from fertilized eggs. They are

extracted very early on in the development of the embryo, typically when it is four to five

days old. In normal development, the cells extracted would normally give rise to all the

cells that would make up the fully developed human. As such, embryonic stems cells are

described as pluripotent, since a single one of these cells can give rise to all cell types in the

body. Unfortunately, since these cells require the destruction of a fertilized human

12 Chapter 2. Culturing and Imaging

embryo, there are ethical issues associated with their use. Figure 1 shows an example

culture of human embryonic stem cells. The stem cells make up the fine grained circular

shaped mass in the centre of the image. The darker spindle shaped cells surrounding them

are 'feeder cells' used to support the colony and are not stem cells.

Figure 1 - Embryonic stem cell colony (round cluster in centre) taken with a 10x objective lens. Note: The

long spindle shaped cells surrounding the colony are 'feeder cells' used to support the colony and are not

stem cells.

Adult stem cells are undifferentiated cells that exist in the developed human or animal

body. They have the ability to differentiate into the cell types that make up the tissue or

organ system from which they originate. As such, they appear to serve as a repair

mechanism for the tissue with which they are associated. Adult stem cells are described as

multipotent, meaning they have the potential to differentiate into multiple types of cells,

although not all the cell types that make up the body. Although more limited in their

differentiation potential than embryonic stem cells, adult stem cells may be extracted from

a living human donor and thus do not have the same ethical concerns associated with their

use. A particular type of adult stem cells, called bone marrow stromal cells (or

mesenchymal stem cells) are the type studied in this thesis. They are discussed in more

detail in the next section. Figure 2 shows an example culture of human bone marrow

stromal cells.

Chapter 2. Culturing and Imaging 13

Figure 2 - Human bone marrow stromal cells - a source of adult stem cells (mesenchymal stem cells). Taken

with a 10x objective lens.

Induced pluripotent stem cells are adult cells that have been artificially induced

(genetically reprogrammed) to behave like embryonic stem cells. Although they are

pluripotent, it is not yet certain whether they may be considered to be the same as

embryonic stem cells for clinical applications. Like adult stem cells, they do not require the

destruction of an embryo, and hence have fewer ethical concerns over their use. However,

research into this type of stem cell is still very much in its infancy, as they were first

reported for mice in 2006 and humans in 2007.

2.1.1 Bone Marrow Stromal Cells

Bone marrow stromal cells (mesenchymal stem cells) are adult stem cells typically found in

bone marrow stroma (structural tissue) and other locations throughout the body [6]. They

have been shown to be able to differentiate into multiple cell lineages including bone,

cartilage, tendon, ligament, fat, muscle (including heart muscle) and neurons [6-8].

Because of this, they offer hope in the regeneration of tissue damaged by such conditions

as osteoarthritis, Parkinson's disease and for the repair of the heart following heart attack.

Although lacking the full range of differentiation possibilities of embryonic stem cells, adult

stem cells do not have the same ethical concerns nor many of the technical problems

currently associated with embryonic cells. When we consider the various types of adult

stem cells, an advantage of mesenchymal stem cells over other types of adult stem cells, is


that they are clinically accessible and thus relatively easy to obtain from a patient [9]. They

have immuno-privileged properties which could make them suitable for transplantation

from one person to another. Hence mesenchymal stem cells are a compelling potential

candidate for use in innovative new medical treatments.

At this point, it should be noted there is still quite some debate about how these cells

should be named [2]. This reflects the fact that there is still much that is unknown about

these cells. Perhaps the two most common names associated with these cells at present

are mesenchymal stem cells and bone marrow stromal cells (or just marrow stromal cells).

Without going into the specifics of the debate we will simply describe what we mean by

the use of those two terms. To all intents and purposes, we use the two terms

interchangeably in this thesis. Technically however, we would say that mesenchymal stem

cells are a subpopulation of cells that exist within the heterogenous population of bone

marrow stromal cells.

We now give a brief history of scientific research into these cells, followed in subsequent

sections by a description of how they are typically extracted and cultured and what

techniques are currently used by scientists to assay the quality of a culture.

Brief History

Definitive evidence for the existence of non-haemopoietic (non blood producing) stem cells

in the bone marrow stroma is generally credited to the work of Friedenstein and colleagues

(for example, see [10-12]) in the 1960s and 1970s. They showed that the bone marrow

contains a population of cells that rapidly adhere to the surface of tissue culture vessels

(unlike the majority of haemopoietic cells present) and had a fibroblast-like appearance. It

was found, for example, that just one of such cells could produce a colony that had the

potential to differentiate into bone and cartilage cells.

Although research into these cells continued in following decades it was not really until the

time of the isolation of human embryonic stem cells reported by Thomson et al [13] in the

journal Science in 1998, and the subsequent publishing of an article demonstrating the

multilineage potential of adult human mesenchymal stem cells by Pittenger et al [8], also in

Science, that worldwide interest mesenchymal stem cells greatly increased [2].


Despite the large amount of research into these cells since that time, there is still no single

antigenic marker that specifically identifies a mesenchymal stem cell [4, 9]. This means

that there is still no chemical fingerprint known that would allow scientists to uniquely and

objectively identify these cells, as distinct from other types of similar cells. Thus research

into mesenchymal stem cells is currently a very active area of research in the scientific

community, not only for the development of new medical treatments, but also in

understanding the basic biology of these cells, including their normal role in the body.

Extraction and Culturing

In order to culture cells they first need to be extracted from the donor. This requires that a

sample of the donor's bone marrow be taken, typically from the iliac crest (hip bone) or

femur (thigh bone). It is typically performed with a large needle (see Figure 3) while the

donor is under general anaesthesia. The sample is then generally filtered and centrifuged

to remove unwanted cells and other components and then incubated in a plastic culture

dish containing a liquid culture medium to provide the necessary nutrients to sustain the

cells. After a period of 24 hours or so, the bone marrow stromal cells will have attached

themselves to the bottom of the culture vessel. The culture media may then be tipped out

and the attached cells rinsed gently. This process removes the non-adherent cells (such as

haemopoietic stem cells) that may have existed in the original sample. Fresh culture media

is then placed in the culture dish and the remaining bone marrow stromal cells are allowed

to proliferate with the culture media being periodically replaced.

Figure 3 - Bone marrow extraction

Once these cells have grown in number (expanded) to completely fill the culture dish they

are said to have reached confluence. Once this happens, the culture media is removed and


the cells are rinsed and then treated with trypsin to detach them from the plastic surface.

They are then typically re-plated (passaged) into a larger culture dish, split into multiple

culture dishes, or frozen in liquid nitrogen for later use.

Assaying

At present, there are a variety of ways in which scientists assess the quality of a bone

marrow stromal cell culture. In the first instance, an expert observing the culture through

a microscope will often be able to form an opinion about the general health of the culture

based on their observations and experience in working with these cells over a substantial

period. Unfortunately, this is rather subjective and not necessarily reliable. A basic

objective test is to count the number of cells each time a culture is passaged and from that

work out the growth rate. It has been observed that faster growing cell cultures/colonies

are generally healthier and have more potential to differentiate into multiple cell lineages

[3, 4]. Along these lines, the colony forming unit assay [3], involves the plating of a portion

of the cells at very low densities and then counting the number of colonies formed after a

period of 10-14 days. The higher the number of colonies formed, the better the quality of

a sample. However, this is not a guarantee that a sample that forms a high number of

colonies will necessarily produce cells that can differentiate into all the desired cell

lineages.

To determine if a cell sample actually has the ability to differentiate into the desired cell

lineages, special tests are performed on cultures based on portions of the original bone

marrow stromal cell sample. For example, to test if the original sample can produce bone

cells (osteoblasts) a culture grown from a portion of the original sample has its normal

culture media replaced with an osteogenic culture media. The osteogenic culture media

contains chemicals that encourage the cells to differentiate into osteoblasts. The culture is

incubated in this new media for a period of time (e.g. several weeks). After this period has

passed, the cell culture is fixed and stained with a chemical that detects characteristic signs

associated with osteoblasts. This is a destructive test. Similar tests are conducted to

determine the adipogenic (fat cell forming) and chondrogenic (cartilage cell forming)

differentiation potential of the original sample.


The above does not cover all the tests that researchers may conduct on a given sample, but

should be sufficient to illustrate the amount of time and effort that is currently required to

ascertain the quality of just one sample of bone marrow stromal cells. It is also important

to note that these tests generally require the services of an experienced cell biologist.

With that in mind, we see that there is a genuine need for a quicker, easier and more cost-

effective way to assess the quality of a bone marrow stromal cell sample. Thus, with this in

mind, it is hoped that the field of image processing and pattern recognition can contribute

to overcoming this challenge, as now will be discussed.

2.2 Computerised Quality Assessment of Stem Cells

The use of computerised techniques for the processing and analysis of medical images is

not new. For example, in the early 1970s Miller [14] developed a complete image analysis

scheme for white blood cell (WBC) segmentation and recognition for specially stained

blood smears (reviewed by Kovalev et al [15]). Sternberg [16] in 1983 reported on the use

of image processing to assist scientists in screening for genetic mutations in infants based

on digitised images of 2-D electrophoretic gels. In 2003, Rodenacker and Bengtsson [17]

presented guidelines for image feature measurements, already based at that time on 25

years of prior image analysis work in the application of quantitative cytology.

More recently, we find a great deal of literature dealing with the analysis of cell and tissue

images, particularly in such areas as diagnostic pathology for the recognition of white

blood cells [15, 18], disease and cancer detection [19-21] as well as for the tracking of living

cells in time-lapse photography [22-25].

Nevertheless, despite decades of computerised medical image analysis and the great flood

of scientific interest in stem cells at present, there has been surprisingly little work done on

computerised quality assessment of such cells [26]. We now review the main approaches

taken by researchers so far in terms of the overall goal of quality assessment. We leave the

discussion of literature relevant to the individual stages of image segmentation, feature

extraction and pattern recognition to later chapters.


2.1.1. Culture Based Assessment

Jeffreys in [26] and summarised by Mangoubi, Jeffreys et al in [27] aim to assess the quality

of human embryonic stem cell colonies based on images taken using phase contrast. In

this case, a healthy colony means one that contains cells that are pluripotent. Based on

expert biologist advice, healthy colonies could be identified based on visual characteristics.

For example, healthy colonies have internal textures that are fine-grained and

homogenous (small closely packed cells) and have colony borders that are sharp and

circular. In contrast, unhealthy colonies had a larger-grained, loose, heterogeneous

internal texture (broader, loosely packed cells) with a less well defined and often less

circular colony border.

In order to make use of this textural information for quality assessment, interior and

border textural features were extracted using a wavelet based decomposition. Such a

technique provides numerical measures of both frequency and spatial information at

multiple resolutions. Features derived from the wavelet detail subband coefficients

obtained from the method were used in conjunction with a support vector machine for

training and classification.

The system was trained on images of colonies that have been graded by a biological expert

into one of four classes, with class 1 representing a fully undifferentiated colony to class 4

representing a fully differentiated colony. When presented with a previously unseen

colony, for which no prior quality information was known, the system correctly classified

the colony 68.8% of the time. It should be noted, that there were only 77 embryonic stem

cell colony images available for both training and testing. When we consider that the

grading system had four classes, this leaves very few images per class available for training,

given that a portion of the images had to be set aside for testing. It may well be that the

accuracy of the system presented would change with a larger dataset.

There are a few issues with the above approach: (1) the training set is based on grading by

a human expert and hence is still subjective. (2) the system still requires a manual

segmentation of the colony interior and border areas and thus still requires a certain

expertise in the location of such areas. Automation of this aspect was a recognised part of

this project that, at the time of publication, still required development.


In terms of applicability to our project, we note that the subject of Jeffreys and Mangoubi's

work are embryonic stem cells which are different to adult stem cells, and specifically,

bone marrow stromal cells. For example, bone marrow stromal cells do not have a clearly

defined colony border, are not as closely packed together, and have a different

morphology (compare Figure 1 and Figure 2). Additionally, as we shall see later, bone

marrow stromal cells taken from different patients can have a different morphological

appearance despite all being relatively healthy. Nevertheless, we see that textural

information based on a collective group of cells, in this case a colony, can be used for

quality assessment without the need to segment individual cells. This is an important

consideration for our project, due to the great difficulty in segmenting individual cells

especially without the benefit of time series information or histological stains.

2.2.1 Single Cell Based Assessment

Extending the work described above [26] [27], Mangoubi et al [28] turn their attention to

images of individual cell nuclei, based on the observation that chromatin (a component of

cell nuclei) in human embryonic stem cells becomes more granular as the cell undergoes

differentiation. However, this technique requires the use of fluorescent markers and a

spinning disc microscope that is able to capture four dimensional movies (3-D time series).

It would also appear to be rather labour intensive since it requires individual nuclei to be

located and monitored making its use for assessing the health of a whole culture

somewhat doubtful, at least for our application.

Shir et al [29] aim to distinguish between healthy and apoptotic (dying) human

mesenchymal stem cells on an individual cell basis. This is based on the observation that

the envelope surrounding the nucleus becomes considerably deformed during apoptosis

compared to the round and flat shape of a normal healthy cell. They do this based on cross

sections of confocal 3-D images of cell nuclei taken using a fluorescent marker. The pixel

intensity levels of these cross-sectional images are fed directly into a support vector

machine classifier.

In this study, the apoptotic cells representing the unhealthy class come from normal cells

that have been induced into apoptosis by the deliberate addition of a chemical. Using a

training set of 2000 images containing both normal cell nuclei and apoptotic cell nuclei


(imaged six hours after treatment with the apoptosis inducing chemical) and a test set of

1040 previously unseen images, they report a classification accuracy of 97% based on a

support vector machine utilising a radial basis function kernel. An advantage of this work,

over Mangoubi's [28] later work with cell nuclei, is that the classes used for training are

derived by an objective means (chemically induced apoptosis) rather than based on more

subjective expert opinion. However, for much the same reasons as Mangoubi's work, we

do not find these methods applicable for our task.

2.2.2 Cell Tracking Based Assessment

Pan, Li, Kanade et al [30], [23], [31], [32] describe the development of a system to track

individual cells (including stem cells) in culture over time. This system uses phase contrast

images taken at 5 minute intervals as input. The system aims to provide complete

spatiotemporal histories of each individual cell, including cell divisions (mitosis). This

would allow the gathering of such information as cell family trees (lineage maps), division

times, motility and death. This information would be able to help researchers understand

more about stem cells themselves as well as the effect of differing culture conditions,

including the effect of drugs on cells.

With this system, researchers need to deduce the quality of the cells from the data

provided, so while undoubtedly useful for investigational purposes, it is not directly

providing an assessment of culture quality. The system also, by its very nature, requires

ongoing imaging at regular (5 minute) intervals and thus would require more sophisticated

and expensive lab equipment , and even more so if multiple cultures are to be monitored.

Because of these reasons, we do not find this work applicable to our project.

2.3 Data Collection

Having given a brief introduction to stem cells as well as the current literature relating to

their quality assessment, we turn our attention now to the culturing and imaging of bone

marrow stromal cells required for use in our project.


Overview of Proposed System

In order to understand the data collection needs for our project, we present a brief

overview of the quality assessment system proposed in this thesis. This is illustrated in a

very simple form in Figure 4 below:

Figure 4 - Proposed quality assessment system (greatly simplified)

What we would ideally like is a system that can be given a bone marrow stromal cell image

as its input and for it to automatically provide us with an indication of the culture quality

(e.g. healthy/unhealthy). The advantage of such a system over the current methods of

quality assessment is that it is quick, objective, does not require an expert cell biologist to

operate, and is inexpensive to use because it only requires phase contrast culture images.

However in order for the above system to be possible, it must first be trained to recognise

healthy and unhealthy bone marrow stromal cell cultures. For this, we need a

representative quantity of images of cell cultures for which we know the quality. In

particular, we need representative quantities of images from cultures that are both good

(healthy) and bad (unhealthy).

When one ponders the image in Figure 3, we see why it may be difficult to obtain a ready

supply of donors willing to donate their bone marrow. But this would seem to be

necessary in order to supply us with a sufficiently large natural variation in bone marrow

stromal cell culture qualities, such that through random chance alone, we would find

sufficient examples of high and low quality samples that could be used to train and test

our system. In order to deal with this issue of limited donor supply, as well as the need to

Output

quality assessment

unhealthy/healthy

Input

BMSC cell images

Quality Assessment

System


have samples clearly representing both healthy and unhealthy cultures, it was decided in

discussion with an expert cell biologist, to grow multiple cultures from a limited supply of

pre-existing frozen BMSC samples and artificially induce half of the cultures to be

unhealthy. Further discussion and details of this will be provided further on in section

2.3.3. For now, we will look briefly at the equipment used for capturing cell culture

images, as well as what was gleaned from earlier pilot cell culture imaging experiments.

This provides the context for the data collection approach taken in our main experiment

(section 2.3.3).

2.3.1 Equipment

This section describes the salient features of key equipment used in the culturing process

and for capturing our image data.

Microscope

The microscope used was a Nikon Eclipse TS100-F (trinocular) inverted phase contrast

microscope (Figure 5). This microscope is especially designed to allow for the attachment

of a camera, without interfering with the normal use of the microscope eyepiece. A knob

just below the eyepiece is used to switch between viewing the specimen through the

eyepiece or through the camera.

Figure 5 - Inverted microscope. A six well plate (culture dish) can be seen on the microscope's stage. The PC

used to control the attached camera is shown on the right.


The phase contrast feature of the microscope is very important when dealing with living

cells. The reason is that living cells appear almost transparent and have very little contrast

compared to the background when viewed through a normal (non-phase contrast)

microscope. The phase contrast system overcomes this problem by passing light through

the sample at different angles such that the differing refractive indices of the cells and

background produce corresponding different changes in phase of the light waves travelling

through the different sections. These different changes in phase result in a much higher

contrast between the cells and the background, making the cells much easier to see.

The microscope used for this project is equipped with three objective lenses, with

magnifications of 4x (non phase contrast), 10x and 20x. As will be discussed in section

2.3.2, the 10x objective is found to be most useful for our application.

Camera

A PixeLINK PL-B686CU 6.6 mega pixel colour microscopy camera was used for image

capture. This camera is connected to a PC via a USB cable and is completely controlled via

software on the PC. Figure 6 shows the camera (left) and a screen shot of the software

used to control it (right). It is a colour camera able to take images at a maximum resolution

of 2208 x 3000 pixels. A key feature of this camera (and its controlling software) is that it

allows a live preview to be seen on the PC screen. This is very important in being able to

obtain a sharp focus when taking images, since the effects of changing the microscope

focus control is immediately observable on screen. Furthermore, once an image is taken,

the captured shot is shown immediately as a confirmation that a good shot has been taken.

Note: What appears to be sharply in focus when looking through the microscope's

binocular eyepiece is not in focus when switching to the camera, as they are split into

different optical pathways. Therefore, the microscope's eyepiece cannot be used

effectively for focussing the image for the camera. This makes the live preview feature of

the camera/software combination indispensable.

Another useful feature, which is a consequence of the camera being PC controlled, is that

the shutter is controlled remotely via the computer's keyboard and not a button on the

camera itself. This is important to ensure that the camera (and the cell culture being

imaged) is not inadvertently disturbed when a shot is taken, resulting in a blurred image.


Figure 6 - PixeLINK digital camera (left) and PixeLINK Capture SE controlling software (right).

In addition to providing a live preview and shutter operation, the software allows camera

settings, such as the exposure to be precisely controlled and saved. This allows for the

settings to be restored at the beginning of each photographic session. This is a useful

feature since other researchers may use the device between sessions and therefore change

settings. Being able to restore the settings used helps achieve consistency in images taken

between sessions.

Also, the ability to be able to instantly name each picture and save it in an appropriate

folder on the PC is very helpful when taking a large set of images that fall into different

categories (e.g. day 1, patient A, 0.2% FCS, well 3). This greatly reduces the risk of an

image being incorrectly categorised, especially when one can double check that each folder

contains the expected number of images.

The final feature worth mentioning here is that an image histogram appears in real time

below the image preview. This allows the exposure to be adjusted to ensure that the

image is not overly saturated and that the majority of image gray levels are within the mid

range. This again helps ensure a more consistent set of images.

2.3.2 Pilot Experiments

Prior to conducting our main data collection (described below in section 2.3.3), a number

of pilot experiments were undertaken to provide the author with experience in culturing

cells. It was also to provide some initial images that could be used for testing the

equipment and for preliminary algorithm design (such as for segmentation), as well as to

decide upon which configuration of equipment would be most suitable for the main


experiment that would follow. During these initial experiments, a number of lessons were

learned which are now discussed below.

Field of View (Selection of Objective Lens Magnification)

The objective lens magnifications available are 4x, 10x and 20x. Figure 7 shows the same

cell culture location viewed under the three different magnifications. The 4x objective was

not a phase contrast lens and was not suitable for living cell images, although it had a very

wide field of view. Conversely, the 20x lens although a phase contrast objective, did not

show a sufficient number of cells per image and hence would have been impractical (too

labour intensive) to take sufficient images manually to be representative of the cells

contained within the well. The 10x objective was thus chosen as the most suitable of those

available for further use, providing a field of view of approximately 0.7 mm by 1 mm.

4x (1.8mm x 2.5mm) 10x (0.72mm x 1mm) 20x (0.36mm x 0.5mm)

Figure 7 - Field of view by objective lens magnification. The white arrow points to the same cell in each

image. Note that in the 4x image (not a phase contrast lens), the spread out cell is almost impossible to see.

Cell changes over 24 hours

It was intended to monitor the progress of multiple cell cultures over a period of six days

by taking images of each culture every 24 hours. Originally, it was thought appropriate to

image the same locations in each culture over the successive days. This was done during

pilot experiments by using a grid marked on the underside of the culture vessels. However,

after conducting these pilot experiments, it was considered not only unnecessary, but also

potentially detrimental to our experiment for the reasons outlined below.


Firstly, after 24 hours, the same location in a culture is unrecognisable based on the cells

present - see Figure 8 below. This figure shows the same location in a culture imaged over

three successive days. From looking at those three images, it is clear that individual cells

cannot be tracked over that time frame. This is little wonder when we consider that

researchers aiming to track individual cells over time used timeframes of just 3 to 10

minutes [23-25]. For this reason there was seen to be little benefit in deliberately imaging

the same precise locations over successive days.

Day 1 Day 2 Day 3

Figure 8 - Same culture dish location imaged at 24hr intervals over 3 days.

Secondly, any permanent marks within the chosen culture locations (e.g. scratches, dried

condensation patterns, etc) could potentially be wrongly associated by the pattern

recognition system with the cells being imaged. This could result in the system 'cheating'

on test results if it learns that the presence of a certain noise pattern is associated with a

particular outcome, rather than making its decision solely on cell related features. This

would be an especially important consideration if using training data composed of images

from more than one day.

Thirdly, should a location become badly scratched or have no cells, then the sequence of

images for that location would potentially be wasted.

A final consideration is that aligning images to a marked grid or some form of jig setup adds

an additional burden to the data collection process, since for each image the user must

make sure the culture dish is correctly aligned in the exact new position. Even when using


a PC controlled automatic microscope stage, other researchers [33] have found that

images still will not be aligned to exactly the same place each time, and will still require

some form of registration mechanism to correctly align successive images. Given that

registration of multiple images (to locate the common portions) causes a loss of useable

area, the amount of area lost when manually aligning image positions on successive days

would be unacceptable for the nature of this type of application.

2.3.3 Method - Main Experiment

In this section we discuss the culturing and imaging process used to create our main set of

data. This is the data, in the form of images of cultures that are known quantifiably to be

healthy or unhealthy, that is used to train and test our BMSC quality assessment system.

Three BMSC samples from different patients were available for use having previously been

obtained and cryopreserved. Under the direction of an expert cell biologist (refer section

1.6), it was decided that six cultures would be grown from each patient's sample, and

monitored over six days by taking nine images of each culture per day. Three of the six

cultures would be grown in culture medium with the normal amount of nutrients - 10%

fetal calf serum (FCS), while the other three would be grown in culture medium with only a

trace amount of nutrients - 0.2% FCS. This would induce half the samples to be unhealthy

and exhibit very poor growth. This means that there would be three healthy cultures and

three unhealthy cultures for each of the three patient samples, giving a total of 18 separate

cultures. It was also decided to verify the actual growth rates achieved by using the nine

images taken of each culture every day over the period of six days. From those images, the

average number of cells per image in each culture could be calculated for each day of

culture growth. These images (in particular, those taken from the first day of culture)

would also be the basis of our image database used for training and testing our system.

In the following sections we discuss the culturing, imaging and validation of the bone

marrow stromal cell samples.

Cell Cultures

Bone marrow stromal cells from three different patients (referred to as A, B and C) were

revised from frozen stock and cultured in three separate culture flasks and allowed to grow


to sufficient numbers in a standard culture medium - Dulbecco’s modified Eagle's medium

(DMEM) containing 10% FCS, 10 U/ml penicillin G and 10 mg/ml streptomycin. They were

subsequently plated in 6 well plates (Greiner Cat. No. 657 160) at a concentration of

approximately 45,000 cells per well and allowed to attach overnight. The media was then

removed using a pipette and the wells rinsed twice with PBS (a biologically neutral

solution). Media (DMEM + penicillin/streptomycin) with concentrations of 10% and 0.2%

FCS were then placed into the wells and the cells returned to the incubator and allowed to

grow over six days with no further changes of media. It was during this final six day culture

period that the images were taken at intervals of approximately 24 hours.

Taking Images

The first set of images (Day 1) was taken approximately 10 hours after the media was

changed, with subsequent image sets (Day 2 to Day 6) taken at intervals of approximately

24 hours thereafter. Based on findings from our pilot experiments (section 2.3.2), the

images were taken using the microscope's 10x phase contrast objective lens.

Since the previous pilot experiments had also shown that the cells move/change so much

in 24 hours that there was no correspondence for images taken in the same location within

a culture, it was decided there was no need to take images from exactly the same location

on successive days. The only requirement was that images taken from each culture on the

same day do not overlap and represent areas from the whole well. To this end, an

approximate 3 x 3 arrangement was used as shown below:

Figure 9 - General order and location of image capture for each culture well (positions are approximate only)

The six well plate was positioned manually for each shot in order shown above with the

indicated positions approximate only. For each shot, the focus was adjusted so that a

1 2 3

4 5 6

7 8 9


sharp image appeared on the screen. Where images had no cells at all or contained

predominately noise (such as a large mark on the outside of the plate), the image would be

manually moved to an adjacent location where a normal shot could be taken. However,

this was a very rare occurrence.

Once an image was taken, it was saved into an appropriate folder and numbered

sequentially from 1 to 9 depending upon its order within the well. The folders were

hierarchically arranged in the following tree structure:

Day (1/2/3/4/5/6)

└── Patient (A/B/C)

└── FCS (10pc/0_2pc)

└── Well (1/2/3)

so that each folder contained 9 images. This arrangement ensured images were not

confused and mistakenly assigned to the wrong category. Additionally, each image was

named in a similar manner. For example the image DAY01_A_002_1_05.bmp represents

an image taken on day 1, from patient A, 0.2% FCS, well number 1, image 5 (of 9). This

image would be stored in the folder path \Day1\A\0_2pc\1\.

Images were taken at the camera's maximum resolution of 2208 x 3000 pixels and saved in

the uncompressed 24 bit windows bitmap format (.bmp) [34] to ensure no data loss from

compression and for speed of manipulation.

Given that there were 18 wells (cultures) and for each well nine images were taken, that

resulted in 162 images taken per day over six days, a total of 972 images.

Issues

Since the cells spend most of their time inside an incubator at 37 degrees Celsius, when

they are removed and placed on a microscope stage in an air-conditioned laboratory (~22

degrees Celsius), condensation soon appears on the upper surface (in this case, the lid) of

the six well plate - see Figure 10. This causes a serious degradation of image quality since

the condensation lies within the optical pathway. When using a culture flask (the case

during pilot experiments), the flask can simply be carefully inverted so that the culture

media can wash over the top surface of the flask, effectively removing the condensation.


However, when using a six well plate where the top surface is the removable lid, this

process cannot be used. In this case, the condensation was removed by placing the plate

in a sterile ventilated hood (Figure 11) and periodically removing the lid, allowing the

condensation on it to evaporate (taking approximately 30 secs) and then replacing the lid.

This was done until no further build up of condensation occurred. Any removal of the lid

was conducted in the sterile environment of the hood to prevent contamination. Once the

condensation had disappeared, the plate could then be placed on the microscope stage

and images taken. After taking images from about half the wells on the plate, further build

up of condensation would appear due to the heat of the microscope lamp. This was

remedied by placing the plate back in the hood and removing the lid to allow the

evaporation of the media once more. This condensation problem could be alleviated by

the use of a portable incubator designed for use on a microscope stage, however no such

device was available for this research project.

Figure 10 - Six well plate showing condensation

Figure 11 - Laminar flow hood (with 6 well plate

inside)

Validation of Cell Outcomes

While it would be expected that the cells with 0.2% FCS would not proliferate well, if at all,

it was desirable to quantify the actual health of the cells at the end of the six day period.

This is important as even among different groups of healthy cells (e.g. from different

patients), the growth rates achieved can be different. It could also be possible that even

for a group of cells placed in 10% FCS, they may still not grow if they were originally

unhealthy to start with, or become unhealthy due to some unexpected intrinsic or extrinsic

factor. So we needed a way to quantify the actual growth pattern achieved and thus gain a

quantification of the health of the cells from each well.


To make such a quantification, the number of cells in each image was manually counted

(using the ImageJ software [35] and its cell counter plug-in [36]), for each well for each of

the six days of growth. Since there were 9 images per well, this gave 9 cell counts that

could be averaged to give an approximation of the number of cells per image per well per

day. These average counts were then used to plot graphs showing the growth in each well

over the six days. The growth rates achieved are shown in Figure 12 below:

Figure 12 - Cell culture growth rates for cultures from patients A, B and C. The three solid red lines in each

graph represent the three wells that had 10% FCS, the three dotted blue lines represent the three wells that

had 0.2% FCS. The vertical axis is the average number of cells/image (over 9 images) and the horizontal axis

is the day in culture. Note: The horizontal bars on each data point represent one standard deviation above

and below the average count.

The graphs show that cells from all three patients grew better (as expected) when cultured

in 10% FCS when compared to the same cells cultured in 0.2% FCS. Note that the initial

slight rise in the number of cells in 0.2% FCS (blue lines) for patients A and B is likely due to

the fact that all cells were initially plated in 10% FCS while they were allowed to attach,

allowing some nutrients to be stored up and used after the media had been changed to

0.2%. The effectiveness of the cells' ability to do this was possibly related to overall quality

of the original cells. For example, cells from patient B achieved a very high growth rate in

10% FCS were presumably healthier (and hence managed some growth in 0.2% FCS)

whereas the cells from patient C achieved significantly lower growth even in 10% FCS and

therefore, not surprisingly, achieved no growth in 0.2% FCS.

Aside from confirming that the amount of FCS did indeed affect cell growth (as expected),

another important point is that the average cells/image on day 1 show that counts are in

the same range, irrespective of the amount of FCS in the culture for each patient. Hence,

0

50

100

150

200

250

1 2 3 4 5 6

Ave

rag

e C

ells

/ I

ma

ge

Culture Days

Patient A

0

50

100

150

200

250

1 2 3 4 5 6

Culture Days

Patient B

0

50

100

150

200

250

1 2 3 4 5 6

Culture Days

Patient C

32

at day 1, cell count is not an indicator of the health state of the culture. Therefore, any

predictions of culture health by our system on day 1 will be due to factors other than the

number of cells.

2.4 Example Images

In this section we will show some examples of the images collected to give the reader a

feel for the similarities and differences among the images when typical examples are

examined.

2.4.1 Cells from different p

The figure below (Figure 13) show cells from the three different patients taken

All three images come from cultures with 10% FCS,

considered 'healthy'. Different cell morphologies can be observed. For example, cells from

patient A, although somewhat similar in appearance to patient B, tend to be larger in size.

Cells from patient C are more distinctive, generally having a

Because of these differences in cell morphology, our system must be able to distinguish

between healthy (i.e. those cultured in 10% FCS) and unhealthy (0.2% FCS) irrespective of

which patient they come from. This also suggests that morpholog

not be the best indicator of cell health since there is considerable variation in morphology

among healthy cells from different people.

Patient A

Figure 13 - Example of different cell morphologies from three different patients. All images are taken from

the healthy class (i.e. 10% FCS) on day 1

spread out. Note: The contrast and brigh

morphology more apparent (especially for Patient C) when viewing on the printed page.

Chapter 2. Culturing and Imaging





patients

show cells from the three different patients taken

All three images come from cultures with 10% FCS, so the cells in these images can be

nsidered 'healthy'. Different cell morphologies can be observed. For example, cells from


Cells from patient C are more distinctive, generally having a more spread out shape.



which patient they come from. This also suggests that morphological measurements may


among healthy cells from different people.

Patient B Patient C

Example of different cell morphologies from three different patients. All images are taken from

on day 1. Patient A - larger, longer, Patient B - more compact, Patient C

. Note: The contrast and brightness in these images have been manually altered to make the cell

morphology more apparent (especially for Patient C) when viewing on the printed page.

Culturing and Imaging





show cells from the three different patients taken on day 1.

so the cells in these images can be

nsidered 'healthy'. Different cell morphologies can be observed. For example, cells from


d out shape.



ical measurements may


Patient C

Example of different cell morphologies from three different patients. All images are taken from

Patient C - more

tness in these images have been manually altered to make the cell


2.4.2 Cells at day 1 - 10% FCS versus 0.2% FCS

The following images show cells from the three different patients A, B and C on day 1 for

those in 0.2% FCS (top row) and 10% FCS (bottom row). The images were taken

approximately 10 hours after the amounts of FCS in the media had been changed.

Although the images of cells at 0.2% FCS may possibly look a little more emaciated,

especially for cells from patient A and B, the difference is difficult to quantify. In practice,

when presented with unlabeled images, it is very difficult to pick the healthy cells from the

unhealthy cells. This was even the case for human experts, as we shall see when we

discuss our results in Chapter 6.

Patient A - Day 1 - 0.2% FCS Patient B - Day 1 - 0.2% FCS Patient C - Day 1 - 0.2% FCS

Patient A - Day 1 - 10% FCS Patient B - Day 1 - 10% FCS Patient C - Day 1 - 10% FCS

Figure 14 - Comparison of cell images for 0.2% FCS (top row) and 10% FCS (bottom row) at Day 1 for patients

A,B and C


2.4.3 Sparse versus dense

The figure below shows two images taken from the same well on day 1. The cells are from

patient B in media of 10% FCS. This illustrates the variation that can appear in the number

of cells viewable in a given image even when taken from the same well at the same time

(the images were taken within minutes of each other).

Figure 15 - Variation in cells numbers present in two concurrent images taken from different locations in the

same culture well. Culture is that of patient B taken on day 1 containing 10% FCS.

2.4.4 Progression over 6 days

The set of images displayed in Figure 16 (see over) show the progression of two cell

cultures from Patient B from day 1 to day 6. Cells on the left are from a well with 0.2% FCS

while those on the right are from a well with 10% FCS. Each image in the figure is a

montage, made from all nine images taken from the well for each day. The increase in the

number of cells for those in 10% FCS is clearly seen over the six days, when compared to

those in only 0.2% FCS. This shows in image form what is indicated by the middle graph

(Patient B) in Figure 12.

2.5 Summary

In the first part of this chapter we have provided a brief introduction to the topic of stem

cells and, more specifically, bone marrow stromal cells. We have looked at how these cells

are generally extracted and cultured in the laboratory as well as the current methods used


by scientists to assay their quality. Following that, we reviewed current research in

automated computerised quality assessment of stem cells. In the second part of this

chapter we looked at how image data necessary for the training and testing of our

proposed BMSC quality assessment system was collected and verified. We concluded the

chapter with a brief qualitative examination of some typical culture images obtained.

Having now described how our validated data was obtained, we turn our attention to the

design of the image processing and pattern recognition system corresponding to the

contents of the blue box illustrated back in Figure 4. The first task of the system when

given an image is to locate which regions of the image contains cells, and which regions

contain background. This is the process of segmentation and it is the subject of the

following chapter.

36

0.2% FCS

Figure 16 - Cell culture growth over six days for 0.2% FCS (left column) and 10% FCS (right column). Each

picture is a montage of the nine images taken from each well. Cells are from patient B.

images has been manually adjusted for better viewing on the printed page.


10% FCS

Day 1

Day 2

Day 3

Cell culture growth over six days for 0.2% FCS (left column) and 10% FCS (right column). Each

picture is a montage of the nine images taken from each well. Cells are from patient B. The contrast of these

images has been manually adjusted for better viewing on the printed page. Figure is continued on next page.


Cell culture growth over six days for 0.2% FCS (left column) and 10% FCS (right column). Each

The contrast of these

Figure is continued on next page.


0.2% FCS


0.2% FCS 10% FCS

Day 4

Day 5

Day 6

37

10% FCS



39

Chapter 3

Segmentation of Phase Contrast

Images of Living Bone Marrow

Stromal Cells

In this chapter we look at the first stage of our proposed system. This stage deals with the

image processing function of our system, and consists of preprocessing and segmentation.

In brief, preprocessing involves processing images to make them more efficient to work

with and more consistent, in terms of brightness and contrast level, from one image to the

next. Segmentation involves locating objects of interest, such as cell regions, within each

image. We begin with a brief overview of our whole system, before developing and

evaluating specific algorithms for preprocessing and segmentation.

3.1 System Overview

In Chapter 2 we briefly introduced our proposed computerised quality assessment system.

To recap, we aim to produce the system shown (in a simplified form) in Figure 17. Ideally,

the system represented by the blue box, can be presented with bone marrow stromal cell

culture images and, in response, indicate whether the culture is healthy or unhealthy. Of

course, in order for the system to provide such an assessment, it must first itself be trained

by presenting it with examples of both healthy and unhealthy culture images. This is

somewhat analogous to a human non-expert (i.e. a person with no a priori knowledge of

cell cultures) being shown a collection of culture images containing labelled examples of

40 Chapter 3. Segmentation

both healthy and unhealthy cultures. From this the human observer will naturally look for

features that appear to differentiate the culture images based on the given health

classification. After the human has been shown a representative sample of images from

both categories, they can then be presented with previously unseen images for which the

classification is unknown, and asked to classify such images. Using the experience gained

from their observations of the training culture images, they will attempt to classify the new

images.

Figure 17 - Overview of proposed quality assessment system

Continuing with the above analogy, one of the first things a human observer will do when

looking at an image (for both training and testing), is to pick out the cells or cell regions

from the background. Trying to get our computerised system to do the same thing is the

process known as image segmentation. The second task, that of looking for discriminative

features among labelled examples, is referred to as feature extraction. Finally, learning to

use such features to correctly classify previously unseen images, following training on a

labelled set, falls under the title of pattern recognition. Thus our quality assessment

system can be represented in more detail, as shown in Figure 18.

With reference to this figure, we note that the input to the image processing block (which

includes preprocessing and segmentation) is an image, while the output is marked regions

within the image. The feature extraction block takes as its input, the marked regions

located by the segmentation process. The feature extraction block takes measurements

from the marked regions and outputs these as a list, termed a feature vector. One feature

Output

quality assessment

unhealthy/healthy

Input

BMSC cell images

Quality Assessment

System

Chapter 3. Segmentation 41

vector is produced for each image. This feature vector is the input to the pattern

recognition stage. In the training phase, the pattern recognition stage is also provided the

classification (e.g. healthy/unhealthy) as input. During this phase, the pattern recognition

stage attempts to determine what feature values are associated with healthy cultures and

what values are associated with unhealthy cultures. During the testing phase, the pattern

recognition stage is only provided with the feature vector and must estimate the

classification, based on its past observations.

Figure 18 - Expanded overview of quality assessment system

Having now given an overview of the major functional components of our system, we

spend the remainder of this chapter looking at the first stage, namely, preprocessing and

segmentation. The remaining two stages, feature extraction and pattern recognition, are

the subject of Chapter 4.

3.2 Preprocessing

The images used for the development and evaluation of our segmentation system were

taken during pilot experiments, prior to the main data collection that is described in

Chapter 2. The culture images used here are also samples of human bone marrow stromal

Image

Processing

Feature

Extraction

Pattern

Recognition

��1�2⋮��

� ℎ�� ℎ��ℎ�� ℎ��

Input

Image

Input

Classification

(training)

� ℎ�� ℎ��ℎ�� ℎ��

Output

Classification

(testing)


cells. They were taken from two different adult patients and are labelled X and Y, to

differentiate them from the samples of patients A, B and C described in Chapter 2. The

cells of sample X (Figure 19a) were considered by expert opinion (refer section 1.6) to be of

a higher quality than those of sample Y (Figure 19b), as they had a higher proliferation rate.

a

b

Figure 19 - Phase contrast images of BMSCs taken at 10x magnification – note the irregular cell morphology

and bright halo around many of the cell borders. The images are typical of those taken from (a) sample X and

(b) sample Y

The microscope and camera setup used was the same as that described in Chapter 2. The

microscope’s 10x phase contrast objective lens was used.

Original images were taken at the camera’s maximum resolution of 2208x3000 pixels.

Images were then converted to 8-bit greyscale and rescaled to a size of 552x750 (using

ImageJ software version 1.40e [35]) to allow for a more manageable speed of processing

and more economical use of storage space. Each image was then normalised, using the

normalisation schemes described below in section 3.2.1. To increase the amount of data

available, each rescaled image was then split into four tiles resulting in images with a final

size of 276x375. Each pixel in these tile images corresponded to an area of 1.3µm x 1.3µm.

3.2.1 Image Normalisation

Image normalisation attempts to reduce the effect of variation in the input images due to

differing lighting, camera exposure and other settings. This is important as we would like

to ensure that differences in our culture images, unrelated to the cells themselves, are


minimised. We would naturally expect that doing so would result in a more robust and

accurate system. The results of testing this hypothesis are presented in section 3.5.1,

where we compare the use of two common normalisation methods on recognition

accuracy, compared to using no normalisation at all.

The two methods of normalisation we investigated were contrast stretching and histogram

equalisation [37], which are now described below.

Contrast Stretching

Contrast stretching applies a linear transformation to the input image so that the intensity

histogram is stretched across the full range of possible pixel intensity values (e.g. 0-255 for

the 8-bit gray level images used here). In order to prevent a relatively small number of

outlier pixel intensities adversely affecting the result, it is normal to allow a certain

percentage (say, 0.5%-3%) of pixels to become saturated (set to 0 or 255). For example,

allowing a saturation of 2% would imply that pixels with an intensity in the 1st percentile of

the histogram would be set to 0, pixels in the 99th percentile would be set to 255, and the

pixels in between would be linearly stretched between the intensities of 1 and 254.

Histogram Equalisation

Histogram equalisation, applies a non-linear transformation to the image histogram, so

that the output image histogram is flattened and approaches a uniform distribution. An

advantage of this method over contrast stretching is that it does not require selection of a

parameter value.

3.3 Segmentation

Segmentation is the process of separating the relevant objects in an image from the

background. In our case, this refers to the separation of cells or cell regions from the

background. This is a necessary precursor to the extraction of features for use in our later

pattern recognition stage. Segmentation is a crucial step since the success of pattern

recognition techniques in image processing systems is known to be very dependent upon

the effectiveness of the preceding segmentation [38].

However, segmentation of living cell images presents a number of challenges. Firstly, the

images must be taken in a non-invasive manner so that the cells may continue to grow


unharmed. This precludes the use of fluorescent markers and histological stains that can

make segmentation much simpler. Furthermore, the nature of images taken using phase

contrast renders standard segmentation approaches ineffective. For example, the bright

halo that appears around many cell borders (Figure 20a) creates false edges when standard

edge detection methods are applied. Furthermore, the background gray levels are a subset

of the cell gray levels and so intensity thresholding cannot be used (Figure 20b).

a

b

Figure 20 - Challenges associated with phase contrast imaging. Image (a) shows the characteristic halo on an

image of bone marrow stromal cells. Image (b) shows a 3-D representation of the same image, with the

vertical axis representing the intensity (gray level) of each pixel.

A variance based method of segmenting living cells has been proposed by Wu et. al. [39],

however being designed for normal bright field (non phase contrast) images, it does not

deal adequately with halo regions. Nevertheless, the variance based approach is still useful

in locating the approximate cell regions, and has been employed in a modified form in our

research. A probabilistic living cell segmentation model [24] has been developed for

segmenting hematopoietic stem cells under phase contrast, however the method takes

advantage of the regular circular morphology of such cells. Unfortunately, the irregular

morphology of BMSCs renders this approach ineffective. A method for segmenting

embryonic and neural stem cells has been proposed in [40] that models such cells as a

centre blob with attached curvilinear processes, which are detected by multi-scale Laplace-

of-Gaussian filters and a multi-scale curvilinear detector, respectively, and combined to

locate individual cells. Once again this method makes assumptions about the morphology

of the cells that may not hold true in the case of BMSCs. Debeir et. al. [41] have recently

Halo

y

x

Inte

nsi

ty


suggested the use of marked weakened watersheds, a method used to deal with the

problem of oversegmentation typical of the watershed approach. This method appears to

deal well with the irregular morphology and phase contrast halo, however it requires that

cell centroids be initially marked. Other applications (for example [30, 31, 42, 43]) designed

to track the movement of living cells over multiple phase contrast image frames also

involve segmentation steps. However, the techniques employed are concerned primarily

with tracking cell centroids rather than producing an accurate contour of the cell regions

themselves and are thus not considered suitable for the present application.

The segmentation approach we have developed involves two stages. Firstly, approximate

cell regions that include the halo areas are located in a manner similar to that proposed in

[39]. We term this step rough segmentation. The second stage, we term refined

segmentation, refines this approximate cell region by excluding the halo and thus providing

a much closer fit to the real cell contour. These two segmentation stages are now

described in the following sections.

3.3.1 Rough Segmentation

A rough segmentation was performed by taking an input image (Figure 21a) and calculating

the standard deviation at each pixel, computed over a circular neighbourhood, or mask,

centred on the pixel. A mask size of radius 3, equivalent to a diameter of approximately

9µm, was found empirically to provide the most suitable result. The precise shape of the

mask is shown in Figure 22b. A smaller mask resulted in more cell areas with little intensity

variation being incorrectly considered background. A larger mask resulted in more

background areas being included in the segmented cell area, as well as a noticeably

reduced performance in the subsequent refined segmentation process. The result of

applying this standard deviation mask can be seen in Figure 21b.

In order to separate cell regions from the background, the standard deviation image was

automatically thresholded using minimum error thresholding [44]. The result of this

process is the binary image shown in Figure 21c. It can be observed from this image that

foreground objects include not only cells, but also small particles, which can be seen as

small white blobs. Small holes can also be seen in the centre of some of the cell regions in

Figure 21c. These holes are caused by contiguous dark cell regions within the input image,

46

where the standard deviation may be very low or even zero, causing them to be

considered part of the background. Finally, image areas that contain the white halo are

considered by this algorithm to be part of the foreground since they have a high local

intensity variation.

a

b

d

e

Figure 21 - Rough segmentation (a) normalised image (b) standard deviation filter applied (c) thresholded

image (d) small objects removed (e) small holes removed (f) roughly segmented region superimposed on

input image

a

b

Figure 22 - Filter mask sizes (a) radius 1, (b) radius 3, and (c) radius 4

Chapter 3. Segmentation


of the background. Finally, image areas that contain the white halo are


b

c

e

f

Rough segmentation (a) normalised image (b) standard deviation filter applied (c) thresholded


b

c

Filter mask sizes (a) radius 1, (b) radius 3, and (c) radius 4

A

B

Segmentation


of the background. Finally, image areas that contain the white halo are


Rough segmentation (a) normalised image (b) standard deviation filter applied (c) thresholded



To clean up the image, the small particles (area ≤ 1024 pixels / ~1700 µm2) and holes (area

≤ 256 pixels / ~430µm2) were removed using the 'Analyse Particles' function of ImageJ [35].

This function uses region based segmentation [37] to separate a binary image into separate

regions representing a set of non-touching objects. An object, in this case represents a

collection of 4-way connected foreground pixels (i.e. white pixels). Once objects are

segmented in this way, the ImageJ algorithm can remove them based on their area, that is,

the number of pixels in each object. First this algorithm is applied to remove small

particles, with the results shown in Figure 21d. To remove the holes, the image of Figure

21d is inverted, so that the background and the holes become foreground objects. The

algorithm is then reapplied, so that the small holes are removed. The result is the inverted

once more, to give the resultant image shown in Figure 21e. The actual area of the

particles and holes to be removed was determined manually by measuring their average

sizes across a range of typical images. This process of measuring was only required to be

performed once, since all subsequent image processing used images taken with the same

camera resolution and microscope objective lens setting.

An outline of the rough segmentation result of Figure 21e is shown superimposed on the

input image in Figure 21f. It can be seen that the rough segmentation result produces a

reasonably accurate contour for cell areas where there is no halo (point A). However, in

areas where the halo is present (point B), the contour balloons outwards to encompass it,

resulting in non-cell regions being included within the contour. The next section details a

simple method of dealing with this effect and producing a refined contour that more

closely matches the actual contour of the cell regions.

3.3.2 Refined Segmentation

An examination of the pixel intensity profiles of cross-sections taken across roughly

segmented areas (Figure 23a) indicates that there is a general increase in pixel intensity in

halo regions as one moves in the direction from the rough segmentation contour toward

the actual cell boundary (Figure 23b). In contrast, areas where the rough segmentation

contour is already following the cell boundary closely, there is a decrease in intensity in the

direction of the cell. The following algorithm was developed that takes advantage of this,

by moving the contour toward the cell until the intensity gradient becomes negative.


a

b

c

Figure 23 - Pixel intensity profile (a) interval AB within roughly segmented area (b) intensity profile of AB

before smoothing, and (c) after smoothing

As a first step, a mean filter of radius 1 (see Figure 22a) is applied to the image to reduce

the effect of noise on the intensity gradient as shown in Figure 23c. Next, the portions of

the smoothed image that lie within the roughly segmented areas are scanned in four

directions (left to right, right to left, top to bottom and bottom to top) with the rough

segmentation contour being moved toward the cell for as long as the smoothed intensity

gradient in that direction is non-negative. Refer to Algorithm 1 (page 50) for a more

detailed description of this process. This produces a more refined binary mask image of the

cell areas as shown in Figure 24b. Finally, a median filter of radius 4 (see Figure 22c) is

applied to the refined mask image to remove spurs caused by noise not fully removed by

the initial mean filter. This gives the final result of the refined segmentation process as

shown in Figure 24c. The outline of this mask is shown superimposed on the original input

image in Figure 24d. It can be seen that the refined segmentation mask produces a much

closer fit to the actual cell boundaries compared to the initial rough segmentation contour

of Figure 21f. In particular, the edge halo areas no longer cause a false ballooning out of

the cell boundary contour.


a

b

c

d

Figure 24 - Refined segmentation (a) mean filter radius 1 applied (b) refined mask (c) refined mask after

median filter radius 4 applied (d) refined segmentation contour

3.4 Performance Evaluation

We have now selected two potential image normalisation methods (contrast stretching

and histogram equalisation) and developed a two stage segmentation process, rough and

refined segmentation. Rough segmentation includes the phase contrast halo, whereas

refined segmentation does not. As the halo is likely dependent in some manner on the

underlying cell structural features [45], it is important to determine if the halo contains

information useful to the assessment of cell culture quality. For this reason, we need to

decide, not only which normalisation method we should use, but also whether to use

rough or refined segmentation.


Algorithm 1 - Refined Segmentation (halo processing)

This box describes in more detail the iterative process used to update the rough segmentation mask as part of

the refined segmentation process described in section 3.3.2.

Our algorithm takes two images as input. The first is a mean filtered normalised input image (e.g. Figure 24a),

described in section 3.3.2, which for simplicity we will call the cell image. This image is used to calculate

intensity gradients and is left unchanged by the algorithm. The second image is the mask image output by

the rough segmentation stage (e.g. Figure 21e), described in section 3.3.1. We will refer to this as the mask

image. This mask image is progressively updated by the algorithm as the white areas of the mask are reduced

in size until they more closely match the actual cell contour.

The algorithm proceeds in four passes. The first pass removes the halo areas from the left side of all

foreground regions of the mask image. The altered mask image from the first pass is then input to the

second pass, which removes the halo areas from the right side. This process is continued with the third and

fourth passes, which continue to alter the mask image by removing the halo regions from the top and bottom

sides, respectively. The result of all four passes, is a more refined mask image, such as that shown in Figure

24b. Applying a median filter to this output image (see section 3.3.2) produces the complete refined

segmentation mask shown in Figure 24c.

We now describe the operation of the algorithm in more detail using the first pass as an example. In the

pseudo-code above (Pass 1) the mask image and cell image are scanned concurrently row by row from top to

bottom in the direction from left to right. In this algorithm, M represents the image width in pixels, and N

represents the image height. The top left pixel in the image is considered to be at position 1,1. It should be

noted that the mask image and the cell image both have the same dimensions. The IsMask(x,y) function

returns true if the pixel at x,y of the mask image represents a foreground (white) pixel. The UpdateMask(x,y)


function updates the (mask image) pixel at x,y to make it part of the background, by setting it to black. The

function I(x,y) returns the pixel intensity at the point x,y of the cell image.

Line 1 loops through each row in the image from top to bottom, while line 3 loops through each pixel on the

current row from left to right, with the exception of the first (leftmost) pixel. It is not included since the

algorithm requires the previous pixel’s intensity to calculate the intensity gradient.

For each pixel (except the leftmost) in the current row that represents a foreground pixel in the mask image,

as determined by the IsMask() function (Line 4), the intensity gradient at the corresponding point within the

cell image is calculated by subtracting the current pixel intensity value from that of the preceding pixel (Line

8). If the gradient is non-negative, meaning that the cell image intensity is getter brighter or staying the same

(indicating a halo region), then the mask image is updated so that the corresponding pixel location is

considered background (Line 10). This is repeated until a negative gradient is encountered in the cell image

(Line 12), meaning that we have reached the real cell boundary. At this point, we skip over any further

foreground pixels in the mask image, until we have encountered at least one background mask image pixel.

This effect is accomplished by setting the SkipToNonMaskRegion flag to true.

As a special case, if the row commences within a foreground region of the mask image (Line 5), then the

algorithm assumes we are within a cell and skips over any further foreground pixels in the mask image, until

we have encountered at least one background mask image pixel.

The process outlined above continues until all pixels in the mask image have been scanned, at which point,

the mask image now has halo areas removed from the left side of all foreground regions. Three more passes

are made over the mask image to remove the halo areas from the right, top and bottom sides. Each

subsequent pass uses the same technique, with adjustments made to alter the direction in which the image is

scanned and the intensity gradient calculated, as shown in the pseudo-code for Pass 2, 3 and 4.


In order to determine the most effective normalisation and segmentation approach (i.e.

the one that maximises pattern recognition performance), the algorithms are evaluated by

measuring their effect on the accuracy of a subsequent pattern recognition system’s ability

to discriminate between images from two BMSC cultures of differing quality, which we call

sample X and sample Y. As part of this process, statistical features are extracted from the

images and segmented regions and these are used as input to the pattern recognition

stage. During training the statistical features are provided to the pattern recognition

system along with the classification (i.e. sample X or sample Y). During testing, the pattern

recognition system is provided with only the statistical features and must estimate the

quality of the cells in the image from which the features were extracted.

In the following section we describe the method of our performance evaluation, including

the image data, the features extracted and the pattern recognition algorithms used. In this

chapter, we only discuss feature extraction and pattern recognition briefly, insofar as it is

necessary to describe how our segmentation system is evaluated. The main discussion of

these topics is covered in Chapter 4. The results of our performance evaluations are

presented and discussed in section 3.5.

3.4.1 Method

To evaluate the normalisation and segmentation algorithms, a database of 128 images (64

of sample X, 64 of sample Y) was split using two thirds (84 images) for training and one

third (44 images) for testing. Equal numbers of both samples were maintained in the two

sets. The effect of the different normalisation and segmentation methods was evaluated

by testing the performance of the system on the training set (i.e. empirical performance)

using stratified 10-fold cross validation [46]. Once the methods that maximised the

empirical performance were determined and applied, the generalised performance of the

resulting system was determined by evaluation on the (previously unseen) test set.

Feature extraction: Statistical measurements of pixel intensity values (mean, mode,

median, standard deviation, skew and kurtosis) were extracted from either the entire

image (when testing normalisation methods) or the segmented regions only (when testing

segmentation methods). The definitions of these statistical measures are provided in

Appendix A. The extracted measurements were used as input to the pattern recognition


stage, along with the class (i.e. sample X or Y). The reason the above statistical features

have been used, rather than morphological measures, such as cell size/length and so on, is

that they only require the segmentation of cell regions, rather than individual cells. This is

an important consideration, since (as illustrated in Figure 25) cells often touch and cannot

be segmented separately by the proposed method. Furthermore, in some cases, only a

portion of a cell is visible, making measurements of cell size and shape unreliable.

Figure 25 - Examples of touching and partially visible cells

Pattern Recognition: Pattern recognition performance was evaluated using four algorithms

previously employed in the literature relating to the analysis of cell images, namely, the

naïve Bayes [18], multi-layer perceptron (MLP) [47], radial basis function (RBF) network

[20] and the support vector machine (SVM) [27]. The Weka system (version 3.5.7) [46] was

used for the implementation of these algorithms. Default parameters provided by Weka

were used and no optimisation of algorithm parameters was performed. The default

parameters used by Weka are designed to give reasonable performance in many

applications and this was found to hold true for this project. A brief description of these

Touching cells

Partially visible

cells


classifiers, along with the default parameters, are now given below, with the reader being

referred to [38] and [46] for a more in-depth coverage.

The naïve Bayes classifier [46] is a simple probabilistic classifier based on the application of

Bayes’ theorem. It is termed naïve because it assumes that the input features are

conditionally independent given the class. Despite this (often incorrect) assumption, the

method often works well in practice [46, 48]. The classifier was used with input features

assumed to follow a normal distribution.

The multi-layer perceptron [46] is an artificial neural network consisting of multiple layers

of artificial neurons with a typically non-linear (e.g. sigmoid) activation function. The

connection weights are trained using backpropagation. The network used here consisted of

one hidden layer containing four neurons, sigmoid activation function, a learning rate of

0.3 and a momentum of 0.2.

The radial basis function network [46] is an artificial neural network incorporating a

weighted sum of (typically Gaussian) radial basis functions. The Weka implementation uses

a k-means clustering algorithm to determine the centres and widths of the Gaussian RBFs

and the weights are determined by logistic regression. The default parameters use two

clusters (per class) with a minimum standard deviation of 0.1.

The support vector machine is a learning method developed from statistical learning

theory [49]. It finds the hyperplane that separates the two classes with the maximum

margin. The Weka implementation uses the sequential minimal optimisation algorithm [50]

for training. The default configuration uses a polynomial kernel of degree one without the

use of lower order terms, and the complexity parameter (C) is set to 1.0.

3.5 Results and Discussion

Here we present and discuss the results of evaluating the performance of the two different

normalisation methods and our two segmentation approaches. Performance here refers

to the classification accuracy achieved by various pattern recognition algorithms

(classifiers) when using a particular preceding normalisation and/or segmentation method.

The accuracy is expressed at the percentage of images correctly classified (as either sample

X or Y).


3.5.1 Effect of Normalisation

It can be seen from Table 1 that the method of normalisation had a very significant impact

on the performance of all the classifiers, with accuracy improvements ranging from 7% to

12% compared to images that were not normalised. Contrast stretching was found to be

the best method of normalisation, with a 1.5% saturation producing the best results for 3

out of the 4 learning algorithms (see Figure 26).

Histogram equalisation resulted in a significant loss of accuracy for all classifiers even

compared to the non-normalised images, and therefore is best avoided. In retrospect, the

reason for the poor performance is clear when we consider that histogram equalisation

flattens the image histogram toward a uniform distribution. This makes the histograms

(and hence the extracted statistical features) of both classes more alike, and therefore,

harder to discriminate.

It is important to note that the absolute accuracies do not necessarily tell us which

classifier is the best, since only the default parameters were used. In fact, we would expect

them all to improve performance when their individual parameters are optimised.

However, it can be seen from the flatter curve in Figure 26 that the MLP was the most

robust technique for dealing with variations in the normalisation process. It maintained a

more consistent degree of accuracy, compared to other methods, when the normalisation

process was not ideal.

Table 1 - Effect of normalisation method vs classifier performance (% images correctly classified)

CS = Contrast Stretching

Norm. Method MLP Naïve Bayes RBF Net SVM

Hist. Eq. 75.00 59.52 59.52 58.33

None 83.33 70.24 76.19 73.81

CS (0.5% sat) 86.90 70.24 77.38 63.10

CS (1% sat) 89.29 79.76 88.10 78.57

CS (1.5% sat) 90.48 80.95 88.10 80.95

CS (2% sat) 90.48 79.76 89.29 79.76

CS (2.5% sat) 90.48 76.19 86.90 77.38

CS (3.0% sat) 86.90 75.00 80.95 73.81


Figure 26 - Classifier performance vs % saturated pixels (contrast stretching)

3.5.2 Effect of Segmentation

Table 2 shows the effect of segmentation on classifier performance using images

normalised using contrast stretching at 1.5% saturation. Accuracy was improved

significantly when using refined segmentation compared to both rough segmentation and

no segmentation. This indicates that the halo is best excluded from segmented cell regions

for best pattern recognition performance.

Table 2 - Effect of segmentation on classifier performance (% images correctly classified)

Segmentation Method MLP Naïve Bayes RBF Net SVM

None 90.48 80.95 88.10 80.95

Rough (inc. halo) 93.98 83.13 84.34 91.57

Refined (excl. halo) 96.34 87.80 92.68 95.12

3.5.3 Performance on Test Data

Table 3 shows the performance of the different learning algorithms on previously unseen

test data when using the combination of contrast stretching at 1.5% saturation and refined

segmentation. The results show that all classifiers generalised very well, achieving

comparable accuracy to that achieved on the training set.


Table 3 - Performance on previously unseen test data

Performance MLP Naïve Bayes RBF Net SVM

Training Set 96.34 87.80 92.68 95.12

Test Set 95.45 93.18 95.45 93.18

3.5.4 Limitations

Our method does have its limitations. Firstly, although the refined segmentation method

generally forms accurate contours around cell regions, it does not attempt to segment

touching cells. It may thus be of limited use, at least in its present form, in applications that

require such separation. Secondly, since the method uses an automatic thresholding

algorithm [44] to segment regions of high intensity variance (cell regions) from regions of

low intensity variance (background), the method fails if an image has insufficient

background area (see Figure 27), as is the case when cells approach confluence. When this

happens, cell body areas that are flat and spread out (low variance) become treated as

background regions, producing an unsatisfactory result. However, this is not considered to

be a major problem for cell quality assessment, since it is intended that images will be

taken early in the culture process, well before confluence is reached.

Figure 27 - Incorrect segmentation on highly confluent culture


3.6 Final Comments

Both image normalisation and segmentation were shown to have a large impact on the

ability of a subsequent pattern recognition system to correctly discriminate between two

bone marrow stromal cell cultures of differing quality. Best results were obtained when

images were first normalised using contrast stretching (at 1.5% saturation) and then

segmented using the refined segmentation method described in section 3.3.2.

Using the above methods the system achieved an average image classification accuracy of

93% on the training data and 94% on previously unseen test images. This represents a large

(~17%) improvement in accuracy compared to that achieved with non-normalised,

unsegmented images.

Furthermore, it was found that including the halo region in the segmented regions hinders

pattern recognition performance. This leads to the conclusion that it does not contain

information indicative of cell culture quality.

Having now developed a suitable image normalisation and segmentation algorithm, the

following chapter will focus on the feature selection and pattern recognition components

along with training and testing the system on a wider range of bone marrow stromal cell

culture images.

59

Chapter 4

Feature Extraction and Pattern

Recognition for BMSC Quality

Assessment

4.1 Introduction

Having established a method of segmentation in the preceding chapter, we now consider

the taking of measurements from the segmented regions and the use of such

measurements with machine learning algorithms, with the goal of discriminating between

healthy and unhealthy cell cultures.

We divide this chapter into three main parts, feature extraction, pattern recognition and

the performance evaluation. The first deals with the selection and extraction of suitable

measurements that are designed to characterise cell culture images in terms of health.

The second part looks at the basic theory behind pattern recognition systems and provides

a basic description of the particular classifiers used in our project along with the salient

parameters that control their operation. The third part deals evaluating the performance

of our entire quality assessment system in terms of how accurately it classifies images of

healthy and unhealthy cell cultures. We then conclude the chapter with a brief summary.

60 Chapter 4. Feature Extraction and PR

4.2 Feature Extraction

As previously stated, the purpose of feature extraction is to take measurements from cell

regions that will allow discrimination of cell images on the basis of health quality. The list

of features extracted from each image is called a feature vector. It is this feature vector,

rather than the image itself, that is presented to the subsequent pattern recognition stage.

In fact, the pattern recognition system has no concept of an image as such. It is concerned

only with the similarities and differences of feature vectors. Thus our feature vector really

represents a summary of the content in the image.

We now consider the matter of what features should make up our feature vector. Since

there are an almost endless number of possible features, we first give a brief overview of

the four broad categories of features relevant to cell image analysis [17]. Following this,

we describe the features chosen for our application and the rationale behind that choice.

4.2.1 Morphometric Features

Morphometric features represent measures relating to size and shape. In the context of

cell image analysis this could typically be such measures as: cell perimeter, cell area, cell

circularity, concavity, convexity, etc, as well as similar measures of the cell nucleus. These

may also be compound measures, such as the ratio of nucleus area to cytoplasm area.

While it may appear at first glance that these would be useful features, other researchers'

experience has shown that morphology-based measures of cell or nucleus shape to be

rarely useful for discriminative purposes [15, 17]. Furthermore, there are practical

considerations that make them difficult to use in our application. Firstly, as mentioned in

chapter 3, our segmentation process does not segment individual cells, but rather cell

regions. As a result, cells that touch are not segmented individually, rendering measures of

size and shape meaningless in these cases. A second consideration is that cells around the

edges of each image are often only partially visible, again rendering such measures

meaningless. This becomes even more problematic if we desire to split the original images

into smaller tiles (to increase the size of our data set), since the amount of partially visible

cells increases as the tile size decreases.

Chapter 4. Feature Extraction and PR 61

4.2.2 Densiometric/Intensity-based Features

These types of features are based on the intensity or grey level of pixel values within the

regions of interest in an image, such as individual cells or cell regions. Since a given region

of interest often contains a large number of pixels, in most cases statistical measures of

intensity are used, rather than the pixel values themselves. Some common measures are

mean intensity, standard deviation (or variance) of intensity, the median, the mode,

skewness, kurtosis, entropy and uniformity [37].

These measurements do not, in themselves, give an indication of spatial relationships and

so provide only a very rudimentary description of textural characteristics. For example, in

Figure 28 below, images a and b represent smooth textures and, in both cases, they have

the same standard deviation but are separable because of their different mean intensity

values. Images a and b can be separated from images c, d and e on the basis of standard

deviation. However, because the measures above do not take spatial information into

account, images c, d and e cannot be separated from each other using such measures,

since the images (c-e) all have the same histogram and therefore the same value for these

measures.

a b c d e

Figure 28 - Texture examples. Images a and b will have different mean intensity values, but equal standard

deviation. Both a and b will differ from c, d and e for both mean and standard deviation. Since densiometric

measures do not include spatial information, images c, d and e all produce the same measure of mean

intensity, standard deviation and so on, since their histograms are identical. The black border around each

image is for display purposes and is not considered part of the image.

However, this problem can be overcome in some measure. Spatial information can be

introduced by subdividing segmented areas into biologically meaningful subregions (e.g.

cell border versus cell interior, cell body versus cell nucleus, etc) and taking the above

measures separately for each type of region [17].

Despite the drawbacks, these types of features have advantages. Firstly, they do not

require individual cells to be segmented and hence are attractive for our project. Secondly,


they can be obtained from any arbitrarily shaped region - there is no requirement for

rectangular patches, etc.

4.2.3 Textural Features

Textural measures are concerned with the spatial distribution of intensity levels within the

region of interest. There are many ways to describe textures such as rough, smooth,

coarse, regular, random, etc. Since these words are very subjective, in practice some form

of mathematical description is typically used. Some common examples are the gray level

co-occurrence matrix and run-length features [17, 37]. Spectral information based on the

Fourier spectrum can also be used to gain a measure of overall global periodicity within an

image or sub-region [37], however this in itself does not indicate the spatial location of a

texture within the area under analysis. More recently, wavelet-based techniques have

been found useful for texture analysis, since they provide both spatial and frequency

information. For example, the related works by Mangoubi [27] and Jeffries [26] use the

mean deviation and variance of wavelet subband coefficients from a two level wavelet

pyramid decomposition as features for embryonic stem cell colony quality assessment.

4.2.4 Structural/Contextual Features

Structural or contextual features describe the relationship between objects or areas within

the region of interest [17]. These features may be things like the number of cells in an

image, the number of dark particles in a cell or cell nucleus, the distance of the cell nucleus

to the edge of the cell, and so on. Clearly, these types of features are very dependent

upon the success and sophistication of the preceding segmentation stage.

4.2.5 Features for this project

Given the promising results achieved in chapter 3 from using intensity based features

(mean, mode, median, standard deviation, skew and kurtosis) it was decided to continue

the use of these. However, initial testing with just these features produced results of

around ~65% accuracy and so it was decided to add some additional features in an effort

to improve accuracy of recognition. Entropy and uniformity were added. In chapter 3, we

used minimum error thresholding as part of our rough segmentation. It was decided to

use this threshold level as an additional feature. It was also decided to incorporate some


spatial information by dividing the segmented regions into inner and outer regions, along

the lines of that described in [17].

The inner and outer regions were located by taking the segmentation mask (Figure 29a)

produced by our refined segmentation stage and applying an erosion operation [37]. The

eroded result produced a mask representing the inner area (Figure 29b). This inner area

mask was then subtracted from the original mask to produce a mask representing the

outer regions (Figure 29c). The three masks for the total cell regions, inner cell regions and

outer cell regions allowed us to extract our statistical features from these areas separately.

a

b

c

Figure 29 - (a) Original segmentation mask produced by our refined segmentation algorithm (b) inner mask

created by eroding the original mask with a circular structuring element (6 pixel radius) (c) outer mask

created by subtracting the inner mask from the original mask.

The use of the erosion operation in generating the inner and outer mask introduces a

parameter, namely, the radius of the erosion mask. To determine an appropriate value, a

number of erosion radii were tested, as shown in Figure 30. From visual inspection, it was

felt that a radius of 6 provided the best separation of outer cell region from inner cell

region in the majority of cases. The precise shape of this erosion mask, also called a

structuring element, is shown in Figure 31. As can be seen from Figure 30, a radius of 6

generally removes the long protrusions from the cells (which thus fall into the outer

regions), while still leaving a sufficient amount of the internal cell body as the inner region.

64

Input Image Radius 2

Radius 6 Radius 8

Figure 30 - Effect of varying the erosion radius on the size of the inner and outer cell regions.

in the mask images represent the outer cell

regions.

Figure 31 - Circular structuring element (radius = 6) used for erosion operation.

Since we now have three different regions, extracting eight statistical measures from each

region, gives 24 different features. As noted above, we also use the minimum error

Chapter 4. Feature Extraction and PR

Radius 2 Radius 4

Radius 8 Radius 10

Effect of varying the erosion radius on the size of the inner and outer cell regions. The white areas

in the mask images represent the outer cell regions, while the black centres inside represent the inner cell

Circular structuring element (radius = 6) used for erosion operation.


, gives 24 different features. As noted above, we also use the minimum error

Feature Extraction and PR

The white areas

centres inside represent the inner cell


, gives 24 different features. As noted above, we also use the minimum error


threshold (calculated in our segmentation stage) as an additional feature, bringing the total

number of features to 25. The definition for the statistical features used can be found in

Appendix A, and the minimum error threshold calculation is described by Kittler and

Illingworth [44]. The complete set of features used are listed in Table 4.

Table 4 - Total feature set used in our quality assessment system

Total Cell Regions Inner Cell Regions Outer Cell Regions

1. Mean (total)

2. Median (total)

3. Mode (total)

4. Standard Deviation (total)

5. Skew (total)

6. Kurtosis (total)

7. Entropy (total)

8. Uniformity (total)

9. Mean (inner)

10. Median (inner)

11. Mode (inner)

12. Standard Deviation (inner)

13. Skew (inner)

14. Kurtosis (inner)

15. Entropy (inner)

16. Uniformity (inner)

17. Mean (outer)

18. Median (outer)

19. Mode (outer)

20. Standard Deviation (outer)

21. Skew (outer)

22. Kurtosis (outer)

23. Entropy (outer)

24. Uniformity (outer)

25. Minimum Error Threshold

4.3 Pattern Recognition

The pattern recognition stage aims to separate the feature vectors (and hence the images

from which they are derived) into two separate classes, namely, healthy cultures and

unhealthy cultures. The algorithm that performs this action is called a classifier. In order

for a classifier to do this successfully, it must be trained on a representative sample of

input feature vectors. During training, the classifier is also given the class of each feature

vector. From this, it attempts to construct a rule that separates the two classes. This

learned rule can then be applied to new (previously unseen) feature vectors in order to

provide a classification.

While there are many different classification algorithms - we discuss two below - they all

share the common task of trying to find a pattern within the training data that will allow

discrimination between the two classes of feature vectors. This pattern, or rule, is often

termed a decision boundary. If the feature vectors are plotted in an n-dimensional space,

where n is the number of features, then the decision boundary is the dividing boundary


that separates vectors of each class. Once this boundary is determined from the training

data, it can then be used to determine the classification of new feature vectors according

to where they lie in the feature space with respect to the decision boundary.

When the classifier constructs this boundary during training, there is a balance to how

much flexibility it should have in the shape of the boundary it constructs. If the boundary

is not flexible enough, it may not be able to adequately represent the data. An example

would be trying to separate data that is not linearly separable with a straight line. On the

other extreme, if the classifier is allowed too much flexibility, it may overfit the boundary

to the training data. This can result in high recognition accuracy on the training data, but

poor performance when tested on previously unseen test data. Different classification

algorithms vary in the flexibility on this decision boundary, and the means by which it is

constructed during training.

Given the large array of possible classifiers available, it is natural to wonder which should

be used for our task. Unfortunately, there is no one single 'best' classifier [38]. Instead, a

lot comes down to how much prior information is known about the distribution of feature

values. In general, it is often a case of 'try and see' [38]. To this end, pattern recognition

systems that have been used with reasonable success for other cell or tissue related

recognition tasks have been selected for use in this project. A sample of such work is

shown in Table 5.

Table 5 - Classifiers used in recent cell or tissue related research

Research Work Classifier

White blood cell recognition [18] Naive Bayesian Classifier

Quality assessment of human embryonic stem

cells [26, 27], Detection of apoptosis in human

mesenchymal stem cells [29]

Support Vector Machine

Lung tissue [20] Radial Basic Function Network

Segmentation of cell nuclei [47] Multilayer Perceptron

In choosing which classifiers to use for our quality assessment system, a number of factors

were considered. Firstly, it was felt appropriate to limit the number of classifiers evaluated


to two, rather than continue with the use of the four used to evaluate our segmentation

algorithm, as described in chapter 3. This was simply a matter of time available. It was felt

better to spend more time evaluating performance with two, rather than attempting to

evaluate four classifiers, and spending less time on each. As previously stated in the

introduction (Chapter 1), our goal is not to find the best classifier, but rather to investigate

whether our quality assessment task is possible with state-of-the-art classifiers. The multi-

layer perceptron and the support vector machine were chosen. The multi-layer perceptron

was chosen because it achieved the highest recognition accuracy in our segmentation tests

and showed the best robustness when dealing with the variation of parameter values in

the contrast stretching algorithm (section 3.5.1). The support vector machine was chosen

due to its use as a classifier in the assessment of embryonic stems cells [26, 27] and human

mesenchymal stem cells [29].

As with the classifiers used in chapter 3, we once again make use of the Weka software

[46] (version 3.5.8) for the implementations of the multi-layer perceptron and support

vector machine. These two classifiers are now discussed, along with the methods used to

determine which parameter settings to use.

4.3.1 Multi-Layer Perceptron

The multi-layer perceptron (MLP) is a feed-forward artificial neural network consisting of

multiple layers of interconnected artificial neurons (nodes). The layers consist of an input

layer containing a number of nodes equal to the number of features in the feature vector,

and output layer containing one node for each class, and one or more hidden layers, each

containing one or more nodes. Each node receives an individually weighted input from all

the nodes in the previous layer. These weighted inputs are summed together in the node

and then input to a typically non-linear (e.g. sigmoid) activation function. The output of

this activation function is then passed individually to each node in the next layer, again via

an individual weighting. Usually, these connection weights are trained using a process

called backpropagation [38]. While the number of nodes in the input and output layers are

fixed as just described, there is no set recipe for how many hidden layers to use, or the

number of nodes to use in each hidden layer - it typically comes down to a combination of

trial and error, combined with previous experience [38]. If there are not enough nodes,


then the classifier may not be able to sufficiently discriminate between the classes, and

thus produce a poor performance on the training set. If there are too many nodes or

layers, then the classifier may tend to overfit to the training data, and thus have good

training set performance, but poor test set performance.

The Weka implementation uses backpropagation to train the classifier, a sigmoid activation

function, and provides control over a number of parameters. The number of hidden layers

and the number of nodes in each hidden layer may be selected. The learning rate and

momentum parameters control the amount by which the weights are updated, and the

momentum applied to the weights during updating, respectively.

In selecting parameter values to use, the following method was followed. Initially, the

learning rate and momentum were set to default values (of 0.3 and 0.2, respectively) and

the number of nodes in the hidden layer was varied from 1 to 15. Only one hidden layer

was used as initial trials showed very little improvement with extra layers when evaluated

under 10-fold stratified cross validation (see section 4.4). Once the number of nodes that

achieved the best performance (in terms of classification accuracy) was found, this value

was held constant and the learning rate was varied from 0.1 to 0.5 in 0.1 steps to find the

best value. Once the best value of learning rate was found, it too was held constant and

the momentum was then changed in the same range to find the best value. In all, this gave

a total of 25 parameter combinations for the MLP.

4.3.2 Support Vector Machine

The support vector machine is a learning method developed from statistical learning

theory [49]. It aims to find the hyperplane (decision boundary) that separates the two

classes with the maximum margin. Since the data may not be linearly separable in the

feature space, the input data is typically remapped to a higher dimensional space with

some appropriate nonlinear kernel function [38]. Common functions used for this purpose

are the polynomial kernel, radius basis function (RBF) kernel and hyperbolic tangent [51].

The Weka implementation uses the sequential minimal optimization algorithm [46, 50] for

training the support vector classifier. The polynomial kernel, Weka's default kernel for the

SVM, was used. This kernel has two parameters, the degree of the polynomial and whether


to use lower order terms or not. The second parameter determines whether the

polynomial of degree q has the form (<x,y>)q or (<x,y>+1)

q, the latter includes lower order

terms. The SVM parameter, C, provides control over complexity, to help limit overfitting.

We have left this value at the Weka default value of 1, and simply adjusted the two

polynomial kernel parameters. We did not carry out trials by varying the C parameter as

quite reasonable results were obtained with the default value.

For the SVM, parameter tuning involved stepping through the polynomial degree from 1 to

8 and for each, trying both with and without the lower order terms. This gave sixteen

parameter combinations. The combination that gave the highest classification accuracy on

the training data (using 10-fold cross validation) was selected for use.

4.4 Evaluating Performance

In order to evaluate the performance of our system (in terms of classification accuracy) we

divided each data set of interest (e.g. all images from Day 1) into two thirds for training and

one third set aside for testing, according to common practice [29, 46]. The split was also

done in a stratified manner to ensure images from all patients, FCS concentrations and

wells were represented in both sets. It should be noted that the test set is kept completely

separate and not used for training the classifier; hence we say it is 'previously unseen'

when tested.

We used our training data to tune the parameters of the pattern recognition algorithm

being used. For each parameter setting, the accuracy on the training set was determined

using 10-fold stratified cross validation [46]. This was done by stepping parameter values

across a range, and selecting the settings that gave the highest accuracy on the training set.

The actual parameters and step increments used for particular classifiers is described in

sections 4.3.1 and 4.3.2. The highest accuracy so achieved, and the parameters used, is

what is reported in Chapter 6 as the training set performance. Once parameters were

determined this way, they were then evaluated using the previously unseen test set. In

this case, the whole training set was used to train the classifier, and then performance was

evaluated on the test set using the parameters decided upon during training. This is what

we report in Chapter 6 as the test set performance.


In order to ensure our results were not simply the result of providence in our division of

training and test set images. We created nine more (i.e. a total of ten) random divisions of

data into training and test sets (two thirds/one third). Each set represents a different two

thirds/one third split of the same data. The technique used above was repeated for each

of these sets, giving a total of ten training performance figures and ten test performance

figures. For each of these sets of results, we have taken the average and standard

deviation as a summary of overall training and test performance for the given

classifier/test case under consideration.

To further validate our test results, we also desired to increase the amount of data for

testing. To do this, we used two different methods. In the first method, we simply divided

each image into four non-overlapping tiles, with each tile being one quarter of the original

image. This increased our dataset by four. In the second method we used a random

window approach where we took four samples from each image using a randomly

positioned rectangular window 80% the size of the original image. Of course, in this case,

there would be considerable overlap between windows taken from the same image. The

advantage of this method over the non-overlapping tiles is that the images are still

reasonably close to the original image size (the size we would expect the system to

normally work with) and yet still allow for variation. This again, produced a dataset four

times larger than the original one.

When considering whole images, for a given culture day there were 162 images available (3

patients x 2 FCS concentrations x 3 wells x 9 images per well). However, a certain number

of these images were not segmented correctly due to the large number of cells present in

those particular images. It was decided to remove these manually, so that the training

would not be influenced by what were clearly bad examples. However, we do quantify the

effect of including these images later in Chapter 6. Naturally, it would be desirable, in

future, to make the removal of these images an automatic step and this is discussed

further in Chapter 7. So, having located and removed some images as 'segmentation

failures', we are left with a total of 139 images. When using non-overlapping tiles, tiles

that exhibited segmentation failure were removed in the same manner as for the whole

images.


4.5 Summary

In this chapter we have examined the method by which we extract features from cell

regions located by our segmentation algorithm, which was discussed in the previous

chapter. We also examined the task of pattern recognition, which involves training a

classifier based on features extracted from images of both healthy and unhealthy cell

cultures. This included a description of the multi-layer perceptron and the support vector

machine classifiers and how we selected suitable parameter values for them. Lastly, we

discussed the method employed for evaluating the performance of the classifiers using

both training and test data.

We are now ready to look at our results in terms of the classification accuracy achieved by

our system as whole, along with a discussion of their significance. However, before doing

so, we will take a brief diversion in the next chapter to describe the software tools we have

developed. These tools were used to automate much of the processes involved in both

Chapter 3 and Chapter 4, and which were instrumental in the formation of our results,

which are presented in Chapter 6.



73

Chapter 5

Software System

5.1 Overview

During this research a large number of images needed to be processed with features

extracted, test and training datasets formed, and pattern recognition tasks carried out

(including parameter selection for the given machine learning algorithms used).

Additionally, this whole process needed to be run for multiple scenarios, such as the

different types of image splitting (whole images, non-overlapping tiles and random

windows). This mandated the use of some form of automatic method to assist with the

task. Originally, processing was performed using a combination of inbuilt ImageJ [35]

functions and custom-made Java plug-ins, with ImageJ macro script used to link them into

a functional pipeline. This was satisfactory during initial research and algorithm

development, such as the development of the segmentation algorithm, but soon proved

cumbersome during the evaluation of our whole system, since changes in parameters

could not be performed without requiring changes to hardcoded values within the macro

script and the various plug-ins. Additionally, ImageJ scripts were slower in execution and

troublesome to debug.

To improve this situation, a number of software tools were developed. The first step was

to combine all the image processing steps up to and including feature extraction,

previously handled by ImageJ scripts as noted above, into one plug-in. This plug-in, written

in Java, is called CellScan, and will be described in the next section. The next requirement

was a database to store and organise the feature data extracted from images by the

CellScan process. A Microsoft Access database application, we have named FeatureDB,

74 Chapter 5. Software System

was created for this purpose. Aside from the storage of data, the database also contains

program code, written using Visual Basic for Applications (VBA), that allows specific subsets

of the data to be exported into the file format used by Weka (.arff format) [46]. This was

necessary in order for it to be used with Weka's machine learning algorithms. In the final

stage, a Java program, called CellAPR, was written to allow the use of Weka's algorithms

via its Java libraries, rather than through its graphical user interface. This provided much

more specific and automated control of the process, especially the determination of

parameter settings for various pattern recognition algorithms. It also allowed training and

test performance figures to be automatically measured and collated, saving very large

amounts of time, and making more thorough investigations possible.

The diagram below illustrates the relationship of the various parts of our developed

system, from the input of sets of unprocessed cell culture images, to the output of

tabulated pattern recognition performance results.

Figure 32 - Software system components

The software components shown above allowed us to evaluate and report on the

performance of our quality assessment system. However, we also desired to test the

performance of human experts when presented with the same set of images as our

system. To facilitate this, another software application was developed for presenting

human experts with cell culture images and providing a means for them to classify each

one. We describe this software application in section 5.5.

CellScan*Process images

*Extract features

*Saves segmented

images (optional)

CellAPRAutomates the PR process

*Parameter selection

*Performance evaluation

*Output performance results

FeatureDB*Store feature sets

*Formation of datasets

*Export as Weka files

Chapter 5. Software System 75

Having given a brief overview of the developed software tools, we now discuss each

component in more detail, beginning with CellScan.

5.2 CellScan

The CellScan program was developed as a plug-in (written in Java) for the ImageJ

application. It provides means to automate image processing and feature extraction of

large batches of images, and provides a simple graphical user interface for the input of

relevant parameters. When the plug-in is installed into ImageJ's plug-ins folder, it may be

activated by selecting CellScan from the ImageJ Plug-ins menu, as shown in Figure 33,

below:

Figure 33 - Activating the CellScan plug-in from ImageJ

The graphical user interface for the CellScan plug-in consists of a single form (Figure 34)

that is displayed when the plug-in is activated. This form allows the user to select the

source of the images to process, what pre-processing, if any, should be done to each

image, as well as what parameters should be used during segmentation and feature

extraction. Extracted feature values may optionally be displayed on the screen or saved

into a nominated Microsoft Access database (i.e. our FeatureDB database). Finally, the

user may elect to have a copy of each processed image (or image tile), with the

segmentation boundaries marked by a yellow line, saved to a nominated output folder.

This option was useful to allow visual inspection of images (via a tool in FeatureDB) to

locate those that were poorly segmented.


Figure 34 - The CellScan graphical user interface

The Java code for the CellScan plug-in is provided in Appendix C. This includes an

implementation of the rough and refined segmentation algorithm described in Chapter 3.

5.3 FeatureDB

A database was required to store and organise the large amount of data produced by the

CellScan process. In addition to this, there was a need to:

• Flag data from images with poor segmentation

• Create multiple training/test set divisions

• Export data in Weka (.arff) format for pattern recognition


We used Microsoft Access 2007 for this purpose, since it allows data storage, ease of data

manipulation, rapid development of graphical user interfaces, and the ability to write

custom program code.

Data extracted by the CellScan process was automatically input into a table within

FeatureDB, which would then be renamed to represent the particular data it contained.

Each time the CellScan process was run, a new table of data would be produced. This

allowed a large collection of feature data to be built up representing features extracted

from cell culture images for different days, for different tiling techniques (whole images,

random windows, etc), and so on. Table 6 enumerates various feature data tables in

FeatureDB that have been populated by running CellScan.

Table 6 - FeatureDB feature data tables (populated by CellScan)

Table Description Number of Images

(extracted feature vectors)

tblDataSets_Day01_Tiles1 Whole images - Day 1 162

tblDatasets_Day02_Tiles1 Whole images - Day 2 162





tblDataSets_Day01_Tiles4 Non-overlapping tiles - Day 1 648

tblDataSets_Day01_TilesRW4 Random window tiles - Day 1 648

In addition to the above tables, three other tables were constructed that can be used to

partition the data in each feature data table into ten separate sets, each set representing a

different random division of training and test data. For each set, images were randomly

assigned to either the training or test sets in a stratified manner to ensure that image

categories (patient, FCS and well) were proportionally distributed between training and

test sets. Three partition tables were required to cater for the three types of image

splitting used, namely, whole images, non-overlapping tiles, and random windows. In

order to obtain a particular dataset for export, a relational join is made (using an SQL


SELECT query) that links the desired data table with the appropriate partition table, along

with a filter condition specifying the particular set number (1 to 10) and whether training

or test data is desired. The result is the subset of feature and classification data that is

ready for export for processing by the pattern recognition stage (CellAPR).

In order to get data to CellAPR for evaluation, it must first be exported from FeatureDB in

the .arff file format required by the Weka machine learning algorithms. A function was

written in VBA for this purpose. The function allows the user to specify:

• Source database table (see Table 6)

• Feature subsets to export (total cell area features, inner cell area features, outer

cell area features, mixture modelling threshold feature)

• Whether to include data from images that did not segment correctly

• Output folder and base file name

When executed, the function creates 20 separate .arff files in the nominated folder,

consisting of 10 training set files and 10 test set files. Each file is given the base file name

specified, prefixed with the set number from 01 to 10, and suffixed with either _training or

_testing. These files are now ready to be automatically read and processed by our CellAPR

program, described shortly.

FeatureDB also included a simple form-based utility (Figure 35) to facilitate visual

inspection of segmented images or image tiles saved by the CellScan process. This allowed

images that were incorrectly segmented to be flagged, so that we could decide whether or

not to include them when exporting datasets to our CellAPR program. The process

involved setting the data source of the form to the relevant table whose images were to be

inspected, inspecting each image (which would be automatically loaded and displayed on

the form), and ticking the 'Segmentation Failure' checkbox if the image was poorly

segmented.

Chapter 5. Software System

Figure

5.4 CellAPR

This final part of the syste

consuming task of parameter selection for the particular machine learning algorithm under

evaluation. The output of this process is tabular data representing recognition

performance achieved

batches of training and test data (exported from FeatureDB). The creation of this program

made it possible to include a much larger analysis of data than would otherwise have been

practicable.

The program is a command

control the machine learning algorithms, without requiring the use of Weka's own

graphical user interface. It allows easy nomination of input data files (.arff files)

and test data, as well as the machine learning algorithm (e.g. SVM, MLP) to use. The Java

source code for the CellAPR program is included in

Software System

Figure 35 - Form for visual inspection of segmentation results

This final part of the system automates the pattern recognition stage, including the time



achieved by a particular machine learning algorithm across 10 separate



e program is a command-line Java program that uses the Weka class libraries to directly


graphical user interface. It allows easy nomination of input data files (.arff files)


source code for the CellAPR program is included in Appendix C.

79

Form for visual inspection of segmentation results

m automates the pattern recognition stage, including the time



a particular machine learning algorithm across 10 separate



line Java program that uses the Weka class libraries to directly


graphical user interface. It allows easy nomination of input data files (.arff files) for training



When the program is run, it processes the 10 training and 10 test .arff data files nominated

(and previously exported from FeatureDB). For each training set, it evaluates each

combination of input parameters and selects the combination that gives the highest

recognition accuracy on the training set, using stratified 10-fold cross validation. The

actual parameters used depended upon the particular learning algorithm being trailed.

The program automates the method of parameter selection described in sections 4.3.1 and

4.3.2.

Once the best performing set of parameters is found, the program then uses them to train

the learning algorithm on the entire training set and then evaluates recognition accuracy

on the corresponding test set. Once this is done, the results are output, including the set

number, the number of images in the corresponding training and test sets, the selected

algorithm parameters, and the training and test set performance. This process is repeated

by the program for all ten sets of training and test data. When complete, the program

calculates and outputs the mean and standard deviation of recognition performance across

all ten sets of training and test data.

The typical output produced by executing the program is shown in Figure 36. This includes

the version number of the Weka class libraries, the learning algorithm selected, the folders

containing the training and test datasets (.arff files), the measured performance data, and

the total execution time.

Weka Version: 3.5.8

SVM

Training Folder:

E:\Results\PR\Output\NewJavaSystem\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFea

tures\

Test Folder:

E:\Results\PR\Output\NewJavaSystem\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFea

tures\

01 (94/45) (<x,y>)^2.0 93.62% 84.44%

02 (93/46) (<x,y> + 1)^2.0 90.32% 86.96%

03 (94/45) (<x,y>)^3.0 88.30% 88.89%

04 (94/45) (<x,y>)^2.0 88.30% 88.89%

05 (94/45) (<x,y>)^2.0 89.36% 82.22%

06 (95/44) (<x,y> + 1)^4.0 85.26% 86.36%

07 (96/43) (<x,y> + 1)^2.0 84.38% 88.37%

08 (91/48) (<x,y>)^2.0 84.62% 87.50%

09 (91/48) (<x,y>)^3.0 90.11% 87.50%

10 (91/48) (<x,y>)^2.0 87.91% 89.58%

Mean 88.22% 87.07%

Std Dev 2.89% 2.26%

Time Taken: 0:01:07.344

Done!

Figure 36 - Tabular output produced by CellAPR


5.5 Expert Performance Evaluation Tool (EPET)

The software tools described above were useful in the research and evaluation of our

quality assessment system, however we also needed to compare the performance of our

system to that of human experts. To that end, another software tool was created (again,

using Microsoft Access) to assist with the process. The key functionality of this tool was to:

• Present an expert with a series of unlabeled BMSC images

• For each image, allow the expert to select a classification of healthy (10% FCS) or

unhealthy (0.2% FCS)

• Automatically mark the results

A screen shot of the tool is shown in Figure 37 below.

Figure 37 - Expert Performance Evaluation Tool


The text below the navigation buttons contains instructions for the expert undertaking the

test. It reads:

Note:

Images are of human BMSC cells and can be from any of 3 different patients.

One group of images are of cells that have been in media containing only 0.2% FCS for ~10

hours while the other group have been in the usual 10% FCS during that time.

Your job is to pick whether each image is of cells in 0.2% or 10% FCS.

The images are randomly ordered.

The tool allows the expert to cycle back and forth through all the images in the set and give

each image, initially flagged as 'Awaiting Classification', a classification of either 10% FCS or

0.2% FCS. The expert's classifications for each image in the set is saved to a database table.

Once the test is complete, the expert's classifications are marked against the ground truth

data to measure their accuracy.

The data gathered with this tool, namely the recognition performance of human experts, in

presented in section 6.3 of Chapter 6.

5.6 Summary

In this chapter we have discussed the software tools developed as part of this research

project. The software tools developed have been instrumental in the production and

analysis of our results, which are discussed in Chapter 6.

Furthermore, a framework is now in place that will speed the progress of future research

investigations, such as the use of other machine learning algorithms, different image

processing parameter settings and new image sets.

83

Chapter 6

Results and Analysis

6.1 Introduction

In this chapter we investigate the performance of the whole system. Specifically, we

investigate the accuracy of the system in correctly distinguishing between images of

healthy and unhealthy cells. To recap, healthy cells were those grown in a culture media

containing the normal amount of nutrients, namely, 10% fetal calf serum (FCS). Unhealthy

cells were those grown in media with only 0.2% FCS. Since it is desirable to determine the

health of cell cultures as early as possible, we concentrate primarily on cell culture images

from the first day of culture.

For our first investigation, we evaluate the system’s performance using whole images, non-

overlapping image tiles and random window tiles, as discussed in Chapter 4 (section 4.4).

For each of these image sets, we test performance using the multi-layer perceptron (MLP)

and the support vector machine (SVM), described in section 4.3.1 and section 4.3.2,

respectively.

We mentioned previously in section 4.4 that we removed images from our training and test

sets where the segmentation failed (either due to a high confluence or absence of cells). In

our next investigation, we quantify the effect that including such images actually has on

the system. We would naturally expect them to have a detrimental effect, which is why

they were removed, however it will be useful to see whether the impact of including them

justifies the future development of an automated method of removing them. For example,

if the impact is minor, it may be preferable just to include them.

84 Chapter 6. Results and Analysis

Following this, we then compare the performance achieved by our system with the

performance achieved by two human experts (cell biologists). This is important since a key

aim of our project is to find out whether it is possible, through the use of image processing

and pattern recognition techniques, to predict the growth potential of a culture of human

mesenchymal stem cells at early stages, before it is readily apparent to a human observer.

Although we are primarily interested in images from day one, we still briefly examine the

performance of the system on images taken over the full six days of culture. It would be

natural to expect that differences between healthy and unhealthy cultures would become

more apparent over time, especially to a human observer, however we test to see whether

this holds true for our automated system. While we mainly envisage the system being

used early in the culture process, it would be still useful to know whether the system can

be used effectively after day one, since a cell researcher may not always have the

opportunity to run a quality check on the first day.

Lastly, we examine the effectiveness of the different feature groups, in order to determine

which cell areas are the most useful for the system in the discrimination of cell quality, and

then conclude the chapter with a brief summary.

6.2 System Performance - Day 1 Images

This section aims to establish the expected performance of our system in terms of

classification accuracy. We initially measure performance on whole images, and then, to

increase the size of our datasets, we split the images into a larger number of smaller image

tiles using two different methods. We also evaluate the system performance on those

larger datasets and compare the results with those obtained from the whole images. For

this section, we measure the system's performance using cell images from day one. As

discussed previously in Chapter 2, these images were taken approximately 10 hours after

the difference in FCS concentration was introduced.

6.2.1 Whole Images

Firstly, we examine the recognition accuracy of the system using whole images. Table 7

shows the training and test set performance achieved by the MLP for each of the 10

Chapter 6. Results and Analysis 85

different training/test set divisions, whose formation is described in section 4.4. Similar

information is shown for the SVM in Table 8.

Table 7 - MLP performance on whole images - Day 1

Set Parameters Training Test

Nodes Learning Rate Momentum

01 (94/45) 2 0.3 0.1 92.55% 80.00%

02 (93/46) 2 0.5 0.3 94.62% 89.13%

03 (94/45) 5 0.3 0.1 88.30% 82.22%

04 (94/45) 2 0.1 0.1 90.43% 84.44%

05 (94/45) 11 0.3 0.1 90.43% 84.44%

06 (95/44) 1 0.3 0.2 89.47% 88.64%

07 (96/43) 6 0.3 0.2 87.50% 83.72%

08 (91/48) 1 0.2 0.2 92.31% 81.25%

09 (91/48) 15 0.3 0.1 89.01% 83.33%

10 (91/48) 10 0.3 0.2 89.01% 95.83%

Mean 90.36% 85.30%

Std Dev 2.20% 4.70%

Time Taken: 0:25:24

Table 8 - SVM performance on whole images - Day 1

Set Kernel Training Test

01 (94/45) (<x,y>)^2.0 93.62% 84.44%

02 (93/46) (<x,y> + 1)^2.0 90.32% 86.96%

03 (94/45) (<x,y>)^3.0 88.30% 88.89%

04 (94/45) (<x,y>)^2.0 88.30% 88.89%

05 (94/45) (<x,y>)^2.0 89.36% 82.22%

06 (95/44) (<x,y> + 1)^4.0 85.26% 86.36%

07 (96/43) (<x,y> + 1)^2.0 84.38% 88.37%

08 (91/48) (<x,y>)^2.0 84.62% 87.50%

09 (91/48) (<x,y>)^3.0 90.11% 87.50%

10 (91/48) (<x,y>)^2.0 87.91% 89.58%

Mean 88.22% 87.07%

Std Dev 2.89% 2.26%

Time Taken: 0:01:31

With reference to these tables, the two numbers in brackets after the set number are the

number of images in the training set and test set, respectively. For example, in set one,

there were 94 training images and 45 test images. The reason the numbers vary across the

different sets is due to the removal of images where segmentation failed. Despite this

variation, in all sets the number of training images plus the number of test images always

equals 139. The parameters that gave the best training set performance, and that were

subsequently used with the test set, are also shown. For the MLP, this consists of the

number of nodes in the hidden layer, the learning rate and the momentum (section 4.3.1).


For the SVM, the form of the polynomial kernel is displayed (section 4.3.2). For example,

(<x,y>)^2.0 represents a kernel of degree two, with lower order terms not included,

whereas (<x,y> + 1)^4.0 represents a kernel of degree four with the lower order terms

included. The mean performance (shown in bold) and standard deviation over all 10

training and test sets is given at the bottom of the table. Additionally, the actual time

taken by the computer to complete all ten sets - including training (with parameter

selection) and testing - is shown below the table on the left in h:mm:ss format. The

computer used was an Intel Core i7 920 CPU 2.57 GHz based machine running 64-bit

Microsoft Windows Vista with 6GB of RAM.

We note that the MLP achieved a higher accuracy (90.36%) than the SVM (88.22%) on the

training sets, but performed slightly worse (85.30% vs 87.07%) on the unseen test sets.

The SVM also was more consistent in its test performance across the ten sets, having a

standard deviation of 2.26% compared to the MLP with 4.70%. From a practical

standpoint, the SVM was also preferable due it's much faster execution time (~ 1.5

minutes) compared to the ~25.5 minutes taken by the MLP. Most of the time taken for

both the MLP and SVM was during training where parameter selection takes place. While

it should be noted that the MLP required 25 training set runs per dataset, compared to 16

runs for the SVM, the MLP still took more than an order of magnitude longer even taking

that into account. Although there is no specific requirement for fast real-time performance

in this pattern recognition application, a fast training performance is still convenient.

Having achieved these results using our whole images, we test our system using are larger

set of data, to see if the level of classification accuracy is still maintained. To that end, we

now investigate the performance on training and test sets made up of non-overlapping

image tiles.

6.2.2 Tiled Images

In order to increase the size of the data available for training and testing, the images were

split into four non-overlapping tiles. Performance testing was done for both the MLP and

SVM with the results shown in Table 9 and Table 10, respectively. From column one, we

see that there were approximately four times the amount of images in the training and test

sets compared to the number in the whole image datasets. For example, set one had 384


training images and 183 test images. As previously mentioned, the variation in the amount

of images for training and testing across the different sets was due to the removal of

incorrectly segmented images, or in this case, image tiles.

We notice that for both the MLP and SVM, increasing the data fourfold significantly

improves the accuracy of the training performance as a predictor of performance on the

unseen test data. The difference between training and test performance being only 0.1%

and 0.04% for the MLP and SVM, respectively, compared to 5.06% and 1.15% for whole

images (Table 7 and Table 8). This is also borne out by the tightening of the standard

deviation for training set performance when going from whole images to tiled images for

both the MLP and the SVM.

Table 9 - MLP performance on non-overlapping tiles - Day 1



01 (384/183) 2 0.1 0.2 84.38% 83.61%

02 (380/187) 3 0.4 0.1 84.47% 80.75%

03 (378/189) 2 0.4 0.2 85.45% 83.07%

04 (379/188) 7 0.3 0.2 84.43% 80.32%

05 (375/192) 1 0.2 0.2 84.53% 84.90%

06 (378/189) 1 0.1 0.3 83.86% 87.83%

07 (373/194) 1 0.1 0.3 84.18% 83.51%

08 (376/191) 8 0.3 0.2 84.04% 84.29%

09 (381/186) 1 0.2 0.2 84.51% 86.02%

10 (380/187) 1 0.4 0.2 84.74% 89.30%

Mean 84.46% 84.36%

Std Dev 0.43% 2.82%

Time Taken: 1:08:19

Table 10 - SVM performance on non-overlapping tiles - Day 1


01 (384/183) (<x,y>)^4.0 88.02% 84.70%

02 (380/187) (<x,y>)^3.0 85.26% 86.63%

03 (378/189) (<x,y> + 1)^3.0 85.45% 85.71%

04 (379/188) (<x,y> + 1)^2.0 86.54% 81.91%

05 (375/192) (<x,y>)^3.0 85.60% 83.33%

06 (378/189) (<x,y>)^4.0 83.07% 83.60%

07 (373/194) (<x,y>)^3.0 85.52% 87.63%

08 (376/191) (<x,y> + 1)^3.0 86.97% 85.86%

09 (381/186) (<x,y> + 1)^3.0 85.56% 88.71%

10 (380/187) (<x,y> + 1)^3.0 83.95% 88.24%

Mean 85.59% 85.63%

Std Dev 1.41% 2.25%

Time Taken: 1:30:34


The lower classification accuracy on training performance when using tiles can be

explained by the fact that there are more data points, which provide a better (less

optimistic) prediction of performance on unseen test data, as discussed in the previous

paragraph. We find a much smaller drop in classification accuracy on the unseen test data

between whole images and tiles. For the MLP this is only a drop of 0.94%, and well within

one standard deviation of its performance on both whole images and tiles. For the SVM a

drop of 1.44%, again well with the standard deviation for both whole and tile images.

We should also remember that the increase in dataset size using tile images does not come

without cost. In this case the trade-off is that the tiles are a quarter of the size of the

whole images. Thus it may be that the slight decrease in test set performance using tiles

could also be partly due to the loss of information resulting from reduced image size. In

initial pilot experiments, not reported here, human cell experts (see section 1.6) were

shown tiled images from two different BMSC cell cultures, one more healthy than the

other. Some experts commented that the small image size made it more difficult for them

to assess the quality since they would typically rely on seeing a larger area of the culture,

and hence be able to see more cells at once in order to form an opinion on the quality. This

could also be the case for the machine learning algorithms. Often the tiles, due to their

small size, only show one cell or only parts of a small number of cells. Since not all cells in a

given culture are necessarily at the same level of health, one may by chance be presented

with (during training or testing) an example of a particularly healthy cell within an

otherwise unhealthy culture and vice versa. Seeing a whole image, rather than a much

smaller tile, allows more cells to be viewed and thus a better appreciation of the culture's

overall health to be gained, and therefore allow for a more accurate prediction to be made.

Because of this potential problem of small image size when using tiles, we now look at an

alternate method of increasing the dataset size, while still maintaining relatively larger tile

sizes.

6.2.3 Random Windows

Here we examine a different method of increasing the data size. Instead of breaking up

each whole image into four non-overlapping (but much smaller) tiles, we create images

using a randomly positioned window 80% the size of the original image to sample each


whole image four times. This produces the same number of images to use for evaluation

as the tiling method, but gives images that are larger and much closer to the image size the

system is intended to be used with. The drawback, however, is that each image overlaps

and is guaranteed to share certain regions in common with each of the other three images

created from the same original whole image. The results achieved by the system with

these images is shown below in Table 11 and Table 12.

Table 11 - MLP performance on random windows - Day 1



01 (372/184) 9 0.3 0.2 97.04% 96.20%

02 (374/182) 15 0.3 0.2 95.72% 96.15%

03 (370/186) 14 0.1 0.3 95.94% 98.92%

04 (378/178) 6 0.3 0.1 96.56% 93.26%

05 (366/190) 9 0.1 0.2 96.99% 93.68%

06 (379/177) 13 0.3 0.2 95.51% 96.61%

07 (369/187) 9 0.3 0.2 96.48% 96.26%

08 (367/189) 9 0.3 0.2 96.18% 96.83%

09 (369/187) 11 0.3 0.2 95.66% 97.33%

10 (371/185) 9 0.3 0.2 97.57% 95.14%

Mean 96.37% 96.04%

Std Dev 0.68% 1.67%

Table 12 - SVM performance on random windows - Day 1


01 (372/184) (<x,y> + 1)^4.0 96.24% 95.65%

02 (374/182) (<x,y> + 1)^3.0 96.52% 93.96%

03 (370/186) (<x,y> + 1)^7.0 94.32% 96.24%

04 (378/178) (<x,y> + 1)^3.0 96.03% 95.51%

05 (366/190) (<x,y>)^4.0 95.36% 93.16%

06 (379/177) (<x,y> + 1)^3.0 95.25% 96.05%

07 (369/187) (<x,y>)^8.0 95.94% 94.65%

08 (367/189) (<x,y> + 1)^4.0 94.28% 97.88%

09 (369/187) (<x,y>)^3.0 93.50% 98.93%

10 (371/185) (<x,y>)^4.0 97.57% 94.59%

Mean 95.50% 95.66%

Std Dev 1.22% 1.75%

When looking at the results we observe a surprising increase in recognition accuracy.

Compared to the average test set performance on the non-overlapping tiles (Table 9 and

Table 10), we see an improvement of 11.68% for the MLP and 10.03% for the SVM.

Additionally, the standard deviation of test set performance is also smaller for both


classifiers using random window images. As with the non-overlapping tiles, the average

performance on the training set is very close to that achieved on the test set (0.33% for the

MLP and 0.16% for the SVM).

The large increase in classification performance when using random windows, over both

whole images and non-overlapping tiles, is likely due to the overlapping nature of the

images themselves. It is quite likely that the four random window images, taken from each

original whole image, share similar statistical feature values. Although it's unlikely any of

the images have exactly the same feature values, effectively the system is still likely to have

encountered very close matches in both training and testing. This means that the test data

is not always truly unseen in this case. This could also hold true for training performance

when using cross validation. In fact, the system may be getting away with a certain degree

of overfitting. This appears to be indicated by the increase in average complexity of the

MLP models and to a lesser extent, the SVM models, generated by the training process

when we compare them to the models generated for both the whole images and the non-

overlapping tile datasets. For the MLP, this can be seen in the higher number of hidden

layer nodes used (average of 10.4 nodes, compared to 5.5 nodes and 2.7 nodes for whole

images and non-overlapping tiles, respectively). For the SVM, the degree of the polynomial

kernel was an average 4.3 for the random windows, compared to 2.4 and 3.1 for the whole

images and non-overlapping tiles, respectively. Due to this apparent overfitting, and hence

over-optimistic performance figures, we abandoned the use of the random window

technique in further analyses.

Having tried three different approaches to measuring classification accuracy of our system,

we now present a few brief comments based on our observation of the MLP and SVM

performance results achieved to this point.

6.2.4 MLP versus SVM

When considering the results achieved so far, the SVM achieved slightly higher accuracy on

the unseen test data for both whole images and non-overlapping tiles (we ignore the

random window results for reasons previously mentioned). The SVM also had less variation

in test performance across the 10 different datasets as can be seen by the lower standard

deviation in test data performance figures, compared to the MLP.


However the purpose of our study is not to determine unequivocally the best classifier to

use for BMSC quality assessment. It could be quite possible that an expert in the design of

multilayer perceptrons could improve the performance to beat that achieved for the SVM

used in this study. What is perhaps most noteworthy - more so than the slight superiority

of the SVM observed in this application - is that both classifiers achieved quite similar

results (within ~2% for training and test data performance) for both whole images and

tiles. This consistency adds to our confidence that the results are representative of what

can be achieved relatively easily using state of the art pattern recognition approaches.

So far we have considered datasets that have had incorrectly segmented images manually

removed. We now quantify the (up till now) assumed benefit of doing this, in the next

section.

6.2.5 Effect of Including Incorrectly Segmented Images

We now quantify our assumption that including incorrectly segmented images, in both

training and test datasets, will adversely affect the classification accuracy of the system.

Intuitively, it makes sense that this would be the case, however, since removing incorrectly

segmented images from the datasets takes time, we desire to know whether it is actually

worth doing. That is, if the impact on accuracy is not great, then we may as well leave the

incorrectly segmented images in the datasets. On the other hand, if the impact is great,

then it would be worth removing them, or more specifically, making some kind of

automated way of removing them.

We evaluate the system using whole images taken from day one. The datasets are the

same as those presented in Table 7 and Table 8, but with the inclusion of images where the

segmentation was deemed (from manually inspection) to be incorrect. Results of including

such images in the datasets are shown in Table 13 and Table 14. We note that because no

images have been removed, there are comparatively more images in the datasets for both

training and testing, and that the training and test sets remain consistent in size with 108

and 54 images each, respectively.

We find that the MLP suffers only a very minor drop in accuracy of 0.30% with the inclusion

of incorrectly segmented images, whereas the SVM suffered a very large drop in accuracy


of 4.66%. We propose that this is due to effect of individual training instances on the two

different types of classifiers. The MLP uses all training instances in its determination of the

decision boundary, whereas the SVM's decision boundary is only affected by those

instances that form the support vectors. It is thought that the MLP, therefore, tends to

average out the effect of unusual feature vectors (that result from the measurements

taken from incorrectly segmented images) and so lessen their impact on system accuracy.

The SVM on the other hand, if it uses some of these unusual feature vectors as its support

vectors, could be more adversely affected since the support vectors define the decision

boundary and hence the classification accuracy. The unusual feature vectors (due to their

unusual measurements) would also appear likely to form support vectors, since they would

tend to be distributed at the outer edges of each class in the n-dimensional feature space.

Table 13 - MLP performance on whole images - Day 1 - Segmentation failures included



01 (108/54) 13 0.3 0.1 90.74% 81.48%

02 (108/54) 9 0.4 0.1 94.44% 81.48%

03 (108/54) 4 0.3 0.2 83.33% 85.19%

04 (108/54) 7 0.3 0.1 87.04% 90.74%

05 (108/54) 5 0.3 0.2 89.81% 83.33%

06 (108/54) 6 0.3 0.2 82.41% 92.59%

07 (108/54) 14 0.3 0.3 88.89% 79.63%

08 (108/54) 13 0.3 0.2 90.74% 83.33%

09 (108/54) 7 0.3 0.2 81.48% 83.33%

10 (108/54) 2 0.1 0.1 87.96% 88.89%

Mean 87.69% 85.00%

Std Dev 4.16% 4.32%

Time Taken: 0:26:01

Table 14 - SVM performance on whole images - Day 1 - Segmentation failures included


01 (108/54) (<x,y>)^3.0 92.59% 83.33%

02 (108/54) (<x,y> + 1)^3.0 91.67% 79.63%

03 (108/54) (<x,y>)^4.0 86.11% 81.48%

04 (108/54) (<x,y> + 1)^3.0 84.26% 85.19%

05 (108/54) (<x,y>)^3.0 88.89% 83.33%

06 (108/54) (<x,y>)^7.0 86.11% 77.78%

07 (108/54) (<x,y>)^3.0 83.33% 85.19%

08 (108/54) (<x,y> + 1)^4.0 85.19% 81.48%

09 (108/54) (<x,y> + 1)^4.0 87.04% 79.63%

10 (108/54) (<x,y> + 1)^4.0 82.41% 87.04%

Mean 86.76% 82.41%

Std Dev 3.38% 2.93%

Time Taken: 0:02:12


We would therefore conclude that it is worth having some mechanism to remove

incorrectly segmented images, either manual or automated, when using the SVM classifier.

An automated system is outlined, in Chapter 7, as a possible future development.

Having examined the performance of the system with day one images, we now turn our

attention to how human experts perform on the same images.

6.3 Comparison with Human Experts

In this section we look at how two human experts (refer section 1.6) performed in

differentiating healthy and unhealthy culture images when tested on the same set of

images as our system. For this we used whole images from day one. As per previous

experiments, the images were split into a training set (for training the system) and a test

set for testing both our system and for testing the human experts. Incorrectly segmented

images were removed from both the training set and test set. This gave a training set of 95

images and a test set of 44 images. The test set consisted of 21 healthy cell culture images

and 23 unhealthy cell culture images. The reason for the different number of healthy

versus unhealthy images was again due to the removal of segmentation failures - if

segmentation failures were actually included, then there would have been an even split.

Note that the human experts were not exposed to the training set in any way, the training

set was used solely to train the system, as a substitute for the experts' acquired

knowledge. Our Expert Performance Evaluation Tool software, described in Chapter 5, was

used to carry out the test.

The overall results are shown in Table 15 below. We see that both the MLP and SVM

outperformed the human experts with similar results in accuracy between the two

algorithms. We also see a very large variation in performance between the two human

experts. The first expert achieved an accuracy quite close to that of our system, however

the second human expert fared considerably worse. In comparison to the 2.28%

performance difference between the two machine learning algorithms, the human experts

differed by 25.00%, a very wide variation. To summarise these results, the human experts'

average classification accuracy was 71.6% (standard deviation 17.7%) compared to an

average of 87.5% and standard deviation of 1.6% for the machine learning algorithms.


Thus, in this test, the use of the machine learning algorithms was both more accurate and

more consistent, than human experts.

Table 15 - Human Expert Performance vs System (% images correctly classified)

Test Subject Performance on previously unseen test set (44 images)

Human Expert 1 84.09% (37/44 correct)

Human Expert 2 59.09% (26/44 correct)

MLP 88.64% (39/44 correct)

SVM 86.36% (38/44 correct)

We look further at the classification profile of the four test subjects (two human, two

machine) by considering the confusion matrices for each subject in Table 16 to Table 19

below. Each matrix compares the classifications made by the test subject (columns) to the

actual (ground truth) classifications of the 44 test images (rows). The shaded cells show

the total images for each row and column. For example, in Table 16 the first human expert

classified 16 of the images as healthy (total of the first column). Of these 16 images that

the expert classified as healthy, 15 of them were actually healthy and one was, in fact,

unhealthy (and therefore an incorrect classification made by the expert). The second

column shows that the expert classified the remaining 28 of the images as unhealthy (total

of the second column). Of these 28 images, the expert was correct for 22 of them, which

were indeed unhealthy, but incorrectly judged six of them to be unhealthy when they were

actually healthy.

When comparing the predictions of the human experts (Table 16 and Table 17) with the

machine learning algorithms (Table 18 and Table 19), we notice that when making

classification errors, the human experts were much more likely to classify healthy cultures

incorrectly as unhealthy, whereas the MLP and SVM were the opposite. The machine

learning algorithms, when misclassifying were much more likely to incorrectly classify

unhealthy culture images as healthy. This is an important finding, since we expect, in most

cases, the cost of classifying an unhealthy culture as healthy would be greater than the

converse. For this reason, we suggest in Chapter 7, the introduction of a cost function to

the automated prediction process to help account for this, if further testing on larger

datasets indicates such a requirement.


Table 16 - Confusion Matrix - Human Expert 1

Expert 1's Classification

Healthy Unhealthy

Act

ua

l

Cla

ssif

ica

tio

n

(gro

un

d t

ruth

)

Healthy 15 6 21

Unhealthy 1 22 23

16 28 44

Table 17 - Confusion Matrix - Human Expert 2

Expert 2's Classification

Healthy Unhealthy

Act

ua

l

Cla

ssif

ica

tio

n

(gro

un

d t

ruth

)

Healthy 8 13 21

Unhealthy 5 18 23

13 31 44

Table 18 - Confusion Matrix - MLP

MLP's Classification

Healthy Unhealthy

Act

ua

l

Cla

ssif

ica

tio

n

(gro

un

d t

ruth

)

Healthy 21 0 21

Unhealthy 5 18 23

26 18 44

Table 19 - Confusion Matrix - SVM

SVM's Classification

Healthy Unhealthy

Act

ua

l

Cla

ssif

ica

tio

n

(gro

un

d t

ruth

)

Healthy 20 1 21

Unhealthy 5 18 23

25 19 44


Having looked at the confusion matrices, we now examine more closely the classification

errors made by the four test subjects. We wish to see if the errors are associated with a

particular patient's cells, or if they are associated with a particular culture well, which

might indicate a problem with a particular culture.

Firstly, in Table 20, we look at the number of classification errors by patient sample (A, B or

C) for each test subject. We see that the human experts made errors distributed across all

three patients' cells, whereas the MLP only had problems with patient C cells and the SVM

had problems with patient B and C cells only. Given the limited size of the test set (44

images) it is hard to draw too much conclusion here, other than that there appears to be

no particular external problem with the data. For example, if we found all errors were

clustered around a particular patient's cells for all of the four test subjects, then we could

conclude there may be some problem with one of our patient samples. Fortunately this

does not appear to be the case, both from these results and from the six day cell culture

growth profiles recorded in Chapter 2.

Table 20 - Test subject misclassifications by patient sample

Sample (Patient) Expert 1 Expert 2 MLP SVM

A 2 7 0 0

B 2 4 0 3

C 3 7 5 3

Finally, we break the classifications errors down further, by individual well, including also

the actual classification and patient sample to which the well belonged. The results are

shown in Table 21. Again, we see no particular anomaly here, as no single well is

consistently misclassified by all the test subjects, which might (if it were true) indicate a

problem with a particular cell culture. With this in mind, we are confident there was no

contamination or other extrinsic factors affecting our results, either for the human experts

or the machine learning algorithms.

We have now examined the performance of human experts compared to our system and

shown that our system outperforms the two human experts in this experiment. However,

this is a small size of test images (44) being evaluated as well as a small number of experts.


We would naturally like to evaluate a greater number of experts, and use a larger set of

test images, in order to gain a more accurate estimation of the classification accuracy that

could be expected by human experts in general. Nevertheless, it provides indication that

our system, as expected, gives a more objective measure of quality than human experts

and is thus worthy of further development.

Table 21 - Test subject misclassifications by actual classification, sample and well

Actual

Classification

Patient

Sample

Well Expert 1 Expert 2 MLP SVM

Healthy

(10% FCS)

A 1 0 2 0 0

2 2 2 0 0

3 0 2 0 0

B 1 0 1 0 0

2 0 1 0 0

3 1 1 0 0

C 1 2 1 0 0

2 0 2 0 1

3 1 1 0 0

Unhealthy

(0.2% FCS)

A 1 0 0 0 0

2 0 0 0 0

3 0 1 0 0

B 1 0 0 0 1

2 1 0 0 1

3 0 1 0 1

C 1 0 1 1 1

2 0 2 1 0

3 0 0 3 1

Having established the initial viability of our system as a BMSC quality assessment tool, we

now look at how the system performs with culture images from days one to six. Following

this, we look at which feature groups are the most helpful in discriminating healthy from

unhealthy cultures, before concluding the chapter with a summary.

6.4 System Performance: Days 1 to 6

Up until now, we have concerned ourselves solely with images taken from day one of the

culturing process, and have not considered images taken from days two to six. We now

examine the performance of the system (using both the MLP and the SVM) for each day

from day one to day six. For this exercise, we used whole images and did not remove


segmentation failures for two reasons. Firstly, the manual removal of them all for six days

worth of images would have been excessively time consuming. Secondly, healthy cells

become more confluent in later days, resulting in more segmentation failures. Thus

removing segmentation failures would leave a much smaller amount of data in our

datasets. In producing our results, we created 10 different training/test data splits as we

have done previously. For each given day, only images taken on that day were used in

training and testing.

For the sake of clarity, we have moved the twelve tables containing the parameters and

training/test performance figures to Appendix B and provided just a summary showing the

average performance for each day achieved by the MLP and the SVM in Table 22 and chart

these results in Figure 38 below.

Table 22 - MLP and SVM Test set performance on whole images - Days 1 to 6

Day MLP SVM

1 85.00% 82.41%

2 89.26% 88.52%

3 87.04% 91.67%

4 86.85% 88.15%

5 92.04% 94.07%

6 93.15% 94.63%

Figure 38 - Performance over Days 1 to 6 (segmentation failures included)

76.00%

78.00%

80.00%

82.00%

84.00%

86.00%

88.00%

90.00%

92.00%

94.00%

96.00%

1 2 3 4 5 6

% C

orr

ect

(U

nse

en

Te

st D

ata

)

Day

MLP

SVM


We note that day one provides the least accuracy compared to all subsequent days and

that as an overall trend, accuracy improves as the culture progresses with accuracy

beginning to level out in days five and six. Within this, however, we notice a dip in the

accuracy midway through the culturing process around day four for the SVM and around

day three and day four for the MLP. The reason for this dip in accuracy is not clear as the

cells do not appear particularly unusual during this period, compared to the other days.

The most probable explanation is that days three and four represent a transition in the

proportion of cell images that do not segment correctly. On days one and two when the

cells are fewer in number, the majority of images are correctly segmented, providing a

more consistent distribution of feature vectors for each class. Likewise, at days five and

six, the majority of cell images (at least for healthy cultures) would not segment correctly

due to the large number of cells in each image (refer 3.5.4), meaning the majority of

feature vectors representing them would be formed from measurements associated with

incorrectly segmented images. Thus, like in days one and two, there would be a consistent

distribution of feature vectors for each class. In days three and four, however, there is

likely to be a more even proportion of correctly segmented images compared to incorrectly

segmented images. This would result in a greater distribution of feature vectors in the

feature space, making classification more difficult.

6.5 Performance by Feature Groups

Up till now, we have used all 25 features (refer 4.2.5) when assessing performance. These

have consisted of eight total cell area features, eight inner cell area features, eight outer

cell area features and the mixture modelling threshold feature. Given the favourable

recognition accuracy achieved by our system using this entire feature set, in comparison to

that achieved by human experts (section 6.3), there did not seem to be an imperative to

perform a rigorous and time-consuming feature selection and evaluation process in an

attempt to improve accuracy. Nevertheless, it was still felt worthwhile to investigate which

of the three cell regions were most indicative of culture quality. Therefore, we now

investigate the effect on accuracy of these feature groups by themselves to see which are

most useful in discriminating healthy and unhealthy cultures.


For these experiments we have used whole images from day one only and have not

included images that were not correctly segmented. Our benchmark is the results

achieved using all features (Table 7 and Table 8). To recap, when using all 25 features, for

the MLP we achieved an average performance of 85.30% and for the SVM an average

performance of 87.07% of images correctly classified, when using previously unseen test

data. We now consider the accuracy achieved using only the total cell area features.

6.5.1 Total Cell Area Features Only

Here we see the performance of the system using only the eight total cell area features in

Table 23 and Table 24 below. Using these features alone gives an accuracy of

approximately 75% for both learning algorithms, less accurate than the entire feature set.

Table 23 - MLP performance on whole images - Day 1 – Total cell area features only



01 (94/45) 6 0.3 0.2 79.79% 66.67%

02 (93/46) 13 0.1 0.2 81.72% 71.74%

03 (94/45) 7 0.3 0.2 78.72% 77.78%

04 (94/45) 11 0.3 0.2 74.47% 77.78%

05 (94/45) 10 0.4 0.2 88.30% 77.78%

06 (95/44) 3 0.1 0.3 75.79% 72.73%

07 (96/43) 11 0.2 0.2 76.04% 79.07%

08 (91/48) 14 0.1 0.2 74.73% 75.00%

09 (91/48) 6 0.2 0.2 79.12% 81.25%

10 (91/48) 3 0.3 0.2 79.12% 72.92%

Mean 78.78% 75.27%

Std Dev 4.11% 4.32%

Time Taken: 0:14:44

Table 24 - SVM performance on whole images - Day 1 – Total cell area features only


01 (94/45) (<x,y> + 1)^4.0 78.72% 75.56%

02 (93/46) (<x,y> + 1)^4.0 81.72% 73.91%

03 (94/45) (<x,y> + 1)^4.0 80.85% 73.33%

04 (94/45) (<x,y> + 1)^5.0 84.04% 77.78%

05 (94/45) (<x,y> + 1)^6.0 84.04% 62.22%

06 (95/44) (<x,y> + 1)^7.0 80.00% 77.27%

07 (96/43) (<x,y> + 1)^8.0 78.13% 76.74%

08 (91/48) (<x,y> + 1)^4.0 79.12% 81.25%

09 (91/48) (<x,y> + 1)^5.0 85.71% 79.17%

10 (91/48) (<x,y> + 1)^8.0 73.63% 81.25%

Mean 80.60% 75.85%

Std Dev 3.53% 5.49%

Time Taken: 0:03:57


6.5.2 Inner Cell Area Features Only

The performance using only the inner cell area features is shown below in Table 25 and

Table 26. As we can see, the inner cell area features are even less indicative of culture

quality than the total cell area features, achieving accuracies of around 74% for the MLP

and around 70% for the SVM.

Table 25 - MLP performance on whole images - Day 1 – Inner cell area features only



01 (94/45) 6 0.5 0.2 80.85% 77.78%

02 (93/46) 5 0.3 0.5 82.80% 76.09%

03 (94/45) 10 0.3 0.2 79.79% 62.22%

04 (94/45) 1 0.3 0.2 79.79% 73.33%

05 (94/45) 5 0.3 0.2 80.85% 71.11%

06 (95/44) 7 0.3 0.2 83.16% 75.00%

07 (96/43) 1 0.3 0.1 79.17% 79.07%

08 (91/48) 1 0.1 0.1 83.52% 70.83%

09 (91/48) 1 0.1 0.1 78.02% 75.00%

10 (91/48) 8 0.5 0.1 74.73% 77.08%

Mean 80.27% 73.75%

Std Dev 2.65% 4.86%

Time Taken: 0:09:13

Table 26 - SVM performance on whole images - Day 1 – Inner cell area features + MMThres only


01 (94/45) (<x,y>)^7.0 79.79% 68.89%

02 (93/46) (<x,y> + 1)^5.0 80.65% 80.43%

03 (94/45) (<x,y> + 1)^8.0 85.11% 68.89%

04 (94/45) (<x,y>)^8.0 81.91% 62.22%

05 (94/45) (<x,y> + 1)^8.0 77.66% 64.44%

06 (95/44) (<x,y> + 1)^4.0 81.05% 77.27%

07 (96/43) (<x,y> + 1)^7.0 79.17% 76.74%

08 (91/48) (<x,y> + 1)^3.0 81.32% 66.67%

09 (91/48) (<x,y> + 1)^2.0 81.32% 62.50%

10 (91/48) (<x,y>)^4.0 73.63% 75.00%

Mean 80.16% 70.31%

Std Dev 3.01% 6.60%

Time Taken: 0:01:47


6.5.3 Outer Cell Area Features Only

Below, in Table 27 and Table 28, we see that the outer cell area features are the most

discriminative of the three cell regions examined, with both the MLP and SVM achieving

around 83% accuracy.

Table 27 - MLP performance on whole images - Day 1 – Outer cell area features only



01 (94/45) 11 0.3 0.2 82.98% 80.00%

02 (93/46) 12 0.3 0.2 84.95% 78.26%

03 (94/45) 2 0.3 0.1 80.85% 86.67%

04 (94/45) 10 0.1 0.1 84.04% 86.67%

05 (94/45) 11 0.3 0.2 90.43% 77.78%

06 (95/44) 2 0.2 0.1 85.26% 81.82%

07 (96/43) 15 0.2 0.2 85.42% 86.05%

08 (91/48) 2 0.5 0.2 84.62% 87.50%

09 (91/48) 9 0.3 0.2 85.71% 85.42%

10 (91/48) 3 0.3 0.1 86.81% 83.33%

Mean 85.11% 83.35%

Std Dev 2.49% 3.67%

Time Taken: 0:10:15

Table 28 - SVM performance on whole images - Day 1 – Outer cell area features only


01 (94/45) (<x,y>)^5.0 90.43% 82.22%

02 (93/46) (<x,y> + 1)^6.0 81.72% 76.09%

03 (94/45) (<x,y> + 1)^7.0 84.04% 84.44%

04 (94/45) (<x,y> + 1)^3.0 85.11% 91.11%

05 (94/45) (<x,y> + 1)^6.0 87.23% 73.33%

06 (95/44) (<x,y>)^6.0 80.00% 90.91%

07 (96/43) (<x,y>)^3.0 86.46% 83.72%

08 (91/48) (<x,y>)^3.0 84.62% 77.08%

09 (91/48) (<x,y>)^4.0 85.71% 79.17%

10 (91/48) (<x,y> + 1)^4.0 82.42% 87.50%

Mean 84.77% 82.56%

Std Dev 2.98% 6.15%

Time Taken: 0:01:43

6.5.4 Mixture Modelling Threshold Feature Only

For completeness, we consider the accuracy using just the mixture modelling threshold

alone. The results are shown in Table 29 and Table 30. Although we would not expect a

single feature to produce a high recognition accuracy, we still see the feature does have


some discriminative power, giving an accuracy in the order of 60% between the two

classifiers. We thus feel its inclusion in our full feature set is justified.

Table 29 - MLP performance on whole images - Day 1 – Mixture Modelling Threshold feature only



01 (94/45) 1 0.1 0.1 57.45% 73.33%

02 (93/46) 2 0.1 0.1 62.37% 63.04%

03 (94/45) 4 0.1 0.1 61.70% 64.44%

04 (94/45) 1 0.1 0.1 64.89% 57.78%

05 (94/45) 2 0.1 0.1 63.83% 57.78%

06 (95/44) 1 0.1 0.1 66.32% 54.55%

07 (96/43) 1 0.1 0.1 63.54% 60.47%

08 (91/48) 2 0.1 0.2 62.64% 62.50%

09 (91/48) 4 0.1 0.1 65.93% 56.25%

10 (91/48) 1 0.1 0.1 60.44% 68.75%

Mean 62.91% 61.89%

Std Dev 2.66% 5.85%

Time Taken: 0:03:47

Table 30 - SVM performance on whole images - Day 1 – Mixture Modelling Threshold feature only


01 (94/45) (<x,y>)^2.0 59.57% 66.67%

02 (93/46) (<x,y>)^1.0 62.37% 56.52%

03 (94/45) (<x,y> + 1)^2.0 62.77% 64.44%

04 (94/45) (<x,y> + 1)^2.0 64.89% 57.78%

05 (94/45) (<x,y> + 1)^8.0 65.96% 53.33%

06 (95/44) (<x,y>)^1.0 66.32% 54.55%

07 (96/43) (<x,y>)^2.0 63.54% 60.47%

08 (91/48) (<x,y> + 1)^4.0 62.64% 62.50%

09 (91/48) (<x,y> + 1)^1.0 65.93% 56.25%

10 (91/48) (<x,y>)^1.0 60.44% 58.33%

Mean 63.44% 59.08%

Std Dev 2.34% 4.35%

Time Taken: 0:00:29

6.5.5 Outer Cell Area Features and Mixture Modelling Threshold Only

Given the outer cell area group of features was found to be the better performer of the

three cell area feature groups, we have combined this group's features with the mixture

modelling threshold. The performance achieved using this nine feature combination is

shown in Table 31 and Table 32. We see the inclusion of the mixture modelling threshold


further improves the accuracy of the outer cell area feature group, giving an overall

accuracy that compares very favourably to the results from our entire feature set.

Table 31 - MLP performance on whole images - Day 1 – Outer cell area features + MMThres only



01 (94/45) 14 0.3 0.2 94.68% 88.89%

02 (93/46) 6 0.3 0.1 88.17% 86.96%

03 (94/45) 7 0.1 0.2 88.30% 95.56%

04 (94/45) 5 0.3 0.2 92.55% 88.89%

05 (94/45) 9 0.2 0.2 89.36% 91.11%

06 (95/44) 3 0.2 0.3 86.32% 84.09%

07 (96/43) 2 0.2 0.2 89.58% 86.05%

08 (91/48) 1 0.3 0.2 92.31% 81.25%

09 (91/48) 4 0.3 0.5 92.31% 93.75%

10 (91/48) 5 0.3 0.2 92.31% 87.50%

Mean 90.59% 88.40%

Std Dev 2.61% 4.29%

Time Taken: 0:09:53

Table 32 - SVM performance on whole images - Day 1 – Outer cell area features + MMThres only


01 (94/45) (<x,y>)^3.0 90.43% 84.44%

02 (93/46) (<x,y> + 1)^3.0 88.17% 95.65%

03 (94/45) (<x,y> + 1)^5.0 86.17% 86.67%

04 (94/45) (<x,y>)^3.0 84.04% 88.89%

05 (94/45) (<x,y> + 1)^3.0 91.49% 82.22%

06 (95/44) (<x,y> + 1)^3.0 88.42% 90.91%

07 (96/43) (<x,y>)^6.0 88.54% 83.72%

08 (91/48) (<x,y>)^2.0 89.01% 77.08%

09 (91/48) (<x,y> + 1)^3.0 91.21% 87.50%

10 (91/48) (<x,y> + 1)^3.0 89.01% 89.58%

Mean 88.65% 86.67%

Std Dev 2.25% 5.14%

Time Taken: 0:01:14

The results achieved using the different feature sets are summarised in Table 33 below.

We note that all feature groups were individually useful, however, of the three cell area

feature groups, the outer cell area group was most useful for discrimination of healthy and

unhealthy cultures. When this feature group was combined with the mixture modelling

threshold feature, the accuracy of the MLP was actually increased by over 3% compared to

its accuracy when using the full feature set. The SVM accuracy remained almost the same

with only a small drop in accuracy of 0.4%. Therefore, it would appear that it is some

aspect of the outer part of the cell bodies that the system finds most useful in culture


health discrimination, at least on day one. This may reflect early signs of the somewhat

emaciated appearance unhealthy cells appear to take on over time, which, based on our

empirical observations, becomes more clearly noticeable on the outer parts of the cells

than within the bodies of the cells.

Given these results, it would be recommended to compare the use this subset of nine

features to the full feature set in future experiments.

Table 33 - MLP and SVM Test set performance on whole images - Day 1 - By feature groups

Features Used MLP SVM

Benchmark - All Features (25) 85.30% 87.07%

Total Cell Area Features Only (8) 75.27% 75.85%

Inner Cell Area Features Only (8) 73.75% 70.31%

Outer Cell Area Features Only (8) 83.35% 82.56%

Mixture Modelling Threshold Only (1) 61.89% 59.08%

Outer Features + MM Threshold (9) 88.40% 86.67%

6.6 Summary

In this chapter we have evaluated the accuracy of our BMSC quality assessment system.

Firstly, we examined the accuracy of the system using whole images, followed by a larger

dataset made up of smaller image tiles extracted from the whole images, for the first day

of culture. Both the MLP and SVM were comparable in performance, achieving

approximately 85% and 87% of images correctly classified, respectively, for the whole

images, and 84% and 86%, correspondingly for the smaller tiles. Given that the use of

smaller image tiles to create a larger dataset also involved a loss of information, due to the

image size reduction, we feel that it may have also slightly lowered the accuracy of the

result as well. With that in mind, we feel it is reasonable to average all four figures to give

an overall indication of our system accuracy as 86% for images taken from day one of

culture. We also examined the effect on system accuracy of including incorrectly

segmented images in both the training and test sets. As expected, this reduced the

accuracy of the system. In the case of the MLP, the reduction in accuracy was almost

negligible (0.30%), whereas the SVM suffered a larger drop in accuracy of 4.66%.


We next compared the performance of our system with two human experts on a set of 44

previously unseen test images. In this test, our system outperformed both human experts.

There was also a very large variation in classification accuracy between the two human

experts (approximately 25%). In addition to this, we examined the classification errors of

both the two human experts and the MLP and SVM. We noted the human experts tended

to mistake healthy cells for unhealthy ones, and the machine learning algorithms tended to

mistake unhealthy for healthy when making incorrect classifications. We therefore

suggested the addition of cost function to our system, to take this into account. We also

examined the classification errors and found they were not limited to particular patients or

culture wells, so we are confident our results are not skewed by any unexpected extrinsic

factor such as contaminated cultures, etc.

Following this, we measured the performance of the system for each day of the six days of

culture. We found that the first day of culture produced the least accurate results with a

general increase in accuracy with time in culture, with the highest accuracy of

approximately 93% and 95% being achieved on day six for the MLP and SVM, respectively.

Lastly, we looked at which feature groups were most useful in our task of BMSC quality

assessment. We evaluated the total cell area, inner cell area, outer cell area and mixture

modelling threshold feature groups, separately. We found the outer cell area feature group

to be the most useful to the system in discrimination of cell culture quality. We found the

combination of the eight outer cell area features and the mixture modelling threshold gave

favourable results compared to the use of the entire feature set, using images from the

first day of culture. We would therefore say that the system relies most heavily on

characteristics associated with the outer part of the cell bodies rather than the deeper

areas. We also recommended that this reduced feature set also be considered in future

trials of system performance.

This includes our presentation and discussion of results. In the final chapter, we present

the conclusions of our thesis and discuss recommendations for future research and

development work.

107

Chapter 7

Conclusions and Future Work

In this final chapter we look back upon the work done in this thesis and draw conclusions

relating to our original aim and the research questions raised in our introduction. We also

review the main contributions of this work and conclude by considering possible avenues

of future research.

7.1 Conclusions

The aim of this thesis was to find out whether it is possible through the use of image

processing and pattern recognition techniques to predict the growth potential of a

culture of human mesenchymal stem cells at early stages, before it is readily apparent to

a human observer. The results of the research covered by this thesis show that it is indeed

possible for a computerised image processing and pattern recognition system to predict

the growth potential of mesenchymal stems cells more accurately and earlier than a

human expert. Our system, when presented with 44 previously unseen bone marrow

stromal cell culture images, outperformed human experts in the ability to correctly classify

healthy and unhealthy cultures. The system correctly classified the health status of an

image 88% of the time compared to an average of 72% of the time for human experts.

Extensive training and testing of the system on a set of 139 normal sized images and 567

smaller image tiles showed an average performance of 86% and 85% correct classifications,

respectively, on previously unseen test images and image tiles.

However, it should be noted that we have demonstrated the feasibility of this in a

constrained situation where the quality of healthy and unhealthy cells was due to

deliberate manipulation of the amount nutrients available to the cells. More work would

108 Chapter 7. Conclusions and Future Work

need to be performed to determine whether the existing system would achieve the same

accuracy in the more generalised case, where the cell health variations are due solely to

natural variation of growth characteristics. Nevertheless, this work has demonstrated that

the potential is there, and worthy of further investigation. Ideas for further development

are outlined later in this chapter (Section 7.3).

In our introduction to this thesis (Chapter 1), a number of research questions were also

raised. We now address those questions below.

7.1.1 How should cell culture images be captured that are likely to contain

useful predictive information?

It was quickly determined (in Chapter 2) that a 10x objective magnification was the most

practical of those available for use in this project. At this magnification it was possible to

capture a reasonable number of cells in each image. Although images were captured at the

camera's highest resolution of 2208 x 3000 pixels, the images were scaled to a resolution of

552 x 750 pixels for ease of processing, and yet were still found to be quite suitable for the

evaluation of cell quality.

Since it was found that a time-series of images was not necessary for the computerised

determination of stem cell quality, there was found to be no need to maintain a

consistency of view over successive days. Furthermore, during pilot experiments where a

grid was used in order to photograph the same locations within a cell culture at 24-hour

intervals, it was found that the cells moved so much within this period that it would be

virtually impossible to track individual cells at this frequency. Since neither the location

nor the tracking of individual cells was found to be required for culture quality assessment,

there was little need to investigate this avenue any further in this thesis.

To summarise, we found that it is only necessary to image the cells once - there is no need

to use a time series of images - in order to classify the quality of cell cultures. A 10x phase

contrast objective lens is suitable for this task. Images taken during the first day of a bone

marrow stromal cell culture can potentially be used successfully by the system to

distinguish between healthy and unhealthy cultures, with a higher accuracy than human

experts.

Chapter 7. Conclusions and Future Work 109

7.1.2 What useful measurable attributes can be extracted from the captured

images?

Prior to extraction of features, it was first necessary to separate the parts of the images

that contained cells from the parts that contained background. To this end, in Chapter 3,

we developed and evaluated an image normalisation and segmentation algorithm. This

algorithm was able to locate cell regions with good success in most cases. It was also able

to deal with the image areas containing the bright halo associated with the use of phase

contrast, such that these areas would not be included within the segmented area.

Nevertheless, the irregular shape of the cells made it very difficult to segment them

individually where they touched. This was more problematic as the cells grew in number.

During development of the normalisation and segmentation algorithm in Chapter 3, a set

of simple statistical features (mean, mode, median, standard deviation, skew and kurtosis)

were used to gauge the effectiveness of the algorithm for the purpose of culture quality

discrimination. It was found the simple statistical measures performed quite well on the

dataset available at that time, hence the use of such features was continued and expanded

further. We also determined that the bright phase contrast halo, which varies across

different parts of cells, does not contain useful information for cell culture quality

assessment.

When our main dataset was formed (as described in Chapter 2), the initial statistical

features did not perform quite as well on the expanded dataset. To deal with this, several

more statistical features were added (entropy, uniformity and the mixture modelling

threshold value from the segmentation algorithm). Additionally, in order to include some

spatial information, cell regions were split into inner and outer regions. The depth of the

outer region was varied with a depth corresponding approximately to the average width of

the cell projections. This new expanded set of features worked well, allowing our system

to outperform human experts in correctly assessing the health of culture images from the

first day of our culture process, some ten hours after culture health changing effects

(altered nutrient levels) were introduced. A particular strength of the statistical features

used was that they did not require individual segmentation of cells and could therefore

deal with touching or partially visible cells.


7.1.3 How can sets of attributes extracted from captured images be classified in

order to predict the outcome of a cell culture?

We decided upon two overall classes for our cell cultures, healthy or unhealthy. Of course,

these terms are subjective, so more specifically, we considered those cells cultured with

the normal amount (10%) of fetal calf serum (FCS) to be healthy, and those cultured in a

greatly reduced amount (0.2%), to be unhealthy. This was felt to be sufficient for the

preliminary nature of this research.

In order to perform this overall classification of an image, it was decided not to attempt to

classify individual cells. This was due to the difficulty in segmenting touching cells and

handling the cases where only parts of cells were visible. Another aspect is that we are

generally interested in the health of the culture as a whole, rather than that of individual

cells, since it would be difficult to extract individual healthy cells from the culture. Finally,

it was shown in our results, that overall culture health could be determined without the

need to individually determine the health of each individual cell.

In terms of the actual algorithms used for the task of classification, we found the support

vector machine (SVM) and multilayer perceptron (MLP) both to be quite suitable. Both

machine learning algorithms outperformed the human experts in our testing and generally

achieved comparable results in terms of overall classification accuracy.

Since the success of the pattern recognition system depends to a large degree on the

quality of the data used to train it, it was important to confirm that our examples of

healthy cells and unhealthy cells were actually healthy and unhealthy cells. To confirm

this, counts of cells in each image for each well were taken for each day in culture over the

six days that the cells were monitored. These counts were then plotted on a graph (see

Chapter 2, section 2.3.3). It could be clearly seen that the cells with only 0.2% FCS were

indeed slower growing and hence less healthy than the cells with 10% FCS, thus validating

our training data.

In testing the accuracy of our system over six successive days of culture, we found that the

system was the least accurate on day one and thereafter generally increased in

performance up to the final day of culture (day six) which produced the most accurate


performance. Nevertheless, even on day one, where the system was least accurate, it still

achieved a relatively high accuracy, outperforming the human experts. The fact that the

system is useful on day one of culture and also that it does not require a time-series of

images is important, since it allows the system to make an assessment very early in the

culture process and with a minimum of fuss, thus increasing its practical usefulness.

7.2 Thesis Contribution

The contribution of this thesis is summarised in the five points below:

• Have demonstrated the applicability and potential of computerised image

processing and pattern recognition techniques to the task of bone marrow stromal

cell culture quality assessment.

• A new segmentation algorithm has been developed for locating cell regions of

irregularly shaped cells in phase contrast images. This is useful, not only as a

precursor to our quality assessment application, but also potentially as a

measurement aid to help cell researchers make more objective measurements of

cell area, perimeter and so on.

• Importantly, we have validated the efficacy of our choice of normalisation and

segmentation method, by demonstrating that both methods quantitatively

improve the classification performance of subsequent pattern recognition

algorithms, in discriminating between cell cultures of differing health status.

• Have established that it is not necessary to segment individual cells in an image of

a bone marrow stromal cell culture, in order for a pattern recognition system to

classify it on the basis of its health state. Segmenting cell regions with the cell

image was found to be sufficient.

• Have proposed a set of features that when extracted from segmented input images

can be used to train current state of the art pattern recognition systems to predict

the quality of bone marrow stromal cell cultures earlier and more consistently than

human experts.


7.3 Future Work

This thesis represents a preliminary, rather than a concluding investigation into the

computerised assessment of bone marrow stromal cells. With that in mind, the following

possibilities for further research are presented below.

7.3.1 Automatic Removal of Segmentation Failures

In section 6.2.5 we found that the inclusion of incorrectly segmented images in our training

and test data did have a detrimental effect on performance, as expected. Removal of

these poorly segmented images has thus far required manual inspection of all images in

the dataset, following the segmentation process. This is a time consuming process and it

would be desirable to remove this burden from the user of the system. A possible system

for removing such images is outlined in Figure 39 below.

Figure 39 - Existing system modified to include extra segmentation validation stage

In Figure 39, a new segmentation validation stage is added to our quality assessment

system. This stage would utilise a pattern recognition function and be similar in operation

to that used for the classification of healthy and unhealthy culture images. However,

instead of classifying images according to culture health, this new stage would classify

them according to whether the segmentation was considered acceptable or not. Those

images classified as having unacceptable segmentation would be rejected and not used for

training or testing of the later culture quality assessment pattern recognition stage. As this

new segmentation validation stage uses pattern recognition, it would also need to be

trained prior to use. The existing manually marked dataset could be used for such a

purpose. We would suggest using the existing feature set for this purpose, with perhaps

� ℎ�� ℎ��ℎ�� ℎ��

Image

Processing

Feature

Extraction

Pattern

Recognition

� ℎ�� ℎ��ℎ�� ℎ��

Input

Image

Input

Classification

(training)

Output

Classification

(testing)

Segmentation

Validation


the addition of some extra features such as the total cell region area and perimeter

(suitably normalised against the total image size), the number of separate objects and so

on.

7.3.2 Expansion of Cell Image Data

As mentioned earlier in this chapter, it would be desirable to expand the available cell

image data to include cells from more donors or even from different sites of the same

donor. It would be hoped that with a larger number of cell samples, there may be

sufficient variation in cell growth rates such that natural examples of both healthy and

unhealthy may be present. This would remove the need to artificially induce cells to be

unhealthy and thus help in determining and improving the general applicability of the

system.

7.3.3 Further Expert Evaluation

We would also like to evaluate the performance of a larger number of cell biology experts

to gain a better measure of general performance for comparison against our system. It

would also be beneficial to present each expert with a larger test set in order to gain a

more accurate measure of the performance of each individual expert.

7.3.4 Further Biological Evaluation

As mentioned earlier in this thesis, the current study is to generate some preliminary data

to investigate the possibility of computerized assessment of bone marrow cultures. For this

purpose, healthy and unhealthy cell cultures were based on the nutrient supply by adding

10% or 0.2% FCS. In our future study, bone marrow samples will be collected and

biologically assessed in cell proliferation, cell metabolism, and cell differentiation

potentials to determine the different healthy status of bone marrow samples. These

biological assessments have been widely used in stem cell cultures and characterization.

7.3.5 Selection and Biological Interpretation of Features

In Chapter 6, we looked briefly at which groups of features in our total feature set were

most helpful in discrimination between healthy and unhealthy cultures. From our results it

appears that the outer cell areas are most helpful. It would be useful to take this

investigation further to see which subset of outer cell area features (mean, mode, median,


standard deviation, skew, kurtosis, uniformity, and entropy) is most helpful. This would

have two benefits as outlined below.

Firstly, using only the most useful features may increase overall system classification

accuracy. Secondly, isolating the most useful features may also help clarify the

corresponding biological significance behind the decisions made by our quality assessment

system. Such understanding may help in training human users during visual inspection of

cell cultures.

7.3.6 Classification Confidence

The system developed in this thesis simply provides a binary classification, e.g. healthy or

unhealthy. It would be advantageous to also provide a measure of confidence, such as the

probability that an image represents a healthy culture versus the probability that it

represents an unhealthy culture. In this way, the researchers could determine at what

level of confidence they will require the classification to be, depending upon their own

application. For example, if the cost of an incorrect classification is high, then they can

correspondingly increase the level of certainty required before they accept the

classification.

Under the present system, this change would output the probability of each classification

with the classification having the highest probability being the one chosen. However, the

actual cost of making a wrong decision is not always the same. For example, incorrectly

classifying a healthy culture as unhealthy may have a different impact compared to

incorrectly classifying an unhealthy culture as being healthy. In order to deal with this

situation, a cost function can be included as part of the prediction process. Under this

scheme, the probabilities of each classification are combined with the cost function so that

the classification that minimises the cost is the one that is chosen. The cost function itself

would need to be determined by the domain experts taking into account the various

impacts of making incorrect classification in terms of lost time, money, risk to patient and

so on, for both types of incorrect classification.


7.4 Final Remarks

This thesis has shown the potential of an automated image processing and pattern

recognition based system for bone marrow stromal cell culture quality assessment. Given

the exciting potential of these cells for the development of innovative new medical

treatments, we hope that the work presented in this thesis may, in some measure, assist in

bringing the reality of such treatments a little closer.



117

Appendix A

Definition of Feature Statistics

This appendix provides definitions of the statistical measures used a features in our

project. The interested reader is referred to [17], [52] and [53] for a more detailed

coverage.

Mean

The mean, is given by the formula:

�� = 1��

In the above, n is the number of pixels in the region for which the mean is being calculated,

and xi is the gray level (intensity) of the ith pixel.

Mode

The mode is the most commonly occurring sample value in a distribution. For our

purposes, it represents the most commonly occurring gray level in an image.

Median

The median is the middle value of a set of data values when the data set is arranged in

ascending order. It is more robust in the presence of outlier samples than the mean.

118 Appendix A.

Standard Deviation

The standard deviation gives a measure of the average distance each sample is from the

mean and gives an indication of the spread of values. It is a little more intuitive to use than

the variance σ2

since its units are the same as the mean. Its formula is given by:

� = �1��(�� − ��)��

In the above, n is the number of pixels in the region for which the standard deviation is

being calculated, xi is the gray level (intensity) of the ith pixel, �� is the mean gray level of all

the pixels in the region, as described earlier.

Skew

The skew is a measure of the symmetry (or lack thereof) of a distribution.

�� = 1��(�� − ��)��

Here n is the number of pixels in the region for which the skew is being calculated, xi is the

gray level (intensity) of the ith pixel, �� is the mean gray level, and � is the standard

deviation of all the pixels in the region, as described earlier.

Kurtosis

Kurtosis is a measure of the peakiness (or flatness) of a distribution. Its formula is:

�� = 1�� (�� − ��) − 3��

Where n is the number of pixels in the region for which the kurtosis is being calculated, xi is

the gray level (intensity) of the ith pixel, �� is the mean gray level, and � is the standard

deviation of all the pixels in the region, as described earlier. Strictly speaking, this form is


called the excess kurtosis, with the subtraction of 3 resulting in a normal distribution

having a kurtosis of 0.

Entropy

Entropy is a measure of the randomness of a distribution. The entropy is 0 for an image of

constant gray level. In this case, the value of a pixel chosen at random from the image is

100% predictable. Its formula is:

= −�"(#�) log� "(#�)'(��)

Here the variable g denotes the gray levels in an image and p(gi), i = 0, 1, 2, ..., (L - 1)

represents the corresponding histogram, where L is number of gray levels (= 256 for an 8

bit gray level image).

Uniformity

Uniformity is a measure of the homogeneity of gray levels in an image. The uniformity

measure is maximum for an image in which all gray levels are the same and decreases from

there. Its formula is given by:

* =�"�(#�)'(��)

Where the variable g denotes the gray levels in an image and p(gi), i = 0, 1, 2, ..., (L - 1)

represents the corresponding histogram, where L is number of gray levels (= 256 for an 8

bit gray level image).

120 Appendix A.


121

Appendix B

Additional Performance Data

This appendix contains results data that are summarised in section 6.4. The data in Table

34 to Table 45 shows the performance of the system for each of the six days of culture.

122 Appendix B.

Performance on Day 1




01 (108/54) 13 0.3 0.1 90.74% 81.48%

02 (108/54) 9 0.4 0.1 94.44% 81.48%

03 (108/54) 4 0.3 0.2 83.33% 85.19%

04 (108/54) 7 0.3 0.1 87.04% 90.74%

05 (108/54) 5 0.3 0.2 89.81% 83.33%

06 (108/54) 6 0.3 0.2 82.41% 92.59%

07 (108/54) 14 0.3 0.3 88.89% 79.63%

08 (108/54) 13 0.3 0.2 90.74% 83.33%

09 (108/54) 7 0.3 0.2 81.48% 83.33%

10 (108/54) 2 0.1 0.1 87.96% 88.89%

Mean 87.69% 85.00%

Std Dev 4.16% 4.32%

Time Taken: 0:26:01



01 (108/54) (<x,y>)^3.0 92.59% 83.33%

02 (108/54) (<x,y> + 1)^3.0 91.67% 79.63%

03 (108/54) (<x,y>)^4.0 86.11% 81.48%

04 (108/54) (<x,y> + 1)^3.0 84.26% 85.19%

05 (108/54) (<x,y>)^3.0 88.89% 83.33%

06 (108/54) (<x,y>)^7.0 86.11% 77.78%

07 (108/54) (<x,y>)^3.0 83.33% 85.19%

08 (108/54) (<x,y> + 1)^4.0 85.19% 81.48%

09 (108/54) (<x,y> + 1)^4.0 87.04% 79.63%

10 (108/54) (<x,y> + 1)^4.0 82.41% 87.04%

Mean 86.76% 82.41%

Std Dev 3.38% 2.93%

Time Taken: 0:02:12

Appendix B. 123





01 (108/54) 9 0.3 0.1 90.74% 87.04%

02 (108/54) 5 0.2 0.1 91.67% 85.19%

03 (108/54) 13 0.1 0.5 90.74% 85.19%

04 (108/54) 3 0.1 0.2 87.96% 92.59%

05 (108/54) 10 0.3 0.2 87.96% 92.59%

06 (108/54) 8 0.2 0.1 88.89% 94.44%

07 (108/54) 8 0.3 0.1 89.81% 92.59%

08 (108/54) 12 0.5 0.2 89.81% 90.74%

09 (108/54) 7 0.3 0.1 91.67% 88.89%

10 (108/54) 11 0.1 0.1 92.59% 83.33%

Mean 90.19% 89.26%

Std Dev 1.59% 3.88%

Time Taken: 0:29:46



01 (108/54) (<x,y>)^3.0 91.67% 88.89%

02 (108/54) (<x,y>)^8.0 94.44% 79.63%

03 (108/54) (<x,y>)^2.0 91.67% 88.89%

04 (108/54) (<x,y> + 1)^2.0 90.74% 88.89%

05 (108/54) (<x,y>)^3.0 89.81% 92.59%

06 (108/54) (<x,y>)^2.0 91.67% 90.74%

07 (108/54) (<x,y>)^3.0 88.89% 92.59%

08 (108/54) (<x,y>)^3.0 87.96% 94.44%

09 (108/54) (<x,y>)^2.0 88.89% 88.89%

10 (108/54) (<x,y>)^2.0 90.74% 79.63%

Mean 90.65% 88.52%

Std Dev 1.87% 5.08%

Time Taken: 0:01:04

124 Appendix B.





01 (108/54) 3 0.4 0.2 85.19% 92.59%

02 (108/54) 14 0.3 0.1 92.59% 83.33%

03 (108/54) 8 0.2 0.1 94.44% 83.33%

04 (108/54) 5 0.2 0.4 90.74% 81.48%

05 (108/54) 4 0.3 0.2 94.44% 90.74%

06 (108/54) 6 0.2 0.2 88.89% 90.74%

07 (108/54) 7 0.3 0.1 88.89% 85.19%

08 (108/54) 10 0.3 0.1 93.52% 90.74%

09 (108/54) 8 0.4 0.2 88.89% 85.19%

10 (108/54) 14 0.3 0.2 94.44% 87.04%

Mean 91.20% 87.04%

Std Dev 3.19% 3.90%

Time Taken: 0:26:08



01 (108/54) (<x,y> + 1)^2.0 94.44% 96.30%

02 (108/54) (<x,y>)^3.0 87.96% 87.04%

03 (108/54) (<x,y>)^2.0 95.37% 85.19%

04 (108/54) (<x,y>)^2.0 95.37% 88.89%

05 (108/54) (<x,y> + 1)^3.0 95.37% 92.59%

06 (108/54) (<x,y>)^4.0 91.67% 94.44%

07 (108/54) (<x,y>)^2.0 91.67% 94.44%

08 (108/54) (<x,y> + 1)^2.0 97.22% 92.59%

09 (108/54) (<x,y>)^2.0 88.89% 94.44%

10 (108/54) (<x,y>)^7.0 93.52% 90.74%

Mean 93.15% 91.67%

Std Dev 3.03% 3.63%

Time Taken: 0:00:47

Appendix B. 125





01 (108/54) 3 0.4 0.1 91.67% 90.74%

02 (108/54) 5 0.3 0.2 91.67% 81.48%

03 (108/54) 13 0.2 0.2 90.74% 87.04%

04 (108/54) 5 0.3 0.2 93.52% 79.63%

05 (108/54) 3 0.5 0.5 91.67% 92.59%

06 (108/54) 6 0.3 0.4 96.30% 81.48%

07 (108/54) 3 0.2 0.4 88.89% 94.44%

08 (108/54) 4 0.3 0.2 95.37% 87.04%

09 (108/54) 4 0.3 0.2 92.59% 85.19%

10 (108/54) 4 0.1 0.1 89.81% 88.89%

Mean 92.22% 86.85%

Std Dev 2.32% 4.97%

Time Taken: 0:23:23



01 (108/54) (<x,y>)^2.0 91.67% 90.74%

02 (108/54) (<x,y>)^2.0 96.30% 83.33%

03 (108/54) (<x,y> + 1)^2.0 88.89% 92.59%

04 (108/54) (<x,y>)^4.0 95.37% 85.19%

05 (108/54) (<x,y>)^3.0 92.59% 92.59%

06 (108/54) (<x,y>)^3.0 91.67% 79.63%

07 (108/54) (<x,y>)^2.0 90.74% 94.44%

08 (108/54) (<x,y> + 1)^3.0 97.22% 88.89%

09 (108/54) (<x,y>)^2.0 96.30% 87.04%

10 (108/54) (<x,y>)^2.0 91.67% 87.04%

Mean 93.24% 88.15%

Std Dev 2.83% 4.64%

Time Taken: 0:00:50

126 Appendix B.





01 (108/54) 4 0.3 0.1 90.74% 94.44%

02 (108/54) 10 0.3 0.2 92.59% 92.59%

03 (108/54) 5 0.1 0.1 91.67% 96.30%

04 (108/54) 4 0.3 0.2 93.52% 90.74%

05 (108/54) 7 0.3 0.2 95.37% 92.59%

06 (108/54) 5 0.2 0.1 97.22% 90.74%

07 (108/54) 15 0.3 0.2 92.59% 92.59%

08 (108/54) 4 0.2 0.1 97.22% 85.19%

09 (108/54) 1 0.1 0.1 93.52% 92.59%

10 (108/54) 3 0.4 0.2 97.22% 92.59%

Mean 94.17% 92.04%

Std Dev 2.43% 2.90%

Time Taken: 0:23:29



01 (108/54) (<x,y>)^2.0 93.52% 96.30%

02 (108/54) (<x,y> + 1)^3.0 96.30% 98.15%

03 (108/54) (<x,y>)^8.0 93.52% 96.30%

04 (108/54) (<x,y>)^6.0 97.22% 94.44%

05 (108/54) (<x,y>)^3.0 95.37% 92.59%

06 (108/54) (<x,y>)^2.0 97.22% 94.44%

07 (108/54) (<x,y> + 1)^2.0 94.44% 96.30%

08 (108/54) (<x,y>)^3.0 98.15% 87.04%

09 (108/54) (<x,y>)^3.0 96.30% 92.59%

10 (108/54) (<x,y>)^2.0 97.22% 92.59%

Mean 95.93% 94.07%

Std Dev 1.64% 3.12%

Time Taken: 0:00:41

Appendix B. 127





01 (108/54) 1 0.1 0.1 94.44% 92.59%

02 (108/54) 3 0.1 0.2 94.44% 87.04%

03 (108/54) 1 0.1 0.1 94.44% 92.59%

04 (108/54) 1 0.2 0.1 95.37% 88.89%

05 (108/54) 3 0.3 0.2 96.30% 94.44%

06 (108/54) 5 0.3 0.1 97.22% 96.30%

07 (108/54) 6 0.2 0.1 95.37% 100.00%

08 (108/54) 2 0.3 0.2 93.52% 94.44%

09 (108/54) 1 0.1 0.1 94.44% 88.89%

10 (108/54) 4 0.2 0.3 95.37% 96.30%

Mean 95.09% 93.15%

Std Dev 1.07% 4.01%

Time Taken: 0:19:40



01 (108/54) (<x,y>)^3.0 96.30% 96.30%

02 (108/54) (<x,y>)^3.0 97.22% 94.44%

03 (108/54) (<x,y>)^2.0 97.22% 98.15%

04 (108/54) (<x,y>)^4.0 98.15% 88.89%

05 (108/54) (<x,y>)^3.0 99.07% 92.59%

06 (108/54) (<x,y>)^3.0 98.15% 96.30%

07 (108/54) (<x,y>)^4.0 95.37% 100.00%

08 (108/54) (<x,y>)^5.0 98.15% 92.59%

09 (108/54) (<x,y>)^1.0 93.52% 88.89%

10 (108/54) (<x,y>)^3.0 96.30% 98.15%

Mean 96.94% 94.63%

Std Dev 1.64% 3.85%

Time Taken: 0:00:41

128 Appendix B.


129

Appendix C

Software Source Code

This appendix contains listings of Java source code for the various functional software units

described in Chapter 5, along with brief notes about their use and dependencies. The

reader (and possible programming purist) should keep in mind that the code presented

here was written with a focus on obtaining research results rather than as an exhibition of

proper coding style. As such, no extensive error handling and other defensive

programming strategies that would befit software with a wide expected distribution, was

developed. In general, the code design is biased toward simplicity and clarity rather than

being extensively optimised for speed.

13

0

Ap

pe

nd

ix C.

CellScan Source Code

The CellScan program is a plug-in for the ImageJ application. It is responsible for the image processing (preprocessing, normalisation,

segmentation) and feature extraction process of our quality assessment system. See section 5.2 for more information on CellScan.

Dependencies

As the program is a plug-in it requires the ImageJ program as its host. ImageJ is an open source application that may be downloaded from

http://rsb.info.nih.gov/ij/.

The graphical user interface was designed using a proprietary third-party tool called JFormDesigner from FormDev Software

(http://www.jformdesigner.com/), however it is not required to compile and execute the code.

Notes

The compiled CellScan program (.jar file) should be placed in the plugins subfolder of the ImageJ installation directory.

Source Code Listing

// CellScan_.java

import java.awt.Checkbox;

import java.awt.Color;

import java.awt.Dialog;

import java.awt.Rectangle;

import java.awt.TextComponent;

import java.io.File;

import java.io.FilenameFilter;

import java.lang.reflect.Field;

import java.util.*;

import java.util.regex.Pattern;

13

1

Ap

pe

nd

ix C.

import cjbdb.AccessDB;

import ij.IJ;

import ij.ImagePlus;

import ij.WindowManager;

import ij.gui.NewImage;

import ij.gui.Roi;

import ij.io.FileInfo;

import ij.plugin.PlugIn;

import ij.plugin.Selection;

import ij.plugin.filter.ParticleAnalyzer;

import ij.plugin.filter.RankFilters;

import ij.process.ImageProcessor;

/**

* @author chrisb

*

*/

public class CellScan_ implements PlugIn {

private ImagePlus impInput;

private boolean hasCurrentImage = false;

private Checkbox chkSaveToDB;

private Checkbox chkProcessCurrentImage;

private Checkbox chkProcessFolder;

private TextComponent txtDBLocation;

private TextComponent txtFolder;

private static final String PLUGIN_NAME = "CellScan";

@Override

public void run(String arg) {

impInput = WindowManager.getCurrentImage();

hasCurrentImage = (impInput==null ? false : true );

InputParameters iParam = new InputParameters();

iParam.hasCurrentImage = hasCurrentImage;

13

2

Ap

pe

nd

ix C.

if (hasCurrentImage) {

iParam.currentImageTitle = impInput.getTitle();

} else {

iParam.currentImageTitle = "";

}

// Input images

iParam.processCurrentImage = hasCurrentImage;

iParam.processFolder = !hasCurrentImage;

iParam.folderToProcess = "E:\\MyImageInputFolder";

// Preprocessing

iParam.normalise = true;

iParam.saturatedPixels = 1.5;

iParam.rescaleImage = true;

iParam.scaleFactor = 0.25;

iParam.splitRowsAndColumns = false;

iParam.numRows = 2;

iParam.numColumns = 2;

iParam.splitRandomWindow = false;

iParam.numRandomWindows = 4;

// Segmentation

iParam.dblVarFilterRadius = 3.0;

iParam.dblMinHoleArea = 257.0;

iParam.blnApplyMeanFilterToInput = true;

iParam.intInputFilterRadius = 1;

iParam.blnApplyMedianFilterToOutput = true;

iParam.intOutputFilterRadius = 4;

iParam.erodeRadius = 6.0;

iParam.showFeaturesOnScreen = false;

iParam.saveFeaturesToDB = false;

iParam.databaseLocation = "E:\\FeatureDB.mdb";

iParam.saveOutputImage = false;

iParam.outputFolder = "E:\\MyImageOutputFolder";

CSDialog myDlg = new CSDialog((Dialog)null);

myDlg.showDialog(iParam);

13

3

Ap

pe

nd

ix C.

if (myDlg.wasCanceled()) {

return;

}

iParam = myDlg.getUserSelections();

if (iParam.processCurrentImage == true) {

if (hasCurrentImage == false) {

IJ.error(PLUGIN_NAME, "There is no image to process!");

return;

}

processSingleImage(iParam);

return;

}

File path = new File(iParam.folderToProcess);

String regExp = ".*\\.(bmp)"; //IJ.getString("Enter a string", "");

int numImagesProcessed = 0;

for (File file : Directory.walk(path, regExp)) {

System.out.println(file.getPath());

impInput = IJ.openImage(file.getPath());

impInput.show();

processSingleImage(iParam);

//close image file

impInput.close();

numImagesProcessed++;

}

IJ.log("Finished. " + numImagesProcessed + " input images processed.");

}

public void processSingleImage(InputParameters iParam) {

FeaturesSet fsWholeCellArea = new FeaturesSet();

13

4

Ap

pe

nd

ix C.

// Record the input file name so it can be saved to a database (if the option is selected)

FileInfo fi = impInput.getOriginalFileInfo();

iParam.inputFileNameAndPath = fi.directory + fi.fileName;

// Rescale image

if (iParam.rescaleImage) {

String origTitle = impInput.getTitle();

IJ.run("Scale...", "x=" + IJ.d2s(iParam.scaleFactor) + " y=" + IJ.d2s(iParam.scaleFactor) + " interpolate

create title=Rescaled");

impInput.changes = false;

impInput.close();

IJ.selectWindow("Rescaled");

impInput = IJ.getImage();

impInput.setTitle(origTitle);


}

// Convert to 8-bit (if not already so)

if (impInput.getBitDepth()!=8) {

IJ.run("8-bit");

}

// Contrast Stretch

if (iParam.normalise && !iParam.normaliseAfterSplit) {

IJ.run("Enhance Contrast", "saturated=" + IJ.d2s(iParam.saturatedPixels,1) + " normalize");

}

// Split image

if (!iParam.splitRowsAndColumns) {

iParam.numRows = 1;

iParam.numColumns = 1;

}

ImagePlus[] impImagesToProcess;

if (iParam.splitRandomWindow) {

impImagesToProcess = splitRandomWindow(impInput, iParam.randomWindowWidth, iParam.randomWindowHeight,

iParam.numRandomWindows);

} else {

// splitTiles() is the default method and also handles the case where there is no split

impImagesToProcess = splitTiles(impInput, iParam);

13

5

Ap

pe

nd

ix C.

}

//close the original


impInput.close();

// Process each tile

for (ImagePlus impTile : impImagesToProcess) {

// Contrast Stretch (only if "normalise after splitting images" option has been selected)

if (iParam.normalise && iParam.normaliseAfterSplit) {

IJ.selectWindow(impTile.getTitle());

IJ.run("Enhance Contrast", "saturated=" + IJ.d2s(iParam.saturatedPixels,1) + " normalize");

}

FeaturesSet fsInnerCellArea = new FeaturesSet();

FeaturesSet fsOuterCellArea = new FeaturesSet();

iParam.imageName = impTile.getTitle();

iParam.imageWidth = impTile.getWidth();

iParam.imageHeight = impTile.getHeight();

iParam.tileNumber = impTile.getProperty("TileNumber").toString();

ImagePlus impRoughSegMask = roughSegmentation(impTile,iParam, fsWholeCellArea);

if (impRoughSegMask == null) {

iParam.roughSegmentationFailed = true;

} else {

iParam.roughSegmentationFailed = false;

ImagePlus impRefinedSegMask = refineSegmentation(impTile, impRoughSegMask, iParam);

extractFeatures(impTile,impRefinedSegMask,fsWholeCellArea);

ImagePlus impInnerMask = getInnerMask(impRefinedSegMask, iParam.erodeRadius);

impInnerMask.updateAndRepaintWindow();

ImagePlus impOuterMask = getOuterMask(impRefinedSegMask, impInnerMask);

impOuterMask.updateAndRepaintWindow();

13

6

Ap

pe

nd

ix C.

extractFeatures(impTile, impInnerMask, fsInnerCellArea);

extractFeatures(impTile, impOuterMask, fsOuterCellArea);

impInnerMask.changes = false;

impInnerMask.close();

impOuterMask.changes = false;

impOuterMask.close();

Roi roi = null;

if ((iParam.processCurrentImage) || (iParam.saveOutputImage == true)) {

// Create a selection from the refined mask

impRefinedSegMask.getProcessor().invert();

IJ.selectWindow(impRefinedSegMask.getID());

Selection selObj = new ij.plugin.Selection();

selObj.run("from");

roi = impRefinedSegMask.getRoi();

impRefinedSegMask.killRoi();

impTile.setRoi(roi);

}

if (iParam.saveOutputImage == true) {

IJ.selectWindow(impTile.getID());

IJ.run("RGB Color");

impTile.getProcessor().setColor(Color.yellow);

roi.drawPixels(impTile.getProcessor());

iParam.outputFileNameAndPath = iParam.outputFolder + "\\" + iParam.imageName + ".png";

IJ.saveAs(impTile, "PNG", iParam.outputFileNameAndPath);

impTile.changes = false;

} else {

iParam.outputFileNameAndPath = "";

}

// Close our rough and refined masks - don't need them anymore

impRoughSegMask.close();

13

7

Ap

pe

nd

ix C.

impRefinedSegMask.close();

}

// Show features on screen

if (iParam.showFeaturesOnScreen == true) {

if (iParam.roughSegmentationFailed==true) {

IJ.log("Rough Segmentation Failed");

}

fsWholeCellArea.displayValues();

}

// Save features to database

if (iParam.saveFeaturesToDB == true) {

saveFeaturesToDB(fsWholeCellArea, fsInnerCellArea, fsOuterCellArea, iParam);

}

// Don't keep images open if we are processing a folder of images

if (iParam.processFolder == true) {

impTile.changes = false;

impTile.close();

}

}

}

private void saveFeaturesToDB(FeaturesSet fsTotal, FeaturesSet fsInner, FeaturesSet fsOuter, InputParameters iParam) {

// Copy features to MS Access database

java.sql.Connection conn = null;

int decimalPlaces = 4;

String dayNum = iParam.imageName.substring(3, 5);

String sample = iParam.imageName.substring(6, 7);

String fcs = iParam.imageName.substring(8, 11);

String well = iParam.imageName.substring(12, 13);

String imageNumber = iParam.imageName.substring(14, 16);

String sqlINSERT = "INSERT INTO tblDataSets ( FileName, Day, Sample, FCS, Well, Image, Tile, RoughSegmentationFailed,

"

+ "Mean, Mode, Median, StdDev, Skew, Kurtosis, Entropy, Uniformity, Area, Perimeter, "

+ "iMean, iMode, iMedian, iStdDev, iSkew, iKurtosis, iEntropy, iUniformity, iArea, iPerimeter, "

13

8

Ap

pe

nd

ix C.

+ "oMean, oMode, oMedian, oStdDev, oSkew, oKurtosis, oEntropy, oUniformity, oArea, oPerimeter, "

+ "pVarFilterRadius, pMinHoleArea, pInputFilterRadius, pOutputFilterRadius, pErodeRadius, MMThres, pImageWidth,

pImageHeight, "

+ "InputFileNameAndPath, OutputFileNameAndPath, pSaveOutputImage, pOutputFolder, "

+ "pRescaled, pRescaleFactor, pNormalize, pNormalizeAfterSplit, pPCSaturatedPixels, pSplitImage, pSplitRows,

pSplitColumns, "

+ "pSplitRandomWindow, pNumRandomWindows, pRandomWindowWidth, pRandomWindowHeight ) "

+ "SELECT '" + iParam.imageName + "', "

+ dayNum + ", "

+ "'" + sample + "', "

+ fcs + ", "

+ well + ", "

+ imageNumber + ", "

+ iParam.tileNumber + ", "

+ (iParam.roughSegmentationFailed ? -1 : 0) + ", "

+ IJ.d2s(fsTotal.mean,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.mode,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.median,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.stddev,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.skewness,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.kurtosis,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.entropy,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.uniformity,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.area,decimalPlaces) + ", "

+ IJ.d2s(fsTotal.perimeter,decimalPlaces) + ", "

+ IJ.d2s(fsInner.mean,decimalPlaces) + ", "

+ IJ.d2s(fsInner.mode,decimalPlaces) + ", "

+ IJ.d2s(fsInner.median,decimalPlaces) + ", "

+ IJ.d2s(fsInner.stddev,decimalPlaces) + ", "

+ IJ.d2s(fsInner.skewness,decimalPlaces) + ", "

+ IJ.d2s(fsInner.kurtosis,decimalPlaces) + ", "

+ IJ.d2s(fsInner.entropy,decimalPlaces) + ", "

+ IJ.d2s(fsInner.uniformity,decimalPlaces) + ", "

+ IJ.d2s(fsInner.area,decimalPlaces) + ", "

+ IJ.d2s(fsInner.perimeter,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.mean,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.mode,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.median,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.stddev,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.skewness,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.kurtosis,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.entropy,decimalPlaces) + ", "

13

9

Ap

pe

nd

ix C.

+ IJ.d2s(fsOuter.uniformity,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.area,decimalPlaces) + ", "

+ IJ.d2s(fsOuter.perimeter,decimalPlaces) + ", "

+ (int)iParam.dblVarFilterRadius + ", "

+ (int)iParam.dblMinHoleArea + ", "

+ iParam.intInputFilterRadius + ", "

+ iParam.intOutputFilterRadius + ", "

+ (int)iParam.erodeRadius + ", "

+ fsTotal.MMThreshold + ", "

+ iParam.imageWidth + ", "

+ iParam.imageHeight + ", "

+ "'" + iParam.inputFileNameAndPath + "', "

+ (iParam.saveOutputImage ? "'" + iParam.outputFileNameAndPath + "'" : "null") + ", "

+ (iParam.saveOutputImage ? -1 : 0) + ", "

+ (iParam.saveOutputImage ? "'" + iParam.outputFolder + "'" : "null") + ", "

+ (iParam.rescaleImage ? -1 : 0) + ", "

+ (iParam.rescaleImage ? IJ.d2s(iParam.scaleFactor,decimalPlaces) : IJ.d2s( 1.0,decimalPlaces)) + ", "

+ (iParam.normalise ? -1 : 0) + ", "

+ (iParam.normaliseAfterSplit ? -1 : 0) + ", "

+ (iParam.normalise ? IJ.d2s(iParam.saturatedPixels,decimalPlaces) : "null") + ", "

+ (iParam.splitRowsAndColumns ? -1 : 0) + ", "

+ iParam.numRows + ", "

+ iParam.numColumns + ", "

+ (iParam.splitRandomWindow ? -1 : 0) + ", "

+ iParam.numRandomWindows + ", "

+ iParam.randomWindowWidth + ", "

+ iParam.randomWindowHeight + ";";

conn = AccessDB.OpenDB(iParam.databaseLocation );

try { AccessDB.runSQL(conn, sqlINSERT);} finally {AccessDB.CloseDB(conn);}

}

// Split images as required

private ImagePlus[] splitTiles(ImagePlus impInputImage, InputParameters iParam) {

String imageTitle = impInputImage.getTitle();

String imageTitleNoExt = removeRightSubstring(imageTitle,".bmp");

int horizSplit = iParam.numColumns;

int vertSplit = iParam.numRows;

14

0

Ap

pe

nd

ix C.

int imageWidth = impInputImage.getWidth();

int imageHeight = impInputImage.getHeight();

int tileWidth = imageWidth / horizSplit;

int tileHeight = imageHeight / vertSplit;

String tileNumber;

ImagePlus[] imgOut = new ImagePlus[(horizSplit*vertSplit)];

for (int x=0; x < horizSplit; x++) {

for (int y=0; y < vertSplit; y++) {

int leftPos = (x * tileWidth);

int topPos = (y * tileHeight);

String selectionParameters = "width=" + tileWidth + " height=" + tileHeight + " x=" + leftPos + " y=" +

topPos;

IJ.selectWindow(imageTitle);

IJ.run("Specify...", selectionParameters);

String tileTitle;

// yes, these are currently redundant - a single image tile is now named as per a split image tile.

if (imgOut.length == 1) {

tileTitle = imageTitleNoExt;

tileNumber = "00";

} else {

tileTitle = imageTitleNoExt + "_" + x + "" + y;

tileNumber = x + "" + y;

}

IJ.run("Duplicate...", "title=" + tileTitle);

IJ.selectWindow(tileTitle);

imgOut[x+(2*y)] = IJ.getImage();

imgOut[x+(2*y)].setProperty("TileNumber", tileNumber);

}

}

return imgOut;

}

/** Creates an array of randomly positioned sub images of the specified size and width from the given

* input image. The number of sub images is specified by numWindows. */

14

1

Ap

pe

nd

ix C.

private ImagePlus[] splitRandomWindow(ImagePlus impInputImage, int windowWidth, int windowHeight, int numWindows ) {

String imageTitle = impInputImage.getTitle();

String imageTitleNoExt = removeRightSubstring(imageTitle,".bmp");

String tileNumber;

ImagePlus[] imgOut = new ImagePlus[numWindows];

int maxXoffset = impInputImage.getWidth() - windowWidth;

int maxYoffset = impInputImage.getHeight() - windowHeight;

Random rand = new Random(System.currentTimeMillis());

for (int i = 0; i<numWindows; i++) {

// get random number for x and y offset

int randXoffset = rand.nextInt(maxXoffset);

int randYoffset = rand.nextInt(maxYoffset);

// copy image from original to new image

String selectionParameters = "width=" + windowWidth + " height=" + windowHeight + " x=" + randXoffset + " y=" +

randYoffset;

IJ.selectWindow(imageTitle);

IJ.run("Specify...", selectionParameters);

String tileTitle;

tileTitle = imageTitleNoExt + "_" + String.format("%02d", i);

tileNumber = String.format("%02d", i);

IJ.run("Duplicate...", "title=" + tileTitle);

IJ.selectWindow(tileTitle);

imgOut[i] = IJ.getImage();

imgOut[i].setProperty("TileNumber", tileNumber);

}

return imgOut;

}

14

2

Ap

pe

nd

ix C.

private String removeRightSubstring(String fileName, String rightSubstring) {

int index = fileName.lastIndexOf(rightSubstring);

String outName = fileName.substring(0, index);

return outName;

}

public ImagePlus roughSegmentation(final ImagePlus impInput, InputParameters iParam, FeaturesSet fs ) {

ImageProcessor ip = impInput.getProcessor();

int w = ip.getWidth();

int h = ip.getHeight();

Rectangle roi = ip.getRoi();

// --- Perform processing ---

int intRadius =(int) iParam.dblVarFilterRadius;

int intSize = (int) ((intRadius * 2) + 1);

String strOutputImageTitle = CJBPluginTools.constructOutputImageTitle("_SD" + intRadius, impInput.getTitle());

// Create circular mask for kernel

boolean[][] blnMask = new boolean [intSize][intSize];

CJBPluginTools.populateRadiusMask(blnMask);

boolean blnShowKernelSize = false;

if (blnShowKernelSize == true) {

CJBPluginTools.displayRadiusMask(blnMask);

}

ImagePlus impSD = NewImage.createByteImage(strOutputImageTitle, w, h, 1, NewImage.FILL_BLACK);

ImageProcessor ipSD = impSD.getProcessor();

// i and j cycle thru all points in the image/roi (square)

for (int j=roi.y; j<roi.y+roi.height; j++) {

for (int i=roi.x; i<roi.x+roi.width; i++) {

// Cycle thru kernal

14

3

Ap

pe

nd

ix C.

int intNumPointsInKernel = 0;

float sngSumOfEachValue = 0;

float sngSumOfEachValueSquared = 0;

float sngVariance = 0;

float sngStdDev = 0;

for (int y = (j - intRadius); y <= (j + intRadius); y++) {

if ((y >= roi.y) && (y < (roi.y + roi.height))) {

for (int x = (i - intRadius); x <= (i + intRadius); x++) {

if ((x >= roi.x) && (x < (roi.x + roi.width))) {

// The current pixel in the kernel lies within the image/roi area

if ( blnMask[x - i + intRadius][y - j + intRadius] == true) {

//we are within the circular area of the filter

float sngValue = 0;

float sngValueSquared = 0;

sngValue = (float)ip.getPixel(x, y);

sngValueSquared = sngValue * sngValue;

sngSumOfEachValue += sngValue;

sngSumOfEachValueSquared += sngValueSquared;

intNumPointsInKernel++;

}

}

}

}

}

// Process results gathered from kernel

sngVariance = (sngSumOfEachValueSquared - ((sngSumOfEachValue * sngSumOfEachValue) /

intNumPointsInKernel)) / intNumPointsInKernel;

sngStdDev = (float)Math.sqrt((double)sngVariance);

14

4

Ap

pe

nd

ix C.

ipSD.putPixel(i, j, Math.round(sngStdDev));

}

}

// Threshold using mixture modelling

int level;

level = CJBPluginTools.getThresholdMixtureModelling(impSD);

fs.MMThreshold = level;

ipSD.threshold(level);

// Use particle analyser to remove holes

ParticleAnalyzer pa = new

ParticleAnalyzer(ParticleAnalyzer.SHOW_MASKS,0,null,iParam.dblMinHoleArea,Double.POSITIVE_INFINITY);

pa.analyze(impSD);

ImagePlus impRoughSegMask = pa.getOutputImage();

//Close the working copy of image (without prompting to save changes)

impSD.changes = false;

impSD.close();

if (impRoughSegMask!=null) {

impRoughSegMask.setTitle("RoughSegMask");

impRoughSegMask.getProcessor().invertLut();

impRoughSegMask.getProcessor().invert();

}

return impRoughSegMask;

}

/**Performs a refined segmentation on the provided ImagePlus object. The object must have a rough segmentation boundary

selected.

* @param impOrig The original image (as an ImagePlus) with the rough segmentation boundary selected.

* @return A new ImagePlus containing the refined segmentation mask

*/

public ImagePlus refineSegmentation(final ImagePlus impInput, ImagePlus impRoughSegMask, InputParameters iParam) {

int intThreshold = 256;

String strOutputImageTitle;

14

5

Ap

pe

nd

ix C.

ImagePlus impWorkCopy = CJBPluginTools.duplicateImage(impInput, "WorkingCopy", true);

ImageProcessor ipWorkCopy = impWorkCopy.getProcessor();

IJ.selectWindow(impWorkCopy.getID());

// remove selection so mean filter is applied to whole image

impWorkCopy.killRoi();

RankFilters rf=new RankFilters();

// Apply mean filter to (entire) input image

if (iParam.blnApplyMeanFilterToInput == true) {

rf.rank(ipWorkCopy, iParam.intInputFilterRadius, RankFilters.MEAN);

}

// Get mask

ImageProcessor mask = impRoughSegMask.getProcessor();

int w = ipWorkCopy.getWidth();

int h = ipWorkCopy.getHeight();

Rectangle r = new Rectangle();

r.x = 0;

r.y = 0;

r.width = w;

r.height = h;

// Create output image

strOutputImageTitle = "RefinedSegMask";

ImagePlus imgOut = NewImage.createByteImage(strOutputImageTitle, w, h, 1, NewImage.FILL_BLACK);

ImageProcessor ipOut = imgOut.getProcessor();

int prevValue = -1;

int curValue = -1;

boolean blnSkip = false;

// Scan from left side moving right

for (int y=0; y<r.height; y++) {

prevValue = -1;

blnSkip = false;

14

6

Ap

pe

nd

ix C.

for (int x=0; x<r.width; x++) {

//scan along row

if (mask==null||mask.getPixel(x,y)!=0) {

// skip roi sections that start on the edge

if (x==0) {

blnSkip = true;

}

if (blnSkip == false) {

curValue = ipWorkCopy.getPixel(x+r.x, y+r.y);

if (prevValue != -1) {

//if ((curValue-prevValue)>= 0) {

//if (curValue >= intThreshold ) {

if (((curValue-prevValue)>= 0) || (curValue >= intThreshold )) {

// halo

ipOut.putPixel(x+r.x-1, y+r.y,128);

} else {

// not halo - skip rest of line

blnSkip = true;

}

} else {

prevValue = curValue;

}


}

} else {

prevValue = -1;

blnSkip = false;

}

}

14

7

Ap

pe

nd

ix C.

}

// Scan from right side moving left


prevValue = -1;

blnSkip = false;

//for (int x=0; x<r.width; x++) {

for (int x=(r.width-1); x>=0; x--) {

//scan along row



if (x==(r.width-1)) {

blnSkip = true;

}







// halo

//ipOut.putPixel(x+r.x+1, y+r.y, (ipOut.getPixel(x+r.x+1, y+r.y) + 64));

ipOut.putPixel(x+r.x+1, y+r.y, 128);

} else {


blnSkip = true;

}

} else {


}


}

} else {

14

8

Ap

pe

nd

ix C.

prevValue = -1;

blnSkip = false;

}

}

}

// Scan from top moving down


prevValue = -1;

blnSkip = false;


//scan along row



if (y==0) {

blnSkip = true;

}







// halo

//ipOut.putPixel(x+r.x, y+r.y-1,128);

//ipOut.putPixel(x+r.x, y+r.y-1, (ipOut.getPixel(x+r.x, y+r.y-1) + 32));

ipOut.putPixel(x+r.x, y+r.y-1, 128);

} else {


blnSkip = true;

}

} else {


14

9

Ap

pe

nd

ix C.

}


}

} else {

prevValue = -1;

blnSkip = false;

}

}

}

// Scan from bottom moving up


prevValue = -1;

blnSkip = false;

//for (int y=0; y<r.height; y++) {

for (int y=(r.height-1); y>=0; y--) {

//scan along row



if (y==(r.height-1)) {

blnSkip = true;

}







// halo

ipOut.putPixel(x+r.x, y+r.y+1, 128);

} else {


blnSkip = true;

15

0

Ap

pe

nd

ix C.

}

} else {


}


}

} else {

prevValue = -1;

blnSkip = false;

}

}

}

// We no longer require our work copy, so close it

impWorkCopy.changes = false;

impWorkCopy.close();

// Fillin roi areas that havent been marked as halo



//scan along row


// in object area

curValue = ipOut.getPixel(x+r.x, y+r.y);

if (curValue == 0) {

ipOut.putPixel(x+r.x, y+r.y,255);

}

} else {

// not in object area


}

}

}

// Set halo areas to black


15

1

Ap

pe

nd

ix C.


//scan along row


// in object area

curValue = ipOut.getPixel(x+r.x, y+r.y);

if (curValue == 128) {


}

} else {

// not in object area


}

}

}

// Perform median filtering on output image

if (iParam.blnApplyMedianFilterToOutput == true) {

rf.rank(ipOut, iParam.intOutputFilterRadius, RankFilters.MEDIAN);

}

imgOut.show();

imgOut.updateAndDraw();

return imgOut;

}

//This presently requires the input image to have a selection marked

public void extractFeatures(final ImagePlus impInput, ImagePlus impRefImagePlus, FeaturesSet fs) {

if (fs==null) {

return;

}

ImageProcessor mask = impRefImagePlus.getProcessor(); //ip.getMask();

ImageProcessor ipInput = impInput.getProcessor();

boolean hasMask = (mask != null);

15

2

Ap

pe

nd

ix C.

// ROI corner coordinates:

int rLeft = 0;

int rTop = 0;

int rRight = ipInput.getWidth();

int rBottom = ipInput.getHeight();

double dblPixelCount = 0;

double dblSumOfEachValue = 0;

double dblSumOfEachValueSquared = 0;

double dblVariance = 0;

double dblStdDev = 0;

double dblMean = 0;

int[] histogram = new int[256];

// process all pixels inside the ROI

for (int y = rTop; y < rBottom; y++) {

for (int x = rLeft; x < rRight; x++) {

if (!hasMask || mask.getPixel(x-rLeft, y-rTop) > 0) {

int intValue = 0;

double dblValue = 0;

double dblValueSquared = 0;

intValue = ipInput.getPixel(x, y);

dblValue = (double)intValue;

histogram[intValue]++;

dblValueSquared = dblValue * dblValue;

dblSumOfEachValue += dblValue;

dblSumOfEachValueSquared += dblValueSquared;

dblPixelCount++;

}

}

}

double dblLog2 = Math.log((double)2);

double[] p = new double[256];

15

3

Ap

pe

nd

ix C.

double dblEntropy = 0;

double dblUniformity = 0;

double dblMode = 0;

double dblMedian = 0;

int maxCount = 0;

double halfCount = dblPixelCount/2.0;

double sum = 0;

for (int n = 0; n <= 255; n++) {

p[n] = (double)histogram[n] / dblPixelCount;

dblUniformity += (p[n]*p[n]);

if (p[n]> 0) {

dblEntropy = dblEntropy + (p[n] * ( Math.log(p[n]) / dblLog2 ) );

}

// Determine mode

if (histogram[n]>maxCount) {

maxCount = histogram[n];

dblMode = n;

}

// Determine median

if (sum <= halfCount) {

dblMedian = n;

sum += histogram[n];

}

}

dblEntropy = -dblEntropy;

dblVariance = (dblSumOfEachValueSquared - ((dblSumOfEachValue * dblSumOfEachValue) / dblPixelCount)) / dblPixelCount;

dblStdDev = Math.sqrt(dblVariance);

dblMean = dblSumOfEachValue / dblPixelCount;

double dblSumOfDev = 0, dblSumOfDev2 = 0,dblSumOfDev3 = 0,dblSumOfDev4 = 0;

// process all pixels inside the ROI

for (int y = rTop; y < rBottom; y++) {

for (int x = rLeft; x < rRight; x++) {

15

4

Ap

pe

nd

ix C.

if (!hasMask || mask.getPixel(x-rLeft, y-rTop) > 0) {

double dblValue = 0;

double dblDev = 0, dblDev2 = 0,dblDev3 = 0, dblDev4 = 0;

dblValue = (double)ipInput.getPixel(x, y);

dblDev = dblValue - dblMean;

dblDev2 = dblDev * dblDev;

dblDev3 = dblDev2 * dblDev;

dblDev4 = dblDev3 * dblDev;

dblSumOfDev += dblDev;

dblSumOfDev2 += dblDev2;



}

}

}

double dblSkewness = 0;

double dblKurtosis = 0;

double dblStdDev3 = dblStdDev * dblStdDev * dblStdDev;

dblSkewness = dblSumOfDev3 / (dblPixelCount * dblStdDev3);

dblKurtosis = (dblSumOfDev4 / (dblPixelCount * dblVariance * dblVariance)) - 3;

// Copy extracted feature to our feature set object as the return value

fs.area = dblPixelCount;

fs.mean = dblMean;

fs.median = dblMedian;

fs.mode = dblMode;

fs.variance = dblVariance;

fs.stddev = dblStdDev;

fs.skewness = dblSkewness;

fs.kurtosis = dblKurtosis;

fs.entropy = dblEntropy;

fs.uniformity = dblUniformity;

calcPerimeter(impRefImagePlus,fs);

}

/**

15

5

Ap

pe

nd

ix C.

* Gets the inner mask (eroded version of impMask)

* @param impMask

* @return

*/

public ImagePlus getInnerMask(ImagePlus impMask, double erosionRadius) {

String erodedImageName = "InnerMask";

ImagePlus impEroded = CJBPluginTools.duplicateImage(impMask, erodedImageName, true);

int[][] se = ImgProcess_.makeDisk(erosionRadius);

ImgProcess_.erode(impEroded.getProcessor(), se);

return impEroded;

}

public ImagePlus getOuterMask(ImagePlus impMask, ImagePlus impInnerMask) {

String erodedImageName = impInnerMask.getTitle();

String maskImageName = impMask.getTitle();

IJ.run("Image Calculator...", "image1=" + maskImageName + " operation=Subtract image2=" + erodedImageName + "

create");

IJ.selectWindow("Result of " + maskImageName);

ImagePlus impOutline = IJ.getImage();

impOutline.setTitle("OuterMask");

return impOutline;

}

/**

* Calculate the total perimeter of the objects in the given mask image - white objects on black background is assumed

* @param impMask

*/

public void calcPerimeter(ImagePlus impMask, FeaturesSet fs) {

String erodedImageName = "Eroded";

String maskImageName = impMask.getTitle();

15

6

Ap

pe

nd

ix C.

ImagePlus impEroded = CJBPluginTools.duplicateImage(impMask, erodedImageName, true);

int[][] se = { {0,1,0},

{1,1,1},

{0,1,0}};

ImgProcess_.erode(impEroded.getProcessor(), se);

impEroded.updateAndRepaintWindow();

IJ.run("Image Calculator...", "image1=" + maskImageName + " operation=Subtract image2=" + erodedImageName + "

create");

IJ.selectWindow("Result of " + maskImageName);

ImagePlus impOutline = IJ.getImage();

ImageProcessor ipOutline = impOutline.getProcessor();

byte[] pixels = (byte[]) ipOutline.getPixels();

int countWhitePixels = 0;

int totalPixelsProcessed = 0;

for (byte pixel: pixels) {

if (pixel != 0) {

countWhitePixels++;

}

totalPixelsProcessed++;

}

fs.perimeter = countWhitePixels;

impOutline.changes = false;

impOutline.close();

impEroded.changes = false;

impEroded.close();

}

}

class InputParameters {

public boolean hasCurrentImage = false;

public String currentImageTitle;

public boolean rescaleImage = false;

public double scaleFactor = 0.25;

public boolean normalise = true;

public boolean normaliseAfterSplit = false;

public double saturatedPixels = 1.5;

15

7

Ap

pe

nd

ix C.

public boolean splitRowsAndColumns = false;

public boolean splitRandomWindow = false;

public int numRandomWindows = 4;

public int randomWindowWidth = 442;

public int randomWindowHeight = 600;

public int numRows = 2;

public int numColumns = 2;

public boolean processCurrentImage = true;

public boolean processFolder = false;

public String folderToProcess = "E:\\MyTestImageFolder";

public double dblVarFilterRadius = 3;

public double dblMinHoleArea = 257;

public boolean blnApplyMeanFilterToInput = true;

public boolean blnApplyMedianFilterToOutput = true;

public int intInputFilterRadius = 1;

public int intOutputFilterRadius = 4;

public double erodeRadius = 6;

public boolean showFeaturesOnScreen = false;

public boolean saveFeaturesToDB = false;

public String databaseLocation = "E:\\JavaTestDB.mdb";

public boolean roughSegmentationFailed = false;

public String imageName;

public int imageWidth;

public int imageHeight;

public boolean saveOutputImage = false;

public String outputFileNameAndPath;

public String outputFolder;

public String inputFileNameAndPath;

public String tileNumber;

void displayValues() {

Class<InputParameters> c = InputParameters.class;

Field[] flds = c.getFields();

for (Field f: flds) {

try {

IJ.log(f.getName() + ": " + f.get(this));

} catch (IllegalArgumentException e) {

// TODO Auto-generated catch block

e.printStackTrace();

} catch (IllegalAccessException e) {



15

8

Ap

pe

nd

ix C.

}

}

}

}

/**

* Contains a set of feature values

*/

class FeaturesSet {

// TODO: Add perimeter extraction

public double entropy = 0;

public double uniformity = 0;

public double skewness = 0;

public double kurtosis = 0;

public double variance = 0;

public double stddev = 0;

public double mean = 0;

public double median = 0;

public double mode = 0;

public double area = 0;

public double perimeter = 0;

public int MMThreshold = 0;

/**

* Displays the feature set to the screen

*/

void displayValues() {

Class<FeaturesSet> c = FeaturesSet.class;

Field[] flds = c.getFields();

for (Field f: flds) {

try {

IJ.log(f.getName() + ": " + f.get(this));

} catch (IllegalArgumentException e) {



} catch (IllegalAccessException e) {



}

}

IJ.log("");

15

9

Ap

pe

nd

ix C.

}

}

class DirFilter implements FilenameFilter {

private Pattern pattern;

public DirFilter(String regex) {

pattern = Pattern.compile(regex,Pattern.CASE_INSENSITIVE);

}

public boolean accept(File dir, String name) {

// Strip path information, search for regex:

return pattern.matcher(new File(name).getName()).matches();

}

}

// AccessDB.java

package cjbdb;

import java.io.PrintStream;

import java.sql.*;

public class AccessDB {

private static final String accessDBURLPrefix = "jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=";

private static final String accessDBURLSuffix = ";DriverID=22;READONLY=true}";

static {

try {

Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");

} catch(ClassNotFoundException e) {

System.err.println("JdbcOdbc Bridge Driver not found!");

}

}

/** Creates a Connection to a Access Database */

public static java.sql.Connection getAccessDBConnection(String filename) throws SQLException {

filename = filename.replace('\\', '/').trim();

String databaseURL = accessDBURLPrefix + filename + accessDBURLSuffix;

return DriverManager.getConnection(databaseURL, "", "");

}

16

0

Ap

pe

nd

ix C.

public static boolean runSQL(java.sql.Connection conn, String sql) {

try {

Statement stmt = conn.createStatement();

if(stmt.execute(sql)) {

printResultSet(System.out, stmt.getResultSet(), "Query Result");

} else {

//System.out.println("SQL/DDL command executed successfully");

}

return true;

} catch(SQLException s) {

System.out.println(s);

return false;

}

}

public static boolean runSQL(String databaseFilenameAndPath, String sql) {


try {

conn = AccessDB.getAccessDBConnection(databaseFilenameAndPath);

return runSQL(conn, sql);



return false;

} finally {

if(conn != null) {

try {

conn.close();

} catch(SQLException ignore) {}

}

}

}

public static java.sql.Connection OpenDB(String databaseFilenameAndPath) {


// Just return the connection object if it is already open

if (conn != null ) {

return conn;

}

// Attempt to open the connection

16

1

Ap

pe

nd

ix C.

try {

conn = AccessDB.getAccessDBConnection(databaseFilenameAndPath);

return conn;



return null;

}

}

public static void CloseDB(java.sql.Connection conn) {

if (conn != null) {

try {

conn.close();

} catch(SQLException ignore) {}

}

}

/** Prints badly ResultSet to PrintStream */

public static void printResultSet(PrintStream p, ResultSet rs, String title) throws SQLException {

if(rs != null) {

ResultSetMetaData metaData = rs.getMetaData();

int cols = metaData.getColumnCount();

p.println("\n--------------------------\n" + title + "\n--------------------------");

for(int i = 1;i <= cols;i++) {

p.print(metaData.getColumnLabel(i) + "\t");

}

p.println("\n--------------------------");

int count = 0;

while(rs.next()) {

for(int i = 1;i <= cols;i++) {

p.print(rs.getString(i) + "\t");

}

p.println("");

count++;

}

p.println("Rows: " + count);

}

}

}

16

2

Ap

pe

nd

ix C.

CellAPR Source Code

The following is the Java source code that implements the CellAPR program. This program automates the pattern recognition (including

parameter selection) stage of our system for a number of pattern recognition algorithms (such as the Support Vector Machine and Multilayer

perceptron). This program is also discussed in section 5.4 of Chapter 5.

Dependencies

This program makes use of the Weka library weka.jar version 3.5.8 (available from http://www.cs.waikato.ac.nz/ml/weka/) and two apache

commons libraries commons-math-2.0.jar and commons-lang-2.4.jar (available from http://commons.apache.org/).

Notes

This program does not have a graphic user interface and in its present stage requires input parameters (input file names and the machine

learning algorithm to use) to be set in code (i.e. hardcoded). To set the file names and paths for the training and test input datafiles (Weka

format .arff files produced from our FeatureDB database, refer section 5.3), go to the following code lines and fill in appropriate values

specifying the file locations:

// -------> SELECT THE DATASETS

// NOTE: the file templates are the arff file names without the leading 01_, 02_, etc

String trainingDataFolder =

"E:\\Results\\PR\\Output\\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFeatures\\";

String trainingFileNameTemplate = "DS_Day01_Tiles1_Erode6_RemovedVisRejects_AllStatFeatures_Training.arff";

String testDataFolder = "E:\\Results\\PR\\Output\\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFeatures\\";

String testFileNameTemplate = "DS_Day01_Tiles1_Erode6_RemovedVisRejects_AllStatFeatures_Testing.arff";

16

3

Ap

pe

nd

ix C.

To select the particular machine learning algorithm to use, go to the code lines shown below and ensure the desired machine learning

algorithm is uncommented (while the ones not being used are left commented). In the example code here, the Support Vector Machine (SVM)

has been chosen.

// -------> SELECT A CLASSIFIER

cClassifier = ClassifierType.SVM;

// cClassifier = ClassifierType.NAIVEBAYES;

// cClassifier = ClassifierType.MLP;

// cClassifier = ClassifierType.MLP_GRIDSEARCH;

Source Code Listing

// CellAPR.java

import weka.core.Attribute;

import weka.core.FastVector;

import weka.core.Instance;

import weka.core.Instances;

import weka.core.converters.ConverterUtils.DataSource;

import weka.classifiers.Classifier;

import weka.classifiers.Evaluation;

import weka.classifiers.bayes.NaiveBayes;

import weka.classifiers.functions.MultilayerPerceptron;

import weka.classifiers.functions.SMO;

import java.util.Random;

import org.apache.commons.math.stat.descriptive.DescriptiveStatistics;

import org.apache.commons.lang.time.StopWatch;

public class CellAPR {

public static Instances isTrainingSet;

public static Instances isTestSet;

public static double[] cDSTrainingPC;

16

4

Ap

pe

nd

ix C.

public static String[] cOptions;

public static String[] cParamDesc;

public static Classifier cModel;

public static double bestTestPC;

public static int bestTestPCIndex;

public static int dataSetNum;

public static int[] cMLP_H;

public static int[] cMLP_L;

public static int[] cMLP_M;

public static int cMLP_H_Value;

public static int cMLP_L_Value;

public static int cMLP_M_Value;

public static int numDataSets;

public static DescriptiveStatistics statsTraining;

public static DescriptiveStatistics statsTest;

public static StopWatch SW;

public enum ClassifierType {

SVM, NAIVEBAYES, MLP, MLP_GRIDSEARCH

}

public static ClassifierType cClassifier;

public static void main(String[] args) {

try {

// -------> SELECT THE DATASETS

// NOTE: the file templates are the arff file names without the leading 01_, 02_, etc

String trainingDataFolder =

"E:\\Results\\PR\\Output\\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFeatures\\";

16

5

Ap

pe

nd

ix C.

String trainingFileNameTemplate = "DS_Day01_Tiles1_Erode6_RemovedVisRejects_AllStatFeatures_Training.arff";

String testDataFolder = "E:\\Results\\PR\\Output\\Day01_Tiles01_Erode06_RemovedVisRejects_AllStatFeatures\\";

String testFileNameTemplate = "DS_Day01_Tiles1_Erode6_RemovedVisRejects_AllStatFeatures_Testing.arff";

// -------> SELECT A CLASSIFIER

cClassifier = ClassifierType.SVM;

// cClassifier = ClassifierType.NAIVEBAYES;

// cClassifier = ClassifierType.MLP;

// cClassifier = ClassifierType.MLP_GRIDSEARCH;

// -------> SELECT NUMBER OF DATASETS TO CRUNCH (OPTIONAL - NORMALLY JUST LEAVE THIS AS 10)

numDataSets = 10;

statsTraining = new DescriptiveStatistics();

statsTraining.clear();

statsTest = new DescriptiveStatistics();

statsTest.clear();

SW = new StopWatch();

SW.reset();

SW.start();

switch(cClassifier){

case SVM:

setupSMO(); System.out.println("SVM"); break;

case NAIVEBAYES:

setupNaiveBayes(); System.out.println("Naive Bayes"); break;

case MLP:

System.out.println("MLP"); break;

case MLP_GRIDSEARCH:

setupMLP_FullGridSearch(); System.out.println("MLP - Grid Search"); break;

}

System.out.println("Training Folder: " + trainingDataFolder);

System.out.println("Test Folder: " + testDataFolder);

for (dataSetNum = 1; dataSetNum <= numDataSets; dataSetNum++) {

String trainingFile = trainingDataFolder + String.format("%02d", dataSetNum) + "_" +

trainingFileNameTemplate;

16

6

Ap

pe

nd

ix C.

String testFile = testDataFolder + String.format("%02d", dataSetNum) + "_" + testFileNameTemplate;

SetTrainingData_ARFF(trainingFile);

SetTestData_ARFF(testFile);


case MLP:

performEvaluationMLP(); break;

default:

performEvaluation(); break;

}

} // end dataset for loop


case MLP: case MLP_GRIDSEARCH:

// These have 3 parameters and so need a two extra tabs to make the mean and std dev columns correctly

// aligned with the data above them

System.out.format("\t\t\tMean\t%4.2f%%\t%4.2f%%\n",statsTraining.getMean(),statsTest.getMean());

System.out.format("\t\t\tStd

Dev\t%4.2f%%\t%4.2f%%\n",statsTraining.getStandardDeviation(),statsTest.getStandardDeviation());

break;

default:

System.out.format("\tMean\t%4.2f%%\t%4.2f%%\n",statsTraining.getMean(),statsTest.getMean());

System.out.format("\tStd

Dev\t%4.2f%%\t%4.2f%%\n",statsTraining.getStandardDeviation(),statsTest.getStandardDeviation());

break;

}

System.out.println();

SW.stop();

System.out.println("Time Taken: " + SW.toString());

System.out.println("Done!");

} catch (Exception e) {



}

}

16

7

Ap

pe

nd

ix C.

private static void performEvaluation() {

try {

Evaluation eTest;

bestTestPC = 0;

bestTestPCIndex = 0;

// Evaluate each option using 10-fold CV on training set to find the best set of parameters

for ( int i = 0; i < cOptions.length ; i++) {

cModel.setOptions(weka.core.Utils.splitOptions(cOptions[i]));

cModel.buildClassifier(isTrainingSet);

eTest = new Evaluation(isTrainingSet);

eTest.crossValidateModel(cModel, isTrainingSet, 10, new Random(1));

cDSTrainingPC[i] = eTest.pctCorrect();

if (cDSTrainingPC[i] > bestTestPC) {

bestTestPC = cDSTrainingPC[i];

bestTestPCIndex = i;

}

}

cModel.setOptions(weka.core.Utils.splitOptions(cOptions[bestTestPCIndex]));


// 10-fold cross validation on training set

double trainingPC;

double testPC;

// Training performance



trainingPC = eTest.pctCorrect();

statsTraining.addValue(trainingPC);

// Testing performance


eTest.evaluateModel(cModel, isTestSet);

testPC = eTest.pctCorrect();

statsTest.addValue(testPC);

16

8

Ap

pe

nd

ix C.

System.out.format("%02d (%d/%d)\t%s\t%4.2f%%\t%4.2f%%\n", dataSetNum,

isTrainingSet.numInstances(),isTestSet.numInstances(), cParamDesc[bestTestPCIndex], trainingPC, testPC );




}

}

private static void performEvaluationMLP() {

try {

Evaluation eTest;

bestTestPC = 0;


// Evaluate each option using 10-fold CV on training set to find the best set of parameters

setupMLP_DetermineNumNodes();










}

}

cMLP_H_Value = cMLP_H[bestTestPCIndex];

// Find best learning rate

setupMLP_DetermineLearningRate();

bestTestPC = 0;

16

9

Ap

pe

nd

ix C.











}

}

cMLP_L_Value = cMLP_L[bestTestPCIndex];

// Find best momentum

setupMLP_DetermineMomentum();

bestTestPC = 0;











}

}

cMLP_M_Value = cMLP_M[bestTestPCIndex];

//Evaluate performance on test set using selected parameters

cModel.setOptions(weka.core.Utils.splitOptions(cOptions[bestTestPCIndex]));

17

0

Ap

pe

nd

ix C.


// 10-fold cross validation on training set

double trainingPC;

double testPC;

// Training performance



trainingPC = eTest.pctCorrect();

statsTraining.addValue(trainingPC);

// Testing performance


eTest.evaluateModel(cModel, isTestSet);

testPC = eTest.pctCorrect();

statsTest.addValue(testPC);

System.out.format("%02d (%d/%d)\t%s\t%4.2f%%\t%4.2f%%\n", dataSetNum,

isTrainingSet.numInstances(),isTestSet.numInstances(), cParamDesc[bestTestPCIndex], trainingPC, testPC );




}

}

private static void setupSMO() {

cModel = (Classifier)new SMO();

cDSTrainingPC = new double[16];

cOptions = new String[16];

cParamDesc = new String[16];

cOptions[0] = "-C 1.0 -L 0.0010 -P 1.0E-12 -N 0 -V -1 -W 1 -K \"weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E

1.0\"";

cOptions[1] = "-C 1.0 -L 0.0010 -P 1.0E-12 -N 0 -V -1 -W 1 -K \"weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 1.0 -

L\"";


2.0\"";


L\"";

17

1

Ap

pe

nd

ix C.


3.0\"";


L\"";


4.0\"";


L\"";


5.0\"";


L\"";


6.0\"";


L\"";


7.0\"";


L\"";


8.0\"";


L\"";

cParamDesc[0] = "(<x,y>)^1.0";

cParamDesc[1] = "(<x,y> + 1)^1.0";


cParamDesc[3] = "(<x,y> + 1)^2.0";


cParamDesc[5] = "(<x,y> + 1)^3.0";


cParamDesc[7] = "(<x,y> + 1)^4.0";


cParamDesc[9] = "(<x,y> + 1)^5.0";

cParamDesc[10] = "(<x,y>)^6.0";

cParamDesc[11] = "(<x,y> + 1)^6.0";

cParamDesc[12] = "(<x,y>)^7.0";

cParamDesc[13] = "(<x,y> + 1)^7.0";

cParamDesc[14] = "(<x,y>)^8.0";

cParamDesc[15] = "(<x,y> + 1)^8.0";

}

private static void setupNaiveBayes() {

cModel = (Classifier)new NaiveBayes();

// Assume normal distribution only


17

2

Ap

pe

nd

ix C.



cOptions[0] = "";

cParamDesc[0] = "Normal";

}

private static void setupMLP_DetermineNumNodes() {

//weka.classifiers.functions.MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H 1

cModel = (Classifier)new MultilayerPerceptron();




cMLP_H = new int[15];

cMLP_L_Value = 3;

cMLP_M_Value = 2;

int i = 0;

for (int H = 1; H <= 15; H++) {

cOptions[i] = String.format("-L 0.%d -M 0.%d -N 500 -V 0 -S 0 -E 20 -H %d", cMLP_L_Value, cMLP_M_Value, H);

cParamDesc[i] = String.format("%d\t0.%d\t0.%d", H, cMLP_L_Value, cMLP_M_Value);

cMLP_H[i] = H;

i++;

}

}

private static void setupMLP_DetermineLearningRate() {






cMLP_L = new int[5];

17

3

Ap

pe

nd

ix C.

cMLP_M_Value = 2;

int i = 0;

for (int L = 1; L <= 5; L++) {

cOptions[i] = String.format("-L 0.%d -M 0.%d -N 500 -V 0 -S 0 -E 20 -H %d", L, cMLP_M_Value, cMLP_H_Value);

cParamDesc[i] = String.format("%d\t0.%d\t0.%d", cMLP_H_Value, L, cMLP_M_Value);

cMLP_L[i] = L;

i++;

}

}

private static void setupMLP_DetermineMomentum() {






cMLP_M = new int[5];

int i = 0;

for (int M = 1; M <= 5; M++) {

cOptions[i] = String.format("-L 0.%d -M 0.%d -N 500 -V 0 -S 0 -E 20 -H %d", cMLP_L_Value, M, cMLP_H_Value);

cParamDesc[i] = String.format("%d\t0.%d\t0.%d", cMLP_H_Value, cMLP_L_Value, M);

cMLP_M[i] = M;

i++;

}

}

private static void setupMLP_FullGridSearch() {






int i = 0;

for (int H = 1; H <= 15; H++) {

for (int L = 1; L <= 5; L++) {

for (int M = 1; M <= 5; M++) {

cOptions[i] = String.format("-L 0.%d -M 0.%d -N 500 -V 0 -S 0 -E 20 -H %d", L, M, H) ;

17

4

Ap

pe

nd

ix C.

cParamDesc[i] = String.format("%d\t0.%d\t0.%d", H, L, M);

i++;

}

}

}

}

private static void displayOptions(Classifier cClassifier) {

String[] options = cClassifier.getOptions();

System.out.println("Options:");

for (int i = 0; i < options.length; i++) {

System.out.print(options[i] + " ");

}

System.out.println("");

System.out.println("=======");

}

private static void SetTrainingData_ARFF(String strPathAndFilename) {

DataSource source;

try {

source = new DataSource(strPathAndFilename);

isTrainingSet = source.getDataSet();

isTrainingSet.setClassIndex(isTrainingSet.numAttributes() - 1);




}

}

private static void SetTestData_ARFF(String strPathAndFilename) {

DataSource source;

try {

source = new DataSource(strPathAndFilename);

isTestSet = source.getDataSet();

isTestSet.setClassIndex(isTestSet.numAttributes() - 1);

17

5

Ap

pe

nd

ix C.




}

}

private static void SetTrainingData_Hardcoded() {

// Declare two numeric attributes

Attribute Attribute1 = new Attribute("firstNumeric");

Attribute Attribute2 = new Attribute("secondNumeric");

// Declare a nominal attribute along with its values

FastVector fvNominalVal = new FastVector(3);

fvNominalVal.addElement("blue");

fvNominalVal.addElement("gray");

fvNominalVal.addElement("black");

Attribute Attribute3 = new Attribute("aNominal", fvNominalVal);

// Declare the class attribute along with its values

FastVector fvClassVal = new FastVector(2);

fvClassVal.addElement("positive");

fvClassVal.addElement("negative");

Attribute ClassAttribute = new Attribute("theClass", fvClassVal);

// Declare the feature vector

FastVector fvWekaAttributes = new FastVector(4);

fvWekaAttributes.addElement(Attribute1);



fvWekaAttributes.addElement(ClassAttribute);

// Create an empty training set

isTrainingSet = new Instances("Rel", fvWekaAttributes, 10);

// Set class index

isTrainingSet.setClassIndex(3);

// Create the instance

Instance iExample = new Instance(4);

iExample.setValue((Attribute)fvWekaAttributes.elementAt(0), 1.0);

iExample.setValue((Attribute)fvWekaAttributes.elementAt(1), 0.5);

iExample.setValue((Attribute)fvWekaAttributes.elementAt(2), "gray");

iExample.setValue((Attribute)fvWekaAttributes.elementAt(3), "positive");

17

6

Ap

pe

nd

ix C.

// add the instance

isTrainingSet.add(iExample);

}

}

177

Bibliography

[1] "Stem Cell Basics," National Institutes of Health, U.S. Department of Health and

Human Services, April 2009. [Online]. Available:

http://stemcells.nih.gov/info/basics/defaultpage [Accessed: Mar 23, 2010]

[2] P. Bianco, P. G. Robey, and P. J. Simmons, "Mesenchymal stem cells: Revisiting

history, concepts, and assays," Cell Stem Cell, vol. 2, pp. 313-319, 2008.

[3] C. M. DiGirolamo, D. Stokes, D. Colter, D. G. Phinney, R. Class, and D. J. Prockop,

"Propagation and senescence of human marrow stromal cells in culture: a simple

colony-forming assay identifies samples with the greatest potential to propagate

and differentiate," British Journal of Haematology, vol. 107, pp. 275-281, 1999.

[4] S. Mareddy, R. Crawford, G. Brooke, and Y. Xiao, "Clonal Isolation and

Characterization of Bone Marrow Stromal Cells from Patients with Osteoarthritis,"

Tissue Engineering, vol. 13, pp. 819-829, 2007.

[5] S. P. Bruder, N. Jaiswal, and S. E. Haynesworth, "Growth kinetics, self-renewal, and

the osteogenic potential of purified human mesenchymal stem cells during

extensive subcultivation and following cryopreservation," Journal of Cellular

Biochemistry, vol. 64, pp. 278-294, 1997.

[6] D. Baksh, L. Song, and R. S. Tuan, "Adult mesenchymal stem cells: characterization,

differentiation, and application in cell and gene therapy," J Cell Mol Med, vol. 8, pp.

301-316, 2004.

178 Bibliography

[7] A. J. Boyle, S. P. Schulman, and J. M. Hare, "Stem Cell Therapy for Cardiac Repair:

Ready for the Next Step," Circulation, vol. 114, pp. 339-352, 2006.

[8] M. F. Pittenger, A. M. Mackay, S. C. Beck, R. K. Jaiswal, R. Douglas, J. D. Mosca, M.

A. Moorman, D. W. Simonetti, S. Craig, and D. R. Marshak, "Multilineage Potential

of Adult Human Mesenchymal Stem Cells," Science, vol. 284, pp. 143-147, 1999.

[9] A. P. Croft and S. A. Przyborski, "Mesenchymal stem cells from the bone marrow

stroma: basic biology and potential for cell therapy," Current Anaesthesia & Critical

Care, vol. 15, pp. 410-417, 2004.

[10] A. J. Friedenstein, J. F. Gorskaja, and N. N. Kulagina, "Fibroblast precursors in

normal and irradiated mouse hematopoietic organs," Experimental Hematology,

vol. 4, pp. 267-274, 1976.

[11] A. J. Friedenstein, R. K. Chailakhjan, and K. S. Lalykina, "The development of

fibroblast colonies in monolayer cultures of guinea-pig bone marrow and spleen

cells," Cell Tissue Kinetics, vol. 3, pp. 393-403, 1970.

[12] A. J. Friedenstein, K. V. Petrakova, A. I. Kurolesova, and G. P. Frolova, "Heterotopic

transplants of bone marrow. Analysis of precursor cells for osteogenic and

hematopoietic tissues," Transplantation, vol. 6, pp. 230-247, 1968.

[13] J. A. Thomson, J. Itskovitz-Eldor, S. S. Shapiro, M. A. Waknitz, J. J. Swiergiel, V. S.

Marshall, and J. M. Jones, "Embryonic stem cell lines derived from human

blastocysts," Science, vol. 282, p. 1145, 1998.

[14] M. N. Miller, "Leukocyte classification by morphological criteria," in Engineering

Foundation Conference on Automatic Cytology, Saxtons River, VT, 1972.

[15] V. A. Kovalev, A. Y. Grigoriev, and A. Hyo-Sok, "Robust recognition of white blood

cell images," Proceedings of the 13th International Conference on Pattern

Recognition, vol. 4, pp. 371-375, 1996.

[16] S. R. Sternberg, "Biomedical Image Processing," Computer, vol. 16, pp. 22-34, 1983.

Bibliography 179

[17] K. Rodenacker and E. Bengtsson, "A feature set for cytometry on digitized

microscopic images," Analytical Cellular Pathology, vol. 25, pp. 1-36, 2003.

[18] D. M. U. Sabino, L. F. Costa, E. G. Rizzatti, and M. A. Zago, "Toward leukocyte

recognition using morphometry, texture and color," IEEE International Symposium

on Biomedical Imaging: Macro to Nano., vol. 1, pp. 121-124, 2004.

[19] T. Markiewicz, S. Osowski, B. Marianska, and L. Moszczynski, "Automatic

recognition of the blood cells of myelogenous leukemia using SVM," Proceedings of

the IEEE International Joint Conference on Neural Networks, vol. 4, pp. 2496-2501,

2005.

[20] I. Maglogiannis, H. Sarimveis, C. T. Kiranoudis, A. A. Chatziioannou, N. Oikonomou,

and V. Aidinis, "Radial basis function neural networks classification for the

recognition of idiopathic pulmonary fibrosis in microscopic images," IEEE

Transactions on Information Technology in Biomedicine, vol. 12, pp. 42-54, 2008.

[21] D. Comaniciu and P. Meer, "Cell image segmentation for diagnostic pathology," in

Advanced algorithmic approaches to medical image segmentation: state-of-the-art

application in cardiology, neurology, mammography and pathology London:

Springer, 2001, pp. 541-558.

[22] S. K. Nath, F. Bunyak, and K. Palaniappan, "Robust tracking of migrating cells using

four-color level set segmentation," Lecture notes in computer science, vol. 4179, p.

920, 2006.

[23] K. Li, E. D. Miller, M. Chen, T. Kanade, L. E. Weiss, and P. G. Campbell, "Computer

vision tracking of stemness," 5th IEEE International Symposium on Biomedical

Imaging: From Nano to Macro, pp. 847-850, 2008.

[24] N. N. Kachouie, P. Fieguth, J. Ramunas, and E. Jervis, "A statistical thresholding

method for cell tracking," IEEE International Symposium on Signal Processing and

Information Technology, pp. 222-227, 2006.

180 Bibliography

[25] A. Pinidiyaarachchi and C. Wahlby, "Seeded watersheds for combined

segmentation and tracking of cells," Proceedings of the International Conference on

Image Analysis and Processing, vol. 3617, pp. 336-343, 2005.

[26] C. G. Jeffreys, "Support vector machine and parametric wavelet-based texture

classification of stem cell images." vol. Master's Thesis Cambridge, MA, USA:

Massachusetts Institute of Technology, 2004.

[27] K. Mangoubi, C. Jeffreys, A. Copeland, M. Desai, and P. Sammak, "Non-invasive

image based support vector machine classification of human embryonic stem

cells," 4th IEEE International Symposium on Biomedical Imaging: From Nano to

Macro, pp. 284-287, 2007.

[28] R. Mangoubi, M. Desai, N. Lowry, and P. Sammak, "Performance evaluation of

multiresolution texture analysis of stem cell chromatin," 5th IEEE International

Symposium on Biomedical Imaging: From Nano to Macro, pp. 380-383, 2008.

[29] O. M. Shir, V. Raz, R. W. Dirks, and T. Back, "Classification of cell fates with support

vector machine learning," in Evolutionary Computation,Machine Learning and Data

Mining in Bioinformatics. vol. 4447: Springer, 2007, pp. 258-269.

[30] J. Pan, T. Kanade, and M. Chen, "Learning to Detect Different Types of Cells under

Phase Contrast Microscopy," Microscopic Image Analysis with Applications in

Biology (MIAAB), 2009.

[31] K. Li and T. Kanade, "Cell Population Tracking and Lineage Construction Using

Multiple-Model Dynamics Filters and Spatiotemporal Optimization," in Proceedings

of the 2nd International Workshop on Microscopic Image Analysis with Applications

in Biology, Piscataway, NJ, 2007.

[32] K. Li, E. D. Miller, L. E. Weiss, P. G. Campbell, and T. Kanade, "Online Tracking of

Migrating and Proliferating Cells Imaged with Phase-Contrast Microscopy,"

Conference on Computer Vision and Pattern Recognition Workshop, pp. 65-65,

2006.

Bibliography 181

[33] K. Althoff, "Segmentation and tracking algorithms for in vitro cell migration

analysis." vol. Master's Thesis Chalmers: Chalmers University of Technology, 2005.

[34] J. D. Murray and W. Van Ryper, Encyclopedia of Graphics File Formats, 2nd ed.:

O'Reilly Media, 1996.

[35] M. D. Abramoff, P. J. Magelhaes, and S. J. Ram, "Image processing with ImageJ,"

Biophotonics International, vol. 11, pp. 36-42, 2004.

[36] K. De Vos, "Cell Counter," University of Sheffield, Academic Neurology, 12th Mar

2006. [Online]. Available: http://rsb.info.nih.gov/ij/plugins/cell-counter.html

[Accessed: 13th June 2010]

[37] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. Upper Saddle

River, NJ: Prentice-Hall, 2002.

[38] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed. New York:

Wiley-Interscience, 2001.

[39] K. Wu, D. Gauthier, and M. D. Levine, "Live cell image segmentation," IEEE

Transactions on Biomedical Engineering, vol. 42, pp. 1-12, 1995.

[40] P. Huiming, Z. Xiaobo, L. Fuhai, X. Xiaofeng, and S. T. C. Wong, "Integrating multi-

scale blob/curvilinear detector techniques and multi-level sets for automated

segmentation of stem cell images," in Biomedical Imaging: From Nano to Macro,

2009. ISBI '09. IEEE International Symposium on, 2009, pp. 1362-1365.

[41] O. Debeir, I. Adanja, N. Warzee, P. Van Ham, and C. Decaestecker, "Phase contrast

image segmentation by weak watershed transform assembly," 5th IEEE

International Symposium on Biomedical Imaging: From Nano to Macro, pp. 724-

727, 2008.

[42] O. Debeir, P. Van Ham, R. Kiss, and C. Decaestecker, "Tracking of migrating cells

under phase-contrast video microscopy with combined mean-shift processes," IEEE

Transactions on Medical Imaging, vol. 24, pp. 697-711, 2005.

182 Bibliography

[43] K. Li and T. Kanade, "Nonnegative mixed-norm preconditioning for microscopy

image segmentation," in Proc. of the 21st Biennial International Conference on

Information Processing in Medical Imaging, 2009, pp. 362-373.

[44] J. Kittler and J. Illingworth, "Minimum error thresholding," Pattern Recognition, vol.

19, pp. 41-47, 1986.

[45] F. Beltrame, B. Bianco, and A. Chiabrera, "Phase recovery from optical phase

contrast microscopy," Proceedings of the IEEE, vol. 71, pp. 270-271, 1983.

[46] I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and

Techniques, 2nd ed. San Francisco: Morgan Kaufmann, 2005.

[47] P. Spyridonos, D. Glotsos, D. Cavouras, P. Ravazoula, V. Zolota, and G. Nikiforidis,

"Pattern recognition based segmentation method of cell nuclei in tissue section

analysis," 14th International Conference on Digital Signal Processing vol. 2, pp.

1121-1124, 2002.

[48] S. J. Russell and P. Norvig, Artificial intelligence: a modern approach, 2nd ed. New

Jersey: Prentice Hall, 2003.

[49] V. N. Vapnik, The nature of statistical learning theory, 2nd ed. New York: Springer,

2000.

[50] J. Platt, "Sequential minimal optimization: A fast algorithm for training support

vector machines," Advances in Kernel Methods-Support Vector Learning, vol. 208,

1999.

[51] S. Theodoridis and K. Koutroumbas, Pattern Recognition, Third ed. San Diego:

Academic Press, 2006.

[52] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, "Textural Features for Image

Classification," IEEE Transactions on Systems, Man and Cybernetics, vol. 3, pp. 610-

621, 1973.

[53] M. Petrou and C. Petrou, Image Processing: The Fundamentals, 2nd ed. Chichester:

Wiley, 2010.

Bibliography 183

Documents

MONITORING MESENCHYMAL STEM CELL CULTURES …eprints.qut.edu.au/43623/1/Christopher_Bradhurst_Thesis.pdf · MONITORING MESENCHYMAL STEM CELL CULTURES USING IMAGE PROCESSING AND PATTERN