Upload
vuongdat
View
234
Download
0
Embed Size (px)
Citation preview
Robotic Manipulation and Characterization of Cells for Drug Screen and Clinical IVF Applications
by
Jun Liu
A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy
Department of Mechanical and Industrial Engineering University of Toronto
© Copyright by Jun Liu 2016
ii
Robotic Manipulation and Characterization of Cells for Drug
Screen and Clinical IVF Applications
Jun Liu
Doctor of Philosophy
Department of Mechanical and Industrial Engineering
University of Toronto
2016
Abstract
Heterogeneity is the hallmark of cell biology. Robotic system and automated methods are
required for characterization and manipulation of a large number of single cells for many
important applications, such as characterizing cell-cell communication for screening drug effects,
processing oocytes or embryos for cryopreservation, and analyzing sperm locomotion behavior
for selection of a high-quality sperm for in vitro fertilization (IVF). However, current robotic
characterization and manipulation systems have two major limitations in search-and-locating
end-effector tips and determination of relative Z position between end-effector tips and single
cells. This research has addressed these two limitations by developing an auto-locating end-
effector tip method and two vision-based contact detection algorithms. By integrating the two
new solutions, I have built two robotic system prototypes for microinjection of a large number of
adherent cells and automatically processing oocytes or embryos for vitrification. The robotic
microinjection system enabled automated gap junction measurement for repurposing
antiarrhythmic drugs to identify optimal treatments for cardiac arrhythmias by assessing each
drug‟s efficacy on rescuing/enhancing gap junctions. Robotic vitrification relieves human
operators from tedious and difficult manual steps, eliminating operator errors and
inconsistencies. It also offers unparalleled timing control based on real-time embryo volume
iii
monitoring. Automation enables the system to process multiple oocytes/embryos with high
efficiency. In addition, automated methods were developed to quantitatively analyze sperm
locomotion behaviors by tracking both sperm head and tail. Experimental results on automated
sperm analysis revealed a significant correlation between sperm head velocity and tail beating
amplitude. By combining automated sperm motility analysis with DNA assessment technique,
this research offers a new solution for identifying single sperms of high quality for IVF
applications. In the end, this research also reviewed future research directions in microsystems
and nanoengineering for intracellular measurement and manipulations and discussed the
promising research for the intracellular study.
iv
Acknowledgments
It has been a relatively long journey since I first came to UofT in 2011. I feel very lucky and
grateful to have a great supervisor, Professor Yu Sun, who guided me through this exploration.
At first, I would like to sincerely thank Prof. Sun for his continuously strong support in my
study, research, personal life and career development. Professor Sun is a wonderful advisor, a
great writer, and a visionary leader. Without his guidance and support, I would not be able to
manage through this tough journey to push the boundary of knowledge for my Ph.D. degree.
Secondly, I want to thank Professor Goldie Nejat, Professor Lidan You, Dr. Jason Maynes and
Professor Jason Gu for serving on my Ph.D. supervisory and defense committees. Their
insightful comments and suggestions are critical for my Ph.D. research. I really appreciate their
time and support in helping me through the completion of my thesis.
Thirdly, I would like to thank all my talented and happy-to-be-around lab colleagues, who
offered generous help in designing research projects, conducting experiments, and drafting paper
manuscripts. Among all my lab colleagues, I should especially thank Zhe Lu and Clement, who
taught me at the very beginning of my study in AMNL. Also, I would like to give my special
thanks to Brandon, Yanliang, Yong Zhang, and Yi Zheng, who provide generous help at the
beginning and throughout my time in AMNL. In addition, I enjoyed working and laughing with
Haijiao, Jun Wen, and Zhuoran. You guys know me. Also, I want to thank Derek, John, Yuan,
Wes, and Devin, who shared with me a lot intercultural funny stuff.
Especially, I want to express my appreciation to all my collaborators, especially Dr. Robert
Hamilton, Dr. Jason Maynes, and Dr. Sevan Hopyan. Their valuable knowledge and insightful
suggestions inspired me to design research projects, conduct experiments and understand results.
It is you doctors who provided such great collaboration opportunities for me to realize my true
value of being an engineer to contribute to our society.
In the end, I would like to thank my parents and my two beloved sisters, who always stand
behind me to support. Lastly but most importantly, I want to thank my dear wife, Qiaorong Fan.
Without her understanding, support, love, and care for our family, I would not be able to travel
this far. My entire Ph.D. work is incomparable to your great achievements of two angels – Alan
and Daniel. It is your endless love that brings me true happiness.
v
Contribution of Co-authors
I am the principle or co-principle author of all the articles on which this thesis is partially based.
I have performed all or the majority of experiments, data collection, analysis, and manuscript
preparation for each article. I would like to acknowledge all of my co-authors for their valuable
contributions. Also, I acknowledge my supervisor, Professor Yu Sun, who provided knowledge,
ideas and directions that were critical in my entire research and he was involved in manuscript
preparation for each article presented here. The following outlines the contributions of each co-
author:
[1] J. Liu, Z. Gong, K. Tang, Z. Lu, C. Ru, J. Luo, S. Xie, and Y. Sun, "Locating end-effector
tips in robotic micromanipulation," IEEE Transactions on Robotics, Vol. 30, pp. 125-30,
2014.
Professor Y. Sun was involved in project design and manuscript preparation. Discussions
regarding research concepts, collection and interpretation of data were continuously
conducted among all contributing authors.
[2] J. Liu, V. Siragam, Z. Gong, J. Chen, M.D. Fridman, C. Leung, Z. Lu, C.H. Ru, S.R. Xie, J.
Luo, R. Hamilton, and Y. Sun, "Robotic adherent cell injection (RACI) for characterizing
cell-cell communication," IEEE Transactions on Biomedical Engineering, vol. 62, no. 1, pp.
119-24, 2015.
Professor Y. Sun and Dr. R. Hamilton were involved in project design and manuscript
preparation. V. Siragam was involved in cell preparation and data interpretation. Z. Gong, J.
Chen, C. Leung, and Z. Lu helped in building the system hardware and programming
software. Discussions regarding research concepts, collection and interpretation of data were
continuously conducted among all contributing authors.
[3] J. Liu, V. Siragam, J. Chen, M.D. Fridman, R.M. Hamilton, and Y. Sun, "High-throughput
measurement of gap junctional intercellular communication," American Journal of
Physiology - Heart and Circulatory Physiology (AJP-Heart), Vol. 306, pp. H1708-13, 2014.
Professor Y. Sun and Dr. R. Hamilton were involved in project design and manuscript
preparation. V. Siragam and M. Fridman were involved in cell preparation and data
vi
interpretation. Discussions regarding research concepts, collection and interpretation of data
were continuously conducted among all contributing authors.
[4] J. Liu, C.Y. Shi, J. Wen, D.G. Pyne, H.J. Liu, C.H. Ru, S.R. Xie, Y. Sun, “Automated
Robotic Vitrification of Embryos,” Robotics & Automation Magazine, vol. 22, pp. 33-40,
2015.
Professor Y. Sun was involved in project design and manuscript preparation. Discussions
regarding research concepts, collection and interpretation of data were continuously
conducted among all contributing authors.
[5] J. Liu, C. Leung, Z. Lu, and Y. Sun, "Quantitative analysis of locomotive behavior of
human sperm head and tail," IEEE Transactions on Biomedical Engineering, Vol. 60, pp.
390-396, 2013.
Professor Y. Sun and Z. Lu were involved in project design and manuscript preparation.
Discussions regarding research concepts, collection and interpretation of data were
continuously conducted among all contributing authors.
[6] J. Liu, C. Moraes, Z. Lu, C.A. Simmons, and Y. Sun, "Single cell deposition," Methods in
Cell Biology, Vol. 112, pp. 403-420, 2012.
Professor Y. Sun and C.A. Simmons were involved in designing the project. All authors
were involved in manuscript preparation.
[7] J. Liu, J. Wen, Z.R. Zhang, H.J. Liu, and Y. Sun, “Voyage inside the Cell: Microsystems
and Nanoengineering for Intracellular Manipulation and Characterization”, Microsystem &
Nanoengineering, vol. 1, p. 15020, 2015.
Professor Y. Sun designed the project and all authors are involved in manuscript preparation.
Discussions regarding research concepts and literature reviews were continuously conducted
among all contributing authors.
viii
Table of Contents
Acknowledgments.......................................................................................................................... iv
Contribution of Co-authors ..............................................................................................................v
Table of Contents ......................................................................................................................... viii
List of Tables ................................................................................................................................ xii
List of Figures .............................................................................................................................. xiii
Chapter 1 Introduction .....................................................................................................................1
Introduction .................................................................................................................................1 1
1.1 Single Cell Characterization and Manipulation ...................................................................1
1.2 Characterization of Gap Junctional Intercellular Communication ......................................4
1.3 Robotics and Automation for in vitro Fertilization..............................................................6
1.3.1 Robotic Vitrification of Mammalian Embryos ........................................................6
1.3.2 Quantitative Analysis of Sperm Locomotion Behavior ...........................................8
1.4 Previous Methods for Robotic Cell Manipulation .............................................................10
1.5 Opportunities for Innovations in Robotic Cell Manipulation ............................................16
1.6 Research Objectives ...........................................................................................................17
1.7 Dissertation Outline ...........................................................................................................18
Chapter 2 Automatically Locating End-effector Tips in Micromanipulation ...............................19
Automatically Locating End-Effector Tips in Micromanipulation ...........................................20 2
2.1 Introduction ........................................................................................................................21
2.2 System Design ...................................................................................................................23
2.2.1 System Architecture ...............................................................................................23
2.2.2 Overall Sequence ...................................................................................................24
2.3 Key Methods for Auto-Locating End-effector Tips ..........................................................25
2.3.1 End-Effector Sweep Pattern Design ......................................................................25
ix
2.3.2 Computer Vision Based End-Effector Detection ...................................................26
2.3.3 Detection of End-Effector Contours ......................................................................27
2.3.4 Autofocusing on End-Effector Tips .......................................................................28
2.3.5 Centering End-Effector Tips via Visual Servo Control .........................................30
2.4 Experimental Results and Discussion ................................................................................31
2.4.1 Overall Performance ..............................................................................................31
2.4.2 Performance Under Different Microscopy Imaging Modes ..................................32
2.4.3 Discussion ..............................................................................................................35
2.5 Conclusion and Guidelines for Implementation ................................................................37
Chapter 3 Robotic Microinjection of Adherent Cells ....................................................................38
Robotic Microinjection of Adherent Cells ................................................................................39 3
3.1 Introduction ........................................................................................................................39
3.2 System Setup ......................................................................................................................42
3.3 Vision Based Contact Detection ........................................................................................44
3.3.1 Contact Detection on Cell Culture Surface ............................................................44
3.3.2 Contact Detection on Cell Membrane....................................................................45
3.4 Robotic Microinjection ......................................................................................................49
3.5 Experimental Results and Discussion ................................................................................51
3.5.1 Performance of Locating Micropipette Tip ...........................................................51
3.5.2 Injection Speed.......................................................................................................51
3.5.3 Injection Success Rate and Cell Survival Rate ......................................................52
3.5.4 Characterization of Cell-Cell Communication ......................................................53
3.5.5 Discussion ..............................................................................................................56
3.6 Conclusion .........................................................................................................................56
Chapter 4 Robotic Vitrification of Mammalian Embryos .............................................................57
Robotic Vitrification of Mammalian Embryos .........................................................................58 4
x
4.1 Cryopreservation of Oocytes and Embryos .......................................................................58
4.2 System Overview ...............................................................................................................61
4.3 Key Methods for Robotic Vitrification ..............................................................................63
4.3.1 Contact Detection...................................................................................................63
4.3.2 Embryo Tracking in Three-Dimensional Space ....................................................64
4.3.3 Placing Embryo on Vitrification Straw ..................................................................68
4.4 Results and Discussion ......................................................................................................70
4.4.1 System Performance ..............................................................................................70
4.4.2 Embryo Volume Measurement ..............................................................................71
4.4.3 Post-Freezing Survival and Development Rates....................................................73
4.4.4 Discussion ..............................................................................................................74
4.5 Conclusion .........................................................................................................................75
Chapter 5 Automated Analysis of Sperm Locomotion Behavior ..................................................76
Automated Analysis of Sperm Locomotion Behavior ..............................................................77 5
5.1 Introduction ........................................................................................................................77
5.2 System Setup and Experimental Design ............................................................................80
5.3 Computer Vision Methods for Sperm Tracking ................................................................81
5.3.1 Overview ................................................................................................................81
5.3.2 Sperm Head Tracking ............................................................................................82
5.3.3 Sperm Tail Tracking ..............................................................................................85
5.4 Experimental Results and Discussion ................................................................................88
5.5 Conclusion .........................................................................................................................93
Chapter 6 Conclusions and Future Research .................................................................................94
Conclusions and Future Research .............................................................................................94 6
6.1 Contributions......................................................................................................................94
6.2 Future Research .................................................................................................................95
xi
6.2.1 Improvement of the Present Research ...................................................................95
6.2.2 Research Outlooks on Intracellular Measurement and Manipulations ..................97
References ....................................................................................................................................100
xii
List of Tables
Table 1 - Previous Methods for Robotic Micromanipulations using Mechanical Strategy .......... 13
Table 2 - Experiment Results of Auto-Locating End-Effector Tips Under Three Different
Microscopy Imaging Modes ......................................................................................................... 33
Table 3 - Effect of Different Threshold Values on Detection Time and Overall Success Rate
(Each Data Point in the Table Was from 100 Trials, All Under Bight Field Imaging) ................ 35
Table 4 - Robotic Microinjection Speed of Three Different Cell Lines ....................................... 51
Table 5 - Injection Success Rate and Cell Survival Rate After Injection ..................................... 52
Table 6 - Embryo Vitrification Experimental Results .................................................................. 71
Table 7 - Dynamic Representation of Sperm Positions in Multi-Sperm Tracking Algorithm ..... 83
Table 8 - Performance Comparison Between MHI-Based Multi-Sperm Tracking and GNN-Based
Multi-Sperm Tracking .................................................................................................................. 88
Table 9 - Average Tracking Error of Sperm Tail Tracking .......................................................... 89
xiii
List of Figures
Figure 1-1 - Schematic illustrations of typical cell manipulation strategies, and techniques for
characterization and manipulation of single cells with mechanical strategy. ............................... 10
Figure 2-1 - Example end-effectors used in micromanipulation. Their micrometer-sized tips must
be located first before micromanipulation initiates. (a) Microgrippers for pick-and-place of small
objects. The grasped particles are 10 µm in diameter. (b) Micropipettes for manipulating
biomaterials. .................................................................................................................................. 21
Figure 2-2 - (a) Schematic illustration of the experimental setup. (b) System picture. ................ 23
Figure 2-3 - Overall sequence for auto-locating end-effector tips contains two major steps. ...... 24
Figure 2-4 - Sweep pattern in X–Y plane (top-down view). (a) When the direction of entering the
FOV is known, end-effector is moved along the direction from which it enters the FOV,
followed by sweeping perpendicularly. (b) When the direction of entering the FOV is not known,
end-effector is swept in a zigzag pattern....................................................................................... 25
Figure 2-5 - Detection of end-effector. Arrows in (b) and (e) show the motion gradient of the
moving end-effectors. (a) Original image of a micropipette. (b) Corresponding MHI of the
moving micropipette. (c) Tip detection result based on active contour is shown in a background-
removed image. (d) Original image of microgripper. (e) Corresponding MHI of the moving
microgripper. (f) Tip detection result based on active contour is shown in a background-removed
image. ............................................................................................................................................ 28
Figure 2-6 - Most in-focus region of the end-effector is detected by using a quad-tree recursive
searching method. (a) The most in-focus region is detected to be on the end-effector body (not on
the tip). (b) The tip is brought to focus (i.e., the most in-focus region is now on the tip). ........... 29
Figure 2-7 - Time of manually locating end-effector tip; auto-locating end-effector tip with
unknown and known entering direction. Within each group (distance to initial focus plane), n =
20 (ten trials to locate micropipette tip and ten trials to locate microgripper tips). ...................... 31
xiv
Figure 2-8 - Active contour detection under three different microscopy imaging modes. (a) bright
field imaging, (b) DIC, and (c) phase contrast. ............................................................................. 34
Figure 2-9 - Auto-locating microgripper tips failed due to other structure appeared in the FOV
first. ............................................................................................................................................... 36
Figure 3-1 - Adherent cells (HL-1) under 20x magnification objective. The inset shows the
schematic of a molecule passing through a gap junction from one cell to its adjacent cell. ........ 39
Figure 3-2 - System setup. (a) Robotic adherent cell injection (RACI) system. (b) A schematic
showing of RACI system. (c) System control architecture........................................................... 42
Figure 3-3 - Contact detection on cell culture surface. (a) Micropipette tip is lowered towards the
cell culture surface while the X-Y stage moves along the X-axis from left to right. (b)
Micropipette tip slides horizontally when it contacts the culture surface. (c) Schematic showing
micropipette is lowered towards cell culture surface. (d) Schematic showing micropipette tip is
sliding along the X-axis. ............................................................................................................... 44
Figure 3-4 - Detection of contact on the cell surface (a) At high cell confluency, the system
detects direct contact of the pipette tip on the cell surface. (b) ROI image shows cell being
deformed: white dot indicates tip position. (c) Modified ROI removing the motion of
micropipette tip. (d) Motion history image showing the deformation of the cell: red arrows
indicate motion gradients. ............................................................................................................. 46
Figure 3-5 - Robotic injection of adherent cells. (a)(b) When the micropipette is inserted into the
cell, material deposition triggered by pressure pulses results in a “shock wave” motion inside the
cell. (c) Fluorescence is present only in the cytoplasm. (d) Fluorescence is present only in the cell
nucleus. ......................................................................................................................................... 49
Figure 3-6 Injection volume calibration results indicate that the injection pressure should be set
lower than 3000 hPa in order to achieve a high cell survival rate. ............................................... 50
Figure 3-7 - Only one cell is injected in an area, and fluorescence dye is transferred from the
injected cell to adjacent cells through gap junctions. ................................................................... 53
xv
Figure 3-8 - Results of dye transfer experiments. (a) Histogram of dye transfer distance for HeLa
cells (n = 200), HEK293 cells (n = 200), and HL-1 cells (n = 400). (b) The number of HL-1 cells
uptaking fluorescence dye from the injected cells. Over 200 cells were injected for each group. 55
Figure 4-1 - The schematic showing manual and RoboVitri approaches. Vitrification involves
multiple steps of cell pick-and-place before freezing in liquid nitrogen. ..................................... 58
Figure 4-2 - (a) The RoboVitri system prototype. The inset is the custom-designed carrier plate.
(b) The system control architecture. ............................................................................................. 61
Figure 4-3 - The detection of contact between the micropipette tip and the multi-well plate
substrate and contact between micropipette tip and vitrification straw surface. (a) and (b) The
schematic of contact detection on the plate substrate and vitrification straw, respectively. (c) and
(d) The system moves the micropipette downward. (e) A further downward movement after
contact induces the micropipette tip‟s sliding motion on the plate substrate surface. (f) A further
downward movement after contact deflects the vitrification straw tip, causing it to become out of
focus. ............................................................................................................................................. 63
Figure 4-4 - Embryo tracking in the VS. Left column: the original images with embryos at
different heights. Middle column: the ROIs containing the tracked embryo target. Right column:
the tracked embryo locations indicated by a red dot in the binarized image. (a) An embryo is
dispensed out of micropipette into the VS solution. (b) An embryo is floating upward, due to
buoyancy, whereas the system performs autofocusing to control focus positions. ...................... 65
Figure 4-5 - A processed embryo placed on the vitrification straw tip. (a) An embryo is deposited
onto the straw by dispensing the VS medium out of the micropipette. (b) The excess medium is
removed by micropipette aspiration under a threshold flow rate to keep the embryo in place. (c)
The schematic showing the embryo dynamics during micropipette aspiration. ........................... 69
Figure 4-6 - The 3-D-tracking results: (a) the z position change of a tracked embryo and (b) the
volume change of three embryos in the same VS measured by the robotic system. .................... 72
Figure 4-7 - Example embryo images before and after vitrification. ........................................... 73
xvi
Figure 5-1 - Experiment design. (a) and (b) Experiment 1: a sperm changed its position between
two frames. (c) and (d) Experiment 2: a sperm was moving toward an HA microdot and then
bound to the HA microdot. ........................................................................................................... 80
Figure 5-2 - Tracking multiple sperm heads. (a) Red dots are tracked sperm heads. (b)
Corresponding motion history image............................................................................................ 82
Figure 5-3 - Sperm tail tracking. (a) Sperm head position is found. (b) STROI is determined. (c)
5×5 windows are scanned to locate the section with the highest intensity sum in the flicker
image. The center point (blue dot in the figure) of the section is considered the tail location. (d)
Based on the blue dot position found in the flicker image, Kalman filter is applied to improve the
accuracy of located sperm tail position. ........................................................................................ 85
Figure 5-4 - Failure cases in sperm head tracking. (a) False positive detection of sperm head in
shape-based tracking is caused by the stationary object/particle with a shape similar to sperm
head. (b) False negative case: shape-based detection algorithm fails to detect the sperm head that
has an abnormal head shape. (c) False positive detection of sperm head in MHI-based detection
is caused by moving debris. (d) False negative case: MHI-based detection algorithm fails to
detect stationary/slow-moving sperms. ......................................................................................... 88
Figure 5-5 - Correlation between sperm velocity and tail beating amplitude. Pearson‟s correlation
coefficient r = 0.7225. ................................................................................................................... 90
Figure 5-6 - Sperms on a PICSI dish with HA microdots. (a) Sperms binding after 5 min. (b)
Sperms binding after 30 min. ........................................................................................................ 91
Figure 5-7 - Velocity comparison between the HA-bound sperms and unbound sperms. ........... 92
Figure 5-8 - Tail beating amplitude of the same sperms before and after binding to HA
microdots....................................................................................................................................... 93
1
Chapter 1 Introduction
This chapter discusses the background knowledge for robotic single cell manipulations and
characterization. In this work, adherent cells, oocytes/embryos and sperm cells are used as
representative examples to demonstrate robotic and automated methods for characterizing cell-
cell communication, picking-and-placing single cells, and analyzing sperm locomotion
behaviors. A review of previous methods is also presented in this chapter.
Introduction 1
1.1 Single Cell Characterization and Manipulation
Population-based techniques, such as flow cytometry, in biological experimentation are unable to
probe the rich information available from the study of single cells [1]. Heterogeneity is a
hallmark of cell biology and is strongly evident in primary cell populations isolated from the
same tissue [2] and well-established cell lines [3], [4]. Furthermore, supposedly identical clonal
cell populations have been shown to deviate in their genetic expression [5] and response to
environmental stimuli over generations of cell division [6]. This diversity has significant
implications in coordinating multicellular behaviors and is of critical importance in
developmental biology, pathobiology, and tissue engineering. For example, the heterogeneous
responses to apoptosis-inducing drugs or ligands play a critical role in discovering new treatment
for various diseases such as cancer, autoimmune disorders and neurodegenerative diseases [7].
Single-cell measurements are key to accurate characterization and modeling of heterogeneous
responses. Therefore, single cell characterization and manipulations are necessary to understand
the cellular basis for population behavior; and can also yield new methods for discovery of
signaling pathway mechanisms and biochemical basis for cellular function [8].
Although single cell studies have attracted researchers‟ extensive focus and efforts, sensing and
quantifying single cell behaviors and properties from bulk analyses are not easy [9]. Their
signals tend to be swamped by noise from the general population and the distinction between
individual cells are blurred, making it difficult or impossible to understand which cell contributes
to what effects in the entire cell populations or tissues [10] [11].
2
Recent advances in robotics and automation, especially at micro/nano scales, have demonstrated
their feasibility and unique contributions in achieving high-throughput characterization and
manipulations of single cells. Compared to manual operation, robotic and automated systems
have the inherent advantages of high speed and high positioning accuracy. For example,
microinjection is a widely used technique to deliver foreign materials (e.g., fluorescent
molecules, DNA/RNA) into single cells. However, manual operation can only inject several to
tens of cells [12], limiting its usefulness when a large number of single cells need to be tested.
Furthermore, manual injection‟s success rate and post-injection cell survival rate are largely
dependent on the operator‟s skills and can vary significantly from case to case.
In addition to the advantages in speed and accuracy, robotic and automated methods can provide
objective results with improved reproducibility. For example, computer-aided sperm analysis can
produce objective quantifying-matrices to assist doctors or embryologists to select high-quality
sperm for in vitro fertilization (IVF). In contrast, manual selection of a single sperm is ad hoc,
largely depending on the operator‟s personal knowledge and expertise.
In general, biological cell lines in vitro can be classified into two groups: adherent cells and
suspended cells. Adherent cells (e.g., cardiomyocytes, endothelial cells), as the major cell types
derived mostly from vertebrates, are anchorage-dependent and need to be cultured on certain
types of substrate that are specially treated to allow cell adhesion and spreading. In contrast,
suspended cells (e.g., oocytes/embryos, blood cells, and sperm cells) are cultured in suspension
mediums and do not require adhesion to the culture substrate. In the group of suspended cells,
sperms are a special type of cells that can generate freely-swimming motion. Typical
manipulations for adherent cells are microinjection and biopsy, whereas manipulations for
suspended cells also include the pick-and-place or translocation of single cells. This research
covers all typical cell manipulations including microinjection of adherent cells, pick-and-place of
suspended cells manipulations, and analysis of sperm locomotion behaviors.
To assist single cell characterization and manipulation, this work focuses on the development of
enabling micro-robotic systems and automated methods for characterizing cellular properties
(e.g., cell-cell communication), analyzing cell behavior, and manipulating single cells. This
research involves the development of the first-of-its-kind robotic adherent cell injection system
for characterization of gap junctional intercellular communication, which has significant
3
implications for screening drug efficacy for cardiac diseases caused by the dysfunction of cell-
cell communication. Additionally, this thesis also introduces a robotic system to pick-and-place
mammalian oocytes or embryos for cryopreservation. Besides, this research also includes the
development of computer vision algorithms for tracking both sperm head and tail to facilitate the
analysis of sperm locomotion behaviors for IVF applications.
4
1.2 Characterization of Gap Junctional Intercellular Communication
Gap junctional intercellular communication (GJIC) exists in most mammalian tissues. Gap
junction is a specialized intercellular channel formed by the juxtaposition of two half channels
called connexons [13], [14]. Gap junctions are critical to several physiological roles, including
impulse propagation in cardiac and neuronal tissue [15], regulation of embryonic development
[16], homeostasis [17], and regulation of cellular proliferation [18]. Disorders of GJIC have been
associated with many pathological conditions and vital diseases, such as cardiac arrhythmias
[19], [20], which are a leading cause of cardiac morbidity and sudden death [21].
Gap junctions provide a direct pathway for electrical and metabolic signaling between
connecting adjacent cells. Similar to ion channels, gap junctions are also ionic conduits [22].
However, unlike conventional ion channels, the selectivity of gap junctions for small ions (either
positively or negatively charged) is minor compared to the selectivity of Na+ and K
+ channels.
The size of gap junction channels is much larger than that of conventional ion channels (100-150
Å vs. 3-5 Å) [23].
Gap junctions facilitate both diffusion of small molecules and conduction of ions between
adjacent cells. Intercellular diffusion is attributed to essentially only through gap junctions,
whereas sodium channels also contribute to conduction in the heart. In addition to ionic
communication, gap junctions are molecular sieves that allow small hydrophilic chemical
molecules, such as metabolites and second messengers (e.g., triphosphate inositol), to pass [24].
Gap junctions are regulated by gating that is referenced as open, semi-open, and closed
configurations. When all the channels in a gap junction gate are closed, diffusion of permeants
from cell to cell is halted. Experimentally, gating is commonly induced by imposing a voltage
gradient across the junction. In addition to gating, the gap junction-mediated intercellular
communication is also regulated by intracellular calcium, pH, and phosphorylation [25].
GJIC is regulated by the number of gap junction channels in the membrane, the functional state
of gap junctions, and their permeability [26]. The measurement of GJIC has been studied for
decades by using a number of methods, such as dye transfer through microinjection [27], the
scrape/scratch method [28], electroporation [29], fluorescence redistribution after photobleaching
[30], and conductance measurement by dual-cell patch clamp [31]. Among these techniques,
5
microinjection of membrane-impermeable, non-toxic tracers into single cells has been the most
commonly used technique for identifying and mapping GJIC for a wide variety of cells [32]. The
microinjection method is considered superior to other techniques because of the following
reasons: 1) microinjection permits the correlation of morphological and functional data from
individual cells; 2) the technique enables kinetic studies aimed at evaluating the transfer rate
from one cell to another; and 3) in microinjection, the level of cell communication is expressed
as number of dye-coupled cells, permitting the direct comparison of GJIC in different cell types
[33].
However, microinjection has stringent skill requirements, low cell viability rate, and poor
reproducibility [12]. Manual microinjection is typically used for injecting a few or tens of cells
per experiment, limiting its usefulness when a large number of cells need to be tested for GJIC
assessment. In this research, I have developed a robotic microinjection system and technique, for
the first time, to enable the injection of hundreds of adherent cells per experiment rapidly and
accurately. The automated system is operated via computer mouse clicking for indicating target
cells for microinjection. Training a user who has no skills in microinjection takes ~15 min, and
after a few hours of operation, the user can readily become proficient at operating the system to
perform microinjection with high success rates.
6
1.3 Robotics and Automation for in vitro Fertilization
The World Health Organization reports that averagely 8~10% of couples worldwide experience
infertility [34]. And data from the US National Centre for Disease Control shows that one out of
eight North American couples seek medical treatment for infertility. Since the birth of Louise
Brown, conceived using in vitro fertilization (IVF) pioneered by Noble Prize Laureate – Robert
Edwards, the outcomes and impact of assisted reproductive theologies (ART) have been
remarkable [35][36]. Despite the significant progress in ART, most operations in IVF clinics are
still performed manually. Treatment outcomes largely rely on expert knowledge and experienced
hands. To address the bottleneck problem in IVF, this research involves the development of
robotic systems and automated methods aiming to standardize two IVF routine procedures: (1)
vitrification of oocytes or embryos, and (2) analysis of sperm locomotive behaviors.
1.3.1 Robotic Vitrification of Mammalian Embryos
Cryopreservation of mammalian reproductive cells is an essential technique in IVF clinics
[37][38]. Oocytes and embryos are routinely frozen and preserved for use in future. Nowadays,
as more and more young ladies choose to focus on career development while they are young,
cryopreservation is becoming a popular technology for preserving high-quality oocytes from
their young age, allowing them to defer their family planning to the future [39]. Additionally,
patients who undergo therapeutic procedures that can place their fertility at risk, such as
chemotherapy, have the option of preserving their oocytes for future use through IVF
cryopreservation. Furthermore, fertilized embryos are often more than needed for transferring
back to mother uterus in one cycle of IVF treatment. The rest of the fertilized embryos are
usually cryopreserved for future use. The length of time an embryo is frozen in liquid nitrogen
has been shown not to have a significant impact on clinical pregnancy, miscarriage, implantation,
or live birth [40].
Techniques of oocyte/embryo cryopreservation are classified into two categories, slow freezing
and fast freezing (i.e., vitrification). Both techniques aim to minimize cell damage that is largely
due to the formation of intracellular ice crystals at low temperature [41]. Slow freezing is a well-
established technique developed during the early 1970s, which makes use of programmable
sequences, or controlled cooling rates. During slow freezing, extracellular water freezes away
from the embryo using a seeding technique, which creates an osmotic gradient that draws water
7
out of the cell until it finally freezes without the formation of intracellular ice crystals [42]. This
procedure requires sophisticated equipment to control the freezing rate, which ranges between
0.3 and 1.0 µ°C/min, and produces a relatively poor survivability rate [43].
On the other hand, vitrification or fast freezing is a more effective cryopreservation method, first
reported in 1985 [44]. Vitrification is considered superior to slow freezing because it vitrifies the
oocyte/embryo with no crystal formation during freezing. The addition of cryoprotectants in
vitrification increases the cytosol viscosity and makes the vitrified oocyte/embryo syrupy. When
directly freezing oocytes/embryos in liquid nitrogen, the syrupy content inside the cell forms
amorphous ice instead of ice crystals, which minimize the vital damage to the cell during
freezing.
At present, oocyte/embryo vitrification is done manually in IVF clinics. An operator looks
through the microscope eyepieces and manipulates oocytes/embryos using a Stipper®
micropipette or mouth pipette. Manual operation for oocyte/embryo vitrification is a demanding
and tedious task, for the following reasons:
(1) vitrification requires precise washing sequences and strict timings in each vitrification
solution (VS) because the cryoprotectants (e.g., DMSO) are toxic and can significantly
damage the cell viability if overexposed in vitrification solution;
(2) because of their small size (about 150 µm), oocytes/embryos can be difficult to observe
and manipulate, especially when the oocytes/embryos locations are dynamically changing
during micropipette aspiration and dispensing;
(3) washing oocytes/embryos with the highly viscous VS can cause osmotic shock to the
cells, which can damage the cell survivability in vitrification;
(4) the manual process has stringent skill requirements, and success rate and cell survival rate
vary across different operators. In IVF clinics, processing an embryo/oocyte in
cryoprotectant medium typically takes a highly skilled embryologist 10 to 15 minutes.
To address the shortcomings of manual vitrification, this research focuses on the development of
a robotic system for automatically pick-and-placing the oocytes/embryos through various
vitrification solutions.
8
1.3.2 Quantitative Analysis of Sperm Locomotion Behavior
In natural fertilization, a healthy sperm overcomes the physiological and biological selection
barriers, actively seeks out and fertilizes an egg. Sperm selection occurs naturally in this
procedure. However, for couples having infertility issues, assisted reproduction technologies are
required to address their reproductive needs. For example, in intracytoplasmic sperm injection
(ICSI), an embryologist selects a single sperm cell and injects it directly into an oocyte (i.e., egg
cell) to overcome issues such as male infertility [45]. These assisted reproduction technologies
bypass the natural sperm selection barriers and demand expertise knowledge from the operator to
select high-quality sperms. The criteria for sperm quality assessment provided by the World
Health Organization are vitality, morphology, and motility [46]. A widely used method for sperm
selection is motile sperm organelle morphology examination (MSOME) [47][48][49][50]. Sperm
motility is also a commonly used criterion for sperm quality assessment. A motility grade is
often used as a specified measure and classified into four grades:
Grade 1: sperm with fast progressive movements;
Grade 2: sperm with slow progressive movements;
Grade 3: sperm with slow non-progressive movements (i.e., with curved motion);
Grade 4: sperms are immobile and fail to move at all.
Besides sperm assessment based on morphology and motility, another method for selecting a
healthy sperm is based on the analysis of sperm DNA integrity. Some of the DNA analysis
methods assess sperm DNA quality directly, such as the TUNEL assay and the sperm chromatin
structure assay. Some other methods indirectly measure sperm DNA quality. For instance,
Huszar‟s group recently proposed a hyaluronic acid (HA) assay [51]. Among a consecutive
series of studies on HA-based sperm assay, researchers in Huszar‟s group indicated that HA
assay permits the selection of healthy sperm with no DNA damage [52][53]. In their research,
sperms that bind to HA microdots are proven to have a higher level of DNA integrity compared
to those unbound sperms. In clinical HA-based sperm selection, a number of healthy sperms‟
heads bind to the HA microdot and lose their progressive movement despite vigorous tail
beating. In this case, the sperm tail beating movement becomes the only indicator to differentiate
the HA-bound sperms from each other.
9
The past few decades have witnessed the development of computer-assisted sperm analysis
(CASA) methods for measuring both sperm morphology and motility [54]. CASA utilizes
automated systems to digitize successive images of sperm, process and analyze the information,
and provide the accurate and objective value for individual sperm cell. Since the 1970s, many
algorithms have been developed to track sperm trajectories, measure sperm velocities, and
analyze sperm morphology. Shi et al. reported a robust single-sperm tracking algorithm based on
a four-class thresholding method to extract a single sperm in a region of interest [55]. The
nearest neighbor method is complemented with a speed-check feature to aid tracking in the
presence of additional sperm or other particles. In another study, Nafisi et al. demonstrated a
template matching algorithm for sperm tracking. The algorithm is insensitive to image
acquisition conditions [56]. Existing algorithms for sperm tracking are largely limited to sperm
head tracking. The small size (≤ 1 µm in thickness) and low contrast of sperm tails under optical
microscopy makes tracking sperm tails challenging.
This research also involves the development of a new automated analysis method by tracking
both sperms‟ heads and tails. With this automated method, users can obtain objective
information about sperm locomotion behaviors. This research has significant applications to
assist the selection of high-quality sperm by unifying the sperm motility analysis with DNA
assessment techniques (e.g., HA assay) in IVF clinics.
10
1.4 Previous Methods for Robotic Cell Manipulation
The past decade has seen significant development of robotic cell manipulation systems and
automated characterization techniques for patterning, grasping, measuring, and injecting
individual cells [57][58][59][60]. In general, the strategies for single cell manipulations and
characterization can be classified into five categories: mechanical, fluidic, electrical, optical and
magnetic.
Figure 1-1 - Schematic illustrations of typical cell manipulation strategies, and techniques
for characterization and manipulation of single cells with mechanical strategy.
The mechanical strategy, which directly acts on target cells with physical tools, is the oldest and
still the most commonly-used method for single cell manipulation in biology labs [61]. In this
category, micropipettes are often used to inject aqueous materials into single cells [62], pick-and-
place single cells [63], and extract inner cell organelles or samples (e.g., polar body biopsy) [64].
In addition to conventional glass or plastic micropipettes, MEMS-based microgrippers have also
been designed and fabricated for picking-and-placing single cells and characterizing their
mechanical properties [65][66]. Mechanical strategy is the simplest approach and easy to use by
directly touching the cells with physical force. However, direct interactions between single cells
and micro tools can often introduce unexpected disturbances to cellular normal functions and
11
damage cell viabilities, especially when the experiments are conducted by users with little
micromanipulation experience.
The second group of manipulation strategies use microfluidic devices to move single cells within
constrained channels [67][68]. In microfluidic manipulations, researchers have applied flow-
induced forces to characterize the cellular mechanical properties [69][70]. For automation in
IVF, such as for vitrification of embryos and sperm quality assessment, microfluidic approaches
have also become increasingly popular. The advantages of using microfluidic devices include the
pre-patterned motion paths, minimal volume requirement, reduction of material costs, and
decreased reaction time with high surface-to-volume ratio [71]. However, the difficulties in
introduction and extraction of single cells on microfluidic devices greatly limit the practicality
for targeting single cells of interest [72]. Moreover, the measured and processed single cells are
difficult to be extracted from the large cell populations.
Different from mechanical and fluidic manipulations, which involve physical contact with single
cells, electrical, optical and magnetic manipulations are capable of interacting with single cells in
an untethered way. Electrical manipulations, such as dielectrophoresis (DEP) [73], generate a
non-uniform electrical field to exert electrical forces on dielectric particles or cells. DEP
manipulates cells without contacting them and is easy to control well [74]. However, the
generated forces strongly depend on surrounding medium, cell electrical properties, cell size and
shapes [75].
Optical manipulations, or optical tweezers, use a highly focused laser beam to produce an
attractive or repulsive force depending on a refractive index mismatch to physically hold and
move micro/nano-objects [76]. Although the optical tweezers can accurately position single cells
with minimal invasiveness, this technique is limited in force generation (pN levels), and
increasing laser power for larger force generation has the risk of laser-induced cell damage.
Magnetic nanoparticles, after introduced into cells, can be moved by magnetic forces generated
by controlled magnetic fields. Because cells do not contain magnetic structures, the force is
specifically applied to the internalized magnetic particles. Magnetic tweezers have been used for
manipulating intracellular structures such as chromatin [77] and phagosomes [78]. Although
most magnetic tweezers reported in literature apply forces in one direction only, there are a
growing number of systems that allow 2D and 3D manipulation [79], [80]. One major limitation
12
of magnetic manipulation lies in the positioning accuracy and force resolutions. To solve this
problem, pioneering researchers in Nelson‟s group at ETH Zurich, Sitti‟s group at Carnegie
Mellon, and Kumar‟s group at UPenn have worked to improve the control system for moving
single [82] or multiple [83][84] untethered microrobots to assemble microobjects [85] or deliver
drugs to single cells and tissues [86]. The magnetic microrobots had also been built with
bioinspired flagellar shapes to increase their mobility in a liquid environment at micro/nano scale
[87][88]. Japanese researchers at Nagoya University also combined the magnetic approach with
microfluidic chips to build an on-chip microrobot system [89] for targeting single cell
mechanical characterization [90].
Compared with manual operation and other cell manipulation strategies (e.g., microfluidics,
magnetic tweezers, laser trapping and dielectrophoresis), robotic cell manipulation systems using
mechanical strategies have been demonstrated to overcome their limitations (e.g., cell damage,
cell loss, poor specificity) and be capable of conducting more complex tasks [57]. Direct
manipulation using mechanical strategies, such as microassembly of microobjects,
microinjection of cells, and pick-and-place of single cells, provides a more intuitive and
straightforward solution.
Robotic micromanipulations using mechanical strategy can be further classified into three
categories depending on their distinct feedback modes. The classifications of robotic
manipulation using mechanical strategy are summarized in Table 1. Early studies in robotic
micromanipulation used visual feedback for automation. Because all micromanipulation tasks
are conducted under microscopes (either optical or electron microscope), visual feedback is the
most useful approach for building closed-loop schemes because imaging feedback can provide
plentiful information for tracking cell locations and positioning manipulation end-effectors.
However, microscopy imaging often has a limited depth of field and a small field of view, posing
critical challenges in searching and locating end-effectors both in the image plane and in depth.
Therefore, early research with visual feedback are mainly focused on microassembly of
relatively large objects or microinjection of large cells such as mouse oocytes and zebrafish
embryos.
13
Table 1 - Previous Methods for Robotic Micromanipulations using Mechanical Strategy
Technical
Classification Methods Description Applications References
Force
Feedback
MEMS-
Based
Devices
silicon-based MEMS devices measure
forces at a high resolution of nano-
Newton level; PDMS-based probes or
micro-posts devices can also measure
forces by visually tracking shape
changes; fabrication processes are
complex; devices are easy to damage.
microassembly;
microinjection;
cellular
mechanical
characterization
[91] [92]
[93] [94]
[95] [96]
[97] [98]
[99] [100]
Piezoelectric
Materials
(e.g., PVDF)
force sensing modules usually
integrated on the micropipette holders
or cell holding devices; relatively
easier to fabricate; measurement
resolution ranges from a few to
several hundreds of micro-Newton.
microinjection;
cellular
mechanical
characterization
[101] [102]
[103] [104]
[105] [106]
[107]
AFM
highest force measurement resolution;
limited applications; High equipment
cost; AFM probes can be modified
into various shapes; applications also
performed under electron microscope
cellular or
intracellular
mechanical
characterization;
microinjection
[108] [109]
[110] [111]
[112] [113]
[114] [115]
[116] [117]
Electrical
Feedback
Impedance
Measurement
Electrode inserting into micropipette
to measure the impedance change;
Monitoring impedance change can
detect cell-pipette contact, pipette tip
breakage, and pipette clogging
microinjection;
selectively
isolate cells
from cell
cultures
[118] [119]
[120] [121]
[119]
I-V Curve
Monitoring +
Visual
Feedback
electrode inserting into glass
micropipette was used for monitoring
cell-micropipette contact and
realizing electroporation
microinjection
of sea urchin
eggs with
fluorescent dye
[122]
Planar Patch-
Clamp
System +
AFM
planar patch-clamp system can both
record electrical signals and exchange
intracellular solution; AFM generates
force stimuli with controllable
patterns and magnitudes; AFM
scanning of the single cell provides
detailed morphological information.
electrical
recordings of
whole-cell
current of
potassium ion
channel
[123]
Visual
Feedback
Computer
Vision +
Visual
Servoing
No extra device/equipment cost;
Optical microscopy has limited depth
of field and small field of view;
Z-position information can only be
estimated; Applications limited to
relative large objects or cells (e.g.,
embryos); Image-based visual
feedback requires system calibration.
microassembly;
microinjection;
pick-and-place
of single cells;
[124] [125]
[126] [127]
[128] [129]
[130] [131]
[132] [133]
[134] [135]
14
As MEMS technology advanced during the first decade of the twenty-first century, force
feedback was introduced into robotic micromanipulation by integrating MEMS-based force
sensors into end effectors. The accurate and high-resolution force measurement provided an
improved solution in the determination of relative Z positions between end-effectors and cells.
Additionally, micromanipulation with MEMS-based force sensors also has the inherent
advantages in characterizing the mechanical properties of single cells because the force-
deformation curve can be easily achieved in experiments. Despite their significant advantages,
MEMS-based force sensors did not reach the large-scale adoption in biological labs or clinics
because of the customized force sensors‟ time consuming and complex fabrication processes. As
an alternative solution to MEMS-based force sensors, researchers also tried using piezoelectric
materials (e.g., PVDF) to make cantilever structures on end-effector holders or thin films on cell
culture devices for measuring forces. PVDF sensors are relatively easier to fabricate, but they
have reduced force measurement resolution, often at several tens of micro-Newtons. The third
type of force feedback uses AFM (Atomic Force Microscopy) systems, which have very high
force measurement resolutions. By modifying the AFM probes into special shapes (e.g.,
nanoneedle, nanofork, and nanoknife), Japanese researchers in Fukuda‟s lab have conducted very
interesting studies, such as the quantification of adhesion forces of single cells, single cell
surgery with modified AFM nanofork or nanoknife. However, AFM is expensive and cannot
achieve many other demanding manipulation tasks including microinjection and pick-and-place
of cells, which limits its practicality in typical biological labs or regular clinics.
In addition to visual and force feedback, other researchers also used electrical feedback to assist
in the detection of cell-micropipette contact. In this method, positive electrodes are often inserted
into micropipettes and connected to electrical amplifying circuits and measurement equipment.
The negative or ground electrodes are placed in the cell culture devices. The system monitors the
changes in micropipette tip resistance/impedance to detect cell-micropipette contact, tip breakage
or tip clogging. In addition to monitoring the changes of electrical signals, this method can also
provide extra functions such as electroporation and patch clamping. However, robotic
manipulations using electrical feedback usually require high-specification measurement
equipment because the weak bioelectrical signals easily obscured by ambient electromagnetic
noise. Moreover, the introduction of an electric field can also disturb normal cellular functions
such as gap junctional intercellular communications.
15
Compared to force and electrical feedback, visual feedback works as the most basic approach to
achieve robotic and automated cell manipulations. Robotic manipulation of cells with visual
feedback does not require additional sensing or measurement equipment, making it the most
suitable approach for large-scale adoption in biology labs and clinics from a cost standpoint.
16
1.5 Opportunities for Innovations in Robotic Cell Manipulation
Although many algorithms and techniques have been developed for robotic cell manipulations,
several key issues still exist which limits the practical adoption of robotic cell characterization
and manipulation technologies in biology labs and clinics. These technical issues are:
In order to realize practical adoption in biology labs and clinics, complicated force
sensors or electrical feedback should be avoided and research needs to be focused on
visual feedback approaches.
To achieve robotic manipulation with visual feedback, novel methods are required to
search and detect low-contrast objects (e.g., end-effector tips with various shapes and
fast-swimming motile sperms) in a small field of view under optical microscopy with a
limited depth of field.
To realize visual-servoing robotic manipulation, the relative vertical positions of end-
effectors and cells (Z-position) must be accurately determined and the positioning errors
in the X-Y plane must be well compensated.
Novel robotic manipulation systems must have high throughput capability in order to
achieve statistically significant results for investigating the heterogeneous cell behaviors.
The cell‟s response to external environment (e.g., embryo volume changes due to osmotic
pressure) and behavior changes (e.g., sperm tail beating increases after binding to HA)
must be closely monitored during cell manipulations in order to achieve the optimal
performance for each target cells.
The prerequisite step for all micromanipulations, which is how to locate end-effector tips,
was performed manually. End-effectors, such as micropipettes and microgrippers, must
be first searched and located under microscopy imaging before any micromanipulation
task is performed. There is no robotic algorithm for automatically locating end-effector
tips.
17
1.6 Research Objectives
In this research, I aimed to solve the aforementioned technical issues and use the new solutions
to build robotic systems and automated methods for characterization and manipulation of single
adherent and suspended cells. Specific objectives of this research include:
To develop automated method for automatically locating end-effector tips.
To develop computer vision based contact detection algorithms for determination of
relative Z-positions of end-effector tips and single cells.
To build an automated system for robotic injection of adherent cells to measure gap
junctional intercellular communications.
To achieve a high throughput in microinjection of adherent cells with a high success rate,
a high reproducibility, and a high post-injection survival rate.
To use the robotic microinjection system for screening the efficacy of selected drug
molecules based on the measurement of gap junctional communications.
To develop algorithms for controlled aspiration and positioning suspended cells inside a
micropipette.
To build an automated system for robotic processing of mouse embryos for vitrification.
To develop 3D tracking algorithms for analyzing the morphological changes of oocytes
or embryos during vitrification.
To develop computer vision tracking algorithms for automated analysis of sperm head
and tail locomotion behavior.
18
1.7 Dissertation Outline
An overview of the ensuing chapters is as follows:
Chapter 2 explains the new solution for automatically locating end-effector tips. Micropipettes
and microgrippers were used as example end-effectors to evaluate the performance of proposed
auto-locating method. Guidelines for implementation of this method are provided at the end of
the chapter.
Chapter 3 presents a robotic injection system for characterization of GJIC. Two new vision-
based contact detection algorithms are discussed in this chapter. The performance of the system
is evaluated by injecting three distinct adherent cell lines. Preliminary testing results on
screening the selected drugs are reported in this chapter.
Chapter 4 describes a robotic approach for automatically pick-and-place embryos for
vitrification. The implementation of auto-locating end-effector tips method and vision based
contact detection is discussed in this chapter. New methods for tracking embryos in vitrification
solutions are explained. Test results on system operation speed, post-thawing survival rate, and
development rate are also presented.
Chapter 5 reports automated analysis method for quantifying sperm locomotion behaviors.
Experimental results on free-swimming sperm and HA-bond sperm are compared in this chapter,
confirming the significant potentials to unify the sperm motility analysis with DNA integrity
assessment.
Chapter 6 concludes the entire thesis work with a summary of contributions. Future research
directions are also discussed in this chapter.
19
Chapter 2 Automatically Locating End-effector Tips in Micromanipulation
This chapter discusses an automated solution for automatically locating end-effector tips in
micromanipulation. The performance of the automated function was tested by using various end-
effectors under three major types of microscopy imaging modes. The newly developed auto-
locating functions are proven to be effective for avoiding tip breakage, saving operation time,
and eliminating human involvement. Special guidelines for implementation of this auto-locating
function are provided at the end of this chapter.
The following section is based on the text from the following publication:
J. Liu, Z. Gong, K. Tang, Z. Lu, C. Ru, J. Luo, S. Xie, and Y. Sun, "Locating end-effector tips in
robotic micromanipulation," IEEE Transactions on Robotics, Vol. 30, pp. 125-30, 2014.
20
Automatically Locating End-Effector Tips in 2
Micromanipulation
In robotic micromanipulation, end-effector tips must be first located under microscopy imaging
before manipulation is performed. The tip of micromanipulation tools is typically a few
micrometers in size and highly delicate. In all existing micromanipulation systems, the process
of locating the end-effector tip is conducted by a skilled operator, and the automation of this task
has not been attempted. This chapter presents a technique to automatically locate end-effector
tips. The technique consists of programmed sweeping patterns, motion history image end-
effector detection, active contour to estimate end-effector positions, autofocusing and quad-tree
search to locate an end-effector tip, and, finally, visual servoing to position the tip to the center
of the field of view. Two types of micromanipulation tools (micropipette that represents single-
ended tools and microgripper that represents multi-ended tools) were used in experiments for
testing. Quantitative results are reported in the speed and success rate of the auto-locating
technique, based on over 500 trials. Furthermore, the effect of factors such as imaging mode and
image processing parameter selections was also quantitatively discussed. Guidelines are
provided for the implementation of the technique in order to achieve high efficiency and success
rates.
21
2.1 Introduction
Micromanipulation tools such as micropipettes and microelectromechanical systems (MEMS)
microgrippers are commonly used in the manipulation of micro-scaled objects. Locating the tip
of these end-effectors under microscopy must be conducted before micromanipulation initiates.
In previous manual and robotic micromanipulation, locating the tip of end-effectors (searching,
positioning, and focusing) is a manual procedure performed by skilled operators. Because of the
small size and fragility of micromanipulation end-effectors, manually locating end-effector tips
has high skill requirements, is time-consuming, and can cause end-effector breakage. Despite the
progress made in robotic micromanipulation [111][136][95][137][138], the automation of the
procedure of locating end-effector tips has not been investigated. Most end-effectors used in
micromanipulation have micrometer-sized tips (single or multi-ended). For instance,
micropipette tips used in cell manipulation and microgrippers for assembly tasks are usually a
few micrometers in diameter (see Fig. 2-1). The tiny tip of these end- effectors is difficult to
locate, particularly under high magnifications in microscopy imaging. When the end-effector
collides into other objects (e.g., wafer substrate, glass slide, petri dish, or other end-effectors)
during the process of locating the end-effector and micromanipulation, the tip can be easily
damaged and requires replacement. Hence, automation techniques to readily locate the end-
effector tips are necessary to reduce human intervention and achieve autonomous robotic
micromanipulation.
Figure 2-1 - Example end-effectors used in micromanipulation. Their micrometer-sized tips
must be located first before micromanipulation initiates. (a) Microgrippers for pick-and-place of
small objects. The grasped particles are 10 µm in diameter. (b) Micropipettes for manipulating
biomaterials.
22
Automatically locating end-effectors requires visual detection and focus estimation to determine
the end-effector‟s position in three dimensions. Several techniques have been reported to visually
detect and track micro objects. Ni et al. proposed an iterative closest point algorithm to track a
microgripper‟s position [139]. The algorithm requires the use of an additional dynamic vision
sensor (silicon retina). This special hardware requirement makes the algorithm unsuitable for
micromanipulation tasks that rely on standard vision systems. Microgrippers with complex
features were also tracked using the template matching method for micro-assembly tasks
[140][141]. Our experimental results indicate that the template matching is ineffective to track
objects without distinct features (e.g., micropipettes). In another related work [142], a
generalized Hough transform was applied to detect end-effector tips. However, the approach can
only detect objects with regular shapes (i.e., lines or circular shapes). Algorithms based on one-
class support vector machines [143], shearlet multiscale directional transform [144], and the
Kalman filter [145] have also been used to detect or track objects under microscopy imaging.
However, these algorithms are only suitable to process in-focus images. In the task of locating
end-effector tips, the end-effector often is partially or entirely out of focus. In addition, the
shapes and end-effectors‟ direction of entering the field of view (FOV) also vary with different
micromanipulation tasks. These unique requirements call for the development of techniques to
automatically locate end-effectors under microscopy.
This chapter presents a technique that is capable of searching for out-of-FOV, out-of-focus, and
low-contrast end-effectors. A detection algorithm based on a motion history image (MHI) and an
active con- tour model is used to search for the end-effector. Through estimating the end-effector
tip‟s location and the use of an adaptive quadtree autofocusing algorithm, the tip of the end-
effector is detected and moved to the center of the FOV and brought in focus. This chapter
describes in more detail algorithm comparisons and experimental results. Micropipettes and
MEMS microgrippers are used as example end-effectors to evaluate the performance of the
technique. Experimental results from the over 500 trials under three common imaging modes
(bright field, differential interference contrast or DIC, and phase contrast) demonstrate that the
technique is capable of automatically locating end-effectors under microscopy imaging with high
efficiency and accuracy.
23
2.2 System Design
2.2.1 System Architecture
Figure 2-2 - (a) Schematic illustration of the experimental setup. (b) System picture.
As shown in Fig. 2-2, the micromanipulation system setup consists of a standard inverted
microscope (TE2000-S, Nikon) with motorized focus control and a CMOS camera connected
24
(scA1300-32gm, Basler; resolution: 1200×900). Objectives of 4×, 10×, and 20×are used and
have depths of field of 55.5, 13.5, 5.5µm, respectively. An end-effector (i.e., a micropipette or a
microgripper in this study) is mounted on a motorized 3-DOF micromanipulator (Sutter MP285)
at a tilting angle of 30◦. Movements performed in the automation procedure consist of X, Y, and
Z translation motions of the end-effector as well as adjustment of the microscope‟s focus.
2.2.2 Overall Sequence
The end-effector is initially at or above the focal plane. The search range is established within a
4-mm cubic workspace (−2 mm ≤ x ≤ +2 mm, −2 mm ≤ y ≤ +2 mm, 0 mm ≤ z ≤ +4 mm) [see
Fig. 2-2(a)]. When setting up an end-effector in micromanipulation, it is feasible for the operator
to readily position the end-effector tip to within this 4-mm cube workspace with unaided eyes
and reasonable care. The automated procedure has two main steps: end-effector detection and
autofocus adjustment. Fig. 2-3 summarizes the overall sequence.
Figure 2-3 - Overall sequence for auto-locating end-effector tips contains two major steps.
(1) End-Effector Detection: In the detection step, the end-effector is swept in the X–Y plane. An
algorithm based on an MHI is used to detect the presence of end-effector in the FOV. In some
cases, the end-effector cannot be detected when it is far away from the focal plane. When this
occurs, the focal plane is moved upwards, and horizontal X–Y sweep is repeated. The step size
of focus adjustment depends on the depth of field of the microscope objectives. In our system,
the step size for 4×, 10× and 20× objectives is set as 50, 10, and 5 µm, respectively. When a
moving object is detected in multiple continuous frames of images during the sweeping of the
end-effector, the end-effector is considered to be present in the FOV.
25
(2) Autofocus Adjustment: In the autofocus step, coarse and fine focus adjustments are
conducted to focus on the end-effector tip. Coarse focus adjustment moves the focal plane in a
large step size until the entire image produces a maximum focus measure value. In fine focus
adjustment, a recursive quadtree autofocusing method is used to accurately focus on the end-
effector tip. After the coarse and fine focus adjustments, the in-focus end-effector tip is moved to
the center of FOV through closed-loop visual servo control. This centering step is designed to
reduce search time when switching to a higher magnification objective.
2.3 Key Methods for Auto-Locating End-effector Tips
2.3.1 End-Effector Sweep Pattern Design
When an end-effector is initially mounted on the micromanipulator, the end-effector is often not
within the FOV. However, the direction from which it will enter the FOV is usually known.
Hence, our system moves the end-effector along this direction, and then, sweeps it
perpendicularly [see Fig. 2-4(a)]. If the body of the end-effector instead of the tip enters the
FOV, the system retracts the end-effector until the tip is found with the algorithm discussed in
Section III-C. If the direction of entering the FOV is not known, the system moves the end-
effector to the bottom left of the plane and sweeps it in a zigzag pattern [see Fig. 2-4(b)].
Figure 2-4 - Sweep pattern in X–Y plane (top-down view). (a) When the direction of
entering the FOV is known, end-effector is moved along the direction from which it enters
the FOV, followed by sweeping perpendicularly. (b) When the direction of entering the
FOV is not known, end-effector is swept in a zigzag pattern.
26
2.3.2 Computer Vision Based End-Effector Detection
When the end-effector is swept in the X–Y plane, the system uses a method based on an MHI to
detect whether the end-effector has entered the FOV. The MHI-based detection is a view-based
temporal method which is robust in representing movements and is employed in a variety of
motion detection applications [146]. When the end-effector is out of focus, it appears blurry and
the motion is not obvious to detect. Since the MHI-based method is able to enhance the motion
representation by accumulating the end-effector‟s movement for a period of time, it is suitable to
use to detect the subtle motion of the end-effector. The original image is first denoised by using a
Gaussian filter [147].
When the end-effector moves in the FOV, a silhouette image is obtained by subtracting two
consecutive frames. The subtraction of two frames effectively suppresses the static noises from
the background. The subtracted silhouette image is then binarized by applying a threshold to
suppress background noise. When the end-effector moves, new silhouettes are captured and
overlaid to the old silhouette that fades over time. The sequentially fading silhouettes record the
motion history of the end-effector. Using this method, the MHI of the end-effector in the FOV is
obtained [see Fig. 2-5(b) and (e)]. The sum of the pixel value in the MHI is used as a measure to
determine the presence of the end-effector. If the sum of the pixel value is above a threshold (δ),
the end-effector is considered to be present in the FOV. The threshold, δ, is calculated
dynamically by analyzing 30 frames of the MHIs in which no end-effector is present.
( 1 )
where µ is the average of pixel value sum of the MHIs in which no end-effector is present, σ is
the standard deviation, and a is a preset parameter which determines the threshold value. By
adding several times of standards deviations (i.e., a · σ) to the mean value, the threshold is able
to reject most of the MHIs that do not have the presence of the end-effector. Accordingly, false
positive detection caused by random noise and the shadow of end-effectors is reduced.
The movement direction of the end-effector, if not known, can be derived by computing the
gradients of the MHI image. The gradients are obtained by applying a 3×3 Sobel gradient filter
to the MHI image. If the detected overall motion gradient correlates with the sweep pattern, the
end-effector is confirmed to be present in the FOV. The system then proceeds to the next step
27
(contour detection). Otherwise, motion detection is regarded as a false positive case, and the
system returns to the sweep step until the end-effector is correctly detected.
2.3.3 Detection of End-Effector Contours
Once the end-effector is detected to be present in the FOV, the end-effector‟s root and tip
locations are estimated [see Fig. 2-5(c)&(f)]. In order to obtain a high estimation accuracy, the
background is first removed by subtracting a background image from the current frame. The
background image is recorded at the time when no end-effector is present in the FOV. The end-
effector‟s contour is then detected by using the active contour method [148].
An active contour is a continuous spline defined by ,where x and y are
image coordinates and s ∈ [0, 1]. The active contour deforms in the spatial domain of an image
to minimize
∫ ( )
( 2 )
where is the internal energy composed of the first and second derivatives of v(s), and is
the external energy
( ) ( ) ( 3 )
where and are gradient and convolution operators, respectively, is a Gaussian filter
with standard deviation σ, and is the image data. The active contour model is initialized
by detecting a rough contour of the last binarized silhouette image overlaid on the MHI. Then,
the initial contour was deformed in the background-removed image under the influence of the
internal and external forces.
With the detected contour of the end-effector [see Fig. 2-5(c)&(f)], the average position of the
contour points on image boundaries is taken as the position of the root of the end-effector. The
point with the largest Euclidean distance to the root location is considered the end- effector‟s tip
position. For end-effectors with multi-ended tips (e.g., microgrippers), tip locations are detected
separately.
28
Figure 2-5 - Detection of end-effector. Arrows in (b) and (e) show the motion gradient of
the moving end-effectors. (a) Original image of a micropipette. (b) Corresponding MHI of
the moving micropipette. (c) Tip detection result based on active contour is shown in a
background-removed image. (d) Original image of microgripper. (e) Corresponding MHI
of the moving microgripper. (f) Tip detection result based on active contour is shown in a
background-removed image.
2.3.4 Autofocusing on End-Effector Tips
In this step, the focal plane is adjusted to focus on the end-effector tip. The normalized variance
method [149] is used to calculate the focus measure. It compensates for the differences in
average image intensity (µ) among different images by normalizing the final output with the
mean intensity [150]. The focus measure, F changes as the system adjusts the focal plane. When
F reaches the global maximum, the end-effector is considered in focus.
29
∑∑
( 4 )
where W and H are the image width and height, respectively, is the pixel intensity at the
point (x, y), and µ is the average image intensity.
With the completion of the aforementioned coarse focusing step, fine focus adjustment updates
the most in-focus region, Rf until the region contains the end-effector tip. Rf is a region with a
predefined size m×n satisfying that the focus measure F of the region Rf is larger than any other
region of the same size in the image.
The detection of Rf is performed by using a quad-tree recursive algorithm. The method
recursively divides a region into four subregions with equal area. Out of the four subregions, the
one with the highest focus measure F is further partitioned into four subregions. This procedure
repeats recursively until the area of the subregion is equal or less than the area m×n of Rf. If the
most in-focus region Rf is found to contain the end-effector body instead of the tip of the end-
effector [see Fig. 2-6(a)], the focal plane is moved downwards since the end-effectors are always
mounted at a tilting angle with tip side down in micromanipulation. The recursive detection of Rf
and focal plane adjustment are then repeated until the end-effector tip is brought in focus [see
Fig. 2-6(b)].
Figure 2-6 - Most in-focus region of the end-effector is detected by using a quad-tree
recursive searching method. (a) The most in-focus region is detected to be on the end-
effector body (not on the tip). (b) The tip is brought to focus (i.e., the most in-focus region is
now on the tip).
30
2.3.5 Centering End-Effector Tips via Visual Servo Control
After the end-effector tip is brought in focus under a low- magnification objective, centering the
tips is performed before the microscope switches to a higher magnification. By centering the tips
under a low magnification, the end-effector is more likely to be present in the FOV when
switching to a higher magnification. Experimentally, this step can significantly reduce the X–Y
sweeping time required to locate the end-effector under higher magnifications. In the system, the
in-focus end-effector tip is visually tracked and centered via closed- loop visual servo control.
Visual tracking algorithms can be classified into point tracking, kernel tracking, and silhouette
tracking [151]. Representative algorithms were implemented in this study, and their performance
was compared. In point tracking, the center point of the most in-focus region is detected and
taken as the initial end-effector tip position. Since the end-effector was moved at a constant
speed, the motion of end-effector tips was modeled as a constant velocity system. A Kalman
filter is then applied to predict and optimize the end-effector tip position. In the kernel tracking
category, template matching was tested to track end-effector tips. A template is manually
captured, and the normalized cross-correlation [152] of the template with real-time images is
calculated as a measure to detect the end-effector‟s tip. Silhouette tracking is typically based on
contour evolution and shape recognition. The active contour tracking approach was chosen and
tested to detect the end-effector tip within a region of interest. For an end-effector with multi-
ended tips, an average position of all tips was used as the overall tip position. Using the visually
tracked tip position as feedback, an image-based PID visual servo controller was used to position
the end-effector tip to the center of the FOV.
31
2.4 Experimental Results and Discussion
In the experiments, a micropipette (1 µm in tip diameter) and a microgripper were used as
example end-effectors. The end-effectors were mounted along the X-axis with a tilting angle of
30◦ to the left side of the FOV. Before the automated procedure started, the end- effector tip was
readily placed within the 4-mm cube workspace [see Fig. 2-2(a)] with unaided eyes, above the
focal plane.
2.4.1 Overall Performance
Figure 2-7 - Time of manually locating end-effector tip; auto-locating end-effector tip with
unknown and known entering direction. Within each group (distance to initial focus plane),
n = 20 (ten trials to locate micropipette tip and ten trials to locate microgripper tips).
The goal is to bring the end-effector tip to the center of the FOV under 4× magnification first,
then under 10× magnification, and, finally, under 20× magnification. The experiments were
categorized into five groups according to the end-effector‟s initial distance (D) to the bottom of
the workspace [see Fig. 2-2(a)].
The experiments were first conducted using bright field imaging under two conditions: with
known and unknown direction of the end- effector to enter the FOV. Fig. 2-7 summarizes the
32
overall time spent on locating the end-effector and bringing its tip to the center of the FOV under
20× magnification. The average time of all five groups with unknown entering direction was
44.7 s, while the average time with known entering direction was only 31.2 s. These results
demonstrate quantitatively that the overall locating time is significantly shorter (p< 0.001) by
using the information of end-effector‟s entering direction.
The task of locating end-effector tips in the five groups was also conducted manually by three
skilled micromanipulation operators. The results shown in Fig. 2-7 demonstrate that the overall
time to auto-locate end-effector tip, with or without known entering direction, is significantly
shorter (p< 0.001) than in manual operation (averagely 64.8 s). In particular, in many
micromanipulation tasks, end-effector‟s entering direction is known. With known entering
direction, the time required to auto-locate the tip is shorter by over 50% compared with manual
operation, throughout the five groups. In all groups of experiments, the deviations of time were
caused by D value differences and the differences in initial lateral distance of the tip to the FOV.
We also experimented using only 4× and 20× magnifications in the locating experiments. After
locating the end-effector tips under 4× magnification, the system directly switched to 20×
(instead of 10×). The average time of 100 trials (under bright field imaging) using only 4×and
20×magnificationswas 44.8 s (with known entering direction), which is much longer than the
average time (31.2 s) of auto-locating the tip by using 4×, 10×, and 20× magnifications
sequentially. This is because the end-effector tip was sometimes out of the FOV when directly
switching from 4× to 20×.When the tip was out of the FOV, the locating time was significantly
lengthened because additional lateral sweeping and focus adjustments were needed under
20×.Experimental results also showed that the locating time under higher magnifications (i.e.,
10×, 20×) is short (<5 s). This is because, after locating under the magnification of 4×, the end-
effector tips are very close to the center of FOV. Accordingly, the searching time for end-
effector tips is greatly reduced under higher magnifications.
2.4.2 Performance Under Different Microscopy Imaging Modes
The end-effector auto-locating technique was then tested under three different imaging modes
(100 trials for each imaging condition): bright field, DIC, and phase contrast. The experimental
results are summarized in Table 2.
33
Table 2 - Experiment Results of Auto-Locating End-Effector Tips Under Three Different
Microscopy Imaging Modes
Imaging mode Overall
success rate
Success rate
of detection
step
Success rate
of autofocus
step
Average tracking error (µm)
Kalman
filter
Template
matching
Active
contour
Bright field 97% 99% 98% 2.12 1.91 0.98
DIC 92% 94% 98% 2.01 1.87 1.34
Phase contrast 87% 98% 89% 2.53 1.82 3.07
Overall success rate refers to the rate of achieving the overall objective (locating and bringing
the tip to the FOV of 20× magnification). The overall success rate under bright field imaging is
higher than under DIC and phase contrast (97% versus92%and 87%). This is because the depth
of field of DIC imaging (2.6 µm for 20×) is smaller compared with bright field imaging
(5.5µmfor 20×), and a halo is present around the end-effector tip under phase contrast [see Fig.
2-8(c)]. The smaller depth of field in DIC caused the end-effector not always to be detected
between two focus levels. The halo in phase contrast caused the system to sometimes fail to
correctly focus on the end-effector tip. Thus, the success rate in the autofocus step under phase
contrast was significantly lower than the other two imaging modes (89% versus 98%).
The tracking algorithms based on the Kalman filter, template matching, and active contour were
evaluated under the three imaging modes by calculating the average tip tracking error across 200
frames during tip centering. The tip tracking error summarized in Table 2 is the Euclidean
distance between the detected tip location and the actual tip position [see Fig. 2-8(c)]. The
average tip tracking errors for Kalman filter tracking and active contour tracking under phase
contrast are the largest because of the halo around the tip which significantly influenced the
focus assessment and reduced the contour detection accuracy. In contrast, there is no significant
performance difference among the three imaging modes for the template matching algorithm.
34
Figure 2-8 - Active contour detection under three different microscopy imaging modes. (a)
bright field imaging, (b) DIC, and (c) phase contrast.
The experimental results also show that the active contour method outperforms Kalman filter
tracking and template matching in terms of average tracking error for brightfield and DIC
imaging. Optimal results for template matching are only possible when there are no significant
changes in brightness and shadowing. Changes in scaling and rotation also can lead to false
detection and loss of the tip position. Additionally, users may change end-effectors during a
micromanipulation task. The slightly different shapes/dimensions across end-effectors of the
same type can render template matching ineffective. Regarding Kalman filter tracking, it heavily
relies on the result from the focus assessment step. However, the center point of the most in-
focus region is not always the same as the end-effector tip location. In summary, the results
demonstrate that active contour tracking under the bright field imaging mode was the most
effective to locate end-effectors under microscopy.
35
2.4.3 Discussion
Table 3 - Effect of Different Threshold Values on Detection Time and Overall Success Rate
(Each Data Point in the Table Was from 100 Trials, All Under Bight Field Imaging)
Threshold value (δ)
MHI detection time (sec) 17.4 23.2 30.6 42.7
Overall success rate 87% 95% 98% 99%
In the experiments for locating micropipettes under bright field imaging, 3 out of 100 trials
failed. The three failure cases were caused by improper selections of the threshold value [δ in Eq.
(1)] in MHI detection. To investigate the effect of threshold value selection on MHI detection,
four groups of experiments were conducted with different threshold values. Table 3 summarizes
the results from using different δ values.
When δ is set too low, the shadow of the micropipette body is detected by the MHI algorithm
much earlier than the body itself. When this occurs, the active contour algorithm forms a contour
around the shadow, and then, the recursive autofocusing algorithm locates a most-in-focus
location on the shadow after the coarse focus adjustment. When the system lowers the focal
plane in order to focus on the micropipette tip, the most-in-focus location on the shadow stays
unchanged (because the shadow and the micropipette body are too far apart). In some cases, with
too low a δ value set, the system was not able to locate the micropipette tip even after reaching
the motion limit of microscope focus.
On the other hand, when the threshold value, δ is set too high, it takes much longer for MHI
detection to complete although the overall success rate in locating the end-effector tip is higher
[see Table 3]. A balance of MHI detection time and the overall success rate must be considered
in practice.
In the microgripper experiments under bright field imaging, 5 out of 100 trials failed. Two
failure cases were also caused by improper selection of the threshold for MHI detection. Other
failure cases occurred in the group of experiments for locating microgripper tips assuming
unknown entering direction, where other structures (rather than the gripping tips) appeared in the
36
FOV first, as shown in Fig. 2-9. Hence, at the completion of the auto-locating process, the first
present structure was considered the microgripper tip and brought to the center of the FOV. This
problem can be avoided in practice since the microgripper‟s entering direction is typically known
in a given micromanipulation system configuration.
Figure 2-9 - Auto-locating microgripper tips failed due to other structure appeared in the
FOV first.
Based on the experimental results and observations, we attempt to establish a set of guidelines to
implement the technique to auto-locate an end-effector in micromanipulation. First, the threshold
value in MHI detection must be properly selected, with a balance of failure rates and time taken.
Second, the threshold value selection is also sensitive to lighting. Histogram analysis in our
experiments confirmed that an average value of pixel intensity, between 80 and 150, in the initial
back- ground image should be used to mitigate the effect of lighting variations on MHI detection.
This can be readily implemented in a control pro- gram to properly set lighting intensity in a
micromanipulation system. Third, the task of auto-locating end-effector tips should be conducted
under bright field imaging (versus DIC or phase contrast). Fourth, the auto-locating sequence
should be performed under gradually increasing magnifications (e.g., 4×, 10×, and 20×) rather
than under abrupt magnification changes (e.g., 4×, and then, immediately 20×).
37
2.5 Conclusion and Guidelines for Implementation
This chapter presented a technique to automatically locate an end-effector under microscopy
imaging. Locating the end-effector tip is the very first step in either manual or automated
micromanipulation. The technique described in this chapter is capable of searching for out-of-
FOV, out-of-focus, and low contrast end-effectors. Experimental results demonstrate that the
search time can be reduced by over 50% with the automated technique. The overall success rate
of the locating system is 97% under the bright field imaging mode. Based on the experimental
results and observations, the following guidelines are established to implement the technique of
auto-locating end-effector tips in order to achieve a high efficiency and success rate.
1. The threshold value in MHI detection must be properly set ([µ + 3σ, µ +6σ]).
2. Lighting intensity must be properly set, for instance, with the average value of pixel
intensity in the initial background image set between 80 and 150.
3. Bright field imaging should be chosen (versus DIC and phase contrast) in the auto-
locating process.
4. The active contour algorithm is most effective under bright field imaging to detect and
track end-effector tips.
5. Abrupt magnification change should be avoided when switching from a low to a high
magnification.
6. The prior knowledge of end-effector entering direction improves both completion time
and the overall success rate.
38
Chapter 3 Robotic Microinjection of Adherent Cells
Compared to robotic injection of suspended cells (e.g., embryos and oocytes), fewer attempts
were made to automate the injection of adherent cells (e.g., cancer cells and cardiomyocytes) due
to their smaller size, highly irregular morphology, small thickness (a few micrometers thick), and
large variations in thickness across cells. This chapter presents a robotic system for automated
microinjection of adherent cells. The system is embedded with several new capabilities:
automatically locating micropipette tips; robustly detecting the contact of micropipette tip with
cell culturing surface and directly with cell membrane; and precisely compensating for
accumulative positioning errors. These new capabilities make it practical to perform adherent
cell microinjection truly via computer mouse clicking in front of a computer monitor, on
hundreds and thousands of cells per experiment (vs. a few to tens of cells as state-of-the-art).
System operation speed, success rate, and cell viability rate were quantitatively evaluated based
on robotic microinjection of over 4,000 cells. This chapter also reports the use of the new robotic
system to perform cell-cell communication studies using large sample sizes. The gap junction
function in a cardiac muscle cell line (HL-1 cells), for the first time, was quantified with the
system.
The following section is based on the text from the following publications:
J. Liu, V. Siragam, Z. Gong, J. Chen, M.D. Fridman, C. Leung, Z. Lu, C.H. Ru, S.R. Xie, J.
Luo, R. Hamilton, and Y. Sun, "Robotic adherent cell injection (RACI) for characterizing cell-
cell communication," IEEE Transactions on Biomedical Engineering, vol. 62, no. 1, pp. 119-24,
2015.
J. Liu, V. Siragam, J. Chen, M.D. Fridman, R.M. Hamilton, and Y. Sun, "High-throughput
measurement of gap junctional intercellular communication," American Journal of Physiology -
Heart and Circulatory Physiology (AJP-Heart), Vol. 306, pp. H1708-13, 2014.
39
Robotic Microinjection of Adherent Cells 3
3.1 Introduction
Intercellular communication is a critical part of cellular activities and coordinates cell functions
[153]. Disorders of intercellular communication are responsible for diseases such as cancer
[154], autoimmunity [155], and diabetes [156]. A standard technique for measuring intercellular
communication is based on monitoring transfer of fluorescent molecules from an individual cell
to adjacent cells through functional gap junctions (see Fig. 3-1). The quantitative measurement of
dye transfer requires injection of fluorescent molecules into single cells. In present gap junction
testing experiments, only a few or tens of cells can be injected due to the limitations of manual
operation, posing a practical hurdle in attaining statistically significant data, for instance, for
testing drug molecules on the alteration of gap junction function.
Figure 3-1 - Adherent cells (HL-1) under 20x magnification objective. The inset shows the
schematic of a molecule passing through a gap junction from one cell to its adjacent cell.
Manually manipulating single cells is tedious and time-consuming and has high skill
requirements. In manual operation, a skilled operator looks through the eyepieces of a
40
microscope while dexterously controlling multiple devices (e.g., micromanipulators, pump,
microscope stage etc.). Robotic cell manipulation technologies progressed significantly over the
past decade. The vast majority of demonstrated robotic systems focused on the manipulation of
suspended cells (i.e., oocytes and embryos) [157]–[162]. Fewer attempts were made to automate
the injection of adherent cells.
Most mammalian cells (e.g., cancer cells and cardiomyocytes) adhere to a culturing surface.
Different from large suspend cells (e.g., mouse oocytes ∼100 µm; zebrafish embryos ∼1 mm),
adherent cells are smaller in size and highly irregular in morphology (vs. spherical for
oocytes/embryos), making robust pattern recognition difficult and automation challenging [163].
Additionally, adherent cells are only a few micrometers thick and vary significantly in thickness,
posing more stringent requirements in robotic positioning.
A few joystick-based systems were demonstrated to assist operators for adherent cell
microinjection [164], [120]. Long training, low success rates, and poor reproducibility make
these systems incapable of injecting more than tens of cells per experiment. For example, the
most popular commercial system for adherent cell injection is Eppendorf‟s InjectMan system that
requires an operator to look into the eyepieces of a microscope and perform microinjection by
controlling joysticks. The injection success rate and cell survival rate of the system are 49.2%
and 49.72%, respectively [164]. The only other joystick-based adherent cell injection system was
demonstrated for the injection of fluorescent molecules into adherent cell lines with a 49%
injection success rate [120]. Due to the difficulty of operating joysticks for fine positioning
control, this system does not permit the injection of a high number of cells (only 82 cells
injected). An automated system integrating a cell detection algorithm reported in [127] produced
a success rate of 67% at a speed of 7-8 cells/minute.
In our previous work in the robotic injection of adherent cells [165], we attempted to transform
joystick-based operation to computer mouse clicking. However, several practical limitations in
the system required the operator to manually perform a few key steps using joysticks and looking
into the eyepieces of the microscope. This hybrid way of system operation (mouse clicking in
front of a computer monitor, and operating joysticks under a microscope) precluded the system‟s
potential for routine use in biology labs. Firstly, the system lacked the critical capability for
automatically locating a micropipette tip. In adherent cell injection, the size of micropipette tip
41
must be kept within a few hundreds of nanometers to ensure a high cell viability. Locating the tip
under optical microscopy requires high skills and extreme care since tip breakage can easily
occur. Additionally, the small tip often gets clogged by large molecules or cell debris,
necessitating regular replacement of micropipette tips (e.g., every 100 cells), demanding the
system to possess the capability for locating micropipette tip automatically.
The second limitation in our previous system was the limited contact detection capability. In
experiments, adherent cells are often cultured on gels or protein-coated (e.g., collagen) surfaces.
Our previously reported contact detection algorithm works well on a bare culturing surface but
has a low success rate on these coated surfaces. Furthermore, the previous contact detection
algorithm is not applicable when cell confluency (i.e., density) is high (>80%). Thirdly, the
previous system was not able to compensate for accumulated positioning errors, which made the
system unable to inject more than 100 cells.
In this work, I present a new robotic system capable of injecting thousands of adherent cells. The
system successfully addresses the limitations of our previous system prototype [15] and has the
following new capabilities: automatically locating micropipette tips; robustly detecting the
contact of micropipette tip with cell culturing surface and directly with cell membrane; and
precisely compensating for accumulative positioning errors. These new capabilities, for the first
time, make it practical to perform adherent cell microinjection truly via computer mouse clicking
in front of a computer monitor, making the robotic system suitable for routine use in biology
labs. I also demonstrate the use of the new robotic system to perform cell-cell communication
studies with large sample sizes (over 1,000 cells vs. a few to tens of cells as state-of-the-art).
Finally, the gap junction function in a cardiac muscle cell line (HL-1 cells), for the first time, was
quantified with the new robotic system.
42
3.2 System Setup
Figure 3-2 - System setup. (a) Robotic adherent cell injection (RACI) system. (b) A
schematic showing of RACI system. (c) System control architecture.
43
This system uses well-calibrated air pressure to deliver aqueous solution into either the
cytoplasm or nucleus of a cell. As shown in Fig. 3-2 (a) and (b), the system consists of a standard
inverted microscope (Nikon TE2000-S, Nikon Microscopes) and a motorized X-Y translational
stage (ProScan, Prior Scientific, Inc.). The X-Y stage has a travel range of 75 mm along both
axes with a resolution of 0.01 µm, a maximum speed of 5 mm/s, and a repeatability of ±1 µm. A
CMOS camera (acA1300-32gm, Basler) is connect to the microscope for visual feedback. A
glass micropipette, laser pulled to have an outer diameter of 500 nm and an inner diameter of 300
nm, is mounted to a 4 degrees-of-freedom (DOF) DC-driven micromanipulator (MX7600,
Siskiyou, Inc.) that has a travel range of 20 mm and a 0.1 µm positioning resolution along each
axis. A host computer executing computer vision microscopy and motion control algorithms
controls all hardware.
The robotic micromanipulator and X-Y stage are cooperatively controlled for positioning the
micropipette along the XYZ axes and positioning cells in the XY plane, respectively. The overall
control architecture of the system is summarized in Fig. 3-2 (c). Microscopy visual feedback is
used to control the positioning of the micromanipulator and X-Y stage, forming an image-based
visual servo control system.
When a micropipette is mounted on the system, the system detects the micropipette tip‟s position
and automatically moves it to the center of the field of view. The system is capable of detecting
the contact in two modes, either on cell culturing surface or directly on the cell membrane, to
determine the relative vertical distance between the micropipette tip and the cells in the Z
direction. After the injection location (inside cytoplasm or nucleus) on each cell is selected via
computer mouse clicking, the system controls the micropipette to deliver a precise volume into
each cell and completes the injection of all selected cells following the shortest path. The system
then controls the X-Y translational stage to bring cells in the next field of view under microscope
imaging for injection.
44
3.3 Vision Based Contact Detection
3.3.1 Contact Detection on Cell Culture Surface
Figure 3-3 - Contact detection on cell culture surface. (a) Micropipette tip is lowered
towards the cell culture surface while the X-Y stage moves along the X-axis from left to
right. (b) Micropipette tip slides horizontally when it contacts the culture surface. (c)
Schematic showing micropipette is lowered towards cell culture surface. (d) Schematic
showing micropipette tip is sliding along the X-axis.
In order to inject materials into a cell, the relative vertical distance between the micropipette tip
and the cell along the Z-direction must be accurately detected. Without the inclusion of an extra
sensor (e.g., tactile or force sensors), which increases hardware complexity, two modes of
computer vision based contact detection algorithms are developed to accurately determine the
relative heights of the micropipette tip and adherent cells. This section describes the first mode,
in which the system detects the initial contact of micropipette tip on the cell culturing surface that
is protein coated (vs. bare surface).
Empty Region Detection: When cell confluency is lower than 80%, empty areas on the
culturing surface can be detected and used for contact detection. The system divides the field of
45
view into sub-regions, the size of which is 100 pixels×100 pixels. The normalized variance of the
pixel value, F, is calculated for each region to represent its smoothness, according to
∑∑
( 5 )
where W and H are the region width and height, is the pixel intensity at the point ,
and µ is the average region intensity. This algorithm effectively compensates for differences in
average intensity among different regions. The region with the lowest variance value is
considered empty and used for performing contact detection.
Contact Detection via Three-Dimensional Motion: The system moves the detected empty
region in the X-Y plane until it is directly under the micropipette tip [Fig. 3-3(a)&(c)]. The
micropipette tip is then lowered by the micromanipulator along the Z direction. When the
micropipette tip contacts the bare surface of a Petri dish, further vertical movement of the
micropipette tip induces horizontal motion [166]. However, in cell culturing, the Petri dish
surface is usually coated with gel or protein (e.g., collagen). The coating makes it difficult for the
micropipette tip to „slide‟ horizontally (it „stabs‟ directly into the coated layer). Therefore, in this
system, the X-Y stage is also servoed simultaneously to move along the direction in which
micropipette tip enters the field of view, while the micropipette tip is being lowered by the
micromanipulator. The X-Y stage‟s motion enables the reliable generation of the pipette tip‟s
horizontal „sliding‟ motion [Fig. 3-3(b)&(d)], resulting in highly reliable contact detection on
protein-coated cell culture surfaces.
3.3.2 Contact Detection on Cell Membrane
When cell confluency is high, empty regions on the culture surface becomes unavailable; hence,
the system performs contact detection by detecting direct contact of the micropipette tip on the
cell membrane [see Fig. 3-4(a)]. After the operator selects a cell via computer mouse clicking,
the system moves the cell to the micropipette tip position in the X-Y plane and then moves the
micropipette tip downwards along the Z axis. When the micropipette tip contacts the cell
membrane, the cell is deformed and a subtle motion appears around the contact point. To detect
this subtle motion, we developed an algorithm based on motion history images (MHI). The MHI-
46
based method enhances motion representation by accumulating cell deformation over a period of
time.
Figure 3-4 - Detection of contact on the cell surface (a) At high cell confluency, the system
detects direct contact of the pipette tip on the cell surface. (b) ROI image shows cell being
deformed: white dot indicates tip position. (c) Modified ROI removing the motion of
micropipette tip. (d) Motion history image showing the deformation of the cell: red arrows
indicate motion gradients.
A region of interest (ROI) is first obtained around the micropipette tip [Fig. 3-4(b)]. Within the
ROI, the image is
( 6 )
47
where is the static background; is the moving object; is the background
noise; and is time. The moving object is extracted via taking the difference between
consecutive frames.
( 7 )
where is the motion region; is noise; and is the difference image
containing the moving object and background aberration due to motion and noise. Background
aberration can lead to incorrect detection (e.g., motion ambiguity and distortion) and make
detection fail when object motion speed is low.
In the system, the motion region contains motion from both the lowering micropipette
and cell deformation due to micropipette-and-cell contact. Hence,
( 8 )
where is the micropipette lowering motion, and is the motion from cell
deformation. When the micropipette is moved along Z axis during contact detection, the tip‟s
location in the X-Y plane, , is constant, and the lowering motion results from the
defocusing phenomenon. Since a defocusing blurred image is equivalent to an exponential decay
function [167], the micropipette lowering motion can be described as
( ) , where ( ) is the initial micropipette‟s motion before cell contact
occurs. Thus, motion from cell deformation is
( ) ( 9 )
After removing micropipette tip‟s motion, the noise term in the difference image can
be removed by a Gaussian low-pass filter and image binarization. The binary image representing
cell deformation-caused motion is
{
( 10 )
48
The motion history image is
{
( 11 )
where the duration τ determines the temporal extent of the movement, and γ is the decay
parameter. The subtle cell deforming motion is greatly enhanced by adding the motion history
information [see Fig. 3-4(d)].
The sum of pixel values in MHI, , is calculated as a measure to determine the contact of cell
membrane. If is above a threshold value δ, the micropipette tip is considered as contacting
cell top membrane. The threshold δ is calculated dynamically by analyzing MHIs in which the
micropipette tip has not contacted cell surface.
( 12 )
where is the average of MHIs‟ sum of pixel values before cell contact occurs; is the
standard deviation; and a is a preset parameter which is experimentally determined. By adding
to the mean value, the threshold δ effectively rejects those MHIs that do not contain cell
deformation. In addition, the motion gradient of the MHI is also used to avoid false positive
detection. The motion gradient is computed by convolution with Sobel filters to yield spatial
derivatives, and . The orientation of the motion gradient at each pixel is
( 13 )
As the micropipette tip touches the cell membrane, deformation occurs around the contact point.
Accordingly, the motion gradients in MHI converge to the contact point.
49
3.4 Robotic Microinjection
Figure 3-5 - Robotic injection of adherent cells. (a)(b) When the micropipette is inserted
into the cell, material deposition triggered by pressure pulses results in a “shock wave”
motion inside the cell. (c) Fluorescence is present only in the cytoplasm. (d) Fluorescence is
present only in the cell nucleus.
In this new robotic system, an operator performs every step throughout the microinjection
process via computer mouse clicking. The operator readily selects injection locations (i.e., cell
nucleus or cytoplasm) since the cell nucleus and cytoplasm are distinct on a computer monitor.
The operator can select a single cell or multiple cells to inject within the field of view. During
computer mouse clicking, cell templates centered at the mouse clicking positions are extracted.
The templates are used for template matching to provide position feedback. After performing
contact detection, the system moves the micropipette tip close to the first target cell and inserts
the micropipette into the target cell along the tilting axis at the maximal speed. The pressure
source is then triggered to apply a positive pressure to deposit the preloaded material into the
cell. Material deposition results in a “shock wave” motion inside the cell, around the injection
location [see Fig. 3-5(b)]. The volume of injected materials is precisely controlled by the
50
injection pressure magnitude and width. Although target cells are selected in a random order by
the operator, the system injects all cells along the shortest path. The system then moves to the
next field of view to continue the injection of more cells. The locations of injected cells are all
recorded by the system, permitting time-lapse imaging of every injected cell.
Injection volume control. Injection volume is determined by the injection pressure and pressure
“on” time (i.e., pulse width). Injection volume was experimentally measured. The micropipette
preloaded with distilled water was immersed into mineral oil. After 50 times of injection with a
pressure “on” time of 0.4 s, a water bubble was formed at the tip of the micropipette. The
injection volume was then calculated based on the size of the water bubble. Figure 3-6 shows the
calibrated relationship between injection pressure and injection volume.
To keep a high post-injection survival rate, the volume of injected foreign materials must not
exceed 5% of the cell‟s cytoplasmic volume [165], and therefore not exceed 30 femtoliters (fL)
for most adherent cells. Accordingly, the injection pressure was set lower than 3,000 hPa in our
system. In addition to injection pressure, the pump also provides a constant positive pressure (50
hPa) throughout the entire microinjection process. The application of this positive pressure
ensures that there is a constant, gentle flow of solution from the micropipette, preventing
undesired dilution of the solution in the micropipette by the medium inside the Petri dish.
Moreover, the positive pressure also reduces the possibility of micropipette tip clogging by
foreign objects in the surrounding medium.
Figure 3-6 Injection volume calibration results indicate that the injection pressure should
be set lower than 3000 hPa in order to achieve a high cell survival rate.
51
3.5 Experimental Results and Discussion
3.5.1 Performance of Locating Micropipette Tip
To evaluate the system‟s performance of automatically locating micropipette, micropipette tip‟s
initial distance to the culture surface was divided into five groups. The task of locating
micropipette tips in the five groups was also conducted manually by three skilled
micromanipulation operators. Experimental results demonstrate that the overall time for auto-
locating micropipette tip (30.2s in average) is significantly shorter (p-value< 0.001) than in
manual operation (averagely 64.8s). Auto-locating micropipette tip was also tested under three
imaging modes: bright field, phase contrast, and differential interference contrast (DIC).
Although the phase contrast or DIC imaging modes are desired for producing a pseudo-3D view
of cells, it was found in experiments that bright field imaging produced a high success rate
because the micropipette tip appears more uniform and “halo” free under brightfield imaging.
3.5.2 Injection Speed
Table 4 - Robotic Microinjection Speed of Three Different Cell Lines
cell line HeLa HEK293 HL-1 overall
injection number 2281 1068 1116 4465
injection time (min) 97 48 52 197
speed (cells/min) 23.5 22.3 21.5 22.7
Injection speed of the system was evaluated by injecting three different adherent cell lines: HeLa
cells, HEK293 cells, and HL-1 cells. The number of injected cells and time consumed are
summarized in Table 4. The injection time in Table 4 is the total experimental time for all steps
including locating micropipette tip, contact detection, cell selection and injection. The results
from the injection of over 4,000 cells show that the injection speed of the robotic system is
consistent across different adherent cell lines. The average injection speed is 22.7 cells/min,
which enables users to inject over a thousand of adherent cells within one hour.
52
3.5.3 Injection Success Rate and Cell Survival Rate
Table 5 - Injection Success Rate and Cell Survival Rate After Injection
injection location injection number success rate survival rate
into cytoplasm 1245 95.2% 97.2%
into nucleus 1036 97.5% 96.5%
In order to evaluate the injection success rate and cell survival rate, a Dextran-Rhodamine
fluorescent dye was injected into the cytoplasm or nucleus of HeLa cells. The Dextran-
Rhodamine dye was chosen because it is highly water-soluble and membrane-impermeable.
Therefore, only the cells that are successfully injected with Dextran-Rhodamine are able to
reveal red fluorescent color. The system permits material deposition into the cytoplasm [Fig. 3-
5(c)] or into the cell nucleus [Fig. 3-5(d)]. When the deposition target (i.e., either cytoplasm or
nucleus but not both) reveals strong red fluorescent signals, the injection was considered
successful. Experimental results summarized in Table 5 show that the success rates for
cytoplasmic (n=1,245) and nuclear (n=1,036) injection were comparable (95.2% and 97.5%).
The success rate of nucleus injection is slightly higher than the cytoplasm injection. That is
because the cytoplasm of the adherent cells appears flatter, while the nucleus is more like a bump
shape. The micropipette tends to penetrate into the nucleus rather than the cytoplasm.
Cell post-injection viability was evaluated by using a cell viability assay kit
(Viability/Cytotoxicity Kit, Life Technologies). The measured cell survival rate after cytoplasmic
injection was 97.2% and after nuclear injection was 96.5%. The survival rate for nucleus
injection is relatively lower because the volume of the nucleus is much smaller than the
cytoplasm volume. Same amount of injected medium will impose greater disruptive effects on
nucleus than on cytoplasm.
53
3.5.4 Characterization of Cell-Cell Communication
Figure 3-7 - Only one cell is injected in an area, and fluorescence dye is transferred from
the injected cell to adjacent cells through gap junctions.
We used the robotic system to perform dye transfer experiments for measuring gap junctional
intercellular communication (GJIC). In the dye transfer experiments, a membrane-impermeable
HPTS (8-Hydroxypyrene-1,3,6-Trisulfonic Acid) fluorescent dye mixing with Dextran-
Rhodamine dye was injected into a cell. HPTS fluorescent dye was chosen because of its small
molecule size and membrane-impermeable property. The small HPTS molecule (MW=524.37
Da) can be transferred from the injected cell to adjacent cells through only gap junctions. In
contrast, the Dextran-Rhodamine molecule, because of its large molecule size (MW=10,000 Da),
cannot pass through gap junctions. Therefore, only the injected cell reveals a red fluorescent
signal, while the injected cell, as well as a number of adjacent cells, reveal green fluorescent
signal (see Fig. 3-7). The exact number of neighboring cells that can reveal green fluorescent
signal is determined by the capability of the cells‟ gap junctions. Thus, gap junctional
intercellular communication was quantitatively investigated by measuring the dye transferred cell
number (i.e., the number of cells uptaking fluorescent molecules from the injected cell through
54
gap junctions) and dye transferred distance (i.e., the furthest distance that the fluorescent dye can
be transferred).
Gap junctional intercellular communication was investigated on three cell lines (i.e., Hela cells,
HEK293 cells, and HL-1 cells). These cell lines are known to have an absent, moderate, and high
expression of gap junctions, respectively. We also used 18-α glycyrrhetinic acid (18-αGA) to
treat HL-1 cells as a demonstration of quantifying drug effect on regulating gap junction
function. It is known that 18-αGA inhibits the role of GJIC in fibroblast growth [168], myoblast
fusion [169], and trophoblast proliferation [170]. In our experiments, the robotic system was used
to assess GJIC of three experiment groups with different doses (25, 50, or 100 µM) of 18-αGA
treatment. Two control groups were also included. Control group 1 (0 µM in Fig. 3-7(b)) had no
18-αGA. Since the 18-αGA stocking solution was dissolved in DMSO, a second control group
(DMSO in Fig. 3-8(b)) with only DMSO treatment was also examined.
The dye transfer distance was measured to quantitatively characterize the GJIC for the three cell
lines. The histogram of dye transfer distance for the three cell lines shown in Fig. 3-8(a) reveals
that HL-1 cells have higher GJIC than the other two cell lines. HL-1 is a cardiac muscle cell line.
The strong GJIC of HL-1 cells is physiologically necessary for impulse propagation in cardiac
tissue [171]. The impulse signal is passed efficiently through gap junctions, allowing the cardiac
tissue to contract at the same tandem. In contrast, Hela cells, a cancer cell line derived from
cervical cancer cells, express little GJIC, confirming they do not pass signals via gap junctions
for cell coordination. Limited GJIC results in poor inhibition of cancer cell mitosis, causing
tumor formation [172].
The experimental data in Fig. 3-8(b) shows that the number of HL-1 cells that uptook fluorescent
dye from the injected cells decreased significantly with a higher dose of GJIC inhibitor. The
results show that 100 µM of 18-αGA effectively blocked almost all the gap junctions. There is no
significant difference between the two control groups of HL-1 cells (i.e., the first two bars in Fig.
3-7(b)).
55
Figure 3-8 - Results of dye transfer experiments. (a) Histogram of dye transfer distance for
HeLa cells (n = 200), HEK293 cells (n = 200), and HL-1 cells (n = 400). (b) The number of
HL-1 cells uptaking fluorescence dye from the injected cells. Over 200 cells were injected
for each group.
56
3.5.5 Discussion
Microinjection of adherent cells permits the direct insertion of foreign materials (e.g.,
DNA/RNA, fluorescent dyes, quantum dots etc.) into single cells. It is a decades-old technology
that is still widely used in cell biology for testing cell-cell communication, studying intracellular
behavior, and gene transfection [173].
Manual microinjection of adherent cells and existing robotic system prototypes are limited to the
injection of a few to tens of cells per experiment at best. This new robotic system described in
this chapter is the first system capable of performing microinjection on hundreds and thousands
of cells per experiment. The system is embedded with strong automation capabilities in every
step of operation, enabling an operator to perform the entire microinjection process via mouse
clicking in front of a computer monitor. Training of a user with no skills in microinjection takes
10-15 minutes, and after a few hours‟ operation, the user becomes highly proficient at operating
the system to perform adherent cell microinjection with high success rates.
3.6 Conclusion
This chapter described a robotic adherent cell injection system equipped with several key
technologies that were recently developed. These new techniques include automatically locating
micropipette tips, robustly detecting the contact of micropipette tip with cell culture surface and
directly with the cell membrane, and precisely compensating for accumulative positioning errors.
The high degree of automation of the system enables users with no microinjection training to
perform large-scale cell injection with high success rates. System operation speed, success rate,
and cell viability rate were quantitatively evaluated. Dye transfer experiments were conducted on
different cell lines to establish a powerful assay for characterizing cell gap junction functions.
57
Chapter 4 Robotic Vitrification of Mammalian Embryos
This chapter reports on the first robotic system for vitrification of mammalian embryos.
Vitrification is a technique used for preserving oocytes and embryos in clinical in vitro
fertilization (IVF). The procedure involves multiple steps of stringently timed pick-and-place
operation for processing an oocyte/embryo in vitrification media. In IVF clinics, the vitrification
is conducted manually by highly skilled embryologists. Processing one oocyte/embryo takes the
embryologist 15–20 min, depending on the protocols chosen to implement. Due to poor
reproducibility and inconsistency across operators, the success and survival rates also vary
significantly. Through collaboration with IVF clinics, we are in the process of realizing robotic
vitrification (RoboVitri) and ultimately aim to standardize clinical vitrification from manual
operation to fully automated robotic operation. This robotic system is embedded with two
contact detection methods to determine the relative z positions of the vitrification micropipette,
embryo, and vitrification straw. A three-dimensional (3-D) tracking algorithm is developed for
visually served embryo transfer and real-time monitoring of embryo volume changes during
vitrification. The excess medium is automatically removed from around the vitrified embryo on
the vitrification straw to achieve a high cooling rate. Tests on mouse embryos demonstrate that
the system is capable of performing vitrification with a throughput at least three times that of
manual operation and a high survival (88.9%) and development rate (93.8%).
The following section is based on the text from the following publication:
J. Liu, C.Y. Shi, J. Wen, D.G. Pyne, H.J. Liu, C.H. Ru, S.R. Xie, Y. Sun, “Automated Robotic
Vitrification of Embryos,” Robotics & Automation Magazine, vol. 22, pp. 33-40, 2015.
58
Robotic Vitrification of Mammalian Embryos 4
4.1 Cryopreservation of Oocytes and Embryos
Cryopreservation of mammalian reproductive cells is an essential technique in IVF clinics [174].
Oocytes and embryos are routinely frozen and cryopreserved. Patients who undergo therapeutic
procedures (e.g., chemotherapy) that can place their fertility at risk have the option of preserving
their oocytes for use at a later time through IVF techniques. Moreover, fertilized embryos are
often needed for more than one cycle of IVF treatment. The rest of the fertilized embryos are
cryopreserved for future use [175].
The techniques of oocyte/embryo cryopreservation are classified into two categories: 1) slow
freezing and 2) fast freezing (i.e., vitrification) [176]. Vitrification, or fast freezing, is proven to
be the most effective method and was first reported in [177]. Vitrification is superior to slow
freezing [44], because it vitrifies the oocyte/embryo with no ice crystal formation during
freezing, resulting in higher cell survival rates. The addition of cryoprotectants in vitrification
increases embryo viscosity and makes the vitrified embryos syrupy. When the vitrified oocytes/
embryos are placed in liquid nitrogen, the syrupy content inside the cell forms amorphous ice
instead of ice crystals, which minimizes the vital damage to the cell during freezing [178].
Figure 4-1 - The schematic showing manual and RoboVitri approaches. Vitrification
involves multiple steps of cell pick-and-place before freezing in liquid nitrogen.
59
At present, oocyte/embryo vitrification is done manually in IVF clinics globally. An operator
looks through the microscope eyepieces and manipulates oocytes/embryos using a micropipette.
An oocyte/embryo is first picked up and removed from the culture dish and washed with the
equilibrium solution (ES) and a series of vitrification solutions (VSs), as shown in Fig. 4-1.
Within each step, timing control has been proven critical. After the many pick-and-place steps,
the processed oocyte/embryo is placed onto a device called a vitrification straw. The volume of
solution remaining around the oocyte/embryo on the straw must be minimal to ensure a high
cooling rate [179]. The vitrification straw is then plunged into liquid nitrogen for freezing and
long-term cryopreservation. Several commercial the VSs and protocols exist; however, their core
steps are largely the same. All the protocols involve multiple washing steps with the ES and the
VS, placing the vitrified oocytes/embryos on vitrification straws, and freezing the vitrification
straws in liquid nitrogen. Manual oocyte/embryo vitrification is a laborious and demanding task
due to the following reasons.
There is a long learning curve, and intense focus is required for embryologists performing
the manual method.
The anti-freezing solutes [e.g., dimethyl sulfoxide (DMSO)] are toxic to
oocytes/embryos. Therefore, the washing time in the VS is critical but can be difficult to
strictly control by an operator.
Because of their small size (~100 µm), oocytes/embryos can be difficult to detect and
manipulate, especially when the medium surrounding the cells is dynamically changing
(e.g., in viscosity) during micropipette aspiration and dispensing.
The manual process has stringent skill requirements, and the success rate and cell
survival rate can vary significantly across operators.
Over the past few decades, the robotics community has made significant progress in
assisting/standardizing clinical procedures, from the transformative da Vinci surgical system to
robotic systems under intensive development for manipulating single cells [180][181]. To realize
RoboVitri for clinical use, a number of tasks must be tackled with clinically acceptable
reliability, for instance, robust embryo tracking in three dimensions in media of different
viscosities and autonomous transfer of processed embryos from liquid environments to the solid
60
vitrification straw tip. Furthermore, timing control in vitrification is much more stringent and
critical than other cell manipulation tasks (e.g., intracytoplasmic sperm injection [157]) to
achieve high cell development rates after embryo thawing.
Although automated vitrification was attempted using microfluidic approaches [72], [182],
[183], embryo loss is a concern because of difficulties in loading and retrieving embryos onto
and from the microfluidic devices. No attempt has been made to automate vitrification using a
robotic approach.
In this chapter, I present a robotic system prototype for automated vitrification and thawing of
embryos. The system has successfully addressed the major challenges in manual operation with
the following technologies:
(1) automated locating of micropipette and vitrification straws;
(2) contact detection methods to determine the relative z-position of the micropipette tip;
(3) autonomous detection and tracking of embryos in three dimensions;
(4) robotically placing vitrified embryos onto vitrification straws and removing excess medium.
These core techniques make it practical to perform automated RoboVitri of embryos. The
experimental results also demonstrate that the embryo survival and development rates after
vitrification and thawing achieved by the robotic system were higher than those of the manual
group. Automated vitrification would free up embryologists to focus on other tasks in IVF clinics
and provide consistently high success and survival rates. With a higher throughput, automated
vitrification also has the potential to become a standard tool for assessing existing vitrification
protocols and developing new protocols.
61
4.2 System Overview
Figure 4-2 - (a) The RoboVitri system prototype. The inset is the custom-designed carrier
plate. (b) The system control architecture.
As shown in Fig. 4-2 (a), the RoboVitri system is built around a standard upright microscope
(Olympus SZX16, Olympus Canada, Inc.) that is equipped with motorized magnification control
and motorized focusing. Mounted on the microscope is an XY-motorized stage (ProScan, Prior
Scientific, Inc.), which has a travel range of 75 mm and a resolution of 0.01 nm along both axes.
A custom-designed carrier plate [Fig. 4-2(a) inset] is placed on the XY stage to hold an embryo
culture dish, a multi-well plate, and multiple vitrification straws. A three- degree-of-freedom
robot (MP285, Sutter, Inc.) carrying a vitrification micropipette (tip diameter: 150 nm) is used to
62
manipulate embryos. A 25-µL glass syringe (Hamilton Company) is mounted on a linear stage
(eTrack, Newmark System, Inc.) for controlled aspiration and dispensing of embryos into or out
of the vitrification micropipette. A camera (scA1300-32gm, Basler, Inc.) is connected to the
microscope to provide visual feedback. A host computer controls all hardware via our custom-
developed control software.
The robotic micromanipulator and X-Y stage are cooperatively controlled for positioning the
vitrification micropipette along the XYZ axes and positioning embryos in the x-y plane,
respectively. The overall control architecture of the automated system is summarized in Fig. 4-
2(b). The techniques for micropipette tip detection and embryo tracking are described in the
“Contact Detection” and “3-D Embryo Tracking” sections, which provide position feedback to
position the micromanipulator and the XY stage, forming an image-based visual servo control
system.
In robotic vitrification, the system first performs an auto-locating of the end effector, a technique
previously reported in Chapter 2, for the robotic system to automatically detect the micropipette
tip. Two contact detection methods are used by the system to determine the relative z-position of
the micropipette tip to the multi-well plate bottoms and the vitrification straw tip. During the
washing steps, a 3-D tracking algorithm integrated with a Kalman filter is used to track the
embryos. After washing in the ES and the VS, the robotic system automatically transfers the
vitrified embryo out of the liquid environment onto the vitrification straw tip and removes the
excess medium to realize the minimum volume requirement in vitrification. The vitrification
straw carrying a vitrified embryo is placed in liquid nitrogen. Similar to the vitrification process,
the frozen embryos are thawed and the system performance is evaluated by quantifying the post-
thaw cell survival and development rate.
63
4.3 Key Methods for Robotic Vitrification
4.3.1 Contact Detection
Optical microscopy has a limited depth of field, making position detection along the z-axis
difficult. The relative z-distance between the micropipette tip and the embryos must be
accurately determined before cell manipulation can start. Since the embryos are always placed on
the bottom of a multi-well plate or on the surface of the vitrification straw tip, the system
determines the relative z-position of the micropipette tip by detecting its contact with the multi-
well plate substrate and the straw surface.
Figure 4-3 - The detection of contact between the micropipette tip and the multi-well plate
substrate and contact between micropipette tip and vitrification straw surface. (a) and (b)
The schematic of contact detection on the plate substrate and vitrification straw,
respectively. (c) and (d) The system moves the micropipette downward. (e) A further
downward movement after contact induces the micropipette tip’s sliding motion on the
plate substrate surface. (f) A further downward movement after contact deflects the
vitrification straw tip, causing it to become out of focus.
64
During contact detection on the multi-well plate substrate, the system moves the micropipette tip
downward to approach the plate substrate [Fig. 4-3(a) and (c)]. When the micropipette tip
contacts the plate bottom [Fig. 4-3(a) and (e)], further downward movement induces the tip‟s
horizontal sliding motion on the plate substrate, which changes the tip‟s position in the x-y plane
and the imaging plane. The system detects the initial x-y position change and, thus, determines
the initial contact of the vitrification micropipette tip with the plate substrate.
The detection of the micropipette tip contact on the vitrification straw surface is different from
contact detection with the multi-well plate bottom. Vitrification straw tips are cantilevers in
structure and have low stiffness compared with plate substrates. Therefore, instead of sliding on
the soft vitrification straw surface, the micropipette tip‟s further downward motion after initial
contact deflects the soft straw tip [Fig. 4-3(b)]. When the straw is deflected by the micropipette
tip, it becomes out of focus in imaging [Fig. 4-3(f)]. Based on the computed focus measure, the
robotic system detects the contact between the micropipette tip and the straw tip surface. This
contact detection step is critical before the robotic system can place the vitrified embryo onto the
straw tip, which will be discussed in the “Placing Embryo on Vitrification Straw” section.
4.3.2 Embryo Tracking in Three-Dimensional Space
The system, after detecting the micropipette tip positions along the XYZ axes relative to the
embryos, performs embryo pick and place to transfer the embryo from one type of solution to
another (e.g., from the ES to the VS) via controlled micropipette aspiration. Due to the changes
of fluid density and osmolarity, the embryo volume changes in the VS, resulting in variations of
buoyancy. In addition, the fluidic flow from the micropipette dispensing also influences the
embryo‟s positions. Thus, the embryos move dynamically in 3-D space when transferred into a
different VS. To avoid losing the embryo and to enable efficient pick and place for ensuring
stringent time control in each type of cryoprotectants, the robotic system must be able to robustly
detect and track embryos three-dimensionally.
65
Figure 4-4 - Embryo tracking in the VS. Left column: the original images with embryos at
different heights. Middle column: the ROIs containing the tracked embryo target. Right
column: the tracked embryo locations indicated by a red dot in the binarized image. (a) An
embryo is dispensed out of micropipette into the VS solution. (b) An embryo is floating
upward, due to buoyancy, whereas the system performs autofocusing to control focus
positions.
When an embryo is dispensed out of the micropipette, a region of interest (ROI) is extracted at
the micropipette tip [Fig. 4-4(a)]. The ROI is denoised through the Gaussian smoothing method
and binarized by applying Otsu‟s adaptive thresholding. A morphological close operation is then
performed to remove noise and small particles that may be present in the ROI. In the binarized
image, the contour of the detected foreground object is computed. The embryo‟s position in the
image plane is detected by calculating the moment of the contour (see Fig. 4-4, right column).
The ROI is then updated to be centered at the embryo‟s centroid. The area of the embryo contour
is also measured to reflect the embryo‟s volume change. For subsequent frames of images, the
system repeats the execution of the process to track the embryo‟s position in the image plane
(i.e., the x-y plane).
To track the embryo‟s floating motion along the Z-axis, the system performs auto-focusing. The
normalized variance method is used to calculate the focus measure. This method is chosen
because it can effectively compensate for differences in the average image intensity (µ) by
normalizing the final output with the mean intensity. The focus measure F changes as the embryo
66
moves in the z direction inside the VS solution. As the focal plane moves close to the embryo,
the contents in the image increase, causing the focus measure to increase. The system adjusts the
microscope‟s focal plane according to maximizing the focus measure, and the position recorded
by the encoder of the focusing motor on the microscope is taken by the system to be the detected
z positions
∑∑
( 14 )
where W and H are the ROI image width and height, respectively, is the pixel intensity
at point , and is the average pixel intensity in the ROI.
The tracked embryo positions from autofocusing can be inaccurate due to the delay of changing
the focal plane. Therefore, a Kalman filter is applied to correct the detected embryo positions.
The embryo movement along the z direction is mainly caused by the dynamic change in
buoyancy. The dynamics of the embryo‟s floating motion in the VS is
( 15 )
where is the liquid density, is the gravitational acceleration, is the mass of the embryo, x
is the embryo‟s position along the z axis, and is the damping ratio of the liquid. Due to osmotic
stress in the VSs, the embryo volume changes over time [16]. The cell volume change in
the VSs can be modeled as
( 16 )
( 17 )
In this model, Eq. (16) describes the change of cell volume over time as a function of the
hydraulic conductivity , surface area , gas constant , temperature , intracellular
permeating , and non-permeating solute concentration in the osmoles, and the
67
extracellular permeating and non-permeating solute concentration. Equation (17) describes
the change in the intracellular moles of permeating solute ns over time as a function of the
DMSO permeability PD. When substituting into Eq. (17), the system analyzes the
dynamics of the embryo‟s 3-D motion and uses the Kalman filter to optimize the tracked results.
With the system dynamics modeled, the changes of embryo state in the VSs can be described by
choosing the XYZ positions and velocities as the state variables
( 18 )
where A is the state transition matrix, represents the noises affecting the actual state of the
embryo caused by the heterogeneous response of the individual embryos, and is assumed to
have a Gaussian distribution .
The embryo‟s position is calculated according to
( 19 )
where H is the output matrix and is the measurement noise, which is also assumed to have a
Gaussian distribution . is chosen based on the estimate of how accurately the
embryo‟s 3-D positions are detected by image processing.
Based on the dynamic model, a priori estimate of the state is computed
. The error covariance is denoted as . The priori estimate for this covariance at time is
then determined by
( 20 )
With the a priori estimate of the state and the measurement (i.e., the detected embryo
position from image processing), the real state of the embryo in the VS is optimized by
( 21 )
where is the Kalman gain and is given by
68
( 22 )
With the optimized embryo 3-D position , the robotic system controls the micropipette tip to
retrieve the embryo out of the solutions via controlled aspiration [184]. A biological advantage
of the embryo detection algorithm is that it enables the system to determine the optimized
processing time based on measuring the volume change of an embryo (i.e., individualized
timing, which is not possible to achieve in manual operation). In the VS, the embryos shrink in
the beginning due to osmotic pressure. Then they re-expand to equilibrate with the VS. Since the
cryoprotectant in the VS is toxic, the equilibration with the VS should be avoided [185].
Therefore, the robotic system retrieves and transfers the embryo out of the VS once its minimum
volume is reached, according to the 3-D tracking results.
4.3.3 Placing Embryo on Vitrification Straw
After washing the embryos in the ES and the VS under controlled timing, the vitrified embryos
need to be placed on a vitrification straw tip. In this step, the excess medium must be removed
from the vitrified embryo to ensure a high cooling rate in the liquid nitrogen. After detecting
contact of the micropipette tip and the straw surface, the system dispenses the embryo with a
relatively large volume of the VS solution onto the straw. The system then moves the
micropipette tip on the straw surface away from the initial dispensing location to form a thin VS
film [Fig. 4-5(a)]. The robotic system controls the motorized syringe to aspirate the VS until the
volume of the embryo droplet stops changing [Fig. 4-5(b)]. When the medium is aspirated into
the micropipette, a friction force acts on the embryo to keep it in the original place. To achieve
this, the fluid speed from micropipette aspiration must be well controlled.
As shown in Figure 4-5(c), the embryo is also acted on by the drag force generated by
micropipette aspiration flow
( 23 )
where is the fluid density, v is the fluid velocity controlled by the motorized syringe, is the
cross-sectional area, and is the drag coefficient. If the drag force is too large, the embryo can
69
be undesirably moved together with the fluid into the micropipette. To keep the embryo in place
on the straw, the drag force must be smaller than the friction force (i.e., ).
Therefore, the aspiration flow rate should be below a threshold value. It was determined
experimentally that the critical aspiration flow rate v must be controlled to be lower than 240
µm/s. Below this threshold value, the minimum volume can be achieved reliably. After removing
the excess medium from around the embryo, the straw is plunged into liquid nitrogen for
freezing. The vitrification straw is then sealed with a plastic cap and put in a liquid nitrogen tank
for preservation.
Figure 4-5 - A processed embryo placed on the vitrification straw tip. (a) An embryo is
deposited onto the straw by dispensing the VS medium out of the micropipette. (b) The
excess medium is removed by micropipette aspiration under a threshold flow rate to keep
the embryo in place. (c) The schematic showing the embryo dynamics during micropipette
aspiration.
70
4.4 Results and Discussion
In the experiments, mouse embryos were gathered from the Canadian Mouse Mutant Repository
in the Toronto Centre for Phenogenomics (Toronto, Ontario). Embryos were produced by super-
ovulating a female and collected ~1.5–2.5 days after conception, which corresponds to the
embryos being in the with KSOM medium (EMD Millipore, Billerica, United States) in a 35-mm
Petri dish and covered with mineral oil to prevent evaporation.
The VS typically contains anti-freezing agents or cryoprotectants such as DMSO, small
molecular-sized glycols (e.g., ethylene glycol), or sucrose. In our experiments, the VS was made
by diluting DMSO in KSOM medium at 20% concentration. The ES was at half the
concentration of the VS (i.e., 10% DMSO). A multi-well plate (Repro Plate, Kitazato
Corporation) was loaded with the ES and the VS for embryo washing. A standard vitrification
straw (Cryotop, Kitazato Corporation) was used as the physical carrier to freeze embryos in
liquid nitrogen. All vitrification experiments followed the Kitazato protocol by washing embryos
in the ES and the VS for 12 min and 90 s, respectively. The robotic system can be readily
reprogrammed to implement other vitrification protocols.
4.4.1 System Performance
The system throughput was evaluated by processing the mouse embryos at the two-, four-, and
eight-cell stages. The capability of automated pick and place of single embryos enabled the
robotic system to perform vitrification of multiple embryos in an optimally scheduled sequence.
Since embryo equilibration in the ES costs minutes, after the first embryo was dispensed into the
ES for equilibration, the robotic system moved back to the culture dish to pick up the next
embryo and place it into the ES of another bath. Repeating this step to process six embryos in the
ES, the system then retrieved the equilibrated embryos from the ES and washed them in the VS
one by one, again following a prescheduled sequence. As a result, the system was able to process
six embryos within 24 min. In comparison, in the manual implementation of the same
vitrification protocol, it was only possible to process two embryos in the same time period, and
the operator was fully occupied in the process.
71
Table 6 - Embryo Vitrification Experimental Results
Method Success Rate Survival Rate Development Rate
Control Group N/A 100% (15/15) 93.3% (14/15)
Manual Group 83.3% (15/18) 73.3% (11/15) 90.9% (10/11)
Robotic Group 90% (18/20) 88.9% (16/18) 93.8% (15/16)
The success rate was also quantitatively evaluated. An experiment was defined to be successful
when the system successfully processed the embryo within the given period of time for each step
of a vitrification protocol. Manual vitrification experiments were also performed by three
operators. The experimental results showed that the robotically vitrified group had a significantly
higher success rate than the manual group (90% versus 83.3%, shown in Table 6). In manual
vitrification, embryos could easily escape from operators‟ monitoring when they floated upward
in the VS, which was the major cause of failure. Embryo loss was effectively avoided by the
system‟s capability of 3-D embryo tracking in RoboVitri. However, failure in RoboVitri arose
when an embryo floated in the VS solution and happened to drift into blind regions of the multi-
well plate. The Repro Plate used in the experiments has inclined sidewalls that produce a dark
blind region. Multi-well plates with vertical sidewalls can help reduce blind regions and, hence,
system failure.
4.4.2 Embryo Volume Measurement
In RoboVitri, the system is able to track embryos in 3-D space and monitor their volume change
in real time for analyzing each individual embryo‟s response to the VS. Fig. 4-6 shows the
tracked embryo position and the volume change of three different embryos in the VS, measured
by the robotic system. When washed in the VS, the embryos first experience a dehydration stage
in which water molecules are drawn out of the cell, causing the embryo to shrink. When the
embryo reaches its minimum volume, the toxic cryoprotectant solutes (e.g., DMSO) start to
penetrate the cell membrane. The embryo ideally should be transferred out of the VS at its
minimum volume.
72
Figure 4-6 - The 3-D-tracking results: (a) the z position change of a tracked embryo and (b)
the volume change of three embryos in the same VS measured by the robotic system.
73
The experimental results summarized in Fig. 4-6(b) demonstrate that the embryos reach their
minimum volume in the VS at different time points. This suggests heterogeneity in embryo
dehydration timing even in the same VS. All existing vitrification protocols stipulate a fixed
washing time for all embryos because measuring the individual embryo‟s volume change is not
feasible to achieve by human operators. In contrast, the robotic system is capable of measuring
an embryo‟s volume throughout the vitrification process and is able to retrieve each embryo from
the VS at its minimum volume point. Therefore, the toxic effects from the cryoprotectant solute
(i.e., DMSO) can be minimized as the embryo is taken out of the VS when embryo dehydration
has just ended.
4.4.3 Post-Freezing Survival and Development Rates
The survival and development rates were defined to quantify the robotic system‟s performance.
Survivability was measured by examining the morphology of the embryo before and after
freezing, as commonly performed in the literature (see [19]). Embryos were considered
unhealthy/dead if they had an abnormal shape, membrane damage, leakage of cellular content, or
degeneration of their cytoplasm.
Figure 4-7 - Example embryo images before and after vitrification.
The development rate was determined by culturing the surviving embryos for an additional 24 h
after thawing (Figure 4-7). If the cell number within the embryo increased, or if the embryo
developed to the blastocyst stage, it was counted as developed. Control samples of non-vitrified
embryos were also cultured to identify the base development rate of the mouse embryo
74
population. Only embryos that had a healthy morphology after freezing were cultured following
similar procedures to other vitrification studies [20].
The experimental results are summarized in Table 6. The robotically vitrified group had higher
survival (88.9% versus 73.3%) and development rates (93.8% versus 90.9%) than the manually
vitrified group. The higher survival and development rates produced by RoboVitri can be
attributed to the optimized processing time achieved by the system through monitoring embryo
volume changes in the VS. The ability of the system to effectively remove excess medium from
around vitrified embryos on the straw (i.e., minimum volume vitrification for a higher cooling
rate when freezing in liquid nitrogen) could also have enabled the robotic system to achieve
higher embryo survival and development rates.
4.4.4 Discussion
Vitrification is an essential technique in IVF for preserving oocytes and embryos. A number of
different the VSs and protocols (e.g., Kitazato, Origio, and Irvine) have been developed and are
commercially available. All of these protocols require multiple steps of embryo washing in
different types of the VS. With the automated capability, our robotic system can be programmed
to test, optimize, and compare different vitrification protocols.
Besides embryo vitrification, oocyte vitrification is also important in IVF practice for preserving
female fertility for future use. Oocyte vitrification requires more steps than embryo vitrification.
For example, in the Kitazato protocol, oocyte vitrification involves three ES steps and two VS
steps, whereas embryo vitrification only has one ES and VS step. The RoboVitri system can also
be programmed to complete oocyte vitrification by simply repeating more washing steps because
of its capability of automated pick and place and stringent time control.
Robotic cell manipulation relieves the human operator from tedious vitrification steps and
eliminates manual operation-caused errors and inconsistencies. It also offers unparalleled timing
control and the ability to leave a minimal volume of the solution on vitrification straws.
Automation also enables the system to process multiple oocytes/embryos with high efficiency.
Efforts will continue to develop the system into an ideal tool for standardizing oocyte/embryo
vitrification in IVF clinics and to achieve improved cryopreservation outcomes.
75
4.5 Conclusion
This Chapter presented an automated RoboVitri system capable of processing embryos with
different the VS. The system is integrated with two contact detection methods to determine the
relative Z-position of the micropipette tip relative to embryos and vitrification straw tips. A 3-D
embryo-tracking technique was developed to prevent embryo loss during multiple washing steps
and achieve vision-guided embryo transfer. The 3-D embryo tracking technique also enables the
system to monitor real-time embryo volume changes in the VS, permitting individualized time
control for each embryo. In the step of placing an embryo onto the vitrification straw tip, the
excess medium was automatically aspirated away from the vitrified embryo to obtain a high
cooling rate. With these technical capabilities, the robotic system successfully achieved a high
throughput and improved post-freezing survival and development rates compared to manual
vitrification.
76
Chapter 5 Automated Analysis of Sperm Locomotion Behavior
Sperm selection plays a significant role in in vitro fertilization (IVF). Approaches for assessing
sperm quality include noninvasive techniques based on sperm morphology and motility as well
as invasive techniques for checking DNA integrity. In 2006, a new device using hyaluronic acid
(HA)-coated dish for sperm selection was cleared by the Food and Drug Administration (FDA)
and entered IVF clinics. In this technique, only sperms with DNA integrity bind to the HA
droplet, after which these bound sperms stop revealing head motion and their tail movement
becomes more vigorous. However, selecting a single sperm cell from among HA-bound sperms
is ad hoc in IVF clinics. Different from existing sperm tracking algorithms that are largely
limited to tracking sperm head only and are only able to track one sperm at a time, this chapter
presents a multi-sperm tracking algorithm that tracks both sperm heads and low-contrast sperm
tails. The tracking results confirm a significant correlation between sperm head velocity and tail
beating amplitude, demonstrate that sperms bound to HA generally have a higher velocity
(before binding) than those sperms that are not able to bind to HA microdots, and quantitatively
reveal that HA-bound sperms‟ tail beating amplitudes are different among HA-bound sperms.
The following section is based on the text from the following publication:
J. Liu, C. Leung, Z. Lu, and Y. Sun, "Quantitative analysis of locomotive behavior of human
sperm head and tail," IEEE Transactions on Biomedical Engineering, Vol. 60, pp. 390-396,
2013.
77
Automated Analysis of Sperm Locomotion Behavior 5
5.1 Introduction
The mechanisms of natural sperm selection are not well understood. It is accepted, however, that
the sperm selection mechanisms play a significant role in the inheritance of superior health traits
such as disease resistance, offspring survival, and fecundity [186][187]. In natural human
conception, sperm selection occurs as a healthy sperm actively seeks out and fertilizes an egg.
However, for couples having infertility issues, assisted reproduction technologies are required to
address their reproductive needs. For instance, in intracytoplasmic sperm injection(ICSI), an
embryologist selects a single sperm and injects it into an oocyte (i.e., egg cell) to overcome
issues such as male infertility [35]. This IVF procedure bypasses the physiologic and biologic
barriers for sperm selection and demands the operator to select high-quality sperms.
The criteria for sperm assessment provided by the World Health Organization are vitality,
morphology, and motility [46]. In IVF, sperm selection is commonly based on sperms‟ motility
and morphology attributes. A widely used method for sperm selection is motile sperm organelle
morphology examination [188][189]. Sperm motility is also a widely accepted criterion for
sperm quality assessment. The past few decades have witnessed the development of computer-
assisted sperm analysis (CASA) methods for measuring both sperm morphology and motility
[190].
Recently, an emerging methodology was introduced for selecting viable sperms with a high level
of DNA integrity. In order to noninvasively select a healthy sperm, Huszar‟s group proposed the
use of a hyaluronic acid (HA) assay [51]. HA is a linear polysaccharide in the extracellular
matrix of cumulus oophorous around the oocyte and plays an important role in natural human
fertilization [191]. A series of studies on HA-based sperm selection confirmed that the HA assay
is able to select healthy sperms with no DNA damage [52][53], and has received FDA approval.
In the HA assay, sperms that bind to HA microdots are proven to have a higher level of DNA
integrity compared to those unbound sperms. When a sperm binds its head to an HA microdot,
the sperm loses its progressive movement and the tail beating motion becomes more vigorous.
The sperm tail beating amplitude becomes the only parameter to differentiate the HA-bound
sperms from each other, calling for techniques to quantify HA-bound sperms‟ tail beating
motion.
78
Several algorithms have been reported for tracking multiple moving objects. Model-based
tracking algorithms incorporate priori information about the objects to develop representations
such as skin complexion [192], shape [193], kinematic skeleton [194], silhouettes [195], or layer
information [196]. Appearance-based approaches apply recognition algorithms to track objects in
eigenspace [197] derived from observations or in kernel space [198]. The correspondence of
multi-object tracking becomes complex with the presence of occlusions, misdetections, entries,
and exits of objects. Algorithms for solving the problems of nearby confuser and objects
occlusion/overlap include deterministic methods [199], single object state estimation (e.g.,
Kalman filter [200], particle filter [201]), and multiple object data association and state
estimation (e.g., global nearest neighbor method [202] and joint probability data association filter
[203]). Due to color uniformity and shape similarity of sperms, multi-sperm tracking has not
been well studied.
Several algorithms have been developed to track sperm trajectories, measure sperm velocities,
and evaluate sperm energetics [54], [204], [205]. Shi et al. reported a single-sperm tracking
algorithm based on a four-class thresholding method to extract a single sperm in a small region
of interest [55]. The method is limited to tracking a single sperm and is incapable of multi-sperm
tracking. Nafisi et al. demonstrated a template matching algorithm for sperm tracking. The
algorithm is insensitive to image acquisition conditions [56]. However, this algorithm relies on
user input to obtain the sperm‟s initial position and cannot track multiple sperms. Existing
algorithms for sperm tracking are largely limited to sperm head tracking. The small size (≤1 μm
in thickness) and low contrast of sperm tails under optical microscopy make sperm tail tracking
challenging. In our previous study [206], a maximum intensity region algorithm was developed
for sperm tail tracking. The tracking algorithm, without proper filtering, can be susceptible to
disturbances, such as overlapping of the target sperm with other sperms or debris and changes in
lighting conditions.
In this chapter, I report an approach for tracking both sperm head and tail. This approach uses a
motion template method to detect and track multiple moving sperms, and integrate a Kalman
filter to the maximum intensity region algorithm to locate the sperm tail‟s position. With the
positions of the sperm head and tail detected, the sperm‟s velocity and tail beating amplitude
were measured. Experimental results demonstrate that there is a significant correlation between
the sperm velocity and its tail beating amplitude. I also analyzed sperm motility and the tail
79
beating movement on HA coated dishes. I found that sperms with a higher level of motility are
more likely to bind to the HA microdots, and the sperm tail beating amplitude significantly
increases after a sperm binds to the HA microdots. Quantitative analysis of sperm tail‟s beating
amplitude can provide useful information for sperm selection.
80
5.2 System Setup and Experimental Design
Figure 5-1 - Experiment design. (a) and (b) Experiment 1: a sperm changed its position
between two frames. (c) and (d) Experiment 2: a sperm was moving toward an HA
microdot and then bound to the HA microdot.
Human sperm samples were tested under a standard inverted microscope (bright field imaging,
Nikon TE2000-S). A 20× objective with a numerical aperture of 0.45 was used (CFI Plan Fluor
ELWD, Nikon). A CMOS camera (601f, Basler; resolution: 640×480) was connected to the
microscope to capture images at a frame rate of 30 frames/second.
We first investigated the correlation between sperm velocity and sperm tail beating amplitude. In
this experiment, human sperms were placed in a Petri dish containing a standard medium
(SpermCatch, NidaCon International). Mineral oil was used to cover the medium to prevent
evaporation. Fig. 5-1(a) and (b) shows a sperm changed its position between the two image
frames. Sperms that had linearly progressive movement for at least 2 s were taken into
81
consideration. To evaluate the linearity of the sperm motion, sperm‟s average VCL and straight
line velocity were calculated by measuring the sperm‟s head position in each frame.
Additionally, the sperm tail beating amplitude was measured using the sperm tail tracking
algorithm.
The second experiment was designed to investigate the difference of sperm velocity and tail
beating amplitude between HA bound sperms and those unbound ones. Human sperms were
analyzed in a PICSI dish (MidAtlantic diagnostic, Mount Laurel, NJ) with HA microdots coated
at the dish bottom. The microdots were first hydrated by placing 10 μL droplets of human tubal
fluid and were added with 10 μL SpermCatch. Human sperm was then placed on the HA
microdots, which were covered with mineral oil to prevent the culture medium from evaporation.
The experiment was conducted at room temperature. Some sperms were observed to bind to the
microdots after 5 min. Fig. 5-1(c) shows that a sperm was moving toward an HA microdot, and
Fig. 5-1(d) shows the same sperm bound to the HA microdot. When a sperm bound to the HA
microdot, its head motion stopped and its tail beat vigorously. In contrast, sperms that might have
DNA defects swam freely on top of the HA microdots without binding. In this experiment, the
head velocity of HA bound sperms before binding and the head velocity of unbound sperms were
measured and compared. The increase of sperm tail beating amplitude after a sperm binds to an
HA microdot was also measured.
5.3 Computer Vision Methods for Sperm Tracking
5.3.1 Overview
In both experiments described in the previous section, the sperm velocity and tail beating
amplitude are calculated in every frame of the image. The algorithm consists of three steps. The
first step tracks the sperm head and uses its position to calculate the sperm curvilinear velocity
(VCL), straight line velocity (VSL), and linearity of the sperm‟s moving path. In the second step,
the sperm tail region of interest (STROI) is extracted. STROI extraction is an extrapolation
process that calculates the region in which the sperm tail is located by using information from the
first step. The STROI is used to capture the tail tip region of the sperm. Once the STROI is
found, the maximum intensity region (MIR) algorithm is used to locate a point on the sperm tail
within the STROI. Finally, a Kalman filter is used to improve the accuracy of the located point
on the sperm tail.
82
5.3.2 Sperm Head Tracking
Figure 5-2 - Tracking multiple sperm heads. (a) Red dots are tracked sperm heads. (b)
Corresponding motion history image.
There are typically multiple sperms moving randomly within a field of view. In order to detect
the sperm heads simultaneously, a multi-target tracking algorithm was developed. For a specific
single sperm, a silhouette image of this sperm is obtained by subtracting two consecutive frames.
The silhouette image is then binarized by applying a threshold (i.e., 60) to suppress the
background noise. When this sperm moves, new silhouettes are captured and overlaid to the old
silhouette that fades over time. The time duration of every silhouette is set to be 0.5 s. The
sequentially fading silhouettes record the motion history of this sperm. Using this method, a
motion history image (MHI) [146] of all the moving objects in the same field of view is obtained,
as shown in Fig. 5-2(b). The position of each moving object is calculated from its central
moment in the MHI. Among these moving objects, there are some objects with very little motion
caused by those sperms with extremely low motility or by Brownian motions of debris. These
objects are excluded by applying morphological transformations (i.e., erosion and dilation) to the
MHI. Figure 5-2(a) shows the detected moving sperms.
To track multiple sperms, the position history of each sperm is recorded and managed in the
multiple sperm tracking algorithm, as summarized in Table 7. When J sperms are detected at
frame , Pij represents the position of the j-th sperm at frame . When a sperm moves close to the
edge of the image and its moving direction is toward the boundary, it is considered swimming
out of the field of view. The algorithm then terminates the corresponding data column for this
83
sperm. As shown in Table 7, the second column disappears after frame i + 1, which means that
the second sperm has moved out of the field of view. On the other hand, when a sperm newly
moves into the field of view, it is added into the data column. For instance, a new sperm moves
into the field of view at frame i + 3. Correspondingly, a new column is added (see the last
column in Table 7).
Table 7 - Dynamic Representation of Sperm Positions in Multi-Sperm Tracking Algorithm
Frame No.
Sperm Index
1 2 J J+1
i
i+1
i+2
i+3
To track an individual sperm, the direction vector of the sperm is used as a unique identifier to
distinguish it from other sperms with similar color and shape. By denoting the position of the j-th
sperm at frame as , the sperm‟s direction vector is , where ∈
. We found experimentally that the maximum moving distance of fast sperms between two
consecutive frames is approximate 3.86 μm (i.e., 8 pixels). Therefore, if the distance between the
j-th sperm‟s position at frame i + 1 and the o-th sperm‟s position at frame i is less than 8 pixels
(i.e., ‖ ‖ ), these two sperms are recognized as the same sperm (i.e., ). There may be
instances in which two or more sperms at frame i + 1 have a distance less than 8 pixels to the j-th
sperm in frame i. The average movement direction in previous 30 frames, , is then used as a
unique identifier to determine which sperm is the correct j-th sperm
84
∑
( 24 )
The candidate sperm that produces the minimum Euclidean distance value is considered the
same j-th sperm in the previous frame i
∈
‖ ‖ ( 25 )
where M is in frame i + 1 the total number of sperms close to the j-th sperm (i.e., ‖ ‖ ) and
is the distance vector between the candidate sperm and the j-th sperm.
Assume that the j-th sperm enters the field of view at frame i and swims out of the field of view
at frame i + N. With the sperm position detected in each frame, the travel distance of the j-th
sperm between two consecutive frames can be determined from its direction vector . The
VCL, which is the average velocity of the sperm head along its actual curvilinear path, is
∑
( 26 )
The VSL, which is the average velocity of the sperm head along the straight line between its first
and last detected position, is
( 27 )
The linearity (LIN) of the sperm‟s curvilinear path is
( 28 )
where LIN is the linearity measure (0 ≤ LIN ≤ 1). A higher LIN value means that the sperm‟s
moving path is more linear. Healthy energetic sperms with progressive/linear movement are
desired (versus those traveling in circles for instance) in sperm selection. In our experiments,
only those sperms having a LIN value greater than 0.9 were considered for further analysis.
85
Figure 5-3 - Sperm tail tracking. (a) Sperm head position is found. (b) STROI is
determined. (c) 5×5 windows are scanned to locate the section with the highest intensity
sum in the flicker image. The center point (blue dot in the figure) of the section is
considered the tail location. (d) Based on the blue dot position found in the flicker image,
Kalman filter is applied to improve the accuracy of located sperm tail position.
5.3.3 Sperm Tail Tracking
After the sperm head position is detected, the sperm tail tracking algorithm extracts an STROI.
As shown in Fig. 5-3(b), the STROI is determined using the sperm head position and the average
direction vector of its movement. The average direction vector Dj is used instead of the direction
vector because the sperm may exhibit abrupt changes in movement direction between two
consecutive frames. By averaging the direction vectors of the sperm across a number of frames
86
(e.g., 30frames), the effect of abrupt changes in the sperm moving direction between frames is
mitigated and the extraction of STROI becomes more robust.
The STROI‟s center position in the frame Tij is determined by subtracting a scaled value of the
direction vector from the sperm head‟s centroid
‖ ‖ ( 29 )
where a is a scalar value determined by the human sperm length. Under the 20× magnification,
the average length of human sperms is approximately 90 pixels (i.e., a = 90). After the center
position is found, a 25×25 region of interest is taken as the STROI. The size of 25×25 provides a
sufficient tail search area that takes into consideration a range of sperm tail length variations and
sperm tail beating amplitudes.
After finding the STROI, the algorithm verifies that a tail is present in the STROI. The
fundamental feature of flicker is extracted by taking the absolute difference between several
consecutive inverted grayscale image frames. A higher number of frames to form the flickering
image results in more enhancement of the image contrast. On the other hand, too many
consecutive frames would add too much sperm tail‟s history information that can influence the
sperm tail tracking accuracy. Experimentally, we determined that six consecutive frames were
appropriate to use for forming a flickering image
∑
( 30 )
where f(i) is the flickering image extracted at frame i, and I represents the grayscale images
containing the sperm of interest in frame i to frame i − 5. Each pixel in the flickering image is
squared to enhance the pixel values of areas in which the tail is present. The sum of the pixel
value in the STROI of the f(i) image is used as a measure to determine the presence of a sperm
tail. If the pixel sum is above a specified threshold value, a tail is considered present. The
threshold value was found experimentally by comparing the pixel-sum values of STROI images
in which a tail exists against cases where no tail exists. An example flicker image is shown in
Fig. 5-3(c). If the pixel sum is below a threshold value, no tail is found inside the STROI. This
87
situation can occur when the sperm of interest moves out of focus, resulting in the disappearance
of the sperm tail.
Once the sperm tail is determined to exist within the STROI, the MIR algorithm uses the
flickering image to locate a point on the sperm tail. By extracting the flicker feature of the sperm
tail, as shown in Fig. 5-3(c), the position of the sperm tail can be detected. This approach
overcomes the challenges that arise from the low contrast image of the sperm tail in a single
frame. The algorithm first finds the location of maximum intensity within the 25×25 STROI of
the flickering image. This is accomplished by evaluating the sum of the intensity values inside a
5×5 window at a spatial sampling interval of 5 pixels in both the x and y coordinates of the
STROI flicker image. The center position of the 5×5 window with the highest intensity is
considered the tail location (i.e., a point on the sperm tail).
The located point on the sperm tail is often inaccurate because the flickering image contains
noises caused by some dark debris or by other sperms entering the STROI. Therefore, a Kalman
filter is applied to correct the measured point on the sperm tail. In order to model the sperm tail
motion, the sperm tail‟s location and velocity in the image coordinate are chosen as state
variables (i.e., ). After the optimized sperm tail‟s position is found, the sperm
tail beating amplitude inside the STROI is computed. The relative position inside the STROI in
frame i is denoted by PTi. The sperm tail beating amplitude A is
∑‖ ‖
( 31 )
where PT is the sperm tail‟s average position inside the STROI and N is the number of frames
until when the sperm tail is successfully detected.
88
5.4 Experimental Results and Discussion
Figure 5-4 - Failure cases in sperm head tracking. (a) False positive detection of sperm
head in shape-based tracking is caused by the stationary object/particle with a shape
similar to sperm head. (b) False negative case: shape-based detection algorithm fails to
detect the sperm head that has an abnormal head shape. (c) False positive detection of
sperm head in MHI-based detection is caused by moving debris. (d) False negative case:
MHI-based detection algorithm fails to detect stationary/slow-moving sperms.
The multi-sperm head tracking algorithm based on MHI was evaluated and compared with the
multiple-object tracking algorithm which uses sperm head shape and the global nearest neighbor
(GNN) data association method. Both algorithms were applied to the same video clips (476 s in
duration) in which more than 200 sperms were present. The evaluation criteria are average
tracking errors (ATE), false positive (FP) rate, false negative (FN) rate, false identified trackers
(FIT), and false identified objects (FIO), as shown in Fig. 5-4 defined in [207].
Table 8 - Performance Comparison Between MHI-Based Multi-Sperm Tracking and GNN-
Based Multi-Sperm Tracking
Algorithms ATE (µm) FP FN FIT FIO
MHI 0.85 3 15 33 28
Shape & GNN 1.23 7 21 54 48
89
As summarized in Table 8 & 9, the average tracking error of MHI-based algorithm (0.85 pixel) is
less than that of shape-based detection algorithm (1.23 pixel). Shape-based multi-sperm tracking
algorithm also has a higher FP rate than the MHI-based tracking algorithm. The FP cases in
shape-based tracking were mainly caused by stationary foreign objects (contaminant particles)
with a similar shape as the sperm head [see Fig. 5-4(a)]. Since these objects were largely
stationary, they had little effect on MHI-based tracking. The FP cases in MHI-based tracking
were caused by cell debris moving with the fluidic flow when sperms were swimming nearby
[see Fig. 5-4(c)]. The experimental results demonstrate that the shape-based tracking algorithm is
more susceptible to background noise and is not always effective in filtering debris/particles that
have similar shapes as the sperm head. On the other hand, the MHI-based multi-sperm tracking
approach is more robust to background noise and disturbances from foreign stationary particles.
The MHI-based tracking algorithm also outperforms the algorithm using GNN data association
in terms of FIT and FIO. The GNN data association method only considers sperm head‟s position
in the last frame. Thus, it is not effective in dealing with abrupt movement changes between two
consecutive frames. In the MHI-based algorithm, the effect of abrupt changes in sperm
movement between frames is mitigated by averaging sperm‟s movement direction vector across a
number of frames.
It needs to be noted, however, that the MHI-based sperm head tracking algorithm as a higher
value of FN than the shape-based multi-sperm tracking algorithm. This is because the MHI-based
algorithm cannot detect stationary sperms. In IVF, only motile sperms are of interest for analysis.
Therefore, this drawback of MHI-based sperm tracking algorithm does not constrain its
applicability in sperm selection. The FN cases in shaped-based and MHI-based algorithms are
shown in Fig. 5-4(b) and (d), respectively.
Table 9 - Average Tracking Error of Sperm Tail Tracking
Video V1 V2 V3 V4 V5 Average
Duration (sec) 3.83 6.67 3.83 4.60 3.87 4.56
MIR (pixels) 1.16 2.73 1.83 2.02 1.98 1.95
With Kalman (pixels) 0.97 2.53 1.05 1.39 1.25 1.43
90
The sperm tail tracking algorithm was evaluated by measuring the Euclidean distance error
between the detected tail point and the actual sperm tail position. The actual position was
carefully identified/input by a user for each frame via computer mouse clicking. Table 9
summarizes the average Euclidean distance error for five additional videos. The overall average
Euclidean distance error for the MIR algorithm integrated with Kalman filtering is 1.43 pixels
(0.69 μm), while the error for the MIR algorithm without Kalman filtering is 1.95 pixels (0.95
μm). This result indicates that Kalman filtering is effective in improving the sperm tail tracking
accuracies of the MIR algorithm. When the MIR algorithm failed to track the sperm tail due to
the overlap with other sperms or occlusion by debris, Kalman filtering was able to estimate the
sperm tail‟s position based on the motion model.
Figure 5-5 - Correlation between sperm velocity and tail beating amplitude. Pearson’s
correlation coefficient r = 0.7225.
To investigate the correlation between the sperm head velocity and the tail beating amplitude, we
chose 30 sperms showing good progressive movements (LIN > 0.9). The tracking algorithms
described in Section 5.3 were used to measure their head velocity and the tail beating amplitude.
We processed the measured data using a linear regression model. As shown in Fig. 5-5, there is a
statistically significant correlation between the head velocity and sperm tail beating amplitude.
Bivariate association between sperm velocity and sperm tail beating amplitude was evaluated by
91
Pearson‟s correlation coefficient. Tracking data captured on the 30 sperms show that Pearson‟s
correlation coefficient was 0.7225 and the p-value was lower than 0.0001. This result
quantitatively demonstrates that the sperm‟s head velocity is proportional to its tail beating
amplitude.
Figure 5-6 - Sperms on a PICSI dish with HA microdots. (a) Sperms binding after 5 min.
(b) Sperms binding after 30 min.
In the HA binding experiment, the heads of the HA-bound sperms became stationary, and their
tail movements became more vigorous. The process of sperm binding to the HA microdots was
recorded at 30 frames/s. Fig. 5-6 (a) and (b) show sperms binding to the HA microdot at 5 and 30
min, respectively. If a moving sperm was detected to stop in the region of HA microdots, this
sperm was considered successfully binding to HA. In contrast, if a moving sperm passed the HA
microdot and disappeared out of the image boundary, it was considered an HA-unbound sperm.
During the process of sperm binding to the HA microdots, the head velocity of the HA-bound
sperms before binding was measured. The HA-bound sperms were compared to those unbound
ones in terms of their head speed. As in the first experiment, only those sperms that exhibited
linearly progressive movement were analyzed. The result as shown in Fig. 5-7 demonstrates that
the HA-bound sperms tended to have a higher head velocity than those unbound sperms. The
HA-bound sperms had an average head velocity of 76.28 μm/s with a standard deviation of
21.25 μm/s, while the average head velocity of those unbound sperms was 50.45 μm/s with a
standard deviation of 15.52 μm/s.
92
Figure 5-7 - Velocity comparison between the HA-bound sperms and unbound sperms.
We also observed in the HA binding experiment that after the sperms bound to the HA
microdots, their tails beat more vigorously. The tail beating amplitude was measured on the same
sperms before and after they bound to the HA microdots. Fig. 5-8 shows sperm tail beating
amplitude before and after binding to the HA microdots. The average amplitude produced by
these 30 HA-bound sperms was 5.31 μm (before binding) and 6.93 μm (after binding). These
results, for the first time, quantitatively reveal an increase in sperm tail beating amplitude before
and after a sperm binds to an HA microdot. The results also quantify differences in sperm tail
beating amplitude across HA-bound sperms. The measurement of sperm tail beating amplitude
can possibly be used as an additional criterion for sperm selection among HA-bound sperms.
93
Figure 5-8 - Tail beating amplitude of the same sperms before and after binding to HA
microdots.
5.5 Conclusion
This chapter presented visual tracking algorithms for tracking both the head and tail of motile
human sperms. The sperm head tracking algorithm is capable of tracking multiple moving
sperms with a high success rate. Based on the sperm head‟s position and its motion direction
vector, the STROI is located. In this region of interest, the MIR algorithm together with Kalman
filtering determines the sperm tail position. The sperm head and tail tracking algorithms enabled
a number of new findings. A significant correlation between sperm head velocity and tail beating
amplitude was found, suggesting that stronger tail propelling produces a higher velocity. The
results also reveal that sperms bound to HA generally have a higher velocity (before binding)
than those sperms that are not able to bind to HA microdots. This discovery “unifies” the
conventional sperm assessment criterion based on sperm velocity/motility and the most recent
HA assay technique. Among the sperms bound to HA microdots, their tails produce different
beating amplitudes. Measuring such amplitude differences quantitatively can possibly be used as
a new, useful sperm selection criterion among HA bound sperms.
94
Chapter 6 Conclusions and Future Research
Conclusions and Future Research 6
This chapter summarizes the major contributions of this research and future research directions.
6.1 Contributions
In summary, this research focused on the development of new robotic systems and automated
methods for characterization and manipulation of single cells (e.g., cardiomyocyte, embryo and
sperm). This research has addressed two practical problems for robotic micromanipulations.
With the new technical solutions, two robotic prototype systems have been designed and built for
high throughput measurement of GJIC and automatically processing embryos for vitrification.
Additionally, quantitative analysis of sperm locomotion behaviors revealed interesting
correlations between sperm head movement speed and the tail beating motions. The combination
of sperm locomotion analysis with HA assay provides new solutions to assist doctors or
embryologists to select high-quality sperms for IVF applications.
The major contributions of this research are summarized as follows:
1. Developed the first automated method for locating end-effector tips. This method solved
the long-lasting practical problem of how to search for and locate end-effector tips for all
micromanipulation tasks. A summary of guidelines was provided for implementation of
this automated method.
2. Developed two modes of computer vision based contact detection algorithms for
determination of relative Z-positions of end-effector tips and single cells. Without the use
of extra sensors (e.g., force sensors or electrodes), these vision based methods were able
to determine the relative Z positions between end-effector tips and cells. The newly
developed methods were integrated into the first-of-its-kind robotic adherent cell
injection system and the robotic vitrification system.
3. Developed an automated system for robotic microinjection of adherent cells to measure
gap junctional intercellular communications. The powerful adherent cell injection system
95
was made possible by integrating several new techniques including automatically
locating micropipette tips, robustly detecting the contact of micropipette tip with the cell
membrane, and precisely compensating for accumulative positioning errors. The system
has been used to quantitatively measure GJIC for screening efficacy of selected drug
molecules.
4. Developed an automated system for robotic processing of mouse embryos for
vitrification. Robotic vitrification relieves the human operator from tedious vitrification
steps and eliminates manual operation-caused errors and inconsistencies. It also offers
unparalleled timing control based on monitoring embryo volume changes in real time.
5. Developed computer vision tracking algorithms for automated analysis of sperm head
and tail locomotion behavior. The analysis of both sperm head and tail locomotion
behaviors revealed a significant correlation between sperm head velocity and tail beating
amplitude, suggesting that stronger tail propelling produces a higher velocity. The results
also showed that sperms bound to HA generally have a higher velocity (before binding)
than those sperms that are not able to bind to HA microdots. These results provided new
information to assist the selection of high-quality of human sperm in IVF clinics.
6.2 Future Research
6.2.1 Improvement of the Present Research
Despite the significant progress of the present research, there are still many opportunities
remaining for further improvement of the robotic systems and automated techniques for
characterization and manipulation of single cells. Several exemplary future research topics are:
Fully automated robotic injection system is required to screen a high number of FDA-
approved drug molecules. In order to achieve a fully automated system, a robust single
cell detection method needs to be developed to recognize the optimal injection locations
on individual cells for injection.
Gap junctions facilitate both diffusion of small molecules and conduction of ions between
adjacent cells. Gap junction conductance measurement can be performed by using a
double whole-cell patch clamp system or optical mapping technique. Future research
96
should involve the development of robotic automation approaches to achieve high
throughput measurement of electrical conduction.
To further increase the throughput of the robotic vitrification system, new functions such
as controlled aspiration of multiple embryos at one time are needed. Along this direction,
future research should focus on microfluidic modeling and controller design to accurately
position multiple embryos with equal distance inside a micropipette.
In the present robotic vitrification system, the last step still needs a technician to
manually pick up and plunge vitrification straws with vitrified embryos into liquid
nitrogen for cryopreservation. To fully eliminate human involvement in the vitrification
procedure, future research can focus on the development of a robotic pick-and-place
manipulator (e.g., a 3-DOF robotic arm with a gripper) to automatically freeze
vitrification straws in liquid nitrogen.
At present, the newly-developed robotic vitrification system was tested by using mouse
embryos. Future work should include more testing on mouse oocytes for oocyte
vitrification.
The current robotic vitrification system is designed to retrieve the embryos out of VS at
their minimum volume to reduce the toxic effects of DMSO. Future testing should
include the performance comparison between the individualized time control in the
robotic system and a fixed VS washing protocol to experimentally validate the theory of
minimizing toxic effects by real-time monitoring oocytes/embryo volume change.
The automated sperm analysis method in this research is able to extract the sperm tail
beating amplitude and use it to investigate the sperm locomotion behaviors. However, the
sperm forward movement also depends on the sperm beating frequency. Further
investigation should be conducted to measure the sperm tail-beating frequency by
developing new computer vision algorithms with high-speed imaging techniques (e.g.,
over 100 fps).
97
6.2.2 Research Outlook on Intracellular Measurement and Manipulation
Beyond single cell characterization and manipulation, there are an increasing number of
researchers starting to focus on the measurement of intracellular properties and manipulation of
sub-cellular structures or organelles. Inside a living cell, numerous biological processes and
biochemical reactions occur in the subcellular organelles, which are often compartmentalized
and dynamically change intracellular physical and chemical properties, for instance, temperature
[208], [209], pressure [210], mechanical [211], [212] and electrical characteristics [213], pH
[214], and concentrations of ions and other molecules [215], [216]. These processes, such as the
production of ATP by mitochondria or protein synthesis by ribosomes, require intracellular
homeostasis to maintain normal cellular functions. Therefore, it is not surprising that each
property is strictly regulated and varies among different intracellular structures and organelles.
Tracking the regulation of these quantities could reveal largely underexplored subcellular
functions and mechanisms. Moreover, an increasing body of evidence has indicated close
correlations between intracellular disorders and diseases. Thus, monitoring intracellular
environments and quantitatively measuring intracellular properties would enable us to better
understand subcellular activities and disease mechanisms and potentially develop new therapies
via rescuing/altering subcellular functions.
Directly measuring the properties of organelles and intracellular structures is difficult. Early
researchers attempted to use glass micropipettes and microelectrodes to measure cytoplasmic pH
[217], pressure [218] and electrical properties [219]. However, these direct measurements were
mostly made on an entire cell due to spatial limitations of these technologies. Compared to the
measurement of whole cells (typically tens of micrometers in size), the characterization of
organelles and intracellular structures requires finer spatial positioning accuracy and much more
miniaturized sensing tips. In addition, signals measured from intracellular structures are often
weak and differ minutely inside a cell. For instance, subcellular temperature variations in
different locations in a cell are only within a tenth of one degree [220]. To measure intracellular
properties, microsystems and nano-engineered tools developed for this purpose must have high
measurement sensitivities and resolutions.
The past few years have witnessed exploratory efforts in the development of new tools and
techniques for direct intracellular measurement and manipulation. For example, because of the
98
small sizes and unique electrical properties, nanowires [221] and nanotubes [222][223] have
been used to build tethered probes for intracellular electrical measurement. AFM tips, modified
via FIB or via direct assembly/growth of nanowires or nanotubes, have been used to quantify
intracellular mechanical characteristics [224] [225]. Untethered nanoparticles and MEMS
sensors, after being introduced into cells using manipulation techniques, have been used to
measure other physical properties (e.g., temperature, pressure) and chemical properties (e.g., pH,
Ca2+ concentration) inside a cell. By combining untethered sensors with tethered probes (e.g.,
SiO2 nanowires), intracellular activities have been transformed into photonic signals and
measured via optical single cell endoscopes [226].
Existing intracellular research started with manipulating and measuring large organelles, such as
cell nuclei, and then moved onto targeting smaller organelles, such as mitochondria. Many
intracellular properties (e.g., pH and temperature) in existing studies have been measured in the
cytoplasm. Future micro- and nanoengineered tools will become even finer in size and more
powerful in function to monitor real-time changes of suborganelle signals, such as pH and
temperature changes during ATP synthesis in mitochondria and calcium storage variations in the
nuclear membrane, reticulum, and Golgi apparatus [227], [228].
New materials, such as graphene, may possibly help in the development of more accurate and
sensitive measurement tools [229]. Graphene field effect transistors have been developed to
monitor action potentials of cardiomyocytes extracellularly [230], [231]. Graphene-based sensors
might be developed and delivered into single cells for intracellular electrical measurements. In
addition to new materials, emerging imaging techniques might also significantly accelerate the
advancement of intracellular measurement and manipulation capabilities. Studies using near-
field imaging enabled by optical nanowires have demonstrated the ability to accurately detect
fluorescent signals with higher resolutions [226]. These new imaging capabilities might enhance
the observation and measurement of subcellular and suborganelle signal changes.
Presently, intracellular measurement and manipulation are manually conducted. Automation
technologies can be developed to help minimize human errors and skill inconsistency [232].
Automation would allow researchers to more easily position tethered devices or more accurately
move untethered sensors inside a cell. In manual microinjection, for example, the number of
injected cells is limited to several or tens of cells[233]. To increase throughput, robotic
99
microinjection systems demonstrated in this research have shown the injection of over 1,000
cells within one hour [234]. The significantly higher throughput enabled quantitative
characterization of gap junction function on a large cell population. To improve the performance
of magnetic or optical tweezers, automated functions are under development to increase the
spatial resolution and accuracy for the manipulation of single cells [235], [236]. These
technologies have direct relevance and might be expanded to enhance intracellular manipulation
and measurement.
Compared to measurements in single cells, the direct measurement and manipulation of
subcellular structures and organelles remain largely underexplored. In the pursuit of better
understanding of intracellular properties, the development of new microsystems and
nanoengineered techniques would transform cell biology by enabling intracellular measurement
and manipulation. These new tools would enable researchers to directly interrogate intracellular
structures, explore the environment inside a cell, and observe and measure intracellular processes
and activities with high spatial and temporal resolutions. The exciting era of intracellular
measurement and manipulation has just begun.
100
References
[1] J. E. Ferrell and E. M. Machleder, “The Biochemical Basis of an All-or-None Cell Fate
Switch in Xenopus Oocytes,” Sci. , vol. 280 , no. 5365 , pp. 895–898, May 1998.
[2] J.-H. Chen, C. Y. Y. Yip, E. D. Sone, and C. A. Simmons, “Identification and
Characterization of Aortic Valve Mesenchymal Progenitor Cells with Robust Osteogenic
Calcification Potential,” Am. J. Pathol., vol. 174, no. 3, pp. 1109–1119, Mar. 2009.
[3] S. D. Carson and S. J. Pirruccello, “HeLa cell heterogeneity and coxsackievirus B3
cytopathic effect: Implications for inter-laboratory reproducibility of results,” J. Med.
Virol., vol. 85, no. 4, pp. 677–683, Apr. 2013.
[4] X. Wang, “Heterogeneity in the HeLa Cell Cycle Response to UVC Analyzed by the
BrdUrd Two-Parameter Method,” Exp. Cell Res., vol. 212, no. 2, pp. 176–189, Jun. 1994.
[5] M. B. Elowitz, “Stochastic Gene Expression in a Single Cell,” Science (80-. )., vol. 297,
no. 5584, pp. 1183–1186, Aug. 2002.
[6] P. D. Bhola and S. M. Simon, “Determinism and divergence of apoptosis susceptibility in
mammalian cells.,” J. Cell Sci., vol. 122, no. Pt 23, pp. 4296–302, Dec. 2009.
[7] X. Xia, M. S. Owen, R. E. C. Lee, and S. Gaudet, “Cell-to-cell variability in cell death :
can systems biology help us make sense of it all ?” vol. 5, no. 5, pp. e1261–11, 2014.
[8] J. Liu, C. Moraes, Z. Lu, C. Simmons, and Y. Sun, “Single Cell Deposition,” 2012, pp.
403–420.
[9] C. Gawad, W. Koh, and S. R. Quake, “Single-cell genome sequencing: current state of the
science,” Nat. Rev. Genet., vol. 17, no. 3, pp. 175–188, Jan. 2016.
[10] O. Schwartzman and A. Tanay, “Single-cell epigenomics: techniques and emerging
applications,” Nat. Rev. Genet., vol. 16, no. 12, pp. 716–726, Oct. 2015.
[11] P. Liberali, B. Snijder, and L. Pelkmans, “Single-cell and multivariate approaches in
genetic perturbation screens,” Nat. Rev. Genet., vol. 16, no. 1, pp. 18–32, Dec. 2014.
101
[12] J. Czyz, U. Irmer, G. Schulz, A. Mindermann, and D. F. Hülser, “Gap-junctional coupling
measured by flow cytometry.,” Exp. Cell Res., vol. 255, no. 1, pp. 40–6, Feb. 2000.
[13] E. L. Hertzberg, T. S. Lawrence, and N. B. Gilula, “Gap Junctional Communication,”
Annu. Rev. Physiol., vol. 43, no. 1, pp. 479–491, Oct. 1981.
[14] N. S. Heyman, D. T. Kurjiaka, J. F. Ek Vitorin, and J. M. Burt, “Regulation of gap
junctional charge selectivity in cells coexpressing connexin 40 and connexin 43,” AJP
Hear. Circ. Physiol., vol. 297, no. 1, pp. H450–H459, Jul. 2009.
[15] P. R. Brink, K. Cronin, and S. V. Ramanan, “Gap junctions in excitable cells,” J.
Bioenerg. Biomembr., vol. 28, no. 4, pp. 351–358, Aug. 1996.
[16] G. H. Kalimi and C. W. Lo, “Gap junctional communication in the extraembryonic tissues
of the gastrulating mouse embryo.,” J. Cell Biol., vol. 109, no. 6 Pt 1, pp. 3015–26, Dec.
1989.
[17] W. R. Loewenstein, “Junctional intercellular communication: the cell-to-cell membrane
channel.,” Physiol. Rev., vol. 61, no. 4, pp. 829–913, Oct. 1981.
[18] M. M. Vance and L. M. Wiley, “Gap junction intercellular communication mediates the
competitive cell proliferation disadvantage of irradiated mouse preimplantation embryos
in aggregation chimeras.,” Radiat. Res., vol. 152, no. 5, pp. 544–51, Nov. 1999.
[19] D. E. Gutstein, “Focal gap junction uncoupling and spontaneous ventricular ectopy,” AJP
Hear. Circ. Physiol., vol. 289, no. 3, pp. H1091–H1098, May 2005.
[20] M. Strom, X. Wan, S. Poelzing, E. Ficker, and D. S. Rosenbaum, “Gap junction
heterogeneity as mechanism for electrophysiologically distinct properties across the
ventricular wall.,” Am. J. Physiol. Heart Circ. Physiol., vol. 298, no. 3, pp. H787–94, Mar.
2010.
[21] S. S. Chugh, K. Reinier, C. Teodorescu, A. Evanado, E. Kehr, M. Al Samara, R. Mariani,
K. Gunson, and J. Jui, “Epidemiology of sudden cardiac death: clinical and research
implications.,” Prog. Cardiovasc. Dis., vol. 51, no. 3, pp. 213–28.
102
[22] J. F. Ek-Vitorin and J. M. Burt, “Structural basis for the selective permeability of channels
made of communicating junction proteins.,” Biochim. Biophys. Acta, vol. 1828, no. 1, pp.
51–68, Jan. 2013.
[23] M. Rackauskas, V. Neverauskas, and V. A. Skeberdis, “Diversity and properties of
connexin gap junction channels.,” Medicina (Kaunas)., vol. 46, no. 1, pp. 1–12, 2010.
[24] D. B. Alexander and G. S. Goldberg, “Transfer of biologically important molecules
between cells through gap junction channels.,” Curr. Med. Chem., vol. 10, no. 19, pp.
2045–58, Oct. 2003.
[25] L. H. Raptis, H. L. Brownell, K. L. Firth, and L. W. Mackenzie, “A novel technique for
the study of intercellular, junctional communication: electroporation of adherent cells on a
partly conductive slide.,” DNA Cell Biol., vol. 13, no. 9, pp. 963–75, Sep. 1994.
[26] J.-C. Hervé, N. Bourmeyster, and D. Sarrouilhe, “Diversity in protein-protein interactions
of connexins: emerging roles.,” Biochim. Biophys. Acta, vol. 1662, no. 1–2, pp. 22–41,
Mar. 2004.
[27] D. J. Fitzgerald and A. W. Murray, “Inhibition of intercellular communication by tumor-
promoting phorbol esters.,” Cancer Res., vol. 40, no. 8 Pt 1, pp. 2935–7, Aug. 1980.
[28] M. H. El-Fouly, J. E. Trosko, and C.-C. Chang, “Scrape-loading and dye transfer,” Exp.
Cell Res., vol. 168, no. 2, pp. 422–430, Feb. 1987.
[29] L. H. Raptis, H. L. Brownell, K. L. Firth, and L. W. Mackenzie, “A novel technique for
the study of intercellular, junctional communication: electroporation of adherent cells on a
partly conductive slide.,” DNA Cell Biol., vol. 13, no. 9, pp. 963–75, Sep. 1994.
[30] M. H. Wade, J. E. Trosko, and M. Schindler, “A fluorescence photobleaching assay of gap
junction-mediated communication between human cells.,” Science, vol. 232, no. 4749, pp.
525–8, Apr. 1986.
[31] M. L. McCain, T. Desplantez, N. A. Geisse, B. Rothen-Rutishauser, H. Oberer, K. K.
Parker, and A. G. Kleber, “Cell-to-cell coupling in engineered pairs of rat ventricular
cardiomyocytes: relation between Cx43 immunofluorescence and intercellular electrical
103
conductance.,” Am. J. Physiol. Heart Circ. Physiol., vol. 302, no. 2, pp. H443–50, Jan.
2012.
[32] M. Abbaci, M. Barberi-Heyob, W. Blondel, F. Guillemin, and J. Didelon, “Advantages
and limitations of commonly used methods to assay the molecular permeability of gap
junctional intercellular communication.,” Biotechniques, vol. 45, no. 1, pp. 33–52, 56–62,
Jul. 2008.
[33] M. H. Juul, E. Rivedal, T. Stokke, and T. Sanner, “Quantitative determination of gap
junction intercellular communication using flow cytometric measurement of fluorescent
dye transfer.,” Cell Adhes. Commun., vol. 7, no. 6, pp. 501–12, 2000.
[34] J. Boivin, L. Bunting, J. A. Collins, and K. G. Nygren, “International estimates of
infertility prevalence and treatment-seeking: potential need and demand for infertility
medical care,” Hum. Reprod., vol. 22, no. 6, pp. 1506–1512, Mar. 2007.
[35] G. Palermo, H. Joris, P. Devroey, and A. C. Van Steirteghem, “Pregnancies after
intracytoplasmic injection of single spermatozoon into an oocyte.,” Lancet (London,
England), vol. 340, no. 8810, pp. 17–8, Jul. 1992.
[36] P. Devroey, “A review of ten years experience of ICSI,” Hum. Reprod. Update, vol. 10,
no. 1, pp. 19–28, Jan. 2004.
[37] J. Konc, K. Kanyó, R. Kriston, B. Somoskői, and S. Cseh, “Cryopreservation of Embryos
and Oocytes in Human Assisted Reproduction,” Biomed Res. Int., vol. 2014, pp. 1–9,
2014.
[38] J. Boldt, “Human oocyte cryopreservation as an adjunct to IVF-embryo transfer cycles,”
Hum. Reprod., vol. 18, no. 6, pp. 1250–1255, Jun. 2003.
[39] M. Kuwayama, G. Vajta, O. Kato, and S. P. Leibo, “Highly efficient vitrification method
for cryopreservation of human oocytes,” Reprod. Biomed. Online, vol. 11, no. 3, pp. 300–
308, Jan. 2005.
[40] R. Riggs, J. Mayer, D. Dowling-Lacey, T.-F. Chi, E. Jones, and S. Oehninger, “Does
storage time influence postthaw survival and pregnancy outcome? An analysis of 11,768
104
cryopreserved human embryos,” Fertil. Steril., vol. 93, no. 1, pp. 109–115, Jan. 2010.
[41] D. E. Pegg, “The relevance of ice crystal formation for the cryopreservation of tissues and
organs,” Cryobiology, vol. 60, no. 3, pp. S36–S44, Jul. 2010.
[42] J. Saragusty and A. Arav, “Current progress in oocyte and embryo cryopreservation by
slow freezing and vitrification,” Reproduction, vol. 141, no. 1, pp. 1–19, Jan. 2011.
[43] G. Vajta and Z. P. Nagy, “Are programmable freezers still needed in the embryo
laboratory? Review on vitrification.,” Reprod. Biomed. Online, vol. 12, no. 6, pp. 779–96,
Jun. 2006.
[44] W. F. Rall and G. M. Fahy, “Ice-free cryopreservation of mouse embryos at -196 degrees
C by vitrification.,” Nature, vol. 313, no. 6003, pp. 573–5, 1985.
[45] G. Palermo, H. Joris, P. Devroey, and a C. Van Steirteghem, “Pregnancies after
intracytoplasmic injection of single spermatozoon into an oocyte.,” Lancet, vol. 340, no.
8810, pp. 17–8, Jul. 1992.
[46] T. G. Cooper, E. Noonan, S. von Eckardstein, J. Auger, H. W. G. Baker, H. M. Behre, T.
B. Haugen, T. Kruger, C. Wang, M. T. Mbizvo, and K. M. Vogelsong, “World Health
Organization reference values for human semen characteristics.,” Hum. Reprod. Update,
vol. 16, no. 3, pp. 231–45, 2009.
[47] B. Bartoov, A. Berkovitz, F. Eltes, A. Kogosovsky, A. Yagoda, H. Lederman, S. Artzi, M.
Gross, and Y. Barak, “Pregnancy rates are higher with intracytoplasmic morphologically
selected sperm injection than with conventional intracytoplasmic injection,” Fertil. Steril.,
vol. 80, no. 6, pp. 1413–1419, Dec. 2003.
[48] H. E. Chemes, “Sperm pathology: a step beyond descriptive morphology. Origin,
characterization and fertility potential of abnormal sperm phenotypes in infertile men,”
Hum. Reprod. Update, vol. 9, no. 5, pp. 405–428, Sep. 2003.
[49] J. Tesarik, C. Mendoza, and E. Greco, “Paternal effects acting during the first cell cycle of
human preimplantation development after ICSI,” Hum. Reprod., vol. 17, no. 1, pp. 184–9,
Jan. 2002.
105
[50] A. De Vos, H. Van De Velde, H. Joris, G. Verheyen, P. Devroey, and A. Van Steirteghem,
“Influence of individual sperm morphology on fertilization, embryo morphology, and
pregnancy outcome of intracytoplasmic sperm injection.,” Fertil. Steril., vol. 79, no. 1, pp.
42–8, Jan. 2003.
[51] G. Huszar, C. C. Ozenci, S. Cayli, Z. Zavaczki, E. Hansch, and L. Vigue, “Hyaluronic
acid binding by human sperm indicates cellular maturity, viability, and unreacted
acrosomal status,” Fertil. Steril., vol. 79, no. June, pp. 1616–1624, Jun. 2003.
[52] A. Jakab, D. Sakkas, E. Delpiano, S. Cayli, E. Kovanci, D. Ward, A. Revelli, A. Ravelli,
and G. Huszar, “Intracytoplasmic sperm injection: a novel selection method for sperm
with normal frequency of chromosomal aneuploidies.,” Fertil. Steril., vol. 84, no. 6, pp.
1665–73, Dec. 2005.
[53] A. Yagci, W. Murk, J. Stronk, and G. Huszar, “Spermatozoa bound to solid state
hyaluronic acid show chromatin structure with high DNA chain integrity: an acridine
orange fluorescence study.,” J. Androl., vol. 31, no. 6, pp. 566–72, 2010.
[54] R. P. Amann and D. F. Katz, “Reflections on CASA after 25 years.,” J. Androl., vol. 25,
no. 3, pp. 317–25, 2004.
[55] L. Z. Shi, J. M. Nascimento, M. W. Berns, and E. L. Botvinick, “Computer-based tracking
of single sperm.,” J. Biomed. Opt., vol. 11, no. 5, p. 054009, 2006.
[56] V. R. Nafisi, M. H. Moradi, and M. H. Nasr-Esfahani, “A template matching algorithm for
sperm tracking and classification.,” Physiol. Meas., vol. 26, no. 5, pp. 639–51, Oct. 2005.
[57] Y. Sun and B. J. Nelson, “Biological Cell Injection Using an Autonomous MicroRobotic
System,” Int. J. Rob. Res., vol. 21, no. 10–11, pp. 861–868, Oct. 2002.
[58] J. L. Tan, J. Tien, D. M. Pirone, D. S. Gray, K. Bhadriraju, and C. S. Chen, “Cells lying
on a bed of microneedles: an approach to isolate mechanical force.,” Proc. Natl. Acad. Sci.
U. S. A., vol. 100, no. 4, pp. 1484–9, Mar. 2003.
[59] H. Lee, Y. Liu, D. Ham, and R. M. Westervelt, “Integrated cell manipulation system--
CMOS/microfluidic hybrid.,” Lab Chip, vol. 7, no. 3, pp. 331–7, Mar. 2007.
106
[60] K. Kim, X. Liu, Y. Zhang, and Y. Sun, “Nanonewton force-controlled manipulation of
biological cells using a monolithic MEMS microgripper with two-axis force feedback,” J.
Micromechanics Microengineering, vol. 18, no. 5, p. 055013, May 2008.
[61] M. Biro and J.-L. Maître, “Dual pipette aspiration: A unique tool for studying intercellular
adhesion,” 2015, pp. 255–267.
[62] P. Mirbod and D. Meng, “Analysis of bolus formation in micropipette ejection systems.,”
Eur. Phys. J. E. Soft Matter, vol. 38, no. 6, p. 59, Jun. 2015.
[63] Z. Lu, C. Moraes, G. Ye, C. a Simmons, and Y. Sun, “Single cell deposition and
patterning with a robotic system.,” PLoS One, vol. 5, no. 10, p. e13542, Jan. 2010.
[64] J. Geraedts and K. Sermon, “Preimplantation genetic screening 2.0: the theory.,” Mol.
Hum. Reprod., Jun. 2016.
[65] K. Kim, X. Liu, Y. Zhang, J. Cheng, X. Y. Wu, and Y. Sun, “Mechanical characterization
of polymeric microcapsules using a force-feedback MEMS microgripper,” in 2008 30th
Annual International Conference of the IEEE Engineering in Medicine and Biology
Society, 2008, pp. 1845–1848.
[66] Yu Sun, Kai-Tak Wan, K. P. Roberts, J. C. Bischof, and B. J. Nelson, “Mechanical
property characterization of mouse zona pellucida,” IEEE Trans. Nanobioscience, vol. 2,
no. 4, pp. 279–286, Dec. 2003.
[67] M. R. Bennett and J. Hasty, “Microfluidic devices for measuring gene network dynamics
in single cells,” Nat. Rev. Genet., vol. 10, no. 9, pp. 628–638, Sep. 2009.
[68] A. Barani, H. Paktinat, M. Janmaleki, A. Mohammadi, P. Mosaddegh, A. Fadaei-Tehrani,
and A. Sanati-Nezhad, “Microfluidic integrated acoustic waving for manipulation of cells
and molecules.,” Biosens. Bioelectron., vol. 85, pp. 714–725, May 2016.
[69] Y. Zheng, M. A. Cachia, J. Ge, Z. Xu, C. Wang, and Y. Sun, “Mechanical differences of
sickle cell trait (SCT) and normal red blood cells,” Lab Chip, vol. 15, no. 15, pp. 3138–
3146, 2015.
107
[70] E. S. Park, C. Jin, Q. Guo, R. R. Ang, S. P. Duffy, K. Matthews, A. Azad, H. Abdi, T.
Todenhöfer, J. Bazov, K. N. Chi, P. C. Black, and H. Ma, “Continuous Flow
Deformability-Based Separation of Circulating Tumor Cells Using Microfluidic
Ratchets,” Small, vol. 12, no. 14, pp. 1909–1919, Apr. 2016.
[71] Y. Liu and H. Lu, “Microfluidics in systems biology-hype or truly useful?,” Curr. Opin.
Biotechnol., vol. 39, pp. 215–20, Jun. 2016.
[72] D. G. Pyne, J. Liu, M. Abdelgawad, and Y. Sun, “Digital microfluidic processing of
Mammalian embryos for vitrification.,” PLoS One, vol. 9, no. 9, p. e108128, Jan. 2014.
[73] P. Y. Chiou, A. T. Ohta, and M. C. Wu, “Massively parallel manipulation of single cells
and microparticles using optical images.,” Nature, vol. 436, no. 7049, pp. 370–2, Jul.
2005.
[74] Y.-C. Wu, T.-H. Wu, D. L. Clemens, B.-Y. Lee, X. Wen, M. A. Horwitz, M. A. Teitell,
and P.-Y. Chiou, “Massively parallel delivery of large cargo into mammalian cells with
light pulses,” Nat. Methods, vol. 12, no. 5, pp. 439–444, 2015.
[75] C. Qian, H. Huang, L. Chen, X. Li, Z. Ge, T. Chen, Z. Yang, and L. Sun,
“Dielectrophoresis for Bioparticle Manipulation,” Int. J. Mol. Sci., vol. 15, no. 10, pp.
18281–18309, Oct. 2014.
[76] A. A. Al Balushi, A. Kotnala, S. Wheaton, R. M. Gelfand, Y. Rajashekara, and R.
Gordon, “Label-free free-solution nanoaperture optical tweezers for single molecule
protein studies.,” Analyst, vol. 140, no. 14, pp. 4760–78, Jul. 2015.
[77] J. S. Kanger, V. Subramaniam, and R. van Driel, “Intracellular manipulation of chromatin
using magnetic nanoparticles.,” Chromosom. Res., vol. 16, no. 3, pp. 511–22, Jan. 2008.
[78] W. Feneberg, M. Westphal, and E. Sackmann, “Dictyostelium cells‟ cytoplasm as an
active viscoplastic body,” Eur. Biophys. J., vol. 30, no. 4, pp. 284–294, Aug. 2001.
[79] A. H. B. de Vries, B. E. Krenn, R. van Driel, V. Subramaniam, and J. S. Kanger, “Direct
observation of nanomechanical properties of chromatin in living cells.,” Nano Lett., vol. 7,
no. 5, pp. 1424–7, May 2007.
108
[80] J. K. Fisher, J. R. Cummings, K. V. Desai, L. Vicci, B. Wilde, K. Keller, C. Weigle, G.
Bishop, R. M. Taylor, C. W. Davis, R. C. Boucher, E. T. O‟Brien, and R. Superfine,
“Three-dimensional force microscope: A nanometric optical tracking and magnetic
manipulation system for the biomedical sciences,” Rev. Sci. Instrum., vol. 76, no. 5, p.
053711, 2005.
[81] S. Erni, F. Ullrich, J. Pokki, D. R. Frutiger, O. Ergeneman, B. E. Kratochvil, and B. J.
Nelson, “Non-contact 3D magnetic biomanipulation for in vivo and in vitro applications,”
pp. 0–1, 2012.
[82] E. B. Steager, M. Selman Sakar, C. Magee, M. Kennedy, A. Cowley, and V. Kumar,
“Automated biomanipulation of single cells using magnetic microrobots,” Int. J. Rob.
Res., vol. 32, no. 3, pp. 346–359, Mar. 2013.
[83] E. Diller, J. Giltinan, and M. Sitti, “Independent control of multiple magnetic microrobots
in three dimensions,” Int. J. Rob. Res., vol. 32, no. 5, pp. 614–631, May 2013.
[84] S. Floyd, E. Diller, C. Pawashe, and M. Sitti, “Control methodologies for a heterogeneous
group of untethered magnetic micro-robots,” Int. J. Rob. Res., vol. 30, no. 13, pp. 1553–
1565, Mar. 2011.
[85] E. Diller and M. Sitti, “Three-Dimensional Programmable Assembly by Untethered
Magnetic Robotic Micro-Grippers,” Adv. Funct. Mater., p. n/a–n/a, Apr. 2014.
[86] B. J. Nelson, I. K. Kaliakatsos, and J. J. Abbott, “Microrobots for minimally invasive
medicine.,” Annu. Rev. Biomed. Eng., vol. 12, pp. 55–85, Aug. 2010.
[87] S. Regnier and M. Sitti, “Rotating Magnetic Miniature Swimming Robots With Multiple
Flexible Flagella,” IEEE Trans. Robot., vol. 30, no. 1, pp. 3–13, Feb. 2014.
[88] L. Zhang, J. J. Abbott, L. Dong, B. E. Kratochvil, D. Bell, and B. J. Nelson, “Artificial
bacterial flagella: Fabrication and magnetic control,” Appl. Phys. Lett., vol. 94, no. 6, p.
064107, 2009.
[89] M. Hagiwara, T. Kawahara, T. Iijima, and F. Arai, “High-Speed Magnetic Microrobot
Actuation in a Microfluidic Chip by a Fine V-Groove Surface,” IEEE Trans. Robot., vol.
109
29, no. 2, pp. 363–372, Apr. 2013.
[90] T. Hayakawa, S. Fukada, and F. Arai, “Fabrication of an On-Chip Nanorobot Integrating
Functional Nanomaterials for Single-Cell Punctures,” IEEE Trans. Robot., vol. 30, no. 1,
pp. 59–67, Feb. 2014.
[91] a. Georgiev, P. K. Allen, and W. Edstrom, “Visually-guided protein crystal manipulation
using micromachined silicon tools,” in 2004 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), 2004, vol. 1, pp. 236–
241.
[92] B. a. Wester, S. Rajaraman, J. D. Ross, M. C. LaPlaca, and M. G. Allen, “Development
and characterization of a packaged mechanically actuated microtweezer system,” Sensors
Actuators A Phys., vol. 167, no. 2, pp. 502–511, Jun. 2011.
[93] Y. Choi, J. Ross, B. Wester, and M. G. Allen, “Mechanically driven microtweezers with
integrated microelectrodes,” J. Micromechanics Microengineering, vol. 18, no. 6, p.
065004, Jun. 2008.
[94] Z. Ni, A. Bolopion, J. Agnus, R. Benosman, and S. Regnier, “Asynchronous Event-Based
Visual Shape Tracking for Stable Haptic Feedback in Microrobotics,” IEEE Trans.
Robot., pp. 1–9, 2012.
[95] F. Beyeler, S. Member, A. Neild, S. Oberti, D. J. Bell, Y. Sun, J. Dual, and B. J. Nelson,
“Monolithically fabricated microgripper with integrated force sensor for manipulating
microobjects and biological cells aligned in an ultrasonic field,” J.
Microelectromechanical Syst., vol. 16, no. 1, pp. 7–15, 2007.
[96] K. Kim, X. Liu, Y. Zhang, and Y. Sun, “Nanonewton force-controlled manipulation of
biological cells using a monolithic MEMS microgripper with two-axis force feedback,” J.
Micromechanics Microengineering, vol. 18, no. 5, p. 055013, May 2008.
[97] Xinyu Liu, K. Kim, Yong Zhang, and Yu Sun, “Nanonewton Force Sensing and Control
in Microrobotic Cell Manipulation,” Int. J. Rob. Res., vol. 28, no. 8, pp. 1065–1076, Aug.
2009.
110
[98] M. Mayyas, P. Zhang, W. H. Lee, D. Popa, and J. C. Chiao, “An active micro joining
mechanism for 3D assembly,” J. Micromechanics Microengineering, vol. 19, no. 3, p.
035012, Mar. 2009.
[99] D. J. Cappelleri and G. Piazza, “A two dimensional vision-based force sensor for
microrobotic applications,” Sensors Actuators A Phys., vol. 171, no. 2, pp. 340–351, Nov.
2011.
[100] X. Liu, R. Fernandes, A. Jurisicova, R. F. Casper, and Y. Sun, “In situ mechanical
characterization of mouse oocytes using a cell holding device.,” Lab Chip, vol. 10, no. 16,
pp. 2154–61, Aug. 2010.
[101] H. B. H. B. Huang, D. Sun, J. K. J. K. Mills, S. H. Cheng, S. Member, J. K. J. K. Mills,
and S. H. Cheng, “Robotic cell injection system with position and force control: toward
automatic batch biomanipulation,” IEEE Trans. Robot., vol. 25, no. 3, pp. 727–737, Jun.
2009.
[102] Y. Xie, D. Sun, H. Y. G. Tse, C. Liu, and S. H. Cheng, “Force Sensing and Manipulation
Strategy in Robot-Assisted Microinjection on Zebrafish Embryos,” IEEE/ASME Trans.
Mechatronics, vol. 16, no. 6, pp. 1002–1010, Dec. 2011.
[103] D. Kim, B. Kim, S. Yun, and S. Kwon, “Cellular force measurement for force reflected
biomanipulation,” IEEE Int. Conf. Robot. Autom. 2004. Proceedings. ICRA ’04. 2004, pp.
2412–2417 Vol.3, 2004.
[104] W. J. Li and N. Xi, “Novel micro gripping, probing, and sensing devices for single-cell
surgery.,” Conf. Proc. IEEE Eng. Med. Biol. Soc., vol. 4, pp. 2591–4, Jan. 2004.
[105] D.-H. Kim, C. N. Hwang, Y. Sun, S. H. Lee, B. Kim, and B. J. Nelson, “Mechanical
analysis of chorion softening in prehatching stages of zebrafish embryos.,” IEEE Trans.
Nanobioscience, vol. 5, no. 2, pp. 89–94, Jun. 2006.
[106] Y. Xie, D. Sun, C. Liu, H. Y. Tse, and S. H. Cheng, “A Force Control Approach to a
Robot-assisted Cell Microinjection System,” Int. J. Rob. Res., vol. 29, no. 9, pp. 1222–
1232, Nov. 2009.
111
[107] Y. Xie, Y. Zhou, Y. Lin, L. Wang, and W. Xi, “Development of a Microforce Sensor and
Its Array Platform for Robotic Cell Microinjection Force Measurement.,” Sensors
(Basel)., vol. 16, no. 4, 2016.
[108] C. D. Onal and M. Sitti, “Visual Servoing-Based Autonomous 2-D Manipulation of
Microparticles Using a Nanoprobe,” IEEE Trans. Control Syst. Technol., vol. 15, no. 5,
pp. 842–852, Sep. 2007.
[109] W. Kang, R. L. McNaughton, F. Yavari, M. Minary-Jolandan, A. Safi, and H. D.
Espinosa, “Microfluidic parallel patterning and cellular delivery of molecules with a
nanofountain probe.,” J. Lab. Autom., vol. 19, no. 1, pp. 100–9, Feb. 2014.
[110] H. Liu, J. Wen, J. Liu, S. Hopyan, C. Simmons, and Y. Sun, “Mechanical characterization
of cancer cell nuclei in situ,” in The 9th IEEE International Conference on Nano/Micro
Engineered and Molecular Systems (NEMS), 2014, pp. 669–673.
[111] T. Fukuda, M. Nakajima, M. Ahmad, Y. Shen, and M. Kojima, “Micro- and
nanomechatronics,” IEEE Ind. Electron. Mag., vol. 4, no. 4, pp. 13–22, Dec. 2010.
[112] Y. Shen, M. Nakajima, Z. Yang, Q. Huang, and T. Fukuda, “Real-time measurement of
the adhesion force by Hybrid System of ESEM and AFM Cantilever,” 2013 13th IEEE
Int. Conf. Nanotechnol. (IEEE-NANO 2013), pp. 325–328, Aug. 2013.
[113] M. R. Ahmad, M. Nakajima, S. Kojima, M. Homma, and T. Fukuda, “In Situ Single Cell
Mechanics Characterization of Yeast Cells Using Nanoneedles Inside Environmental
SEM,” IEEE Trans. Nanotechnol., vol. 7, no. 5, pp. 607–616, Sep. 2008.
[114] M. R. Ahmad, M. Nakajima, M. Kojima, S. Kojima, M. Homma, and T. Fukuda,
“Nanofork for single cells adhesion measurement via ESEM-nanomanipulator system.,”
IEEE Trans. Nanobioscience, vol. 11, no. 1, pp. 70–8, Mar. 2012.
[115] Y. Shen, M. Nakajima, S. Kojima, M. Homma, M. Kojima, and T. Fukuda, “Single cell
adhesion force measurement for cell viability identification using an AFM cantilever-
based micro putter,” Meas. Sci. Technol., vol. 22, no. 11, p. 115802, Nov. 2011.
[116] M. R. Ahmad, M. Nakajima, M. Kojima, S. Kojima, M. Homma, and T. Fukuda,
112
“Instantaneous and Quantitative Single Cells Viability Determination Using Dual
Nanoprobe Inside ESEM,” IEEE Trans. Nanotechnol., vol. 11, no. 2, pp. 298–306, Mar.
2012.
[117] Y. Shen, M. Nakajima, Z. Yang, S. Kojima, M. Homma, and T. Fukuda, “Design and
characterization of nanoknife with buffering beam for in situ single-cell cutting,”
Nanotechnology, vol. 22, no. 30, p. 305701, Jul. 2011.
[118] P. Kallio and J. Kuncová, “Manipulation of living biological cells: Challenges in
automation,” in Workshop on microrobotics for biomanipulation in the IROS’03, 2003.
[119] P. Kallio, T. Ritala, M. Lukkari, and S. Kuikka, “Injection Guidance System for Cellular
Microinjections,” Int. J. Rob. Res., vol. 26, no. 11–12, pp. 1303–1313, Nov. 2007.
[120] K. Viigipuu and P. Kallio, “Microinjection of living adherent cells by using a semi-
automatic microinjection system.,” Altern. Lab. Anim., vol. 32, no. 4, pp. 417–23, Oct.
2004.
[121] K. Kovanen, M. Lukkari, S. Purmonen, T. Ylikomi, J. Viitanen, and P. J. Kallio, “Electric
impedance assisted micropipette aspiration.,” Conf. Proc. IEEE Eng. Med. Biol. Soc., vol.
2007, pp. 5823–6, Jan. 2007.
[122] K. Sakaki, N. Dechev, R. D. Burke, and E. J. Park, “Development of an Autonomous
Biological Cell Manipulator With Single-Cell Electroporation and Visual Servoing
Capabilities,” IEEE Trans. Biomed. Eng., vol. 56, no. 8, pp. 2064–2074, 2009.
[123] C. Zhang, P. Li, L. Liu, Y. Wang, Z. Gao, and G. Li, “Development of
Mechanostimulated Patch-Clamp System for Cellular Physiological Study,” IEEE/ASME
Trans. Mechatronics, vol. 19, no. 4, pp. 1138–1147, Aug. 2014.
[124] G. Yang, J. a. Gaines, and B. J. Nelson, “Optomechatronic Design of Microassembly
Systems for Manufacturing Hybrid Microsystems,” IEEE Trans. Ind. Electron., vol. 52,
no. 4, pp. 1013–1023, Aug. 2005.
[125] Y. L. Zhang, M. L. Han, C. Y. Shee, T. F. Chia, and W. T. Ang, “Self-calibration method
for vision-guided cell micromanipulation systems.,” J. Microsc., vol. 233, no. 2, pp. 340–
113
5, Feb. 2009.
[126] Z. Yi, T. KokKiong, and H. Sunan, “Software Based Vision System for Automated Cell
Injection,” 2008 Int. Conf. Biomed. Eng. Informatics, pp. 718–722, May 2008.
[127] G. Becattini, L. S. Mattos, and D. G. Caldwell, “A fully automated system for adherent
cells microinjection.,” IEEE J. Biomed. Heal. informatics, vol. 18, no. 1, pp. 83–93, Jan.
2014.
[128] G. Becattini, L. S. Mattos, and D. G. Caldwell, “A visual targeting system for the
microinjection of unstained adherent cells.,” Comput. Biol. Med., vol. 43, no. 2, pp. 109–
20, Feb. 2013.
[129] L. S. Mattos, E. Grant, R. Thresher, K. Kluckman, and D. G. Caldwell, “Experiments with
a teleoperated system for improved bio-micromanipulations,” in The 2010 IEEE
International Conference on Information and Automation, 2010, pp. 1–6.
[130] B. Tamadazte, S. Dembele, G. Fortier, and N. Le Fort-Piat, “Automatic
micromanipulation using multiscale visual servoing,” in 2008 IEEE International
Conference on Automation Science and Engineering, 2008, pp. 977–982.
[131] Y. H. Anis, M. R. Holl, and D. R. Meldrum, “Automated Selection and Placement of
Single Cells Using Vision-Based Feedback Control,” IEEE Trans. Autom. Sci. Eng., vol.
7, no. 3, pp. 598–606, Jul. 2010.
[132] H. Uvet, A. Hasegawa, K. Ohara, T. Takubo, Y. Mae, and T. Arai, “Vision-Based
Automated Single-Cell Loading and Supply System,” IEEE Trans. Nanobioscience, vol.
8, no. 4, pp. 332–340, Dec. 2009.
[133] E. Avci, C.-N. Nguyen, K. Ohara, Y. Mae, T. Arai, and C. Gobel, “Vibration analysis of
microhand for high speed single cell manipulation,” 2012 IEEE Int. Conf. Mechatronics
Autom., pp. 75–80, Aug. 2012.
[134] K. Inoue, T. Tanikawa, and T. Arai, “Micro-manipulation system with a two-fingered
micro-hand and its potential application in bioscience.,” J. Biotechnol., vol. 133, no. 2, pp.
219–24, Jan. 2008.
114
[135] Zenan Wang, Win Tun Latt, S. Yih Min Tan, and Wei Tech Ang, “Visual Servoed Three-
Dimensional Cell Rotation System,” IEEE Trans. Biomed. Eng., vol. 62, no. 10, pp. 2498–
2507, Oct. 2015.
[136] B. K. Chen, Y. Zhang, and Y. Sun, “Active release of microobjects using a MEMS
microgripper to overcome adhesion forces,” J. Microelectromechanical Syst., vol. 18, no.
3, pp. 652–659, Jun. 2009.
[137] D. O. Popa and H. E. Stephanou, “Micro and meso scale robotic assembly,” J. Manuf.
Process., vol. 6, no. 1, pp. 52–71, Jan. 2004.
[138] M. Sitti, “Microscale and nanoscale robotics systems,” IEEE Robot. Autom. Mag., vol. 14,
no. 1, pp. 53–60, Mar. 2007.
[139] Z. NI, A. Bolopion, J. Agnus, R. Benosman, and S. Regnier, “Asynchronous Event-Based
Visual Shape Tracking for Stable Haptic Feedback in Microrobotics,” IEEE Trans.
Robot., vol. 28, no. 5, pp. 1081–1089, Oct. 2012.
[140] Y. H. Anis, J. K. Mills, and W. L. Cleghorn, “Vision-based measurement of
microassembly forces,” J. Micromechanics Microengineering, vol. 16, no. 8, pp. 1639–
1652, Aug. 2006.
[141] B. Tamadazte, N. L.-F. Piat, and S. Dembélé, “Robotic micromanipulation and
microassembly using monoview and multiscale visual servoing,” IEEE/ASME Trans.
Mechatronics, vol. 16, no. 2, pp. 277–287, Apr. 2011.
[142] T. Kasaya, H. T. Miyazaki, S. Saito, K. Koyano, T. Yamaura, and T. Sato, “Image-based
autonomous micromanipulation system for arrangement of spheres in a scanning electron
microscope,” Rev. Sci. Instrum., vol. 75, no. 6, pp. 2033–2042, 2004.
[143] H. Kuba, K. Hotta, and H. Takahashi, “Automatic particle detection and counting by one-
class SVM from microscope Image,” in Advances in Neuro-Information Processing, G.
Köppen, Mario and Kasabov, Nikola and Coghill, Ed. Springer Berlin / Heidelberg, 2009,
pp. 361–368.
[144] S. Yi, D. Labate, G. R. Easley, and H. Krim, “A shearlet approach to edge analysis and
115
detection.,” IEEE Trans. Image Process., vol. 18, no. 5, pp. 929–41, May 2009.
[145] G. Ligorio and A. M. Sabatini, “A Novel Kalman Filter for Human Motion Tracking With
an Inertial-Based Dynamic Inclinometer.,” IEEE Trans. Biomed. Eng., vol. 62, no. 8, pp.
2033–43, Aug. 2015.
[146] M. A. R. Ahad, J. K. Tan, H. Kim, and S. Ishikawa, “Motion history image: its variants
and applications,” Mach. Vis. Appl., vol. 23, no. 2, pp. 255–281, Oct. 2010.
[147] E. R. Davies, Machine Vision: Theory, Algorithms, Practicalities, 3rd ed. San Francisco,
US, 2004.
[148] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: active contour models,” Int. J. Comput.
Vis., vol. 1, no. 4, pp. 321–331, 1988.
[149] T. Yeo, S. Ong, and R. Sinniah, “Autofocusing for tissue microscopy,” Image Vis.
Comput., vol. 11, no. 10, pp. 629–639, Dec. 1993.
[150] Y. Sun, S. Duthaler, and B. J. Nelson, “Autofocusing in computer microscopy: selecting
the optimal focus algorithm.,” Microsc. Res. Tech., vol. 65, no. 3, pp. 139–49, Oct. 2004.
[151] A. Yilmaz, O. Javed, and M. Shah, “Object tracking,” ACM Comput. Surv., vol. 38, no. 4,
p. 13–es, Dec. 2006.
[152] Z. Yang, “Fast Template Matching Based on Normalized Cross Correlation with Centroid
Bounding,” 2010 Int. Conf. Meas. Technol. Mechatronics Autom., vol. 2, no. 1, pp. 224–
227, Mar. 2010.
[153] T. Pawson, “Protein modules and signalling networks.,” Nature, vol. 373, no. 6515, pp.
573–80, Feb. 1995.
[154] P. Friedl and K. Wolf, “Tumour-cell invasion and migration: diversity and escape
mechanisms.,” Nat. Rev. Cancer, vol. 3, no. 5, pp. 362–74, May 2003.
[155] I. Stefanová, J. R. Dorfman, and R. N. Germain, “Self-recognition promotes the foreign
antigen sensitivity of naive T lymphocytes.,” Nature, vol. 420, no. 6914, pp. 429–34, Nov.
116
2002.
[156] K. M. Abdullah, G. Luthra, J. J. Bilski, S. A. Abdullah, L. P. Reynolds, D. A. Redmer,
and A. T. Grazul-Bilska, “Cell-to-cell communication and expression of gap junctional
proteins in human diabetic and nondiabetic skin fibroblasts: effects of basic fibroblast
growth factor.,” Endocrine, vol. 10, no. 1, pp. 35–41, Feb. 1999.
[157] Z. Lu, X. Zhang, C. Leung, N. Esfandiari, R. F. Casper, and Y. Sun, “Robotic ICSI
(intracytoplasmic sperm injection).,” IEEE Trans. Biomed. Eng., vol. 58, no. 7, pp. 2102–
8, Jul. 2011.
[158] H. B. Huang, D. Sun, S. Member, J. K. Mills, and S. H. Cheng, “Robotic Cell Injection
System With Position and Force Control : Toward Automatic Batch Biomanipulation,” pp.
1–11, 2009.
[159] A. Pillarisetti, M. Pekarev, A. D. Brooks, and J. P. Desai, “Evaluating the effect of force
feedback in cell injection,” IEEE Trans. Autom. Sci. Eng., vol. 4, no. 3, pp. 322–331, Jul.
2007.
[160] K. Sakaki, N. Dechev, R. D. Burke, and E. J. Park, “Development of an autonomous
biological cell manipulator with single-cell electroporation and visual servoing
capabilities.,” IEEE Trans. Biomed. Eng., vol. 56, no. 8, pp. 2064–74, Aug. 2009.
[161] Q. Zhao, M. Wu, M. Cui, Y. Qin, J. Yu, M. Sun, X. Zhao, and X. Feng, “A novel
pneumatic micropipette aspiration method using a balance pressure model.,” Rev. Sci.
Instrum., vol. 84, no. 12, p. 123703, Dec. 2013.
[162] H. Ladjal, J.-L. Hanus, and A. Ferreira, “Micro-to-Nano Biomechanical Modeling for
Assisted Biological Cell Injection.,” IEEE Trans. Biomed. Eng., vol. 60, no. 9, pp. 2461–
71, Sep. 2013.
[163] H. Esmaeilsabzali, K. Sakaki, N. Dechev, R. D. Burke, and E. J. Park, “Machine vision-
based localization of nucleic and cytoplasmic injection sites on low-contrast adherent
cells.,” Med. Biol. Eng. Comput., vol. 50, no. 1, pp. 11–21, Jan. 2012.
[164] S. N. Lim, N. A. Zeenathul, M. L. Mohd Azmi, O. Abas Mazni, and O. Fauziah, “Effect
117
of protein concentration and injection pressure in microinjection delivery of maltose
binding protein into breast cancer cells,” Pertanika J. Sci. Technol., vol. 19, no. 2, pp.
273–283, 2011.
[165] W. Wang, Y. Sun, M. Zhang, R. Anderson, L. Langille, and W. Chan, “A system for high-
speed microinjection of adherent cells.,” Rev. Sci. Instrum., vol. 79, no. 10, p. 104302,
Oct. 2008.
[166] W. H. Wang, X. Y. Liu, and Y. Sun, “Contact Detection in Microrobotic Manipulation,”
Int. J. Robot. Res., vol. 26, no. 8, pp. 821–828, Aug. 2007.
[167] D. Ziou and F. Deschênes, “Depth from defocus estimation in spatial domain,” Comput.
Vis. Image Underst., vol. 81, no. 2, pp. 143–165, 2001.
[168] W. Martin, G. Zempel, D. Hülser, D. Hãlser, and K. Willecke, “Growth Inhibition of
Oncogene-transformed Rat Fibroblasts by Cocultured Normal Cells : Relevance of
Metabolic Cooperation Mediated by Gap Junctions Growth Inhibition of Oncogene-
transformed Rat Fibroblasts by Cocultured Normal Cells : Relevance of Metabol,” pp.
5348–5354, 1991.
[169] R. M. Mège, D. Goudou, C. Giaume, M. Nicolet, and F. Rieger, “Is intercellular
communication via gap junctions required for myoblast fusion?,” Cell Adhes. Commun.,
vol. 2, no. 4, pp. 329–343, Aug. 1994.
[170] T. Nishimura, C. Dunk, Y. Lu, X. Feng, a Gellhaus, E. Winterhager, J. Rossant, and S. J.
Lye, “Gap junctions are required for trophoblast proliferation in early human placental
development.,” Placenta, vol. 25, no. 7, pp. 595–607, Aug. 2004.
[171] S. Rohr, “Role of gap junctions in the propagation of the cardiac action potential.,”
Cardiovasc. Res., vol. 62, no. 2, pp. 309–22, May 2004.
[172] J. W. Holder, E. Elmore, and J. C. Barrett, “Gap junction function and cancer.,” Cancer
Res., vol. 53, no. 15, pp. 3475–85, Aug. 1993.
[173] T. K. Kim and J. H. Eberwine, “Mammalian cell transfection: the present and the future.,”
Anal. Bioanal. Chem., vol. 397, no. 8, pp. 3173–8, Aug. 2010.
118
[174] G. Vajta and M. Kuwayama, “Improving cryopreservation systems.,” Theriogenology,
vol. 65, no. 1, pp. 236–44, Jan. 2006.
[175] U.-B. Wennerholm, V. Söderström-Anttila, C. Bergh, K. Aittomäki, J. Hazekamp, K.-G.
Nygren, a Selbing, and a Loft, “Children born after cryopreservation of embryos or
oocytes: a systematic review of outcome data.,” Hum. Reprod., vol. 24, no. 9, pp. 2158–
72, Sep. 2009.
[176] K. E. Loutradi, E. M. Kolibianakis, C. a Venetis, E. G. Papanikolaou, G. Pados, I. Bontis,
and B. C. Tarlatzis, “Cryopreservation of human embryos by vitrification or slow
freezing: a systematic review and meta-analysis.,” Fertil. Steril., vol. 90, no. 1, pp. 186–
93, Jul. 2008.
[177] W. H. Tsang and K. L. Chow, “Cryopreservation of mammalian embryos: Advancement
of putting life on hold.,” Birth defects Res. Part C Embryo today Rev., vol. 90, pp. 163–
175, 2010.
[178] M. Kasai, J. H. Komi, A. Takakamo, H. Tsudera, T. Sakurai, and T. Machida, “A simple
method for mouse embryo cryopreservation in a low toxicity vitrification solution, without
appreciable loss of viability.,” J. Reprod. Fertil., vol. 89, pp. 91–97, 1990.
[179] M. Kuwayama, “Highly efficient vitrification for cryopreservation of human oocytes and
embryos: the Cryotop method.,” Theriogenology, vol. 67, no. 1, pp. 73–80, Jan. 2007.
[180] G. T. Sung and I. S. Gill, “Robotic laparoscopic surgery: a comparison of the da Vinci and
Zeus systems,” Urology, vol. 58, no. 6, pp. 893–898, Dec. 2001.
[181] H. Matsuoka, T. Komazaki, Y. Mukai, M. Shibusawa, H. Akane, A. Chaki, N. Uetake,
and M. Saito, “High throughput easy microinjection with a single-cell manipulation
supporting robot,” J. Biotechnol., vol. 116, no. 2, pp. 185–194, Mar. 2005.
[182] Y. S. Heo, H.-J. Lee, B. a Hassell, D. Irimia, T. L. Toth, H. Elmoazzen, and M. Toner,
“Controlled loading of cryoprotectants (CPAs) to oocyte with linear and complex CPA
profiles on a microfluidic platform.,” Lab Chip, vol. 11, no. 20, pp. 3530–7, Oct. 2011.
[183] L. Meng, X. Huezo, B. a. Stone, K. Baek, G. Ringler, and R. P. Marrs, “Development of a
119
microfluidic device for automated vitrification human embryo,” Fertil. Steril., vol. 96, no.
3, p. S207, Sep. 2011.
[184] X. Zhang, C. Leung, Z. Lu, N. Esfandiari, R. F. Casper, and Y. Sun, “Controlled
aspiration and positioning of biological cells in a micropipette.,” IEEE Trans. Biomed.
Eng., vol. 59, no. 4, pp. 1032–40, Apr. 2012.
[185] M. H. Johnson and S. J. Pickering, “The effect of dimethylsulphoxide on the microtubular
system of the mouse oocyte.,” Development, vol. 100, no. 2, pp. 313–24, Jun. 1987.
[186] W. V Holt and K. J. W. Van Look, “Concepts in sperm heterogeneity, sperm selection and
sperm competition as biological foundations for laboratory tests of semen quality.,”
Reproduction, vol. 127, no. 5, pp. 527–35, May 2004.
[187] C. H. Ramlau-Hansen, a M. Thulstrup, J. Olsen, and J. P. Bonde, “Parental subfecundity
and risk of decreased semen quality in the male offspring: a follow-up study.,” Am. J.
Epidemiol., vol. 167, no. 12, pp. 1458–64, Jun. 2008.
[188] B. Bartoov, F. Eltes, M. Pansky, H. Lederman, E. Caspi, and Y. Soffer, “Estimating
fertility potential via semen analysis data.,” Hum. Reprod., vol. 8, no. 1, pp. 65–70, Jan.
1993.
[189] A. Berkovitz, F. Eltes, S. Yaari, N. Katz, I. Barr, A. Fishman, and B. Bartoov, “The
morphological normalcy of the sperm nucleus and pregnancy rate of intracytoplasmic
injection with morphologically selected sperm.,” Hum. Reprod., vol. 20, no. 1, pp. 185–
90, Jan. 2005.
[190] L. Larsen, T. Scheike, T. K. Jensen, J. P. Bonde, E. Ernst, N. H. Hjollund, Y. Zhou, N. E.
Skakkebaek, and A. Giwercman, “Computer-assisted semen analysis parameters as
predictors for fertility of men from the general population. The Danish First Pregnancy
Planner Study Team.,” Hum. Reprod., vol. 15, no. 7, pp. 1562–7, Jul. 2000.
[191] C. G. Petersen, F. C. Massaro, A. L. Mauri, J. B. a Oliveira, R. L. R. Baruffi, and J. G.
Franco, “Efficacy of hyaluronic acid binding assay in selecting motile spermatozoa with
normal morphology at high magnification.,” Reprod. Biol. Endocrinol., vol. 8, no. 1, p.
120
149, Jan. 2010.
[192] C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, “Pfinder: real-time tracking of the
human body,” in Proceedings of the Second International Conference on Automatic Face
and Gesture Recognition, pp. 51–56.
[193] Jinman Kang, I. Cohen, and G. Medioni, “Object reacquisition using invariant appearance
model,” in Proceedings of the 17th International Conference on Pattern Recognition,
2004. ICPR 2004., 2004, pp. 759–762 Vol.4.
[194] C. Sminchisescu and B. Triggs, “Kinematic jump processes for monocular 3D human
tracking,” in 2003 IEEE Computer Society Conference on Computer Vision and Pattern
Recognition, 2003. Proceedings., pp. I–69–I–76.
[195] K. M. G. Cheung, S. Baker, and T. Kanade, “Shape-from-silhouette of articulated objects
and its use for human body kinematics estimation and motion capture,” in 2003 IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, 2003.
Proceedings., pp. I–77–I–84.
[196] Hai Tao, H. S. Sawhney, and R. Kumar, “Object tracking with Bayesian estimation of
dynamic layer representations,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 1,
pp. 75–89, 2002.
[197] M. J. Black and A. D. Jepson, “EigenTracking: Robust Matching and Tracking of
Articulated Objects Using a View-Based Representation,” Int. J. Comput. Vis., vol. 26, no.
1, pp. 63–84, 1998.
[198] D. Comaniciu, S. Member, and V. Ramesh, “Kernel-Based Object Tracking,” vol. 25, no.
5, pp. 564–577, 2003.
[199] C. J. Veenman, M. J. T. Reinders, and E. Backer, “Resolving motion correspondence for
densely moving points,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 1, pp. 54–
72, 2001.
[200] D. Beymer and K. Konolige, “Tracking People from a Mobile Platform,” Exp. Robot. VIII,
vol. 5, pp. 234–244, 2003.
121
[201] M. Isard and A. Blake, “Condensation - conditional density propagation for visual
tracking,” Int. J. Comput. Vis., vol. 29, no. 1, pp. 5–28, 1998.
[202] P. Konstantinova, A. Udvarev, and T. Semerdjiev, “A study of a target tracking algorithm
using global nearest neighbor approach,” in Proceedings of the 4th international
conference conference on Computer systems and technologies e-Learning - CompSysTech
’03, 2003, pp. 290–295.
[203] H. A. P. Blom and E. A. Bloem, “Probabilistic data association avoiding track
coalescence,” IEEE Trans. Automat. Contr., vol. 45, no. 2, pp. 247–259, 2000.
[204] L. Z. Shi, J. M. Nascimento, C. Chandsawangbhuwana, E. L. Botvinick, and M. W. Berns,
“An automatic system to study sperm motility and energetics.,” Biomed. Microdevices,
vol. 10, no. 4, pp. 573–83, Aug. 2008.
[205] V. R. Nafisi, M. H. Moradi, and M. H. Nasr-esfahani, “Sperm Identification Using Elliptic
Model and Tail Detection,” World Acad. Sci. Eng. Technol., vol. 6, pp. 205–208, 2005.
[206] C. Leung, Z. Lu, N. Esfandiari, R. F. Casper, and Y. Sun, “Automated sperm
immobilization for intracytoplasmic sperm injection.,” IEEE Trans. Biomed. Eng., vol. 58,
no. 4, pp. 935–42, Apr. 2011.
[207] K. Smith, D. Gatica-Perez, and J. Odobez, “Evaluating Multi-Object Tracking,” in 2005
IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR’05) - Workshops, 2005, vol. 3, no. c, pp. 36–36.
[208] B. B. Lowell and B. M. Spiegelman, “Towards a molecular understanding of adaptive
thermogenesis.,” Nature, vol. 404, no. 6778, pp. 652–60, Apr. 2000.
[209] D. Patel and K. A. Franklin, “Temperature-regulation of plant architecture.,” Plant Signal.
Behav., vol. 4, no. 7, pp. 577–9, Jul. 2009.
[210] H. Jiang and S. X. Sun, “Cellular pressure and volume regulation and implications for cell
mechanics.,” Biophys. J., vol. 105, no. 3, pp. 609–19, Aug. 2013.
[211] P. J. Ehrlich and L. E. Lanyon, “Mechanical strain and bone cell function: a review.,”
122
Osteoporos. Int., vol. 13, no. 9, pp. 688–700, Sep. 2002.
[212] D.-H. Kim, P. K. Wong, J. Park, A. Levchenko, and Y. Sun, “Microengineered platforms
for cell mechanobiology.,” Annu. Rev. Biomed. Eng., vol. 11, pp. 203–33, Jan. 2009.
[213] W. Moody, “Effects of intracellular H+ on the electrical properties of excitable cells.,”
Annu. Rev. Neurosci., vol. 7, pp. 257–78, Jan. 1984.
[214] W. J. Waddell and R. G. Bates, “Intracellular pH.,” Physiol. Rev., vol. 49, no. 2, pp. 285–
329, Apr. 1969.
[215] I. A. Silver, J. Deas, and M. Erecińska, “Ion homeostasis in brain cells: differences in
intracellular ion responses to energy limitation between cultured neurons and glial cells.,”
Neuroscience, vol. 78, no. 2, pp. 589–601, May 1997.
[216] T. Kinashi, “Intracellular signalling controlling integrin activation in lymphocytes.,” Nat.
Rev. Immunol., vol. 5, no. 7, pp. 546–59, Jul. 2005.
[217] R. C. Thomas, “Intracellular pH of snail neurones measured with a new pH-sensitive glass
mirco-electrode.,” J. Physiol., vol. 238, no. 1, pp. 159–80, Apr. 1974.
[218] S. M. Kelly and P. T. Macklem, “Direct measurement of intracellular pressure.,” Am. J.
Physiol., vol. 260, no. 3 Pt 1, pp. C652–7, Mar. 1991.
[219] W. R. Loewenstein and Y. Kanno, “Some electrical properties of the membrane of a cell
nucleus.,” Nature, vol. 195, pp. 462–4, Aug. 1962.
[220] K. Okabe, N. Inada, C. Gota, Y. Harada, T. Funatsu, and S. Uchiyama, “Intracellular
temperature mapping with a fluorescent polymeric thermometer and fluorescence lifetime
imaging microscopy.,” Nat. Commun., vol. 3, p. 705, Jan. 2012.
[221] B. Tian, T. Cohen-Karni, Q. Qing, X. Duan, P. Xie, and C. M. Lieber, “Three-
dimensional, flexible nanoscale field-effect transistors as localized bioprobes.,” Science
(80-. )., vol. 329, no. 5993, pp. 830–4, Aug. 2010.
[222] M. G. Schrlau, E. M. Falls, B. L. Ziober, and H. H. Bau, “Carbon nanopipettes for cell
123
probes and intracellular injection.,” Nanotechnology, vol. 19, no. 1, p. 015101, Jan. 2008.
[223] M. G. Schrlau, N. J. Dun, and H. H. Bau, “Cell electrophysiology with carbon
nanopipettes.,” ACS Nano, vol. 3, no. 3, pp. 563–8, Mar. 2009.
[224] H. Liu, J. Wen, Y. Xiao, J. Liu, S. Hopyan, M. Radisic, C. A. Simmons, and Y. Sun, “In
situ mechanical characterization of the cell nucleus by atomic force microscopy.,” ACS
Nano, vol. 8, no. 4, pp. 3821–8, Apr. 2014.
[225] R. Singhal, Z. Orynbayeva, R. V. Kalyana Sundaram, J. J. Niu, S. Bhattacharyya, E. a
Vitol, M. G. Schrlau, E. S. Papazoglou, G. Friedman, and Y. Gogotsi, “Multifunctional
carbon-nanotube cellular endoscopes.,” Nat. Nanotechnol., vol. 6, no. 1, pp. 57–64, Jan.
2011.
[226] R. Yan, J.-H. Park, Y. Choi, C.-J. Heo, S.-M. Yang, L. P. Lee, and P. Yang, “Nanowire-
based single-cell endoscopy.,” Nat. Nanotechnol., vol. 7, no. 3, pp. 191–6, Mar. 2012.
[227] A. E. Rossi and R. T. Dirksen, “Sarcoplasmic reticulum: the dynamic calcium governor of
muscle.,” Muscle Nerve, vol. 33, no. 6, pp. 715–31, Jun. 2006.
[228] M. Micaroni, “Calcium around the Golgi apparatus: implications for intracellular
membrane trafficking.,” Adv. Exp. Med. Biol., vol. 740, pp. 439–60, Jan. 2012.
[229] A. K. Geim and K. S. Novoselov, “The rise of graphene.,” Nat. Mater., vol. 6, no. 3, pp.
183–91, Mar. 2007.
[230] L. H. Hess, M. Jansen, V. Maybeck, M. V Hauf, M. Seifert, M. Stutzmann, I. D. Sharp, A.
Offenhäusser, and J. a Garrido, “Graphene transistor arrays for recording action potentials
from electrogenic cells.,” Adv. Mater., vol. 23, no. 43, pp. 5045–9, 4968, Nov. 2011.
[231] T. Cohen-Karni, Q. Qing, Q. Li, Y. Fang, and C. M. Lieber, “Graphene and nanowire
transistors for cellular interfaces and electrical recording.,” Nano Lett., vol. 10, no. 3, pp.
1098–102, Mar. 2010.
[232] J. Liu, Z. Gong, K. Tang, Z. Lu, C. Ru, J. Luo, S. Xie, and Y. Sun, “Locating End-
Effector Tips in Robotic Micromanipulation,” IEEE Trans. Robot., vol. 30, no. 1, pp. 125–
124
130, Feb. 2014.
[233] J. Liu, V. Siragam, J. Chen, M. D. Fridman, R. M. Hamilton, and Y. Sun, “High-
throughput measurement of gap junctional intercellular communication.,” Am. J. Physiol.
Heart Circ. Physiol., vol. 306, no. 12, pp. H1708–13, Jun. 2014.
[234] J. Liu, V. Siragam, Z. Gong, J. Chen, M. D. Fridman, C. Leung, Z. Lu, C. Ru, S. Xie, J.
Luo, R. M. Hamilton, Y. Sun, and S. S. Member, “Robotic adherent cell injection for
characterizing cell-cell communication.,” IEEE Trans. Biomed. Eng., vol. 62, no. 1, pp.
119–25, Jan. 2015.
[235] M. Hagiwara, T. Kawahara, Y. Yamanishi, T. Masuda, L. Feng, and F. Arai, “On-chip
magnetically actuated robot with ultrasonic vibration for single cell manipulations.,” Lab
Chip, vol. 11, no. 12, pp. 2049–54, Jun. 2011.
[236] Y. Tan, D. Sun, J. Wang, and W. Huang, “Mechanical characterization of human red
blood cells under different osmotic conditions by robotic manipulation with optical
tweezers,” IEEE Trans. Biomed. Eng., vol. 57, pp. 1816–1825, 2010.