Eindhoven University of Technology MASTER Simulation of ... · Simulation of drift control for an electron microscope stage using image processing in the loop Feng, F. Award date:

Eindhoven University of Technology

MASTER

Simulation of drift control for an electron microscope stage using image processing in the loop

Feng, F.

Award date:2013

Link to publication

DisclaimerThis document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Studenttheses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the documentas presented in the repository. The required complexity or quality of research of student theses may vary by program, and the requiredminimum study period may vary in duration.

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

https://research.tue.nl/en/studentthesis/simulation-of-drift-control-for-an-electron-microscope-stage-using-image-processing-in-the-loop(28351e65-86f3-4955-af7a-ec5a8c1ee62d).html

Technische Universiteit Eindhoven

Master Thesis

Department of Mathematics and Computer Science

Simulation of Drift Control for an Electron MicroscopeStage Using Image Processing in the Loop

Author:Fan Feng(0786187)

Supervisor:Dr. ir. Pieter Cuijpers (TU/e)Dr. ir. Jeroen de Boeij (FEI)

August 19, 2013

Contents

1 Introduction 11.1 Overview of Electron Microscope . . . . . . . . . . . . . . . . . 1

1.1.1 Mechanics of Electron Microscopy . . . . . . . . . . . . . . 21.1.2 Imaging Acquisition Principles . . . . . . . . . . . . . . . 2

1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . 31.2.1 Disturbances Summary . . . . . . . . . . . . . . . . . 31.2.2 Drift Influence on image quality . . . . . . . . . . . . . . . 4

1.3 Literature Review . . . . . . . . . . . . . . . . . . . . . . 51.3.1 Passive Isolation . . . . . . . . . . . . . . . . . . . . 51.3.2 Post Processing . . . . . . . . . . . . . . . . . . . . 61.3.3 Motion Compensation . . . . . . . . . . . . . . . . . . 6

1.4 Contribution . . . . . . . . . . . . . . . . . . . . . . . . 61.5 Outline of thesis . . . . . . . . . . . . . . . . . . . . . . 6

2 Hardware Architecture Overview 82.1 Hierarchical architecture . . . . . . . . . . . . . . . . . . . . 82.2 System Work flow . . . . . . . . . . . . . . . . . . . . . . 11

2.2.1 Optics Configuration . . . . . . . . . . . . . . . . . . 122.2.2 Viewing Navigation Process . . . . . . . . . . . . . . . . 122.2.3 Image Acquisition Process . . . . . . . . . . . . . . . . 132.2.4 Image Processing . . . . . . . . . . . . . . . . . . . 152.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . 15

2.3 Architecture Proposal . . . . . . . . . . . . . . . . . . . . 162.3.1 Architecture Stress . . . . . . . . . . . . . . . . . . . 162.3.2 New Architecture Proposal . . . . . . . . . . . . . . . . 17

2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 18

3 Proposed Method 193.1 Image Acquisition and Drift Measurement . . . . . . . . . . . . . . 19

3.1.1 SEM . . . . . . . . . . . . . . . . . . . . . . . . 193.1.2 TEM . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2 Motion Prediction Controller . . . . . . . . . . . . . . . . . . 243.3 Actuator . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.3.1 Piezo Actuator . . . . . . . . . . . . . . . . . . . . 273.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 283.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 29

4 Model Implementation 304.1 Image Acquisition . . . . . . . . . . . . . . . . . . . . . . 314.2 Image Deformation . . . . . . . . . . . . . . . . . . . . . 324.3 Image Processing . . . . . . . . . . . . . . . . . . . . . . 344.4 Motion Prediction . . . . . . . . . . . . . . . . . . . . . . 344.5 Interconnections . . . . . . . . . . . . . . . . . . . . . . 364.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 37

5 Simulation and Evaluation 385.1 Metrics for Evaluation . . . . . . . . . . . . . . . . . . . . 38

5.1.1 Closed-loop image versus Reference image . . . . . . . . . . . 385.1.2 Drift versus Stage Movement . . . . . . . . . . . . . . . . 38

5.2 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . 395.2.1 Simulation Environment . . . . . . . . . . . . . . . . . 395.2.2 Experiment Configuration . . . . . . . . . . . . . . . . . 39

i

5.3 Correctness Evaluation . . . . . . . . . . . . . . . . . . . . 40

5.3.1 TEM Simulation Result . . . . . . . . . . . . . . . . . . 405.3.2 SEM Simulation Result . . . . . . . . . . . . . . . . . . 42

5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 47

6 Conclusion 486.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 486.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . 496.3 Final words . . . . . . . . . . . . . . . . . . . . . . . . 50

ii

List of Figures

1.1 System schematic of TEM and SEM . . . . . . . . . . . . . . 31.2 Drift distortion . . . . . . . . . . . . . . . . . . . . 51.3 Drift blur e�ect . . . . . . . . . . . . . . . . . . . . 5

2.1 Block diagram . . . . . . . . . . . . . . . . . . . . 92.2 Microscope column . . . . . . . . . . . . . . . . . . . 102.3 Accessory Cabinet . . . . . . . . . . . . . . . . . . . 112.4 Optics Cabinet . . . . . . . . . . . . . . . . . . . . 112.5 Optics Con�guration process . . . . . . . . . . . . . . . . 122.6 Viewing Navigation Process . . . . . . . . . . . . . . . . 132.7 Imaging Acquisition Process . . . . . . . . . . . . . . . . 142.8 Communication diagram . . . . . . . . . . . . . . . . . 162.9 Proposed architecure . . . . . . . . . . . . . . . . . . . 172.10 Drift Compensation Diagram: arrow indicates the excuation sequence for each block,

N indicates the frame index, its incremental by 1 in each iteration. . . . . . 18

3.1 Schematic diagram of closed-loop system: . . . . . . . . . . . . 193.2 Raster scan . . . . . . . . . . . . . . . . . . . . . 203.3 Drift e�ect in raster scan . . . . . . . . . . . . . . . . . 213.4 SEM sensor data acquisition rate . . . . . . . . . . . . . . . 223.5 Drift e�ecet in TEM: . . . . . . . . . . . . . . . . . . 233.6 TEM sensor data acquisition rate . . . . . . . . . . . . . . 233.7 SEM controller . . . . . . . . . . . . . . . . . . . . 253.8 Operation rate of SEM controller in closed loop . . . . . . . . . . 253.9 TEM controller . . . . . . . . . . . . . . . . . . . . 263.10 Operation rate of TEM controller in closed lop . . . . . . . . . . . 263.11 Closed loop system of Piezo actuator: . . . . . . . . . . . . . 273.12 Bode plot of approximate piezo actuator closed-loop system . . . . . . . 273.13 Operation rate of piezo actuator in SEM closed loop . . . . . . . . . 283.14 Operation rate of piezo actuator in TEM closed loop . . . . . . . . . 28

4.1 SEM simulator . . . . . . . . . . . . . . . . . . . . 314.2 TEM simulator . . . . . . . . . . . . . . . . . . . . 314.3 Model of PIA . . . . . . . . . . . . . . . . . . . . . 324.4 SEM image acquisition . . . . . . . . . . . . . . . . . . 334.5 SEM image deformation . . . . . . . . . . . . . . . . . . 334.6 TEM image deformation . . . . . . . . . . . . . . . . . 344.7 SEM motion prediction . . . . . . . . . . . . . . . . . . 354.8 TEM motion prediction . . . . . . . . . . . . . . . . . . 354.9 SEM simulator interconncetion . . . . . . . . . . . . . . . 364.10 TEM simulator connection . . . . . . . . . . . . . . . . . 36

iii

LIST OF FIGURES

5.1 TEM linear drift result . . . . . . . . . . . . . . . . . . 405.2 Di�erence image between aligned image and reference image, non-algned image and

reference image . . . . . . . . . . . . . . . . . . . . 415.3 Linear drift TEM . . . . . . . . . . . . . . . . . . . . 415.4 Linear drift: blur image . . . . . . . . . . . . . . . . . . 425.5 SEM step response . . . . . . . . . . . . . . . . . . . 435.6 SEM step response:fast scan . . . . . . . . . . . . . . . . 435.7 SEM step response: slow scan . . . . . . . . . . . . . . . . 435.8 Overshoot redution in fst scan . . . . . . . . . . . . . . . . 445.9 SEM linear drift result . . . . . . . . . . . . . . . . . . 445.10 SEM linear drift result . . . . . . . . . . . . . . . . . . 455.11 Sensitivity plot: fast scan with a dwell time of 5e-7 second . . . . . . . 465.12 Sensitivity plot: slow scan with a dwell time of 5e-6 second . . . . . . . 46

6.1 Proposed new closed-loop schematic . . . . . . . . . . . . . . 506.2 Optics Con�guration process usecase1:Magni�cation settings . . . . . . . 526.3 Optics Con�guration process usecase2:Focus settings . . . . . . . . . 526.4 Viewing Navigation Process usecase1:Large scale stage movement . . . . . 536.5 Viewing Navigation Process usecase2:small scale stage movement . . . . . 536.6 Viewing Navigation Process usecase3:Electron beam de�ection . . . . . . 536.7 Imaging Acquisition Process usecase1:TEM mode . . . . . . . . . . 546.8 Imaging Acquisition Process usecase1:SEM mode . . . . . . . . . . 546.9 SEM simulator . . . . . . . . . . . . . . . . . . . . 556.10 TEM simulator . . . . . . . . . . . . . . . . . . . . 56

iv

List of Tables

1.1 Comparison between TEM and SEM . . . . . . . . . . . . . . 41.2 Disturbances sheet . . . . . . . . . . . . . . . . . . . 4

5.1 TEM experiment con�gurations for linear drift . . . . . . . . . . . 395.2 step response test for SEM . . . . . . . . . . . . . . . . . 395.3 Linear speed drift for SEM . . . . . . . . . . . . . . . . . 395.4 Experiment for rejection ratio test . . . . . . . . . . . . . . 405.5 Result for image comparision . . . . . . . . . . . . . . . . 405.6 Result for averaging image comparision . . . . . . . . . . . . . 425.7 Result for image comparision in slow scan . . . . . . . . . . . . 455.8 Result for image comparision in fast scan . . . . . . . . . . . . 45

v

Abstract

The main objective in this research is to develop a primary model for simulating the impact ofdrift during image acquisition process and to propose a real time closed-control loop scheme forcompensating the e�ect of drift. Image acquisition process in electron microscopy is vulnerable tointernal and external disturbances, which in turn produces an arti�cial image. Among the uncer-tainties, external noises can be reduced by placing the instrument in a well-controlled environment.Nevertheless, internal disturbances that induced by temperature variation and beam-specimen in-teraction cannot be compensated automatically. Such kind of disturbances are referred to drift, andtheir non-linear and slowly movement will lead to a blur or distorted image under di�erent imagingmodes. Our work is mainly about solving the problem illustrated above. In the �rst place, hardwarearchitecture has been printed out at the level of system engineering. The current system architectureis evaluated and formalized with di�erent point of views. Among them, the main contribution is toinvestigate any possible innovation points in the current system . Secondly, at modelling level, thesimulation model has been built to validate our proposed methodology which is the most signi�cantof research in this thesis. The overall model which contains important functionalities has been in-volved in image acquisition process. And then, the performance of the scheme has been evaluatedaccording to the simulation results. After that, the future directions will be proposed in the end.

Keywords: Drift compensation Simulation Closed-loop

vi

Acknowledgement

I would like to express my sincere gratitude to my advisor dr.ir. P.J.L. (Pieter) Cuijpers, who hassupported me throughout my thesis with great patience and immense knowledge. His stimulatingsuggestions and encouragement helped me to coordinate my project especially in writing this thesis.Furthermore, I would like to acknowledge with much appreciation the important role of the FEIcompany engineers, Jeroen de boeij, Janssen Bart and Kees Kooijman who give me the extraordinarysuggestions through out the whole project. I would also like to thank all the working sta�s in FEIcompany for your great supporting and helpful guidance. Last but not the least, I give best thankto my families and my girl friend for their dedication, love and persistent con�dence in me.

vii

Chapter 1

Introduction

The background of our research is mainly about electron microscope which has been widely utilizedin the area of civil and military application. The objective is to develop a primary model for simu-lating the impact of drift during image acquisition process and to propose a real time closed-controlloop scheme for compensating the e�ect of drift. In this chapter, the basic knowledge of electronmicroscope has been presented in the �rst place. It is intended to give a high level description ofthe di�erent electron microscopy types which serves as the foundations of the overall research. Afterthat, the problem statement is focused on thermal drift, together with the proposed solution inliterature. At the end, the organization of the whole thesis is presented.

1.1 Overview of Electron Microscope

An Electron microscopy (EM) is a type of microscope that is designed based on the principle ofelectron optics. An EM uses electron beam and electron lens instead of light and glass lens toilluminate the specimen, and generates magni�ed image. The fundamental di�erence between elec-tron microscopy and optics microscopy lies in the wavelength of the source of illumination. It isknown that the wavelength of the electrons is 100,000 times shorter than visible light. As a result,electron microscopy has a resolution of 0.5 nm which is more than 4000 times better than a opticalmicroscope. Such an achievement in resolution enables researcher to examine the �ne structure ofmaterial at a quite high magni�cation.Electron Microscopy is widely used in various areas around world, such as life science, electronics,industry and scienti�c research. Researchers and engineers use electron microscopy to explore theinternal infrastructures of the samples and detect defects during manufacturing process. Recent�nds suggests that modern electron microscopy can be used not only visualize a specimen in nanoscale, but also move it, position it and even disassemble and recompose it.The electron microscopy can be classi�ed into two basic types: transmission electron microscopy,and scanning electron microscopy [2]. As the name implies, TEM generates high accelerating voltageelectrons which pass through the ultra-thin samples and image is acquired due to the interactionbetween electrons and specimen. In reality, not all samples can be made thin enough for penetration;alternatively, scanning electron microscopy can be adopted for surface analysis. Scanning electronmicroscopy (SEM) generates the image by scanning surface of a specimen using a focused electronbeam. The electron beam scanning in a raster pattern through the surface of the sample. TheSEM can achieve a resolution up to pico meter scale. Some other electron microscopy types includescanning transmission electron microscopy (STEM) ,which is a microscopy technique combining theprinciple of TEM and SEM, and focused ion beam and dual beam microscopy ,which replace theelectron source by a beam of ion. In this thesis, the proposed strategy and model are based on theprinciple of TEM and SEM. Following section, we will discuss the mechanism of TEM and SEM in

1

CHAPTER 1. INTRODUCTION

detail.

1.1.1 Mechanics of Electron Microscopy

• Transmission Electron Microscopy ( TEM)

There are four primary sections in TEM: an electron source, an electronic magnetic lens system,and sample holder and an imaging system. The schematic of TEM is shown in �gure 1.1 .

i. Electron source: The electron source is an important part in a microscopy system. Thebrightness of the source dominates the intensity of electron beam, and it will have greatimpact on the resolution, contrast and signal to noise capabilities of the imaging system. Theelectronic gun at the top of the system is designed to create electrons and then accelerate themin a beam towards the specimen. Three main types of electron source are used in electronmicroscopy: tungsten, lanthanum hex boride (LaB6), and �eld emission gun (FEG). Each ofthem is a combination of bene�ts and cost.ii. Electron magnetic lens system: The electron magnetic lens system consists of a series ofelectromagnetic lenses and apertures. These components are used to focus and de�ect theelectron beam. The condensers lens are designed to control the diameter of the beam as wellas focus the beam spot at the specimen. The apertures are used to control the intensity ofbeam by �ltering out unwanted beam before hitting the specimen.iii. Sample holder: The sample holder includes a mechanical arm for holding the specimenand controlling its position in three dimensions, an air lock is equipped to allow inserting thesamples without pulling in additional pressure in the microscope column. To manipulate thespecimen into the �eld of view, it is often required to position the specimen in x, y and zdirection and tilt along one direction. A mechanical motor is used to achieve the specimenmanipulation in micro meter scale.iv. Imaging system: The imaging system includes series of electromagnetic projection lensesand a screen. The projection lens is used to refocus the electron beam which emerges from thespecimen, enlarge and project the image to screen. TEM uses a �uorescent screen which emitslight on impact of the transmitted electrons. A CCD or CMOS camera is used to capture theimage in a way similar to photography.

• Scanning electron microscopy (SEM)

Similar to TEM, there are also four main parts in SEM: an electron source, an electron magneticlens system, a detector system and scan system. The schematic of SEM is show in �gure 1.1.

i. Electron source: The electron source is same as in TEM. Main di�erence is that the accel-erating voltage is smaller than in TEM.ii. Electron magnetic lens system: Also similar to TEM.iii. Detector system: The detector system is designed to collect the beam electrons created byinteraction with specimen surface.iv. Scan system: The scan system consists of electromagnetic scan coils. The scan coils areused to position the electron beam on the specimen. These scan coils de�ect the beam before itpasses to the objective lens. In SEM, the image is produced by scanning the specimen surfaceline by line over a certain area.s

1.1.2 Imaging Acquisition Principles

We distinguish three elements present in an image acquisition process: a sample, a radiation sourceand an image acquisition system. The di�erent Imaging acquisition modes depend on the energyof radiation source, physics of electrons-specimen interaction and image acquisition system. We

2


Figure 1.1: System schematic of TEM and SEM. Right hand side is the SEM and left hand side isTEM.

will give a brief introduction on the imaging principle of TEM and SEM, the table 1.1 provide acomparison of TEM and SEM.

• TEM

In TEM, the light source is replaced by electron gun which emits a beam of electrons withhigh accelerating voltage. After emerging from electron gun, the beam is condensed intoparallel before hitting the specimen. The beam will be focused and magni�ed through a set ofelectromagnetic coils, then it will penetrate the ultra-thin samples and �nally the transmittedbeam is refocused and magni�ed to the �uent screen.

The TEM image acquisition is the result of the scattering, di�raction and absorption of theelectron beam as they pass through the specimen. Di�erent region of the specimen scatter thebeam by numerous degrees where the extent depends on the thickness and local compositionand density of the specimen.

• SEM

In SEM, the electron gun produces an electron beam that is focused into a �ne spot over thespecimen surface. At each point, the spot dwells for a certain period of time during which theelectrons of beam interact with the specimen surface. The focused spot scans the specimensurface in a raster pattern and simultaneously registration the signals. The collected signalsare ampli�ed and modulated to the intensity value , and �nally a SEM image is formed.

1.2 Problem Statement

1.2.1 Disturbances Summary

Modern electron microscopy is used to explore and evaluate the internal infrastructure of the samplesdown to atomic level. At high magni�cation, the imaging process is vulnerable to interfere fromvarious types of disturbances. The external disturbances are caused by the outside surroundings,such as �oor vibrations and acoustic environment. Such kind of disturbances usually stimulateunpredictable movements of stage or samples and the e�ects are presented in acquired image. The

3


Table 1.1: Comparison between TEM and SEM

Types TEM SEM

Electron Beam Broad, static beams Beam focused to �nepoint

Voltage needs High voltage 60,000-300,000 Low Voltage 400-30,000

Sample preparation Specimen must be very thin Wide range of specimens al-lowed

Interaction of beam-specimen Electrons must pass throughand be transmitted by thespecimen

Information needed is col-lected near the surface of thespecimen

Imaging Transmitted electrons are fo-cused by the objective lensand magni�ed to create a realimage

Beam is scanned along thesurface of the sample to buildup the image

internal disturbances are induced inside the column, such as temperature variation and specimencharging. All disturbances will contribute to the misalignment between where we expect to acquireand the actual position of the electron beam, hence, the image is deformed. The 1.2 lists sometypical disturbances properties and e�ects.

Table 1.2: Disturbances sheet

Type Inducer FrequencyProperty

E�ect

Stick-Slip Friction < 100 Hz Degrade the smoothness ofstage movement

Acoustic environment Unwanted sound 10 Hz-10KHz

Exciting stage resonance

Floor Vibration Walking, talking around col-umn

0.1 Hz- 10Hz

�aggy characters shown in im-age

Thermal Drift Heating or cooling specimenresult in temperature varia-tion. Internal column temper-ature variation

0.001 Hz -0.01 Hz

Integrating miss-positionedpoint during image scan-ning. Image distortion andblurred(averaging imageframes)

For modern electron microscopy product, the external uncertainties can be dealt with by placingthe instrument in a well-conditioned environment. Internal disturbances like specimen charging canbe avoided by appropriate preparation techniques. But for thermal drift, the e�ect will increasewith time, and cannot be compensated automatically. For that purpose, we will focused ourselveson the drift compensation in the rest of thesis.

1.2.2 Drift Influence on image quality

Drift is a slow, non-linear and continuous movement of the specimen. The thermal drift appearswhen inserts the specimen into electron microscopy column, and heating or cooling the specimen.

4


Since it is not practical to create a perfect environment for electron microscopy, the drift e�ect isunpreventable. As a result, images that acquired under drift e�ect will be deformed. We will explainthe image defects for TEM and SEM respectively in the following section.

Drift in�uence in Scanning modeDrift is a time dependent procedure. In�uenced by thermal drift, the specimen starts move withrespect to optical column and the position of the spot-probe will change, resulting in a displacementof �eld of view. With the impact of drift, there is no longer one-to-one correspondence between scan-ning pattern and image. In another words, the spot probe integrates mis-positioned point whichleads to image distortion.The distortion is shown in the �gure below:

Figure 1.2: Drift distortion. Image on the left side is the normal image without distorion, image onthe right side is distoed in horizontal direction.

Drift in�uence in Transmission modeDue to a completely di�erent imaging mechanism, the drift presents di�erent e�ects in TEM. Com-pared to SEM, a TEM image will never distorted but only blur. At low magni�cation, this results ina slow movement of the content in the image stream. At high magni�cation, the drift shift becomespronounced and leads to image blurring. The blurring a�ects is shown in the �gure 1.3

Figure 1.3: Drift blur e�ect. Image on the left side is the normal image without blur, image on theright side is the blurring image which is taken by averaging 128 frames which are acquired in a TVrate (1 seconds per frame).

1.3 Literature Review

In literature, several methods have been proposed to eliminate the drift e�ects. The approaches canbe grouped into two folds: passive isolation and post processing.

1.3.1 Passive Isolation

Thermal drift can be eliminated by placing the instrument in a specially designed laboratory withperfect isolation mechanisms [3], and also carefully setting up the experiment conditions. It is given

5


that the researcher invests heavily in construction of such kind of laboratory which is not worthwhile.Even with a perfect experiment conditions, it still takes time until the specimen reaching the thermalequilibrium, which is not compatible with the high throughput imaging procedure.In FEI , a so called 'clamshell' is designed which is mounted around the specimen holder. It is usedto reduce the sensitivity of the stage to temperature and pressure variation and prevent air �ow pushinto the column. Such kind of equipment can be used to reduce the thermal drift to some extent,but the result is still not satisfactory.

1.3.2 Post Processing

The post processing method refers to using image restoration techniques[1] to remove the e�ect ofdrift. Advanced digital image processing algorithms such as cross correlation, phase correlation,are applied to estimate the shift vectors via series of images. Individual image is aligned andreconstructed based on the estimation. The result of this method [4] is limited by the accuracy ofthe image processing algorithm. Also, it is not compatible with the fast drift rate. For instance, assoon as the point of interest is moving out of image, it is no longer possible to restore the image.As a matter of fact, the post processing approach only yields satis�able results when drift rate isrelatively low.

1.3.3 Motion Compensation

In literature, several real-time motion compensation methods have been proposed for drift correction.In [5], the drift is assumed to be constant over time, it is compensated with a series of consecutivestage adjustments. In reality, the drift always present non-linearity, the proposed method is not copewith such kind of disturbances. Some other methods, in [6] , propose an develop an on-line driftcompensation scheme for TEM only. The result yields satisfactory performance, but e�ort needs tobe continue to take SEM into consideration.

1.4 Contribution

In this thesis, we propose an on-line closed loop compensation strategy for drift correction in bothTEM and SEM. The main contributions can be classi�ed into 3 parts:1. Evaluate the current system architecture overview and propose a new architecture for drift-compensation purpose.2. Propose a closed loop control strategy to compensate the drift e�ect during image acquisition.Image-based sensing is applied to extract drift movement by means of cross correlation. A �rst orderhold prediction strategy is designed to estimate the drift behaviour. The compensation is achievedby driven the stage towards opposite direction of drift during image acquisition.3. Construct a model to simulate the the closed-loop system, analyse the performance of the proposedstrategy and provide future work directions.

1.5 Outline of thesis

The remaining report is organized as follows. Chapter 2 presents an architecture overview of anexisting electron microscopy product including hardware composition, components communication,and work �ow. The system overview serves as the foundation for the project; it helps understandingthe work-�ow of electron microscopy and assesses the feasibility of the proposed solution in actualsystem.In Chapter 3, the proposed correction method is discussed. The strategy is described step by stepwith a control point of view which including the control law and necessary elements in the closedloop.

6


In Chapter 4, we give the detailed implementation of the proposed approach using Matlab/ Simulinktools. For both TEM and SEM, the fundamental mechanism of the model such as image formation,image processing and motion compensation are discussed.In Chapter 5, the performance of the approach is examined by simulating di�erent kinds of con�g-urations with di�erent drift types.In Chapter 6, we summary the result and give a conclusion about the overall project. Possible futurework directions and remarks are presented as well.

7

Chapter 2

Hardware Architecture Overview

In this chapter, we are going to provide a comprehensive architecture overview of the system, usinga number of di�erent architecture views to depict di�erent aspects of the system. It is intended toillustrate the fundamental compositions of the system and provide a high level representation of thesystem work �ow.The architecture overview provides us a deeper understanding of the system design. It starts bydiscussing the high level architecture of the system with block diagram and internal diagram repre-sentation.It helps us quickly familiar with functionality of di�erent components. Then we describethe fundamental work �ows aiming at assessing the current system and �nd available sources forour drift correction usage. Lastly, we propose the new architecture for drift compensation purpose.

2.1 Hierarchical architecture

1. Block DiagramThe electron microscopy is composed of several hardware components, this section will specifydi�erent sub-systems in the electron microscopy. The block diagram of the overall system is givenin 2.1

• Power Cabinet: The electron microscopy power supply system.

• HT tank: High accelerating energy electron beam generator.

• Accessory: The central electron microscopy system. It is the bridge between user desktopand microscopy column. It receives the operator command from the end user desktop and isresponsible for coordinating di�erent sub-systems to complete the given tasks, such as imageacquisition and viewing navigation.

• Optics Cabinet: The optics con�guration center for electron microscopy system. It receivesthe parameter from user and coordinate the optics con�guration process.

• Microscope Column: The main body of the electron microscopy. It provides the environmentfor the interaction between electron beam and specimen. It contains several sub-systems insidethe column, such as electron gun, series of electron lens, specimen holder, detector and etc.The column must be maintained at a lower pressure level, typically on the order of 1e-4 Pa.

• Camera Cabinet: The primary imaging equipment power supply units.

8

CHAPTER 2. HARDWARE ARCHITECTURE OVERVIEW

Figure 2.1: Block diagram

2.Internal diagramIn previous section, we de�ned the block diagram of the electron microscopy system which enablesus to determine the composition of system at a high level. In this section, we are going to explorethe internal structure of di�erent components together with its communication interface.

I. Microscopy ColumnFigure 2.2 is the internal structure of microscopy column. The di�erent blocks are ordered

according to their particular functionality.

• Electron Gun: At the top of the column, generating the high accelerating voltage beam.

• Condenser lens system: In TEM, the condenser lens system is used to condensing the electronbeam into parallel. In SEM, it is used to condensing the beam into a single spot.

• Beam de�ection system(above stage): De�ect the beam onto target position. In SEM, it isalso responsible for de�ecting beam across the specimen surface in a raster pattern.

• Objective lens system(above stage): Focusing the beam onto specimen surface.

• Specimen stage: Specimen holder used to control the position of the specimen which is drivenby mechanical motors.

• Apertures: Filtering high angel scattered electrons.

• Objective lens system(below stage): Defocusing the beam that penetrated through the speci-men. Only activated in TEM mode.

• Projection lens system:Projecting and magnifying the image to the screen. Only activated inTEM mode.

9


Figure 2.2: Microscope column

II. Accessory Cabinet 2.3

• DC motion controller board: A closed control system to control the position of the stage bytuning the DC motor voltage.

• Vacuum: Maintain and control the vacuum degree inside the column.

• Piezo Motor control board: A closed control system to control the position of the stage bytuning the Piezo motor voltage.

• PIA board: short for pattern imaging acquisition. It generates scanning pattern for beamde�ection, and simultaneously samples the detector to acquire pixel signals.

Accessory cabinet is core component which in charge of stage position control and image acquisitionprocess control in electron microscopy. It provides the interface between desktop PC and terminaldevices include stage motor, imaging system inside the column. It is shown in �gure 2.3 thatthe operator command is routed to accessory cabinet via Ethernet connection. The stage positioncommand is parsed to motor voltage, and the scanning pattern is parsed to the voltage of beamde�ector coils. For our drift compensation purpose, the stage motion system and imaging systemwould plays an important roles in the proposed architecture, we will give details about this in thefollowing sections.

10


Figure 2.3: Accessory Cabinet

III. Optics Cabinet 2.4

• Optics rack 1: controlling the magnetic �eld of condenser lens system and objective lens system.

• Optics rack 2: controlling the magnetic �eld of projection lens system.

• Optics rack 3: controlling the magnetic �eld of de�ection system.

Figure 2.4: Optics Cabinet

Optics cabinet is used to setting the magnetic �eld of the lens system inside the column. It providesan e�ective solution for dampening ambient magnetic �elds, allowing high resolution performanceof electron microscope. The cabinet consists of a magnetic �eld control unit, magnetic �eld sensorand communication cables. The signals are fed to the control unit where ampli�ers drive currentsthrough the magnetic lens, thus the magnetic �eld is e�ciently adjusted. Typically, the parameterscon�gured at optics cabinet will remain constant over the imaging time, but the proper adjustmentis crucial for acquiring a high quality image.

2.2 System Work flow

The scope of the project is to propose an automated drift correction strategy which is implementablein the current electron microscope system. Regarding automating drift correction, the goal is to

11


changing the architecture such that the system is capable for autonomous correction without userinterference, which involves modifying or updating part of the work �ows in current system. Giventhat we want to provide the drift-free image, we also need a way to determine if the system possesavailable sources to perform the correction. For these purposes, we will evaluate the fundamentalwork �ows in the following sections

2.2.1 Optics Configuration

Optics con�guration is the basis for electron microscopy operation. Typical use cases include mag-ni�cation settings, focus settings and electron gun settings, shown in �gure 2.5.

• Magni�cation settings: A group of electron magnetic coils are involved in magni�cation set-tings. After the electron beam emits from the electron gun, it �rst passes through and isfocused by the condenser lens and objective lens system. The cross-section of beam is demag-ni�ed with a factor of 10 to 100 and then focused into a spot. See �gure 6.2.

• Focus settings: the primary purpose of focus setting is to focus the electron beam spot on thetarget position. This can be achieved via three ditches: (1) DC de�ector: the DC de�ector isan electron magnetic coil which used to de�ect the beam. By adjusting the surface beam spotposition, the �eld of view is calibrated; (2) DC motor: the DC motor is used to adjust thestage position in vertical direction with a [-375,375] um range. As a result, the depth of focusis changed and ultimate spot size; (3) Piezo motor: the Piezo motor is also a stage positionactuator, the stage position can be tuned in vertical axis with a [-300,300] nm ranges which is�ne adjustment compared to DC motor. See �gure 6.3.

• Electron Gun settings: The brightness of the image is determined by the electron beam en-ergy. The brighter spot indicates more intense electron-specimen interaction and yields morespecimen information. The energy of the beam can be tuned by changing the current densityof the electron gun.

Figure 2.5: Optics Con�guration process

2.2.2 Viewing Navigation Process

Proper operation of electron microscope demands precise specimen orientation. Since the specimen ismounted inside the column, the navigation process is done automatically without user intervention.

12


Typical use cases include large scale movement, �ne scale movement and slow movement,shown in�gure 2.6.

• Large scale stage movement : The DC motor is applied to change the stage position in a largescale range, from -1 to 1 mm in horizontal plane. The step size is 2 um and the bandwidth is10 Hz. See �gure 6.4.

• Small scale stage movement: The Piezo motor has the same functionality to DC motor, butwith �ne accuracy. The position can be adjusted from -1 um to 1 um in the horizontal planewith a step size of 20 pm. The bandwidth is 10 Hz. See �gure 6.5.

• Electron beam shift : Changing the beam spot position can also regarding as a navigationaction. This is done by changing the current of DC de�ector. The reason we named it slowmove is that there is an AC de�ector in column which takes the same action with DC de�ectorbut with an extreme fast frequency which up to 1 MHz. We will discuss the usage of the ACde�ector in next subsection. See �gure 6.6.s

Figure 2.6: Viewing Navigation Process

2.2.3 Image Acquisition Process

Image acquisition is the primary goal of the electron microscope. All the con�gurations and adjust-ments are for acquiring a satisfactory and high quality image. In chapter 1, we have mentioned twoimaging modes, namely transmission and scanning mode. Each mode possesses di�erent imagingmechanisms, and as a result, di�erent hardware are involved which is shown in �gure 2.7.

• TEM: The image acquisition process is a single-button-press activity. After exposure timecon�gurations and brightness adjusted, the button is pressed and the acquired image is per-ceived from the cameras. Basically, a �uorescent screen is mounted at the bottom of the TEMcolumn, which emits light on impacted of the transmitted electrons; then a camera is used torecord the image in a normal photography process. See �gure 6.7.

13


• SEM: Image acquisition is a more complex process in scanning mode. It is done by scanning thespecimen area pixel by pixel. The electron beam position is controlled by AC de�ector, whichcoordinates the spot sweep cross the sample surface in a raster pattern. The scanning processconsists of a series of horizontal line scans, with a fast sweep back when the beam reaches theend of line range. Meanwhile, the detectors positioned around the stage are synchronized tocollect the emitted electrons for further image assembling. See �gure 6.7.

Figure 2.7: Imaging Acquisition Process

14


2.2.4 Image Processing

After the image acquisition process, the image is assembled and stored in the desktop PC. Thedigital image can be subjected to sophisticated analysis and processing using image manipulationtechniques. The image quality can be further improved by contrast enhancement, �ltering, colorcoding and etc.

2.2.5 Discussion

From the above it is clear that to be able to acquire an image with suitable quality, each work �owplays an important role. Part of the processes are performed by the operator such as optics con-�guration and viewing navigation. The parameters con�gured at these stage are assumed constantduring image acquisition. Remaining part of the processes include image acquisition process andimage processing is proceeded automatically.Regarding automated drift correction, resources includes sensor, actuator and controller are inde-fensible. Given the work �ow, we found that:

• Stage motion can be used to compensate the specimen drift by driven the stage position towardsopposite direction of drift. However, stage position is kept static during image acquisition,meanly there is no connection between imaging system and stage motion system.

• There is no dedicate sensor for drift measuring in current system, only sensor available is theinstrument itself. Sensor data can be acquired by processing the produced image. However,no image processing hardware involved in image acquisition for drift measurement purpose,as a result, data has to be sent to desktop PC for processing which is time consuming andine�cient.

By evaluating di�erent work �ow scenarios, a communication infrastructure of current system is pre-sented in �gure 2.8. The problems founded in previous discussion can be identi�ed in communicationdiagram which depicted the system with a layered fashion. The desktop PC is at the top level, it isdirectly connected to the imaging system and middle level controller. The terminal devices includesstage motors and optics lens are controlled by the middle layer controller. It is clearly that there isno direct connection between imaging system and motion system. The only route is through desktopPC which will lead to sensor delay and heavy payload. According to these architecture limitations,new architecture proposal is expected which we will discuss in next subsection.

15


Figure 2.8: Communication diagram

2.3 Architecture Proposal

2.3.1 Architecture Stress

In current system, the stage position adjustment is taken in optics con�guration and viewing navi-gation process, it did not participate in the image acquisition process. Meanly, the stage stands stillduring image acquisition and there is no available communication channel between imaging systemand motion system, therefore, real-time motion compensation is infeasible. Due to this drawback,new architecture is proposed in next section.

16


2.3.2 New Architecture Proposal

Figure 2.9 is a high level block diagram of proposed system architecture. A scanning FPGA generatesan x,y coordinates which are used to drive the scan coil that moves over specimen in a rasterpattern. The scattered electrons are collected by a group of collectors which are registered to thex,y position of the beam on the sample. Acquired pixels are assembled and streamed directly to theimage processing unit, where drift vectors are calculated. The concurrent calculated drift vectorsprovide a temporal component that allows to make prediction on how drift will be in coming events.The prediction of coming drift is processed in the motion prediction block and generates the setpoint of the stage. As a result ,stage is driven towards the opposite direction of the drift duringimage acquisition process and the drift e�ect is eliminated. Apparently, the proposed architectureintroduces a communication channel between imaging system and motion system which allows afast response and processing rate for drift correction. The accuracy of the correction will rely on themotion prediction unit which will be elaborated in chapter 3.Figure 2.10 is a �ow chart illustrating the steps performed for proposed drift compensation method.The compensation process starts by acquiring a reference image(we assume the index of referenceimage is 0). Once reference image is located, it is bu�ered in the image processing unit and the�rst frame(l = 1) is starting acquired. It should be noti�ed here that the drift vectors computationstarts after the �rst frame is acquired, so no drift compensation action is taken during �rst frameacquisition. As soon as the drift vectors are available, the drift compensation procedure is activated,and executes in parallel with the next image acquisition process. The compensation process executesrepeatedly until a satisfactory image is acquired.Now, we have introduced the proposed drift compensation architecture with a block representationtogether with its work �ow. A crucial problem needs to be stressed out here which is refereed to assensor delay. Intuitively, the proposed communication channel between motion system and imagingsystem will ensure a fast data delivery rate, but due to the nature of image-based sensing andcontinuity of drift, the sensor delay still dominate the performance of the method. We will explainthis e�ect and give the propose solution in chapter 3.

Scanning FPGA

Sampling FPGA

Optics cabinet

Beam deflector

Beam detector

Column

Scanning Pattern

Deflector current

Pixel signals

PIA

Image Processing

Motion Prediction

Image FrameShift

estimation

Stage motion controller

Piezo Stage

Stage motion control system

Stage set point

Figure 2.9: Proposed architecure

17


Start Image Acquisition Process

Acquired Reference Image

Compute Drift vectors

Predict Drift Movement & Drift CompesnationAcquired Next Image (l=N)

Assemble Next Image (l=N)

Assemble Reference Image(l=0)

Figure 2.10: Drift Compensation Diagram: arrow indicates the excuation sequence for each block,N indicates the frame index, its incremental by 1 in each iteration.

2.4 Conclusion

In this chapter, we provide the hardware architecture overview of current system. It aims at pro-viding fundamental composition description of the electron microscope product. Key work �ows aregiven together with the occupied hardware components.By evaluating the architecture, it is found that there is no direct connection between image acqui-sition system and stage motion system which indicates the incapability for real time drift compen-sation. For that purpose, a new architecture is proposed in line with the closed-loop methodology.It contains a communication channel between the two separate systems and additional modulesincluding image processing and motion prediction are connected in between. In next chapter, wewill give a depth analysis of image acquisition process under drift e�ect and propose a predictionmethod for drift compensation.

18

Chapter 3

Proposed Method

In this chapter, method for closed-loop system design is presented. The overall structure of theclosed-loop system is shown in �gure 3.1, in total three parts are included: a plant includes actuator,sensor and image acquisition system; a controller and an external disturbances.Image acquisition is the process that drift disturbances would impact on. In�uenced by the drifte�ect, the specimen starts moving during image acquisition and the acquired image is distorted orblurred. Once image is acquired and assembled, the specimen drift vector is measured by calculatingthe relative displacement of image content in each acquired image with respect to the reference image.Given the drift vector, the controller predicts the future drift using �rst order hold prediction, andgenerates the stage set points by linear interpolation strategy. Eventually, the stage position isdriven towards the opposite direction of the drift, thus, the imaging �eld is adjusted and the driftis compensated.

Controller(Drift motion

predictor)+ue

Referenceimage

y

Actuator(Stage motion system)

Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T1


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T2 T2

Image Processing

T2 T2

+ueReference

imagey


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1

+ueReference

imagey


Image Acquisition

system

Acquired Image

d

Plant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1 T1 T1


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

d

Plant

-

(Sensor)Image

Assembling

DisturbancesProcess

T2 T2

Image Processing

T2 T2 T2 T2

+

d

DisturbancesProcess

Figure 3.1: Schematic diagram of closed-loop system: Closed-loop system includes controller, plantand external disturbances. e is the pixel shift between reference image and acquired image, u is thestage setpoints, d is the drift disturbances, y is the system output(acquired image).

In following sections, we are going to give a detail explanation for each conceptual blocks togetherwith its operation rates.

3.1 Image Acquisition and Drift Measurement

3.1.1 SEM

1.Image AcquisitionRaster Scanning : In scanning mode, the electron beam sweeps across the surface of specimen fol-lowing a raster pattern. �gure 3.2 illustrates the raster scanning process, it includes a series of line

19

CHAPTER 3. PROPOSED METHOD

scan across the specimen pixel by pixel, with a fast sweep back when the spot reaches the end of thehorizontal �eld. The spot point keeps static at the scanning point for a set of time before steppinginto next position which referred to as the dwell time.

Scanning

Sweep back

Fast scan Horizontal

Slow scanVertical

i

j

Figure 3.2: Raster scan: the scaning starts from the upper right corner, ends in the bottom leftconner. The pattern consists of a fast scan in horzional axis and slow scan in vertical axis.

We �rst construct a coordinate system, pixel P(i,j) refers to the pixel position, i presents the hori-zontal coordinate and j presents the vertical coordinate respectively. The time instance T for acquirepixel P(i,j) can be presented as follows:

T (P (i, j)) = i∗tdwell + j∗trow (3.1)

trow = Pixelrow ∗ tdwell + tline (3.2)

tdwell is the beam dwell time for each pixel, trow is the line time which equals to the multiplicationof dwell time and number of pixels per row. tline refers to the back track time for electron beam.Pixelrow is the number of pixel per row.In SEM, the acquired pixels are �rst stored in a packet before assembling into a complete imagewhich is called packet-based acquisition. The packet can accommodate pixels that acquired withincertain period of time, typically con�gured to 20 millisecond in current FEI system. For example,if the frame time is 1 second, then in total 50 packets are created. By the time when scanningoperation �nished, the packets will be assembled into image. Suppose the packet index is k andpacket time is Tpacket , we can derive:if

k ∗ Tpacket ≤ T (P (i, j)) < (k + 1) ∗ Tpacket (3.3)

then P (i, j) ∈ packet k and k ≤ Numpacket =⌈Tframe

Tpacket

⌉At the time when all packets are acquired, a complete image frame l is assembled. Taking imageframe index l into consideration, the equation 3.1 and 3.2 can be reformulated as follows:

T (P (i, j, l)) = i∗tdwell + j∗tline + l ∗ tframe (3.4)

tframe = Pixelframe ∗ tdwell + tline ∗ numrow (3.5)

where tframe is the frame time, Pixleframe is the total number of pixels per frame. numrow is thenumber of rows per frame.Based on equation 3.4 and 3.5, the coordinates i and j are represented as function of time t. Givenany time instance t, the spot position (i,j) can be uniformly identi�ed by equation 3.6 and 3.7. Sup-pose the dwell time, back track time and frame time remain constants after con�guration, we canconclude that the scanned positions with respect to pixel coordinates are monotonically increasing

20


with time instance t.

j (t, l) =

⌊t− l ∗ t_frame

trow

⌋(3.6)

i (t, l) =

⌊(t− l∗tframe)− j (t, l) ∗ trow

tdwell

⌋(3.7)

2.Drift e�ectWe have de�ned the image acquisition model which presents the pixel position as function of timeinstance t. Next step, the drift e�ect is taken into consideration. Ideally, the specimen shouldstand still during image acquisition process. For instance, the pixel position P (i (t), j (t)) atframe l and time t will appear at the same position P(i(t

′),i(t

′)) at frame l

′and time t

′,where

t′= (l

′ − l) ∗ tframe + t.In reality, the thermal drift causes a movement of the specimen with respect to stage. As a result,the actual scanned position will change which leads to a displacement of �eld of view. The drift canbe formalized with a vector as a function of time t,

Dr (t) =

{Dx(t)Dy(t)

}(3.8)

The actual position can be formalized as follows, and the e�ect is shown in �gure 3.3.

Pact(i(t), j(t) = P (i(t) +Dx(t), j(t) +Dy(t)) (3.9)

x

y

x

yDrift effect in raster scan

Figure 3.3: Drift e�ect in raster scan: left hand side �gure shows the scanning image without drift,right hand side �gure shows the scanning image with a drift towards positive x and y directions.Di�erent pixel position has di�erent drift shift vector, and last scan position su�ers heavest drift.

3. Drift measurementAs the deformed image is acquired, the next step is to extract the drift information. It should benoted that sensor data cannot be measured within single deformed image, it should be estimatedbetween pairs of images. In literature, various of image processing algorithms are developed to copewith the shift measurement between pairs of images, such as phase correlation, cross correlation,block matching [12]. In our closed loop, cross correlation [8] is chosen as the image processing al-gorithm. Given reference image A with dimension (Ma,Na), deformed image B with dimension(Mb,Nb), the cross correlation matrix is given by equation 3.10, the peak position appearing in thecross correlation matrix C reveals the image shift value. Consequently, drift vector is extracted fromthe reference image and deformed image.

C(i, j) =

(Ma−1)∑m=0

(Na−1)∑n=0

A(m,n)conj((B(m+ i, n+ j))); (3.10)

21


where 0 ≤ i ≤Ma+Mb− 1, 0 ≤ j ≤ Na+Nb− 1.

Up to now, the SEM image acquisition process together with drift e�ect and measurement areall presented, short remarks are given below:Remarks

• It follows from equation 3.6,3.7 that the extent of image distortion signi�cantly depends onthe dwell time. Intuitively, a minimum dwell time is expected to limit drift within an image,but signal to noise ratio cannot be guaranteed. On the contrary, the longer the dwell time is,the lower de�ection rate the spot position has, as a result, the more pronounce distortion isdepicted in the image.

• The drift distortion function is unknown to us, but it has been identi�ed that drift is a slowlychanging, continues process. Therefore, we assume the drift can be modelled by a linearfunction with constant speed or a non-linear function with low frequency down to sub Hz.

• Based on above discussion, it is concluded that SEM image acquisition is a multi rates pro-cess. The pixel intensity values are collected in a packet rate and complete frame is available inframe rate. As a consequence, a rate transition is required to handle transfer of data betweenthese two operating rates. Figure 3.4 shows the timing domains of image acquisition togetherwith image processing unit in the closed loop system.


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image Assembling

T1

Image Processing

T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2 T1

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2T1

T1T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

Rate transition Rate transition

ue y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

Controller

T2T2

T2

+

d

DisturbancesProcess

Rate transition

Referenceimage

Rate transition

Rate transition

Rate transition

Rate transition

Figure 3.4: SEM sensor data acquisition rate:Image acqusition is running in packet rate(T1). Sensordata assembling and image processing is running in frame rate(T2).

3.1.2 TEM

1. Image acquisitionCompared to the complexity of image acquisition process in SEM, acquiring image in TEM is muchsimpler. The image acquisition process is a single-button-press activity, after con�guring exposuretime and adjusting illumination, a button is pressed and all pixels are acquired in parallel. The timeinstance T for acquire pixel P(i,j) can be presented as follows:

T (P (i, j)) = tframe (3.11)

Taking the frame index l into consideration, the equations are rewritten as:

T (P (i, j, l)) = l ∗ tframe (3.12)

22


2. Drift e�ectDuring TEM image acquisition, a CCD or CMOS camera will integrate all incoming electrons overexposure time. Under drift e�ect, image presents blur and becomes severe in long exposure time.Meanwhile, image averaging is frequently applied to increase signal to noise ratio among series ofimages. Because image averaging process takes time, a slight drift shift will also blur the averagedimage.In this thesis, the TEM image acquisition process is assumed to be done at TV rate (10 frames/seconds)which indicates a short exposure time. As a result, single image blur is assumed negligible. Underthis assumption, the image shift will be consistent for all pixels within single frame. The drift e�ectis shown in the �gure 3.5.

Figure 3.5: Drift e�ecet in TEM: Compared with SEM, no image distortion presents in TEM image,every pixel has same amount of shift.

Remark

• Similar to SEM, drift shift between acquired image and reference image is estimated by meansof cross correlation method.

• TEM image acquisition can be simpli�ed as a subset of SEM image acquisition process. InSEM, single complete image consists of multiple packets acquisition. In TEM, the packet timeis assumed to be equal to frame time , thus, all image acquisition and image processing willproceed in same rate, shown in �gure 3.6.


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

Rate transition

T2 T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

+ueReference

imagey


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1

+ueReference

imagey


Image Acquisition

system

Acquired Image

d

Plant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1 T1 T1


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

d

Plant

-

(Sensor)Image

Assembling

DisturbancesProcess

T2 T2

Image Processing

T2 T2 T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

Figure 3.6: TEM sensor data acquisition rate :In TEM, sensor data acquisition and image processingunit update in a frame rate.

23


3.2 Motion Prediction Controller

In the proposed closed-loop system, the image shift value, which calculated in current step, is themeasurement of the drift shift of the previous sample step. As drift is a continues process, any driftwithin current frame acquisition cannot be measured and compensated, thus, gives rise to controlperformance degradation. Meanwhile, the long frame time will also contribute to sensor value de-lay, especially severe in SEM with long dwell time. Due to this limited and unpreventable sensordelay, the future drift measurement is expected to be successfully approximated by linear-predictionmethod based on the previous sensor values. In this section, a �rst order hold prediction scheme isproposed. For prediction purpose, the drift process is reconstructed as a piecewise linear approxi-mation to the sensor value sampled. Following sections we will explain the prediction strategy foreach imaging mode.

SEMThe controller is divided into two parts with di�erent processing rates. Shown in �gure 3.7, the �rstpart receives the sensor value and exports the drift speed estimation and set point at the beginningof each new frame with a frame rate. These two outputs will be held in a zero order hold unitduring frame acquisition. Based on the estimation of �rst part controller, the second part controllercalculates the set point of stage using linear-interpolation method and updates in a packet rate. Themathmatical model is given below:

A. Controller1Input: Estimation of shift in horizontal and vertical axis. Ex, EyOutput: Drift speed prediction in horizontal and vertical axis. Vx, VyStarts point at each new frame. Ux,UyControl Law:

V x (l + 1) = V x (l) +Kp ∗ Ex(l+1)Tframe

, V x (0) = 0

V y (l + 1) = V y (l) +Kp ∗ Ey(l+1)Tframe

, V y (0) = 0(3.13)

Stage set point at each new frame. Ux,Uy

Ux (l + 1) = Ux (l) + V x (l) ∗ Tframe + Ex (l + 1) , Ux (0) = 0Uy (l + 1) = Uy (l) + V y (l) ∗ Tframe + Ey (l + 1) , Uy (0) = 0

(3.14)

l presents the frame step, Kp = 1 stands for the proportional gain of speed adjustment.

B. Controller 2Input: Drift Speed estimation in horizontal and vertical axis. Vx, VyStarts point at each new frame. Ux, UyOutput: set point of stage in horizontal and vertical axis. Sx, SyControl Law:

Sx (k + 1) = Ux (l + 1) + V x (l + 1) ∗ (k%Numpackets) , Sx (0) = 0;Sy (k + 1) = Uy (l + 1) + V y (l + 1) ∗ (k%Numpackets) , Sy (0) = 0;

(3.15)

k presents the packet step, l presents the frame step p where

l =

⌊k

NUMpackets

⌋

24


Controler1

Controller2

Ex,EyVx,Vy

Ux,Uy

ZOH

ZOHSx,Sy

Controller

Figure 3.7: SEM controller

Remark

• Given the sensor sampling period Tframe, and the predictor update period Tpacket , Tframe =h ∗ Tpacket, where h is a integer which lager or equal than one, the sensor-to-controller delayis compensated using �rst order hold method. The stage position correction is dealt with byintroducing linear-interpolation.

• Once the loop is closed, calculated shift value between closed-loop image and reference imagewould be the error between predicted drift and actual drift. This error information is used toupdate the drift speed prediction, as a result, stage motion speed is adjusted as well.

• In equation 3.13, the speed adjustment gain Kp is set to one, which indicates a full gain. If Kp

larger then one, then it is expected that a smaller shift between acquired image and referenceimage will lead to a large adjustment of the stage position which result in a large overshoot.If the controller repeatedly make prediction under a large Kp, the output stage position wouldoscillate around the desire position and the system will lost stability over time. Conversely,a Kp with value less than one will weaken the controller prediction e�ort which results in asmall overshoot but the settling time is expected increase. So, for our controller design, theKp is set to one and the in�uence of di�erent choice of Kp is measured in chapter 5.

• Figure3.8 shows the timing domains of the controller for SEM. It is noted that a rate transitionproperty is presented in the second part of the controller which is due to the nature of �rstorder hold prediction. As discussed above, the second part controller takes the drift estima-tion(�rst controller output, update in frame rate) as the input, update the stage set pointsusing linear interpolation method, as a result, the frame rate is tuned back to packet rate.


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-


T1

Image Processing

T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2 T1

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2T1

T1T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess


ue y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

Controller

T2T2

T2

+

d

DisturbancesProcess

Rate transition

Referenceimage

Rate transition

Rate transition

Figure 3.8: Operation rate of SEM controller in closed loop

25


TEMTEM controller is shown in �gure 3.9. Given the shift estimation from image processing, the con-troller updates stage set point at each frame step. The overall controller design is followed by a zeroorder hold. The controller model is given as below:

Controller:Input: Estimation of shift in horizontal and vertical axis. Ex, EyOutput:Stage set point at each new frame. Sx,Sy

Sx (l + 1) = Sx (l) + Ex (l + 1) , Sx (0) = 0Sy (l + 1) = Sy (l) + Ey (l + 1) , Sy (0) = 0

(3.16)

l presents the frame index,Kp = 1 stands for the proportional gain of speed adjustment.

ControllerEx,Ey Sx,Sy

TEM motion prediction

Figure 3.9: TEM controller


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

Rate transition

T2 T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1

+ueReference

imagey


Image Acquisition

system

Acquired Image

d

Plant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1 T1 T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

Figure 3.10: Operation rate of TEM controller in closed lop

Remark

• We assume TEM has a shorter frame time(10 frame/seconds) which indicates a higher sensorsampling rate. As a result, TEM is treated as a subset of SEM, meanly the frame time equalsto packet time and the timing domains of TEM controller is equal to frame rate (T2), see�gure 3.10.

3.3 Actuator

Actuator is a device that transforms the control signal into motion. In our proposed method, thestage motor is adapted to manipulate the position of the specimen such that it is preserved withinthe �eld of view during image acquisition. In current system, two motors are used to manipulate thestage orientation operation which are DC motor and piezo actuator. From the discussion in chapter2, it is found that piezo actuator has high accuracy and small step size compared to DC motor,thus,it is the best option for the drift compensation purpose.

26


3.3.1 Piezo Actuator

Piezo actuators are designed based on the inverse piezoelectric e�ect. The advantages of piezoactuators include large force, high sti�ness and fast response time. The step size of piezo actuatoris up to 20 Pico meters, and the band width is 10 Hz(open loop). Shown in �gure 3.11, a feedbackclosed loop is used to control the motion of the motor which could eliminate non-linear behaviourof piezoceramics such as hysteresis and creeps. In this thesis, the actuator closed -loop system isapproximated as a second order dynamical system, its transfer function is shown in equation 3.17.See from �gure 3.12, the bandwidth of the closed-loop is about 200 Hz, which would su�ciently highfor our set point update rate, as a result, the traceability of the stage motion is guaranteed.

G(s) =ωn

2

s2 + 2βωns+ ωn2

(3.17)

where ω is chosen to 500 and β is chosen to 1.2.

Controller(PI controller)

Plant(Amplifier + piezo mechanics + strain

sensor)

+r

y-

Ve

Figure 3.11: Closed loop system of Piezo actuator:r is the reference position, v is the voltage, y isthe measured position.

Figure 3.12: Bode plot of approximate piezo actuator closed-loop system

27


3.4 Discussion

So far, all closed-loop relevant elements are discussed. By incorporating piezo actuator, we have thecomplete closed-loop system together with its timing domains for each unit, which are shown in 3.13and 3.14.The overall closed-loop of SEM is a multi rates system, given by a fast rate actuator alignmentand slow rate image sensing. To cope with the sensor delay, a �rst order hold linear-interpolationcontroller is designed. At time t when l ∗Tframe < t < (l+1)∗Tframe, the stage position is updated(T1 packet rate) in a linear way without knowledge of the sensor measurement. The linear rampprediction (speed of stage movement) will be updated till t = Tframe ,when new frame is acquired.With this prediction scheme, the sensor-controller delay is weakened.In TEM, a single rate system is depicted in �gure 3.14. All functional blocks including image sensing,controller prediction and motor actuation are proceeded in same rate (frame rate). Consequently,sensor-delay issue is released.


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-


T1

Image Processing

T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2T1

T1T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess


Figure 3.13: Operation rate of piezo actuator in SEM closed loop


predictor)+ue

Referenceimage

y


Image Acquisition

system

Acquired Image

dPlant

-

(Sensor)Image

Assembling

DisturbancesProcess

T1 T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

Image Processing

Image Processing

Rate transition


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-


T1

Image Processing

T2


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

Rate transition

T2 T2

C1 c2

Controller

T2 T1

ueReference

imagey


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T1 T2

Image Processing

T2 T2

C1 c2

Controller

T2T1

T1T1


predictor)ue

Referenceimage

y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2 T2

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess

+

d

DisturbancesProcess


ue y


Image Acquisition

system

Acquired Image

Plant

-

(Sensor)Image

Assembling

T2 T2

Image Processing

T2 T2

Controller

T2T2

T2

+

d

DisturbancesProcess

Rate transition

Referenceimage

Figure 3.14: Operation rate of piezo actuator in TEM closed loop

28


3.5 Conclusion

In this chapter, the mechanism of proposed method is discussed. Fundamental control elementsinclude sensor, controller and actuators are details explained.Given the image formation model together with drift vectors, the distortion and blur e�ects arefurther analysed. Image-based sensing is taken by measure the drift between pairs of images and theaverage pixel shift value is used as sensor data for drift estimation and compensation. The controlleris designed to predict the dynamic behaviour of drift motion and update the stage position andmotion speed such that the drift can be traced and compensated. Driven by piezo actuator, thestage move towards the opposite direction of the drift, thus, the drift e�ect is eliminated.

29

Chapter 4

Model Implementation

In this chapter, a simpli�ed simulation model of closed-loop motion compensation system is devel-oped. It includes all relevant elements at an appropriate modelling level: a model of image acquisitionprocess, a drift motion predictor, a approximation model of piezo motion system and a model ofimage processing algorithm. The model enables us to validate the performance of the closed-loopsystem under di�erent disturbances frequencies and di�erent system con�gurations. Their designsand implementations are guided by the hardware architecture overview (chapter 2) and proposedcompensation strategy (chapter 3). The schematic of the models are shown in �gure 4.1 and �gure4.2 :It is important to point that simulink model is a approximation of real system, not all functionalitiesand dynamic behaviours can be simulated in the model. In order to simplify the model and promotethe e�ciency, certain assumptions should be take into consideration �rst.

• In TEM and SEM, image acquisition is a time dependent and continues process. Processesinclude beam-specimen interaction, electron scattering and electrons collection are relevant inimage acquisition process which present extreme complex dynamical property. Regarding ourautomating drift-correction purpose, only drift e�ect is incorporate in our model, namely theimage shift and image distortion. Consequently, �rst assumption is made as follows:Assumption 1: In the model, the image acquisition process is assumed to be a discrete process.Every sample step, a packet of pixels is acquired in SEM, and an image frame is acquired inTEM.

• In our model, the image shift and distortion e�ect are modelled by picking up mis-alignedpixel positions. Since pixel coordinates are all integers, thus assumption is made:Assumption 2: All image shifts are limited at pixel accuracy, any sub pixels shift will berounded to nearest integer value.

• The objective of the simulink model is to model the image acquisition process under driftand validate the correctness of the proposed strategy. Thus, any algorithm processing timeand data transfer delay are assumed negligible in the simulation. In reality, these factors areimportant regarding the system performance, we will leave this to the future work.Assumption 3: Both image processing and data transmission are done in zero time. Hence,the total simulation time equals to the period of interest (multiple frame times).

• As discussed before, single image blur is assumed negligible in fast frame time (10 frames/second),thus assumption 4 is given:Assumption 4: Image blur only occur after averaging multiple frames in TEM, single imageblur is assumed negligible.

30

CHAPTER 4. MODEL IMPLEMENTATION

Scanning Pattern

GeneratorDisturbances

Motion Compensation

Scanning Engine

Image Assembling

Image Processing

Motion Prediction

Piezo Stage system

T1 T1

T1

T2

T1 T1

T1

Rate Transition

T2Reference

Image

Rate Transition

T2T1

T1

Figure 4.1: SEM simulator: The sample time is set to packet time (T1). Every sample step, a packetof pixels is acquired and stage position is adjusted. The Rate Transition block is used to transformthe packet rate to frame rate (T2). A detailed explanation for each functional block will be givenin the following sections.

Scanning Pattern

GeneratorDisturbances

Motion Compensation

Scanning Engine

Image Assembling

Image Processing

Motion Prediction

Piezo Stage system

Rate Transition

Reference Image

DisturbancesMotion

CompensationImage

Assembling

Image Processing

Motion Prediction

Piezo Stage system

T2T2

T2 T2

T2

T2

Reference Image

Figure 4.2: TEM simulator: The sample time is set to frame time (T2). Every sample step, a frameis acquired and stage position is adjusted. A detailed explanation for each functional block will begiven in the following sections.

4.1 Image Acquisition

SEMBefore explain the model of the SEM image acquisition, let us �rst take a look at 'PIA'. A PIA is shortfor Pattern Imaging Acquisition, which is the scanning device in FEI electron microscope product.The PIA consists of two main functionalities: 1. generate the scanning pattern for de�ecting theelectron beam; 2. sample the detectors to collect intensity value for each pixel. Afterwards, thesignals are sent to desktop PC for assembling and processing. In the simulator, we model thePIA device by dividing it into three parts: scanning pattern generator, scanning engine and imageassembling unit. Figure 4.3 shows the relationship between the model and real system.

• Scanning pattern generator: Each sample step, the generator produces the pixel coordinateswith respect to horizontal and vertical position and stores them into two vectors (indexx, indexy).The pixel coordinates are referred to as scanning pattern and their sequence decides the wayof scanning (raster scanning). The vector size depends on the dwell time and packet time, e.g.dwell time = 1e− 7, packet time= 1e− 3, vector size=packet time/dwell time= 1e4.

• Scanning engine: In the model, the plant and sampling device are combined into one functional

31


Scanning FPGA

Sampling FPGA

Communication & Control

Plant

X,Y

Signal

Pixel stream

Host PC

PIA

Scanning pattern

generator

Scanning Engine

Image Assembling

X,Y Pixel stream

X,YSimulator

Figure 4.3: Model of PIA: A comparision between the PIA device and image acquisition model.

block which is called scanning engine. The sampling process is done by picking up the pixelintensity value from reference image under scanning pattern coordination. Afterwards, theintensity values are stored into a packet for further assembling.

• Image assembling: Each sample step, a slice of reference image is sampled and a packet ofpixel intensity value is produced. The pixel stream is relayed to the image assembling unitfor bu�ering. When image scanning �nalized, the image assembling unit outputs the completeimage frame. The image acquisition process is shown in �gure 4.4, for simplicity, we take a5*5 image to explain:

The persudo code is given below:

for f = 0→ fnum dofor p = 0→ ptotlapackets do

for t = 0→ tpacketstime docompute the scanning pattern for packet p ;Take the intensity value from reference image and store the pixels to packet p;

end forend for

end for

TEMAs mentioned before, there is no complex mechanism in TEM image acquisition, only action is totake image content from reference image at each sample step.

4.2 Image Deformation

In this section, we are going to add drift disturbances model to the image acquisition process. Indisturbances block, the drift behaviour is modelled and con�gured by adding linear or non linear

32


86 87 9 150 151

62 25 24 18 123

4 88 96 185 124

36 38 9 255 254

222 221 206 203 247

1 2 3 4 5

1

2

3

4

5

Horizontal_coordinates=[1,2,3,4,5,1,2]

Vertical_coordinates=[1,1,1,1,1,2,2]

86 87 9 150 151

62 25Pixel

Stream=[86,87,9,150,151,62,25]

Scanning pattern

generator

Scanning Engine

Image Assembling

X,Y Pixel stream

X,YSimulator

1 2 3 4 5

1

2

3

4

5

Figure 4.4: SEM image acquisition: The packet size is 7. Each sample step, seven paris of coordinatesare generated and a packet of 7 pixels are acquired from reference image. To acquire a completeimage, 4 packets are needed in total.

functions . The deformation process is explained in the following sections.SEMIn order to model the drift e�ect, a disturbances block is added between scanning pattern generatorand scanning engine. The scanning pattern generator exports the pixel coordinates that is the areasuppose to be scanned. Before start the scanning process, the pixel coordinates are �rst sent tothe disturbances block. Within disturbances block, the drift is sampled and added to the scanningpattern (pixel coordinates). As a result, the output coordinates are turned into the actual scannedpositions under drift e�ect. It should be stressed out the image deformation is achieved by pixelcoordinates manipulation which is limited to integer accuracy. Therefore, any image shift with subpixel accuracy in real system will be rounded to nearest integer in the model. The image deformationprocess is illustrated in �gure 4.5.

Scanning Pattern Generator

Disturbances

Scanning Engine Image Assembling

Original pixel coordinates(X,Y)

Shifted Pixel Coordinates(X+dx,Y+dy)

Pixel Stream

Image Deformation SEM

Figure 4.5: SEM image deformation

The persudo code is given below:

for f = 0→ fnum do

33


for p = 0→ ptotlapackets dofor t = 0→ tpacketstime do

Sampling the drift value at time instant t;Add the drift shift to the scanning pattern coordinates;Take the intensity value from reference image and store the pixels to packet p;

end forAssemble the packet p;end forAssemble the frame f;

end for

TEMAs discussed in chapter 1, the drift e�ect presents image blurring in transmission mode, especially inlong exposure time and image averaging. In the simulator, we only implement the image averagingdeformation, the long exposure time deformation is left as future work. Followed by the samemechanism in scanning mode, the drift is �rst sampled before image acquisition. The whole referenceimage is shifted a number of pixels which equal to the drift value. The deformation process is shownin 4.6

DisturbancesImage

acquisitionDrift value

(Dx,Dy)Shifted image

Image Deformation TEM

Figure 4.6: TEM image deformation

4.3 Image Processing

In previous sections, we have implemented the image acquisition process under drift e�ect. Nextprocess is to calculate the translation value between deformed image and reference image using imageprocessing techniques. In the simulator, cross correlation method is taken.The cross correlation result reveals translational shift between two images at peak position and theresult can be used as the global estimation of the shift vector between reference image and deformedimage. In order to e�ciently compute the cross correlation and optimize the simulation speed, weuse Fourier transform to compute the cross correlation in the frequency domain.The image processing procedure is as follows:

1. Take the Fourier transform of the deformed image and reference image, which is done by usingthe Matlab 2D Fast Fourier transform operation (�t2).

2. Calculate the conjunction of the deformed image and reference image in frequency domain.3. Transform the result into spatial domain using Matlab inverse Fourier transform operation

(i�t2).4. Get the coordinates of the peak of the cross correlation which de�ne the shift vector between

deformed image and reference image.

4.4 Motion Prediction

After obtaining the drift measurement at the current sample step, the drift behaviour in the nextsample step is estimated in the motion prediction unit by means of linear-interpolation.SEM

34


In chapter 3, we have discussed the mathematical model of the control law with a two level structure.Conceptually, two frequencies exist in closed-loop system, namely frame rate(sensor sample rate)and packet rate(controller update rate). Intuitively, the predictor uses zero order hold for receivedsensor data and uses �rst order hold for output data. In simulator, the overall model is running witha packet time step. The motion prediction unit is dispatched under a packet coordinator which sendsthe trigger signal at every packet instant. The signal which updated in a packet rate is triggeredevery sample step. The signal which updated in a frame rate frequency is determined when thetriggered signal equals to the total packet number. In that sense, the conceptual model is simpli�edby combining two controllers into one unit. The process is depicted in �gure 4.7.

Packet1 Packet2 Packet n. . .

Stage speed updateStage start position

update

Stage position update Stage Position

update

Frame (n)

Packet Coordinator

Stage Speed Update

Stage position Update

Packet counter==total number of packets

NO

Yes(New Frame)

Packet1 Packet2 Packet n. . .

Stage speed updateStage start position

update

Stage position update

Stage Position update

Frame (n+1)

Figure 4.7: SEM motion prediction: The packet coordinator generates the packet counters. If thecounter equals to the total packet number, then a frame is acquired and stage speed is updated.Otherwise, only stage position is updated.

The control law is implemented using Matlab level 2 S-function, the pseudo code is as follows, krefers to the sample step with respect to packet time:

if packetcounter(k) == totalnumpacket thenUpdate driftspeed(k + 1);Update startpoint(k + 1);

elseDriftspeed(k + 1) = driftspeed(k);Startpoint(k + 1) = startpoint(k);

end ifCalculate stage set point(k+1);

TEMAs discussed in chapter 3, the stage position is updated in frame rate.

Frame sequence

Frame n Frame n+1 Frame n+m. . .

Stage position update Stage position update Stage position update

Figure 4.8: TEM motion prediction

35


4.5 Interconnections

In previous sections, all functional blocks are established and explained. The �gure below is adiagram of components together with a description of their interconnection topology and data �ow.

Scanning Pattern

Generator

DisturbancesPixel coordinates packet

(X,Y)

Scanning Engine

Image Assembling

Pixel intensity packetMotion

Compensation

Shifted Pixel coordinates packet(X+dx,Y+dy)

Compensated pixel coordinates packet(X+dx-Sx,Y+dy-Sy)

Piezo Actuator

Stage set point(Sx,Sy)

Image Processing

Complete ImageMotion

Prediction Unit

Image shift estimationEx,Ey


Reference Image

Figure 4.9: SEM simulator interconncetion

Disturbances

Image Assembling

Motion Compensati

on

Drift Vectors(dx,dy)

Compensated pixel coordinates packet(dx-Sx,dy-Sy)

Piezo Actuator


Image Processing

Complete ImageMotion Prediction

Unit

Image shift estimationEx,Ey


Reference Image

Figure 4.10: TEM simulator connection

36


4.6 Conclusion

The proposed drift correction model is presented in this chapter. Fundamental components includesensor (image acquisition and image processing), actuator (stage motion system) and controller(motion prediction) are explained, the implementation mechanisms and pseudo codes are given aswell.The overall model is built in Matlab/Simulink environment. The execution of the model is calledfrom a script, which allows for modi�cations between di�erent system con�gurations and di�erentdisturbances. The screen-shot of the simulink models are given in �gure 6.9 and 6.10. In nextchapter, we are going to perform several experiments based on our simulink model. The purpose ofthe tests is to validate the correctness and performance of the proposed closed loop compensationmethod.

37

Chapter 5

Simulation and Evaluation

In this chapter, we will simulate and evaluate the proposed closed-loop motion compensation method.All simulations were conducted using the constructed model in Simulation. The performance of theproposed method is tested in di�erent imaging modes and under di�erent types of disturbancese�ects.

5.1 Metrics for Evaluation

To evaluate the performance, it is straight forward to give a comparison between reference imageand closed-loop image. But from a control perspective, the image comparison is not enough. Itcan only be referred to as the correctness checking but lacks information about response time anddisturbances rejection ratio which are critical factors for measuring the closed-loop performance.In order to give an intuitive measurements for correctness checking and in-depth measurements forperformance checking, we propose the following metrics:

5.1.1 Closed-loop image versus Reference image

Numbers of image comparison metrics are available to give quantitative indicators about di�erencebetween images. In our experiments, the closed-loop image and reference image are compared usingroot mean square error (RMSE). Low mean square error indicates high similarity between images.The mathematical formulas are given below:

RMSE =

√∑Mm=1

∑Nn=1 [R (m,n)− I(m,n)]2

MN(5.1)

Where R is the reference image, and I is the closed-loop image. M is the number of pixels per rowand N is the number of pixels per column.

5.1.2 Drift versus Stage Movement

As discussed before, motion compensation action is taken in a �xed sample step during imageacquisition process. To validate the performance of the compensation correction and stability ofthe closed loop against di�erent disturbances frequency, the curve that presents the stage positionversus drift is included.

38

CHAPTER 5. SIMULATION AND EVALUATION

5.2 Experiment Setup

5.2.1 Simulation Environment

All experiments are carried out in the Matlab/Simulink 2012a environment. To facilitate the exper-iments, a short script is written to set the imaging con�gurations and control the running fashion ofthe model (single running or batch fashion). Taking advantage of Matlab functions, the simulationdata can be easily logged for further processing and analysing.

5.2.2 Experiment Configuration

The tables 5.1 , 5.2 , 5.3 and 5.4 list the simulation settings. For each experiment, the imaging con-�gurations (imaging mode, imaging time) and disturbances types (linear or non-linear, dynamicalbehaviour) are de�ned. We �rst apply linear drift in each imaging mode for correctness checking.Then a step response simulation is taken to evaluate the closed-loop performance in SEM. Finally,non-linear drift with variety frequencies are added to the SEM simulator to acquire the disturbancesrejection ratio.

A. TEM

Table 5.1: TEM experiment con�gurations for linear drift

Drift Type drift parameters Frame time Frame numbers

Constant Speed 10 pixels/second 0.1 second 50

B.SEM

Table 5.2: step response test for SEM

Drift Type drift parameters dwell time simulation time

Step function Amplitude 50 pixels 5.00E-07 2 seconds

Step function Amplitude 50 pixels 5.00E-06 20 seconds

Table 5.3: Linear speed drift for SEM

Drift Type drift parameters dwell time simulation time

Linear Drift 50 pixels/second 5.00E-07 1.5 seconds

Linear Drift 15 pixels/second 5.00E-06 16 seconds

39


Table 5.4: Experiment for rejection ratio test

Drift Type drift parameters dwell time frame time simulationtime

Non-linear Drift(Sine function) (1e-3 5)Hz, 200pixels amplitude

5.00E-07 0.13s 1/f

Non-linear Drift(Sine function) (1e-3,0.5)Hz,200pixels amplitude

5.00E-06 1.3s 1/f

5.3 Correctness Evaluation

5.3.1 TEM Simulation Result

In this section we discuss the closed-loop performance for TEM, table 5.1 lists the experimentcon�gurations. As depicted in �gure 5.1, the �rst row images shows the drift with a linear speedat di�erent frame instance with respect to time. The �nal image is displaced about 50 pixels inboth directions with respect to the reference image. The second row images are corrected using realtime motion compensation method proposed before. The e�ectiveness of the compensation e�ort isclearly visible from �gure 5.1 and 5.2. Table 5.5 gives the quanti�cation about non-correction andcorrection results by means of image comparison. Obviously, the correction image shows low RMSEwhich indicates a high similarity with reference image.

Frame number#1(reference) #25 #50

Non correction

Correction

Figure 5.1: TEM linear drift result: �rst row images show the result without correction. Secondrow images show the result that applying closed-loop compensation.

Image Measurement

Table 5.5: Result for image comparision

Image frames RMSE Drift shifts(pixels)

Non-correction(25) 71.11 23.89


Correction(25) 7.84 0.91

Correction(50) 7.84 0.91

Real Drift VS Predicted drift(opposite to stage position)The plot 5.3 shows the drift process and the stage movements(opposite direction). Ideally, the

40


Non correction Correction

Figure 5.2: Di�erence image between aligned image and reference image, non-algned image andreference image: Value approach to zero occurs where the reference image has a feature that notpresent in the acuqired image.

drift process should be fully tracked via the stage movement(opposite), and the error between driftestimation and real drift should converge to zero eventually. However, in simulation, the predicteddrift oscillates around the real drift, and the error never converges to zero. Reasonable explanationsgo into two folds:(1) Image processing resolution accuracy: It is expected the image processing unit can generatethe exact error between real drift and predicted drift. The cross-correlation is ideal for uniformtranslations, but for large shift in deformed image, features may move out of �eld of interest andthe estimation result is no longer consistent.(2) Model accuracy limitation: The overall model is designed with pixel accuracy. Any sub-pixelcalculation or manipulation is rounded to the nearest integer value, thus, sub pixel shift cannot becompensated.

Figure 5.3: Linear drift TEM

Blurring e�ectAs mentioned before, image averaging is frequently applied to increase signal to noise ratio in TEM.Under drift e�ect, the image su�ers severe blur in both directions after applying image averaging.With the motion compensation correction, the blur defect is eliminated which results in a stable

41


image,meanwhile, the signal to noise ratio is also improved compared with the correction image 5.1.Again, it con�rms the correctness of the method.

Correction Non-correction

Figure 5.4: Linear drift: blur image

Quanti�cation:

Table 5.6: Result for averaging image comparision


Non-correction 51.26 22.92

Correction 5.41 2.42

5.3.2 SEM Simulation Result

A. Step ResponseFrom a control perspective, a step response is the system output response when their inputs changefrom 0 to 1 in a very short time. Knowing the step response will give information about the stabilityof the closed loop control system. In our case, a step response refers to apply pixels shift to referenceimage at time 0 and see how the closed loop response. The images 5.5 are the step response results.At time zero, the reference images are shifted with 50 pixels in horizontal and vertical direction. Afteracquiring about 14 images, the error between the reference image and scanning image is converge tocertain point which indicates the system reaches the stable state. The �gure 5.6 and 5.6 show thestep response for fast scan and slow scan respectively. Let us discuss the result with the followingaspects:I. Settling timeThe settling time is the time elapsed from initial point to the time when the error between theoutput and controlled variable entered into a stable band. In our case, it refers to the time elapsedto fully tracking the drift. In control theory, the settling time can be changed by tuning the feedbackgain. As in our case, the frame time can be treated as the gain. Namely, the shorter the frame timeis, the larger the gain is and the shorter the settling time elapsed. It is shown in the �gure 5.6 and5.7, the fast scan takes about 1.6 seconds to be settled compared to the slow scan which is about 20seconds.II. Stable stateWhen entering stable band, we expect the error equals zero which implies the drift has been fully

42


#1(reference) #2 (stpe) #5 #14

Fast scan

Slow scan

Figure 5.5: SEM step response

Figure 5.6: SEM step response:fast scan

Figure 5.7: SEM step response: slow scan

tracked. In simulation result, the error between drift and controller outputs never converges to zero.As discussed before, one possibility is the inaccuracy of image processing computation and anotherlies in the resolution limitation of the model.III. overshootThe overshoot refers to the output value exceed its reference value. In our closed-loop system, itrepresents a over-correction of stage position. As shown in �gure 5.7 and 5.6, the overshoot is about50 pixels for slow scan and 40 pixels for fast scan. According to the control law given in chapter3, the �rst stage position adjustment happens as soon as �rst frame acquired, the calculated setpoint equals to the �rst drift measurement which is 50 pixels in our case, then linear-interpolationcompensation starts working. As a result, overshoot is inevitable in the step response. Theoretically,the overshoot can be reduced by weaken the drift speed correction. The drift speed correction isformalized in equation 5.2, the Kp is set to 1 in our controller design. We can imagine that if Kp isless than 1, the overshoot will be reduced, which is shown in �gure 5.8, where Kp is set to 0.2, andthe overshoot is reduced approximately 20 pixels in fast scanning. However, it is clearly identi�ed

43


from the �gure that the settling time is increased caused by the weaken speed alignment. Thereforewe can conclude that tuning parameter Kp is a trade-o� between settling time and overshoot, andcarefully set Kp will give a better performance of the closed-loop system.

Driftspeed = Kp ∗Driftmeasuremnt

Tframe(5.2)

Figure 5.8: Overshoot redution in fst scan

B. Linear DriftFigures 5.10 shows the simulation result with a linear drift in scanning mode. First takes a look atthe non-correction image series. Due to long dwell time, it increases the likelihood that the speci-men move with respect to beam spot during scanning, thus slow scan images su�er heavy shift anddistortion compared to fast scan. From table 5.7 and 5.8, the correctness is identi�ed given by lowRMS and extreme small shift compared to non-correction image.

#1 (reference) #5 #10

Non-correction

correction

#1 reference #5 #10 Frame number

Non correction

Correction

Fast scan Slow scan

Figure 5.9: SEM linear drift result

Image ComparisionLinear drift VS stage movement

44


Table 5.7: Result for image comparision in slow scan




Correction(5) 53.50 -6.06

Correction(10) 16.77 -0.74

Table 5.8: Result for image comparision in fast scan




Correction(5) 29.76 -2.04

Correction(10) 12.91 -0.09

Figure 5.10: SEM linear drift result

C. Disturbances rejection ratio analysisRather than linear or step behaviour, the drift disturbance usually shows non-linear properties inrealistic environment. In this section, we are going to explore how performance varies with di�erentfrequencies disturbances , which often regards as the disturbances rejection ratio analysis. The ex-periment settings can be found in table 5.4. In simulation, we choose the Sine wave as the non-lineardisturbances function, its frequency varies from 1e-3 Hz up to 10 Hz. Under each frequency, the rootmean square error value (shown in equation 5.3) between real drift and estimated drift(( ˆdrift)) iscalculated, and the result is as shown in �gure 5.11 and 5.12:

RMSE( ˆdrift) =√MSE( ˆdrift) =

√E(( ˆdrift− drift)2) (5.3)

Each data point indicates the drift prediction performance obtained by calculate the root meansquare value of error data between real drift and predicted drift. With ideal compensation, the truedrift is expected to be equal to the estimated drift which results in a zero rms value. In reality, theprediction scheme shows a low pass �lter characteristic. The disturbances, at low frequency downto sub Hz, is well estimated, but the predictor lost traceability at high frequency disturbances. Thisis due to the low pass property of �rst order hold predictor.By comparing the sensitivity plot between fast scan and slow scan, it can be concluded that thedisturbances rejection ratio is inverse proportional to the dwell time. Given small dwell time, highsensor sample rate is guaranteed, therefore, high frequency disturbance can be measured and com-

45


pensated.

Figure 5.11: Sensitivity plot: fast scan with a dwell time of 5e-7 second

Figure 5.12: Sensitivity plot: slow scan with a dwell time of 5e-6 second

46


5.4 Conclusion

To test the performance of proposed closed-loop drift compensation strategy, di�erent experimentsare tested in chapter 6. An linear drift experiment for TEM and SEM shows the correctness ofthe closed-loop strategy given by low RMSE value and less than 1 pixel image shift. For the stepresponse test in SEM, the closed-loop system performance is evaluated. The conclusion goes intotwo folds:(1) The settling time is linear proportional to the dwell time, for instance, the settlingtime of scan(tdwell=5e-7s) is 1.6 seconds which is 10 times faster than a slow scan(tdwell=5e-6s) of 16seconds. (2) The overshoot can be reduced by weaken the drift speed correction, but su�ers incrementof settling time. By analysing the disturbances rejection ratio, the RMSE value between predicteddrift and real drift are measured. The real drift is modelled by sine wave function with di�erentfrequencies. The result reveals that the closed-system has a low-pass nature and its disturbancesrejection ratio is inverse proportional to the dwell time. For instance,with a fast scan of dwell timeis 5e-7 seconds, the disturbance with frequency under 0.5 Hz is well estimated and compensated,whereas a disturbance with frequency under 0.05 Hz will be well compensated for a slow scan ofdwell time 10 times slower than fast scan. Given this result, the disturbances rejection ratio willbe determined by the dwell time, thus, any kind of disturbances with di�erent frequency can becompensated using our strategy when dwell time is well chosen.

47

Chapter 6

Conclusion

6.1 Discussion

Drift is a major disturbance in electron microscopy. It causes unintended and non-linear movement ofthe specimen during image acquisition process and leads to distortion or blurring image. Nowadays,the most common way to eliminate the drift e�ect is to wait until the thermal stable state is reached,which is not e�cient. In literature, some image processing techniques are incorporate for image postprocessing purpose. Such proposed method yields positive results only with small shift betweenimage frames. The result is not valid concerning large shift, especially when the object is move outof the interest �eld. To solve the problem and enhance the performance of electron microscopy, weproposed an on-line closed loop motion compensation strategy to deal with drift, the developmentare based on two ideas:

• Image-based sensing: The drift measurement is done by comparing a pairs of images by meansof cross correlation which translates the movement to pixels shift.

• Motion prediction controller: Given the measured drift in current sample step, the motionprediction controller estimates the drift in next sample step, updates the stage motion speedand decides the stage position. By adjusting the stage position towards opposite directionwith respect to drift during image acquisition, the drift shift is eliminated and the object willstay within point of interest.

The experiments results con�rm our original expectations. The proposed method is robust againstdi�erent kinds of imaging mode and various system con�gurations. The corrected image qualityshows a great improvement compared to the uncorrected images. However, the model also ownslimitations:

• From the disturbances rejection ratio analysis, we can conclude that the strategy is only validat lower disturbances frequency range, normally down to sub Hz. For a slow scan with dwelltime of 5e-6 second, the disturbance with frequency lower than 0.05 Hz is well compensated.For a faster scan with dwell time of 5e-7 second, the disturbances rejection frequency is increaseto 0.5 Hz.

• Though the specimen stays with in the �eld of view during image acquisition, but local �nershifts still exist. From control view, the error between tracked disturbance and stage movementnever converge to zero. This can be explained with the image processing inaccuracy and modellimitations.

Given those limitations, future work directions will be provided in next sections

48

CHAPTER 6. CONCLUSION

6.2 Future work

Removal drift e�ect during image acquisition process is a long term research for the future electronmicroscopy product. The proposed method in this thesis presents a research direction adaptingreal time motion compensation method. There are plenty areas which could yield performanceimprovement for the proposed approach. The proposed future investigations are grouped into twocategories:

Towards Model

• All image manipulations are restricted to pixel accuracy, including image shift, image process-ing and image compensation. The purpose of doing this is to simplify the model and improvingthe simulation speed. As a result, it limited the performance of the method. As shown in thesimulation result, sub pixel shift cannot be compensated due to this restriction. One workingdirection is to update the model with sub pixel accuracy.

• As mentioned before, the long exposure time will result in a blurring image in transmissionmode. In the model, we only consider the image shift in TEM mode, the image blurringoccurred only after image averaging. For the sake of the completeness and concreteness, it iscrucial to incorporate the drift e�ect under long exposure time con�guration.

• In the model construction procedure, we focused ourselves on the compensation strategy im-plementation and validation with a control perspective, thus, all kinds of computation andcommunication time are neglected. In reality, timing constraints play an important role in anykind of real time system. Since the operation in electron microscopy is operated in sub secondsor even down to Nano seconds, any event of delay could degraded the overall system perfor-mance. For instance, delays in image processing calculation or lags in data transmission willpostpone the sensor data. As a result, the closed loop settling time is increasing or even loststability. To facilitate the timing analysis, delays can be easily added between the functionalblock in the current model. Future work is to formalize the timing requirements using timeautomaton or hard real time analysis. Also generate the timing requirements speci�cation forhardware implementation (e.g. communication delay,computational delay in hardware).

Towards compensation strategy

• Sensor Aspects: From control perspective, improve the sensor acquisition time and accuracyis key method to promote the closed loop performance. In the proposed method, Image basedcontrol is the core. Compared to the hardware measurement, the image based measurementis more sensitive to the overall system con�gurations. For instance, the system con�gurationwill determine the frame time and further determine the sensor data acquisition time. In ourmodel, the sensor date acquisition is dominated by the frame time, and the accuracy dependson image processing. For future research, we propose following direction:

I. Advanced image processing

First step towards image processing improvement is to apply some more algorithms such asblock matching which measures the local shift in prede�ned block rather than global measure-ment.

The on line compensation can guarantee the specimen stay within the �eld of view duringimage acquisition, but locally �ner movement are unable to eliminate. Future work can makea two tier image processing architecture, �rst tier is to perform on-line compensation andsecond tier is to do image alignment after image acquisition o�-line.

• Actuator Aspect: To achieve the drift compensation purpose, stage motor is used as theactuator in the proposed method. Given the set point of the stage, it can trace the drift

49


movement and compensate it by moving towards the opposite direction with respect to drift.However, the stage motor is not the only choice, other unit such as beam de�ectors is alsofeasible, which are used to de�ect the electron beam during image acquisition process. We canimagine the new loop as a photography scenario. The stage is the background and the cameralens is the beam de�ectors. We can adjust the point of view by either changing the locationof background or adjusting the lens position. In the current loop, shift lens is not available.By incorporate beam de�ectors, we can construct a two level closed loop strategy, a possibleschematic is proposed in �gure 6.1.

Image Acqusition Image Processing Motion Prediction

Stage motor

Beam deflectors

Stage pos

Beam pos

Drift Disturbances +

Figure 6.1: Proposed new closed-loop schematic

• Controller Aspects

I. Control Algorithm

The control algorithm we developed follows the idea of prediction. Meanly, take the currentmeasurement into account, future drift dynamics is estimated. Since the drift dynamics isunknown to us, the controller behaves as a signal reconstruction and the estimation error isuncontrollable. The �rst step towards future work is to learn the dynamic behaviour of thedrift and formalize it using system identi�cation technique. Then some modern control theoriessuch as Kalman �ltering and Modern predictive control[11] can be adapted to the problem.Using the receding horizon principle, the estimation error can be minimized and the overallperformance will be improved.

II. Updating strategy

In the current method, the shift vector is measured between deformed image and referenceimage. Other alternative such as consecutive images comparison is also worthy investigation.We can imagine the relative shift between consecutive images is smaller than the shift betweendeformed image and reference image over long period of time. The image processing accuracywill be improved by taking new updating strategy.

6.3 Final words

The objective of the project is to establish an accurate, drift-free and stable image by the strategy ofthe so-called �st-order hold prediction for closed-loop compensation in electron microscope. The typeof correction method which presented in the thesis can also be applied to reject other disturbanceswith high frequency, using a well-chosen image processing method and fast sampling rate. We really

50


hope this work could point out a right research direction for the followings and could be applied inreal system in the near future.

51

Appendix A

Figure 6.2: Optics Con�guration process usecase1:Magni�cation settings

Figure 6.3: Optics Con�guration process usecase2:Focus settings

52

Appendix A

Figure 6.4: Viewing Navigation Process usecase1:Large scale stage movement

Figure 6.5: Viewing Navigation Process usecase2:small scale stage movement

Figure 6.6: Viewing Navigation Process usecase3:Electron beam de�ection

53

Appendix A

Figure 6.7: Imaging Acquisition Process usecase1:TEM mode

Figure 6.8: Imaging Acquisition Process usecase1:SEM mode

54

Appendix B

Scanni

ng Pa

ttern G

enerato

r

Scanni

ng Mo

de Se

lection

crop_i

ndex_x

crop_i

ndex_y

ori_ind

ex_x

ori_ind

ex_y

Scanni

ng En

gine

index_

x

index_

y

close_

horizo

ntal_c

oordin

ate

close_

vertica

l_coor

dinate

open_h

orizont

al_coo

rdinate

1

open_v

ertical

_coord

inate2

close_

Pixel_

stream

open_P

ixel_s

tream1

Refere

nce_im

age_lo

ading

Refen

rence

Image

Piezo

Actuat

or Syst

em

pos_x

pos_y

res_x

rex_y

Motion

_Com

pensat

ion

drift_h

or

drift_v

er

stagep

os_hor

stagep

os_ver

close_

hor

close_

ver

open_h

or

open_v

er

Motion

Predi

ction U

nit

In1 In2

Out1

Out2

Out3

Image_

assem

bling

index_

x

index_

y

close_

Pixel_

stream

open_P

ixel_s

tream

Closed Op

en

Image_

Proces

sing_U

nit

im_pro

cessin

g_58

Disturb

ance

Index_

x

Index_

y

shift_h

or

shift_v

er

drift_h

or

drift_v

er

Const

ant1

double

[1x400

01]

double

[1x400

01][1x

40001]

int8 int8

double

[1x400

01] [1x400

01]dou

ble [1x

40001] [1x400

01]

double

(2)

2

double

double

[1x400

01]

[1x400

01]

[1x400

01]

[1x400

01]

uint32

[1x400

01]

[1x400

01]

[1x400

01]

uint32

[1x400

01]

double

double

[512x5

12]

double

[1x400

01]

[1x400

01]

double

[1x400

01]

[1x400

01]

double

[1x400

01] int8 (2)

2

int8 int8

double

[1x400

01]

[1x400

01]

double

[1x400

01]

[1x400

01]

double

double

double

doubledou

ble

double

[512x5

12]

Figure 6.9: SEM simulator

55

Appendix B

Refer

ence_

Imag

e_Lo

ading

yTe

st_ima

ge_lo

ading

Piezo

Actua

tor Sy

stem1

pos_x

pos_y

res_x

rex_y

Motion

_Vect

or_ge

nerat

or

Drift_

motion

_vecto

r(x,y)

Motion

Pred

iction

Unit

x_est

y_est1

x y s

Imag

e_Pro

cessin

g

refere

nce

Fast_

Scan

_imag

e

Motion

_vecto

r_x

Motion

_vecto

r_y

Imag

e_Ac

quisit

ion_P

lant

Refer

ence_

Imag

e_in

Exter

nal D

isturba

nce

vx vy

Refer

ence_

image

_out

defor

med_

image

alignm

ent_im

age

motion

_gen

erator

Figure 6.10: TEM simulator

56

Bibliography

[1] R. Gonzalez and R. Woods. Digital Image Processing. USA: Addison-Wesley, 1992..

[2] FEI company. An introduction of electron microscopy. FEI company, 2010.

[3] Chad N.Himmel. Concrete pier foundation and building desigh for electron microscope installa-tions with vibration disturbances from train, truck, and bus tra�c. inter noise, 2010.

[4] Michael T. Snella. Drift Correction for Scanning-Electron Microscopy. MIT, 2010.

[5] M.A.O'Keefe, B.Parvin, D.Owen, J.Taylor, K.H.Westmacott, W.johnston, U.Dahmen. Automa-tion for on-line remote-control in situ electron microscopy. . Scanning Microsc.11,229-239(1997),

[6] Alina Tarau, Pieter Nuij , maarten Steinbuch. Hierarchical ocntrol for drift correction in trans-mission electron microscopes. . Processings of the 19th IEEE international Conference on Controlapplications,(2011) pp. 351-356

[7] Arturo Tejada , ArnoldJ.denDekker , WouterVandenBroek. Introducingmeasure-by-wire,the sys-tematic use of systems and control theory intransmission electron microscopy. Ultra microscopy,2011.

[8] Douglas Lyon. The Discrete Fourier Transform, Part 6: Cross-Correlation. JOURNAL OFOB-JECT TECHNOLOGY, 2010.

[9] http://www.mathworks.nl/products/simulink. MathWorks.

[10] Wolfgang Huber Visualisation of genomic data with the Hilbert curve. Wolfgang Huber, Brixen2011.

[11] Abed C. Malti (a), Sounkalo(a), Nadine Piat(a), Claire Arnoult (b), Naresh Marturi (a) Towardfast calibration of the global drift in scanning electron microscopes with respect to time andmagni�cation. FEMTO-ST/AS2M,

[12] Yih-Chuan Lin , Shen-Chuan Tai. Fast Full-Search Block-Matching Algorithm for Motion-Compensated Video Compression.. IEEE TRANSACTIONS ON COMMUNICATIONS, VOL.45, NO. 5, MAY 1997,

57

Documents

Eindhoven University of Technology MASTER Simulation of ... · Simulation of drift control for an electron microscope stage using image processing in the loop Feng, F. Award date: