A Probabilistic Approach toConceptual Sensor Modeling

Examensarbete utfort i Bildbehandlingvid Totalforsvarets forskningsinstitut

och Tekniska Hogskolan i Linkoping av

Mattias Sonesson

Reg nr: LiTH-ISY-EX-3428Linkoping 2004

A Probabilistic Approach toConceptual Sensor Modeling

Examensarbete utfort i Bildbehandlingvid Totalforsvarets forskningsinstitut

och Tekniska Hogskolan i Linkoping av

Mattias Sonesson

Reg nr: LiTH-ISY-EX-3428

Supervisor: Lars Forssell

Examiner: Klas Nordberg

Linkoping 18th May 2004.

Avdelning, InstitutionDivision, Department

Institutionen för systemteknik581 83 LINKÖPING

DatumDate

2004-05-18

SpråkLanguage

RapporttypReport category

ISBN

Svenska/Swedish

X Engelska/English

Licentiatavhandling

X Examensarbete ISRN LITH-ISY-EX-3428-2004

C-uppsats

D-uppsats Serietitel och serienummerTitle of series, numbering

ISSN

Övrig rapport

____

URL för elektronisk versionhttp://www.ep.liu.se/exjobb/isy/2004/3428/

Titel

Title

A Probabilistic Approach to Conceptual Sensor Modeling

Författare Author

Mattias Sonesson

SammanfattningAbstract

This report develops a method for probabilistic conceptual sensor modeling. The idea is to gener-

ate probabilities for detection, recognition and identification based on a few simple factors. The

focus lies on FLIR sensors and thermal radiation, even if discussions of other wavelength bands

are made. The model can be used as a hole or some or several parts can be used to create a sim-

pler model. The core of the model is based on the Johnson criteria that uses resolution as the

input parameter. Some extensions that models other factors are also implemented. In the end a

short discussion of the possibility to use this model for other sensors than FLIR is made.

NyckelordKeyword

Johnson criteria, probabilistic, conceptual, sensor model

Abstract

This report develops a method for probabilistic conceptual sensor modeling. Theidea is to generate probabilities for detection, recognition and identification basedon a few simple factors. The focus lies on FLIR sensors and thermal radiation,even if discussions of other wavelength bands are made. The model can be usedas a hole or some or several parts can be used to create a simpler model. Thecore of the model is based on the Johnson criteria that uses resolution as the inputparameter. Some extensions that models other factors are also implemented. Inthe end a short discussion of the possibility to use this model for other sensors thanFLIR is made.

Keywords: Johnson criteria, probabilistic, conceptual, sensor model

i

ii

Acknowledgment

This thesis was done at the Autonomous Systems department, Division of SystemsTechnology, at the Swedish Defence Research Agency, to whichs staff I would liketo express my gratitude.

Specially I would like to thank my supervisor Lars Forssell, for showing me how“things are done” and boosting my self esteem. Thanks to Leif Hagstedt for hishelp with practical details, computers and other stuff. Peter Stromback for cheeringme up when motivation was low, with software and many fruitful discussions.

Finally, I express my gratitude to all the guys still fighting the battles in Riss-nehallen.

iii

Abbreviations and acronyms

C2NVEO United States Army Command Center for Night Vision and Electro-OpticsCCD Charge Coupled DeviceCycle Two adjacent lines of pixels, used in target to probability transformsFIR Far Infra RedFLIR Forward Looking Infra RedFOV Field Of ViewGPS Global Positioning SystemHot spot Strong signal source on a uniform backgroundLOS Line Of SightLP Line Pair, equivalent to cycleLP/TGT Line Pair per targetNIR Near Infra RedOSIC Optimal Sensor Integrated ControlRL Reinforced LearningSCR Signal to Clutter RatioSNR Signal to Noise RatioTIR Thermal Infra RedUAV Unmanned Aerial VehicleUV Ultra Violette

iv

Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Modeling 52.1 General concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Classic sensor modeling . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Preliminary list of contributing factors . . . . . . . . . . . . . 62.3 Conceptual sensor modeling . . . . . . . . . . . . . . . . . . . . . . . 72.4 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4.1 Decomposing the system . . . . . . . . . . . . . . . . . . . . . 82.5 Initial approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.6 Noise is not always noise . . . . . . . . . . . . . . . . . . . . . . . . . 92.7 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.7.1 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.7.2 Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.7.3 Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.8 Refining the approach . . . . . . . . . . . . . . . . . . . . . . . . . . 102.8.1 Necessary factors for detection . . . . . . . . . . . . . . . . . 102.8.2 Signature characteristics . . . . . . . . . . . . . . . . . . . . . 102.8.3 Decision making . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 System decomposition 133.1 Size does matter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Inverse square and cube laws . . . . . . . . . . . . . . . . . . . . . . 133.3 Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.4 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.5 The Johnson criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.6 Extensions to the Johnson criteria . . . . . . . . . . . . . . . . . . . 193.7 Adding another factor to the equation . . . . . . . . . . . . . . . . . 203.8 Are the results only applicable for detection probabilities? . . . . . . 22

v

vi Contents

3.9 Hot Spots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4 Development of a model for the effects of noise 254.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.2 Preparations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.3 Conducting the experiments . . . . . . . . . . . . . . . . . . . . . . . 264.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.5 Suggestions for further studies . . . . . . . . . . . . . . . . . . . . . . 26

5 Sensors 295.1 TV/Video sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.2 History of IR sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.2.1 Reticles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.3 Pinhole camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.4 Advanced camera model . . . . . . . . . . . . . . . . . . . . . . . . . 315.5 Rayleigh criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.6 Further enhancement of the sensor model . . . . . . . . . . . . . . . 32

6 Signal generation 336.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.2 Blackbody radiation . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.3 Interaction with materials . . . . . . . . . . . . . . . . . . . . . . . . 346.4 Optical windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.5 Thermal variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.6 Calculating the clutter . . . . . . . . . . . . . . . . . . . . . . . . . . 366.7 Vegetation model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386.8 Further enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . 38

7 Atmosphere 397.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.2 Scattering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.3 Absorption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407.4 Merging absorption and scattering . . . . . . . . . . . . . . . . . . . 417.5 Rain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417.6 Snow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427.7 Dust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427.8 Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8 Information and data fusion 458.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458.2 Different kinds of fusion . . . . . . . . . . . . . . . . . . . . . . . . . 468.3 Fusion modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Contents vii

9 Summary and conclusions 499.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519.4 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

9.4.1 Other sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.4.2 Visual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.4.3 Radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.4.4 Laser radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

List of Figures

3.1 Principals of angles in the FOV . . . . . . . . . . . . . . . . . . . . . 143.2 A scene with a box. . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.3 Target to LP transformation . . . . . . . . . . . . . . . . . . . . . . 173.4 Target equivalent bar patterns . . . . . . . . . . . . . . . . . . . . . 183.5 Visualization of the C2NVEO table. . . . . . . . . . . . . . . . . . . 203.6 Example of cells in a scene with two black targets. . . . . . . . . . . 213.7 Detection probabilities for different clutter levels . . . . . . . . . . . 23

4.1 Scenes used in acquisition experiments. . . . . . . . . . . . . . . . . . 274.2 Examples of images used in the acquisition experiments. . . . . . . . 28

5.1 Layout of pinhole camera. . . . . . . . . . . . . . . . . . . . . . . . . 31

6.1 The radiation from a blackbody of different temperatures. . . . . . . 34

7.1 An examples of scattering rays. . . . . . . . . . . . . . . . . . . . . . 40

8.1 Different fusion methods. . . . . . . . . . . . . . . . . . . . . . . . . 47

List of Tables

3.1 The Johnson table of cycles . . . . . . . . . . . . . . . . . . . . . . . 183.2 Resolution per level of acquisition . . . . . . . . . . . . . . . . . . . . 193.3 The C2NVEO LP to probability transform . . . . . . . . . . . . . . 193.4 Probability of detection in clutter . . . . . . . . . . . . . . . . . . . . 223.5 Detection probabilities for different clutter levels Pdet [15][11]. . . . . 22

6.1 Reflectance for some materials . . . . . . . . . . . . . . . . . . . . . 35

7.1 Extinction Coefficient of Atmospheric Obscurants (low altitude). . . 427.2 Explanations to Table 7.1. . . . . . . . . . . . . . . . . . . . . . . . . 42

viii Contents

Chapter 1

Introduction

1.1 Background

This work is done as a Master Thesis at the System Technology Division of TheSwedish Defence Research Agency (FOI). The background of the thesis originatefrom the needs in two different research projects, Collaborating Missiles and Pre-cision Engagement, conducted on behalf of the Swedish Armed Forces at the Au-tonomous Systems Department. Within the two projects, the need for models ofdifferent types of target seeking sensors have been foreseen, primary focused towardthe image generating type, i.e. IR/TV-sensors. Some of the future planned activi-ties within the projects are research within the field of Optimal Sensor IntegratedControl (OSIC), which can be described as designing controllers that enable thesensors to get a optimal view of the target, or studies on artificial intelligence algo-rithms like Reinforced Learning (RL). Common for all these activities is the needof a accurate sensor model, suitable for use in conjunction with dynamical modelsof different types of vehicles, in technical and or tactical simulation emphasizingthe beneficial that are possible from a system perspective.

Physical validated models of target seekers are unfortunately very complex,detailed and computational expensive and therefore of little practical use in simu-lations on a higher system level. An alternative approach is the one used within thewar gaming community, see [4], based on probabilistic modeling. This, approach,however does not have enough fidelity to be used in research in the field of controlsystems for autonomous vehicles. There is therefore an urge of a novel approachbased on a conceptual approach and on physical insight in the detailed behavior ofadvanced image generating sensors and the signal processing associated with thetarget detection algorithms.

This master thesis is a first step in a development plan which aims to be ableto model most of the existing types of sensors, both passive and active, in differentfrequency bands. The thesis is aimed at investigation of the possibility to enhanceprobabilistic sensor models in order to get an increased accuracy without increasingthe computational complexity associated with execution of the model.

1

2 Introduction

1.2 Objectives

The objective in this thesis is to propose a methodology for developing conceptualmodels and produce an model for different kinds of sensors on a system level. Manymodels exist today but almost all are very low level and based directly on the lawsof physics. This means that they become very detailed and complex.

Therefore it is vital that a new sensor model is produced that is not so detailedand complex as the near-physics models, and not as blunt as the existing near-system models.

Hopefully this IR-sensor model can lead the way for a new package of modelsfor passive sensors, ranging from UV1 to IR2, and active sensors like radar andlaser radar.

1.3 Limitations

The first attempt was to construct a model of a FLIR3 sensor mounted on anaerial vessel looking down on stationary ground objects. But it soon turned outthat the concepts used are valid for many other types of sensors, mostly other TV-like sensors, looking in other directions than down towards the ground. It couldalso be used to look at objects in the sky from an observer ground based (theopposite of what was initially planned).

Only stationary objects are considered in this report, since it seems as if movingobjects are always detected in IR. The recognition and identification phases does,on the other hand, not gain much information from the fact that an object ismoving, and some suggestions are made on how to model moving objects.

Although all necessary factors might not be included in the model, sometimesintentionally, sometimes not, hopefully enough factors to make the model valid incontext are included. The basic idea is to make it simple, but still with a higherfidelity than the existing probabilistic sensor models used today.

The intention is not to calculate all factors separately, but rather try to showthe possibility of merging factors into other descriptive units.

The entire model is based on human acquisition models. This is of course alimitation and a degrading factor of its validity. However, as stated below, theproblem of modeling the signal processing algorithms leads to the choice of humanacquisition as the foundation. Modeling sensors must take its start somewhere,and this might be it. Other models should be compared to this one in order for itto really reach its full potential. In that way, the conceptual model could be tuned(by changing a few parameters) to model certain sensors better.

1Ultra Violette, short wavelenght light2Infra Red, long wavelength light3Forward Looking Infra Red

1.4 Outline 3

1.4 Outline

The initial chapters analyzes sensor modeling, then in the following chapters thedifferent properties are modeled. The basics rules of resolution should always beimplemented. The validity of the model can then increased by adding the effectsof another chapter. The structure of the report is:

Chapter 2 (Modeling) is an introduction to modeling and efforts are made toshow some of the difficulties in modeling, as well as setting up the rules of the gameand define the concept of conceptual modeling.

Chapter 3 (System decomposition) analyzes the capabilities of detection, recog-nition and identification, and how they are affected by the background.

Chapter 4 (Development of a model for the effects of noise) discusses and at-tempts to develop a model for the effects of noise in human acquisition. Theattempted model is coherent with the models developed in earlier chapters. Detec-tion, recognition and identification of objects are investigated through experiments,to complete the overall model.

The following chapters tries to analyze the physics involved in the sensor aswell as draws conclusions of what to model and how.

Chapter 5 (Sensors) starts calculating one of the descriptive features discussedin Chapter 3, namely the resolution. This would be the basics that should beimplemented into any model.

Chapter 6 (Signal generation) continues with the descriptive features. Thesignal strength and some geometric properties of the object are modeled and cal-culated in the signal to clutter ratio. This chapter can be included to build a moresophisticated model.

Chapter 7 (Atmosphere) constructs a model for how the signal is effected by theatmosphere. Rain, fog and snow are some of the situations modeled. To completethe model the effects of the atmosphere should be implemented.

Chapter 8 (Information and data fusion) suggests a model for data and infor-mation fusion that could be used for multiple sensors.

Chapter 9 (Summary and conclusions) takes the experiences and summarizesthem in an attempt to build a conceptual sensor model. A discussion of how tomodel other sensors completes the chapter.

4 Introduction

Chapter 2

Modeling

This chapter contains some basics of modeling and introduces an initial approachfor the conceptual sensor model.

2.1 General concepts

To make a model of a physical object one has to divide it into subsystems or smallerparts, for example shape and function. Each of these smaller parts can in themselves be considered as objects to model and it is therefore possible to divide themfurther into smaller parts. At some level it is no longer meaningful to divide theparts into smaller units. In this stage it is time to concentrate on the function ofeach part.

The models based directly on the laws of physics, the near physics models,are constructed in this way. First a geometrical model is made, in an artificial3D environment. To make it useful some kind of discrimination is made to thegeometrical model, and this is where the real modeling starts. Objects that are bysome standard considered to be unimportant are left out, this might for instance bethe inside of a car or details to small to make a difference. What can be consideredunimportant is of course dependent on the situation, naturally a more detaileddescription of the real world can be more exact than a crude model. At some levelit is not meaningful to increase the complexity of the model because it does notadd any extra information. This is dependent on the governing algorithms of thesimulation.

A simulation of the trajectory of free falling spherical objects can for instance bedescribed by the shape (spherical with radius r), the mass (m), the material (wood,rubber, plastic, ...), the color and so on. But as Galileo stated objects of differentmass fall with the same speed. So the mass can be ignored in our simulation, aswell as the color. Does the shape of the object matter? Well, yes! Different shapesmakes the air stream behave differently around the object. But does this affectour example simulation? It depends on whether the air is a part of the model. Is

5

6 Modeling

the governing algorithm given by Eq. (2.1)? In that case the air is not a part ofthe system and therefor the shape does not matter. Eq. (2.1) limits the necessaryfactors to the initial height, gravity and time (h0, g, t).

h(t) = h0 − gt2

2(2.1)

Another model might however take the air stream into account, and give abetter result, and then the shape of the object is of interest.

In designing a model of a physical object the overall question is for what purposeit should be used.

2.2 Classic sensor modeling

Traditionally sensor modeling has been made either very simple, or very complex.They have been modeled to determine whether an object is present and what it isfrom real data.

One way is to make use of the real signal processing algorithms and try to createan artificial input signal that resembles the real world as closely as possible. Thisis what we call near physics modeling. Creating the signal can be made by somekind of ray tracing, or equivalent technique, for every frame. The frames can thenbe joined to form a sequence that can be used as an input signal or the framescan be used individually. This is a very good technique for trying out new signalprocessing algorithms and for validating a system. The problems are that details inthe input signal, that are important for testing the performance of the algorithms,are expensive in terms of time and computational complexity.

For simulations where the signal processing algorithms are not to be tested thedemand for speed is crucial. In simulation of cooperative components, such assensors and flight mechanics of an UAV, the importance is not whether a certainobject, say a landing field, is determined exactly right at every given moment.What is interesting might be whether it is possible at all to see the landing strip,and approximately when it is detected. A simpler form of sensor modeling canthen be used.

Such simpler sensor models are used today, but they are too simple. In factsome known sensor models, are so simple that they check only the distance to thetarget, and when the distance to the target is smaller than the operating radius ofthe device, the target is detected. All objects within this limit will be found. Thismodel is for some purposes useful, but it certainly has a lot of disadvantages.

2.2.1 Preliminary list of contributing factors

To decide what factors that should be included in the model is a large part of thework. A first glimpse of the problem of modeling forward looking sensors of TVtype and a brainstorming phase will give some ideas of what factors could be used.Some of these factors are:

2.3 Conceptual sensor modeling 7

• range to target

• humidity

• attitude of sensor

• light conditions

• sun reflexes

• line of sight to target

• target temperature

• surrounding temperature (air, ground)

• weather

• area of target

• sensor specifics

• ...

Soon there are so many parameters in the equation that it starts to get outof hand. All these are important, as a lot more are. But how should they becombined? Calculating rays and making a pseudo image requires a signal processingalgorithm do draw conclusions from it. This problem will be addressed later.

2.3 Conceptual sensor modeling

Conceptual modeling is more about trying to fake a system that operates in ap-proximately the same way as the real system, but is a lot cheaper in terms of timeand complexity. Conceptual modeling is to find the essence of a system, and modelit with as few parameters as possible. The output of such a sensor model is not todetermine the answer to a question with a direct answer, but instead try to give aprobability of whether the question could be answered. The decisions are left to ahigher level to make.

2.4 Hypothesis

What is interesting in a signal? How can this interest be quantified? From infor-mation theory we know [14] that information is defined as the deviation from aknown behavior. How can the information in a scene be quantified in a way so thatconclusions of detection, recognition and identification can be drawn.

It is time to start thinking in new patterns. How does the interesting part in asignal differ from the rest of the signal? Can that be measured and modeled?

8 Modeling

2.4.1 Decomposing the system

An estimate of how well a system works depends not only of certain physical factors,but also of what kind of algorithm is used for the signal processing. This leads upto a scenario where not only physical factors but also algorithms must be evaluated.If it is possible to determine some kind of measurement on how well an algorithmworks due to some basic descriptive factors, for instance SNR1 and some other kindof signature characteristics, the sensor model could be reorganized.

A suggestion for a conceptual model could then be . . .

rangehumidity

...

⇒{

SNRsignature characteritics

}

Detection algorithm characteristics

⇒ Probability

where the implication arrows would be the part to concentrate on. The first arrowrepresenting a transformation of the physical properties to some descriptive factors.The second arrow is the calculation of the probability based on the descriptive fac-tors and properties of a detection algorithm. But what are “signal characteristics”and how should it be measured and modeled?

2.5 Initial approach

To start with the signal characteristics are ignored and all efforts are concentratedon deciding the SNR. But what is SNR? The signal to noise ratio is the quotientbetween the energy of the signal (S) and the noise (n1), see Eq. (2.12). The signalis defined as in Eq. (2.2), where x and y represents the pixel coordinates in theimage. Usually the SNR is measured in decibel, a logarithmic scale.

S(x, y) = o(x, y) + n1(x, y) (2.2)

SNR = 10log

∑S2∑n2

1

(2.3)

1Signal to noise ratio

2.6 Noise is not always noise 9

2.6 Noise is not always noise

In Eq. (2.2) the signal is divided into two parts. One part containing the informa-tion we are looking for (object) and the other containing “the rest” (noise). Thiscould be modeled as suggested in section 2.4.1, but a more thorough investigationof the noise property could be worthwhile. In [15] a distinction between clutter andnoise is made. It is stated that temporal noise is stationary, ergodic and Gaussian,as well as time dependent. Clutter on the other hand is non-stationary, non-ergodicand non-Gaussian. The n1(x, y) term is made from the the static background andsurroundings of the object of interest, as well as a distortion term of both the signaland clutter. In principle the object signal (o) has no values where the clutter isdefined, and vice versa.

Based on the statements above lets define noise as temporal and clutter asspatial. Different measures can be used to eliminate their effects. This leads to amodified version of Eq. (2.2), where o(x, y) still is the signal of interest, c(x, y) isthe clutter and n(x, y) is the noise.

S(x, y) = o(x, y) + c(x, y) + n(x, y) (2.4)

2.7 Background

In the specification a model based on three independent probabilities of targetacquisitions was mentioned, the idea is that these relatively easy can be estimatedone by one and joined together as can be seen in Eq. (2.5).

Peffect = Pdetection · Precognition · Pidentification (2.5)

2.7.1 Detection

The general principle of detection is to determine if there are any objects in thescene that could be separated from the noise and clutter. If there are, these willbe investigated further in terms of recognition and identification.

2.7.2 Recognition

If the detection phase indicates that there is something in the scene it is now up tothe recognition (or classification) stage to determine what it is. Is it an aeroplane,a car, a house? Here the categorie of the object should be determined.

2.7.3 Identification

Identification is the process in which to determine the object to the limit of thesystems knowledge, a refinement of the recognition. Then we also know whetherthe object detected and classified is a friend or an enemy. This stage is highlydependent on the recognition.

10 Modeling

2.8 Refining the approach

In section 2.4.1 a general layout was presented and refined in Eq. (2.4). Now it istime to refine that model further by dividing it into three different parts, one foreach probability. Some of the factors that may contribute to one of the differentprobabilities might not be interesting for the others.

2.8.1 Necessary factors for detection

What is relevant for detection of an object in a scene? Some might say colors,others are perfectly happy with a gray scale image, yet others want clearly visibleedges, but what is more important is the energy of the signal, relative to the noise.The most important factor is if the object is above the noise or not in the signal.What is the SNR of the signal, 40 dB or -1 dB?

What is the minimum difference before an algorithm can decide whether thereis something in a scene or not? The answer is that it depends, for instance on ifit is a single image or if it is sequence of images. But this is not enough. Theresolution is of utter importance. It might be obvious that you have to “see” theobject before you can find it, but when you are modeling a sensor that has to betaken into account.

If the object produces a strong signal compared to the rest of the scene, it mightgain a good SNR even if it is only one pixel large. From that pixel it is nearlyimpossible do decide anything, it might just be a deviation in the CCD-array, ora neutrino decay changing a part of the memory. But if it is seen for a period oftime and determined not to be a deviation in the CCD it can be recognized as adetected object.

A larger (or a closer) object will of course be easier to detect even from onesingle image.

The conclusion is then that if the object is small the energy from the objectneeds to be higher, and if the object is bigger a smaller SNR can be accepted andstill produce a good result.

2.8.2 Signature characteristics

The possibility to classify an object is also dependent on the SNR as well as theresolution, or the size of the object measured in pixels. The requirements of theresolution (size) are stronger, a single pixel can not lead to the classification of anobject however strong it might be.

What is then the minimum size of the object, measured in pixels? That shoulddepend on the objects physical size compared to the size in pixels, the resolution.

In image and signal processing different characteristics of the signal are calcu-lated in order to classify an object. Different distinctive features can be calculatedin order to determine the properties of an object. Below there are a few commonalgorithms presented, some of them are dependent on the distance to the objectthey are evaluating and some are not. The idea is that perhaps some of these

2.8 Refining the approach 11

features can be used to help model a sensor or a signal processing algorithm, forfurther information on them and other algorithms see for example [1], [2] and [7].

If a rectangle is put around the object, the smallest possible rectangle stillcontaining the object, the long side of the rectangle is called dmax, and the shortside dmin.

dmin

dmax(2.6)

Standard deviation over the target silhouette is another measurement, where σI

is the standard deviation of the intensity of the target silhouette and Imean is themean value of the target area (defined in 2.11)

σI

Imean(2.7)

To determine the shape of the object the relationship between the area of theobject and the perimeter can be translated to a specific measurement. Where A isthe area of the object, P the perimeter, Z(A) is a factor for normalization dependenton the area.

A

P 2 · Z(A)(2.8)

Mean edge strength can be measured as below, where Ek = 1 if pixel is declaredas edge and 0 if not and k is defined in the target region.∑

k Ek

A(2.9)

Yet another descriptive factor is the FORM-factor [2], this one is not dependenton the resolution and calculates the ratio between area and mean shortest distanceto the perimeter.

σA

d2(2.10)

Some of the feature measurements are dependent on the distance to the object.For instance the mean intensity, where A is measured in number of pixels, is verysensitive to absolute intensity.

Imean =∑

k Ik

A(2.11)

As mentioned earlier the ratio between object signal and noise is interesting. TheSNR has been defined earlier but could also be defined more appropriate to thesituation as below. Bmedian is the median of the surrounding background and σdenotes the corresponding variances. This quantity relates the objects intensityand fluctuations in intensity to the background and will later on be defined as theclutter.

12 Modeling

Imean −Bmedian√σ2

I + σ2B + ε

(2.12)

The two quantities directly dependent on the distance (not possible to use withoutexact knowledge of distance) are dmax (as defined in Eq. (2.6)) and the area,measured in pixels.

2.8.3 Decision making

The quantities defined above are calculated and the results are used, one by oneas well as together, and compared to existing information on targets in a databaseor “information central”. This database or “information central” can be made upfrom trained back propagation neural networks, comparisons with a database andthe closest (by some standard) target entry is chosen as candidate, if it is closeenough.

Bayes rules can be used as one of many possible algorithms in the determination.Here in the conceptual model we have prior information of the object. In fact weknow exactly what it is. The model will use the probability of a correct detection,recognition and identification of an object. No incorrect classifications will be madeby the system. There are exemples in the literature [3] of false target detectionsbut not mentioned further in this report.

Chapter 3

System decomposition

In this chapter a model for describing the signal emitted by the objects in the FOV1, S(x, y), is presented in the first part. It is then in the latter part of the chapterextended to take the clutter in to consideration.

3.1 Size does matter

A truck standing a few meters in front of you a sunny day, is not a very difficultobject to detect. As the truck is moved further away it becomes increasinglydifficult to detect, and at some point the truck will no longer be possible to see.

If the truck was doubled in size, it would be possible to see it at an even greaterdistance. We can therefore conclude that size and distance are important factorswhen detecting an object.

A digital camera working with a two-dimensional array of detector elements willdepict the truck reasonably correct at short distance. As the distance to the truckfrom the camera grows, the truck will be depicted with fewer and fewer pictureelements (pixels). At some point there will be only one pixel representing theentire truck. If it is moved even further away only half the pixel will describe thetruck and the other half will be describing the surroundings of the truck, the twohalves will be weighted together to produce an output. It is now not possible todetermine what it is the pixel is representing. But first lets take a few steps back.

3.2 Inverse square and cube laws

The resolution of a camera, independently of what frequency range is used (mea-sured in pixels) is constant, but the size of the projected FOV onto the image-planein the camera will be dependent on the distance to the objects in the FOV. If theangular size of the FOV is held constant the depicted scene will increase in size

1field of view

13

14 System decomposition

with the square of the distance to the observed objects. This leads to the fact thatthe resolution of an object will decrease with the square of the distance.

Consider an airborne sensor at height h over the ground and at distance r froma scene (as seen in Fig. 3.1). Let the object of interest be represented by a boxof a size so that the object is circumscribed, with the area ab = AT . If the FOV,2α× 2β, is small then it is possible to write Eq. (3.1) when considering the similartriangles in Fig. 3.1.

αβ

s

b

a ar

h

α

c

s

Figure 3.1. Principals of angles in the FOV

h

s=

c

a(3.1)

2α = arcsin(c

s) ≈ c

s(3.2)

And similarly for β.

2β = arcsin(b

s) ≈ b

s(3.3)

The solid angle covering the object is approximately given by φ.

φ =cb

4πs2=

s2αs2β

4πs2=

αβ

π(3.4)

From the Eq. (3.1)-(3.3) it follows that

φ =1π

c

s

b

s=

hAT

πs3=

hAT

π(h2 + r2)32

(3.5)

Furthermore, if the probability of detection is proportional to the solid angle assuggested by [13], we get

Pdet = k1 ·hAT

π(h2 + r2)32. (3.6)

Now k1 represents all the other factors that has influence on probability ofdetection, besides resolution. The question is, how can it be determined?

3.2 Inverse square and cube laws 15

b

a

y

Ax

A

Figure 3.2. A scene with a box.

For r � h the probability is approximately proportional to the inverse cube ofr, and for h � r it is approximately proportional to the inverse of the square of h.

It is easily understood that the resolution of the sensor is of great importance.The proportionality k1, should then be dependent of the resolution of the sensorarray, i.e. the n×m pixels. Lets define resolution, ρ, as number of pixels per meterin the scene, or angular resolution as pixels per radians. The resolution can the bedefined as Eq. (3.7).

ρ = ρxρy =n

c· m

b(3.7)

k1 = k2 · ρ = k2nm

cb(3.8)

Now let the FOV grow so that it covers more than the object of interest in thescene (as seen in Fig. 3.2), i.e. 2sα · 2sβ = cb > AT = AxAy.

Pdet = k1 ·hAT

π(h2 + r2)32

= k2nm

cb· hAT

π(h2 + r2)32

= (3.9)

= k2nm

s2α · s2β· hAT

π(h2 + r2)32

= k2nmAT

4αβ· h

π(h2 + r2)72

(3.10)

All objects with the same area, AT , will not be detected with the same probabil-ity. Some objects may be thin and long, so that they do not cover a whole pixel inheight and will therefore be difficult to detect, compared that to a square object itwill probably be harder to detect. Hence the area of object (as seen by the sensor)is not the most important factor. Therefor the shape of the object has influence onthe proportionality constant. There are a few different ways of measuring form asmentioned in section 2.8.2 (P2A or FORM for instance), see [2]. Another, easier,way would be to regard only the smallest side of the object towards the sensor.

Let a rectangle circumscribe the object, as shown in Fig. 3.3. The largest sideof the rectangle corresponds to the easiest detectable dimension, Ay, but this willnot be a true representation of the objects shape. Let instead the smallest side ofthe rectangle be the decision factor to use, Ax.

16 System decomposition

For most type of objects the smallest side of a circumscribing rectangle will bea good measure, however for extreme concave objects this might no be the bestsolution. Ways of correcting this include letting parts of the object fall outside therectangle and letting the rectangle be placed within the object.

Let the smallest side of the rectangle be represented by Ax, Eq. (3.9) thenbecomes

Pdet = k2 ·nAx

2α

1(h2 + r2)

12· mAy

2β

1(h2 + r2)

12· h

π(h2 + r2)12

(3.11)

It is here assumed that the y-dimension is much easier detectable and in fact thedifficulty lies in detecting the x-dimension. Note that the sensor angular resolutionis what is actually regarded.

3.3 Sampling

According to the sample theorem a signal should be band limited in order for asampled signal to be accurately reconstructed. It has to limit the bandwidth tomaximum half that of the sample frequency (fs). That is, −fs/2 ≤ f ≤ fs/2 isrequired, otherwise a reduction of information will be the result.

An image can be viewed as a two-dimensional signal. When taking a photoa sampling of the world is done. For the sampled image, spatial frequencies canbe calculated (compare with the frequencies for an audio signal which can be saidto be temporal), and they do not differ in any significant way from “ordinary”(temporal) frequencies.

Since no band limiting filters can2 be used, a camera will never satisfy theconditions of the sampling theorem. Aliasing will be the result, which is the samething as what happened when the truck and its surroundings was represented byonly one pixel (in the example above). All details to small to be noticed in thesampled reality (the image) will still be represented by the pixel, however averagedwith the rest of the details not noticeable.

When the truck is moved further away, each pixel will cover a larger area ofthe truck, corresponding to a lower sample rate. This effectively corresponds toa low pass filtering of the image since the sampling is performed with rectangularfunctions. When an image is low pass filtered details tend to disappear, leading tothe fact that it will be harder to recognize the depicted object.

Even if it is possible to detect an object in the image, it might still be difficultto determine what it is, i.e. classify it. Recognition (or classification) can be seenas a special case of detection; the ability to detect whether an object is a memberof group A or B. The same goes for identification.

2Band limited filters can be applied, but requires sets of gitters, lenses and apertures

3.4 History 17

3.4 History

During the Second World War a new discipline in mathematics was founded, Op-erations Research. To take advantage of Operations Research during and afterthe war mathematical models for soldiers, tanks and army divisions were needed,and also developed. Questions like “When will a ship be spotted?” needed amathematical answer.

One of the first who tried to determine statistical detection models of targetswas H.R. Blackwell. In an article from 1946 he presented a study of circular targetson uniform background. He tried to find factors like contrast, size and brightnessto describe possibility of target acquisition. In the research he let 19 women makemore than 200,000 observations. Later on Blackwell and Taylor determined con-trast limits for acquisition.

3.5 The Johnson criteria

In 1958 John Johnson presented a method for determining whether an object wouldbe detected when using image intensifiers [10]. Johnson transformed the objectsto be represented by bar patterns. Each object, its distance, and orientation wasrepresented with the number of black and white bars (a pair of a black and a whitebar is named a cycle), covering the objects minimum dimension towards the viewer.The size of the bars will be a result of the resolution of the image enhancer, thedistance to and size of the object.

θ

φ

Figure 3.3. Illustration of target to LP transformation. φ angular size of one pixel-row(0.5 LP). θ angular size of targets (maximum) dimension. The object is 2.5 pixel-rowshigh and that transforms to 1.25 LP/TGT.

Every line of pixels can be said to have an angular spreading, and that spreadingin its turn can be represented by a size at some distance from the observer, closeto or over the object.

Johnson defined the resolution in the device used as the width of two adjacentlines, called a cycle. He then compared the angular size of the objects minimumdimension (towards the viewer) with the angular size of one cycle. This way he gotthe equivalent of the object as seen by the observer in cycles. By letting several

18 System decomposition

Figure 3.4. Classic figure from [10] describing target equivalent bar patterns.

TARGET RESOLUTION PER MINIMUM DIMENSIONBroadside View Detection Orientation Recognition Identification

Truck 0.90 1.25 4.5 8.0M48 Tank 0.75 1.2 3.5 7.0Stalin Tank 0.75 1.2 3.3 6.0

Centurion Tank 0.75 1.2 3.5 6.0Half-track 1.0 1.5 4.0 5.0

Jeep 1.2 1.5 4.5 5.5Command Car 1.2 1.50 4.5 5.5

Soldier (Standing) 1.5 1.8 3.8 8.0105 Howitzer 1.0 1.5 4.8 6.0

Average 1.0± 0.25 1.4± 0.35 4.0± 0.8 6.4± 1.5

Table 3.1. The Johnson table of cycles, as presented in [10].

observers view scenes containing a bar target object a table could be made with thenumber of cycles necessary for detection, orientation, recognition and identification,see Table 3.1. The term cycle seems to have been replaced, more and more, by theequivalent term line pair (LP3).

The conclusion is that everything needed to determine if an object is visible,and to what extent, is its angular size, as seen from the observer. But what is theoutput of Table 3.1? In fact it is not clearly stated in the article, but it is consideredto be the measure of the 50% probability of a correct decision, as mentioned in [17].

3One cycle or LP is two adjacent rows of pixels

3.6 Extensions to the Johnson criteria 19

Detection Orientation Recognition IdentificationAverage number of cycles 1 1.4 4 6.4

± 25% ± 0.25 ± 0.35 ± 1.00 ± 1.60Ratio compared to detection 1.0 1.4 4.0 6.4

Table 3.2. Resolution per level of acquisition according to [10].

Number of resolvable lines across targetProbability Detection Recognition Identification

1.00 1.5-3 9-12 18-240.95 1-2 6-12 12-160.80 0.75-1.5 4.5-6 9-120.50 0.5-1 3-4 6-80.30 0.375-0.75 2.25-3 4.5-60.10 0.25-0.5 1.5-2 3-40.02 0.125-0.25 0.75-1 1.5-2

Table 3.3. C2NVEO’s version of the Johnson Criteria, from [17].

As can be seen in Table 3.1 no target diverges substantially from the averagein each category. All targets are within 25% of the average and we can concludethat the average in each category is a good approximation of decision boundaries.

The result with respect to resolution states, the rather obvious, that to classifyan object we need higher resolution than just to detect it. In fact the resolutionhas to be four times higher to classify, and six times higher to identify an objectthan to detect it, see Table 3.2.

3.6 Extensions to the Johnson criteria

The Johnson Criteria is very good in its simple form to transform an object into aprobability. However it can only give the 50% limit, a better model that can giveprobabilities in a few more ranges would be even more useful.

From C2NVEO4 a more advanced model can be found, see Table 3.3, withLP/TGT levels for different probabilities.

It is states that the values from Johnson are pessimistic in [5] according to [11],and we will see later on that others have come to the same conclusion.

According to [15] the initial (Johnson’s) performance measure were conductedin significant clutter. What that means is unclear. Noticeable is that today SNRand SCR 5 is considered to be important factor in the decisions.

4U.S. Army Command Center for Night Vision and Electro-Optics5signal to clutter ratio

20 System decomposition

10−1

100

101

102

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Line Pairs over minimum dimension

Pro

bab

ility

C2NVEO

detectionrecognitionidentification

Figure 3.5. Visualization of the C2NVEO table.

3.7 Adding another factor to the equation

It can be stated that the lack of understanding how noise and clutter effects theJohnson-like methods, is considered as a deficiency. Some research has been doneto investigate what role clutter has to the detection performance.

If clutter should work as an input parameter we have to define a mathematicalmeasure for it. According to [15] there had been some attempts to define clutter,but nothing that really fitted to the Johnson like methods. So instead they definedthere own clutter measure, see Eq. (3.12).

The calculations are made over cells in the scene. The cells are square with theside approximately twice the hight of the targets. It is probably chosen because it isusually the targets minimum dimension, and it will produce high clutter values forobjects of approximately the same size as the target. According to the authors, itseemed to give higher values for subjectively more cluttered scenes and it containsboth spatial and intensity information.

A center cell is placed so that the target is centered in it, the outer cells areplaced around the center cell. Clutter is then defined as

clutter =( N∑

i=1

σ2i

N

) 12

(3.12)

where N is the number of contiguous cells and σi is the radiance standarddeviation over the ith cell, it is probably an ordinary standard deviation calculatedfrom the radiance property of the materials in every pixel in the cell, see Eq. (3.13).

3.7 Adding another factor to the equation 21

σ2i =

∑x,y

(gi(x, y)− gi)2 (3.13)

In this case gi(x, y) is the intensity in pixel (x,y) in the ith cell and gi is themean of the cell.

Figure 3.6. Example of cells in a scene with two black targets.

The signal to clutter ratio (SCR) is then calculated as in Eq. (3.14) and fornegative contrasts (i.e. when the signal from the surroundings is stronger than theinteresting object) (3.15). g0 represents the object of interest.

SCR =|g0,max − gmean|

clutter(3.14)

SCR =|g0,min − gmean|

clutter(3.15)

Noticeable is that high clutter levels (low SCR) will result in detection prob-abilities lower than 0.9 (Pdet < 0.9), for all resolutions. For example 6 LP/TGTgives 0.8 possibility of recognition from Table 3.3, and a pretty good identificationaccording to Table 3.1, but at high clutter (SCR = 0.33) detection possibility dropto 0.84. So at high clutter levels detection levels are closing in on recognition levels.

On the other hand for very low clutter (high SCR) detection probabilities stayclose to 1, even for as low resolution as 0.25 LP/TGT. It can be noticed that thisis better than the usual 0.5 LP/TGT for “Hot Spot”-targets.

Further, it does seem as if clutter plays a more important role for resolutionslower than 2.0 LP/TGT. The conclusion, made by the authors of [15], is that clutter

22 System decomposition

Number of resolvable lines across targetSCR 0.25 0.5 1.0 2.0 3.0 4.0 6.00.33 0.00 0.00 0.32 0.67 0.77 0.841.2 0.04 0.46 0.96 0.98 1.0 1.05.8 0.38 0.87 0.95 1.0 1.0 1.039 0.93 0.96 1.0 1.0

Table 3.4. Pdet from observations with different resolutions and SCR levels, from [15].

Number of line pairs across targetDetection Low Clutter Medium Clutter High Clutterprobability (SCR > 10) (1 ≤ SCR ≤ 10) (SCR < 1)

1.0 1.7 2.8 -0.95 1.0 1.9 -0.90 0.9 1.7 7.00.80 0.75 1.3 5.00.50 0.5 1.0 2.50.30 0.3 0.75 2.00.10 0.15 0.35 1.40.02 0.05 0.1 1.00.00 0.0 0.0 0.0

Table 3.5. Detection probabilities for different clutter levels Pdet [15][11].

can be divided into three distinct regions. The first, for high clutter levels (SCR≤ 1) a strong increase in resolution is needed even for a little decrease in SCR.

The second region, for medium clutter levels (1 < SCR ≤ 10), a doubling ofresolution is needed when SCR drops from 10 to 1, to maintain the same level ofprobability.

In the third region, low clutter levels (10 < SCR) it is assumed that the prob-ability reaches one, asymptotically, as the SCR reaches infinity. Resolution seemsto play a minor role here.

Since there are three homogeneous regions where resolution does not vary muchin order to get the same level of probability, SCR can be calculated without greaterprecision, just to determine what region it belongs to is enough. This is good sincedetermining the clutter level is a hard and tiresome work. It seems as if it is possibleto avoid the problems of defining a correct and exact clutter measure.

3.8 Are the results only applicable for detectionprobabilities?

In [15] the influence of clutter is investigated for detection. A suggestion of howto calculate clutter levels is proposed, and the procedure is said to be possible to

3.9 Hot Spots 23

10−2

10−1

100

101

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Line Pairs over minimum dimension

Pro

babi

lity

Schmieder−Weathersby

low cluttermedium clutterhigh clutter

Figure 3.7. Visualization of Table 3.5.

simplify.Maybe it is not necessary to further expand the model with recognition and

identification in clutter? If an object has been detected in clutter, it has beendiscriminated from false targets already, and therefor recognition and identificationis not dependent on clutter, but only on noise. But if detection only reports signalsources and it is up to the recognition to determine weather it is an interestingobject or not, a clutter extension is necessary for improvement. So the best (easiest)way to handle this is in the detection phase, by using Table 3.5.

3.9 Hot Spots

Hot spot is the name for strong signal sources on a uniform background, with ahigh target to background ratio. An aeroplane in the sky for instance. They aregenerally easier to detect than other objects, usually 0.5 LP/TGT is used, but asseen above even lower resolutions are possible. They could therefor be modeled asobjects without clutter.

3.10 Summary

The model presented in Eq. (2.4) has now been modeled to some extent. A descrip-tion of how to model s and c has been presented. The model for the object signalis concentrated on the resolution of the object, using the same models presentedby Johnson and C2NVEO. From the list of different characteristics in section 2.8.2

24 System decomposition

we can see that Eq. (2.6), (2.7), (2.11) and (2.12) are all used. Eq. (2.8), (2.9) and(2.10) has not been used in the model so far. The background and nearby objectshas been taken into consideration, as well as intensities and contrasts.

Just because the objects position relative to the sensor is known, it does notautomatically mean that it can be detected using the model presented above. Theline of sight to the object must be considered. The object might be occluded byanother object, for instance a car parked partly behind a bus. Is the object at allvisible from the sensors position and to what extent? All of it or just a fraction?By letting the bounding box vary in size so that it fits to the objects circumstances,this problem can be handled. This is only possible for small adjustments to thesize of the bounding box.

The model has so far not taken into consideration how the signal is corruptedon its way from the objects in the scene to the sensor. The signal should be effectedby the atmosphere, dependent on humidity, rain, fog and other factors. How tomodel this will be considered in the following chapters.

A conversion table or function correspoding to the Schmieder-Weathersby (asseen in Fig. 3.7) for clutter would be nice for noise. However no such models hasbeen found and an investigation presented in the next chapter attempted to findsuch a transform.

Chapter 4

Development of a model forthe effects of noise

In this chapter the experiences from an investigation of human acquisition in noisyimagery are discussed and suggestions for further studies are made.

4.1 Introduction

As stated earlier in this report, a transformation from resolution to probability isused in the this model. This basic transform is enhanced with the introduction ofthe SCR, which also adds additional functionality. The effects of noise in perceptionand target acquisition is however important. In order to find a transform resemblingthe Johnson criteria, when noise is present, a series of experiments was conducted.

All the transforms provided in this report so far, are based on human perfor-mance. The attempts to produce a transform here will therefore also model humanbehavior.

4.2 Preparations

To start with, four images were selected, all showing different scenes, see Fig. 4.1.The images contained vehicles and humans, considered as interesting objects, ina natural environment. From the images several, or all, of the interesting objectswere measured in terms of the number of pixels in the short side of a boundingbox. The scenes contain from one to nine different interesting measured objects.

Since the dependence between resolution and probability of acquisition is wantedfor different noise levels, a resolution pyramid1 for every scene is produced. Differ-ent objects in the scene will have different resolution due to their size and therefore

1A collection of the same image in multiple resolutions

25

26 Development of a model for the effects of noise

many different resolutions can be obtained from one picture in the resolution pyra-mid. Resolutions from 0.2 to 56 LP/TGT were obtained.

The clutter and the noise are considered to be independent of each other andcan therefore be investigated independently. Note, that two objects in the samescene does not necessarily have the same clutter values.

To each scene noise is added before the resolution is changed, in order to getdisturbances of the images in the same resolution as the object. Four different setsof images were compiled, containing 36 images. Nine of each scene, with differentresolutions and noise levels, ordered so that the higher resolutions were shown last.Two examples can be seen in Fig. 4.2

4.3 Conducting the experiments

Each set was shown to a group of people with varying age, background, educationand sex. The viewers had unlimited time, and were instructed to mark the part ofthe image they thought contained a vehicle or a person. If they thought they knewwhat type of object it was they were instructed to write that down, and when theobject presented was identified it was marked in the same way. Nearly 40 personswere used, making over 6500 observations.

4.4 Results

The hypothesis is that the probability of detection, recognition and identification ofobjects would decrease with increasing noise. The idea was to produce a transform,similar the one for clutter [15].

The analysis of the collected data revealed no significant changes for the proba-bilities investigated that could be directly liked to the noise. The initial hypothesisseems to be correct, but no numbers for a transform can be presented. It seems asif the wrong noise interval was investigated. There are some differences observedin the data but not enough to produce a transform based on resolution and SNR.

Initially it seemed as if the transform for recognition moved closer to the onefor identification, which would have been expected, but this could not be verified.It simply seems as if more test subjects (viewers) should be used and that lowerSNR levels should be investigated.

The fluctuations in the data are large, and can not be determined to result onlyfrom different SNR and resolutions.

The conclusions from the test must be that it seems as if there are differencesdue to noise, and that it should be investigated further.

4.5 Suggestions for further studies

When the noise was added before the change in resolution, the effect was that thenoise was low pass filtered, resulting in a lower degree of noise. A different way

4.5 Suggestions for further studies 27

(a) Jeep traveling downhill. (b) Four-wheeled motorcycles.

(c) Israeli checkpoint. (d) Tanks on a field.

Figure 4.1. Scenes used in acquisition experiments.

of doing this would be to add noise with a spatial size in the same order as thewanted resolution. The noise would then be clearer to the viewer. Now the viewerdid not understand that many of the images shown were altered with additionalnoise. It could not be seen except when comparing noiseless and noisy images. Forthe images of high resolution it was still possible however. But when trying to addlarger noise, it can not have the effect of altering the clutter levels.

More viewers could also be used in order to get more stable data. The fluctua-tions in the data should then be less and the correlations to clutter and resolutionshould be more significant.

The number of images in the resolution pyramid could probably have beenfewer. It seems as if nine was too many, since there were objects of different size,and therefore resolution, in the scenes. Extra images could instead be used tointroduce more scenes.

The most important factor however is the noise levels used. It seems as if SNR

28 Development of a model for the effects of noise

100 200 300 400 500 600

50

100

150

200

250

300

350

(a) Israeli checkpoint, with added noise. (b) Tanks on a field, with added noise.

Figure 4.2. Examples of images used in the acquisition experiments.

below 10 dB should be in focus for further investigations.

Chapter 5

Sensors

The model is now established and it is time to discuss the sensors effects of theresolution. As stated above, this study is concentrated on a flying sensor platformlooking down on the ground. The platform (may it be a missile, a UAV or a mannedaircraft) is equipped with one or several sensors.

5.1 TV/Video sensors

The evolution of digital cameras are well known, and today it is possible to findone with reasonably good quality and resolution at almost every supermarket. Theprinciples of different TV/Video imaging systems will not be analyzed in detail,but one can easily determine that the basic function of the cameras are the sameregardless of wavelengths detected and the frequency of image updates. The con-cepts discussed in this chapter are focused on IR, but can just as well used forvisual light.

5.2 History of IR sensors

As early as around 1930 the first thermal imaging system was developed, the Evap-orograph [18]. From the 1940’s two different types emerged, the first was a scanned,TV-like, system with discrete detector elements. The second used a scanning elec-tron beam on an surface that was positively charged from incident light, that wasthe infrared Vidicon. During this period one detector element was usually usedand optics and mechanics took care of the scanning.

The development of semiconductors lead to faster and cheaper sensors, espe-cially in cooperation with a cooling system. In the 1960’s many different ForwardLooking Infra Red (FLIR) sensors evolved. During the late 1970’s and the early1980’s FLIR technology matured and two dimensional arrays were developed. Theybecame bigger, faster and more sensitive due to new compounds (HgCdTe for in-stance). For further information on Vidicons, image intensifiers and materials see

29

30 Sensors

[18].Three wavelength bands are used in IR, those are near IR (NIR) 3-5 µm, thermal

IR (TIR) 8-12 µm which corresponds to blackbody radiation from about 300 K,and finally the far IR (FIR) 30-1000 µm. Cooling is very important for thermal IRwhich can cause problems. Cooling systems are large, heavy and energy consumingand are not suitable for portable use, however some do exist. The TIR sensors haveto be cooled or they will start detecting their own heat signature, and the imagequality will degrade.

5.2.1 Reticles

For one time use (as in missiles), two dimensional sensor arrays have not been usedin the past because of the price, that is now starting to change. Instead a muchsimpler target tracking device has been used, namely reticles [9].

In front of a single detector element, a rotating disc with holes or transparentsectors, is placed. The incident light beam passing through the reticle will bemodulated by its rotation. The direction to the target in front of the sensor will beeasy to determine by analyzing the frequency or the phase of the signal, dependingon what type of reticle is used. The amplitude of the signal has to be limited by anautomatic gain control (AGC). This makes reticles ideal for hot spot detection, i.e.a strong signal source on a uniform background, and is therefor used in air-to-airand ground-to-air applications. It can also be used in air-to-ground, if the contrastof the signal source is strong compared to the background.

Detection of a object is easy, as long as it discriminates it self from the back-ground. Recognition and identification of targets are nearly impossible.

Reticles are not modeled further in this thesis, but there are as stated abovemany similarities with hot-spot detection, modeled in section 3.9.

5.3 Pinhole camera

The pinhole camera is an ideal camera, with an infinitesimal small aperture (in thefocal point), which in theory leads to clear images. But this also leads to a problemof too little light passing through the aperture.

The layout of the pinhole camera can be viewed in Fig. 5.1. The distance f,known as the focal length, and the size of the sido of the light sensitive area A, i.e.film or detector array, are the factors determining the quality of the camera, andultimately. These two will determine the size of the FOV.

FOV

2= arctan

(A/2f

)(5.1)

The size of the scene (d) possible to depict is directly proportional in size tothe relationship between R and f.

5.4 Advanced camera model 31

d

focal point

R

FOV

f

A

Figure 5.1. Layout of pinhole camera.

With a detector array, made of N ×M sensor elements (pixels), the resolutionof the camera can be determined as the size of the scene projected onto each pixel.This part of the scene projected on to the pixel will be represented by its average.

To calculate the resolution of a pinhole camera with a detector array we haveto know something about the world on the outside. In this case we will assumethat the world around the camera is flat. The position of the camera, its directionand the size of the FOV have to be known.

If the camera is located at a great distance from the scene the resolution fromone corner of the FOV to the other can be considered constant. But if the sceneis viewed from a 45 degree angle, with a large FOV, the resolution might differsomewhat.

5.4 Advanced camera model

A modern camera is more complicated than the pinhole camera, but works inapproximately the same way. The use of lenses has solved the problem of too littlelight on the detector array by allowing a larger aperture. This has however leadto the fact that only a part of the scene can be viewed in focus, the rest will beblurry. The size of the FOV is no longer determined solely on the physical size ofthe detector array and the focal distance, but also on the lenses. All those featuresare compared to the pinhole camera and measured by their D/f number, and thesize of the FOV is determined by an angle.

Since there are no ideal lenses or cameras the incoming rays will scatter some-what. This scattering will lead to an averaging in every pixel, that can be seen asa convolution with a Gaussian filter, hpsf , defined in Eq. (5.2), where σx and σy

are physical factors defining the lenses.

hpsf (x, y) = e− 1

2 ( x2+y2

σ2x+σ2

y)

(5.2)

This will effect the image as low pass filtering, and as a consequence the res-olution of scene is altered. It is believed that the Gaussian filter can model this

32 Sensors

satisfactory, where σ is a factor determining the quality of the lenses and the effectof the point spread function. If the cut off frequency of the filter is much lowerthan the resolution of the detector array, than the averaging can be ignored.

5.5 Rayleigh criteria

At a great distance it will not be possible to detect an object, simply because of lowresolution. One way to handle this is to zoom in or narrow the FOV, or increasethe number of pixels on the sensor array, but there is a limit for how far awayobjects can be seen. The limit is given by the Rayleigh criterion [16] [12] whichstates that the smallest separation possible to detect is ∆θ,

sin(∆θ) =1.22λ

D(5.3)

where λ is the wavelength and D is the diameter of the lens. With a known D/frelationship Eq. (5.3) turns into Eq. (5.4) which gives the minimum separation onthe sensor array.

With a lens diameter of 122 mm and a wavelength of 10 µm, details larger than0.1 m can be seen on an object 1000 m away.

∆xmin = f sin(∆θ) = f1.22λ

D(5.4)

For a known f/D the smallest separable distance on the sensor array can bedetermined. The Nyquist theorem states that the sampling distance should be halfthat of the smallest detail in order for it to be correctly depicted. That means thatif the distance between two adjacent detector elements is smaller than or equal to∆xmin/2, the effects of the Rayleigh criteria can be ignored in the calculation ofthe resolution.

Usually the sensor array is the factor limiting the resolution in a real system,not the Rayleigh criterion, but it can be worth checking.

5.6 Further enhancement of the sensor model

A ground that acts as natural terrain (non flat) is of course more interesting thanthe idealized flat version. To model that one has to take the projection of theobject on to the sensor array in consideration. A real projection does not have tobe performed but a calculation of the minimum dimension of the object as seen bythe sensor.

If the FOV is large an effect called vignetting will occur [2]. It is simply adamping of the signal dependent of the angle of incident on the lens, and it isfurther modeled in section 6.6.

Chapter 6

Signal generation

To enhance the model some properties specific for the signals used should be im-plemented.

6.1 Introduction

After a model for analyzing the signals has been presented it is time to concentrateon generating those signals. But as stated earlier it will not be in the form of areal signal, but rather calculate the characteristics of such a signal.

The model has so far been independent of wavelength, no difference has beenmade between IR and visual light. When generating the signals the differencein wavelengths are important. If the signal is supposed to be a thermal infrared(8−12µm) it will be based mostly on the temperature of the objects in the scenes.If it is in the visible spectra, the sun or an artificial light will be reflected on theobjects and into the sensor. In the first case the signal is generated by the objects,in the latter it is changed by the object. First lets consider thermal IR.

6.2 Blackbody radiation

All bodies (with a temperature above 0 Kelvin) radiate energy. A perfectly blackbody absorbs all incoming radiation and this will lead to an increase in temperature.The energy radiated from a black body is proportional to the temperature to thepower of four, known as Stefan-Boltzmann’s law Eq. (6.1).

M = σT 4 (6.1)

The radiation emitted from the body comes with all wavelengths. Thoughdifferent temperatures will lead to different composition of wave lengths in theradiation Eq. (6.2) and the radiant exitance will then become Eq. (6.3).

33

34 Signal generation

Mλ =2πhc2

λ5

( 1

ehc

λkT − 1

)(6.2)

M =∫ ∞

0

Mλdλ =∫ ∞

0

2πhc2

λ5

( 1

ehc

λkT − 1

)dλ (6.3)

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

x 10−5

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Wavelength, λ [m]

Spe

ctra

l rad

iant

exi

tanc

e, M

λ [W

/m2 ]

400 K

350 K

300 K

270 K

Figure 6.1. The radiation from a blackbody of different temperatures.

Since a black body is a perfect radiator nothing can radiate more than thisand all ordinary objects will always radiate less. The emissivity, ε, is the quantitydescribing how good radiator the object is, and it is defined as Eq. (6.4). If theemissivity is independent of wavelength one refers to the object as a gray body.

ε =M

Mblack body(6.4)

Radiant exitance (M) is the energy from one area unit per time unit, andthe corresponding entity for received radiation is called irradiance (E), and is theradiation on one area unit per time unit. Since they are similar they both have theunit Wm−2, and are measures of intensity.

6.3 Interaction with materials

Some surfaces are seen as shiny and some as subdued this gives us the three prop-erties reflectance (r), absorbance (a) and transmittance (t). These tree propertiesare somewhat dependent on each other, for example: a perfect reflector will not

6.4 Optical windows 35

Material Visible Near IR Thermal IR0.4 to 0.7 µ m 3 to 5 µ m 8 to 12 µ m

Olive (paint) 5-10 8-10 17-23White (paint) 70-80 10-35 5-10

Vegetation 2-18 5-20 5-10Loam 2-20 2-10 0-20Ice - 1-8 2-6

Table 6.1. Reflectance for some materials, in % of incident (or self generated) radiation,from [17]. Olive and white represent measurements from the paint used for vehicles inthe US Army (olive is the normal color and white is used for winter).

absorb anything, nor will it let anything through; a perfect absorber will not re-flect, nor transmit anything. This lead up to Eq. (6.5). According to Kirchhoffslaw ε and a must be the same [17].

r + a + t = 1 (6.5)

One of the most common approximations made is that a, r and t are constants[17], and some more sophistication can be gained by setting ε to a value less thanunity. Since transmittance often can be neglected, the relationship between emis-sivity and reflectance becomes ε = a = 1− r, which is a good start. Some values ofr can be found in Table 6.1. The intensity of a signal transmitted from an object,Iobject, will be a fraction of the blackbody radiation, I0.

Iobject = εI0 (6.6)

Today large efforts are made to investigate the possibility of constructing ve-hicles and clothing in materials that are hard to detect [8]. By letting an objectbe covered by a material with high reflective and low emissive properties in TIR,the object will appear to have the same temperature as the surrounding terrain,and will thereby be harder to detect. A clear day it might however reflect the sky,with a cooler temperature than the surroundings, and the object will have a largenegative contrast to the background. A cloudy day, it will almost appear invisible.

6.4 Optical windows

For some wavelength the atmosphere is a good absorber, and for some it is a goodtransmitter. These different transparent frequency bands are called windows. Thechoice of wavelength (8-12 µm), is a compromise between the fact that the heatradiation is at its peak there for temperatures around 300 K see Fig. 6.1, and thefact that the radiation can pass through the atmosphere reasonably well.

36 Signal generation

6.5 Thermal variation

In the TIR region one has to take account for the heating of the sun during the day.Some objects are better at absorbing the heat than other. For instance the leavesof a tree might change the temperature faster than a large stone. The temperaturerise due to the sun and daylight is known as the thermal daily cycle. When the sunis up the objects will become warmer, and when it is not, the temperature of theobjects fall. The time of day is thereby a factor with importance when generatingthe signals of the background.

6.6 Calculating the clutter

Since the clutter is relative to the object of interest [6] it is necessary to relate thesize of an object to the vegetation, or to define the size of all trees to a certainvalue.

Recalling the equations defining clutter Eq. (3.12) - Eq. (3.15) adopted withthe signal definitions from Eq. (2.4) the equations become as below.

σ2i =

∑x,y

(ci(x, y)− cimean)2 (6.7)

clutter =( N∑

i=1

σ2i

N

) 12

(6.8)

SCR =omax − cmean

clutter(6.9)

The variance Eq. (6.7) should be calculated over a cell with approximately twicethe size of the object. If the variance of the terrain is known around the object theclutter can be determined as the square root of a weighted sum of variances, wherethe weights are dependent on occurrence instead of the formal cells. The SCR isthen determined from the strongest (or weakest) signal from the object comparedto the mean intensity of the surroundings and normalized with the clutter.

clutter =( ∑

i

kiσ2i

) 12

(6.10)

∑i

ki = 1 (6.11)

This means that the variance is the parameter defining the surroundings orbackground. When specifying the vegetation a map containing the variance iscreated, needed is also a the emissivities of materials in the area. From thesethe SCR can be calculated with enough precision to determine which of the threeSCR levels that should be used. These simplifications are in line with what wassuggested in [15].

From Eq. (6.2) and (6.3) we get Eq. (6.12) in the wavelenght region [λ1, λ2].

6.6 Calculating the clutter 37

Mobject =∫ λ2

λ1

2πhc

λ5· 1

ehc

kT λ − 1dλ (6.12)

When hν = hcλ � kT Wien’s radiation law is applicable, which turns Eq. (6.12)

into Eq. (6.13).

Mobject =∫ λ2

λ1

2πhc

λ5· e− hc

kT λ dλ (6.13)

But to calculate the SCR omax and cmean are also needed.Some surfaces radiate energy in only certain directions, but most radiate in

all directions, these are called Lambertian surfaces. For them Lambert’s cosinelaw Eq. (6.14) and (6.15) can be used [12], where R is the distance and θ and ϕrepresents the angles to the LOS1. Iϕ is the radiation an observer would detectat an angle ϕ to the normal of the radiating surface. The recieved radiation at asurface, Es, is dependant on the incidant signals angle, θ, from the normal.

Iϕ = Iobject cos ϕ (6.14)

Es =Iϕ cos θ

R2(6.15)

For reflected radiation Mreflection = rE, and for the thermal radiation M = I0,so Iobject = εMobject.

The illumination from the object on the sensor will become

Esensor =Iobject cos ϕ cos θ

R2(6.16)

if this is a pinhole camera. But for a camera with lenses it will turn into Eq. (6.17).As mentioned earlier the intensity is altered if the angle of the incident ray is

large compared to the normal of the lens. From [2] we get that it is damped withcos4(φ), where φ is the angle to the center of the lens. A 30o angle will almosthalf the signal intensity, but on the other hand 30o is a large angle. At 10o 94%remains.

Esensor =Iobjectcos

4(θ)R2

(6.17)

Now we see that the signal is dependent on the size of the object, the anglesbetween the object and the sensor, the distance between sensor and object (thedistance is considered equal to all sides of the box, i.e. the box dimensions are smallcompared to the distance) and the emissivity. What remains is the temperature ofthe object.

Since we approximated the object with a bounding box, it is considered to behomogeneous and that leads to Eq. (6.18).

1line of sight

38 Signal generation

omax = Esensor = ε · Mobjectcos4(θ)

R2(6.18)

We now know how much energy is passed from the object to the sensor. Thesame equation can be used to determine the energy from the background, the con-tiguous background cells. Hence the omax and cmean in Eq. (6.9) are determined.

6.7 Vegetation model

To be able to calculate an accurate clutter level the surrounding terrain of theobject is of interest. Not only the vegetation but also if there are any buildingsand/or other objects. The clutter measure defined in Eq. (3.12) must be used incooperation with the information of the surroundings.

A specification of terrain with buildings, vehicles and other objects is necessary.The most direct and easy solution is probably to specify a “map” over the simulatedarea. This map should contain what type of vegetation is present as well as itsintensity and variance. Between the specified points an interpolation could be used.

The variance in a scene is dependent on the resolution. Lower resolution isequal to averaging and probably lower variance. The variance defined in the mapmust be normalized to a certain resolution and then some sort of scaling dependenton resolution can be used.

So far we have considered the objects as visible although close to other objectsand vegetation, but not occluded by it.

6.8 Further enhancements

An object located in a forest will be occluded by trees and leaves. This couldprobably be modeled as noise, dependent on what type of forest and how far in theobject is located. A transmittance could be modeled. This has however not beeninvestigated.

We have also assumed that the quantization of the sensor is unlimited and thatno problems will arise from it. A detector array will however use discrete levels forthe intensity of each pixel. This will induce a minimum temperature detectable forthe sensor, if the visibility is devided into discrete levels.

Chapter 7

Atmosphere

The last stage of the model is presented here. It discusses how the medium betweenthe sensor and scene effects the signal.

7.1 Introduction

How electro magnetic radiation is effected by the atmosphere is difficult to say ina general way. The simplest answer is that different things happens to differentwavelengths. Some of the signal energy is lost on the way to the sensor, this isnot only an effect of the increasing projection area due to the distance but alsodependent on interaction with the atmosphere. A ray might hit some particleand change direction or it may simply be absorbed. The two major effects areabsorption and scattering, and these will be the focused of this chapter.

If there are no significant disturbances for a specific wavelength or an intervalthis is refered to as windows, or transmission windows.

7.2 Scattering

When the medium (the air) contains particles, the EM-radiation sometimes collideswith them, when traveling along its path towards the sensor. The photon will thenbounce off the particle into a new direction. This effect is mainly caused by largerparticles, and generally leads to a weaker signal detected by the sensor.

Scattering comes from molecules and aerosols in the atmosphere. In aerosolswe include amongst others rain, dust, fog and smoke. The molecular scattering isneglectable for wavelength shorter than 2 µm [11]. If single rays of light would passthrough the atmosphere without any distortion everything would be perfectly clearall the time. If the ray is reflected out of the path towards the sensor, the totalenergy at the sensor will decrease. Other objects not in the FOV also transmitradiation, some of this radiation might be scattered in to the FOV of the sensor,

39

40 Atmosphere

increasing the total energy. For further information on scattering see for instance[18].

When the ray is scattered it is most likely scattered in another direction thanfrom which it came, but that is not certain. The easiest way to model this isto assume, without any further clarification, that all angles could be consideredequally likely for scattering.

in

out

Figure 7.1. An examples of scattering rays.

To determine the amount of rays scattered out of, and into, the line of sight(LOS), between the sensor and the object, an integration over the path must beperformed. Along the path the probability of hitting a particle must be considered.Can a photon be scattered out of the LOS and then back in? A background withstrong signal will effect the object signal more than a dark background. This in-and out-scattered signal is not a part of what is wanted and can be seen as a partof the n in Eq. (2.4), the additive noise.

It would however take a large amount of integrals to evaluate this, and we wouldbe back at ray tracing methods, dependent on how many deflections (scatterings) ofthe signal should be considered. It could be calculated ones and for all for differentatmospheric conditions and put in a table. That is, however, not necessary sincethere are experiments conducted, and tables already made. Let the parameterdefining the amount of scattering of the signal be Kσ, a quantity defined perdistance.

7.3 Absorption

Absorption is caused by the effects of molecules absorbing certain wavelengthscorresponding to discrete energy levels (in the internal states of the molecule).This leads to a dependency on wavelength. Examples of absorbers, of differentwavelengths, are oxygen, water vapor and carbon dioxide.

There are two types of absorption, the constant absorption due to carbon diox-ide, nitrogen and methane from the air, and a variable from water and ozone.

7.4 Merging absorption and scattering 41

Carbon dioxide, nitrogen and methane levels are usually very stable and can beconsidered as constant. The water and ozone on the other hand can vary a lot.

To quantify the damping of the signal a quantity, transmittance, is defined, it isthe coefficient that determines what amount is still left of the original signal, andit is defined as Eq. (7.1) where Kα defines the absorption and R the distance.

τalpha(λ) = e−KαR (7.1)

Consider an image with an object depicted in it, using visible light. Let’s saythere is a fog when the picture is taken. A lot of the energy from the depictedobject will be absorbed and scattered by the fog. If the picture is taken at daytimethe fog will probably be visible as a milky semi-transparent object. This is becauseof rays of other signal sources being scattered by the fog, in to the FOV of thecamera. These different rays can emanate from the depicted object itself.

7.4 Merging absorption and scattering

By joining Kα and Kσ into one unit γ(λ), the extinction coefficient, the effects ofatmospheric obscurants can easily be modeled.

γ(λ) = Kα(λ) + Kσ(λ) (7.2)

τ(λ) = e−γ(λ)R (7.3)

γ is the extinction coefficient (or effect attenuation coefficient) and is measuredin nepers, Np/km, and can be translated into dB/km as Np/km = 0.23dB/km.Kα is absorption, and can be divided into molecular and aerosol parts.

The transmittance has values ranging from 0 to 1, where 1 denotes no distur-bance of the signal and 0 that nothing of the original signal is transmitted to thesensor. The calculations are of course dependent on the distance between the signalsource (the object) and the sensor, see Eq. (7.3).

τa =1

λ2 − λ1

∫ λ2

λ1

e−γ(λ)Rdλ (7.4)

A common simplification made is that the extinction coefficient can be consid-ered to be independent of wavelength for the interval (λ1 and λ2), the mean Eq.(7.4) then becomes equal to Eq. (7.3).

7.5 Rain

Rain affects target-to-background contrast for IR in two ways. The first is bycooling the object (the signal source) and the second is that the signal itself isattenuated. The cooling of the object is not modeled here, but should be im-plemented in the signal generation in Chapter 6. It has been investigated and

42 Atmosphere

Approximate Ranges of Extinction Coefficients (Np/km)Atmospheric Visible Near IR Thermal IRObscurant 0.4 to 0.7 µ m 3 to 5 µ m 8 to 12 µ m

Gases ∼= 0.02 0.25 - 0.73 0.03 - 0.8Haze 0.2 - 2.0 0.02 - 1.0 0.02 - 0.4Fog 2.0 - 20 1.0 - 20 0.4 - 20Rain 0.3 - 1.6 0.3 - 1.6 0.3 - 1.6Snow 2.0 - 12 2.0 - 12 2.0 - 12Dust 0.2 - 4.0 0.2 - 4.0 0.2 - 4.0

Table 7.1. Extinction Coefficient of Atmospheric Obscurants (low altitude).

Extinction Coefficient Descriptive Term Visual Range< 0.1 Np/km Very low > 30 km, very clear

0.1 - 0.5 Np/km Low 6 - 30 km, clear to hazy0.5 - 2 Np/km Medium 2 - 6 km, hazy> 2 Np/km High < 2km, foggy

Table 7.2. Explanations to Table 7.1.

modeled with good accuracy (according to [11]). Of course heavier rain has a moreprofound impact resulting in a higher γ. For thermal IR rain can be modeled withan extinction coefficient of 0.3 to 1.6.

Vehicles traveling on a wet road will produce a fog like situation behind them.This can be modeled by adding some local fog near the car. Note that we onlyhave considered stationary objects so far in this description and that this is becausemoving objects are considered to be detected at all times. But for recognition andidentification purposes, the concept of “local fog” can be useful.

7.6 Snow

Not surprisingly, snowing can also effect the signal. It seems as if transmittance isless influenced for shorter wavelengths than for longer. As intuition also says thetransmittance decreases when the snowing increases. Cooling of the signal sourcein thermal IR is a factor to consider, it should be dealt with in the same way as inthe equivalent case for rain.

For cold and snowy conditions, an extra snowfall could be added behind a fastmoving ground vehicle.

7.7 Dust

Cars traveling on dirt roads in dry conditions will create a dust cloud behind them,making it more difficult to see objects behind it. The dust could also be a result

7.8 Noise 43

from explosions or storms. As seen in Table 7.1 the extinction coefficient seems tobe independent of wavelength for the dust, and hold values from 0.2 to 4.

7.8 Noise

If there was no fog (or other obscurant) present, and the conditions otherwise areequal to when the photo, mentioned above, was taken, the signal from the object(o) would be almost undistorted. The difference between these two photos (foggyand clear) can be described as a distortion by the fog, and it could be considered asnoise. This noise could then in its turn be modeled by extensions to the Johnsoncriteria as suggested in Chapter 4. The same model can also be applied for rain,snow and so on.

In order for this to work there has to be a connection between the extinctioncoefficient (or the transmittance) and the SNR. A perfectly clear day, one could saythat τ = 1 and that it means no noise (SNR=∞). In a perfect fog, where nothingof the original signal is detected at the sensor, τ=0, is represented massive noise(SNR =-∞). In the foggy example above where some of the signal (s = o + c) isdelivered to the sensor 0 ≤ τ < 1 (−∞ < SNR < ∞). If half the original signalis detected that corresponds to the signal being equally strong with the noise, andhence τ = 0.5 (SNR=0)

So the remaining part of the object signal that is received by the sensor is,s = o− n, and the noise is n=τ · o.

SNR = 10logE[s2(x, y)]E[n2(x, y)]

= 10logE[(τo(x, y))2]

E[(1− τ)o2(x, y)]= (7.5)

= 10logτ2 · E[(o(x, y))2]

(1− τ)2E[o2(x, y)]= 20log

τ

1− τ(7.6)

The signal to noise ratio is then linked to the extinction coefficient through Eqs.(7.3) and (7.5), and becomes Eq. (7.7).

SNR = −20log(eγR − 1) (7.7)

Note that atmospheric conditions can vary from one area to another, and thatthe signal from the scene to the sensor can pass through many of those.

44 Atmosphere

Chapter 8

Information and data fusion

How to treat multiple data and information that fits to this general probabilisticconceptual sensor model is discussed here.

8.1 Introduction

Today the technological advances have lead to more complex and cheaper sensors.The trend is to integrate many types of sensors in a system in order to get morecomplete information about the situation and to avoid the effects of camouflage.A missile could for instance be equipped with a GPS and an inertial navigationsystem cooperating to give a more accurate position, the position is then comparedwith a map database and the forward looking TV-sensor can verify the position.For night flight TV-systems are not the ultimate choice, but instead a radar canbe used. For some purposes a FLIR can be chosen. To make the systems moreuseful, independent of weather and time of day, radar, IR and visible sensors canbe used together. This generally leads to better information. The data from thethese sensors then have to be merged to form an output.

Information from one source, whether a sensor, a database or an operator canbe regarded independently, one by one, or their inputs together can be seen as one.

When looking at data from different sources describing the same object humanstend to use parts of information from all the sources to create a more accurate,or for us more suiting, picture. The data from the different inputs are somehowweighted together to form one “super data” to help in decision making. Today thereare many systems designed to extract different kind of information from differentsources and compile them into a single unit. Research is also conducted in multisensor fusion to improve performance and new models are developed.

45

46 Information and data fusion

8.2 Different kinds of fusion

The general way of processing data and information for fusion can be divided intothree groups, according to [11], sensor-level fusion, central-level fusion and hybridfusion.

Sensor-level fusion is when each sensors data is processed independently and theresult from every such process is joined together, see Fig. 8.1(a). This is possiblythe simplest and most straight forward approach. Since processing is made on eachsensor, a lot of hardwear is needed which increases costs but reduces the time forprocessing.

In central-level fusion very little is done to the information from each sensor,perhaps some simple filtering and baseline estimation but not much more. Theidea is to merge relatively unprocessed data into the decision algorithms, see Fig.8.1(b). This will generally lead to a better result than from the sensor-level fusionprocesses. However it has a higher computational complexity and might thereforbe slower than the sensor-level fusion.

The third kind, hybrid fusion, is a mix of the two models above. Every sensorsends it raw data to a central-fusion process as well as processing the data itself andreporting the results to a sensor-level process. The two, sensor-level and central-level processes, make their calculations mostly independent but could also shareinformation, see Fig. 8.1(c).

8.3 Fusion modeling

First lets engage in some basic probability theory. Consider relation Eq. (8.1) forthe events Xi and Xj .

P (Xi ∪Xj) = P (Xi) + P (Xj)− P (Xi ∩Xj) (8.1)

If Xi and Xj can be considered independent, Eq. (8.1) turns into Eq. (8.2).

P (Xi ∪Xj) = P (Xi) + P (Xj)− P (Xi)P (Xj) (8.2)

Sensor-level fusion is possibly the simplest to model from a conceptual view-point. Since every sensor processes its data independently, no cumbersome regardsto other sensors must be made, if the data from each sensor and its processing isconsidered to be independent from each other.

Consider the possibility of sensor i making a correct decision (i.e. detects anobject or puts the object in the right category and so on), and denote the probabilityfor that with P (Xi) = pi. The probability that at least one of two sensors willmake a correct decision is then P (Xi ∪Xj). Let P ({Xi) = qi = 1− pi denote thepossibility of failing to detect a presented object. The possibility of both sensorsfailing to classify will then be P ({(Xi∪Xj)) = P ({Xi∩{Xj), and if the events areindependent qiqj = (1− pi)(1− pj).

The possibility of at least one of the sensors being able to classify the objectwill then be, as seen in Eq. (8.3) and (8.4).

8.3 Fusion modeling 47

Fusion

Processor

Sensor N

etc.

Feature extr.Transducer

Sensor 1

etc.

Feature extr.Transducer

(a) Outline of the sensor-level fusion process.

Sensor 1

Sensor N

FusionProcessor

cue

data

(b) Outline of the central-level fusion process.

Fusion

ProcessorSensor N

Processing N

Sensor-levelfusion

Sensor 1

Processing 1

(c) Outline of the hybrid fusion process.

Figure 8.1. Different fusion methods.

P2 = 1−Q2 = 1− qiqj = 1− (1− pi)(1− pj) (8.3)

For N sensors this turns into

48 Information and data fusion

PN = 1−QN = 1−N−1∏i=0

1− pi = PS−L (8.4)

and Eq. (8.4) will be the conceptual description of sensor-level fusion, PS−L.Central-level fusion must be considered to have greater accuracy than sensor-

level, since the one of the extreme forms of central-level fusion could be identicalto sensor-level fusion. On the other extreme there is a fusion combining the datafrom every sensor in order to make decisions, if the data is used correctly this wayshould be better than the sensor-level type for every case, or at least equal. Ifconsidered from an information perspective the sensor-level fusion discards someinformation early in the processing. This information might not be necessary forthe individual sensor processing, but it might be of help for another type of sensor.Center-level fusion has the ability to use all of the information, and should thereforbe better. This leads up to an approximation of center-level fusion, according toEq. (8.5), the probability is of course always smaller than one.

1 ≥ PC−L ≥ PS−L = 1−N−1∏i=0

1− pi (8.5)

For hybrid fusion no direct expression will be presented. One could concludethat it should be better than sensor-level fusion, and that it has also the possibil-ity to perform a more accurate result than center-level fusion since both of thesesystems are components of the hybrid fusion. Of course the performance of thesystems will depend most certainly on their implementations. A smart implemen-tation of sensor-level fusion can of course outperform a poorly designed center-levelfusion system.

Chapter 9

Summary and conclusions

9.1 Summary

The objective for this thesis was to investigate the possibility of a conceptual sensormodel, and to try to find a method that could be possible to use for different kindof sensors. The investigations has been concentrated to a staring forward lookinginfra red sensor, especially in the wavelengths band 8-12 µm, known as thermal IR.

The developed model starts with calculating the resolution that the sensor ob-serves close to an object in a scene. The resolution is dependant of the geometry,the angles and distances between sensor and objects, as well as the physical prop-erties of the sensor. Then the transform used to convert resolution to probabilityhas to prepared with some regards to the terrain and atmospheric conditions.

The terrain and objects in a scene has to be measured in order to quantifythe clutter, and its influence on object detection. This is done by relating thesurroundings of an object, by comparing the size and intensities of other objects,to the object itself. The result, describing the similarities of the object comparedto its background, then changes the resolution to probability transform by shiftingit towards higher or lower resolution.

The atmospheric conditions will also effect the possibility to detect an object.Rain, snow and dust should of course change the transform as well. This is doneby approximating the loss of signal to the sensor and relating that to a signal tonoise ratio. The SNR is then used to shift the resolution to probability transformin a similar way to that of clutter.

Since sensors are getting cheaper and the demands on performance under vary-ing conditions are getting higher the use of multiple sensors are more common. Tomerge information from many sensors can be done by probability theory for thisconceptual sensor model. This gives a sensor platform the possibility of finding ob-jects under varying conditions when one or many sensors have poor performances.This is getting more and more common in modern technical platforms.

49

50 Summary and conclusions

9.2 Discussion

This work attempted to model a FLIR, with some hope of finding a model todescribe sensors in general. The conclusions are that this is possible, and that thismodel could be used for other sensors as well.

The objectives were to develop a conceptual sensor model, that means that itshould approximate the behaviour of the sensor but not necessarily work in thesame way. By choosing the tree factors that this model is based on, the sensormodel turns into a simpler entity to model and is still remaining true to the objec-tive. These three factors are suitable with the initial directives and plans for theinvestigation.

When no pseudo images or sequences of images has to be generated the speedincreases, and the time to calculate one step of the sensor is shortened. This furtheradds to the positive performance of the sensor model.

When generating the probabilities the idea is that a system will not alwaysrespond the same way to a given situation, but if the same situation is tested overand over again it will in most of the cases respond in the same way. This will leadto more realistic testing of a scenarios. This was the initial thought of how thesensor model should work.

Since the technique the model is based on, the Johnson criteria and similarmethods, has been used for 50 years this model should work as well. In fact, byadding atmospherical effects and distortion of the signal a substantial improvementhas been done. This will make it easy to test ”the same scenario” under differ-ent conditions, and also find out how rain or dust influence the outcome. Thismight however be the weakest part of the model since it is new and not verified orcompared to other models. A comparison should be done to physically or experi-mentally validated models.

The most difficult thing to model was the signal (image) processing done onan image created by a 2D detector array. The efficiency of a signal processingalgorithm is very difficult to quantify in a general case. Instead a discussion ofinformation possible to retrieve from those images lead up to the importance ofthe resolution compared to the size of the object. About the same time a modelfor human acquisition was found, the Johnson criteria. John Johnson had alreadyfound a connection between the information found by the resolution and possibili-ties of detection, recognition and identification. This model was adopted althoughit modelled human behaviour, this because of it simplicity and the fact that thesame ideas could be used for computerised sensors as well (maybe with some scalingof the resolution to probability transform).

With the Johnson criteria well established, enhancements could be made. Oneof those enhancements, the dependence of clutter, were also found in the literaturebut the other, dependence of noise, could not be found. An attempt to develop itwas made.

The generation of the characteristics of the signals is quite straight forward.The effects of the atmosphere has been modelled in many ways in the past, andthere were many found in the literature. None of them seemed to fit to this model

9.3 Conclusion 51

though. By introducing the effects (scattering and absorption) as disturbances ofthe signal, it could be defined as noise and linked to the resolution to probabilitytransform dependent on noise.

The experiments mentioned in Chapter 4 could not verify the independencebetween clutter and noise. The author however still believe that it holds.

The down side of the model is that the actual resolution to probability trans-forms are missing for an automated system, since the tables presented in the reportare based on human observers. If an object is masked in the terrain, let’s say innarrow forests, it should naturally be harder to detect than in an open field. Howthis should be accomplished is not determined in here, nor was it planned to.

As shown in this thesis it is possible to construct a probabilistic conceptualsensor model by calculating a few descriptive features. The object searched for andits characteristics compared to the nearest terrain and other objects are included aswell as how the signal is changed on its path to the sensor. Those descriptive termsare resolution, signal to clutter ratio (SCR) and signal to noise ratio (SNR). Thedetector array has not been considered much but some of the extensions suggestedat least try to model the detector.

9.3 Conclusion

This model should not be seen as only a model for FLIRs. The factors (resolution,SCR and SNR) can be used for other types of sensors, as well as for FLIRs, tomodel their behaviour. The three factors are not specific for one sensor type butalmost universal. Their descriptive features fits a lot of sensors although that theymight not be calculated in the same way. Other TV-like sensors should be straightforward to implement, but radar and laser radar might be more complicated sincethey are more of three dimensional character.

Based on this work and experiences it is found possible to build a conceptualsensor model based on a few descriptive features. The features used here areresolution which aim at describing the level of details possible to extract from ascene. Clutter (or SCR) is used to find the resemblance of the object comparedto the surroundings. The noise (or SNR) that tries to model the atmosphericconditions and physical properties of the sensor.

9.4 Future work

A suggestion for future work in this area would be to make investigations to deter-mine the probabilities of acquisition in noise, in order to complete the model forhuman observers. The next stage should be to implement it. Evaluations shouldbe made, comparisons to actual sensors or other validated models, to produce theresolution to probability transforms necessary.

Investigations on false target detection should be made and implemented. Somework regarding false target detection has been found for instance [3]. By adding

52 Summary and conclusions

false target indications the model gets even closer to a real system and performanceshould increase.

9.4.1 Other sensors

The basic descriptive features extracted are not specific for TV-like systems andcould be used for other sensors, even for active sensors like radar and laser radar.

9.4.2 Visual

For sensors using visual wavelengths colors can be added to the model. A suggestionof how that could be done is to construct a conceptual sensor model for eachwavelength band. Three colors (red, blue and green) could then be merged withsomething like the methods presented in chapter 8.

9.4.3 Radar

A commonly used active sensor is the radar. Since the 1920 the radar has beendeveloped and used. The big breakthrough came in the Second World War, and ithas since then been an important factor for military applications, offensive as wellas defensive. There are many different types of radar including pulsed, modulatedcontinuous wave and compressed waveform.

The resolution of a radar is dependent on what type of radar is used. Say forinstance that some pulse radar is modelled, then the resolution, ra, is dependenton pulse length, angle to the nadir and the antenna.

ra = 2Rsin(θa

2)

=2hsin(θa/2)

cosθ(9.1)

rr = cτ

2sinθ(9.2)

Where ra is the resolution in the direction of travel, dependent on the antenna,and rr is the resolution perpendicular to ra.

The terrain and objects in the vicinity of the target could be modelled withsome kind of clutter, in the same way as for IR. Needed is a quantity relating thespecific features of an object to the surroundings, a clutter measure.

Noise will also be present for a radar. The atmospheric model created in thisthesis should be possible to use for radar as well. The properties of the atmosphereare similar to those for IR, in fact the wavelengths used in radar applications arebecoming shorter and shorter, and are today close to those of IR.

9.4.4 Laser radar

Resolution is also here a crucial factor for the performance. The laser radar takesthree dimensional pictures of a scene, instead of using a two dimensional projection.

9.4 Future work 53

The resolution will become a question of how close the detector elements can bearranged, what the minimum time for detection of one element is and the distance.

The background is modelled by clutter in the same way as for TV-like systems,by size and other similarities to the object of interest, it will however be a 3Dclutter measure.

Again the atmosphere can be modelled as noise, and probably occlusion byvegetation too in the same way as in this thesis. The wavelengths used in laserradar are no different from the visual or infra red.

54 Summary and conclusions

Bibliography

[1] Leif Carlsson, Magnus Gustafsson, Patrik Hermansson, Mikael Karlsson, NilsKarlsson, Andris Lauberts, Fredrik Nasstrom, and Mathias Wilow. Multisen-sormalsokare - klassificering av markmal med datafusion. Technical report,FOI-R–0652–SE, Sensorteknik, 2002.

[2] Per-Erik Danielsson, Olle Seger, and Maria Magnusson Seger. Bildanalys.Department of Electrical Engineering, Linkopings Universitet, 2001.

[3] Theodore J. Doll and David E. Schmieder. Observer false alarm effects ondetection in clutter. Optical Engineering, 32(7), July 1993.

[4] James F Dunnigan. Wargames Handbook, Third Edition. Writers Club Press,2000. ISBN: 0-595-15546-4.

[5] ed. L. Biberman. Resent Psychological Experiments and the Display Signal-to-Noise Ratio Concept, Perception of displayed Information.

[6] J.E. Nygaards et al. The measurements of stimulus complexity in high-resolution sensor imagery. Technical report, Aerospace Medical Research Lab-oratories, Wright-Patterson AFB, 1964.

[7] Jorgen Karlholm et al. A survey of methods for detection of extended groundtargets in eo/ir imagery. Technical report, FOI, 2003.

[8] Forsvarsmakten. Telekrig. Forsvarsmakten, 1997. M7746-168001.

[9] Andreas Hornedal, Per Westterlund, and Johan Ohrmalm. Sensorer. Technicalreport, FFAP-F-200 Flygtekniska Forsoksanstalten, 1999.

[10] John Johnson. Analysis of image forming systems. Technical report, U.S.Army Engineer Research and Development Laboratories, Fort Belvoir, Vir-ginia, 1958.

[11] Lawrence A. Klein. Millimeter-Wave and Infrared Multisensor Design andSignal Processing. Artech House, Inc., ISBN: 0-89006-764-3.

[12] Carl Nordling and Jonny Osterman. Physics Handbook, Fifth edition. Stu-dentlitteratur, 1996. ISBN: 91-44-16575-7.

55

56 Summary and conclusions

[13] J. S. Przemieniecki. Mathematical methods in defens analyses, 3rd ed. Officeof Naval Research, Department of the Navy, Washington, DC, 1989. ISBN0-9603590-1-X.

[14] Kahlid Sayood. Introduction to Data Compression, 2nd ed. Morgan Kauf-mann, 2000. ISBN: 1-55860-558-4.

[15] David E. Schmieder and Marshall R. Weathersby. Detection performance inclutter with variable resolution. IEEE Transactions on Aerospace and Elec-tronic Systems, AES-19(4), July 1983.

[16] Frank L. Pedrotti S.J. and Leno S. Pedrotti. Introduction to Optics, SecondEdition. Prentice-Hall, Inc., 1993. ISBN: 0-13-016973-0.

[17] Gary Waldman and John Wootton. Electro-optical Systems modeling. ArtechHouse, Inc., 1978. ISBN: 0-89006-541-1.

[18] William L. Wolfe and George J. Zissis. The Infrared Handbook, 3rd ed. Officeof Naval Research, Department of the Navy, Washington, DC, 1989. ISBN0-9603590-1-X.

På svenska Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under en längre tid från publiceringsdatum under förutsättning att inga extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/ In English The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances.

The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.

According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.

For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/ © Mattias Sonesson