FVSysID ShortCourse 6 Validation

AIAA Short Course: Flight Vehicle System Identification in Time Domain, Aug.2006 Validation/1Dr. Ravindra Jategaonkar

ParameterAdjustments

Model Response

ResponseError

ActualResponseInput

Maneuver

ModelValidation

ComplementaryFlight Data

Identification Phase

Validation Phase

OptimizedInput

Flight Vehicle

IdentificationCriteria

EstimationAlgorithm /Optimization

Data Collection& Compatibility

easurementsM

ethodsM

odelsM

A Priori Values,lower/upperbounds

Model Structure

Validation of Flight Vehicle System Identification

-+

MathematicalModel /

Simulation


This page is left intentionally blank.


Data Base Validation (1)

System identification:“Given the system inputs and responses, what is the model?”

Model Validation:“How do you know that you got the right answer?”

Definition:“… Validation refers to the process of confirming that the conceptual model is applicable or useful by demonstrating an adequatecorrespondence between the computational results of the model and the actual data (if it exists) or other theoretical data.”

Broad classification:

1) Statistical properties of the estimates,

2) Residual analysis, and

3) Model predictive quality


Data Base Validation (2)

Several criteria1) Standard derivations

2) Correlation among the estimates

3) Goodness of fit

4) Plausibility of estimates (WT data base)

5) Statistical analysis of residuals (bias, variance, covariance, and PSD)

6) Model deficiencies in terms of residual control inputs (inverse simulation)

7) Model predictive capability

"ACID TEST“Simulation and comparison with flight data not used in identification

- Criteria to be used in conjunction with each other

- Basic philosophy remains same for simple as well as for global models


Statistical Accuracy of Parameter Estimates

Measure of accuracyClues into the effectiveness, or lack thereof, of model parameters

Fischer information matrix provides a good approximation to the parameter error covariance matrix P:

Standard deviations (Cramer-Rao bounds):

Correlation coefficients:

( ) ( )1

1

1−

=

−

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧∑ ⎥⎦

⎤⎢⎣⎡

Θ∂∂

⎥⎦⎤

⎢⎣⎡

Θ∂∂

≈N

kk

Tk tyRtyP

iii p=Θσ

ji

ijji pp

p=ΘΘρ


Statistical Accuracy: Practical difficulties

Standard deviations:- Estimated error bounds are too optimistic

-> use fudge factor of 5-10(ad hoc approach to bridge the gap between theory and practice).

-> Scatter in the estimates larger than standard deviations(non trivial issue; scatter can not be avoided)

- Variances of the estimated parameters and the covariances(off diagonal elements of the error matrix) must be small.

- Correlation coefficients:-> Linear dependency (> 0.9; >0.95; >0.98…..)


Residual Analysis: Goodness of fit

Cost function value:- ML estimation: determinant of R (covariance matrix of residuals) - Most direct way to evaluate model quality- Cost function value should be small:

Value depends on the number of system outputs, noise level, and unitsof the variables (radians or degrees; g or m/s2; m/s or kts)

- Absolute value alone of limited use

- No consistent workable criteria has been put forward- Old adage: Goodness of Fit is no criteria.

It is a necessary but not a sufficient condition.

Peculiarity of cost function:Very low value does not guarantee equally good matching of all variables; In worst case a perfect match of one variable yields minimum.


Residual Analysis: Overall Fit and Decomposition of Fit Error

Theil’s inequality coefficient

Ratio of the root mean square fit error and the root mean squarevalues of the measured and estimated signals summed together

TIC is normalized: between 0 and 1

0 --> perfect fit1 --> worst case ==> two time series are very significantly different

Thumb rule: Acceptable TIC value depends on the application, in general 0. 25 to 0.3 indicates a good agreement

∑=

+∑=

∑=

−=

⎥⎦⎤

⎢⎣⎡

⎥⎦⎤

⎢⎣⎡

⎥⎦⎤

⎢⎣⎡

N

1k

2)kt(yN

1N

1k

2)kt(zN

1

N

1k

2)kt(y)kt(zN

1TIC


Residual Analysis: Decomposition of Fit Error

Bias proportion: measure of systematic

error in the identified model

Variance proportion: represents the model’s ability

to duplicate the variability inthe true system

Covariance proportion: measure of non-systematic

error

and denote the mean vales; and the standard deviations and correlation coefficients.

Bias and variance proportions should be very small (typically less than 0.1), in an ideal case, zero;

Covariance proportion should be close to one.

∑=

−

−= N

kkiki

iiMi

tytzN

yzU

1

2

2

)]()([)/1(

)(

∑=

−

−= N

kkiki

iyzSi

tytzNU i

1

2

2

)]()([)/1(

)( σσ

∑=

−

−= N

kkiki

yiziCi

tytzNU i

1

2)]()([)/1(

)1(2 σσρ

1=++ Si

Ci

Mi UUU

iz iyσ ρ


Inverse SimulationSimulation: given inputs u and system model f, find system output y

Inverse Process of calculating desired controls, for the givensimulation system model and response.

Closely similar to the control problem (namely, given system model f and output y, find control input u).

Subtle difference:Classical control problem does not require measured system responses.Inverse simulation explicitly needs measured responses, and leads to

controls based on residuals.

Aircraft

Feedbackcontroller

Mathematicalmodel

Inputu

Measured Responsez

yComputed Response

Δu+_


Elevator Aileron

Angle of attack, deg0 10 20

-.16

-.28

-.20

-.24

/rad

Clξ

Angle of attack, deg0 10 20

-1.0

-1.6

-1.2

-1.4

/rad

Cmη

Rudder

Angle of sideslip, deg0 10-10

-.14

-.18

-.16

/rad

Cnζ

Wind tunnel / analytical prediction

Estimates from flight data

Model Plausibility (1)

Example: Control surface effectiveness - ATTAS VFW-614


Rolling motionShort period motion

Mach number

-4.0

-10.0

-6.0

-8.0

/rad

0 0.2 0.60.4

Cmq

Cmα.

Mach number

-.80

-1.1

-.90

-1.0

/rad

Clp

0 0.2 0.60.4

Dutch roll motion-.10

-.70

-.40

/rad

Mach number

Cnr

Cnβ.

0 0.2 0.60.4

Wind tunnel / analytical prediction

Estimates from flight data

Damping Derivatives - ATTAS VFW-614



0

1

P RP

HI

PSI

VK QTH

EU

KW

K H

Eigenvalue = ( -1.61196 , + - 0 )

0

1

P RP

HI

PSI

VK QTH

EU

KW

K H

Eigenvalue = ( -0.965076 , + - 1.75372 )

0

1

P RP

HI

PSI

VK QTH

EU

KW

K H

Eigenvalue = ( -0.142527 , + - 1.54814 )

0

1

2

3

4

5

6

-1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0

Im[ra

d/s]

Re [1/s]

0.3 0.2 0.1

00

Eigenvalues and eigenvectors completely characterize the system behavior

Linearized system matrix


Dutch rollShort period


Model Predictive Capability (1)

Proof-of-Match (POM):

An important part of flight simulator certification and acceptance.

Compare the flight measured system responses with those predicted by the model for the same (“identical”) control inputs.

In this POM process, the identified (aerodynamic) model is kept fixed.

Other important issues:

1) Proper choice of the data set to compare against,

2) Initial conditions on the state variables (trim), and

3) Criteria to check adequate correspondence between model response and measured aircraft outputs.



Proof-of-Match (POM):

Data set: Complementary flight data, i.e., flight maneuvers not usedin the estimation the of aerodynamic (“Acid test”)FAA has defined a set of roughly a little more than 100 different cases, covering different modes of aircraft motion, and configurations

Initial conditions: Ideally, the simulation is to be started from the same initial conditions as in the flight

Criterion: FAA tolerances (to avoid subjective evaluation)

Non trivial task: Measurements corrupted by noisePresence of turbulenceModeling errors

Allow small biases on the selected initial conditions and on themeasured control deflections



ReversibleFlight Control

Dynamics

Measured Aircraft MotionVariables

MeasuredPilot InputForces

Control SurfaceDeflection

Aircraft MotionVariables

Rigid BodyDynamics

Flight controls stand-alone

Measured Control SurfaceDeflect.

Rigid-body stand-alone

ReversibleFlight Control

Dynamics

Rigid BodyDynamics

Integrated model

Control SurfaceDeflection

Aircraft MotionVariables

Pilot InputForces

Stand-alone and integrated models



Elevatordeflection

Pitch rate

Verticalacceleration

time sec

deg/s

deg

g

10

0

-5

15

0

-10

-2.5

0.5

0 2.5 5 7.5 10

Time domain verification:

Majority of VT in time domaine.g., 1.5°/s for rates, 0.1 g for accelerations

Few are in terms of dampingratio and frequency; e.g.,0.02 damping ration, 10% period

Recent effort:Frequency domain criteria

Validation test 2c11: Short period dynamics


Frequency domain criteria

Bode plots of measured to model estimated responsee.g., qm/q (error function)

Ideally, for a perfect match:0 dB magnitude and 0 deg phase angle

over the frequency range

Boundaries are based on the LOES (Low Order Equivalent System) mismatch criteria: unnoticeable dynamics

Brings our more clearly the range of model applicability

Important for high authority FCS where aeroservoelasticeffect may be dominant


0.3 1 10 20Frequency rad/s

Phase

deg

-80

-40

0

40

-10

-5

0

5

10

Magnitude

dB

Validation test 2c11: Short period dynamics

More restrictive tolerance band in the range of 1 to 5 rad/spilot cross-over frequency, allows more error outside of it (LOES phase-lead leniency), and are asymmetric at low frequencies (allow more phase lead error than phase lag error) to have better fidelity with respect to lag error which might cause pilot-induced oscillations.



-30

-20

-10

05

P / AILERON and P / AILERON

Magnitude

-360

-300

-240

-180

1 10Frequency

dB

rad/s

Phase

deg

From Identified model:

Linearized system matrices A, B, C, D

Frequency response matrix

From measured flight responses:

Approximation of frequency responsesthrough FFT techniques

Noisy measured data: smoothing

DB)AIj(C)j(G +−−= 1ωω


ReferencesJategaonkar, R. V., Flight Vehicle System Identification: A Time Domain Methodology,Volume 216, AIAA Progress in Astronautics and Aeronautics SeriesPublished by AIAA Reston, VA, Aug. 2006, ISBN: 1-56347-836-6http://www.aiaa.org/content.cfm?pageid=360&id=1447

Hamel, P. G. and Jategaonkar, R. V., “Evolution of Flight Vehicle System Identification”, Journal of Aircraft, Vol. 33, No. 1, Jan.-Feb. 1996, pp. 9-28.

Hodgkinson, J. and Mitchell, D., “Handling Qualities”, in Flight Control Systems, Pratt, R. W. (Ed.), AIAA Progress in Astronautics and Aeronautics Series, Vol. 184, 2000, Chapter 4.

Jategaonkar, R.V., “Determination of Aerodynamic Characteristics from ATTAS Flight Data Gathering forGround-Based Simulator”, DLR-FB 91-15, 1991.

Murray-Smith, D. J., “Methods for the External Validation of Continuous System Simulation Models: A Review”, Journal of Mathematical and Computer Modelling of Dynamical Systems, Vol. 4, No. 1, 1998, pp. 5-31.

N.N., “Airplane Simulator Qualification”, FAA Advisory Circular, AC 120-40C,Interim Version, Jan. 1995.

N.N., “ Joint Aviation Requirements - Aeroplane Flight Simulators”, JAR-STD 1A, Westward Digital Ltd., Cheltenham, England, April 1997.

Tischler, M. B., “System Identification Methods for Aircraft Flight Control Development and Validation”,NASA TM 110369, Oct. 1995.

see also the References from the Section “Examples”

Documents

FVSysID ShortCourse 6 Validation