JETSIM Simulator DevelopmentAuthor: Shamaz Khan

Flight Science and Technology Lab (FS&T), Liverpool

Abstract

This paper summarises the development of JETSIM simulator in University of Liverpool Flight Science and Technology Lab. The fixed base simulator was developed to assist students enrolled in pilot studies degree. The simulation runs X-Plane home version, and comprises of actual aircraft hardware (Jetstream 31). The aircraft flight model for Jetstream was developed using Plane Maker in X-Plane, and validated against the flight test data. Handling qualities analysis was performed on the model to validate against more sophisticated FLIGHTLAB model. This was done to ensure that model does not exhibit serious limitations before installed on the simulator.

Introduction

The quantification of simulation fidelity underpins the confidence required for the expanding use of flight simulators in design, research and pilot training. Whilst regulatory simulators standards exist and new standards are in development, but the research has shown that these standards do not provide fully quantitative approach for assessing simulation fidelity, even in research environment. This report progress on developments of JETSIM Simulator at the University of Liverpool, and its subsequent use in a research project (Lifting Standards) aimed at creating new predicted and perceived measures of simulator fidelity assessment. The analysis used throughout the report, has been evolved from handling qualities engineering. The results from flight test of Cranfield University’s Jetstream 31 research aircraft, JetSim simulator equipped with X-Plane simulation software and HELIFLIGHT piloted simulation trials are presented to show the strong connection between handling qualities engineering and fidelity assessment. The issue of pilot perceived fidelity is examined, along with the pilot-in-the loop analysis and

development of new metrics discussed for the aircraft landing task.

In context on training simulators, regulatory authorities have produced functional performance standards, along with associated training credits, to provide a framework for the acceptance of the synthetic training device. Documents such as JAR-FSTD 1A and JAR-STD A describe the qualifying criteria and procedures for the fixed wing flight training device and detail the component fidelity required to achieve a fit for purpose approval. Whilst these standards serve a vital role in regulatory process, the influence of the cueing environment on the pilot opinion during qualification needs to be understood better. Currently, there are no quantitative methods used to assess the fidelity of the overall simulator, with pilot performing the task.

For a fidelity assessment procedure to be demonstrated, a pilot-vehicle structure was created for the nominal flight vehicle. This includes a pilot dynamic model in both primary and outer loops. Later the simulation limitations were included within the model. Flight simulator limitations will include two sets of motion dynamics (no motion and

1

motion), with the visual cue quality model, the latter can be implemented in X-Plane using the Instructor Operating system (IOS). A control strategy for a particular task and vehicle was developed for the Landing manoeuvre. The results obtain from this model were used to address the simulator’s limitation on pilot’s primary control loop. Within this sort of model, the visual time delays within the simulator were modelled using appropriate gain values.

Overall Methadology

The handling qualities of the Jetstream 31 aircraft were evaluated in two phases, first for the longitudinal axis, and second for lateral axis. The analysis was carried out using FLIGHTLAB and X-Plane, in order to develop an initial framework for the simulator flight model. Within FLIGHTLAB, a standardised Jetstream 31 model is available and allows both linear and nonlinear responses to be modelled. Data from FLIGHTLAB and X-Plane was then exported for visualisation and analysis in Matlab. The obtained data was compared and assessed according to the MIL-1797A standards, and the aircraft was then defined in terms of predicted handling qualities per these specifications. Offline analysis is intended to produce predicted HQRs – these are not the actual handling qualities of the aircraft, but are intended to be used as a guide to estimate how the aircraft will perform online. The sections of offline analysis are:

Stability analysis (Longitudinal and

Lateral)

Control Anticipation parameter (CAP)

Control Power Analysis

Bandwidth analysis (longitudinal pitch

attitude and roll attitude)

Static stability assessment

Mode Excitation

One underlying idea of this analysis is the notion that pilot is part of the system that is intended to accomplish a mission. The different task difficulties were distinguished by defining Flight Phase categories A, B, and C. Category A consists of demanding tasks such as air-to-air combat and refuelling, Category B includes less demanding tasks such as cruise and climbs and descents, and Category C includes terminal tasks such as landing and takeoff. It is the formal category, which is a main focus of this research.

In this phase, the Jetstream aircraft model was modelled in a full-motion base flight simulator where it carried out a set of prescribed Mission Task Elements (MTEs). Simulation trials were run within JetSim and HELIFLIGHT simulators, latter incorporating motion. This gives us an additional test metric, to differentiate the rating under the influence of motion. In order to acquire a pilot rating, the desired and adequate bounds of piloting task were defined to the test pilot. The Pilot rated each MTE using the Cooper Harper rating scale, justifying his decisions. These HQRs are known as the assigned handling qualities. The data recorded from these MTEs was then analysed in the same way as the data exported from the FLIGHTLAB software; plotted using Matlab and compared with the MIL-1797A requirements alongside the desired and adequate performance limits stipulated in the MTEs. Correlation between the pilot experience and the simulation data were also checked as both are equally merited in flight handling qualities assessment.

Offline Analysis

A number of methods are employed to define the short period dynamics criteria, such as bandwidth criterion, together with its

2

components, phase delay and dropback criterion. The analysis uses the nonlinear aircraft models, from both simulators. Better understanding of the nature of the manoeuvre and vehicle can be obtained by studying the control input transfer functions. For longitudinal dynamics, the two modes are phugoid (generally with the frequencies below 1 rad/sec) and short period (with frequencies between 1 to 10 rad/sec). Both of these modes represent the flying qualities of a conventional aircraft, or if augmented with active control one that is on a conventional response type. Short period flying qualities are of more interest here, since the mode is relatively rapid that governs the transient changes in angle of attack, pitch, flight path and normal load factor that occurs following a rapid control or gust input. Forward speed here remains relatively constant, and the mode usually is stable underdamped second order oscillation. These modes were triggered in both simulators by applying step inputs.

The control anticipation parameter is used in place for the thumb-print short period requirement, since the pilot’s ability to control the short period motion is affected by both the short period mode and the numerator. Therefore, this numerator effect has been incorporated into a criterion in the form of Control Anticipation Parameter (CAP).

It has been pointed out that CAP is proportional to the manoeuvre margin. The analysis for CAP was conducted for both simulators, but obtaining the natural frequency and damping of short period mode, according to the MIL-1797A specifications. These standards define the CAP as the ratio of initial pitch acceleration to steady state normal acceleration. The results for Control Anticipation Parameter, from FLIGHTLAB and X-Plane are illustrated in Figure 1. Both simulations were run at different speeds, for category B flight phase (cruise). The CAP values were evaluated based on the natural

frequency and damping ratio from the time history.

10-1

100

10-2

10-1

100

101

Damping Ratio SP

CAP

Control Anticipation Parameter Criteria CAT-B

Level 1

Level 2 X-PlaneFLIGHTLAB

Figure 1: short period frequency boundaries for Category B flight phase. This data represents the

eight different flying speeds (150 knots to 240 knots), from both of the simulators.

100

101

102

10-1

100

101

102

n -1 (g/rad)

nS

P (rad

/sec

)

Short Period Frequency Requirement - CAT B

Level 2 & 3

Level 1

Level 1Level 2

X-PlaneFLIGHTLAB

Figure 2: Frequency against Load Factor for CAT B Flight

It must be noted that all the configurations by FLIGHTLAB model, falls within the Level 1 Category. For X-Plane model, it is evident that at low speed the flight model is rated Level 2 or worse, but at higher speeds (above 170knots) the model is rated Level 1. The boundaries correctly predicted pilot ratings about 80% of the time. The data in Figure.2 represent those cases for which the short period natural frequency and Fs/n were within the present Level 1 boundaries.

3

A bandwidth frequency for an aircraft is a frequency at which the close loop tracking can take place without threatening the stability of the aircraft. It follows that an aircraft capable of operating at a large enough value of bandwidth will have a superior performance when regulating against disturbances. A bandwidth criterion is particularly useful for highly augmented aircraft in which the response characteristics are non-classical in form. In other words, it is defined as the highest frequency at which the pilot can double his gain or allow a 135 degree phase lag between control input and aircraft attitude response without loss of stability. If the gain bandwidth is sufficiently lower than the phase bandwidth the aircraft is more prone to pilot induced oscillations (PIOs).

MIL 1797A sets out separate handling quality levels for the pitch, roll, and yaw bandwidth. Different charts are available for high and low speed flights where the boundary between the two is set at 150kts.

The bandwidth values are obtained differently for each of the simulator. For FLIGHTLAB a linearised model was extracted from X-analysis in order to obtain values for the bandwidth of the aircraft and the phase response. However, for X-Plane, the bandwidth values were calculated directly from the time histories obtain from the step responses. The Fourier Transform (FFT) was applied to translate these time histories into a frequency domain data to obtain phase and amplitude plots. The results from these analyses are displayed on the associated charts defining the level of the bandwidth.

0 1 2 3 4 5 6 7 8 9 100

2

4

6

8

10

12

14

16

18

20

BW (rad/sec)

p (s

ec)

Pitch Attitdue Bandwidth Criteria FLIGHTLAB

Level 1

Level 2

Level 3 80kts100kts130kts160kts180kts

0 1 2 3 4 5 6 7 8 9 100

2

4

6

8

10

12

14

16

18

20

BW (rad/sec)

p (s

ec)

Pitch Attitdue Bandwidth Criteria X-Plane

Level 1

Level 2

Level 3 80kts100kts130kts160kts180kts

Figure 3: Pitch Axis Bandwidth handling qualities from XPLANE and FLIGHTLAB model

0 1 2 3 4 5 6 7 8 9 100

2

4

6

8

10

12

14

16

18

20

BW (rad/sec)

p (s

ec)

Pitch Attitdue Bandwidth Criteria

Level 1

Level 2

Level 3 X-PlaneFLIGHTLAB

Figure 4: Pitch attitude bandwidth Criteria

All of the results in the charts shown above indicate a phase response of zero; this is likely due to the linear model used to obtain the results as it does not have a time delay.

4

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

BW (rad/sec)

p (s

ec)

Pitch Attitdue Bandwidth Criteria Low Speed

Level 3 Level 2 Level 1

Susceptible if flight path BW insufficient

No PIOPIO tendency

X-PlaneFLIGHTLAB

Figure 5: Longitudinal axis bandwidth requirements MIL-1797A

Figure 5 illustrates the pitch attitude bandwidth criteria for level C flying phase. The bandwidth hypothesis is that the pilot can adequately follow input commands with frequencies up to the bandwidth frequencies. If he tries aggressively to follow higher frequency commands, this will lead him to instability. It is clear from the above figures that for pitch attitude, for lower speeds the bandwidth frequency is in level 2, while the higher speeds (not presented on the figure) have much higher bandwidth frequencies. This allows for high frequency inputs in these axes without concern for a lack response from the aircraft.

With reference to MIL 17917A, and natural frequencies obtained from above set of results, it is evident that for X-Plane model at higher speeds the 1/T 2is much higher than the natural frequency of the short period mode, unlike the FLIGHTLAB model, in which the 1/T 2

remains consistently lower than the open loop natural frequency. If 1/T 2is sufficiently higher than ωSPthe pitch and the attitude response may not be separated enough to give pilot the additional cues he needs in order to control the outer slower loop.

For Roll axis, the recommended values of bandwidth for roll axis according to the MIL 1797A standards depends on a piloting task associated with certain mission and mission phases. For roll axis phase crosses 180 degree.

Table 1: Required Bandwidth for CAT A and CAT C

There were some difficulties calculating the roll bandwidth, as the aircraft does not return to its steady state condition. The time and frequency response to step aileron input for 200kts speed is shown below for reference purpose.

10-1

100

101

102

-15

-10

-5

0

5

Ampl

itude

(dB)

Positive FFT Frequency Response

10-1

100

101

102

-3

-2

-1

0

1

Frequency (Hz)

Phas

e (d

eg)

Figure 6: Roll response to lateral input (frequency response – X-Plane)

The Dutch roll damping according to MIL-1797 specifications for X-Plane are given in Figure 7.

0 0.5 1 1.50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dutch Roll Freq dr (rad/sec)

Dut

ch R

oll D

ampi

ng

dr

Dutch Roll Requirement CAT-C Phases

Level 3

Level 2

Level 1

Worsethan Level3

X-Plane

Figure 7: Dutch Roll Damping Requirements

Perceived Handling Qualities

5

Based on the above observations, results and analysis the handling qualities ratings for the math model of the aircraft from both simulators in summarised below. The analysis is divided into three colours (red, yellow, green) representing level 3, 2 and 1. It must be notes that these ratings were used to predict the accuracy of Jetstream 31 flight model used during the testing phase, not to assess the aircraft handling qualities. Of particular interest is the X-Plane model (running on JETSIM). These results show, the satisfactory performance by the aircraft though minor improvements are required in lateral axis, which were found slightly sluggish. Since the selected MTE requires testing to be done in longitudinal axis only, these limitations were ignore, but may become a part of further research in assessing the handling qualities of the aircraft with improvements, such that the model is error free and meet Level 1 qualities before being tested. Such level of accuracy will allow, to pick up simulator limitations (in all axis) more accurately. The results from Precieved handling qualities are given in Appendix of this paper.

Piloted Assessments

The results from test flights flown by both pilots are shown for landing task in different simulator configurations (motion and no motion). Two landings were performed by each pilot (excluding the trail landings) in each simulator. The results are given for each pilot performance in both simulators.

The landings were performed under clear visual conditions, thus the approaches were performed without using any navigational aid. This allowed testing the visual cue quality in both simulators. The tests were done to check the aircraft dynamic response characteristics, and how the motion cue affects the pilot performance and workload. The performance standards desired and adequate are given in Table 2 and Figure 8

Performance standards Desired X Adequate X Maintaining ILS glide slope of ±X dots

1/2 1

Keep aircraft aligned with the runway centerline

Minimum Normal Landing Speed VREF +X knots 5 10Table 2: Landing Mission task element performance

standards

Figure 8: Landing Mission Task Element

Analysing the results shown above for JETSIM simulator, a less frequency but high frequency stick movements are apparent by the pilot. Considering the longitudinal stick inputs (shown in Figure 5.11) the trends clearly shows that pilot struggles to maintain the aircraft attitude near the flare point, this is confirmed by the time history for the pitch attitude (shown in Figure 5.13) at time greater than 85seconds. Furthermore, during the approach phase strong variations in the pitch attitude and rate are apparent. Pilot late commented that it was difficult for him to capture the glide slope, due to the seating

6

position in the simulator. Unlike HELIFLIGHT simulator, the JETSIM have control column operating by hydraulics system, and is twin seat simulator. Thus pilot position does seem to affect the result, seriously effecting pilot eye point view of the runway.

There was some serious simulator limitations came in front after performing the test flight. Some of the prominent issues picked up during the test are:

Poor visual drift (20-30°)

Delays between pilot input and aircraft

responses

Inaccuracies in aircraft instrument

hardware

Strong coupling between lateral and

longitudinal axis, requiring pilot to

apply corrections

Limitations in pitch attitude estimation

Although there may be many reasons contributing to above limitations, it was obvious from the results seen so far, that vertical acceleration is a key parameters in pilot’s performance. Although, the approach was performed visually, there were some discrepancies observed in altitude and airspeed indicator. Latter is particularly true for pilot station instruments, not correlating with the actual simulator instruments.

Hard landing was seen, and the task does not meet the requirements. Pilot gave the HQR of 7 for the task, commenting on the sluggish hardware controls, and large pilot eye point. Referring to the offline handling qualities, it seems that there seems to exist little or no deficiencies in the flight mode. Major deficiencies arise from the simulator hardware.

Based on the MIL 1797A specifications, for short period frequencies both models exhibit Level 2 ore better rating at higher speeds, but very sensitive at lower speeds. Both

FLIGHTLAB and X-Plane model were awarded Level 3 for short period frequency requirements.

Overall Assessment

The online and offline analysis were conducted primarily to address the simulator fidelity issue instead of assessing the handling qualities for the aircraft. However, during the offline analysis phase, flight model handling qualities were indeed assessed to quantify and understand the nature of the accuracy of the flight model that will later be used as a part of flight testing. Analysing the offline and online analysis, several deficiencies have been identified, which are discussed below.

Offline analysis shows that there are severe problems in X-Plane’s Jetstream flight model in lateral axis, and there is an evidence of strong coupling between the two axes, which seem to affect the pilot workload during the task performance. This coupling is in Level 3 for X-Plane and Level 2 for FLIGHTLAB model. The rating becomes better (excluding bandwidth) as the speed increases for both simulators, hence showing that coupling affects the flight at lower speeds than higher.

For the X-Plane model, bandwidth analysis shows that the model is in Level 2 overall, but FLIGHTLAB model approaches level 3 at higher speeds. However, although it is in level 1, but low bandwidth frequency has serious limitations on the aircraft performance.

Results: Motion Influence

The direct comparison of the frequency response behaviour provides a clear picture of model fidelity as a function of frequency. This is critical for validating piloted simulations since the requirements on the pilot cueing accuracy are also frequency dependant. The

7

separate display of magnitude and phase responses allows the sources of the simulator discrepancies to be more easily determined. For example an excessive time delay in simulation hardware causes a linear phase shift with a frequency. Scaling errors in a simulation model appear as a clear vertical shift (in dB) in the magnitude curve. These effects are all combined in the time domain and therefore are not easily discernible in a conventional time response comparison.

10-2

10-1

100

101

-20

-10

0

10

20

30

Frequency (Hz)

Erro

r (dB

)

Simulation DataProposed Level D Fidelity Criteria

10-2

10-1

100

101

-300

-200

-100

0

100

200

Frequency (Hz)

Erro

r (de

g)

Simulation DataProposed Level D Fidelity Criteria

Figure 9: Jetstream 31 math model error function and proposed fidelity criteria

Figure 9 re-plots the frequency data for longitudinal stick inputs at 200knots, in terms of magnitude and phase errors as a function of frequency. Here 0 dB magnitudes and 0 deg phase indicate perfect tracking of the JETSIM simulation with HELFILIGHT simulation results. Also shown in the figure are the mathematical model mismatch boundaries proposed herein for the highest fidelity training simulations (FAA Level D). These boundaries correspond to the LOES mismatch criteria from the fixed wing handling qualities criteria.

The Jetstream 31 simulation math model compiles with the proposed Level D fidelity criteria (more accurate however at lower frequencies). The result is consistent with the pilot comparison of a simulator and the flight behaviour of Jetstream 31 aircraft. However, a serious mismatch in phase is evident at higher frequencies (frequencies greater than 6.2rad/sec).

Furthermore, a direct comparison of stability and control derivatives identified from the simulator or flight test with values identified mathematical models can be used to derive correction factors for significantly improving the model fidelity. It has been seen and discussed previously that identification tools provide a systematic and accurate approach to determine these correction factors which are excessively and routinely used by the simulation community. Comprehensive simulation studies are often used to define the flight control system hardware requirements.

Pilots were asked to comment to following parameters:

Primary Aircraft Responses: this

includes response magnitude, control

sensitivity, control efficiency and

control position

Secondary Aircraft Responses:

response magnitude, and undesirable

secondary effects (such effects

involves the pilot’s ability to make

corrections)

Task Execution: to assess the

difficulty of task execution, stability

characteristics of the aircraft,

similarities and differences between

two simulator environments, and

configuration (motion and no motion)

The developed rating scale (PRS) goes from scale of 1 to 6, where 6 represent the highest similarity. The difficulty of execution (DOE) scale goes from 1 to 5, again maximum 5, represents the highest similarity, and 1 represents the largest differences in task execution.

Mathematical Model vs Piloted Assessment

8

Direct frequency response comparison of end-to-end performance of a complex simulation model with the conceptual design model (X-Plane aircraft) and its specifications constitutes an important dynamic check, which often exposes unexpected processing delays. These delays may arise from the numerical integration techniques used by the simulator, or errors in the digital implementation of the control laws. Furthermore, this technique is also useful in exposing degradation in aircraft control system performance or other hardware dynamics modelled in the advanced design simulation model that may have been ignored at model conceptual design stage.

10-2

10-1

100

101

102

-50

-45

-40

-35

-30

-25

-20

-15

-10

-5

0

Frequency [Hz]

Am

plitu

de (d

B)

Math ModelSimulation

10-2

10-1

100

101

102

-800

-700

-600

-500

-400

-300

-200

-100

0

100

Frequency [Hz]

Pha

se (d

eg)

Math ModelSimulation

Figure 10: Frequency response (Amplitude and Phase) for Simulator versus math model

Figure 6.1 compares the longitudinal axis pitch rate response to longitudinal stick input from fixed based JETSIM against the mathematical model. Both responses exhibit the properties of second order system with no zeros and two poles. In the 0.03-0.3 Hz (0.19-1.88rad/sec), the simulator visual drive response follows the math model, although the visual is less than

one to one, as seen by the vertical shift in magnitude plot. Comparing the phase response, at lower frequencies both math model and visual response remains the same (at zero degree phase). At higher frequencies, the visual drive is unable to follow rapid commands to the aircraft model, resulting in a larger phase lags (as seen by the downward shift in the phase curve).

Model Limitations and Conclusion

Although both model and the simulator proved to be adequate for the wide range of flight conditions, several issues were not covered during the frequency response analysis and identification analysis. Some of these issues are:

Flaps configuration: although JETSIM

allow pilot to select the flaps up to 30°

(required at landing), the

HELIFLIGHT does not give pilot this

luxury. Based on the mathematical

model, the flaps can only be selected

as OFF (0 deg) and ON (50 deg)

Lateral Trim effects: such as increase

in drag due to sideslip angle, are not

modelled

Instrumentation: there were some

limitations and discrepancies in

simulator instruments which was not

addressed in the report, increasing

pilot workload

Acknowledgement

This document is the summary overview of flight model validation process, thus omits some details and results, and is extracted from the Thesis written for MSc Simulation in Aerospace Engineering Course “Closed Loop

9

Assessment of Flight Simulator Fidelity”. The developed JETSIM is shown in Appendix.

Appendix

Perceived Handling Qualities Results

Bare Airframe - X-PlaneSpeed (knots)

Pitch Attitude Bandwidth, L1,2,3 130 148 160 180 200 230 250

X-Plane L2 L2 L2 L2 L1 L1 L1FLIGHTLAB L2 L2 L2 L2 L2 L3 L3

Bare Airframe - X-PlaneShort Period Frequency Requirements-N/Alfa

Speed (knots)

Level 1, 2 and 3 CAT-B 130 148 160 180 200 230 250X-Plane L3 L2/L3 L2/L3 L2/L3 L1 L1 L1

FLIGHTLAB L3 L3 L2 L2 L2/L1 L2/L1 L2/L1

Bare Airframe - X-PlaneShort Period Frequency Requirements-N/Alfa

Speed (knots)

Level 1, 2 and 3 CAT-C 130 148 160 180 200 230 250X-Plane L3 L2/L3 L2/L3 L1 L1 L1 L1

FLIGHTLAB L3 L3 L2 L2 L2 L2 L2

Roll Axis Control Power

Speed Limits X-PlaneFLIGHTLA

B

L L2L2

M L1L1

H L1L1

Bare Airframe - X-PlaneSpeed (knots)

10

Short Period Damping Ratio 130 148 160 180 200 230 250Angle of Attack Response

Levels 0.208 0.242 0.367 0.41 0.64 0.689 0.91CAT A L3 L3 L2/L1 L1 L1 L1 L1CAT B L2 L1 L1 L1 L1 L1 L1 CAT C L3 L3 L2 L2 L1 L1 L1

Bare Airframe - X-PlaneSpeed (knots)

Stick Forces per g Requirement 130 148 160 180 200 230 250

X-Plane L3 L3 L2 L1 L1 L1 L1FLIGHTLAB L3 L3 L2 L1 L1 L1 L1

JETSIM Simulator

11

12