1

QUASAR New star in ATSEP assessment

Author Petr Dvořák

Version 1.8

Date February 13, 2020

Slides 23

2

slide 2

QUASAR

3

slide 3

…in ATSEP assessment

New star…

4

slide 4

Questionnaire

for

ATSEP

Standard

Assessment

Routines

5

slide 5

Context

6

slide 6

QUASAR Initiative

Inter-FAB cooperation

on ATSEP assessment harmonisation

Internationally recognised group

of experts on ATSEP Training

Ambitions to become European quasi-standard

Outstanding example

of both intra- and inter- FAB cooperation

Formal status coming from FAB CE HR SubC activities

and FABEC Training Task Force

What?

7

slide 7

Harmonisation of ATSEP training

Common training requirements / objectives are the same ― Eurocontrol ATSEP CCC Initial Training [SPEC-132]

― EU Regulation 2017/373 + EASA AMC/GM (under review of RMT.0719)

Training process may be different ― Training materials, duration, techniques, …

― Different local environment, …

― Different organisational structure, culture, etc.

Training output shall be the same ― Competent ATSEP on comparable level

Assessment shall be done ― “establish, implement and document”

Why?

8

slide 8

Harmonisation of ATSEP training

Common training material ― Not possible for various reasons

Common assessment ― Common final test (question database)

― Common standardised rules (Terms of Use)

― Tool for independent evaluation of training

QUASAR Examination

European-wide scope

How?

9

slide 9

Common database of final test questions for initial training ― Direct mapping to the defined training objectives

― Each objective covered by multiple questions

― Question complexity relates to taxonomy level

― Simple, understandable, unambiguous and consistent language (EN)

― Using multiple choice questions (single or multiple correct answers)

Questions cover all parts of ATSEP Initial Training

Outputs

Training part Total number of

questions

Number of training

objectives

Basic Training 621 198

Qualification Training – Shared 166 71

Qualification Training – COM 223 66

Qualification Training – NAV 312 149 (+34 MLS)

Qualification Training – SUR 255 100

Qualification Training – DPR 227 71

Qualification Training – SMC 61 40

TOTAL 1865 695

10

slide 10

Terms of Use ― How to generate and evaluate tests

― Defined standard pass mark (75 %)

― Rules for organising assessments

― Rules on quality monitoring (feedback) and maintenance

― Rules for handling of QUASAR material (confidentiality, copyright)

Outputs

11

slide 11

Complex development, review and approval process ― Collection of currently used material from participating organisations

― Review of questions and creation of new ones to cover all training

objectives

― Series of reviews incl. pilot runs, validation and feedback

― Release (common approval in QTF)

― Expert review (experts in participating organisations)

― Final approval (implementation of gathered comments)

― Working environment with interactive Access database

created by Austro Control and hosted by LPS SR

― Around 100 experts participated in the process

― Regular 2-3 day meetings every 2 months

― Offline work (”Homeworks”), distribution of tasks, use of SMEs

― Gathered a total of 2677 questions / 1865 approved

― About 30 % of gathered questions were dropped out

(not meeting high quality requirements)

Process

12

slide 12 01/2013 01/2014 01/2015 01/2016 01/2017 01/2018 01/2019 01/2020

QUASAR Task Force / Timeline

FAB CE (2013)

― ANS CR

― Austro Control

― BHANSA

― Croatia Control

― Hungarocontrol

― LPS SR

― Slovenia Control

FABEC (2015)

― DFS

― Skyguide

― ENAC/DSNA

― mandate from other FABEC members

(ANA Luxembourg, Belgocontrol, LVNL, MUAC)

NATS (2015)

AVINOR (2017)

12 core member organisations (not all participating all the time)

Who?

13

slide 13

QUASAR Task Force (QTF) *highlighted currently

active members

14

slide 14

Statistics

15

slide 15

Next Steps Continuous use of QUASAR examinations

Incorporation of changes in EU regulation 2017/373 and

EASA AMC/GM based on the work of EASA RMT.0719 ― Introduction of Information System Security subject to Shared

― Introduction of new topics (e.g. SWIM, Virtualisation)

― Move of Functional Safety to Shared

Maintenance of QUASAR material ― Keeping the database up to date

― Incorporation of feedback

― Updates of Terms of Use

Gathering feedback and statistics

Providing QUASAR examination to 3rd parties

16

slide 16

Use and Feedback Several validation runs performed (since 2016)

QUASAR Examination already in use (since 2017)

― as main means of assessment or for further validation

― Skyguide

― ENAC

― DFS

More than 240 participants in Basic Examination ― Average score of 83.7 %

― Average reached minimum 70.5 %, maximum 94.2 %

― Similar numbers for Shared Examination (more than 80 participants)

Very good feedback on quality and usability

― NATS

― Austro Control

― ANS CR

17

slide 17

Available to 3rd parties through QTF members ― Common administrative fee of 95 EUR per student

(applicable only if not combined with training provision)

― Electronic version or paper-form test

― On-site or remote provision

― Material remains confidential under supervision of QTF

― Non-disclosure agreement mandatory

QUASAR Examination

18

slide 18

Implementation of QUASAR examination in CANI LMS ― Full implementation of all parts according to ToU

― Allows detailed analysis of student’s performance

― Detailed test protocols for documentation and archival

― Provides group’s results for different parts / subjects

― Evaluation of training (training gap analysis)

― Available online

Local Implementation

19

slide 19

High quality standard examination

for ATSEP Initial Training

Transparency through clearly defined rules

and direct mapping of each question to defined objectives

Allows compliance with EU regulation 2017/373 ― ATSEP.OR.300/305 Competence Assessment:

A service provider employing ATSEP shall:

(a) establish, implement and document processes for:

(1) assessing the initial and ongoing competence of ATSEP;

…

(b) define the following criteria against which initial and ongoing competence

shall be assessed: …

(3) knowledge.

Available also to 3rd parties through QTF members

Summary

20

slide 20

Contacts

Petr Dvořák dvorakp@ans.cz

Robert Horak robert.horak@austrocontrol.at

Thorsten Brikey thorsten.brikey@dfs.de

Chris Cox chris.cox@nats.co.uk

… or any other QTF member

More info: quasar.cani.cz

21

slide 21

Working interactive demo available

at ANS CR / CANI booth #885

Unique possibility to check your ATSEP knowledge

and have hands-on experience with QUASAR examination

Demo

22

slide 22

Demo

23

slide 23

Invitation to participate in celebrating the success story

with core QUASAR members at CANI booth #885

today at 18:15

Invitation

“The power radiated by quasars is enormous.”

24

Thank you for your attention!

See you at CANI booth #885