Examination system for the aviation area - Trafik-, Bygge- …/media/Dokumenter/04 … · · 2017-01-23Examination system for the aviation area ... 11 Technical requirements Requirements

Technical requirements

Examination system for the aviation area

January 2017

3 Technical requirements Contents

Contents

1 Introduction 5

1.1 Contents and scope of task 5

1.2 System requirements 5

2 Standard system 8

3 Requirements for CQB, etc. 9

3.1 Question databases 9

3.2 Questions 9

3.3 Selection masks 9

4 Requirements for the exam 11

5 Requirements concerning security 13

6 Requirements concerning logging and statistics 15

7 Requirements concerning documentation 17

8 Miscellaneous non-functional requirements 19

9 Technical requirements 21

10 Commissioning plan 23

11 Requirements for suppliers 25

Annex 1 27

Glossary 27

Annex 2 28 Data format for EASA CQB 28

5 Technical requirements IntroductionData format for EASA CQB

1 Introduction

1.1 Contents and scope of task

Danish Transport, Construction and Housing Authority wishes to procure and

implement a browser based standard IT system for planning and holding theoretical aviation exams in compliance with EU Regulation 216/2008 and its provisions for implementation for pilot educations.

As regards exams for commercial pilots, the exams for the various modules will be based on questions from EASA’s CQB. For exams for private pilots, TBST must be able to establish and maintain a CQB.

The system must support the organization of exams by aviation schools, and that

these may be carried out online via the examinees own IT equipment (BYOD: laptops, Macs or tablets) and use as few human resources as possible for exam surveillance.

Safety/security facilities must minimize the risk that examinees during the exam, can use their exam equipment for locating answers to the exam questions via irregular means, and that the exam questions are leaked from the CQB’s and are made available via the internet after the exams.

The system must as a minimum be applicable for holding exams for the above examination categories in the aviation area but also other types of exams within other

professional areas, such as e.g. driver training programmes and drone certificates, if possible.

The Technical requirements are based on an illustration of the task, Figure 1. The figure shows an overview and flow of the elements included in the task. In relation to the exam system, the following types of users are identified:

Examinee Aviation school Exam supervisor Exam administrator (Danish Transport, Construction and Housing Authority

employee handling fees and complaints, amongst others) CQB administrator (Danish Transport, Construction and Housing Authority

employee administrating question banks) System administrator (Danish Transport, Construction and Housing Authority

employee administrating technical matters)

A glossary of terminology and abbreviations used is shown in Annex 1.

1.2 System requirements

Clauses 2-11 describe the requirements made by the Danish Transport, Construction

and Housing Authority for the exam system. Mandatory requirements are marked with (mr), while other requirements are marked with (r).

A mandatory requirement (mr) is a requirement relating to the content of the bid,

which must be met/fulfilled, as deviations from the mandatory requirements are not accepted. The tenderer must comply with the mandatory requirements without any reservations, and bids that do not meet all mandatory requirements will be rejected as not conforming.

Other requirements (r) are an expression of the requirement of the Danish Transport, Construction and Housing Authority with regard to the delivery, and are part of the


competitive parameters which the tenderers can compete upon. They are included in the evaluation of the bid as described in the tender conditions, see clause 9.

Figur 1. Illustration of task


8 Technical requirements Standard systemData format for EASA CQB

2 Standard system

1. The Danish Transport, Construction and Housing Authority requires a browser based standard IT system (r).

2. The supplier must describe how the system complies with the task, see Figure 1. (r)

When evaluating the bid, the assessment of compliance with the requirements in this clause weighs 15%.

9 Technical requirements Requirements for CQB, etc.Data format for EASA CQB

3 Requirements for CQB, etc.

3.1 Question databases

Annex 3 shows the data format CQBs.

Composition/structure and maintenance. 3. The system must have capacity to contain several CQBs each consisting of up

to 40,000 exam questions. (mr) 4. The system must have capacity to handle two versions of each CQB as a

minimum. (mr)

5. The Danish Transport, Construction and Housing Authority must be able to import, create, change, export and delete a CQB. (mr)

6. Only the CQB administrator must be able to see, create, change and delete the contents of a CQB. (mr)

3.2 Questions

Composition/structure 7. Each exam question in a CQB must consist of.

a. a minimum of four answers (according to the ”multiple choice” principle) each containing a text with a suggested answer to the exam question. (mr)

b. Potential enclosed Annexes (one or more) containing information

which is necessary in order to answer the exam question. (mr) c. Annexes which must be made up of text, visuals, video and audio

clips in standard formats and resolution adaptable for PCs and portable units. (mr)

8. The same annex must be connectable to more than one exam question. (r) 9. Languages: Exam questions must be formulated in two languages (Danish and

English) (mr)

Meta data 10. Blockings: The Danish Transport, Construction and Housing Authority must be

able to mark an exam question which by the Danish Transport, Construction and Housing Authority has been found unsuitable for certain modules – possibly all modules – so that it cannot be selected for exams in these modules. This marking (block) must be completely separate from the EASA marking of the questions use for various modules. (mr)

11. Exclusion list: For each question it must be possible to sustain an exclusion list, which denotes which other questions that are not to be selected if the first

question is selected. The list will, amongst others, be used to prevent that the answer to one question will appear from another question. (mr)

Maintaining exam questions 12. It must be possible for the CQB administrator to add, change or delete

questions in a CQB including meta data. (mr)

3.3 Selection masks

Composition/structure

13. Exams in the various modules must be definable via selection masks. A selection mask, defines how many exam questions are to be selected from the various subject groups and sub groups. (mr)

14. It must be possible to set up individual selection masks for each module exam. (mr)

10 Technical requirements Requirements for CQB, etc.Data format for EASA CQB

Maintaining masks 15. A selection mask for a module must be editable (set-up, changes, and

deletion) by the CQB administrator only. (mr) 16. It must be possible to import and export selection masks. (r)

When evaluating the bid, the assessment of compliance with the requirements in this clause weighs 5 %.

11 Technical requirements Requirements for the examData format for EASA CQB

4 Requirements for the exam

17. The supplier must describe how the exam is held online in the system provided, including how the solution provided supports that aviation schools can carry out exams and that exams can be taken via the examinees’ own devices (BYOD), see Figure 1. (r)

18. It must be possible for the Danish Transport, Construction and Housing Authority (and no one else) to print out an exam set along with annexes on paperin case of prolonged loss of internet connection to the exam place) (r)

19. When generating an exam for a class, the questions must be presented in differing orders for the examinees. (mr)

20. The same question is not to be selected more than once per examinee in the individual module exam. (mr)

21. The Danish Transport, Construction and Housing Authority shall recommend

each student to the individual modules in a sitting. At the same time a unique password is generated for the examinee. (mr)

22. At random times, before the exam has been initiated, it must be possible to change or cancel exam settings. (mr)

23. It must be registered, which exam supervisor handled which module of an

exam. (mr) 24. The examinee must be able to go back and forth between questions in the

exam set. (mr) 25. It must be possible for the exam supervisor to submit comments regarding the

execution of the module exam – for instance whether there were technical difficulties or students who failed to appear. (r)

26. For each exam question, there must be a field for marking whether the

examinee wishes to complain about the question after the exam. (mr) 27. For a configurable period after the exam, the examinee must be able to look

through (but not correct) his or her examination paper, with a view towards being able to complain about questions. (mr)

28. The individual examinees exam module with questions and answers, must be uniquely identifiable, for instance via the aid of an ID number. (mr)

29. The individual examinee’s module exam including questions and answers must

be stored until it has been deleted by the exam administration, so that it afterwards is only possible for the exam administrator (and no one else) to see the module exam, for instance in connection with the handling of complaints.


13 Technical requirements Requirements concerning securityData format for EASA CQB

5 Requirements concerning security

30. The supplier must describe how the solution minimises the risk that examinees during the exam are using the devices (BYOD) to cheat. (r)

31. The system must offer a logon facility with the use of a username and password. (mr)

32. The system must offer the possibility of awarding differentiated rights that support different user types, see the glossary. (mr)

33. The system must be able to use role based rights management and only the Danish Transport, Construction and Housing Authority ’s employees must be able to assign rights. (mr)

34. It must not be possible after finishing the exam to retrieve exam questions from log files and similar system on the exam equipment. (mr)


15 Technical requirements Requirements concerning logging and statisticsData format for EASA CQB

6 Requirements concerning logging

and statistics

35. The system must log access to and changes made to exam questions and exam results. (mr)

36. It must be possible to retrieve and print statistics for e.g. correct answers for the individual questions, percentage of correct answers per school, school placement, etc. (mr)

17 Technical requirements Requirements concerning documentationData format for EASA CQB

7 Requirements concerning

documentation

37. The system must be delivered with the following documentation in Danish or English:

38. Installation procedure (mr) 39. Operational documentation (mr)

40. User manuals customised to individual user types, see the glossary (r) 41. Screen reliant online help for users (r) 42. System documentation (mr)

When evaluating the bid, the assessment of compliance with the requirements in clauses 7, 8, 9 and 11 weighs 5% in total.

19 Technical requirements Miscellaneous non-functional requirementsData format for EASA CQB

8 Miscellaneous non-functional

requirements

43. Capacity – the system must be able to handle at least 100 simultaneous examinees at any given time distributed across the various CQBs. (r)

44. The system must execute suitable error handling, including giving appropriate and telling error messages. (mr)

45. Input from users and systems must be validated and an appropriate error message must be given, if requirements to syntax and content are not met. (mr)

46. User interfaces for the individual user types must be formed simply and clearly and under the terms of best practice. (r)

47. Response times must be comparable to response times in comparable

dialogue-oriented web systems. (mr)

48. The examinee must be able to choose between Danish or English user interfaces (the IT system itself – not the questions). (r)


21 Technical requirements Technical requirementsData format for EASA CQB

9 Technical requirements

49. The system must be parameter operated in order that the Danish Transport, Construction and Housing Authority may configure changes, including duration of module exams, requirements to passing exams, labels and creation of new modules. (r)

50. The system must have an API which allows the system to easily communicate with other IT systems. (mr)

51. Communication must take place in an XML format and must support all communications following completion of the task, see Figure 1. (mr)

52. The following data must be exportable from the system through the API: a. The module result per student (percent of correct answers) (mr) b. Changes of exam results due to complaints. (r)

c. Any other data concerning the exams. (r)

d. Raw data for the use of e.g. data warehouse. (r) 53. The system must be operable via a browser. (mr) 54. The browser must be operable via PC and tablet on the following platforms:

iOS, Windows and Android. (mr)

If the system is to be supported by the Danish Transport, Construction and Housing Authority, the following requirements must be met:

55. The system must be handled on an MS Windows Server 2012 R2 with connected SQL Server (Microsoft product). (mr)


23 Technical requirements Commissioning planData format for EASA CQB

10 Commissioning plan

The tenderer must as part of the bid prepare a commissioning plan. The plan must describe in detail the following:

How the transition from the present supplier to a new supplier will be

planned and carried out, including handling ongoing sittings.

How many resources the Danish Transport, Construction and Housing Authority’s is expected to supply during the implementation period.

The supplier’s proposed test plan. The supplier must describe the overall

test plan and method, including providing a description of the test environment. A more detailed test plan must be prepared during the

implementation project, containing various test scenarios, script for completion of the test, test cases and descriptions of test data and other elements included in the test. The test must contain a test of the exam system integration into the Danish Transport, Construction and Housing Authority’s system for administration of pilot licences via the API.

Schedule with important milestones until commissioning of the system

as at 01.04. 2017

The supplier must use IT tools for completion, management and documentation of trials and tests. Test tools must be accessible for the Danish Transport, Construction and Housing Authority’s employees.

The plan must, amongst others, ensure that the Danish Transport, Construction and Housing Authority after the transition can conduct exams under this contract and thereby without delivery subject to the current contract.

The time schedule must, as a minimum, include the following key milestones:

Tabel 1. Overview of milestones

Date Short description

20.03.2017 Commissioning plan approved

01.04.2017 Taking over test approved

01.05.2017** Commissioning of exam system (PPL)

01.05.2017** Commissioning of exam system (other exam categories)

*)

Test on completion

*) Period agreed after conclusion of the contract.

**) Subject to changes

56. The tenderer must prepare a commissioning plan that complies with the above. (r)


25 Technical requirements Requirements for suppliersData format for EASA CQB

11 Requirements for suppliers

57. The supplier shall introduce changes to the system so that the system complies with new EASA requirement within three months. (mr)

58. The Danish Transport, Construction and Housing Authority must have ownership of its own versions of the CBQs with questions, including exclusion

list, blockings and annexes. (mr) 59. The Danish Transport, Construction and Housing Authority must have

ownership of the finished exams and the results. (mr) 60. The supplier must within normal office hours make available a support

function. (r) 61. Debugging must be commenced within 4 hours measured within normal office

hours. (r)


27 Technical requirements Data format for EASA CQB

Annex 1

Glossary

API Application programming interface. A set of procedures in the system, which an external program can call, in order to exchange data.

Browser based IT system

An IT system that is accessible online via an Internet browser. It means that it is not software or an application that will be installed.

BYOD ”Bring your own device”. Use of personal IT equipment (laptops, Macs and tablets).

CQB Central Question Bank = question data bank with exam question, including meta data.

EASA European Aviation Safety Agency

Exam category: This means theoretical exams in the following exam categories, see EU Regulation 216/2008 and its provisions for implementation for pilot educations:

PPL/SPL Private/Sailplanes Pilot License A/H/S/B/Core subjects

ATPL Airline Transport Pilot License A/H/HIR

CPL Commercial Pilot License A/H

IR Instrument Rating A/H

EIR Enroute Instrument Rating

CBIR Competent Based Instrument Rating

For each exam category, a series of modules are attached with independent exams. For instance, with PPL 8 modules are mandatory, and with ATPL 12 modules are mandatory.

Class: The examinees who are gathered to take part in an exam.

The supplier The supplier of the exam system.

NemID The national Danish log-in solution for online banking and government websites. (see https://www.nemid.nu/dk-da/).

Sitting The period in which a series of modules, related to an exam category must be passed (at present 10 days).

Standard IT system An IT system which has already been sold to several independent clients, and which will be possible to implement at the Danish Transport, Construction and Housing Authority with no or few adaptations apart from establishment of interfaces and integration into other systems.

TBST Trafik- og Byggestyrelsen (The Danish Transport, Construction and Housing Authority )

https://www.easa.europa.eu/

https://www.nemid.nu/dk-da/


Annex 2

Data format for EASA CQB

Extracts from mail from EASA (June 2016)

I have considered not only the current multiple choice question type, but also fill-in-the-blank type questions, which we hope to also introduce into the ECQB in the coming 1-2 years. This was one of the items discussed at the ECQB Workshop last November in Cologne. It would make sense also to add information about the QTI

format, which offers so many more advantages compared to the MDB format that has been used so far. One of these advantages is that it provides a reference

between the question ID and the annex file. This is not contained within the MDB format. Finally, the tool will need to manage not only the current syllabus (based on the recently published AMC/GM to Part-FCL) but also the new, revised syllabus that we expect to have in place in the next 2 years (following completion of rulemaking task RMT.0595).

The syllabus (blueprint) Please bear in mind that this will need to be updated, with a fully revised syllabus expected to come out at the end of this year, with the completion of rulemaking task RMT.0595. It is also probable that the 5th level of LO will also be numbered,

meaning that the syllabus will go down to a 5th level of detail (it currently goes only to 4 levels). In the long term, the syllabus/blueprint should be editable, to enable further changes to the syllabus to be reflected in your question bank in the future.

Question files

MDB format

This is still used for ECQB releases, in the standard output and the extended output. The table below lists the fields in the extended output – the only difference to the standard output is the three additional fields listed at the end of the table. Please note that this format does not contain a reference between a question and annex.

It is important to note that only multiple choice format questions are currently contained in the ECQB. However, regulatory changes are being proposed that will allow the ECQB to contain other question types, hopefully in ca. 2 years. Please therefore do ensure that your provider is able to manage additional question types

(fill-in-the-blank as a certain new type, and others to be discussed at ECQB workshops).

Description of the contents of the MDB files for ECQB releases – extended output

Column Name Data Type Comment / Description

compteur AutoNumber

Auto Increment Integer with no physical meaning. Not a primary key but unique in the table nonetheless.




number Number The External ID of the question. This is unique across all domains.

subject/section Number Subject / Section of the domain

topic Number B = Topic

paragraph Number C = Paragraph

subpara Number D = Sub Paragraph

provisional Number E = Provisional (not currently used)

reference Number Not used

texte de la question Memo The text of the question.

réponseA Memo The first answer to the multiple choice question.

This is always the correct answer.

réponseB Memo The second answer to the multiple choice question.

réponseC Memo The third answer to the multiple choice question.

réponseD Memo The fourth answer to the multiple choice question.

variability Number Not used

knowledge Text(100) Not used

check mark Number Not used

significance Text(20) Not used

prov n° Number Not used

code Number Not used

date Date/Time The creation date of the question.

subj chairman Text(30) Not used

ia n° Number Not used

ez n° Number Not used

ass doc Number Not used

kind ass doc Number Not used




ref_annex Number Not used

suf_annex_a Text(50) Not used

suf_annex_b Text(50) Not used

suf_annex_c Text(50) Not used

suf_annex_d Text(50) Not used

suf_annex_e Text(50) Not used

kind_ass_doc_a Number Not used

kind_ass_doc_b Number Not used

kind_ass_doc_c Number Not used

kind_ass_doc_d Number Not used

kind_ass_doc_e Number Not used

validation Yes/No Not used

type Number Not used

catchword Text(50) Not used

application_ATPLA Yes/No -1 = Yes, 0 = No

length-atpla Number This is the length in minutes allowed in order to answer the question.

score-atpla Number This is the score for a correct answer.

application_ATPLH Yes/No -1 = Yes, 0 = No

length-atplh Number This is the length in minutes allowed in order to answer the question.

score-atplh Number This is the score for a correct answer.

application_ATPLHIR

Yes/No -1 = Yes, 0 = No

length-atplhir Number This is the length in minutes allowed in order to answer the question.

score-atplhir Number This is the score for a correct answer.

application_CPLA Yes/No -1 = Yes, 0 = No

length-cpla Number This is the length in minutes allowed in order to




answer the question.

score-cpla Number This is the score for a correct answer.

application_CPLH Yes/No -1 = Yes, 0 = No

length-cplh Number This is the length in minutes allowed in order to answer the question.

score-cplh Number This is the score for a correct answer.

application_IRA Yes/No -1 = Yes, 0 = No

length-ira Number This is the length in minutes allowed in order to answer the question.

score-ira Number This is the score for a correct answer.

application_IRH Yes/No -1 = Yes, 0 = No

length-irh Number This is the length in minutes allowed in order to answer the question.

score-irh Number This is the score for a correct answer.

application_PPLA Yes/No Not used

length-ppla Number Not used

score-ppla Number Not used

application_PPLH Yes/No Not used

length-pplh Number Not used

score-pplh Number Not used

application_EIR Yes/No -1 = Yes, 0 = No

length-eir Number This is the length in minutes allowed in order to answer the question.

score-eir Number This is the score for a correct answer.

application_SEE Unknown Not used

application_GPL Unknown Not used

application_VLAP Unknown Not used

application_CPL_PARA

Unknown Not used




application_PPL_PARA

Unknown Not used

application_P_PHOTO

Unknown Not used

flag_epac Text(50) Not used

lot Text(50) Not used

remarks Text(50) Not used

wording_national Memo Not used

answerA_national Memo Not used

answerB_national Memo Not used

answerC_national Memo Not used

answerD_national Memo Not used

flag_delete Yes/No Not used

work_case Yes/No Not used

work_num Number Not used

Justification Memo An explanation of why the correct answer is indeed correct, and the other answers are incorrect

QuestionArea Text Unique ID related to the 5th level of the syllabus

References Memo A reference to support the question, e.g. ICAO Annex standard, widely used text book


QTI format The QTI format is also being currently tested by the ECQB Team and those NAAs that expressed an interest. As it is being tested, it’s not possible to provide clear details at present. However, the following should help you:

The output is QTI 1.2 compliant

Each subject is expected to be managed as a separate folder/archive

Output includes a reference to any annex that is required to answer a question

Output also should include annexes that are used by the questions contained in

the release

At present only the multiple choice type has been tested for QTI output

The following metadata are part of the ECQB questions in the QTI output

(please note that this detail is not finalised):

NAME CONTENT

ATPL(A) No / Yes

ATPL()H/IR No / Yes

ATPL(H)/VFR No / Yes

CPL(A) No / Yes

CPL(H) No / Yes

IR(A) and (H) No / Yes

CB-IR(A) No / Yes

EIR No / Yes

Creation Date Calendar date DD/MM/YYYY but sometimes with a further 2 digits

Justification Block text, in 1 or more paragraphs

Property Rights JAA / Agency

Question Area Single line text

Question score 1 / 2 / 3 / 4 / 5

Time limit 1 / 2 / 3 / 4 / 5

Source Not currently possible to export this in the QTI package

Annexes At present these are released simply as a folder of images. See above where the QTI output could be used to provide a package of questions plus related annexes together (though I stress again that we’re still testing this).

Trafik-, Bygge- og

Boligstyrelsen

Edvard Thomsens Vej 14

DK-2300 København S

[email protected]

www.trafikstyrelsen.dk

Technical Requirements

ISBN Write ISBN here

Documents

Examination system for the aviation area - Trafik-, Bygge- …/media/Dokumenter/04 … · · 2017-01-23Examination system for the aviation area ... 11 Technical requirements Requirements