Advanced Organiser

The following introductory slides provide:• A definition of Open Book in the SP context

• Possible benefits of using an open book exam format

• Criteria for using an open book format

• A algorithm to help you decide on whether or not to use

an open book format

Definition of the Open Book examination format

An open book exam is not a method of assessment, but aformat for assessment.It is similar to a closed book in thatstudents complete the test items:• within a defined time period• in a specified setting determined by the examiners• Under invigilation.The defining characteristic is that for all or some ofthe test items, students can:• access a range of resources determined by the examiners

in answering these questions

Benefits of the open book format

• The examination of performance mirrors more authentically the conditions under which professionals are accustomed to working in the real world, as compared to a closed book format

• As specified resources are made available, there is less emphasis on students’ needing to memorise large bodies of data. The focus can be more on the application of knowledge to real world problem solving.

Open Book Examinations offers a thorough discussion on potential benefits of using open book examinations and the type of questions to be used

When to use the open book exam formatThe open book format is particularly useful where assessment focuses on

application/higher application learning outcomes.

The following situations are most conducive for using open book:

• Authenticity of student performance is absolutely required. ( N.B. You want to be sure that the work produced is solely the students own work. This cannot be assured with take home ICA’s)

• The assessment activity necessitates the use of a range of resource materials in order for students to demonstrate competence. (For example, any real world task that involves accessing charts, tables, regulations,etc)

The next slide provides an organiser that may help you in

deciding the usefulness of the open book format for your module

Organiser for deciding on using open book exam format

Am I intending to assess student learning thatrelates to application/higher application learningOutcomes as specified in the Module Document ‘Table of Specifications’?

Do I want to assess student l,earning under Examination conditions to ensure authenticityOf performance in the time frames allocated?

Is student learning more validly assessed if they canaccess a range of resource material typically usedby professional in real world contexts

Design an appropriate test item(s) for an open book examination format

It is very unlikely than an open book format is appropriate

It may be better to use a Take home ICA for thisParticular assessment

Consider a range of test items using a closed book format

Advanced Organiser

The following slide set focuses on key

implementation issues:

• Preparing students and staff for an open book exam

• Deciding on resource access for students

• Types of test items appropriate for open book exams

• Administration of an open book exam

Key issues in implementing the Open Book examination format

• As open book items are likely to be longer and more complex, design considerations take on greater importance

• The use of resource materials must follow the guidelines laid down by SP

• Sufficient practice in tackling such items must be provided to students by way of formative assessment

• Sufficient time is allowed in the examination. This may involve more piloting of items to ensure time parameters are fair

Guidelines for selecting resource materials

• There is no numerical figure that can be placed on the number of pages that can be accessed

• Resources must be those typically used by professionals in doing these real world activities

• Resources must be sufficiently complete to facilitate task completion, but do not provide direct answers to any part of a question

• The selected resource range does not lead to the unfair advantage of some students as compared to others. (For example, students who have faster internet access or more money to access better resources)

• Clear guidance concerning what materials can be used, when they are provided and accessed, and under what conditions they are to be used

Preparing students for open book exams

• As for any type of assessment, students will need appropriate and sufficient practice – formative assessment - in order to be fully familiar with the expectations, demands and conditions of open book exams.

• Students must be clearly and consistently briefed that the resources available will not provide direct answers to the question. All relevant learning must have been achieved prior to the examination

Types of test items for open book exams• In principle any type of test item can be used in an open book

examination. However, as objective based test items (e.g., MCQ’s) and short answer items mainly involve knowledge recall and the use of specific types of thinking (e.g., analysis, inference and interpretation), these are more suitable for a closed book format.

• Extended essay and performance based items provide the most appropriate assessment opportunities for the open book format. Producing a design to specifications and meeting regulations; making inferences and interpretations from a range of cases; preparing a voyage plan - are some typical examples.

Administration of open book exams

• An open book exam must follow the same SP Examination Board procedures as for a traditional closed book exam, with the exception that specified resources can be accessed

• In cases where an exam contains both closed and open book items, the module coordinator is responsible to ensure that resource materials used for open book items are unavailable or of no aid to closed book items.

Advanced Organiser

The following 3 slides focus specifically on:

• Test item design – general considerations

• Steps in the design of valid and useful test items for an open book exam

Test item design

• It is important to remember that the open book format is not an assessment method – but a framework and set of conditions in which assessment is conducted.

• It is important, therefore, to follow the same design considerations as for a closed book format with the exception that the test item will incorporate access to specified resource material. The following slides identify the key steps involved in designing extended essay and performance based items

• However, as such items are likely to be larger and more complex, it is particularly important to ensure that:– Clear and concise notes of guidance are provided to ensure students fully

understand what is required for successful completion– The level of difficulty of the tasks does not exceed the standard set by the

module learning outcomes

Designing test items

Step 1: Identify clearly the knowledge, skills andprocess to be incorporated into the item

For this step it is important to:• Choose specific topic areas in your curriculum that contain knowledge

essential for key understanding of the subject. For example, central concepts, principles and models.

• Identify the types of thinking that are important for promoting student understanding and subsequent competence in these topic areas. For example, analysis, evaluation, etc.

• Identify specific vocational skills that are important for competence in the real work context. For example, design skills, statistical techniques, software application, etc

• Identify other process skills that are important for promoting learning in the identified areas. For example, skill in searching for and summarising information, etc.

Designing test items

Step 2: Produce the assessment itemIt is important that the item:

• Clearly involves the application of the knowledge, skills and processes identified from Step 1, e,.g., relates to identified learning outcomes in the module

• Reflects as far as is possible real world problem solving

• Clear notes of guidance are provided, which:– Identify the products of the task and what formats of presentation are

acceptable (e.g. short report, drawing, model, essay, etc)

– Specify the parameters of the activity (e.g. length, areas to incorporate, resources provided, etc.

Designing Authentic Assessment offers extended guidance Open Book Examinations may again be a useful referenc epoint

Advanced Organiser

The following are some examples of items used by

SP lecturers in a open book exam:

1. Voyage Planning

2. News Release

3. Network Design

Example 1: Voyage Planning In Module No.3 Passage Planning, ‘Advanced Diploma of Applied Science (Shipmaster)’ Learning outcome 3, specifies the following competency: • A passage plan is compiled for a short international journey In order to meet this competency students would need to demonstrate essential underpinning maritime knowledge as well as a range of process skills, for example: • Accessing relevant sources of information, e.g. charts/publications/regulations, etc• Analysing, making inferences from and evaluating sources of information• Written communication The following test item, using an open book assessment format is designed to assess this area of competence

Task scenario You are the Master of a Panamax type bulk carrier, which is presently docked in Singapore. Your owners have instructed

you to proceed to Yokohama in Japan. They need the ship to be there by 12 th Sept.

 1. Prepare what you consider to be the most suitable voyage plan. 2. Justify your selection of route, identifying the important decision making factors that you have taken into account

in producing this plan. You are provided with the following essential factual information and reference sources that need to be accessed inanswering this question. Carrier Details Publications / manuals providedForward draft 9 metres Guide to Port EntryAft draft 9.8 metres Pilot TablesFuel oil consumption 25 tons Tide TablesAir draft 15 metres Ocean Passages of the WorldManoeuvring characteristics - Mariners’ HandbookProvided as attachment Ocean Currents of the World

Load Line ZonesList of LightsList of Radio stations

 Produce a short report of approximately 800 words documenting the following:          the options considered,         the chosen route,         all reasons for the choice made. You may include diagrams/charts and any other supplementary material that illustrates the above. The report must followaccepted conventions in terms of writing style.

Example2: News Release

In the Language and Communication module, ‘Media Writing’, the following learning objectives are specified: • Identification of the ‘news angle’ in presenting a story• Selection and organisation of the relevant facts in support of a news angle• Use of language according to the journalistic writing style for the right emphasis In meeting these objectives, students would need to demonstrate both essential underpinning knowledge of constructing and presenting news items, as well as good thinking in the following areas: • Making inferences and interpretations from available data• Analysis of text• Creativity in presenting information The following test item, using an open book assessment format, is designed to assess these learning outcomes

Task You are a Public Relations Consultant with Royal Diamond Co. Ltd., which has recently launched a web site to sell diamonds online. The web site at www.RoyalDiamond.com. has received 2,000 hits and sold diamonds to more than 100 customers within its first week of being set up. So far Royal Diamond Co. Ltd. is the only company selling diamonds online in Singapore. You have been told to prepare a news release on the successful launch. You have interviewed Mr. David Ong, the Manager of the Company. Based on the transcript of the interview below, write the news release (12-15 paragraphs) that will be distributed to the press as part of a press kit. Give your story a headline and explain the news angle you have chosen.

Example 3: Network Design

In module ‘Building Services Systems, the following learning objective is specified: • Design lighting protection system for buildings in accordance with the guidelines stated in CP 33 In meeting this objective, students would need to demonstrate essential underpinning knowledge of lighting protection systems as well as skill in certain specific types of thinking, for example, analysis, comparison and contrast and evaluation. The following test item, using an open book assessment format, is designed to assess these learning outcomes:

Task Design and draw a complete network of lighting protection system for the high rise building shown in Figure 2 (not included here), using the rolling sphere criterion with sphere radius of 45m. Your design shall include the air-terminations, down conductors and earth terminations. For the purpose of this design, use horizontal air termination only.