opus.lib.uts.edu.au read… · Web viewWith the increasing accessibility of technology, the need to memorise and describe theories and concepts diminishes. Emphasis should therefore

STRUCTURED ABSTRACT

Student Perceptions of Open-Book and Closed-Book Exams in Postgraduate Engineering Management SubjectsHiyam Al-Kilidar, Alan Sixsmith, Rene Leveaux, Grant Mooney.Faculty of Engineering and IT, University of Technology Sydney

Corresponding Author Email: [email protected]

CONFERENCE SUB-THEME (select only one most appropriate theme and delete the rest)3. Beyond the classroom – tailoring engineering to meet the needs of all stakeholders. (External Industry requirement)

CONTEXT Our postgraduate engineering management programs cater for cohorts from diverse cultures and engineering backgrounds. The curriculum is designed to equip graduates with (1) hard skills - referred to as technical skills, consisting of a body of knowledge appropriate to the field of practice, and (2) soft skills - consisting of interpersonal-driven skills such as leadership, communication, negotiation, teamwork, and relationship building.

Formal exams (Closed Book Exams (CBE) and Open Book Exams (OBE)) have long been used in various disciplines as a means of assessing student learning. CBE often focus on recall of content and formulae, while OBE require questions to be written in a manner which requires interpretation of content, and as such, reflects the true learning outcomes of the student cohort.

With the increasing accessibility of technology, the need to memorise and describe theories and concepts diminishes. Emphasis should therefore be focused on the application of such knowledge into various situations, including the use of skills such as critical thinking, problem-solving, and ability to organise, retrieve and handle information. This requires students to provide responses to scenario-based questions or problems through the assimilation and application of learnt knowledge and information from various sources as they would in the workplace.

PURPOSE The purpose of this paper is to address perceptions of students from diverse cultural and discipline backgrounds, of OBE and CBE.

APPROACH A pre-exam and post-exam survey will be conducted to gauge students’ perceptions of before and after sitting an OBE.

RESULTS Results will examine student perceptions of OBE in regards to: (1) cognitive matters such as levels of exam anxiety and needed effort and (2) the application of problem solving, analytical thinking and scenario based reasoning.

CONCLUSIONSThis paper will draw conclusions of student perceptions of technical and cognitive matters relating to sitting OBEs.

KEYWORDSExamination, active learning, Open Book exams, Closed Book exam

AAEE2018 CONFERENCE Hamilton, New Zealand

INTRODUCTIONIncreasingly, postgraduate engineering management programs are catering for student cohorts of diverse cultural and discipline-based engineering backgrounds. The goal of such programs is to imbue graduates with the knowledge and skills required to act in higher positions within engineering or technology related management positions. While there is ongoing discussion about priorities and approaches, ultimately, most agree that such skills divide into two broad categories:

Hard skills – the measureable and quantifiable technical skills, process expertise and body of knowledge appropriate to the field of practice;

Soft skills – less tangible interpersonal proficiencies around leadership, communication, negotiation, teamwork and relationship building.

Attempting to satisfactorily address the collective needs of various stakeholders of the learning process (including students, accreditation bodies, organisation, etc.) can present challenges for education institutions. Hard skills are most often traditionally approached via memorising formalised content as part of the learning contract, while soft skills are formed and nourished through more interactive learning strategies. Both approaches may be further complicated depending upon factors such as context, cohort and cultural behaviours around instruction delivery. Accuracy in measuring the actual degree of understanding gained by students is arguable.

Formal exams have long been used in various disciplines as a means of assessing student learning. Closed Book Exams (CBE) and Open Book Exams (OBE) are examples of assessment methods used by educators to evaluate student performance and understanding in subjects. CBE often verge on recall of content and formulae, while OBE entail questions to be written in a manner which requires interpretation of content, and as such, reflects the true learning outcomes of the student cohort.

With the increasing accessibility of information through technology, the need to memorise and describe theories and concepts diminishes. Emphasis ought therefore be placed more on the application of such knowledge into various situations including the use of skills such as critical thinking, problem-solving and ability to organize, retrieve and handle information. Replacing questions that may rely on memorising in CBE, OBE instead aims to assess situational skills. It requires students to provide responses to scenario-based questions or problems through the assimilation and application of learnt knowledge and information from various sources. As an assessment item, OBE may potentially better affirm student capability in tune with skills needed, and roles offered, in the workplace.

BACKGROUNDKnowledge and the access to knowledge has dramatically increased with the evolution of technologies, which is more evident in the highly technical domains where innovative demands create constant change. A cycle that continues to evolve since the 1950’s. As business environments continue in becoming more of the survival of the fittest, graduates in technical fields must not only come readily equipped with the necessary technical knowledge, but also to be able to navigate in a changing environment. To successfully enable this, tertiary preparation needs to better address learning, and subsequently the examination environment to ensure graduates are equipped with a deeper and practical knowledge.

Theophilides and Koutselini (2000) state that closed book examinations (CBE) are a well-established process which is widely and frequently used in testing knowledge. In theory, CBE require the student to answer questions, use recalled knowledge and, to some extent, reproduce rote-learnt information. Open book exams (OBE) on the other hand, permit students to consult notes, text books and any other materials within the exam format itself and which the student considers to be of aid – while mostly excluding devices that can be employed for external communication such as computers, mobile phones and the like.

Furthermore, open book examinations normally fall into one of two categories – partial (or restricted) open book or fully open book. Partial open book examinations permit the student to take notes or texts

Proceedings, AAEE2018, Hamilton, New Zealand


into the examination, however the amount and type of materials are restricted (usually to one or two pages of notes). This encourages the student to prioritise value and importance of knowledge, and condense it prior to entering the exam. Fully open book examinations have no such restriction on the volume of permitted materials, thus allowing wider prioritisation during the exam itself which may come at the cost of the time available for question response in the examination.

There has been considerable studies and discussion on the preparation, reaction and success of open book examination, which highlight that both the positive and negative aspects must be considered when using or considering the use of OBE in differing circumstances, as may be seen in works by Theophilides and Dionysiou (1996), Jehu, Pincton, and Cher (1970), Koutselini (1997), Feldhusen (1961), Michaels and Kieran (1973), Weber, McBee and Krebs (1983) in Rakes (2008).

From the positive perspective of OBE, they can reduce exam tension and stress, provide fairer assessment of conceptual understandings and provide a pathway for more lasting learning outcomes (Feldhusen, 1961; Jehu et al., 1970; Michaels and Kieran, 1973; Weber et al., 1983 in Theophilides, C. and Koutselini, M., 2000). They also reduce the need to memorization of facts and encourage more constructive practical learning (Feldhusen, 1961; Theophilides & Dionysiou, 1996). Vanderburgh (2005) also identifies innovative use of gained knowledge and a deeper probing of the subject material, thus encouraging students to summon higher levels of critical thinking.

Sambell & McDowell (1998) have indicated that with OBE there is an attempt to have the students understand and apply insight to real-life situations, rather than simply employ memory recall to regurgitate course materials. Through the use of OBE, deeper comprehension of subject matter allows the student to apply course materials to more complex problems, providing a more genuine assessment of learning gained. Extant literature indicates value of implementing authentic assessment as a means of testing conceptual thinking but that this also needs to be developed around a set of authentic and real-life tasks (Dochy 2001; Gielen 2003 in Whitelock and Simon Cross, 2012). OBE most naturally fits this framework.

Several negative problems were also recognized in the use of OBEs. Firstly, students often spend excessive examination time attempting to locate the relevant information consuming available time to formulate responses to questions (Vanderberg 2005; Koutselini, 1997; Theophilides & Dionysiou, 1996). Additionally, OBE can lead to a potential reduction in exam preparation through false confidence of a possible outcome of the exam, leading to the student being insufficiently prepared for the exam, as noted by Kalish (1958 p200), who points out, “makes students believe that they will be successful in the course” when this may not necessarily be the case. It has also been noted (Vanderberg 2005) that OBEs increase in academic overheads linked to both preparing and marking of OBEs.

Studies related to student’s study behaviour, attitudes and approaches comparing each type of examination are wanting. Such studies provides valuable information regarding the specific behaviours that may influence a student’s performance in a course by uniquely identifying the features of both OBE and CBE examinations. The purpose of this study was to investigate and identify the behaviours and perceptions of students when addressing the two different types of exams. The focus also includes the different stages of the exam process – the preparation, during the exam and post exam.

APPROACHThis pilot research approach was quantitative in nature employing two similar voluntary and anonymous surveys administered before and after sitting an OBE in June 2018 to students in three post-graduate engineering and information technology subjects within the University of Technology Sydney. These subjects were chosen because they are reflective of the cohorts of professional postgraduate programs. Participants were of mature working age with previous experience of closed book examinations.

The student cohort in the three subjects surveyed was quite diverse. From a total of 356 enrolled across the three surveyed subjects, approximately 20% were female. The actual gender breakdown is shown in table 1.



Gender and Cultural Statistics of Student Cohort

Students Female Male Total Female percentageTotal 71 285 356 19.94

Table 1: Gender categorisation

While each of the three subjects have a diverse cohort with students coming from 29 countries. The five countries which contributed approximately 84 % of the cohort (299 of 356 students) are indicated in table 2.

Country Students Female Male % of Cohort

India 128 26 102 36.0China 94 22 72 26.4

Australia 28 3 25 7.9

Pakistan 26 0 26 7.3Nepal 23 3 20 6.5Total 299 54 245 84.1

Table 2: Top five countries of student cultural backgrounds

The first survey was offered to students prior to sitting an OBE for the first time. Most questions could be answered on a sliding scale that addressed student pre- OBE views associated with:

OBE preparation compared to CBE; Expectations of an OBE experience compared to CBE; Expectations of resources required (notably materials and time) to successfully sit an OBE; Expected difficulties that may arise during an OBE compared to a CBE; Adequacy of an OBE to test understanding compared to a CBE.

The second survey sought comparative views post the conclusion of the OBE experience. Of particular interest were capturing students’ observations of whether study preparations were adequate, prior expectations of an OBE were met, plus any unforeseen/unexpected difficulties that were encountered.

While both surveys looked for specific response weightings in the key areas, respondents could also add further clarification in their own words if they so wished. This allowed participating students to add qualitative depth to quantitative answers.

The validity of perception-based metrics is endorsed by Tang (2005) who suggested that such measures are most valuable when assessing comparative situations over wide-ranging domains – although his specific research was aimed at activities in commercial environments it could just as easily be applied to academic ones. The partnering of quantitative perceptions to focused qualitative responses can be enlightening as well as permitting cross-correlation of common themes.

FINDINGS The number of respondents to the pre-exam survey was 105. However only, only 89 responds were usable as the other 16 were either not fully completed or were submitted after the formal exam period has commenced. In the post exam survey 77 fully completed responses were received during the two week period the survey was opened. Of those who responded to the post- exam survey 35 of 77 did not undertake the pre-exam survey.



1. Students Overall Preference of Exam TypeWhen comparing open book to closed book exams student perceptions were compiled both before (pre exam survey) and after sitting the open book exam (Post exam survey). The results are shown in tables 3 and 4.Students’ Preferred Exam Type Closed book

examNo Preference

Open book exam

Pre-OBE exam survey 19% 34% 47%Post OBE exam survey 27% 55% 18%Table 3: Students’ preferred exam type before and after sitting OBE

It is interesting to note from Table 3 that almost half the students (47%) were in favour of sitting an OBE compared to only 19% who favoured a CBE. However, after actually sitting the OBE, student preference for OBE was reduced to 18%, while preference for CBE increased to 27%. Equally interesting is the 21% increase in the ‘No Preference’ result once students experienced sitting an OBE indicating that they no longer preferred OBE to CBE.

Table 4 illustrates student perceptions of OBE from the pre and post exam surveys. It is curious to note that after sitting an OBE, students’ perceptions of OBE difficulty was significantly reduced and they also deemed OBE to be less stressful than CBEs. This is understandable as students would feel more ”secure” having some subject content with them that would provide a sense of assurance/ “safety net’.

CriteriaPre-exam survey

Post-exam survey

Open book exams are More Difficult than closed book exams 44.1% 28.8%

Open book exams are Less Stressful than closed book exams 41.6% 50.8%

Open book exams are the Same as closed book exams 14.3% 20.4%

Table 4: Student perceptions of open versus closed book exams

2. Difficulties of OBEIn the pre-exam survey students were asked to identify what difficulties they perceived they may face when sitting an open book exam. Of the six (6) listed criteria in this question, four (4) were considered as high potential difficulties in an open book exam. They were

1. ‘finding relevant material to answers questions’ (66%) 2. ‘time to answer questions’ (55%), 3. ‘validating my initial thoughts of the answers’ (46%).4. ‘determine what to take into the exam room’ (45%),

These difficulties point out that prior to sitting an OBE, students perceived that OBEs may be harder than CBEs and/or that OBE require more exam time and effort than CBE.

Intriguingly the highest percentage referred to “finding relevant material to answers questions’ (66%)”, this may indicate that students did not properly prepare for the exam because of the ability to take material into the exam.

For the post exam survey respondents identified the actual difficulties they experienced when sitting the open book exam. The top difficulties identified were:

1. ‘time to answer questions’ (31%) 2. ‘I took more resources into the exam than I needed’ (22%) 3. ‘validating my initial thoughts when compiling answers’ (16%) 4. ‘finding relevant material to answers questions’ (14%).

While ‘validating my initial thoughts when compiling answers’ aligns with being prepared for the exam, and ‘finding relevant material to answer questions’ potentially indicates that a number of respondents



were clearly underprepared for the exam and thought that having material with them in the exam would overcome limited exam preparation.

3. Material Taken into the Exam RoomRegarding what to take into an open book exam, student perceptions in the pre exam survey focused on ‘‘personally complied notes or materials’ (88%) and lecture notes’ (81%). Students also considered the ‘textbook’ (50%) and ‘other reference material’ (39%) as valid sources of material for the exam. In the post exam survey students were asked what ‘they took into the exam’. Again the predominant sources of material were ‘lecture notes’ (41%) and ‘personally complied notes or materials’ (30%) while ‘textbook’ (11%) and ‘other reference / reading material’ (18%) were also supported.

In comparing this set of pre-exam and post-exam data it would appear that, in actuality, students predominantly took only one source of material into the open book exam. With ‘lecture notes’ and ‘personally complied notes or materials’ being the preferred choice of material (and in fact these may well have been the same set of content).

In regards to the use of material ‘to take in’ (pre-exam survey) or ‘taken in’ (post-exam survey) student responses were similar for two criteria, namely ‘quick reference’ (41.5% pre-exam and 38.5% post-exam) and ‘validation of answers’ (19% pre-exam and 17.5% post exam). Two matters of note come from the data on this question.

1. There was higher response rate in the post-exam survey to ‘searching for answers’ (23.5% as compared to 11% in the pre-exam survey) and a lower response rate for ‘re-enforce learned knowledge’ (28% pre-exam but only 20% post-exam). This may indicate a lack of preparation from students before the exam or that exam questions required analysis and interpretation for the answer rather than descriptive answers.

2. One student indicated they ‘did not use any of the material they took into the exam’.

In the post-exam survey, students were asked about their perceptions of how they may prepare for future OBEs. Not surprising, is that potentially not having sat an OBE before, students now understand the need to prepare themselves for an open book exam as 61% indicated they ‘will study, review and refine all the material I collate to take into future exams’, rather than simply rely on the materials taken in and ‘search for answers’. A further 26% of students indicated they ‘will collate the material they take into future exams’ and 12% of those students also indicate they would ‘review that material’.

In regards to whether students perceived that having material in the exam would hinder their performance by reducing the time to answer questions. In the pre-exam survey students indicated that this would not be the case with an approximate 60% (No impact) and 40% (Yes, an impact). Conversely, in the post-exam survey approximately 66% of respondents indicated that having material in the exam room did impact on their performance, while 34% indicated there was no impact or difference.

4. Impact on Examination Preparation TechniquesIn regards to whether students believe that open book exams will change examination preparation techniques, the pre-exam survey requested respondents to indicate preference to a range of options. Results are shown in Table 5.

Do you believe that open book exams will change your examination techniques?

Percentage

Yes Maybe No Blank

No change to closed book exam 19.1 44.9 31.5 4.5Prioritizing (Order of answering questions) 39.3 38.2 16.9 5.6Time management (Distributing time amongst questions and use of reference material) 66.3 22.5 7.9 3.4The depth of an answer (providing more details from reference material) 64.0 28.1 5.6 2.2The way you approach problem solving questions (using reference material rather than relying on memory) 67.4 23.6 6.7 2.2



Table 5 Change in exam preparation technique – pre-exam perceptions

Table 6 below illustrates the results to the counterpart question in the post-exam survey.

Do you believe that having sat an open book exams it will now will change your examination technique for future open book exams?

Percentage

Yes Maybe No Blank

No change to closed book exam 55.8 20.8 23.4 0.0Prioritizing (Order of answering questions) 67.5 20.8 11.7 0.0Time management (Distributing time amongst questions and use of reference material) 76.6 14.3 7.8 1.3The depth of an answer (providing more details from reference material) 72.7 19.5 7.8 0.0The way you approach problem solving questions (using reference material rather than relying on memory) 66.2 23.4 7.8 2.6Table 6: Change in exam technique – post-exam perceptions

There has been a shift in student perceptions across all categories, except ‘the approach to problem solving’ following the completion of the open book exams. This indicates that in future exams, students perceive that they will not only be prepared before entering the exam (see on the previous page where 61% of respondents indicated ‘I will study, review and refine all the material (including my personally compiled notes I collate to take into future exams’) but will also develop strategies for answering questions (around the dimensions of ‘prioritizing’, ‘time management’ and ‘depth of answer’) to ensure they complete the exam whilst giving themselves the best opportunity to do well.

Qualitative analysisIn analysing the free format comments made by the respondents (27 from 89 in the pre-exam survey and 25 from 77 in the post-exam survey) the following main themes emerged – 1) Writing, 2) Time Management, 3) Question Types and 4) Preparedness.

1. WritingWriting was a concern identified from a number of the comments in both surveys. This, in some way, was to be expected given the highly diverse student cohort made up from 29 countries enrolled in the three subjects. This was identified by comments such as:

“Local students with good English definitely will be benefited by open book. It is a bit unfair compare with international students. International students need some time to think of something before writing the answer ...” (pre-exam)

“we are not do hand writing in daily life any more, the hand writing will be the most challenge thing in exam … especially for the student from non-English native background … “ (pre-exam)

“As an international student, writing lots of words during exam could be hard. Honestly, writing skill is a pre-skill that is required” (post-exam)

2. Time ManagementWhile all students knew the exam time limit was two hours (a University directive), time to answer questions was still a factor identified in the free format comments such as:

“Main thing is we cannot manage time” (post-exam).

“The questions asked too much and I didn't have enough time to finish it” (post exam)



“There isn't always enough time to validate the answers since the question paper is set more difficult keeping in mind that students have notes during the examination” (pre-exam).

3. Types of QuestionsThe type of questions perceived or that were encountered in the exam surfaced as an issue among the free format comments for example:

“Descriptive questions should be included” (post-exam)

“… the question paper is set more difficult keeping in mind that students have notes during the examination” (pre-exam).

“through my experience, I did not have enough [time] to answer all questions comprehensively, because, during 2 hours, I needed to read the cases, analysis the issues, consider my draft answers, organise my words and write down … That was a bit challenging” (post-exam)

4. PreparednessFinally and most importantly the need to prepare for the exam was identified as factor influencing open book exams in the free format comments such as:

“Open book exam is good enough to improve your knowledge. But the students need to understand the topics clearly and should prepare for exam otherwise it will be difficult” (pre-exam)

“Less stressful to take, but more anxious to prepare and to anticipate the results” (pre-exam)

“Open book exams, sort of make students lazy, not to be fully prepared, so guess students just prepare the material and wait for the exam day to refer to them and answer the questions from them” (pre-exam)

“To prepare for open book exam we need to have a good understanding in all the topics … This open book exam will help the students to think rather than studying” (post-exam).

CONCLUSIONOne of the drivers for open book exams came from a component of our University’s model of learning which is to conduct authentic assessment items. From the survey results it appears that this was also a learning experience for them which will hold them in good stead as they progress through their studies and future careers. This is supported by several free form comments, for example: “I believe they better reflect the real-world where interpretation of information is more crucial than what has been retained in memory” (pre-exam) and “Open book exams are more reflective of the real world where I may know where to look for an answer, obtain the answer and then synthesis the responses needed for a client, project or report. The trick is having exam questions appropriately tailored for open book” (post-exam)

Our surveys showed that student perceptions of OBE difficulty and stress levels changed. Prior to sitting OBEs, students perceived them to be more difficult than CBEs. However after sitting OBE the perception of difficulty was significantly reduced. Students also reported having lower stress levels after sitting an OBE when compared to stress levels of CBEs.

Finally to wrap up, in regards to students perceptions of OBE versus CBE, the last word belongs to this perceptive student who made the following comment in the pre-exam survey:



“It doesn't matter whether open book or close book if we prepare well for the exam it should be ok”.

REFERENCESDochy, F., 2001. A new assessment era: different needs, new challenges. Learning and instruction, 10(suppl. 1), pp.11-20.

Feldhusen, J.F., 1961. An evaluation of college students' reactions to open book examinations. Educational and Psychological Measurement, 21(3), pp.637-646.

Kalish, R.A., 1958. An experimental evaluation of the open book examination. Journal of Educational Psychology, 49(4), p.200.

Koutselini, M., 1997. Contemporary trends and perspectives of the curricula: Towards a meta-modern paradigm for curriculum. Curriculum Studies, 5(1), pp.87-101.

Rakes, G.C., 2008. Open book testing in online learning environments. Journal of Interactive Online Learning, 7(1), pp.1-9.

Sambell, K. and McDowell, L., 1998. The construction of the hidden curriculum: messages and meanings in the assessment of student learning. Assessment & Evaluation in Higher Education, 23(4), pp.391-402.

Tang, J. (2005) “Competition and innovation behaviour”, Research Policy, 35(1):68-82.

Theophilides, C. and Dionysiou, O., 1996. The Major Functions of the Open-Book Examination at the University Level: A Factor Analytic Study. Studies in Educational Evaluation, 22(2), pp.157-70.

Theophilides, C. and Koutselini, M., 2000. Study behavior in the closed-book and the open-book examination: A comparative analysis. Educational Research and Evaluation, 6(4), pp.379-393.

Vanderburgh, P.M., 2005. Open-book tests and student-authored exam questions as useful tools to increase critical thinking. Advances in physiology education, 29(3), pp.183-184.

Whitelock, D. and Cross, S., 2012. Authentic assessment: What does it mean and how is it instantiated by a group of distance learning academics?. International Journal of e-Assessment, 2(1), p.9.

Williams, J.B. and Wong, A., 2009. The efficacy of final examinations: A comparative study of closed‐book, invigilated exams and open‐book, open‐web exams. British Journal of Educational Technology, 40(2), pp.227-236.


Documents

opus.lib.uts.edu.au read… · Web viewWith the increasing accessibility of technology, the need to memorise and describe theories and concepts diminishes. Emphasis should therefore