Assessing the eff ectiveness of a fl ipped classroom in foundational engineering dynamics

Report

Dr Paul DochertyDr Wendy Fox-TurnbullPinelopi Zaka

May 2017

Research Team

Paul D Docherty - Department of Mechanical Engineering

Wendy Fox-Turnbull - School of Education

Pinelopi Zaka - e-Learning Support

Acknowledgements

The authors would like to thank Bridget O’Regan and Ako Aotearoa for their support of the

project, as well as student participants that shared their experiences with the flipped

approach. We also express gratitude to the independent reviewer who improved the

document via their thoughtful input.

This project was funded through the Ako Aotearoa Southern Hub Regional Hub Fund 2016.

More information is available at https://akoaotearoa.ac.nz/projects/flipped-classroom-

foundation-engineering

Published by Ako Aotearoa National Centre for Tertiary Teaching Excellence, PO Box 756,

Wellington 6140.

ISBN: 978-0-947516-72-7

June 2017

This work is published under the Creative Commons 3.0 New Zealand Attribution Non-commercial Share Alike Licence (BY-NC-SA). Under this licence you are free to copy, distribute, display and perform the work as well as to remix, tweak, and build upon this work non-commercially, as long as you credit the author/s and license your new creations under the identical terms.

A flipped classroom approach has been touted as a future teaching methodology in the

Department of Mechanical Engineering at the University of Canterbury. A pilot

implementation of the approach was trialled in a small cohort of first year engineering

students. This research assessed the efficacy of the approach.

Qualitative and quantitative methodologies were used to determine the effectiveness of the

approach. The quantitative approach used a researcher independent metric of academic

success. Semi-structured interviews were used to gather student impressions of the

approach. The quantitative approach showed that the student performance was not

significantly changed by the flipped approach. However, the qualitative analysis showed that

the approach was appreciated by the students. In particular, the student’s liked the

collaborative environment of the help sessions, and the increased direct communication with

the lecturer, as well as the flexibility that independent learning provided to them.

The expansion of the approach within the mechanical engineering undergraduate teaching is

feasible. The implementation of the approach was not particularly arduous and the overall

student experience was enhanced by the approach.

Contents

1. Introduction .................................................................................................................... 5

1.1. Motivation for the Study .......................................................................................... 5

1.2. Literature Review .................................................................................................... 5

1.2.1. The Flipped Classroom Defined ....................................................................... 5

1.2.2. Rationale for and Characteristics of Flipped Classroom in Tertiary Education .. 6

1.2.3. Students’ Perceptions of the Flipped Approach to Learning ............................. 7

2. Study design .................................................................................................................. 9

2.1. Overview of Degree Structure ................................................................................. 9

2.2. Course Content First-year Mechanics Course ......................................................... 9

2.2.1. Semester 2 - Traditional Delivery ..................................................................... 9

2.2.2. Summer semester – Partial Flipped Delivery ................................................... 9

3. Qualitative Methodology ............................................................................................... 11

3.1. Qualitative Findings .............................................................................................. 11

3.1.1. Students’ Perspectives of Teaching ............................................................... 12

3.1.2. Students’ Perspectives of Learning ................................................................ 14

3.2. Discussion of Qualitative Outcomes ...................................................................... 15

4. Quantitative Methodology ............................................................................................ 17

4.1.1. Content of Second-year Dynamics Course .................................................... 18

4.1.2. Analysis Methodology .................................................................................... 18

4.2. Quantitative Results .............................................................................................. 19

4.3. Discussion of Quantitative Outcomes.................................................................... 21

5. Recommendations ....................................................................................................... 23

5.1. Recommendations for lecturers ............................................................................ 23

5.1.1. Materials development and provision ............................................................. 23

5.1.2. Lecturer attitude ............................................................................................. 23

5.1.3. Quality of materials ........................................................................................ 24

5.1.4. Availability and accessibility of content........................................................... 24

5.1.5. Student preparation ....................................................................................... 24

5.2. Recommendations for students............................................................................. 24

5.2.1. Student preparedness .................................................................................... 24

5.2.2. Receptivity to approach ................................................................................. 24

5.2.3. Time management ......................................................................................... 24

6. Impacts of research ..................................................................................................... 25

6.1. Impact on Learners ............................................................................................... 25

6.1.1. Retrospective ................................................................................................. 25

6.1.2. Prospective .................................................................................................... 25

6.2. Impact on the teacher ........................................................................................... 25

6.2.1. Retrospective ................................................................................................. 25

6.2.2. Prospective .................................................................................................... 25

6.3. Impact on institution .............................................................................................. 25

6.3.1. Retrospective ................................................................................................. 25

6.3.2. Prospective .................................................................................................... 25

6.4. Impact on Research Team .................................................................................... 26

6.4.1. Retrospective ................................................................................................. 26

6.4.2. Prospective .................................................................................................... 26

7. Conclusions ................................................................................................................. 27

8. References .................................................................................................................. 28

1. Introduction

The efficacy of the flipped classroom approach for teaching engineering in a New Zealand

tertiary environment was assessed in a mixed methods study. An early formalisation of the

approach was published in 1982 (Baker, 2000) but has become popular in recent times due

to availability of technology, malleability of teaching methodologies, and fiscal restrictions

and pressures, pressures on time and resources within universities, and an increased

student demand for student-centred teaching. The flipped classroom approach has the

potential to provide educators with an opportunity to fulfil these varied influences on modern

practice. However, it is important to directly access the applicability and efficacy of the

methodology within the proposed contexts it may be expanded to cater for. Hence, student

feedback, impressions, and achievement must be considered.

This study investigated engineering students’ impressions of the flipped classroom approach

and their ongoing academic achievement in follow-on courses. The approach was utilised to

teach a summer school paper to two separate cohorts over two consecutive years. The

course content covered foundational engineering dynamics and was in the first year of a four

year Bachelor of Engineering (with honours) degree programme in New Zealand.

1.1. Motivation for the Study

Recently, the tertiary teaching sector has had an increased uptake of alternative methods of

teaching and learning (Blair et al., 2016). There is some literature reporting success and

failure of the flipped classroom approach (Blair et al., 2016, Love et al., 2014, Lavelle et al.,

2013, Gannod et al., 2008). In an effort to assess whether the flipped classroom is beneficial

or detrimental for engineering students, a flipped classroom approach was trailed in two

summer school cohorts over a period of two years. This study gathered the student

impressions and ongoing academic success as well as recorded suggestions for making

further improvements to the course and wider implementation of the methodology.

Students are increasingly expecting self-paced learning within the tertiary sector. The

readiness of information allows students to quickly compare and contrast educational

institution practices. Thus, in order to remain competitive, institutions must adapt and uptake

the most well-accepted and successful teaching methodologies. There is also increased

pressure from qualification authorities and governments for universities and other tertiary

institutions to perform more effectively and efficiently (Bishop and Verleger, 2013, O'Flaherty

and Phillips, 2015b).

1.2. Literature Review

1.2.1. The Flipped Classroom Defined

The flipped classroom approach uses two distinct stages: (1) course content introduction

and delivery at the students’ own time and pace (often delivered online). Course content is

presented as readings, videos, graphical presentations or quizzes (Lavelle et al., 2013, Blair

et al., 2016, Hanson, 2016). Students engage with this material at their own time and pace.

(2) These ‘lectures’ are followed with workshops within which students engage with the

recently delivered course materials (Bishop and Verleger, 2013). The workshop sessions

often include elements of interactive, collaborative and applicative engagement with the

course content and solidify the concepts delivered by the materials (Lavelle et al., 2013,

Bishop and Verleger, 2013, Hanson, 2016).

1.2.2. Rationale for and Characteristics of Flipped Classroom in Tertiary

Education

Blair et al. (2016) suggest that recent technological advances facilitated the shift in tertiary

teaching from the traditional teacher-centred approach to a student-centred approach. Mayer

(2002) notes that the approach facilitates knowledge uptake and retention as opposed to

simple transfer of knowledge and facts. The shift to student-centred pedagogy resonates

with the sociocultural learning theory of Vygotsky (1978). In particular, the ‘Zone of Proximal

Development’, is directly relevant to student learning within the flipped classroom.

Sociocultural approaches to education contrast with traditional approaches that suggest the

students’ learning unfolds via orderly stages. In contrast, the sociocultural theory focuses on

the roles teachers and peers play in education with an emphasis on group interactions and

collaborative learning (Richardson, 1998, Daniels, 1996).

Guided participation has been recognised as essential to uptake of thought patterns that are

consistent with individuals that are proficient in particular fields (Bishop and Verleger, 2013).

This important learning methodology is intrinsic within the flipped classroom approach as

learning occurs both collaboratively and independently (Lavelle et al., 2013). In particular,

while students engage with course content independently, scheduled class times enable

guided participation via collaboration and direct lecturer engagement. Sociocultural conflict

theory is thus particularly relevant to the flipped classroom approach to learning. This theory

suggests that theoretical discrepancy or conflict across individuals sparks cognitive

development. Furthermore, socio-cognitive conflict theory recognises conflict as essential to

cognitive change. Doise and Mugny (1984) demonstrated that students working in pairs

solve higher level problems than those working in isolation; regardless of their partner’s

ability. When students were challenged with alternative opinions, their own performance was

enhanced, regardless of the validity of the opposing points-of-view; thus, the social

interaction of working in partnership enhanced and augmented the learning and performance

of the students.

Engineering inherently requires real-world skills that must be taught (Choi, 2013). The

flipped classroom has exhibited positive outcomes in lab-based classes (Lavelle et al.,

2013). The increased direct student-teacher engagement ultimately facilitates better

mentoring of practical skills. The mentoring occurs as required and is thus relatively efficient

(Lavelle et al., 2013).

Johnson and Renner (2012), Lavelle et al. (2013), and Blair et al. (2016) identify a number of

advantages for the flipped classroom. Course materials can be altered to provide for a

diverse range of students with differing learning styles and can also be adapted to suit those

with disabilities. Formative assessment can be utilised to provide ongoing support and

feedback to students, in a way that fits with the natural flow of the help sessions. In contrast,

such practice in traditional lectures is not well-received by students. Choi (2013) argues that

the flipped classroom promotes specific and immediate feedback for optimal education

settings.

Barker (2000) noted the shift of the teacher to a ‘facilitator’ rather than a ‘director’ of learning

in the flipped classroom. This was coupled with concurrent increases in: direct

communication between the teacher and students, use of active learning, focus on

understanding and application, provision of student control, a greater sense of student

responsibility over learning, and collaboration.

The flipped classroom approach has some intrinsic disadvantages. Notably, Johnson and

Renner (2012) suggest that students must be disciplined to succeed in a flipped classroom.

Furthermore, the approach assumes that students have internet access. Some students are

not inclined to view the materials prior to their workshop classes and thus the collaborative

environment can become ineffective. Also, material development and editing is generally

very labour intensive (Lavelle et al., 2013).

1.2.3. Students’ Perceptions of the Flipped Approach to Learning

Student perception of the flipped classroom varied broadly (Johnson and Renner, 2012, Blair

et al., 2016, Love et al., 2014, Nguyen et al., 2015, Hanson, 2016). Blair and Colleagues

(2016) report that after engineering students engaged in a flipped classroom they were keen

for a wider deployment of the methodology. Student perceptions of the approach were

heavily influenced by the online material quality (Blair et al., 2016). Hanson (2016) noted that

increased understanding through dialogue in the face-to-face component, wider and deeper

thinking, the ability to pause and replay the online lecture material and the flexibility of time

to avoid conflicting commitments and students also acknowledged a reduced sense of

isolation and disengagement were all benefits of the approach. A number of students

indicated that the flipped classroom requires different pedagogical approaches to ensure

sustained academic performance and recognised the overall approach requires self-

discipline from the students. Somewhat unexpectedly, students also recognised that the

flipped approach was more efficient than the traditional lecture approach (Blair et al., 2016,

Nguyen et al., 2015, Hanson, 2016, O'Flaherty and Phillips, 2015b).

Nguyen et al. (2015) noted student feedback that communicating with fellow students while

watching online lecture materials was beneficial and an important aspect of good pedagogy.

They also recognised that engagement with the recorded online material was critical to

success (Nguyen et al., 2015, Pierce and Fox, 2012, Blair et al., 2016). Students in several

studies noted the reviewability of the materials as a significant positive for the flipped

classroom (Pierce and Fox, 2012, Love et al., 2014, Blair et al., 2016).

Hanson (2016) reported that there lacked uniform positivity for the approach. In particular,

some students felt unprepared for the approach. O’Flaherty and Philips (2015b) found

similar responses where students were initially ambivalent to the need to engage in both

workshops and the online materials. Nguyen (2015) and Zhu et al. (2016) suggest that

students must be informed of the value of discipline when engaging with flipped education.

The literature indicates varied academic achievements from students of the flipped

classroom. O'Flaherty and Phillips (2015) reviewed a number studies and found that, in

general, there was an improvement in students' academic achievement concurrent with the

use of flipped approaches. In studies typically measured either with pre vs post test scores

or course grades compared to those of previous years where traditional approaches were

used. However, both metrics are highly investigator dependent, and thus, the statistical

validity of the outcomes is tenuous. Kerr confirmed that there is some improvement in

engineering students' academic achievement when flipped approaches are being used

(Kerr, 2015). Yelamanthi and Drake compared the achievement of 17 students who

participated in a flipped version of a first year Digital circuits course to 22 students

participating in the traditional version of the same course and found a significant

improvement in students' academic success from students of the flipped classroom

(Yelamanthi and Drake, 2015). However, the ability of the lecturer to provide individualised

assistance to students of the small class limits the applicability of the outcomes to larger

cohorts. Wilson compared student results across a flipped cohort and a previous years'

traditional class cohort in an engineering management course (Wilson, 2013) and found a

lack of statistically significant differences in student achievement across methodologies. This

is despite a clear indication that students preferred the flipped approach. Similarly, Mason,

Shuman and Cook, in their study of flipped approaches in a senior year Control Systems

course, found that students in the flipped classroom only performed better than students in

the traditional classroom when solving certain problems (Mason et al., 2013). The authors

conclude that the flipped approach "at best improved students’ understanding of engineering

concepts, and at worst did no harm”. Statistically, this metric and interpretation is invalid and

unfounded, as if one measures enough outcomes, one may expect significant outcomes in

some cases as a matter of statistical likelihood. This is an example of ‘p-hacking’ (Selvin and

Stuart, 1966).

Most importantly, it is a reasonable assumption that publication bias obscures the true

impact of the flipped classroom on the academic success of students (Pigott et al., 2013, van

Hilten, 2015, Dickersin, 1990). In particular, publication bias can occur when studies that had

negative or ambiguous outcomes due to the intervention decide not to publish their findings.

This loads the academic literature with a biased collection of outcomes and

recommendations. Pigott et al. (2013) assessed a number of educational intervention thesis

to their published versions and found that statistically significant outcomes were more likely

to be published than non-significant outcomes. The authors urge researchers to report all

findings in order to generate a more even body of academic literature in educational

interventions. Such a culture change will lead to successful and representative meta-

analysis of outcomes, and thus data driven policy change can be undertaken without fear

that the data is affected by bias or statistical anomalies.

2. Study design

This study utilised a mixed method methodology to capture the qualitative and quantitative

outcomes of student achievement and impressions of the flipped classroom approach. The

class analysed was a mechanics class that is foundational in the degree structures of civil

and mechanical engineering at the University of Canterbury. The flipped classroom was

restricted to only the second half of the course that concentrated on engineering dynamics.

2.1. Overview of Degree Structure

Engineering at the University of Canterbury uses a structure with two semesters of four

papers in each. A bachelor of engineering with honours degree (BE (hons)) has 32 papers

and is a four year program (assuming a full course load). All classes are compulsory and

students are required to repeat classes if they fail them. The first-year (termed the

‘intermediate year’) includes foundational engineering and physical science content. The

following years are labelled the first, second and third professional years. The professional

years prepare the students to practice as professional engineers. Entrance to the first

professional year of engineering is competitive. Student intermediate year grades are the

basis for comparative entry to their preferred engineering disciplines. International exchange

students are sometimes allowed to enrol in selected courses within the program as well.

2.2. Course Content First-year Mechanics Course

The statics section of the first-year mechanics course taught analysis of rigid body forces,

distributed forces, trusses, and three dimensional vector operations. The dynamics section

taught simple kinematic, equations of motion via Newton’s Laws, work, impact, collisions and

energy methods.

2.2.1. Semester 2 - Traditional Delivery

The traditional delivery of the course involved 48 lectures that had a 50 minute duration. The

first twenty four lectures delivered the statics content and final 24 lectures delivered the

dynamics section. There were four lectures and one two-hour tutorial per week and a two-

week break between the statics and dynamics sections. Online quizzes provided formative

assessment and there were three graded labs. There was a statics test in the first week of

the dynamics section and an exam approximately two weeks after the dynamics section.

The dynamics lecturer annotated prepared notes under a document camera with occasional

physical demonstration. The students had access to the unannotated notes and copied

annotations while the lecturer verbally described the content.

2.2.2. Summer semester – Partial Flipped Delivery

There were a total of 24 lectures that were 110 minutes in duration. There were three

lectures per week and the course was divided into two four-week blocks either side of a two-

week Christmas break. Hence, the delivery rate of taught materials was faster in the summer

occurrence of the course than the semester 2 occurrence. This increased pace was deemed

acceptable as students were limited to enrolling in only two summer papers. Only the

dynamics section of the summer occurrence was taught using the flipped delivery. In

contrast, the statics teaching used the traditional lecture model.

The flipped environment provided the students with video lectures that were prepared in a

PowerPoint framework. The videos used the same notes as the traditional delivery and

similar annotation would appear in the videos as the lecturer orally described the content. A

50 minute lecture would typically condense to a ~35 minute video. The students watched

two video lectures before attending each scheduled workshop. Students generally annotated

their notes mimicking the annotations of the videos and brought their annotated notes to the

scheduled workshop. Scheduled workshops started with question/answer sessions.

However, since students can resist in-class participation (Magnuson-Martinson, 1995, Eble

and McKeachie, 1985, Hollander, 2002), the lecturer encouraged a culture of participation by

planting questions before the first session. The lecturer also required students to work on

problems in groups. The question and answer part of the face to face session generally

lasted 30-40 minutes, depending on the issues that arose. The remainder of the session was

dedicated to example problems that required the content presented in the previous lectures

to complete the problems. Opportunities for student feedback (online and face to face) were

also frequently provided, in order to inform the lecturer’s approach with the flipped classroom

and to identify possible challenges that the students needed further support on.

3. Qualitative Methodology

Qualitative methods were used to evaluate the students’ perceptions and impressions of the

flipped classroom methodology. Interpretivism is the study of why social action occurs by

first understanding how emotions and world views motivate individuals (Neuman, 2000). The

aim of the interpretive paradigm is to understand the subjective individual experiences with

the integrity of the subject. The paradigm also aims to understand how people construct

meaning from their environment (Neuman, 2000, Taylor and Bogdan, 1998). The

interpretivist paradigm enabled the researchers to examine student perspectives on the

flipped classroom, their interaction with their lecturer, the technologies used, and other

culturally situated tools were used to construct engineering dynamics knowledge and

understanding.

Twenty students in the flipped classroom participated in semi-structured, focus-group

interviews. Students were recruited via emails that were addressed to the 2014 and 2015

summer cohorts. Students were offered $50 to $100 remuneration to cover costs of

attending the interviews and to encourage participation. In total eight interviews were

undertaken and scheduling limitations required two individual interviews.

Interviews were undertaken by either Dr Fox-Turnbull or Ms Zaka. External agencies were

used to transcribe recorded interview audio. Aspects of this research were ethically

restrictive (Cohen et al., 2013) as Dr Docherty was the course lecturer and had ongoing

authority over the students. In particular, he had roles that involved grading and pastoral

support. Due to these ethical considerations, Dr Docherty did not undertake any of the

interviews, and received anonymised interview transcripts.

Semi-structured interviews explore peoples’ motivations and actions (Taylor and Bogdan,

1998). Focus groups can yield insight not otherwise accessible in structured interviews as

the participants can be prompted by the contributions of others (Cohen et al., 2001).

However, focus group interviews must be interpreted in terms of the group dynamics and the

impact of other participants on individual responses (Taylor and Bogdan, 1998). Interview

transcripts interviews were coded and analysed to identify several key factors that influenced

students’ experiences and perspectives of the flipped classroom approach.

3.1. Qualitative Findings

Students were predominantly positive towards the flipped classroom experience. However,

one student was notably negative. The outcomes showed that student perceptions fell into

two broad categories (1) teaching and (2) learning. Each category exhibited a number of

themes which are summarized below. Factors, such as student responsibility for time

management, or working collaboratively were perceived as both positively and negatively.

Perspectives of Teaching

• Course Materials

• The Process

• Lecturer Approach

• Preparation of Student

Perspectives of Learning

• Reviewability

• Independence

• Time Management

• Working Collaboratively

3.1.1. Students’ Perspectives of Teaching

Some students noted that course materials were easily accessed via the designated Online

Learning Environment (based on the Moodle platform) and fitted well into their schedules.

However, nearly all students experienced frustration due to errors in the video material.

Students accepted that mistakes would be time consuming to fix; while others offered a

solution:

“Have you guys watched The Khan Academy? so… [the lecturer] was

complaining that if he has an error in his recordings then he has to start from

scratch. He was having editing issues, but what Sal [in the Khan videos] does he

just puts a little text box at the bottom and says, hey look, this is wrong and this is

what I meant to say. It would have reduced the time and effort that we spend

quite dramatically” (Student D).

There was also a recommendation that the videos should be developed by people who

specialise in video production.

Students felt empowered by the emphasis on problem solving in workshops. Many students

would have preferred earlier access to lecture material. Similarly, the worked examples to be

undertaken in workshops were requested in advance: “Maybe write a list of questions out

and then you know what you will ask at the next tutorial. Just keeping up-to-date” (Student

N). However, the lecturer noted that providing such examples in advance would provide little

benefit to the students. In particular, there are many examples in the video lectures that

students could engage with prior to the flipped classroom, and providing examples prior to

the worked session would mitigate the lecturer’s ability to contribute to student learning in

real-time. Others mentioned that they would prefer a variety of materials for asynchronous

learning. One student suggested that an additional online could improve peer-to-peer

learning. Furthermore, some students experienced frustration with duration of the videos and

the time taken to watch “for me it was at least three hours in the evening after I came home

from work…you finish that and you are exhausted” [Student A].

Most students indicated that they experienced increased engagement in the course due to

the flipped approach.

“When you’re taking notes in real time lectures you focus more on taking notes

and you don’t have time to digest the information and they find it quite useful; that

[in the flipped classroom] they could pause and think about what they just wrote

and the content sunk in better” (Student O).

Over half of the students mentioned that they like the frequency of the tutorials, which were

held three times a week over the four week section of the course. A number of students

mentioned the approach forced their engagement in the course. Most students considered

the increased engagement a positive aspect of the approach; “compared to my first year

grade I was normally [understanding course content] on a higher level so yeah I was more

engaged, I was willing to work harder” (Student F). Another student suggested that while the

course took more time than the traditional approach, she knew the material better and

therefore required less effort to prepare for the end of year examination. Perhaps

surprisingly, it was only a minority of students that mentioned the increased time

commitment of the approach as a negative. Some noted that using the flipped approach for

more than one course per semester would be challenging.

Three students noted improved academic achievement due to the approach: “I’d say just

that the teacher and the way he taught was very effective and reflected in my results”

(Student L). Nearly all students were positive about the collaborative nature of the

workshops which provided unique opportunities to interact with the lecturer on a one-to-one

basis. They noted that the small class sizes and the resulting group dynamics, “We were

sitting in groups and the girls in my group really helped me out” (Student F), “you kind of

need someone who learns with you” (Student F).

Students in the study were very positive about the lecturers approach during the course and

noted the increased engagement during the flipped classroom; “I’d say just the teacher and

the way he taught it was very effective, and that reflected in my results” (Student L).

Students valued the direct engagement with the lecturer during the workshop sessions: “[It

was] easier to get that one-on-one time, easier to get the help you needed” (Student M). Two

students specifically noted their appreciation for the lecturer’s time investment in making the

videos. Some students suggested that the lecturer utilise both his own material and other

readily available resources. In particular, the Khan Academy was mentioned several times:

“Lecturers tend not to use that resource [internet], they don’t give you links, they don’t

engage with the thing” (Student A).

Several students felt that they should have been better prepared for the process. In

particular, they noted that the approach required an entirely different approach to learning.

“You need to purchase that approach with a different approach as well as think about it. You

need to explain that material to a person who doesn’t really understand. You need to guide

them through several courses” (Student A).

Overall, a majority of students preferred the flipped method of teaching over traditional

lectures.

3.1.2. Students’ Perspectives of Learning

One major noted benefit of the approach was the ability to view and re-engage with course

content until full comprehension of the threshold subjects. “I really enjoyed it coz I was able

to pause it if I didn’t know something, pause it, google it, pause it so something else, come

back to it and it didn’t feel like you had to stay there the whole time” (Student I). Most

students appreciated that workshops followed immediately after the associated lectures.

Students indicated that the increased independence of the approach and gained a sense of

empowerment from the process:

“The flipped classroom is very self-directed learning. You have to sit down at

home, you have to watch the lecture, you have to take notes by yourself, there is

no one telling you that you have to be there in class, you can do whatever you

want, you don’t have to go to the tutorial, you go there if you want help, if you

want to push yourself and you want to learn more … but I think I really benefitted

from it in the end. I really enjoyed the style and gained a lot more from, you get

the materials beforehand because I think it gave me time to process it, digest it,

to understand what I was looking at before I actually went in a did it” (Student M).

A majority of the students liked the self-paced nature of the course. However, a few felt that

self-paced learning disadvantaged them: “So it forces me to invest more time than I want to,

than I can” (Student A). Two students noted that the approach does not prepare students for

the schedules expected in professional practice. In particular, they noted that professionals

typically do not have flexible timeframes to prepare at their own pace before they solve real-

life problems. “For quite a lot of jobs that you’re going to get coming out of a degree like this,

you don’t wake up whenever you want and go to work” (student J).

Nearly all students noted a need for self-responsibility and discipline in a flipped classroom:

“It takes a lot of self-control” (Student C). While some students thought the self-responsibility

was advantageous, others recognised their own immaturity meant that they were not as

successful as they had hoped in the teaching approach. A few students indicated that the

flipped classroom approach is generally most suitable for students with a high level of self-

discipline “I wouldn’t have been able to do it when I was 18. There wouldn’t have been a

chance” (Student P).

Students recognised the importance of time management in the flipped approach. A number

of students found that the online lectures took too long a time to watch and the flipped

approach generally took considerably more time than the traditional approach: “At some

point I was struggling to keep up with watching the videos … also a bad thing because you

spent a lot more time going over it” (Student L). A number of students mentioned the need to

engage in the prescribed course material in a timely matter: “the moment you missed one

and you go to the tutorial and have no idea what’s going on and then you kind of not waste

that tutorial but you can’t fully engage in it so you have to re-watch it that night to catch-up”

(Student O). Another student felt that the increased time investment was not reflected in

improved grades.

A majority of the students liked the collaborative nature of the workshops and enjoyed the

resulting group dynamics. Most also appreciated the small class-sizes and increased

lecturer interaction compared to traditional classes “I definitely feel that working with a

partner or small group really helped. As soon as I hit a roadblock or something that I didn’t

understand, I could bounce ideas off [student name] and he could do the same for me”

(Student D). Two students indicated that they felt responsibility to the group and thus,

ensured that they were prepared to make a contribution at the workshops. One student

mentioned watching and discussing the online material with a classmate.

“I was doing it with a very good friend of mine. We spent a lot of time we’d watch

them together and we’d hit a problem and we’d both work through what [the

lecturer] was trying to get towards, and the next day we’d be very confident in

what we’d got” (Student D).

However, other students mentioned that they would not like to watch the video with peers.

3.2. Discussion of Qualitative Outcomes

This study concurs with pertinent literature on student perspectives of the flipped classroom

(Hanson, 2016, Nguyen et al., 2015) and lecturer behavioural factors (Blair et al., 2016). This

section will discuss our findings in the context of prior research.

One rationale for adoption of flipped classrooms in tertiary settings recognises trends

towards student-centred, problem based learning. This trend conforms to Vygotsky’s (1978)

sociocultural theory. Like the students in Nguyen et. al’s (2015) study, students in this cohort

noted the positive aspects of working collaboratively. Nguyen et al. also commented that the

group dynamics developed during the workshop caused students to study harder to ensure

they can contribute to group discussions. Students indicated that the flipped approach took

more time to engage with. However, the reviewability of the materials and the collaborative

nature of the workshops enabled a greater understanding of the material. Working

collaboratively typically generates a range of viewpoints and conforms to the sociocultural

conflict theory (Doise and Mugny, 1984). Social conflict theory states that all learners’

cognitive development, regardless of academic ability is benefitted by understanding

conflicting views of peers. A number of students also mentioned the value of collaborative

work when engaging with course materials and in workshops. This aligns with the findings of

Mayer’s study that suggested increased value of face-to-face interaction both with the

lecturer and peers supports the apprenticeship of thinking theory (Bishop and Verleger,

2013, Lavelle et al., 2013, Doise and Mugny, 1984).

Another advantage of the flipped approach observed in this study and in other studies was

that students could adjust the course content delivery rate and reviewed at the students’

discretion (Blair et al., 2016, Johnson and Renner, 2012, Lavelle et al., 2013). These

findings support the success and importance of student-centred learning theory (Vygotsky,

1978). However, some students in this study felt that the flipped approach took more time

than other courses and indicated that undertaking too many courses at one time using the

flipped approach would be arduous. Some students in this study acknowledged that

although the course took a lot of time, they made efficient use of the time. Hence, they were

able to decrease time in exam study.

Hanson (2016) suggested that students should be prepared for the flipped classroom

approach. This study determined that students declared a desire to have the philosophy and

pedagogy underpinning the flipped approach explained to them prior to engagement of the

process. Negative elements of the flipped approach that emerged in this study (such as lack

of time management, self-discipline and inability to work collaboratively) were observed

previously (Johnson and Renner, 2012, Lavelle et al., 2013, Mayer, 2002).

The quality of the lecturer developed materials for this cohort impacted students’

perspectives of the flipped approach and was also found by Blair et al. (2016). The video

materials developed for this course had a number of mathematical and formulaic errors that

consumed a lot of time for some students. While students understood the difficulty in fixing

the errors in on-line materials, some indicated that the errors damaged their confidence.

However, others saw the mistakes as an advantage as they were forced to critically think

about the materials and the differences between their ideas and those presented. Thus, the

videos provided contrasting views and consolidated the ideas of the students (Doise and

Mugny, 1984). The collaborative nature of the workshop sessions were very well received by

students. In workshop sessions, students received immediate feedback on their level of

comprehension (Choi, 2013, Maykut and Morehouse, 1994).

4. Quantitative Methodology

A quantitative analysis was undertaken to determine whether the flipped classroom had a

significant impact on academic performance. In order to determine the ongoing academic

success of individuals from a flipped cohort, the performance of the students in a

subsequent-year dynamics paper was considered. In particular, the grades of students that

undertook the flipped classroom of the foundational paper was contrasted to the grades of

students that undertook the same class in a typical, traditional delivery.

Those recruited into the mechanical or mechatronics program undertook the second-year

dynamics paper in their first professional year (or second-year of study). There were 188

students enrolled in the second-year dynamics class. Prerequisite knowledge for the

second-year dynamics class was predominantly covered in a first-year mechanics class of

which there are two evenly divided parts: a statics section and a dynamics section. The

dynamics section was taught by Dr Paul Docherty (author).

Most of the students that enrolled in the second-year dynamics course undertook a first-year

mechanics class during the prior year (N=170). Eleven students used the direct entry (DE)

route into the professional engineering programme, one was an international exchange

student, and six had failed the second-year class in a prior attempt and were resitting the

class. Since the validity of including the individuals that did not take the first year mechanics

course in the year prior is statistically ambiguous, these eighteen DE, exchange, and

repeating students were omitted from the statistical analysis. Figure 1 shows the recruitment

pathways of the students into the second-year dynamics class.

Most students in the second-year dynamics class were successful in their first attempt of the

first-year mechanics course during the previous year. The Semester 2 occurrence of the

first-year mechanics course was taught using traditional lectures. Students in the summer

cohort typically either failed the semester 2 occurrence (N=10) or were required to undertake

remedial classes in their first year due to a lack of prerequisite high-school qualifications for

first year courses. The dynamics section of the summer occurrence was taught using the

flipped classroom delivery method.

Figure 1. Pathways of students into the second-year dynamics class. (*Three of 13 were

also enrolled in the traditional mechanics class)

4.1.1. Content of Second-year Dynamics Course

The second-year dynamics course taught advanced kinematics and dynamics of machine

elements, vibrations and an introduction to numerical (computational) methods to evaluate

differential equations. Hence, the dynamics section of the first-year mechanics class formed

a foundation for successful participation in the second-year class. Formative assessment of

the second year dynamics paper included weekly homework, a test, and a project. A two-

hour exam formed the summative assessment. The second year dynamics class was taught

using a traditional lecturing style. The lecturers of both the second year dynamics and first

year mechanics papers had autonomous control over the teaching style of the classes and

did not influence the outcomes of the others classes. In addition, the second-year lecturer

was agnostic to the recruitment path of specific students. Hence, there was no potential for

conscious or subconscious bias in grading based on whether the student underwent the

flipped or traditional content delivery in the previous year. Hence, the second year lecturer

was an independent assessor of the efficacy of the flipped environment for the retention of

foundational dynamics concepts.

4.1.2. Analysis Methodology

The final exam marks and the aggregate marks from the second-year dynamics class were

analysed and contrasted across students that undertook the flipped or traditional versions of

the first year class. The traditional and flipped cohorts achieved significantly different grades

200 level dynamics

N=188

100 level mechanics

(flipped) N=53

100 level mechanics (traditional)

N=432

Other

157 18

10

Analysis N=155 traditional

N=13 flipped

13*

18 ineligible 2 dropped out

in the first year, they were poorly matched cohorts and thus violated assumptions of

hypothesis testing. The average academic preparedness for those in the flipped cohort was

less than the preparedness of the traditional cohort. In particular, the average intermediate

GPA for those in the flipped environment was significantly lower than the average

intermediate GPA from those in the traditional occurrence (Table 1). In order to achieve

statistical equivalence, the cohorts were statistically normalized. First order linear regression

using least squares estimation was used to determine the relationship between the second-

year performance and aggregated first-year grade point averages (GPA) across both

cohorts. The second-year outcomes were normalized by subtracting the expected outcome

as determined by their intermediate GPA and the linear regression. This eliminated the bias

introduced by the lower university preparedness of the flipped cohort. Since the data groups

had uneven numbers, was unpaired and not normally distributed, the Wilcoxon rank sum

and Kolmogorov-Smirnov statistical tests were used. Rank sum is a comparison of

magnitude and the Kolmogorov Smirnov test measures the significance of across

distribution. A significance level of 5% was used.

4.2. Quantitative Results

Overall, there were no statistically significant distinctions between the performances of the

students of the flipped classroom and the traditional classroom in the follow-up class. Figure

2 shows the relationship between student outcomes from a second-year dynamic course

correlated to the intermediate GPAs for the class. There was a reasonably high correlation

(R=0.691) between incoming GPA and the second- year aggregated course mark of the

cohort. A moderate correlation (R=0.506) was observed between the second-year dynamics

exam marks and the incoming GPA. Note that there is only one regression line per analysis

in Figure 2. This was necessary to allow the corrected marks to yield the appropriate

distinction across the traditional and flipped cohort outcomes.

Figure 2 Regression and Pearson correlation between exam mark and intermediate GPA

(left) and the correlation between aggregated course score and intermediate GPA (right).

The flipped and traditional cohort are shown as ‘o’ and ‘+’ symbols, respectively.

The cumulative distribution functions of the second year course outcomes for the two

cohorts are shown in Figure 3. The rank sum test measures the significance of

nonparametric bias across the samples, and the Kolmogorov Smirnov test determines the

significance in differences in the global distribution of the curves. Table 1 summarizes the

outcome statistics across the cohorts.

Figure 3 Cumulative distribution functions for second-year dynamics outcomes across those

that undertook the first-year mechanics class using the flipped and traditional delivery. (A)

uncorrected exam performance. (B) corrected exam performance. (C) uncorrected course

mark. (D) corrected course mark

Table 1 shows a lack of distinction in the second year grades across the cohorts.

Furthermore, the most significant distinction across cohorts was in the incoming GPA. This

confirms the need for correction of grades. Overall, there were no significant changes across

the cohorts when the grades were corrected. However, this does not imply the flipped

classroom did not affect grades. In contrast, this outcome shows that the changes, assuming

that there are changes, were undetectable in the cohorts of this study.

TABLE 1. summary statistics

Traditional Median (Q1, Q3)

Flipped Median (Q1, Q3)

Rank sum p-value

Kolmogorov-Smirnov p-value

Intermediate GPA

5.5 (4.27, 7.00)

3.9 (3.10, 5.12)

0.012* 0.022*

Uncorrected exam mark

47.5 (39.5, 55.4)

43 (35.25, 51.6)

0.21 0.55

Corrected exam mark

0.77 (-5.90, 5.33)

0.99 (-3.26, 7.72)

0.65 0.82

Uncorrected course mark

117.5 (102, 129.6)

106 (89.9, 115.1)

0.031* 0.12

Corrected course mark

0.59 (-6.58, 9.14)

0.82 (-6.87, 5.93)

0.85 0.96

* denotes a statistically significant difference across cohorts – p<0.05

4.3. Discussion of Quantitative Outcomes

This analysis shows that the flipped classroom had no observable positive or negative effect

on the ongoing student performance in follow-up classes. It does confirm though that the

flipped approach in a summer cohort did manage to provide students with the foundational

skills to succeed in the following year. Figures 3B and 3D show that the corrected grades

were indistinguishable. Table 1 shows that it is impossible to declare significance in any

tested metric other than the uncorrected course marks (ranksum p=0.031). However, the

use of uncorrected course marks is statistically dubious. The observed differences in the

CDF curves shown in Figures 3A, 3B, and 3D were too small to be statistically significant.

Hence, while the teaching methodology is most likely to have had a small effect on learning

and retention of dynamics, the effect was too small to capture in this present cohort.

This small effect indicates that a prospective analysis using follow-on courses as a

benchmark may be difficult. Assuming a 5% contrast with a 20% variance in the

performance across cohorts in follow-on performance in the follow-up course would require a

study with approximately 250 students in each arm to achieve an 80% likelihood of achieving

statistical significance [4]. Such numbers are infeasible for prospective analysis at mid-sized

engineering faculties.

Educational research is used to inform decisions on educational policy change. Hence, it is

critical that all research outcomes are reported in the academic literature. In this case, the

outcome indicated that the flipped classroom provided no discernible ongoing benefit or

detriment in academic performance in a follow-on course. This outcome contradicts positive

outcomes previously published (O'Flaherty and Phillips, 2015a, Kerr, 2015). However, to

avoid contributing to the publication bias that is known to affect educational research, studies

with neutral results must be reported (Dickersin, 1990, Pigott et al., 2013, van Hilten, 2015).

The use of grades attained in a follow-on course to determine the efficacy of the flipped

teaching methodology enabled a fully independent analysis. In contrast, directly measuring

course marks across first-year traditional and flipped cohorts would not be researcher

independent. The independence of the second-year examiner provides validity to this

analysis. Furthermore, the most important factor for determining the academic success of a

foundational course is ongoing academic success of its students. Hence, this analysis

directly targeted the primary focus of educational policy makers.

5. Recommendations

When considering implementation of the flipped approach, the broader, total student

experience needs to be considered. Students should be offered a balanced varied

programme, using a range of best practice delivery methods that have been designed to

cater for a variety of needs. This study identified a number of recommendations aimed at

improving students’ experience in the flipped classroom. These recommendations, listed

below are divided into two categories: recommendations for lecturers and recommendations

for students.

Recommendations for Lecturers

• Balance of Delivery

• Lecturer Approach

• Quality and Variety of Materials

• Preparation of Students

• Workshop Materials in Advance

• Prepare students for methodology

Recommendations for Students

• Student Preparedness

• Student Engagement

• Time Management

5.1. Recommendations for lecturers

5.1.1. Materials development and provision

Varied and balanced materials should be presented to students using a range of strategies

to ensure that students remain engaged in the course. In particular, strategies could include:

lecturer developed videos, use of existing materials available on the internet (such as Khan

Academy), online forum discussions to enhance understanding in a collaborative way, post

video quizzes and authentic collaborative problem solving and other innovative ICT tools as

they emerge.

5.1.2. Lecturer attitude

A positive lecturer approach to students’ potential to learn is essential. Understanding that

the lecturers approach to their students learning and the material they are teaching has a

huge impact on ongoing student achievement. If all students have the potential to learn then

it stands to reason that lecturer approach and attitude are the main contributing factors in

students’ achievement.

5.1.3. Quality of materials

Students need to have confidence that the course materials are of a high quality, succinct

and engaging and without any errors. Students in this study suggested professional video

editors should have been consulted during development of the course materials

5.1.4. Availability and accessibility of content

Students recommended that all workshop and video materials be available to the students

well in advance so that they could better manage their time and self-paced learning. Along

with this we would recommend a very clear schedule of the face-to-face workshops and with

a detailed outline of content so that students are adequately prepared for each workshop.

5.1.5. Student preparation

Students need to be well prepared for the flipped approach. Most students have never

experienced similar teaching approaches in the past. Hence, the process, its rationale,

philosophy, advantages and inherent risks should be clearly explained to all participants

prior to engagement. It should not be assumed that students have the maturity to effectively

manage their time for effective self-paced learning. Opportunities for frequent student

feedback on the effectiveness of the approach will also provide valuable feedback to the

lecturer with regards to student readiness to learn in a flipped classroom, as well as areas

that need particular attention.

5.2. Recommendations for students

5.2.1. Student preparedness

Students themselves need to be prepared for the flipped classroom. Students in this study

recognised the considerable difference to other approaches, and the need to adapt. The

students need to take ownership of their learning and commit themselves to taking on board

strategies to succeed in the alternative flipped approach.

5.2.2. Receptivity to approach

Students’ receptiveness to the alternative strategy and active engagement with peers and

lecturer is critical to efficient learning in the flipped classroom. Students recognised the

benefit of interacting with the lecturer and peers in the workshop and in some cases, while

engaging with the videos.

5.2.3. Time management

Flipped classroom students must develop time management strategies and be self-

disciplined to ensure materials are engaged with in a timely manner. Failure to engage with

course material in a timely manner renders the help sessions useless. When non-attendance

is ongoing and the student becomes further behind, the flipped classroom can lead to a

compounding problem for the student to catch up.

6. Impacts of research

6.1. Impact on Learners

6.1.1. Retrospective

The qualitative analysis of this research indicated that the application of the flipped

classroom had significant benefit in terms of student experience. However, the quantitative

analysis showed that the approach had little ongoing effect on the academic performance of

the students. These outcomes confirm the efficacy of the approach, in particular, for

enhancing student experience.

6.1.2. Prospective

This research has shown that there is benefit in the flipped approach in the summer cohort.

Hence, the approach will continue to be used in future iterations of the class. Furthermore,

the recommendations of this study have led to small scale uptake of the approach in other

engineering classes at the University of Canterbury. In particular, there has been a recent

emergence of flipped approaches for small modules within courses. This limited uptake is

partially due to the considerable time cost of producing online materials, and the time burden

of learning in a flipped environment.

6.2. Impact on the teacher

6.2.1. Retrospective

The lecturer of the flipped classroom has a more in depth understanding of pedagogy as a

result of facilitating the flipped classroom. Ultimately, it has provided an ongoing resource for

teaching and learning that becomes increasingly efficient in terms of lecturer effort in each

iteration of the class.

6.2.2. Prospective

The critical analysis of the approach made available by this research will lead to

improvements in the facilitation of the course. In particular, the lecturer is more empowered

to provide the students with detailed recommendations for successful engagement with the

methodology.

6.3. Impact on institution

6.3.1. Retrospective

The University of Canterbury utilises best practice teaching approaches. However, the

university must be cautious when implementing new approaches for teaching on a wide

scale. While the academic literature almost universally indicates positive outcomes from

flipped approaches the potential for publication bias remains an ongoing concern. The use of

flipped approaches has been touted as a possible strategy for teaching into the engineering

program and the course analysed here was one of the pilot courses of the approach.

6.3.2. Prospective

The outcomes of this research implied that the ubiquity of positive quantitative outcomes in

the academic literature regarding flipped classrooms may indeed be enforced by publication

bias. However, this research also confirmed the significant benefit in terms of the student

experience. Ultimately, students are very mobile, and often have the choice to attend a

variety of tertiary institutions. Hence, there will be an increasing pressure on universities to

provide teaching styles that are preferred by students.

6.4. Impact on Research Team

6.4.1. Retrospective

The research team was brought together for this research and was an entirely new

collaboration. The skills of the team members were very distinct and cross pollenated to

yield a much more comprehensive evaluation of the flipped classroom than any one

researcher in the team would be able to produce. In particular, Dr Docherty was immersed in

qualitative research and thus developed experience and expertise in a research

methodology he has not previously been involved with. Ms Zaka and Dr Fox-Turnbull

developed quantitative skills and experience working with engineering students who

represented a very different culture to students the researchers have been involved with in

the past.

6.4.2. Prospective

This research has provided the research team with the skills, experience, and appetite to

undertake ongoing research into educational practice. Future research will involve ongoing

development and improvements to the flipped classroom in the first year dynamics class.

The researchers will also monitor and support the emergence of the flipped approach within

the College of Engineering at the University of Canterbury. The outcomes of this study will

be published and presented to disseminate the outcomes to the wider academic community.

7. Conclusions

The study aimed to investigate student experience and ongoing success after a flipped

classroom approach to learning dynamics in an undergraduate engineering programme. The

study determined that students were generally positive about the approach. Students’

perceptions were divided into two categories. Perceptions of teaching, and perceptions of

learning.

This analysis also assessed the ongoing academic performance of students from a flipped

classroom in a follow-on course. The findings indicated that the performance of students

who undertook the foundational class using a flipped teaching approach achieved similar

academic outcomes in the follow-on course to those that undertook the foundational class

using traditional methods. While ongoing academic performance is a very important metric

for assessing educational success of foundational classes, the student impression and

experience is also critically important to consider during the design of educational policy.

The study led to a series of recommendations for future implementations of the approach in

tertiary teaching of engineering.

8. References

BAKER, J. W. 2000. The “classroom flip”: Using web course management tools to become a

guide by the side. The 11th international conference on college teaching and learning,

Jacksonville, FL.

BISHOP, J. L. and VERLEGER, M. A. 2013. The flipped classroom: a journey of research.

120th ASEE Annual Conference and Exposition. American Society for Engineering

Education.

BLAIR, E., MAHARAJ, C. and PRIMUS, S. 2016. Performance and perception in the flipped

classroom. Education and Information Technologies, 21, 1465–1482.

CHOI, E. M. 2013. Applying inverted classroom to software engineering education.

International Journal of e-Education, e-Business, e-Management and e-Learning, 3, 121-

125.

CLARKE, S. 2008. Active learning through formative assessment, London, Hodder

Education.

COHEN, L., MANION, L. and MORRISON, K. 2001. Research methods in education,

London, Rutledge Falmer.

COHEN, L., MANION, L. and MORRISON, K. 2013. Research methods in education,

London, Routledge.

DANIELS, H. 1996. An introduction to Vygotsky, London, Routledge.

DICKERSIN, K. 1990. The existence of publication bias and risk factors for its occurrence.

JAMA, 263, 1385-1389.

DOISE, W. and MUGNY, G. 1984. The Social Development of Intellect, Oxford Pergamon

Press.

EBLE, K. E. and MCKEACHIE, W. J. 1985. Improving Undergraduate Education through

Faculty Development. An Analysis of Effective Programs and Practices, ERIC.

GANNOD, G. C., BURGE, J. E. and HELMICK, M. T. Using the inverted classroom to teach

software engineering. ICSE '08, May 10 - 18 2008 Leipzig, Germany. ACM New York, NY,

USA, 777-786.

HANSON, J. 2016. Surveying the experiences and perceptions of undergraduate nursing

students of flipped classroom approach to increase understanding of drug science and its

application to clinical practice. Nurse Education in Practice 16, 79-85.

HOFSTEDE, G. and BOND, M. H. 1984. Hofstede's Culture Dimensions: An Independent

Validation Using Rokeach's Value Survey. Journal of Cross-Cultural Psychology, 15, 417-

433.

HOLLANDER, J. A. 2002. Learning to discuss: Strategies for improving the quality of class

discussion. Teaching Sociology, 317-327.

JOHNSON, L. W. and RENNER, J. D. 2012. Effect of the flipped classroom model on a

secondary computer applications course: student and teacher perceptions, questions and

student achievement. EdD Dissertation, University of Louisville.

KERR, B. The flipped classroom in engineering education: A survey of the research. 2015

International Conference on Interactive Collaborative Learning (ICL), 2015 2015 Florence,

Italy. IEEE, 815-818.

KHOO, E. G., COWIE, B. and PETER, M. 2015. Re-engineering an engineering course:

How flipped classrooms afford transformative teaching, learning, and workplace

competency.

LAVELLE, J. P., STIMPSON, M. T. and BRILL, E. D. Flipped Out Engineering Economy:

Converting a Traditional Class to an Inverted Model. In: CHAN, A. K. W. K. V., ed. Industrial

and Systems Engineering Research Conference 2013. 397-406.

LOVE, B., HODGE, A., GRANDGENETT, N. and SWIFT, A. W. 2014. Student learning and

perceptions in a flipped linear algebra course. International Journal of Mathematical

Education in Science and Technology, 45, 317-324.

MAGNUSON-MARTINSON, S. 1995. Classroom computerization: Ambivalent attitudes and

ambiguous outcomes. Teaching Sociology, 1-7.

MASON, G. S., SHUMAN, T. R. and COOK, K. E. 2013. Comparing the Effectiveness of an

Inverted Classroom to a Traditional Classroom in an Upper-Division Engineering Course.

IEEE Transactions on Education, 56, 430-435.

MAYER, R. E. 2002. Rote versus meaningful learning. Theory Into Practice, 41, 226-232.

MAYKUT, P. and MOREHOUSE, R. 1994. Beginning qualitative research: a philosophic and

practical guide, London, The Falmer Press.

NEUMAN, L. W. 2000. Social research methods: qualitative and quantitative approaches,

Needham Heights, Allyn and Bacon.

NGUYEN, B., YU, X., JAPUTRA, A. and CHEN C-H. S. 2015. Reverse teaching: Exploring

student perceptions of “flip teaching”. Active Learning in Higher Education, 17, 51-61.

O'FLAHERTY, J. and PHILLIPS, C. 2015a. The use of flipped classrooms in higher

education: A scoping review. Internet and Higher Education, 25, 85-95.

O'FLAHERTY, J. and PHILLIPS, C. 2015b. The use of flipped classrooms in higher

education: A scoping review. Internet and Higher Education, 25, 85-95.

PIERCE, R. and FOX, J. 2012. Vodcasts and active-learning exercises in a "flipped

classroom" model of a renal pharmacotherapy module. American Journal of Pharmaceutical

Education, 76, 1-5.

PIGOTT, T. D., VALENTINE, J. C., POLANIN, J. R., WILLIAMS, R. T. and CANADA, D. D.

2013. Outcome-Reporting Bias in Education Research. Educational Researcher.

RICHARDSON, K. 1998. Models of cognitive development, Hove, U.K., Psychology Press

Ltd.

SELVIN, H. C. and STUART, A. 1966. Data-Dredging Procedures in Survey Analysis. The

American Statistician, 20, 20-23.

TAYLOR, J. T. and BOGDAN, R. 1998. Introduction to qualitative research methods: a

guidebook and resource, New York, John Wiley and Sons Inc.

VAN HILTEN, L. G. 2015. Why it's time to publish research “failures” [Online]. [Accessed

8/3/2016].

VYGOTSKY, L. S. 1978. Mind in society: the development of higher psychological

processes. In: COLE, M., JOHN-STEINER, V. and SOUBERMAN, E. (eds.). London:

Harvard University Press.

WILSON, H. The use of Flipped Learning in an Engineering Technician Management

Course. In: AAEE, ed. Proceedings of the 2013 AAEE Conference, 2013 Gold Coast,

Queensland, Australia.

YELAMANTHI, K. and DRAKE, E. 2015. A flipped first-year digital circuits course for

Engineering and Technology students. IEEE Transactions on Education, 48, 179-186.

ZHU, Y., WING, A. and YATES, G. 2016. University students' self-control and self-regulated

learning in a blended course. Internet and Higher Education, 30, 54-62.