Effectiveness of flipped classrooms in Chinese baccalaureate nursing education… · 2018-12-15 · 2 Abstract Background: In recent years, the flipped classroom approach has been

Accepted Manuscript

Title: Effectiveness of flipped classrooms in Chinesebaccalaureate nursing education: A meta-analysis ofrandomized controlled trials

Authors: Rujun Hu, Huiming Gao, Yansheng Ye, Zhihong Ni,Ning Jiang, Xiaolian Jiang

PII: S0020-7489(17)30274-2DOI: https://doi.org/10.1016/j.ijnurstu.2017.11.012Reference: NS 3062

To appear in:

Received date: 7-8-2017Revised date: 28-11-2017Accepted date: 28-11-2017

Please cite this article as: Hu, Rujun, Gao, Huiming, Ye, Yansheng, Ni, Zhihong, Jiang,Ning, Jiang, Xiaolian, Effectiveness of flipped classrooms in Chinese baccalaureatenursing education: A meta-analysis of randomized controlled trials.InternationalJournal of Nursing Studies https://doi.org/10.1016/j.ijnurstu.2017.11.012

This is a PDF file of an unedited manuscript that has been accepted for publication.As a service to our customers we are providing this early version of the manuscript.The manuscript will undergo copyediting, typesetting, and review of the resulting proofbefore it is published in its final form. Please note that during the production processerrors may be discovered which could affect the content, and all legal disclaimers thatapply to the journal pertain.

https://doi.org/10.1016/j.ijnurstu.2017.11.012

https://doi.org/10.1016/j.ijnurstu.2017.11.012

1

Effectiveness of flipped classrooms in Chinese baccalaureate

nursing education: A meta-analysis of randomized

controlled trials

Rujun Hu, Huiming Gao, Yansheng Ye, Zhihong Ni, Ning Jiang, Xiaolian Jiang*

Author information:

Rujun Hu, PhD Candicate

West China Hospital/West China School of Nursing, Sichuan University, Chengdu

610041, Sichuan, China.

Huiming Gao, M.S.

School of Nursing, Zunyi Medical College, Zunyi 563000, Guizhou, China.

Yansheng Ye, PhD Candicate



Zhihong Ni, PhD Candicate



Ning Jiang, PhD Candicate



Xiaolian Jiang, PhD, Professor



Correspondence:

Xiaolian Jiang, PhD, Professor, West China Hospital/West China School of Nursing,

Sichuan University, Chengdu 610041, Sichuan, China.

Tel: +86 028 85422070; fax: +86 028 85422070.

Email address: [email protected]；[email protected]

ACCEPTED MANUSCRIP

T

2

Abstract

Background: In recent years, the flipped classroom approach has been broadly

applied to nursing courses in China. However, a systematic and quantitative

assessment of the outcomes of this approach has not been conducted.

Objective: The purpose of the meta-analysis is to evaluate the effectiveness of the

flipped classroom pedagogy in Chinese baccalaureate nursing education.

Design: Meta-analysis of randomized controlled studies.

Data source: All randomized controlled trials relevant to the use of flipped

classrooms in Chinese nursing education were retrieved from the following databases

from their date of inception through September 23, 2017: PubMed, EMBASE, the

Cochrane Central Register of Controlled Trials, CINAHL, the China National

Knowledge Infrastructure, the Wanfang Database, and the Chinese Scientific Journals

Database. Search terms including “flipp*”, “inverted”, “classroom”, and “nurs*” were

used to identify potential studies. We also manually searched the reference lists of the

retrieved articles to identify potentially relevant studies.

Review Methods: Two reviewers independently assessed the eligibility of each study

and extracted the data. The Cochrane risk-of-bias tool was used to evaluate the quality

of the studies. RevMan (Version 5.3) was used to analyze the data. Theoretical

knowledge scores and skill scores (continuous data) were synthesized using the

standardized mean difference (SMD) and 95% confidence interval (CI). The statistical

heterogeneity of the included studies was analyzed by calculating the I2 statistic and

applying a chi-square test. Publication bias was assessed by funnel plots. The quality

ACCEPTED MANUSCRIP

T

3

of the combined results was evaluated using the Grading of Recommendations

Assessment, Development and Evaluation system.

Results: Eleven randomized controlled trials published between 2015 and 2017 were

selected. All the included studies had a moderate possibility of bias due to low

methodological quality. The meta-analysis indicated that the theoretical knowledge

scores and skill scores were significantly higher in the flipped classroom group than

in the traditional lectures group (SMD=1.06, 95% CI: 0.70-1.41, P<0.001, and

SMD=1.40, 95% CI: 0.46-2.34, P<0.001). There was no significant publication bias

indicated in the primary analysis. Sensitivity analysis showed that the results of our

meta-analysis were reliable. The evidence grades of the results regarding the

theoretical knowledge and skill scores were low and very low, respectively.

Conclusion: Flipped classroom pedagogy is more effective than traditional lectures at

improving students’ theoretical knowledge and skill scores. Given the limitations of

the included studies, more robust randomized controlled trials are warranted in a

variety of educational settings to confirm our findings.

Keywords: flipped classroom, nursing, meta-analysis, review

1. Introduction

The flipped classroom is a term that was coined by Bergmann and Sams in 2007,

and it is defined as a teaching method in which “that which is traditionally done in

class is now done at home and that which is traditionally done as homework is now

ACCEPTED MANUSCRIP

T

4

completed in class” (Bergmann and Sams, 2012, Davies et al., 2013). Specifically, in

the flipped classroom model, the content that is traditionally presented in the

classroom setting is instead assigned as homework to be accomplished before class,

while during class time, students engage in active learning, such as case studies,

learning laboratories, games, and simulation-based learning under the guidance of the

teacher (Herreid and Schiller, 2013). Flipped classroom pedagogy aims to help

students transition from passive learning to active, self-directed learning and to

improve their analytical, integration, and critical thinking skills to better address the

complexity of contemporary health care (Hamer, 2000). Because of the characteristics

of this approach, flipped classroom pedagogy has attracted the attention of educators

and has been widely implemented and studied in many disciplines, including nursing

(Galway et al., 2014, Gilboy et al., 2015, Hsu et al., 2016, Lichvar et al., 2016,

Liebert et al., 2016, Mason et al., 2013, Mattis, 2015, McLaughlin et al., 2013,

Mortensen and Nicholson, 2015, Park and Howell, 2015).

However, the implementation of the flipped classroom teaching model in higher

education for nursing remains controversial (Geist et al., 2015, Harrington et al.,

2015, Schwartz, 2014, Simpson and Richards, 2015). Some studies have found that

the flipped classroom model in nursing education presents certain advantages over

traditional lecture-based learning in terms of improving students’ examination scores,

grades and satisfaction (Critz and Knight, 2013, Geist et al., 2015, Missildine et al.,

2013, Yacout and Shosha, 2016). Conversely, some researchers have not found the

flipped classroom model to be superior to traditional models with respect to student

ACCEPTED MANUSCRIP

T

5

examination scores and satisfaction (Geist et al., 2015, Harmon and Hills, 2015,

Harrington et al., 2015, Simpson and Richards, 2015, Tian et al., 2015).

In China, the flipped classroom model has been used in higher nursing education

settings since 2014, and the annual number of published Chinese studies focusing on

the application of flipped classrooms has increased exponentially from 6 items in

2014 to approximately 160 items in 2016. However, a systematic and quantitative

assessment of the outcomes of the flipped classroom teaching model has not been

conducted to date. Considering that different education systems and cultural

differences among different countries may influence the effectiveness of the flipped

classroom method (Frambach et al., 2012, Kim, 2008, Sharma et al., 2014), we

performed a meta-analysis with the target population that was limited to Chinese

baccalaureate nursing students to assess the effects of the flipped classroom model.

The goals were to provide a scientific basis for evaluating the necessity and feasibility

of flipped classroom learning in higher nursing education and to disseminate the

findings more widely to international nursing educators and researchers. In addition,

this research will provide a valuable reference for teaching-learning innovation and

for researchers in different education systems and cultural backgrounds who want to

assess the effectiveness of the flipped classroom model.

2. Methods

The data used in this systematic review and meta-analysis were from previously

published studies; therefore, ethical approval and student consent were not necessary.

2.1 Inclusion and exclusion criteria

ACCEPTED MANUSCRIP

T

6

Two independent raters (RJH and YSY) screened and selected all the studies.

The inclusion criteria were as follows: (1) types of participants: baccalaureate nursing

students in China; (2) types of interventions: using flipped classrooms as the

educational approach; (3) types of controls: traditional face-to-face lecture without

supplementary teaching methods that could affect the results; (4) types of outcomes:

theoretical knowledge scores (scores acquired by theoretical knowledge examination)

and/or skill scores (scores acquired by operation skill examination); (5) types of study

designs: randomized controlled trials (RCTs); and (6) language: English and Chinese.

Studies were excluded if they (1) had incomplete data or (2) were conference

abstracts. Any disagreements between the two raters (RJH and YSY) were resolved

by discussion with a third reviewer (XLJ).

2.2 Search strategy

This study was conducted by following the guidelines in the Preferred Reporting

Items for Systematic Reviews and Meta-Analyses statement (PRISMA) (Moher et al.,

2009). To identify the relevant studies, we searched for publications in the following

databases from their date of inception through September 23, 2017: PubMed,

EMBASE, the Cochrane Central Register of Controlled Trials (CENTRAL),

CINAHL, the China National Knowledge Infrastructure (CNKI), the Wanfang

Database, and the Chinese Scientific Journals Database (VIP). Medical subject

headings (MeSH) or key words including “flipp*”, “inverted”, “classroom”, and

“nurs*” were used to identify potential studies (See Supplementary File 1). We also

ACCEPTED MANUSCRIP

T

7

manually searched the reference lists of the retrieved articles to identify potentially

relevant studies.

All the identified studies, including the titles and abstracts, were downloaded

into Endnote X8 for review. We removed all duplicate studies. The references of the

potential papers were examined to identify any additional papers that met the

inclusion criteria and may have been missed by the search strategy. The titles and/or

abstracts of the retrieved studies from the search strategy and from additional sources

were screened independently by two raters (RJH and YSY) to identify studies that

potentially met the inclusion criteria outlined above. We searched and downloaded the

full papers of potentially eligible studies, which were assessed independently on the

basis of inclusion criteria by the same two raters. Disagreements between the two

raters were resolved by discussion with a third reviewer (XLJ).

2.3 Data extraction and quality assessment

Two raters (NJ and ZHN) used a pre-designed data collection form (Microsoft

Office Excel 2013) to extract all the data independently. The following information

was extracted: first author, publication year, study design, subjects, course type,

participant characteristics, sample size (intervention group/control group), educational

approach in the intervention group, educational approach in the control group,

outcomes, and the duration of study.

Two reviewers (NJ and ZHN) independently evaluated the methodological

quality and risk of bias of each selected study, and disagreements were resolved by

ACCEPTED MANUSCRIP

T

8

discussion with a third reviewer (XLJ). We used the Cochrane risk-of-bias tool to

assess the following items: random sequence generation (selection bias), allocation

concealment (selection), blinding of participants and personnel (performance bias),

blinding of outcome assessment (detection bias), incomplete outcome data (attrition

bias), selective reporting (reporting bias), and other bias. Each domain was

categorized as exerting a low risk of bias (unlikely to seriously alter the results), a

high risk of bias (seriously weakens confidence in the results), or an unclear risk of

bias. The Cochrane Collaboration risk of bias tool is available online at

http://handbook.cochrane.org/.

2.4 Statistical methods

Two reviewers (RJH and HMG) separately input the data, and we used Review

Manager Software 5.3, which was developed by the Cochrane Collaboration, to

conduct the heterogeneity tests and meta-analysis. Before pooling the study results,

the statistical heterogeneity of the included studies was analyzed by calculating the I2

statistic and applying the chi-square test (Higgins et al., 2003). If I2<50% and P>0.10,

we considered the heterogeneity to be low, and a fixed effects model was used to pool

the data. Otherwise, we used a random effects model to summarize the results. If

heterogeneity was present, a sensitivity analysis was performed by excluding the

study with the largest sample size and then recalculating the pooled estimates for the

remaining studies to assess whether this exclusion significantly altered the meta-

analysis results. Continuous data were synthesized using the standardized mean

ACCEPTED MANUSCRIP

T

http://handbook.cochrane/

9

difference (SMD) and 95% confidence interval (CI). A two-sided P<0.05 was

considered to indicate a significant difference in the overall effect. We used funnel

plots to assess the possible publication bias, evidence of asymmetry, and other small

study effects (Hguyatt et al., 2011). In addition, we ranked the evidence quality using

the Grading of Recommendations Assessment, Development and Evaluation

(GRADE) system (https://gradepro.org/).

3.Results

3.1 Literature search results

A total of 840 records from these electronic databases were identified. After the

titles, the abstracts of these studies were reviewed, and 789 studies were excluded

either because they were duplicated (464 studies) or they did not match the inclusion

criteria (325 studies), such as not addressing undergraduate baccalaureate nursing

students in China (186 studies) and non-RCT (139 studies). Then, we further

reviewed the full texts of the remaining 51 studies, 40 of which were excluded

because they did not address undergraduate baccalaureate nursing students in China

(25 studies), they were non-RCT (8 studies), or they had unrelated outcomes (4

studies) or incomplete data (3 studies). Finally, 11 studies were included in the

qualitative synthesis and meta-analysis (Deng, 2016, Ji et al., 2016, Li, 2016, Li et al.,

2017, Liu, 2016, Tao et al., 2016, Tian et al., 2015, Yang, 2016, Zhang et al., 2017,

Zhao et al., 2016, Zhong et al., 2016). The literature screening process and results are

depicted in Fig. 1.

ACCEPTED MANUSCRIP

T

10

3.2 General study characteristics

The 11 included RCTs were all published between 2015 and 2017 in Chinese

journals; all were published in Chinese and were performed in China. The studies

examined the teaching methods in different nursing courses; four pertained to the

fundamentals of nursing, two addressed introductions to nursing, and one of each

involved preventive medicine, emergency and critical care, geriatric nursing, surgical

nursing, and obstetrics and gynecological nursing. The participants in the 11 included

studies were all undergraduate baccalaureate nursing students. The sample size ranged

from 62 to 242 participants, and the pooled sample size was 1484 (742 in the flipped

classroom group and 742 in the control group). All the studies employed the flipped

classroom as the educational approach in the intervention group and traditional

lectures as the approach in the control group. All the study outcomes were measured

using theoretical knowledge and/or skill scores obtained after applying flipped

classroom teaching. The study duration varied from one semester to two semesters.

The characteristics of the included studies are summarized in Table 1.3.3 Risk of bias

in the included studies

We used the Cochrane risk-of-bias tool to assess the risk of bias for each study.

We presented the results using a “risk of bias summary” (Fig. 2). All the studies were

described as “randomized”, but only 4 of them reported the randomization methods

they used. None of the studies described their allocation concealment or blinding of

participants, personnel and outcome data. All the studies clearly reported all the

expected results to limit reporting bias. Moreover, all the studies reported that there

ACCEPTED MANUSCRIP

T

11

were no statistically significant differences in age or sex between the flipped

classroom and control groups at the baseline (P>0.05). Finally, the shape of the funnel

plot for the primary “theoretical knowledge scores” outcome of the 9 studies was

nearly symmetrical (Fig. 3). There was no significant publication bias indicated in the

primary analysis.

ACCEPTED MANUSCRIP

T

12

3.4 Meta-analysis results

3.4.1 Theoretical knowledge scores

Nine studies (Deng, 2016, Li, 2016, Li et al., 2017, Liu, 2016, Tian et al., 2015,

Yang, 2016, Zhang et al., 2017, Zhao et al., 2016, Zhong et al., 2016) including 1180

students (590 in the flipped classroom group, 590 in the control group) reported their

students’ theoretical knowledge scores. One study (Tian et al., 2015) showed no

statistically significant difference in student theoretical knowledge scores between the

flipped classroom and the control group, while the others (Deng, 2016, Li, 2016, Li et

al., 2017, Liu, 2016, Yang, 2016, Zhang et al., 2017, Zhao et al., 2016, Zhong et al.,

2016) showed significant differences. A high level of heterogeneity was observed

between the nine studies (I2=87%, P<0.001), and thus, a random effects model was

utilized for the meta-analysis. The pooled effect size showed a significant difference

between the flipped classroom and traditional lecture groups (SMD=1.06, 95% CI:

ACCEPTED MANUSCRIP

T

13

0.70-1.41, P<0.001) (Fig. 4). A fixed effect model was also applied to pool the data,

and the pooled effects still favored the flipped classroom group (SMD=1.04, 95% CI:

0.91-1.16, P<0.001). We observed heterogeneity between studies upon reporting

theoretical knowledge scores; therefore, a sensitivity analysis was performed to verify

the reliability of the results. After excluding the study with the largest sample size (Li

et al., 2017) from the analyses, the pooled effect size favored the flipped classroom

group (SMD=1.15, 95% CI: 0.82-1.47, P<0.001), and the effects observed in the

primary analysis were not changed.

3.4.2 Skill scores

Five studies (Ji et al., 2016, Li, 2016, Tao et al., 2016, Zhang et al., 2017, Zhao et

al., 2016) involving 810 students (406 in the intervention group, 404 in the control

group) reported the student skill scores. One study (Zhang et al., 2017) showed no

statistically significant difference in student skill scores between the flipped

classroom and the control group, while the others (Ji et al., 2016, Li, 2016, Tao et al.,

2016, Zhao et al., 2016) showed significant differences. There was a high degree of

heterogeneity (I2=97%, P<0.001), and a random effects model was therefore used.

ACCEPTED MANUSCRIP

T

14

The pooled effect size showed a significant difference between the flipped classroom

and traditional lecture (SMD=1.40, 95%CI: 0.46-2.34, P<0.001) (Fig. 5). A fixed

effect model was also applied to pool the data, and the pooled effects still favored the

flipped classroom group (SMD=1.56, 95% CI=1.40-1.72, P<0.001). Given the

observed heterogeneity between the studies that reported skill scores, a sensitivity

analysis was performed to verify the reliability of the results. After excluding the

study with the largest sample size (Ji et al., 2016) from the analyses, the pooled effect

size favored the flipped classroom group (SMD=1.21, 95% CI: 1.03-1.40, P<0.001),

and the effects observed in the primary analysis were not changed.

3.4.3 Quality of evidence

The quality of evidence was evaluated with the GRADE system. As shown in

Table 2, the evidence grade of the results regarding theoretical scores and skill scores

was low and very low, respectively.

4.Discussion

4.1 Summary of main findings

To the best of our knowledge, this is the first meta-analysis to examine the

effectiveness of the flipped classroom approach in Chinese baccalaureate nursing

ACCEPTED MANUSCRIP

T

15

students. A comprehensive search of the literature comparing the flipped classroom

with traditional lectures was performed, and eleven studies met the predefined

inclusion criteria. All the studies used a flipped classroom as the teaching approach in

the intervention group and traditional lectures as the approach in the control group.

The results indicated an improvement in students’ theoretical knowledge scores and

skill scores through the flipped classroom approach.

Although flipped classroom pedagogy has been widely applied in nursing

education around the world, the use of this teaching method in China is still in its

initial stages, especially in higher nursing education. Some nursing colleges and

universities have made tentative steps towards utilizing the flipped classroom

approach in nursing education (especially baccalaureate nursing education) since

2014, and more and more nursing educators have realized the advantages of flipped

classrooms and are trying to use the new pedagogy (Zhang et al., 2017). Because

different education systems or cultural backgrounds may influence the effectiveness

of the flipped classroom method (Frambach et al., 2012, Kim, 2008, Sharma et al.,

2014), the target population was limited to Chinese nursing students in the Chinese

education system to evaluate the potential effectiveness of the flipped classroom

approach in China. In addition, we focused on two outcomes (theoretical knowledge

scores and skill scores) in this meta-analysis on the basis of Kirkpatrick's framework

of evaluation, which assesses outcomes on the following four different levels: level 1

(reactions), level 2 (learning), level 3 (behavior), and level 4 (results) (Ameh and Van,

2015). Among these levels, evaluation at level 2, or determining participant

knowledge and skills in a test setting, is widely used (Yardley and Dornan, 2012). In

addition, theoretical knowledge scores and skill scores are relatively objective and

reliable for assessing the effectiveness of flipped classrooms compared with other

subject outcomes.

The results of the current meta-analysis revealed that the flipped classroom

pedagogy had positive effects on the students’ academic performance, which is

consistent with the previous research (Betihavas et al., 2016, Njie-Carr et al., 2017,

ACCEPTED MANUSCRIP

T

16

Presti, 2016). There are several possible reasons to explain the findings. First,

according to Bloom’s revised taxonomy of learning, the traditional lecture has

primarily involved spending classroom time (when teachers and students interact

face-to-face) to promote low-level cognitive work (the acquisition and comprehension

of factual knowledge). However, the flipped classroom offers the opportunity for

students to engage in higher-order cognition (application, analysis, evaluation, and

synthesis of knowledge) through the use of active learning strategies, such as

problem-based learning, simulation, think-pair-share activities, student presentations

and discussions, and others (Krathwohl, 2002, Mclaughlin et al., 2014), which can

foster students’ motivation to acquire knowledge and skills (Prince, 2004). Second,

the flipped classroom emphasizes individualized student-centered learning, and

students can learn according to their own learning style by studying the key and

difficult points repeatedly before class. In the face-to-face class phase, students use

their knowledge with instructors’ direction to address challenging problems in a

setting that promotes teamwork, which helps students acquire knowledge and skill

better than traditional classroom lectures alone (Zhong et al., 2016). Third, Chinese

students have received traditional lecture-based instruction since primary school.

Instructors are usually dominant, and students are only a passive audience, which

greatly affects learning enthusiasm among the students (Liu et al., 2015). However,

compared with traditional learning, the flipped classroom approach is a novelty for

them and has greatly motivated their learning interest (Yan and Xie, 2015).

The purpose of this meta-analysis was to assess the overall effectiveness of

flipped classrooms in Chinese baccalaureate nursing education. All the studies

included in this meta-analysis used the flipped classroom as the intervention and

theoretical and/or skill scores as the outcome to assess the intervention effectiveness;

this methodological consistency reduced the heterogeneity to some extent. However,

the educational environments, courses, course contents, course duration,

concentration, examination criteria, and others were different among included studies,

which resulted in high heterogeneity among the included studies for theoretical

ACCEPTED MANUSCRIP

T

17

knowledge and skill scores. Therefore, a random effects model was utilized for the

meta-analysis. Sensitivity analyses were performed by excluding the study with the

largest sample size, while the result did not change the effects observed in the primary

analysis. This result indicated that our result was stable and reliable even though there

was high heterogeneity.

All the included studies had a moderate possibility of bias due to the low

methodological quality, with 7 studies lacking a description of the randomization

method and no studies mentioning allocation concealment. Therefore, selection bias

was present. Due to the nature of the flipped classroom teaching approach, it was

difficult to blind the students and faculty, but none of the studies reported whether the

outcome assessors were blinded. We did not identify any attrition bias or reporting

bias based on the information reported in the original studies. In addition, the funnel

plot was almost symmetrical, and there was no publication bias in the meta-analysis.

Based on the findings above, we did not obtain high-quality evidence in the present

meta-analysis, given that the summarized evidence ranged from very low to low.

4.2 Comparisons with other published reviews

Three review articles (Betihavas et al., 2016, Njie-Carr et al., 2017, Presti, 2016)

to date have summarized and analyzed studies that were previously conducted on the

impact of the flipped classroom model in higher nursing education. These reviews

involved only descriptive analyses and no meta-analysis. Betihavas’s (Betihavas et al.,

2016) systematic review revealed that the use of the flipped classroom in higher

nursing education yielded neutral or positive academic outcomes and mixed results

regarding satisfaction. In that review, although some students were satisfied with the

flipped model approach, whether this satisfaction translated into improved final

examination scores remained unknown. Presti’s (Presti, 2016) integrative review

indicated that the flipped classroom approach yielded positive outcomes, but

quantifiable, significant changes in the nursing students’ knowledge, skills, and

attitudes in response to the flipped learning approach were lacking. Finally, Njie-

ACCEPTED MANUSCRIP

T

18

Carr’s (Njie-Carr et al., 2017) integrative review concluded that the flipped classroom

model improved student examination scores, course performance, and satisfaction.

However, the results were mixed, because they were derived primarily from non-

experimental studies that employed various methods in the classroom and a variety of

outcomes, such as single examination scores or course grades. In summary, these

findings hinted that flipped classroom yielded neutral or positive outcomes in nursing

education, and our findings further confirmed previous results.

4.3 Strengths and limitations

4.3.1 Strengths of the review

Our meta-analysis had some strengths. First, we conducted a broad search using

both MeSH terms and keywords that covered flipped classroom and nursing

education, and this search was conducted in multiple Chinese and English databases.

In addition, the sensitivity analyses indicated that the results of the meta-analysis were

credible. Finally, there was no evidence of publication bias in our meta-analysis.

4.3.2 Limitations of the review

Several potential limitations of our study should be acknowledged. First, because

our research focused only on undergraduates in higher nursing education in China, the

results may be applicable to situations in China only. Second, our study assessed only

two objective outcomes, theoretical knowledge scores and skill scores, and we did not

evaluate other subjective outcomes, such as student satisfaction, critical thinking

ability, and teamwork ability, etc. Third, although the search strategy was extensive

and inclusive, we did not search the unpublished literature, and hence the related data

in these investigations might be omitted.

4.3.3 Limitations of the included studies

Some limitations may be present in the included studies. First, all the included

studies had low methodological quality, such as inaccurate randomization methods,

ACCEPTED MANUSCRIP

T

19

allocation concealment and blinding of assessors, etc. that may have resulted in

selection bias and detection bias. Second, there were no standard measurement tools

used to assess the effectiveness of the flipped classroom pedagogy, and the different

instruments with different validity and reliability values may have led to measurement

bias. Third, the sample size in some included studies was small, which may affect the

intervention effects. Lastly, the courses, contents, duration, concentration,

examination criteria, and other factors vary among the included studies, which

resulted in significant heterogeneity.

4.4 Implications for nursing education and future research

Our findings suggest that the flipped classroom is a promising teaching

approach. However, Morton et al. (Morton and Colbert-Getz, 2017) and Tian et al.

(Tian et al., 2015) found that flipped classroom students outperformed lecture

classroom students on analysis items but there were no differences in performance

between the two groups in terms of knowledge and application. Therefore, not all the

contents are appropriate for flipping, such as courses conveying large amounts of

factual content, but flipping may benefit retention when students are expected to

analyze material. In addition, pre-class work prepares students for in-class work, but

if students do not prepare fully pre-class, the learning effect of the in-class phase will

be affected (Ramnanan and Pound, 2017). Hence, it is very important to improve

students’ learning interest and compliance (Chen et al., 2017, Heitz et al., 2015).

Some suggestions to achieve this improvement are as follows: refining learning

materials to minimize student fatigue and distraction; and paying attention to process

evaluation. For example, instructors should record and evaluate students’ pre-class

performance and give quizzes to supervise and facilitate learning.

This review reveals suggestions for future research. Given the methodological

drawbacks of the included studies, high-quality studies with accurate randomization

and blinding to reduce the risk of bias are needed. In addition, all the included studies

in this review assessed the effects of the flipped classroom across three of

ACCEPTED MANUSCRIP

T

20

Kirkpatrick’s classification measures as follows: perception, attitude and change in

knowledge and skill sets, and none of the studies conducted a process evaluation of

the flipped classroom intervention. Therefore, there is a need to conduct studies with

evaluations of the long-term effects of flipped classrooms regarding retention and

transfer of knowledge to professional practice and patient care (the highest two levels

of Kirkpatrick’s measures) and to perform a process evaluation to better examine the

effects of flipped classrooms. Moreover, further studies with a large sample size are

warranted. Lastly, few studies applied flipped classrooms to laboratory courses.

Therefore, more studies are needed to confirm the effect of flipped classrooms in

laboratory courses.

5. Conclusions

This meta-analysis is the first evidence-based study to include all RCTs used to

evaluate the effects of the flipped classroom approach in baccalaureate nursing

students in China. The results showed that the flipped classroom pedagogy is an

effective learning approach to improve students’ theoretical knowledge and skills.

This finding suggests that flipped classrooms can be integrated into nursing education

to improve academic performance. However, due to the limitations discussed earlier,

additional studies with large samples and high methodological quality are warranted

in a variety of educational settings to confirm our findings.

Conflict of interests

The authors report no conflict of interests.

Funding

This research did not receive any specific grant from funding agencies in the

public, commercial, or not-for-profit sectors.

Author Contributions

ACCEPTED MANUSCRIP

T

21

RJH, HMG, YSY, ZHN and NJ were responsible for conceiving and designing

the experiments, collecting and analyzing the data, and writing and revising the

manuscript. RJH and HMG were responsible for the data interpretation. XLJ made

important intellectual contributions to the research design, provided technical

guidance and revised the manuscript. All the authors read and approved the final

version of the manuscript.

What is already known about the topic?

The flipped classroom approach has been widely used in higher nursing

education settings in China since 2014.

Many studies have examined the effects of the flipped classroom approach on

theoretical knowledge scores and skill scores in nursing students, but the findings

have been mixed.

Previous reviews from other countries showed a positive relationship between the

flipped classroom approach and improved examination scores in nursing students.

These studies performed only descriptive analyses and did not involve a meta-

analysis.

What this paper adds

The flipped classroom approach was more effective than traditional lectures at

improving students’ theoretical knowledge scores and skill scores.

The studies included in this review exhibited some common weaknesses, such as

low methodological quality (inaccurate randomization methods, allocation

concealment and blinding, etc.), small samples, non-uniform assessment tools,

ACCEPTED MANUSCRIP

T

22

and a lack of long-term effectiveness assessments and process evaluations on

flipped classrooms.

Acknowledgments

We thank American Journal Experts (AJE) for the English language editing. This

manuscript was edited for English language usage by AJE.

ACCEPTED MANUSCRIP

T

23

References

Ameh, C.A., Van, d.B.N., 2015. Making It Happen: Training health-care providers in emergency

obstetric and newborn care. Best Practice & Research Clinical Obstetrics & Gynaecology

29 (8), 1077-1091.

Bergmann, J., Sams, A., 2012. Flip your classroom: reach every student in every class every day.

International Society for Technology in Education, Eugene.

Betihavas, V., Bridgman, H., Kornhaber, R., Cross, M., 2016. The evidence for 'flipping out': A

systematic review of the flipped classroom in nursing education. Nurse Educ Today 38,

15-21.

Chen, F., Lui, A.M., Martinelli, S.M., 2017. A systematic review of the effectiveness of flipped

classrooms in medical education. Medical Education 51 (6), 585-597.

Critz, C.M., Knight, D., 2013. Using the flipped classroom in graduate nursing education. Nurse

Educ 38 (5), 210-213.

Davies, R.S., Dean, D.L., Ball, N., 2013. Flipping the classroom and instructional technology

integration in a college-level information systems spreadsheet course. Educ Technol Res

Dev 61 (4), 563-580.

Deng, W.F., 2016. Application of the flipped classroom teaching mode based on constructive

theory in introduction to nursing. Chin J Prac Nurs 32 (18), 1429-1433.

Frambach, J.M., Driessen, E.W., Chan, L.C., Cp, V.D.V., 2012. Rethinking the globalisation of

problem-based learning: how culture challenges self-directed learning. Med Educ 46 (8),

738–747.

Galway, L.P., Corbett, K.K., Takaro, T.K., Tairyan, K., Frank, E., 2014. A novel integration of online

and flipped classroom instructional models in public health higher education. BMC Med

Educ 14 (1), 181.

Geist, Melissa, J., Larimore, David, Rawiszer, Hannah, Sager, Al, A.W., 2015. Flipped Versus

Traditional Instruction and Achievement in a Baccalaureate Nursing Pharmacology

Course. Nurs Educ Perspect 36 (2), 114-115.

Gilboy, M.B., Heinerichs, S., Pazzaglia, G., 2015. Enhancing student engagement using the flipped

classroom. J Nutr Educ Behav 47 (1), 109-114.

Hamer, L.O., 2000. The Additive Effects of Semistructured Classroom Activities on Student

Learning: An Application of Classroom-Based Experiential Learning Techniques. Journal

of Marketing Education 22 (22), 25-34.

ACCEPTED MANUSCRIP

T

24

Harmon, R.B., Hills, R.L., 2015. Transforming Psychiatric Mental Health Nursing Education With

Team Based Learning. Arch Psychiatr Nurs 29 (6), 413.

Harrington, Ann, S., Bosch, Vanden, M., Schoofs, Nancy, BeelBates, Cynthia, Anderson, Kirk, 2015.

Quantitative Outcomes for Nursing Students in a Flipped Classroom. Nurs Educ Perspect

36 (3), 179-181.

Heitz, C., Prusakowski, M., Willis, G., Franck, C., 2015. Does the Concept of the “Flipped

Classroom” Extend to the Emergency Medicine Clinical Clerkship? Western Journal of

Emergency Medicine 16 (6), 851-855.

Herreid, C.F., Schiller, N.A., 2013. Case Studies and the Flipped Classroom. J Coll Sci Teach 42 (5),

62-66.

Hguyatt, G., Doxman, A., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic,

B., Atkins, D., Falck-Ytter, Y., 2011. GRADE guidelines: 5. Rating the quality of evidence—

—publication bias. J Clin Epidemiol 64 (12), 1277-1282.

Higgins, J.P.T., Thompson, S.G., Deeks, J.J., Altman, D.G., 2003. Measuring inconsistency in meta-

analyses. BMJ 327 (7414), 557-560.

Hsu, S.D., Chen, C.J., Chang, W.K., Hu, Y.J., 2016. An Investigation of the Outcomes of PGY

Students’ Cognition of and Persistent Behavior in Learning through the Intervention of

the Flipped Classroom in Taiwan. Plos One 11 (12), e0167598.

Ji, H.X., Liang, A., Zhang, H.Y., 2016. Application of the flipped learning model based on micro-

lectures in experimental teaching of surgical nursing. Modern Vocational Education (14),

152.

Kim, S., 2008. Silent Participation: East Asian International Graduate Students' Views on Active

Classroom Participation. Journal on Excellence in College Teaching 19, 199-220.

Krathwohl, D.R., 2002. A Revision of Bloom's Taxonomy: An Overview. Theory Into Practice 41 (4),

212-218.

Li, C.Y., 2016. Application the flipped learning method based on micro-lectures in fundamentals of

nursing teaching. Journal of Xiangnan University (Medical Sciences) (03), 56-58.

Li, Q., WANG, Y.N., LI, X.Y., Shan, W.Y., 2017. Application of flipped classroom in obstetrics and

gynecological nursing education. Health Vocational Education 35 (16), 121-123.

Lichvar, A.B., Hedges, A., Benedict, N.J., Donihi, A.C., 2016. Combination of a Flipped Classroom

Format and a Virtual Patient Case to Enhance Active Learning in a Required Therapeutics

Course. Am J Pharm Educ 80 (10).

Liebert, C.A., Lin, D.T., Mazer, L.M., Bereknyei, S., Lau, J.N., 2016. Effectiveness of the Surgery Core

Clerkship Flipped Classroom: a prospective cohort trial. Am J Surg 211 (2), 451.

ACCEPTED MANUSCRIP

T

25

Liu, F., 2016. Application of flipped classroom teaching mode based on micro-lectures in teaching

of introduction to nursing. Chinese Nursing Research 30 (27), 3406-3407,3408.

Liu, Y., Yang, G.H., Yang, C.Y., Chang, Z.X., 2015. Application of cooperative learning in the course

of professional English among student nurses. Chin J Nurs Educ 12 (7), 540-542.

Mason, G.S., Shuman, T.R., Cook, K.E., 2013. Comparing the Effectiveness of an Inverted

Classroom to a Traditional Classroom in an Upper-Division Engineering Course. IEEE T

Educ 56 (4), 430-435.

Mattis, K.V., 2015. Flipped Classroom Versus Traditional Textbook Instruction: Assessing Accuracy

and Mental Effort at Different Levels of Mathematical Complexity. Technology,

Knowledge and Learning 20 (2), 231-248.

McLaughlin, J.E., Griffin, L.M., Esserman, D.A., Davidson, C.A., Glatt, D.M., Roth, M.T.,

Gharkholonarehe, N., Mumper, R.J., 2013. Pharmacy student engagement, performance,

and perception in a flipped satellite classroom. Am J Pharm Educ 77 (9), 196.

Mclaughlin, J.E., Roth, M.T., Glatt, D.M., Gharkholonarehe, N., Davidson, C.A., Griffin, L.M.,

Esserman, D.A., Mumper, R.J., 2014. The flipped classroom: a course redesign to foster

learning and engagement in a health professions school. Academic Medicine 89 (2), 236.

Missildine, K., Fountain, R., Summers, L., Gosselin, K., 2013. Flipping the classroom to improve

student performance and satisfaction. J Nurs Educ 52 (10), 597-599.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., 2009. Preferred reporting items for systematic

reviews and meta-analyses: the PRISMA statement. BMJ 339 (7716), 332-336.

Mortensen, C.J., Nicholson, A.M., 2015. The flipped classroom stimulates greater learning and is

a modern 21st century approach to teaching today's undergraduates. J Anim Sci 93 (7),

3722-3731.

Morton, D.A., Colbert-Getz, J.M., 2017. Measuring the impact of the flipped anatomy classroom:

The importance of categorizing an assessment by Bloom's taxonomy. Anatomical

Sciences Education 10 (2), 170-175.

Njie-Carr, V.P., Ludeman, E., Lee, M.C., Dordunoo, D., Trocky, N.M., Jenkins, L.S., 2017. An

Integrative Review of Flipped Classroom Teaching Models in Nursing Education. J Prof

Nurs 33 (2), 133-144.

Park, S.E., Howell, T.H., 2015. Implementation of a flipped classroom educational model in a

predoctoral dental course. J Dent Educ 79 (5), 563-570.

Presti, C.R., 2016. The Flipped Learning Approach in Nursing Education: A Literature Review. J

Nurs Educ 55 (5), 252-257.

Prince, M., 2004. Does Active Learning Work? A Review of the Research. Journal of Engineering

ACCEPTED MANUSCRIP

T

26

Education 93 (3), 223-231.

Ramnanan, C.J., Pound, L.D., 2017. Advances in medical education and practice: student

perceptions of the flipped classroom. Advances in Medical Education & Practice 8, 63.

Schwartz, T.A., 2014. Flipping the statistics classroom in nursing education. J Nurs Educ 53 (4),

199-206.

Sharma, N., Lau, C.S., Doherty, I., Harbutt, D., 2014. Medical school pedagogy should be culture-

dependent. Academic Medicine Journal of the Association of American Medical Colleges

89 (2), 194.

Simpson, V., Richards, E., 2015. Flipping the classroom to teach population health: Increasing the

relevance. Nurse Educ Pract 15 (3), 162.

Tao, W.W., Liu, Y.N., Ding, S.Z., Sha, L.Y., Liu, Y., 2016. Effect of flipped classroom and micro-

lectures on educational environment and academic emotions of nursing students.

Chinese Nursing Management 16 (9), 1221-1224,1225.

Tian, Z.J., Jin, R.H., Liu, C.F., 2015. Application of flipped classroom in geriatric nursing education.

Chin J Nurs Educ 12 (5), 333-335.

Yacout, D., Shosha, A., 2016. Nursing students' perceptions towards flipped classroom

educational strategy. Journal of American Science 12 (2), 62-75.

Yan, L.X., Xie, H.Y., 2015. Application of flipped classroom in nursing comprehensive skill training.

Journal of Nurses Training 30 (7), 626-628.

Yang, S.G., 2016. Effectiveness of the flipped classroom applied in preventive medicine teaching.

China Medical Education Technology 30 (5), 577-579,580.

Yardley, S., Dornan, T., 2012. Kirkpatrick's levels and education 'evidence'. Medical Education 46

(1), 97–106.

Zhang, H.M., Wu, X.W., Shao, Z.Z., 2017. Application of micro learning resource-based flipped

class model in the teaching of basic nursing. Nursing Practice and Research 14 (2), 127-

129.

Zhang, W., Zhang, G.J., Xue, Y.Z., 2017. Quantitative analysis of the literature on flipped classroom

in nursing area in China. Health Vocational Education 35 (1), 125-126.

Zhao, H., Yao, P.B., Liu, Y.H., Wu, M.C., 2016. Application of the flipped learning model based on

micro-lectures in fundamentals of nursing. Journal of SNAKE（Science & Nature） 28

(1), 108-109.

Zhong, M.L., Jiang, D.Y., Wei, Q.M., JIang, Z.X., Chen, S.Z., Hu, R.J., 2016. The effect of flipped

classroom in primary life support education. Chin J Nurs Educ 13 (4), 273-276.

ACCEPTED MANUSCRIP

T

27

Fig. 1. Flow diagram of study selection.

RCT=randomized controlled trial, CENTRAL=Cochrane Central Register of Controlled Trials,

CNKI=China National Knowledge Infrastructure, Wanfang=Wanfang Database, and VIP=the

Chinese Scientific Journals Database.

Scre

enin

g In

clu

ded

El

igib

ility

Id

enti

fica

tio

n

Full-text articles excluded, with

reasons

(n=40)

Not addressing undergraduate

baccalaureate nursing students

(n=25)

Non-RCT (n=8)

Unrelated outcome (n=4)

Incomplete data (n=3)

Records identified through database searching

(n=840)

PubMed (n=39)

EMBASE (n=33)

CENTRAL (n=0)

CINAHL (n=46)

CNKI (n=257)

Wanfang (n=266)

VIP (N=199)

Additional records identified

through other sources

(n=0)

Records after duplicates removed

(n=376)

Records screened

(n=376)

Records excluded

(n=325)

Not addressing undergraduate

baccalaureate nursing students in

China (n=186)

Non-RCT (n=139)

Full-text articles assessed

for eligibility

(n=51)

Studies included in the

qualitative synthesis

(n=11)

Studies included in the

quantitative synthesis

(meta-analysis)

(n=11)

ACCEPTED MANUSCRIP

T

28

Fig. 2. Risk of bias summary: a review of the author evaluations of each risk of bias item for

each one included in the study.

ACCEPTED MANUSCRIP

T

29

Fig. 3. Funnel plot of comparisons: flipped classroom versus traditional

lecture; outcome: theoretical knowledge scores. SE=standard error,

SMD=standardized mean difference.

ACCEPTED MANUSCRIP

T

30

Fig. 4. Forest plot of comparison: flipped classroom versus traditional lecture; outcome: standardized

mean difference for the theoretical scores. CI=confidence interval, SD=standard deviation.

ACCEPTED MANUSCRIP

T

31

Fig. 5. Forest plot of comparisons: flipped classroom versus traditional lecture, outcome: standardized

mean difference of the skill scores. CI=confidence interval, SD=standard deviation.

ACCEPTED MANUSCRIP

T

32

Table 1 Characteristics of the included studies

Study ID Study

design Subjects Course type Participant characteristics

Sample size

(IG/CG) Intervention Control Outcomes

Duration of

study

Tao et al 2016

(Dalian, China)

RCT Fundamentals

of nursing

Laboratory

course

Sophomores at a university.

Mean age IG: 20.64 (SD 0.77), CG: 20.57 (SD 0.86).

Gender IG: Female 28 (87.5%), CG: Female 27 (90%). Admission scores IG: 537.44 (SD28.87), CG: 528.53

(SD33.86).

62 (32/30)

Used flipped classrooms as the educational

approach.

Pre-class: interactive online modules; prerecorded

mini-videos (5-8 min); textbook reading; pre-assignments.

In-class: group-developed presentations; group

discussion.

Traditional

lecture

Skill scores (final

examination) One semester

Yang 2016

(Henan, China) RCT

Preventive

medicine Theory course

Sophomores at a college.

Gender IG: Male 5 (7.8%), CG: Male 4 (6.3%). 128 (64/64)


approach.

The detailed implementation process was not described.

Traditional

lecture

Theoretical

knowledge scores

(final examination)

One semester

Zhong et al

2016 (Guizhou,

China)

RCT Emergency and critical

care

Theory course Juniors at a college. 72 (36/36)


approach.

Pre-class: interactive online modules; voice-over PowerPoint; prerecorded mini-videos; textbook

reading; pre-assignments.

In-class: PowerPoint presentations; role play; games.

Traditional lecture

Theoretical knowledge scores

(unit test)

One semester

Tian et al 2015

(Shanxi,

China)

RCT Geriatric

nursing Theory course

Juniors at a university.

Mean age IG: 21 (SD 1.14), CG: 20.72 (SD 0.85).

Gender IG: Female 30 (93.8%), CG: Female 30 (93.8%)

64 (32/32)


approach.

Pre-class: prerecorded mini-videos (10-15 min); pre-

assignments.

In-class: discussion; presentations; role play; case studies.

Traditional

lecture

Theoretical

knowledge scores

(final examination)

One semester

Deng 2016

(Hubei, China) RCT

Introduction

to nursing Theory course

Freshmen at a university.


Gender IG: Female 28 (82.4%), CG: Female 27 (84.4%).

Admission scores IG: 490.88 (SD7.23), CG: 491.97

(SD8.26).

66 (34/32)


approach.

Pre-class: prerecorded lecture videos; PowerPoint

lecture; classical cases; pre-assignments.

In-class: group-developed presentations; discussion;

question and answer sessions.

Traditional

lecture

Theoretical

knowledge scores

(final examination)

One semester

Liu 2016

(Shaanxi,

China)

RCT Introduction

to nursing Theory course

Freshmen at a university.

104 (53/51)

Used flipped classroom as the educational approach.

Pre-class: prerecorded mini-videos; review materials;

pre-assignments.

In-class: group-developed presentations; discussion;

question and answer sessions.

Traditional

lecture

Theoretical

knowledge scores

(final examination)

One semester

Zhao et al

2016 (Hunan,

China)

RCT Fundamentals

of nursing

Theory course

& Laboratory

course


Age IG: Ranging from 18 to 22, CG: Ranging from 19 to 22.


229 (115/114)


approach.

Pre-class: prerecorded mini-videos (approximately 10

min); interactive online modules; pre-assignments.

In-class: discussion; instructors’ demonstration;

standard patients.

Traditional

lecture

Theoretical

knowledge and

skill scores (final

examination)

Two

semesters

ACCEPTED MANUSCRIP

T

33

Ji et al 2016

(Hunan,

China)

RCT Surgical

nursing

Laboratory

course Sophomores at a college. 242 (120/122)


approach.

Pre-class: review teaching materials; practice;

discussion; self-developed mini-video (approximately 5

min).

In-class: instructors comment on students’ self-

developed mini-video and discussion.

Traditional

lecture

Skill scores (final

examination) One semester

Li 2016 (Hunan,

China)

RCT Fundamentals

of nursing

Theory course & Laboratory

course

Sophomores at a college. 182 (90/92)


Pre-class: interactive online modules; prerecorded mini-videos; PowerPoint lecture; pre-assignments.

In-class: problem-based learning; case studies;

discussion; instructors’ demonstration; practice.

Traditional

lecture

Theoretical knowledge and

skill scores (final

examination)

One semester

Li et al 2017

(Hebei, China) RCT

Obstetrics and

gynecological

nursing

Theory course

Juniors at a college.



240 (117/123)


approach.


mini-videos (8-30 min); pre-assignments.

In-class: problem-based learning; group-developed

presentations; discussion; question and answer sessions.

Traditional

lecture

Theoretical

knowledge scores

(final examination)

One semester

Zhang et al

2017

(Guangdong,

China)

RCT Fundamentals

of nursing

Theory course

& Laboratory

course




95 (49/46)



videos; PowerPoint lecture; pre-assignments.

In-class: problem-based learning; discussion;

instructors’ demonstration; practice.

Traditional

lecture

Theoretical

knowledge and

skill scores (final

examination)

One semester

RCT, randomized controlled trial. IG, intervention group. CG, control group. SD, standard deviation

ACCEPTED MANUSCRIP

T

34

Table 2 Quality of evidence for the included studies.

Variable

Risk of bias Inconsistency Indirectness Imprecision Publication bias Quality of evidence Outcomes

Theoretical knowledge scores

Serious (-1) Serious (-1) No No Undetected Low

Skill scores Serious (-1) Serious (-1) No Serious (-1) Undetected Very low

ACCEPTED MANUSCRIP

T

Documents

Effectiveness of flipped classrooms in Chinese baccalaureate nursing education… · 2018-12-15 · 2 Abstract Background: In recent years, the flipped classroom approach has been