Promising Practices in STEM Teaching and Learning: A Meta ...aggiestem.tamu.edu/wp-content/uploads/2016/12/... · STEM literacy includes and integrates literacy in all four subject

Texas A&M University 1

Promising Practices in STEM Teaching and Learning: A Meta-Synthesis

Document Authors

Mary Margaret Capraro

Robert M. Capraro

Sandra Metoyer

Sandra Nite

Cheryl Ann Peterson


Document prepared Under the Direction of

STEM Collaborative for Teacher Professional Learning

Project Members in Alphabetical Order

Mary Margaret Capraro

Robert M. Capraro

Tim Scott

Jackie Stillisano

James Morgan

Hersh Waxman

This material is in part based upon work supported by the The Higher Education Coordinating

Board under Grant No. 11307. Any opinions, findings, and conclusions or recommendations

expressed in this material are those of the STEM Collaborative for Teacher Professional

Learning at Texas A&M University and do not necessarily reflect the views of The Higher

Education Coordinating Board.


Table of Contents

Promising Practices in STEM Teaching and Learning: A Meta-Synthesis ........................... 4 Research Question ......................................................................................................................................... 4

Article Coding Procedures ........................................................................................................................ 5

Category 1: Reform-Based Teaching and Learning ................................................................... 5

Category 2: Informal STEM ...............................................................................................................11

Category 3: Teacher Factors.............................................................................................................13

Category 4: Technology......................................................................................................................19

Category 5: School Factors Influencing STEM Learning ............................................................23

Other STEM Interventions ..................................................................................................................24

References ..............................................................................................................................................26


Promising Practices in STEM Teaching and Learning: A Meta-Synthesis

The term STEM was first used in the 1990s and was frequently used to label anything that involved one or more

of the following four disciplines: science, technology, engineering, or mathematics (Bybee, 2010). Mathematics

and science have been the focus of practical applications of science, technology, engineering, and mathematics

(STEM), while technology and engineering have taken a back seat. Often technology teachers claim that they

integrate the T and E in STEM, but they continue to think of STEM as a set of four separate subjects (Sanders,

2009). Technology influences the lives of people every day in a myriad of ways, but the typical person

understands little about it. An increased emphasis of technology in education would position students to better

understand the changes in the world around them and how the use of technology is integral to science,

engineering, and mathematics applications (Bybee, 2010). Advocates of STEM education have encouraged

increased integration of technology and engineering in the K-12 curriculum. Technology is more than just

computer literacy; it includes handheld devices and instruments that can be applied in science and engineering.

STEM literacy includes and integrates literacy in all four subject areas. Scientific literacy can be defined in

terms of knowledge and processes applied to decisions in the natural world. Technological literacy involves not

only the ability to use new technologies but also an understanding of how they are developed and how they

affect our lives. Engineering literacy includes using the engineering design process to solve problems that cross

discipline lines. Mathematical literacy requires students to be able to analyze and reason in order to solve

problems and interpret solutions (Hanover Research, 2011). Interdisciplinary STEM education creates a synergy

expanding beyond the four individual subject areas toward the solving of problems that overlap the four

disciplines and their subcategories. Unfortunately, the integration of STEM education in the K-12 classrooms

has been slow. Many high schools claim to implement a STEM program, but the subjects continue to be taught

in isolation with few connections between them. Although these efforts may be a start in the right direction,

they need to move more definitively in the direction of Project-Based Learning (PBL) using the engineering

design process to solve problems that include other subject areas (Lantz, 2009). One event that may help to

move the education field toward integrated STEM is the inclusion of technology literacy on the National

Assessment of Educational Progress (NAEP). Requiring STEM education for all students will be a step toward

better preparation of students for the highly technological and quickly changing world today (Bybee, 2010;

Dugger, 2010).

Providing a high-quality education that prepares students for majors and careers in STEM fields remains

challenging for educators. Over the last 8-10 years, there has been an increased interest in exposing students to

integrated studies in STEM areas to better prepare them to solve 21th century problems that require knowledge

in multiple fields. New or modified teaching strategies that emulate real world work situations may be required

to successfully implement the new experiences in learning. The purpose of this study is to determine what

STEM strategies have been effective in increasing student knowledge and ultimately an interest in STEM fields.

Research Question

What are the promising practices in middle and high-school STEM teaching and learning?


Article Coding Procedures

To answer the research question, a comprehensive search of STEM practices in teaching middle school and

high school was conducted using the following search terms for the title: "STEM practice" OR teaching OR

learning OR education OR "high school" OR "middle school" OR research NOT cell NOT cells

The idea of integrated STEM was first prominently used in classrooms around the United States in 2005; as a

result, the criterion for the time period surveyed was set for January 1, 2005 through the date of the search,

August 28, 2013. We did not require the word “STEM” in the title because the term was not widely used in the

earlier years. We considered any article that included some integration of at least two of the STEM fields to be

STEM.

We used two comprehensive search engines available through the Texas A&M University Library System:

Google Scholar and EBSCO. The Google Scholar search returned 1,128 hits, and the EBSCO Academic Search

Complete (with medical journals eliminated) returned 7,621 hits. We screened studies to eliminate those related

to an agricultural or medical meaning of “STEM” (e.g., plant stems, stem cell research), elementary level,

undergraduate level, or graduate level STEM education, and studies dealing only with STEM careers. We also

checked references on each codable study to locate additional articles. Studies used in this meta-analysis

consisted of journal articles, paper and poster presentations, dissertations, reports, and book chapters. We

collected all studies available that appeared to relate to middle school or high school STEM education, resulting

in a compilation of 509 artifacts with 61 of these artifacts fulfilling the criteria to be included in this meta-

synthesis. In the search for integrated STEM, there were likely articles missing that used technology in

mathematics or science education because there were no doubt thousands of them. Although the word “STEM”

was not a required criterion, and artifacts that integrated any two of the STEM subjects were accepted, there

would be many articles that use technology in mathematics or science education that would not meet the search

criteria (e.g., science simulation, math program). Sixty-one artifacts were classified into five categories Reform

Based Teaching and Learning (see Table 1), Informal Education (see Table 2), Teacher Factors (see Table 3),

Technology (see Table 4), and School Factors (see Table 5) and commonalities within the five categories were

sought. The rest of the report will detail these five categories.

Category 1: Reform-Based Teaching and Learning

Educational reformers who are interested in improving teaching and learning encourage teachers to use

practices that are student-centered and constructivist in nature such as inquiry-, Project-, and problem-based

learning. These practices enable students to (1) feel excited about the world around them; (2) engage

knowledgeably in public discussion about issues of scientific and technological concern; and (3) increase their

economic productivity as a result of knowledge and skill acquisition (NRC, 1996). Furthermore, reform-based

teaching and learning practices have a history of producing positive outcomes (Anderson, 2002) such as

increases in cognitive achievement and skills (Shymansky, Kyle, & Alport, 1983), scientific literacy,

vocabulary knowledge, conceptual understanding, critical thinking, and positive attitudes (Haury, 1993). The

results of these practices, however, are mixed. Strobel and van Barneveld (2009) conducted a meta-synthesis of

extant meta-analyses comparing problem-based learning to traditional classroom instruction. They found that

problem-based learning promoted long-term retention of content knowledge, developed skills, and satisfied both

teachers and students, whereas traditional practices were more effective for short-term retention of knowledge

which was measured by standardized exams. When Kirschner, Sweller, and Clark (2006) analyzed extant

research related to reform-based practices such as inquiry, problem-based learning, and other constructivist


approaches, they found that less able learners had a decrease in content knowledge after the reform-based

practices were implemented but reported these same students enjoying the experiences.

Artifact coding. The meta-synthesis on reform-based teaching and learning practices resulted in 25 artifacts

from the years 2005-2013 (see Table 1). The practices were classified as inquiry, engineering-design, Project-

Based Learning (PBL), problem-based learning, Legacy Cycle, and hands-on activities. Students were exposed

to reform-based practices in a variety of settings at different grade levels, explored a variety of STEM-related

subjects, and were immersed in these STEM-related learning environments for different lengths of time. In

addition, different target groups of students were the focus of the studies and some teachers received

professional development (PD). Students were exposed to these reform-based strategies in both formal and

informal settings. Fourteen studies examining students in formal classrooms and the remaining 11 in informal

settings such as afterschool and weekend programs (3), summer programs (5), or combined afterschool and

summer programs (3). Students were in middle school (12), high school (11), or both (2). These students were

involved in different subjects related to integrated STEM, Science, Mathematics, Engineering, and Technology

along with writing, reading, and social studies. Students were engaged in STEM-related projects using six

different reform-based practices for one week or less (3), two to five weeks (6), six weeks to one semester (6),

or more than one year (6). Four of the studies did not describe the length of time that students were engaged.

Groups of females (10) and minorities (8) were targeted in the various studies. In nine of the 25 studies, the

students’ teachers received PD designed to aid them in the implementation of the reform-based practices. Most

of these reform-based teacher and learning practices focused on the following: 1) enhancing students’ content

knowledge (17), 2) developing students’ skills (8), 3) increasing students’ use of technology (3), 4) promoting

students’ interests in STEM-related college majors and careers (8), 5) examining students’ perceptions and

attitudes (12), and 6) providing rich learning environments for students (1).

Findings. In the meta-synthesis studies, inquiry was the most used reform-based teaching and learning practice.

Though it is a widely-used practice, inquiry has a myriad of meanings. These meanings change depending on

the context (e.g. Aulls & Shore, 2008; Grandy & Duschl, 2007; NRC, 1996). Inquiry was used in nine of the

meta-synthesis studies, and it was not defined in four of these studies (Duran, Hoft, Lawson, Madjahed, &

Orady, 2013; Heggen, Omokar, & Payton, 2012; Hylton, Otoupal-Hylton, Campbell, & Williams, 2012;

Wimpey, Wade, & Benson, 2011). Little and Leon de La Barra (2009) only briefly described inquiry as

questioning and hands-on minds-on. Ricks (2006) provided additional descriptions of inquiry as a student-

centered, active learning practice focusing on questioning, critical thinking, and problem-solving. Ricks (2006)

also stated that inquiry should mirror as closely as possible the enterprise of practicing real science. Three of the

studies described inquiry according to the National Science Education Standards’ definition of scientific inquiry

(Ketelhut, 2006; Kim et al., 2011; Williams, Ma, Prejean, Ford, & Lai, 2007). The National Science Education

Standards defines scientific inquiry as

…the diverse ways in which scientists study the natural world and propose explanations based on

the evidence derived from their work. Inquiry also refers to the activities of students in which they

develop knowledge and understanding of scientific ideas, as well as an understanding of how

scientists study the natural world. (National Research Council, 1996, p.23)

This definition captured both the role of inquiry in scientific endeavors and the activities of students as they

developed knowledge and understanding of scientific activities and how scientists study the natural world.

Engineering-design was used in five of the 23 research studies included in this meta-synthesis. The authors of

two studies stated that they had used an engineering-based practice; however, they failed to define what they


meant (Hudson, English, & Dawes, 2012; Klahr, Triona, & Williams, 2007). The authors of the remaining

studies defined engineering-design too differently but the core of the descriptions remained the same - the

identification and design of a solution to address the needs of the problem and the testing of the solution

(Mehalik, Doppelt, & Schuun, 2008; Schnittka, 2008; Richards, Hillock, & Schnittka, 2007). The engineering-

design practice was described concisely by Mehalik, Doppelt, and Schuun (2008) as a practice that systems

engineers used in the design and analysis of systems with iterative stages where students articulated their own

needs for a design. Students followed the following seven stages of a design process where they describe the

current situation, identify needs, develop criteria, generate alternatives, choose an alternative, create and test

prototypes, and reflect and evaluate.

Studies incorporating PBL were also found (4). While two of the four studies contained claims that students

engaged in PBL, they neither defined what they meant by PBL nor provided sufficient detail (Brown et al.,

2010; Kampe & Oppliger, 2011). The two studies that contained definitions both recognized PBL as a

constructivist practice where students had the opportunity to design their own projects and find solutions to

open-ended problems collaboratively (Lou, Liu, Shih, & Tseng, 2011; Olivarez, 2012). Olivarez (2012) also

discussed how PBL encourages student motivation and promotes academic rigors along with encouraging

multiple subject area integration and proper scaffolding by the instructor.

Reform-based practices that were not used as often as other practices, according to this meta-synthesis, included

problem-based learning, (3) and the Legacy Cycle (2) along with general hands-on (2) activities. Lou, Shih,

Diez, and Tsend (2011) defined PBL as a student-centered practice used in a meaningful learning situation that

was focused on the solution to a problem taken from a real situation. Students took initiative to construct

knowledge and effectively developed a solution to the problem by providing the resources, guidance, and

opportunities for exploration. Duran and Sendig (2012) described their study as design-based inquiry; however,

they did define what they meant and contextually their student intervention appeared to incorporate problem-

based learning. Zhe, Doverspike, Zhao, Lam, and Menzemer (2010) did not define problem-based learning.

Two other studies incorporated the Legacy Cycle. Klein and Sherwood (2005) defined the Legacy Cycle as a

practice where a strong contextually based challenge was issued to students and they had to generate ideas, view

different perspectives, research, complete formative self-assessment, and share their product, whereas Kanter

and Schreck (2007) discussed the benefits of the Legacy Cycle. These benefits included the student learning

concepts via inquiry in answering the driving questions and motivating students. The two studies using hands-

on practices did not provide enough detail to determine how specific reform-based practices were used

(Menzemer, 2007; Mosina, Belkharraz, & Chebanov, 2012). However, students in both studies engaged in their

own projects and explored, probed, observed, and collected data.

Reform-based learning practices were the focus of 25 studies explored in the meta-synthesis. Inquiry and

engineering-design were the most widely used practices; however, Project-Based, problem-based, Legacy

Cycle, and hands-on were also represented. Though we were able to classify these studies, with the exception of

hands-on, according to the practice used, many of the studies either did not define the practice used or rarely

provided definitions for the practice that closely matched the definitions found in other studies. Only three of

the studies (inquiry) contained the same definition for practice because of their use of the National Science

Education Standards. Though care was taken in categorizing these practices, the results are uncertain due to the

lack of uniformity or the dearth of definitions provided when the studies labeled the reform-based learning

strategy used.


Conclusions. Student outcomes from reform-based teaching and learning practices centered around six

subcategories: content, skills, technology use, majors and careers, perceptions and attitudes, and learning

environments.

Content Knowledge: One of the main goals of the reform-based teaching and learning strategies found in the

studies appeared to be increasing students’ content knowledge (17). For studies exploring the effects of inquiry,

four out of six showed positive gains pre- to posttest on students’ content knowledge regarding STEM, science,

technology, and math (Duran et al., 2013; Hylton et al., 2012; Ricks, 2006; Williams et al., 2007). Heggen et al.

(2012) found that students who participated in the inquiry-based afterschool program had higher science and

mathematics scores than their peers. Wimpey et al. (2011) had mixed results when they compared the pre- and

posttest gains between students in an inquiry setting versus students in a traditional setting. Inquiry students

showed greater posttest gains for Algebra than the traditional students; however, these students showed lower

gains for Algebra II than traditional students. The inquiry students’ science scores did show greater gains than

traditional students. Little and Leon del la Barra (2009) did not report any gains for their students’ engineering

and technology knowledge.

Engineering-design based practices appeared to offer benefits for students’ development of content knowledge.

Three of the four engineering design-based studies encompassing content knowledge also demonstrated

increases in students’ content knowledge in science, engineering, and math (Mehalik, Doppelt, & Schuun, 2008;

Schnittka, 2012; Richards, Hallock, & Schnittka, 2007). Furthermore, Mehalik et al. (2008) found that science

content knowledge increased more on the posttest for engineering design-based projects than inquiry. In

addition, the engineering design-based approach helped low achieving African Americans more than any other

group of students. Klahr et al. (2007), however, did not find any difference from pre- to posttests between

virtual engineering design-based learning environments and physical hands-on learning environments for

students’ engineering content knowledge.

Unlike the previous practices, the outcomes expressed in the PBL studies did not emphasize content knowledge.

Only one of the four PBL studies examined the results of the teaching and learning practice for content

knowledge. Lou, Liu, Shih, and Tseng (2011) found that there were significant positive differences between

students’ content and procedural knowledge in STEM when the students participated in PBL rather than

traditional practices. The focus of the other reform-based teaching and learning practices was increasing content

knowledge, with four of the six studies providing results. The results, however, were mixed with only Klien and

Sherwood (2005) and Menzemer (2007) reporting positive results. Klien and Sherwood found that students who

were taught using the Legacy Cycle to study an interdisciplinary biomedical engineering curriculum had

significantly greater gains in biology, physics, and anatomy and physiology content knowledge than students in

traditional classrooms. Menzemer (2007) reported that both typical students and Special Learning Disability

students showed gains in content knowledge related to STEM when engaged in hands-on activities. Studies by

Lou, Shih, Diez, and Tseng (2011) and Kanter and Schreck (2007) had mixed results for increasing content

knowledge. The qualitative results from Lou et al.’s study revealed that students had mixed thoughts on the

development of their STEM content knowledge. Some students reported that problem-based learning helped

them develop their content knowledge, whereas some students said that it did not. Students were, however, able

to connect and apply mathematics knowledge to their scientific knowledge. Kanter and Schreck, when

implementing the Legacy Cycle, found that students who had a greater initial understanding of biological

concepts scored higher on the more cognitively difficult concepts and application and were more likely to show

results of meaningful understanding. Students who demonstrated a low level of biological content knowledge

understandings were only able to show gains in basic biological understanding.


Skills: Another student outcome that researchers examined while exploring the effects of reform-based practices

was the development of skills related to STEM. Overall, the reform-based teaching and learning practices

increased students’ skills in various STEM related subjects. Studies examining Engineering-Design Based

practices did not explore students’ STEM-related skills. Skill development was studied in three of the eight

inquiry studies. Both Duran et al. (2013) and Little and Leon de la Barra (2009) found using pre- and posttests

that students’ technology skills increased while engaged in inquiry activities. Williams (2007) reported that

inquiry skills did not improve between pre- and posttests. They believed this was a result of students not

engaging in a systematic process, not being able to focus on a specific design, goal, or process, and that the

facilitators lacked the knowledge to guide their students through the inquiry process. PBL had a positive impact

on students’ writing, technology (Brown et al., 2010), and STEM-related workforce skills (Kampe & Oppliger,

2011). Olivarez found that when PBL practices were compared to traditional practices that students’ math,

science, and reading skills differed significantly with students having higher gains as a result of PBL (2012).

Problem-based learning practices had a positive impact on students’ skills development such as critical

thinking, inference, and inductive reasoning (Duran & Sendag, 2012). Students also used engineering skills to

guide their problem-solving skills (Lou, Shih, Diez, & Tseng, 2011).

Technology Use: The use of increased technology was another student outcome that was studied by researchers

examining the implementation of STEM reform-based practices. This increased use of technology was

supported by Heggen et al. (2012) who found that inquiry-based practices increased students’ use of both

mobile phones and computers. The increase in technology use included solving mathematical and scientific

problems. Duran et al. (2013) found that when students engaged in inquiry-based learning the results of their

technology use were mixed. Students increased their use of common technology during the intervention but

only half of the students increased their use of more advanced STEM technologies. Students’ use of basic

STEM tool sets stayed the same through the intervention. Students increased their use of technologies while

engaged in PBLs, such as software for databases, robotics programing, modeling, and computer game

development along with communicative technologies such as blogs, podcasting, and social networking (Kampe

& Oppliger, 2011).

Majors and Careers: A goal of many of the implementations of reform-based practices was to increase

students’ interests in STEM majors and future careers. As a result of being engaged in problem-based learning

practices revolving around STEM, females realized that they could have a career related to STEM due to their

knowledge development, activities they participated in, and interest towards STEM which was fostered during

the intervention. They also realized that they needed to promote their own abilities so they could have a career

in STEM (Lou, Shih, Diez, & Tseng, 2011). Problem-based learning programs also increased both male and

female students’ desire to participate in and confidence towards having a STEM-related career (Zhe et al.,

2010). For hands-on practices, 81% of underrepresented minority students who were involved in a youth center

promoting STEM chose to enter a STEM-related major. PBLs also had positive impacts on students’ desire to

enter STEM fields; Kampe and Oppliger (2011) found that 64% of their students had an interest in having a

STEM career. While other reform-based practices showed positive results related to STEM majors and careers,

inquiry-based practices had mixed results. After engaging in inquiry-based practices students were more

positive about having a science-related (Heggen et al., 2012; Ricks, 2006) or computing-related (Heggen et al.,

2012) career, did not demonstrate a change towards their desire to become an engineer (Little & Leon de la

Barra, 2009), and were mixed in their desire to pursue a STEM career (Duran et al., 2013). Students who had

previously thought about having mathematics as a career continued their desire to enter a mathematics field,

whereas students who were uncertain about mathematics as a career became less inclined to enter the field after

engaging in inquiry-based practices (Duran et al., 2013).


Perceptions and Attitudes: Many of the researchers examined student perceptions and attitudes towards STEM

related-activities, interests, attitudes towards STEM in general or a specific STEM subject, attitudes towards

people involved in STEM careers, and their self-efficacy regarding their abilities to be successful towards

STEM content and skills. Overall, students who engaged in inquiry-based practices had positive attitudes and

perceptions towards STEM. These students had developed increasingly positive attitudes towards learning

science (Ricks, 2006) and computing (Heggen et al., 2012) and strengthened their confidence in mathematics

and science skills (Hylton et al., 2012). Furthermore, if attitudes towards science or STEM were overwhelming

positive, these attitudes did not change during the intervention (Duran et al., 2013; Heggen et al., 2012). One of

the studies found that students with higher self-efficacy initially collected more data using their scientific

inquiry skills in a virtual environment; however, after several visits self-efficacy no longer had an impact since

both low and high self-efficacy groups were collecting data equally. Also, self-efficacy did not have an impact

on the gathering of different sources within the virtual environment (Ketelhut, 2006). Another study found that

though the attitudes towards science and scientists were positive for males after the intervention, females were

just anxious towards science though their views towards science became more positive. The females also

reported that technology made science learning interesting, made the data more accurate, and helped with

visualization and understanding (Kim et al., 2011). Studies using engineering design-based practices showed

mixed results. In classrooms where engineering design and science content were integrated, attitudes towards

engineering were more positive as a result of the interventions with female students showing greater gains

(Schnittka, 2012). However, in a physical science and engineering environment using either hands-on or virtual

activities, the confidence level towards the ability to use engineering was the same for students participating in

the activities, but the female students’ confidence was consistently lower than males (Klahr et al., 2007). PBL

practices also showed mixed results. Students had positive attitudes towards their summer workshop experience

(Kampe & Oppliger, 2011) and STEM PBL practices had a positive influence on students’ behavioral

intentions, attitudes, and desire to learn (Lou, Liu, Shih, & Tseng, 2011). However, PBL did not have an impact

on students’ self-efficacy and caused it to decrease towards social studies (Brown et al., 2010). When hands-on

practices were used, both typical and Special Learning Disability students were satisfied with their STEM

intervention, were more interested in STEM, and had higher self-efficacy towards STEM. The Special Learning

Disability students, however, showed lower results on perceptions and attitudes towards STEM, but nonetheless

these results were still positive.

Learning Environments. Reform-based teaching and learning practices can promote students’ STEM learning in

an interactive and technology-based environment. Hudson et al. (2012) examined female students in an

engineering design-based learning environment. They found that their students exhibited the following

behaviors: 1) sought opportunities to clarify engineering terms that enabled them to enter into discourse around

their design and construction of an engineering prototype, 2) connected the task to their conceptual

understandings, 3) asked questions that advanced their project design and construction, and 4) discussed the

practicalities of their design and debated ideas without being judged. As a result of the type of learning

environment that enabled the female students to engage in those types of activities, the students were successful

in the engineering design-based project and had positive STEM-interactions.


Category 2: Informal STEM

Informal STEM learning environments generally provide occasions for scientific learning minus the time

constraints generally found in more formal settings (Hofstein & Rosenfeld, 1996). Informal learning settings

(i.e., museums, zoos, science centers, and science camps) often have the tools, resources, and expertise to

support STEM learning opportunities. The advantages of flexible time constraints in informal learning

environments facilitate greater chances to augment conceptual learning, reflection time, assessment of subject

matter, and informal discussions. These environments provide opportunities to facilitate student understanding

and transform science processes and concepts. Within informal science settings there are many opportunities for

scaffolding student science knowledge, attitudes, and science and STEM career options.

The TIMSS report suggested that because the United States places considerable emphasis on STEM instruction

and learning in formal settings (schools), they may overlook opportunities for rich informal science education

resources for reinforcement (Lee, 1998). Gerber, Cavallo, and Marek (2001) found that a large percentage of

students’ science learning can happen in informal learning environments outside formal classroom walls in

places like homes, camps, museums, after school programs, or in everyday experiences (Gerber et al., 2001).

When students engage in valuable informal STEM activities, they possess higher scientific reasoning abilities

than those who are not participating (Gerber et al., 2001).

National education groups have examined the impact of informal science on STEM knowledge. The National

Research Council (NRC, 1996) suggested that informal science education can complement and scaffold STEM

teaching and student learning. The implementation of programs intended to increase K-12 student involvement

in STEM, both formal and informal, was a priority for the mitigation of the shortage of students going into

STEM careers and measuring the effectiveness of these programs was necessary to be ensured of the programs’

impact (National Science Board [NSB], 2010). The NSB (2010) demonstrated the demand for equality across

students (race, gender, ethnicity) within STEM programs. The National Academy of Sciences (NAS

Committee, 2007) advocated for increasing participation in STEM of underrepresented minority and low-

income students. This participation may lead to further success of the STEM initiative. The NAS stated that

summer inquiry-based research programs were one venue for increasing participation of underrepresented

minority and low-income students.

Informal science learning has gained prominence as a possible contributor to student learning. These informal

environments also can be viewed as mechanisms for linking formal and informal science education efforts to

produce further collaborative endeavors focused on improving student learning in the STEM areas (Falk,

Osborne, Dierking, Dawson, Wenger, & Wong, 2012; NRC, 1996). Research has shown us that students and

adults pursue STEM understandings in and out of school using community resources (Bell, Lewenstein, Shouse,

& Feder, 2009).

Artifact Coding. The meta-synthesis on informal education resulted in 22 artifacts from the years 2006 through

2012 (see Table 2). The venues for the studies which were classified as informal learning ranged from mainly

after-school programs/clubs (8) to summer camps (14) with some including follow-up mentorships. The lengths

of the activities ranged from 3 hours (1), 40 hours (2), 80 hours (1), two days (1), three days (1), four days (1),

one week (1), two weeks (5), one month (2), seven weeks (1), ten weeks (2),18 months (1) to a longitudinal

three-year (1) study. A total of 1,835 participants were studied within 20 of the studies that provided


demographics with a range from 21 to 239 participants within each study. Some of these participants were from

underserved, underrepresented, low SES, and minority populations (6 studies/576 students); some were chosen

randomly or by lottery (3 study/390 students); while others only attended if they had a high aptitude, high

STEM interest and/or high scores in mathematics and science (6 studies/ 420 students). Many informal

activities had more than one focus with one using as many as seven different teaching pedagogies. These

pedagogies included the following: inquiry, hands-on activities, PBL, technologies, small and large group

activities, field trips, modules, roles models, discussions, collaboration, science projects, and mentors. Most of

these informal activities contained a combination of two or more of these pedagogical strategies. Subject areas

included science, mathematics, engineering, technology, STEM, music, and robotics with some containing more

than one of these specific subject areas in the informal settings. Most of these informal activities focused on

specific objectives: 1) increasing student knowledge and understanding, 2) increasing the STEM pipeline by

developing a wider breadth of understanding for STEM careers, and 3) improving student attitude and

confidence in STEM areas.

Findings. Results indicated that informal venues enhanced content knowledge and understanding, career and

major choices, and attitudes and confidence.

Content knowledge: Williams et al. (2007) showed that students’ physics content knowledge increased during a

robotics summer camp with middle school students. The majority of secondary students after being exposed to

science careers in a 4-day summer STEM camp increased on measures of science content, motivation,

knowledge, and development of science (Marle, Decker, Kuehler, & Khaliqi, 2012). Ricks (2006) demonstrated

statistically significant gains in participants’ science knowledge after a month-long camp including fieldtrips

and hands-on activities. Johnson, Hayden, Farmer, Hataway, Reynolds, and McConner (2013) using a summer

research program approach with talented underserved students found that 97% of them enriched their learning

in STEM.

Careers and majors: Camp students were more likely to enroll in STEM classes and choose STEM majors,

increasing the STEM pipeline. Hubelbank, Demetry, Nicholson, Blaisdell, Quinn, Rosenthal, and Sontgerath,

(2007) demonstrated that middle school girls in their camp chose a greater number of elective mathematics and

science courses in high school and more of these girls later chose engineering as a major. In one camp with

challenging engineering tasks, 67% of the students improved their GPA and 98% went to college (Hylton et al.,

2012). In an after school and summer camp (Mosina, Belkharraz, & Chebanov, 2012), as many as the 81.8% of

campers chose STEM majors while 95% of the talented secondary students in another 2-week camp (Johnson et

al., 2013) thought about majoring in STEM. Camp students in a ten-week informal environment were also more

motivated to consider a career in STEM (Zhe et al., 2010). Retrospectively, Ricks (2006) discovered that a

significant number of high-STEM interested previous camp participants selected STEM majors. In general,

after school and summer camps focusing on STEM topics had a positive effect on students’ impressions of

STEM careers and were highly associated with students wanting to pursue a STEM career. What is not clear is

how many students already possessed a predilection for STEM, and this might have been manifested in their

choosing to attend an after school or summer STEM camp.

Attitudes and confidence: As a result of informal STEM interventions students’ experiences improved their

attitudes about STEM and self-confidence within respective STEM disciplines. Instructors noted an

improvement in middle school students’ self-confidence during their one month camp (Hylton et al., 2012). A

significant increase in positive student attitude toward and knowledge about engineering careers was found by

Hirsch, Kimmel, Rockland, and Bloom (2006) during their engineering focused camp. Likewise, Zhe et al.

(2010) noted that high school students in their camp increased their self-confidence in STEM and improved


their independent research skills. Ricks (2006) saw an improvement in science attitudes after a 4-week summer

camp. Hispanic campers had higher gains in their increased confidence of success in and intentions toward

science careers as they developed a science social niche and greater motivation and attitudes toward science

(Marle et al., 2012). Hoyles, Reiss, and Tough, (2011) demonstrated that there should be a STEM club in every

school because they promoted positive impacts for students.

Some studies found both positive and negative effects. In one intervention, there was a drop in scores after one

semester because camp students faced their misconceptions about the topic, but after more instruction and

reflection on the misconception, there was increased scientific understanding (Miller, Ward, Sienkiewicz, &

Antonucci, 2011). There were no significant differences in engineering self-efficacy (Hubelbank et al., 2007);

however, qualitative data showed no gender differences in self-efficacy or knowledge gains – even though

certain sessions in the camp appealed to specific genders and their effect and knowledge gained were not

compromised (Marle et al., 2012).

Conclusions. Findings from these informal activities indicated that camps and after school clubs and activities

produce positive results. Most of these informal activities improved content knowledge about STEM subjects,

students’ attitudes toward STEM, and choice of STEM majors. Some recommendations included having STEM

clubs in every school. It was also suggested that girls preferred workshops with social aspects. Thus, it is

important to provide more activities to engage girls. Many of the studies suggested a support system should be

included for students and parents using mentors and/or role models if at all possible. The studies on informal

education show some encouraging results when the use of these role models and mentors were included in their

camps and semester long research projects.

Category 3: Teacher Factors

Teacher factors are attitudes, knowledge, beliefs, and practices of the teacher that impact student learning and

instructional practices. Many factors contribute to a student’s academic success, but research has suggested

that, among school-related factors, teacher factors matter most (Rand Corporation, 2012). Teacher factors

included, for example, teachers’ willingness to integrate technology (Yoon, 2010), or teachers’ perceptions of

the “type” of student that they would encourage to pursue engineering studies (Nathan, Tran, Atwood, Prevost,

& Phelps, 2010). Teacher factors also included classroom factors that were determined by the attitudes, beliefs,

or practices of the teacher, for example fostering students’ team skills through teacher designed collaborative

learning activities. Some overlap of the artifacts may be present among the categories—such as technology.

Artifacts for the meta-synthesis were categorized as teacher factors if the artifact, in some way, addressed

student outcomes related to teacher factors.

Artifact Coding. Twelve artifacts were retained from the literature search that pertained to the influence of

teacher factors on student success in STEM secondary education (see Table 3). Themes related to teacher

factors were identified through an iterative process of constant comparison among the 12 artifacts. Early in the

analysis process, tentative linkages were developed between theoretical themes and evidence of effectiveness as

demonstrated in student outcomes. As the coding progressed and themes for effective teacher factors for STEM

teaching and learning were categorized, the coding process shifted towards verification. Artifacts were revisited

and reviewed again as additional themes and evidence of teacher factors emerged.

Findings. Six principle themes for teacher factors were identified within the 12 artifacts analyzed: a) high

teacher content knowledge, b) deep understanding of STEM teaching practices (effective pedagogy), c) frequent

and effective integration of technology, d) effective use of team skills and collaborative learning, e) high teacher


self-efficacy, and f) emphasis on “deliberate instructional practice.” Evidence among the artifacts indicates

positive student outcomes (e.g., improved content knowledge, improved skills, or improved attitudes towards

science) for the six teacher factors. Each of the six themes is discussed in more detail in the following sections.

Teacher content knowledge: Teacher content knowledge had a substantial impact on student outcomes. Five of

the artifacts reviewed established a positive relationship between teachers’ content knowledge and students’

gains in content knowledge (Moskal, Skokan, Kosbar, Dean, Westland, Barker, & Nguyen, 2007; Silverstein,

Dubner, Miller, Glied, & Loike, 2009; Ragusa, 2012; Lambert, 2007; Hotaling et al., 2012). The higher the

teachers’ knowledge of the subject matter, the higher the students’ performance, after instruction, on concept-

inventory type tests. Instruments used to measure content knowledge included state standardized tests,

researcher developed content-specific tests, and passing rates on state mandated science or mathematics tests. In

four of the five cases exploring the influence of teacher content knowledge, the research on student outcomes

followed a teacher PD series that included a component of instruction dedicated to building content knowledge

in specific areas of current research and technology. All four PD series, however, also included in-depth

instruction of STEM teaching strategies (for example problem-based learning, inquiry, or engineering design).

In most cases, it was not possible to separate the effects of increased teacher content knowledge on student

outcomes from improved teaching strategies or other teacher factors. Clear evidence of this relationship,

however, was demonstrated in one study by Hotaling et al. (2012). Researchers grouped the teacher

participants by posttest content scores into three categories: low content knowledge, medium content

knowledge, and high content knowledge. They then compared student scores, following an instructional

intervention, by teacher category. High posttest teacher scores significantly predicted higher posttest student

scores, especially for the weaker students. In contrast, both weaker and stronger students had lower posttest

scores if their teachers’ scores were low, but the weaker students did very poorly when this was the case.

Teachers may take several years to translate new content knowledge into educational practices. Silverstein et

al. (2009) found that it took three to four years, following a PD experience focused on authentic research

experience, to see an increase in the teachers’ students’ scores on the state assessment for science. In years three

and four following the research experience for teachers, students of the participating teachers scored

significantly higher (10.1 percent) on the state assessment for science than students of non-participating

teachers. Integrated teacher content knowledge positively affected student learning. Teachers with a broad

content base tended to teach with a greater degree of science discipline integration, and the students of these

teachers outperformed students of teachers who taught with a discipline specific focus (Lambert, 2007).

Based on evidence from the artifacts reviewed, teacher content knowledge appeared to be the most critical

teacher factor contributing to student success in STEM secondary education. Teacher content knowledge was

most effective when it was broad, covered a range of disciplines, and integrated. STEM teachers should have

an understanding of concepts in mathematics, physical sciences, life sciences, and geoscience. And, once

teachers acquired new content knowledge, it took additional time to translate the content knowledge to

educational practices.

STEM teaching practices: Teachers’ understanding and effective use of STEM teaching practices had a positive

impact on student outcomes. A deep understanding of STEM teaching practices was explored by five of the 12

artifacts reviewed (Moskal et al., 2007; Silverstein et al., 2009; Mji, 2006; Finson, 2006; Ragusa, 2012).

Teacher instructional practices that modeled authentic scientific and engineering practices fostered improved

student content knowledge, increased skills, and improved attitudes towards STEM disciplines and careers.

These instructional practices included integration of content and problem-solving tasks (Moskal et al., 2007);

use of constructivist education practices (Silverstein et al., 2009); data-driven instruction (Silverstein et al.,


2009); and engineering inquiry-based learning (Ragusa, 2012). As an example, Silverstein et al. (2009)

described a PD activity in which teacher participants acted as researchers in a faculty mentor’s lab for 16 weeks

divided between two summers. Specific elements of the research experience, that Silverstein et al. (2009)

contributed to improved STEM teaching strategies, were the teachers’ lab management skills and confidence

gained from the research experience (self-efficacy), and the connections teachers made between studying an

authentic contemporary science problem and classroom applications of that problem. Teachers who were more

competent and confident in lab experiences and lab management skills were more likely to implement regular

demonstrations and experiments for their students. Students with regular exposure to experiments were then, in

turn, more likely to perform at a high level on the science portion of the National Assessment of Educational

Progress (NAEP) (Wenglinsky, 2000; Braun et al., 2009).

Unlike specific teaching strategies for STEM education, general teaching styles did not appear to have much

effect on students’ perceptions of science or scientists. Finson, Pedersen, and Thomas (2006) explored the

influence of teaching styles (didactic, conceptual, or exploratory) on students’ perceptions of scientists using

students’ drawings. It was hypothesized that students of the exploratory teaching style would have a broad

perception of scientists, whereas students of the didactic teaching style would have a stereotypical perception.

No correlation could be made between general teaching style and students’ perceptions of scientists. These

results, in combination with the other study results, inferred that specific teaching strategies, rather than

teaching styles, influenced student outcomes in STEM education. STEM teaching strategies should model

authentic scientific and engineering practices.

Integration of technology: The teacher’s ability to effectively integrate student use of technology is often

thought to have a positive impact on student outcomes in STEM education. Several artifacts mentioned teacher

use of technology during PD activities, but only three of the 12 artifacts analyzed for teacher factors directly

addressed the effects of teachers’ abilities and/or confidence for technology integration in the classroom on

student outcomes (Yoon & Liu, 2010; Hoyles et al., 2011; Hotaling et al., 2012). Hotaling et al. (2012) found

that teachers who were successful in utilizing technology (sensors) for the classroom had spent significant time

using the technology during a summer training period, had sufficient follow-up during the academic year with

support from the university, and had a high level of engagement with the learning community established

during the summer PD activity.

Yoon and Liu (2010) found a similar, but opposite trend. Teachers participating in a summer PD activity

demonstrated poor adoption of the technology used in their PD (simulation and visualization tools) due to lack

of time, lack of institutional support, and low engagement with a learning community. Yoon and Liu (2010)

made two critical distinctions in the study. First, communication technology differed from education

technology. Education technology was similar to “data-driven instruction” (Silverstein et al., 2009) where the

students were acting as scientists using technology as a tool to collect, organize, and interpret information.

Adoption and integration of education technology appeared to be more difficult. This may be due to the

simultaneous cognitive demands of teaching a new technology while also teaching skills for data analysis and

interpretation. Second, Yoon and Liu (2010) emphasized the importance of “negotiating the middle ground”

among the teacher, the institution (e.g., their school and district), and the students. Too often, contextual factors

were not considered when designing PD related to technology integration for secondary STEM education.

Among the artifacts analyzed for the meta-synthesis, integration of technology appeared to have a mixed effect.

Positive outcomes, due to technology, appeared to co-vary with other factors such as teacher content

knowledge, the presence of campus support, or active engagement within a learning community. Use of

education technology that encouraged data-driven instruction appeared to be the preferred method used in the


described PD activities. This may be because data-driven instruction reflects authentic science and engineering

practices.

Team skills and collaborative learning: Collaborative learning is a social process of knowledge building that

requires students to work as an interdependent team towards a clear objective resulting in a well-defined final

product, consensus, or decision (Wright et al., 2013). Collaboration requires team skills, and collaborative

learning requires structure and guidance from the teacher as the facilitator. Five artifacts out of the 12 artifacts

reviewed discussed teacher factors necessary to facilitate collaboration: such as grouping structure, grouping

process, or activity design (Duran et al., 2013; Moskal et al., 2007; Nag, Katz, & Saenz-Otero, 2013;

Nourbakhsh et al., 2005; Zhe et al., 2010).

Moskal et al. (2007) intentionally focused on collaboration during a PD event for teachers. Mathematics and

science teachers, who collaborated during the PD event to understand a STEM problem or to create a STEM

integrated lesson, were observed frequently utilizing collaborative learning and STEM integration in their

classroom by external evaluators. Students of the participating teachers were more likely than students in a

comparison group to indicate an interest in pursuing a STEM-related degree or career and demonstrated slightly

better scores on the state assessment test (Moskal et al., 2007). The effects of team skills and collaborative

learning cannot be isolated in this example from the effects of gains in teacher content knowledge or

pedagogical skills. Teachers, however, often teach the way they were taught. Collaborative learning in the PD

event increased the likelihood of the teacher fostering a collaborative learning environment in their classroom.

Poorly structured group activities may foster competition and isolation rather than team skills and collaboration.

Nag et al. (2013) explored a structured series of gaming competitions among students working remotely from

diverse locations across the United States. In order to incentivize collaboration, they used a model of “layered

collaboration.” Students worked together as a team, worked outside of their team in alliances, and worked with

opponents during competitions to achieve game objectives. The game was structured in an effort to encourage

collaboration within competition. Results of the tournament scores indicated alliances of teams scored higher on

average by more than one standard deviation than individual teams. In addition, the majority of game

participants agreed, through survey responses, that the competition had improved their leadership and team

skills, and had increased their interest in leadership. These results indicated the greater the degree of

collaboration, the greater the gains in student content knowledge and skills, in this case computer programming,

engineering-based problem solving, and leadership skills. Collaborative learning, a factor determined by a

teachers’ own learning experiences and skills for developing activities that encouraged students’ team skills,

supported positive student outcomes for STEM education.

Teacher self-efficacy: is a person’s belief in his or her ability to succeed in a particular situation. The most

effective way of developing self-efficacy, in a specific area, is through mastery experiences (Bandura, 1994).

The most effective way of developing self-efficacy for teaching STEM is to apply STEM content knowledge in

the context of real-world problem solving through authentic STEM research experiences (Silverstein et al.,

2009). Four of the 12 articles reviewed for teacher factors included evidence of a positive relationship between

teacher efficacy and improved STEM teaching and learning (Hoyles et al., 2011; Ragusa, 2012; Silverstein et

al., 2009; Yoon & Liu, 2010).

Silverstein et al. (2009) found that, following two summers of research experiences for teachers, the teachers’

use of constructivist education practices increased, their confidence to acknowledge their own gaps in content or

skills increased, and their skills in lab management skills increased. The authors attributed gains in teacher self-

efficacy for teaching STEM to the authentic contemporary research experiences and to the regard by the


scientists of the teachers as researchers (i.e., increased sense of professionalism). This increase in teacher self-

efficacy eventually translated to increased science scores for the students of the teacher participants. Similar to

other studies, however, it was not possible to attribute the positive student outcomes solely to teacher self-

efficacy. Rather, it inferred a combination of teacher content knowledge, pedagogical skills, and self-efficacy.

Deliberate instructional practice: is the intentional use of a set of activities designed to improve student

performance, challenge the learner, and provide feedback to the learner and the teacher through formative

assessment techniques (Marzano, 2011). Expertise is acquired by extensive engagement in practice activities

that are related to the broader learning goal, and individual differences in performance can often be attributed to

differences in the amount of deliberate practice (Ericsson, Krampe, & Tesch-Romer, 1993). Deliberate

instructional activities are designed by the teacher to help students improve specific aspects of their overall

performance. The deliberate practice is typically scheduled for a fixed period of time, on a regular schedule, and

of a limited duration. The activities must be appropriate, yet challenging, and the level of difficulty should allow

for successive refinement of skills through repetition, reiteration, and informative feedback (van Gog, Ericsson,

Rikers, & Paas, 2005).

Two of the artifacts reviewed for teacher factors contributing to success in STEM made an explicit association

between deliberate instructional practices and positive student outcomes. The deliberate instructional practices

investigated included the following: a) an emphasis on learning to read content-area text (Ragusa, 2012), and b)

teachers’ modeling of reflection (Hotaling et al., 2012).

An emphasis on learning to read content-area text, also referred to as disciplinary literacy (Shanahan &

Shanahan, 2008), was used to improve teacher performance and student learning in a quasi-experimental study

by Ragusa (2012). Disciplinary literacy, in contrast to content area literacy, is advanced literacy instruction

embedded within content-area classes. The difference is that content literacy emphasizes general reading and

communication techniques that a novice might use to make sense of text, while disciplinary literacy emphasizes

the unique strategies used by experts to read, understand, and communicate within a specific discipline.

Although Ragusa (2012) did not mention the term disciplinary literacy, the reading strategies taught were

utilized specifically for understanding and communicating STEM-disciplinary text and could be considered

instruction in disciplinary literacy.

In order to instruct students on the practices needed to understand the structure and use of scientific

informational text, teachers were taught, through a PD program, how to utilize question generation strategies

(QGS) and question answer relationship strategies (QAR) (Raphael & Au, 2005) specific to STEM-disciplinary

text (Ragusa, 2012). The teachers (n=53) then integrated these deliberate practices in their classroom instruction

with their students (~ 5,000). The result was higher student science knowledge, improved student science

literacy, and increased enthusiasm for science. Other factors, individually or synergistically, may have

contributed to these outcomes. In addition to deliberate practice on learning to read STEM-specific text, the PD

also incorporated teacher content knowledge and STEM pedagogical practices (specifically inquiry instruction).

A second deliberate instructional practice exhibiting significant positive student outcomes was that of teachers

modeling reflection. Reflection has been defined as a mode of understanding that requires “becoming critically

aware of one’s own tacit assumptions and expectations, and those of others, and assessing their relevance for

making an interpretation” (Mezirow, 2000).

Hotaling et al. (2012) described a cohort of teachers that was encouraged to model reflection by discussing

pretest results for algebra and electricity units with their students. The pretest results for each unit were put


online and were immediately available to be shared, aggregated by class, with the students. The results were

displayed in a graphical format. About half of the teachers participating in the study reported that they discussed

the pretest results with their students. Following the units of study, students’ posttest results (for content

knowledge) were compared by teachers that discussed the pretest results versus those that did not and by

weaker versus stronger students. Discussing the results was “correlated with significantly higher scores for both

stronger and weaker students. Even more important, it had the greatest impact on the weaker students, in some

cases almost equalizing the scores between the two groups” (Hotaling et al., 2012, p. 29). The authors did not

refer to the strategy of discussing pretest results as “modeling reflective thinking.” However, through discussion

of pretest results teachers were encouraging students to become aware of assumptions and expectations and to

self-assess their existing knowledge of the content—this was reflection.

Specific strategies or methods the teachers used to discuss the pretest results were not described. A better

understanding of how they facilitated reflection through discussion of the pretest results would have been

beneficial. Deliberate practice with disciplinary literacy, utilizing QGS and QAR strategies in order to model

and facilitate student reflection, could feasibly yield significant positive outcomes for STEM learning.

Conclusions. Six principle themes for teacher factors were discussed in this section: a) high teacher content

knowledge, b) deep understanding of STEM teaching practices (effective pedagogy), c) frequent and effective

integration of technology, d) effective use of team skills and collaborative learning, e) high teacher self-

efficacy, and f) emphasis on “deliberate instructional practice.” The central point condensed from review of the

12 artifacts related to teacher factors is that teacher content knowledge appears to be the most important teacher

factor impacting student outcomes. Other essential points related to teacher content knowledge include the

following: a) teacher content knowledge is a significant predictor for student academic success, b) gains in

teacher content knowledge are typically facilitated through summer PD activities, c) many of the PD programs

span a long period of time (up to two years) and include support for the teachers during the academic year, and

d) it takes a long time (two plus years) for teachers to transfer new content knowledge into actions and practices

in the classroom.

Effective STEM pedagogy appears to be the second most important teacher factor impacting student outcomes.

Teacher content knowledge and effective STEM pedagogy were often taught through the same PD program,

and positive student outcomes were attributed to both. Therefore, it is not feasible to state the impact of

improved STEM pedagogical practices separate from the influence of increased teacher content knowledge.

Effective STEM instructional practices included integration of content and problem-solving tasks (Moskal et

al., 2007); use of constructivist education practices (Silverstein et al., 2009); data-driven instruction (Silverstein

et al., 2009); and engineering inquiry-based learning (Ragusa 2012). Results indicate these instructional

practices foster improved student content knowledge, increased skills, and improved attitudes towards STEM

disciplines and careers.

Technology adoption and integration by the teacher appears to have a mixed effect on student outcomes.

Positive outcomes, contributed to technology, appear to co-vary with other factors such as teacher content

knowledge, the presence of campus support, or active engagement within a learning community. Use of

education technology that encourages data-driven instruction is the preferred method used in the described PD

activities. This may be because data-driven instruction reflects authentic science and engineering practices.

However, this is also the type of technology that has a low adoption rate among teachers due to factors such as

lack of time, lack of institutional support, and low engagement with a learning community. Positive student

outcomes as a result of teacher integration of educational technology depend more on the content knowledge,

self-efficacy, and STEM instructional methods of the teacher than they do on the technology itself.


Team skills and collaborative learning were discussed in five of the 12 artifacts reviewed for teacher factors. Of

these five, three made casual references to the importance of collaboration and how they structured

collaborative teams within the outreach and/or PD programs, but they did not measure student outcomes based

on levels or degree of collaboration. One artifact, however, described in detail how they structured teams,

activities, and criteria for the primary purpose of fostering multiple layers of collaboration while discouraging

passive cooperation and negative competition (Nag et al., 2013). They categorized the participants based on the

layer of collaboration they had achieved (up to 3 layers) and compared the score results among the categories.

They found that the greater the degree (or layer) of collaboration, the greater the gains in student content

knowledge and skills. Collaborative learning is a teacher factor that can enhance STEM student outcomes.

One artifact attributed gains in teachers’ self-efficacy to positive student outcomes (Silverstein et al., 2009).

However, others inferred the influence of teacher confidence for content, teacher confidence for a specific

technology, or teacher confidence for managing a lab/classroom to positive student outcomes. Although teacher

self-efficacy appears to be an important teacher factor for STEM success, the artifacts reviewed inferred that it

was strongly associated with teacher content knowledge and STEM instructional practices.

Deliberate instructional activities were designed by the teacher to help students improve specific aspects of their

overall performance. Deliberate teacher practices discussed by two of the artifacts reviewed for teacher factors

were an emphasis on learning to read content-area text and teacher modeling of reflection. Both artifacts

presented results from quasi-experimental research indicating significant positive student outcomes associated

with these practices.

Teacher factors are a significant contributor to student academic success in STEM. Teacher content knowledge

is the most crucial of the teacher factors, followed by STEM instructional practices (or pedagogy). Integration

of technology, teacher self-efficacy, and team-building skills are closely associated with teacher content

knowledge and STEM instructional practices. PD programs, therefore, should concentrate on building STEM

content knowledge in an authentic context (e.g., research) paired with instruction/modeling on how to modify

the content and research practices into STEM instructional practices. PD that prioritizes building teacher

content knowledge should include multiple sessions over an extended period of time with support provided

during the academic year. Lastly, program directors and stakeholders should not expect an immediate direct

impact on student learning resulting from increased teacher content knowledge. It takes considerable time to

transfer new content and skills into classroom practices.

Category 4: Technology

Artifact Coding. There were 25 articles, presentations, posters, or other documents that discussed integrating

technology with mathematics, science, or engineering, resulting from the search for STEM teaching and

learning artifacts from 2005 through 2013 (see Table 4). There are doubtless many more articles that describe

the use of technology in teaching mathematics, science, or engineering, but they did not meet the search criteria

and thus were not included. For example, the search terms would not have identified an article about

simulations in science or animations in mathematics. There are many technology-related terms that were not

included in the search; the main focus was on STEM integration. Determining the categorization of the studies

and activities that used technology was problematic because technology can cut across all types of educational

settings (formal and informal), was used in different teaching strategies (e.g., problem-based learning, PBL,

inquiry, engineering design), was used for different purposes (e.g., increase content knowledge, improve


attitudes towards STEM), and there were many different types. There were no other particular themes that

emerged common among studies using technology other than the integration of STEM-related subjects. The

decision was made to discuss these artifacts by type of technology used though many used a combination of

different types of technology.

Robotics. More than half (13 of 25) of the artifacts that discussed the use of technology as part of STEM

integration described robotics projects. Three of the studies addressed content knowledge, eight focused on

technology skills, nine discussed STEM interest, and nine others concentrated on 21st century skills. All of the

robotics projects were implemented in informal environments, in summer camps, and after school programs.

Long-term goals were generally related to improving student attitudes toward STEM and increasing student

interest in STEM majors and careers (Adamchuk, 2009; Duran et al., 2013; Javidi & Sheybani, 2010; Kampe &

Oppliger, 2011; Moskal et al., 2007; Nugent, Barket, White, & Grandgenettt, 2011; Stephen, Bracey, & Locke,

2012; Strautmann, 2011; Welch, 2010). Increased content knowledge in mathematics, science, and engineering

was a goal of several of the robotics programs, but whether or not goals were met was generally measured

through student self-reports (Adamchuk, 2009; Moskal et al., 2007; Nag et al., 2013). Students were provided

opportunities to improve 21st century skills, such as problem solving and collaboration, by using robotics to

solve the problems presented to them (Adamchuk, 2009; Duran & Sendag, 2012; Duran et al., 2013; Kampe &

Oppliger, 2011; Moskal et al., 2007; Nag et al., 2013; Nourbakhsh et al., 2005; Nugent et al., 2011; Welch,

2010). Increasing confidence in programming computers or handheld devices was a positive outcome of

robotics activities (Nourbakhsh et al., 2005), while some programs sought only to increase student use of and

confidence in the use of technologies in general (Duran & Sendag, 2012; Duran et al., 2013; Javidi & Sheybani,

2010; Kampe & Oppliger, 2013; Moskal et al., 2007; Nugent et al., 2011; Strautmann, 2011).

Computer Simulations. The use of simulations for training in the corporate world as well as for educational

applications has increased as the cost of computers has decreased. Simulations in the real world save time and

money while effectively training employees for potentially dangerous or complicated tasks. In the educational

environment, simulations provide experiences to students they would not normally be able to access. There

have been a few literature reviews on the ways simulations assist in student learning, but they were not focused

on STEM learning. In May 2013, a brief report was published, introducing an upcoming meta-analysis on

computer-based simulations for STEM learning; however, the final report is not yet available. The preliminary

findings indicated that simulations did improve student learning, and additional scaffolding increased the effect

(D’Angelo, Rutstein, Harris, Haertel, Bernard, & Borokhovski, 2013).

Nine of the 25 documents examined for this report described projects that used simulations of some type. Many

were not well characterized; however, one specifically mentioned robotics (Nag et al., 2013) and one involved

simulations in games (Sumners, Handron, & Jacobson, 2012). One project used a five-week simulation project

to address national standards for middle school students in persuasive writing, social studies, and science. The

simulation addressed water resources and solving a crisis in the availability of clean water. For students in both

Connecticut and Chicago, there were positive changes in self-efficacy in technology use. The project originated

with a single five-week simulation and has grown to a 14-week project with five different simulations (Brown

et al., 2010; Lawless, Brown, & Boyer, 2011).

Another simulation project was an action research project designed with a specific mathematical academic goal

in mind—to deepen understanding of scaling and proportional reasoning among middle school students.

Students participated in a four-week web-based interactive activity supported by group work and discussions.

Students believed the activities improved their understanding of scaling and proportions, increased their

confidence in their abilities to be successful in mathematics, and enjoyed the simulations (Chapman, 2012).


Simulations in the form of collaborative games were used to increase knowledge about the geological clock

(Johnson-Glenberg, Birchfield, Savvides, & Megowan-Romanowicz, 2011), to compare results of learning in

physical versus virtual hands-on science activities (Kalhr, Triona, & Williams, 2007), and to increase student

interest in STEM careers (Strautmann, 2011; Tan, Ngo, Chandrasekaran, & Cai, 2013).

Curricular units aligned with standards for high school biology and physical science included a summer

workshop for teachers, a small contingent of students for a summer pilot, and implementation in the classroom

with continued support for teachers. There were a number of difficulties encountered in such a project. First of

all, teachers needed to be familiar with and comfortable using the educational technologies, connecting the

activities with high school content, understanding nanotechnology as a different science rather than an addition

to existing sciences. They likewise required access to download necessary software on classroom computers.

Overall, teachers reported increased comfort levels with technology, but students did not (Yoon & Liu, 2010).

Handheld Technology. Besides the handheld calculator, many projects described in the artifacts collected used

other handheld devices that worked with either calculators or computers to collect data. The handheld

technologies used included cameras, motion sensors, various types of probes, global positioning systems (GPS),

geographic information systems (GIS), and other data collection devices. Often the handheld devices were used

to collect data to be analyzed on the graphing calculator or computer. Many projects were not specific as to how

and through what means data were analyzed, but the authors did state that they were using GPS or GIS devices

or other devices to collect information about the environment, such as conditions in the water or the air.

Among the artifacts that discussed the use of handheld technology, all but two also included robotics,

programming, or both. The projects that involved robotics with handheld devices focused on increasing student

interest in STEM subjects and/or STEM majors and careers. In addition, students had opportunities to improve

21st century skills, particularly problem solving and collaboration (Adamchuk et al., 2009; Kampe & Oppliger,

2011; Moskal et al., 2007; Stephen et al., 2012). Several studies included providing students with experience in

programming as one of their goals (Adamchuk et al., 2009; Hotaling et al., 2012; Moskal et al., 2007).

In one program, students first constructed a set of sensors from common electronics components, as teachers led

them through the science and mathematics principles necessary to build the sensors. They then tested the

sensors, programmed them, and used them to collect data to monitor water quality. Later, they used

programming skills to collect data and send it wirelessly to computerized sensor stations and formed a wireless

sensor network. Teachers were provided with 120 hours of PD to support the implementation of the integrated

STEM curriculum (Hotaling et al., 2012).

The use of motion sensors to create position-time, velocity-time, and acceleration-time graphs provided

interactive learning for students through visualization of science and mathematics concepts. Students gained

more content knowledge by use of the open source SmartGraphs software, and teacher experience with the

sensors and software contributed to even greater gains (Kay, Zucker, & Staudt, 2013).

The development of the Mobile Application Development for Science (MAD Science) curriculum centered

around using sensors to gather and analyze data from the environment. The program was designed for middle

school students in an after-school program. Students collected data samples related to a civic problem in the

community, analyzed the results, and presented their findings. The main goal of the program was to increase

student engagement and continued interest in science and technology (Heggen et al., 2012).


Programming. Seven of the nine articles that mentioned programming used it in connection with a robotics

project, and those were discussed briefly in the section on robotics. One of the remaining articles, where

students constructed sensors as well as programming them, was described in the previous section (Hotaling et

al., 2012). Another study described a one-week summer music technology program in which students worked

with computer-based instruments. They built a speaker and learned about sound transduction, sound waves and

sinusoidal function, frequencies and the inverse relationship to wave length, and computerized interfaces with

music. The purpose was to engage and interest students under-represented in STEM through music, an integral

part of their lives (Kim et al., 2011).

Other Technologies. One artifact involved a more traditional use of technology through computer-aided

instruction. Rather than just using the computer instruction to directly teach content, the system was used to

confront students with misconceptions about decimal numbers. Feedback was provided for wrong answers to

assist students in recognizing illogical concepts that they held. Students’ understanding and use of decimal

numbers improved following the instruction, and the retention of knowledge was high (Huang, Liu, & Shiu,

2008).

Another project’s goal was to increase middle school students’ interest in STEM careers by providing

information about what STEM professionals do in their jobs. Video interviews of STEM professionals were

created and edited for appeal to teenagers and were shown over an eight-week period. STEM career interests

were measured before, after half of the videos were shown, and after all of the videos were shown. The effect

was small but positive, and researchers noted that combining the videos with other methods might have

produced stronger results (Wyss, Heulskamp, & Siebert, 2012).

Findings. Of the technologies described in the 25 artifacts, robotics was used most frequently; however, this

type of technology was used only within informal settings, not in secondary classrooms. Many other types of

technologies were incorporated into the robotics after school programs, and summer camps and competitions,

such as simulations, hand-held devices, and programming. Simulations were the second most frequently used

technology application, providing virtual access to real-life situations and games for learning. Technology use

enabled students to demonstrate positive gains in content knowledge, enhance technology related skills,

experience a variety of teaching strategies, contemplate entering STEM majors or careers, become interested in

and enjoy STEM-related subjects, and develop 21st century skills.

Conclusions. There were considerably more studies that had stated goals of engaging students in STEM

learning, growth in knowledge and comfort with various technologies, increasing interest in STEM studies and

careers, and improving student attitudes about STEM courses than there were with main goals to increase

students’ mathematics or science content knowledge. In general, long-term projects working on relevant

problems produced positive results for student interest in STEM majors and careers, but these projects were

conducted in informal environments, usually with voluntary involvement. For studies that addressed content

knowledge, the strongest results were seen in science content, with fewer gains in mathematics content.

For projects that involved STEM teaching and learning in the formal school setting, teacher PD was generally

addressed. Teacher content knowledge seemed to be the most critical piece in success, but that knowledge was

minimally useful without the pedagogical knowledge to implement the integrated STEM curriculum. At times,

the limitations in the school settings to access and fully implement the necessary technology diminished the

success of the project. With a high level of fidelity of implementation, goals related to STEM teaching and

learning were realized.


Category 5: School Factors Influencing STEM Learning

Coding Artifacts. Ten artifacts of the meta-synthesis were based on school factors (see Table 5). The general

methodological choice was quasi-experimental. The samples were small, highly idiosyncratic, and without clear

criteria related to any modern reporting practices (American Educational Research Association, 2006; American

Psychological Association, 2001). Three studies used nationally representative data while the rest used

convenience samples (Legewie & DiPrete, 2012; Lynch, 2009; Maltese, 2008). Of the studies using convenient

sampling, only two included sufficient information about the sample in hand to determine a population and

none explicitly compared their sample to the population. The study durations were generally short with low

intensity. The studies ranged from 90 minutes to 16 hours and only one study included the administered survey,

while most studies did not use outcome measures but relied on informal interviews or researcher notes. Of the

four studies that incorporated some type of curriculum, only one reported on providing PD for teachers and

none of the studies included student-learning outcomes.

Conclusions. Among those factors, course taking in high school, particularly in 11th and 12th grades, were

highly associated with earning a STEM degree in 5 years. Students who took a calculus course were most

likely to earn a STEM degree. With regard to taking science courses, most students earning a STEM degree had

taken physics as compared to only 9% who took chemistry alone (Tyson et al., 2007). Contradictorily, a

nationally representative sample indicated that when controlling for student characteristics, neither parental nor

high school characteristics had any statistically significant effect on post-secondary STEM enrollment. Rather,

gender, race, and science and math achievement and interest were the strongest predictive factors in STEM

enrollment when holding parental and school characteristics constant. Once students entered post-secondary

education, Blacks, Hispanics, and Asians were all more likely to major in an NCES-defined STEM field than

Whites (Lynch, 2009).

The results for course taking were different for males and females. Females took more rigorous mathematics

and science courses in high school. However, only half the number of females went on to earn a STEM degree

as compared to males (Legewie & DiPrete, 2012; Tyson et al., 2007). This trend was persistent regardless of

which combination of high school mathematics and science courses were taken (Tyson et al., 2007). This was

aligned with students who had a strong identity in a particular field and those who also had a strong penchant to

claim an interest in a STEM field (Hazari, Sonnert, Sadler, & Shanahan, 2010).

There were apparent differences by ethnicity, shedding new light on the issues of equity. Nearly 33% of Asian-

American students earned a STEM degree, only 12.8% of White students did so, but Hispanic and Black

students tended to perform on par with their counterparts once in college. When considering SES, students on

free lunch in high school who obtained a baccalaureate degree were more likely to be in STEM fields than those

students who paid for their lunch (Tyson, et al., 2007).

Whole school STEM programs that applied the engineering design process and addressed Standards for

Technological Literacy developed by the International Technology and Engineering Educators Association

(ITEEA) (Goodwin, Brawley, Ferguson, Price, & Whitehair, 2013) achieved about on par with non-whole

STEM schools. Whole school programs required PD because teachers had little or no prior experience with

engineering design or the applications of various technologies. After two four-day PD sessions that included

hands-on experience with tools, electronics, materials selection, and STEM lesson planning, students were

engaged and enthusiastic, and all teachers incorporated the engineering design process, Standards for


Technological Literacy, and cross-curricular activities on a regular basis. Students indicated that the Whole-

School STEM program helped them learn, increased their interest in school, and increased the frequency with

which they used STEM tools. In particular, underachieving students were particularly responsive to STEM

activities, and demonstrated levels of leadership, enthusiasm, and success that were much higher than in other

academic subjects. Building a STEM identity was within the capabilities of the teacher by focusing the class on

conceptual understanding, conducting labs that addressed students’ beliefs about the world, discussing currently

relevant science and the benefits of being a scientist, and encouraging students to take science classes. The first

finding was that a conceptual understanding focus was positively related to physics identity, dovetailed with a

host of other research that found a link between physics conceptual understanding and performance (Hazari et

al., 2010).

Other STEM Interventions

The use of attitudinal and dispositional change agents was the dominant factor represented. Students who had

the disposition or inclination to consider post-secondary STEM matriculation were more likely to have a

positive attitude toward a STEM career after voluntarily participating in 16 hours per year of workshops

conducted by university STEM professors (Cantrell et al., 2009).

The following was the original question driving this literature review: What are the promising practices in

middle and high-school STEM teaching and learning? The most promising practice identified was increasing

teacher content knowledge, followed by improving teachers’ pedagogical practices for STEM teaching and

learning. Many of the studies focused on students’ STEM learning followed by a PD event for teachers that

focused on teacher content knowledge, reform-based STEM teaching practices, and/or STEM research

experience for teachers. Five artifacts reviewed established a positive correlation between teacher content

knowledge and student performance. However, it was difficult to attribute student performance solely to teacher

content knowledge. Direct empirical evidence of this relationship was demonstrated in one of the five studies

(Hotaling et al., 2012). Teacher content knowledge had a substantial impact on student outcomes.

Promising pedagogical reform-based practices included PBL, problem-based learning, inquiry-based learning,

engineering design, hands-on practices, and the legacy cycle. Of these reform-based practices, inquiry was the

most often used followed by engineering design. One of the main goals of the reform-based teaching and

learning strategies for 12 out of 23 artifacts appeared to be increasing students’ content knowledge, and—for

the most part—student content knowledge was enhanced in programs focused on that objective. Of the 12

artifacts focused on improving student content knowledge, ten showed a positive effect. Of those ten studies,

four utilized inquiry, three utilized engineering design, one used PBL, one used the legacy cycle, and one

focused on hands-on strategies. Inquiry represents the most often studied reform-based practice, and these

results indicate that it may also be the most promising.

However, caution is necessary. Inquiry was poorly defined in several of the studies, and it is sometimes used as

a catchall term. In addition, even though it may indicate average positive gains in content knowledge, prior

research has shown these gains to be highly heterogeneous with different outcomes for low versus high

performers. In other words, students struggling with STEM courses do not seem to perform as well with inquiry

as students with a high interest or aptitude for STEM (Kirschner et al., 2006; Tai & Sadler, 2009). Keeping in

mind the need for quality differentiated instruction, reviewers were especially interested in promising practices

for less able and less motivated learners. Engineering design appears to show the most promise in this area with

research indicating that low achieving African Americans gained substantially using this reform-based practice

(Mehalik et al., 2008).


Considerably more of the artifacts reviewed had stated goals other than increasing students’ mathematics or

science content knowledge. Most of those reviewed had the following non-content related objectives: engaging

students in STEM learning, expanding students’ knowledge and/or comfort with various technologies,

increasing interest in STEM studies and careers, and improving attitudes about STEM. In general, long-term

projects working on relevant problems produced positive results for student interest in STEM majors and

careers, but these projects were most often conducted in informal environments, usually with voluntary

involvement. Among the 22 items reviewed related to informal STEM learning, findings support the use of

informal learning environments to enhance content knowledge, skills, attitudes, and interest for STEM.

However, caution is again necessary. Equity issues related to “closing the gaps” involve strategies for access to

equal participation as well as strategies for access to equal success. Even though STEM summer camp

opportunities and after school activities attempt to recruit underrepresented and/or low achieving students, the

reality is that access to informal STEM activities is often based on students’ expressed interests, prior academic

achievement, teacher recommendation, time availability and flexibility, travel availability and flexibility, and

overall levels of ambition/motivation. It is difficult, then, to attribute success to an elective program when

factors such as teachers’ expectations and student motivation have not been controlled for. Promising practices

that may be gleaned from the studies on informal education could be transitioned and adopted in formal

education settings such as the typical classroom. These practices include students identifying and solving

authentic problems, content-focused field trips, interactions with experts in STEM fields, and long-term projects

that require true STEM integration terminating with results and conclusions.

For projects that involved STEM teaching and learning in the formal school setting, teacher PD was generally

addressed. Teacher content knowledge seemed to be the most critical piece in success, but that knowledge was

minimally useful without the pedagogical knowledge to implement the integrated STEM curriculum. At times,

the limitations in the school settings to access and fully implement the necessary technology diminished the

success of the project. With a high level of fidelity of implementation, goals related to STEM teaching and

learning were realized.


References

*Adamchuk, V., Nugent, G., Barker, B., & Grandgenett, N. (2009). The use of robotics, GPS and GIS

technologies to encourage STEM-oriented learning in youth. Proceedings of the 2009 Midwest Section

Conference of the American Society for Engineering Education. University of Nebraska, Lincoln.

*Adamchuk, V., Nugent, G., Barker, B., & Grandgenett, N. (2009). The use of robotics, GPS and GIS

technologies to encourage STEM-oriented learning in youth. Teacher Education Faculty Proceedings &

Presentations. Paper 12. University of Nebraska, Lincoln.

Anderson, R. D. (2002). Reforming science teaching: What research says about inquiry. Journal of Science

Teacher Education, 13(1), 1-12.

Aulls, M. W., & Shore, B. M. (2008). Instruction and inquiry instruction (Vol. 1). New York, NY: Erlbaum.

Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human behavior, 4 pp. 71-81.

New York, NY: Academic Press.

Bell, P., Lewenstein, B., Shouse, A. W., & Feder, M. A. (2009). Learning sciences in informal environments:

People, places and pursuits. Washington, DC: National Academies Press.

*Brown, S. W., Lawless, K. A., Boyer, M. A., Cutter, A., Brodowinska, K., O'Brien, D., Williams, G., Powell,

N., Enriquez, M. F., & Mullin, G. (2010). Global Ed2: Using PBL to promote learning in science and

writing.

Bybee, R. W. (2010). Advancing STEM education: A 2020vision. Technology and Engineering Teacher, 70(1),

30-35.

*Cantrell, P., & Ewing‐ Taylor, J. (2009). Exploring STEM career options through collaborative high school

seminars. Journal of Engineering Education, 98(3), 295-303.

*Chapman, L. A. (2012). Active learning of scale and proportionality: A STEM approach. (Master's thesis).

Montana State University-Bozeman, Graduate School.

D’Angelo, C., Rutstein, D., Harris, C., Haertel, G., Bernard, R., & Borokhovski, E. (2013). Review of computer-

based simulations for STEM learning in K-12 education. Menlo Park, CA: SRI International.

Dugger Jr, W. E. (2010). Evolution of STEM in the United States. In 6th Biennial International Conference on

Technology Education Research in Australia. Retrieved from http://www.

iteea.org/Resources/PressRoom/AustraliaPaper.pdf

*Duran, M., & Sendag, S. (2012). A preliminary investigation into critical thinking skills of urban high school

students: Role of an IT/STEM program. Creative Education, 3(2), 241-250.

*Duran, M., Höft, M., Lawson, D. B., Medjahed, B., & Orady, E. A. (2013). Urban high school students’

IT/STEM learning: Findings from a collaborative inquiry-and design-based afterschool program.

Journal of Science Education and Technology, June, 1-22.

*English, L., Hudson, P., & Dawes, L. (2011). Perceived gender differences in STEM learning in the middle

school. International Journal of Engineering Education, 27(2), 389–398.

Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of

expert performance. Psychological Review, 100, 363-406.

Falk, J., Osborne, J., Dierking, L., Dawson, E., Wenger, M., & Wong, B. (2012). Analysing the UK science

education community: The contribution of informal providers. London, UK: Wellcome Trust.

*Finson, K. D., Pedersen, J. E., & Thomas, J. (2006). Comparing science teaching styles to students' perception

of scientists. Faculty Publications: Department of Teaching, Learning and Teacher Education. Paper 88.

University of Nebraska-Lincoln.

Gerber, B. L., Marek, E. A., & Cavallo, A. M. L. (2001). Development of an informal learning opportunities

assay. International Journal of Science Education, 23(6), 569-583.


*Goodwin, M., Brawley, M., Ferguson, P., Price, D., & Whitehair, J. (2013). A whole-school approach to

STEM education: Every child, every class, every day. 3rd IEEE Integrated STEM Education

Conference, Princeton, NJ.

Grandy, R., & Duschl, R. (2007). Reconsidering the character and role of inquiry in school science: Analysis of

a conference. Science & Education, 16(2), 141-166.

Hanover Research. (2011). K-12 STEM education overview. Washington DC: Author.

Haury, D. L. (1993). Teaching science through inquiry. Striving for excellence: The national education goals

(Vol. 2). Washington, DC: Educational Resources Information Center.

*Hazari, Z., Sonnert, G., Sadler, P. M., & Shanahan, M. C. (2010). Connecting high school physics experiences,

outcome expectations, physics identity, and physics career choice: A gender study. Journal of Research

in Science Teaching, 47(8), 978-1003.

*Heggen, S., Omokaro, O., & Payton, J. (2012). MAD Science: Increasing engagement in STEM education

through participatory sensing. The 6th International conference on mobile ubiquitous computing,

systems, services and technologies. Barcelona, Spain.

*Hirsch, L. S., Kimmel, H., Rockland, R., & Bloom, J. (2006, October). Using pre-engineering curricula in

high school science and mathematics: A follow-up study. Presented at the ASEE/IEEE Frontiers in

Education Conference, San Diego, CA.

Hofstein, A., & Rosenfeld, S. (1996). Bridging the gap between formal and informal science learning. Studies in

Science Education, 28, 87-112.

*Hotaling, L., Lowes, S., Stolkin, R., Lin, P., Bonner, J., Kirkey, W., & Ojo, T. (2012). Sense it: Teaching

STEM principles to middle and high school students through the design, construction and deployment of

water quality sensors. Advances in Engineering Education, 3(2), 1-34.

*Hoyles, C., Reiss, M., & Tough, S. (2011). Supporting STEM in schools and colleges in England: A report for

universities. London, UK: Universities UK.

*Huang, T., Liu, Y., & Shiu, C. (2008). Construction of an online learning system for decimal numbers through

the use of cognitive conflict strategy. Computers & Education, 50, 61-76.

*Hubelbank, J., Demetry, C., Nicholson, S. E., Blaisdell, S., Quinn, P., Rosenthal, E., & Sontgerath, S. (2007).

Long term effects of a middle school engineering outreach program for girls: A controlled study. In

Proceedings, American Society for Engineering Education Annual Conference & Exhibition.

*Hudson, P., English, L. D., Dawes, L. (2012, November). Catapulting into STEM Education: Female students’

conceptual understandings of a middle school engineering project. Paper presented at the STEM2012

conference, Beijing, China.

*Hylton, P., Otoupal-Hylton, W., Campbell, W., & Williams, D. (2012). Science bound: Success story of STEM

education. Paper presented at the ASEE/IEEE Frontiers in Education Conference, 1-5.

*Javidi, G., & Sheybani, E. (2010, October). Making youth excited about STEM education. CCSC:

Northwestern Conference, Newberg, OR.

*Johnson, D., Hayden, L., Farmer, C., Hataway, J., Reynolds, A., & McConner, M. (2013). The impact of 2006-

2012 CRESIS summer research programs that influence student’s choice of a stem related major in

college. Conference, 1-5.

*Johnson-Glenberg, M. C., Birchfield, D., Savvides, P., & Megowan-Romanowicz, C. (2011). Semi-virtual

embodied learning-real world STEM assessment. In L. Annetta & S. C. Bronack (Eds.), Serious

educational game assessment (pp. 241–257). Rotterdam, Netherlands: Sense.

*Kampe, J., & Oppliger, D. E. (2011, June). Effectiveness of team-based STEM project learning to recruit

minority high school students to STEM. Paper presented at the annual conference and exposition of

American Society for Engineering Education, Vancouver, B.C., Canada.


*Kim, Y., Batula, A., Migneco, R., Richardson, P., Dolhansky, B.,Grunberg, D., Morton, B., Prockup, M.,

Schmidt, E., & Scott, J. (2011) Teaching stem concepts through music technology and dsp. Paper

presented at the Digital Signal Processing Workshop and IEEE Signal Processing Education Workshop

(DSP/SPE).

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work:

An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based

teaching. Educational Psychologist, 41(2), 75-86.

*Klahr, D., Triona, L. M., & Williams, C. (2007). Hands on what? The relative effectiveness of physical versus

virtual materials in an engineering design project by middle school children. Journal of Research in

Science Teaching, 44(1), 183-203.

*Klein, S. S., & Sherwood, R. D. (2005). Biomedical engineering and cognitive science as the basis for

secondary science curriculum development: A three year study. School Science and Mathematics,

105(8), 384-401.

*Lambert, J. (2007). High school marine science and scientific literacy: The promise of integrated science

course. International Journal of Science Education, 28(6), 633-654.

Lantz, H. B., Jr. (2009). Science, technology, engineering, and mathematics (STEM) education: What form?

What function? Retrieved from http://www.currtechintegrations.com/pdf/STEMEducationArticle.pdf

*Lawless, K., Brown, S. W., & Boyer, M. (2011). A socio-scientific approach to STEM education: The

GlobalEd2 Project. In T. Amiel & B. Wilson (Eds.), Proceedings of World Conference on Educational

Multimedia, Hypermedia and Telecommunications 2012 (pp. 395-400). Chesapeake, VA: AACE.

Lee, J. (1998, April). Missing links in international education studies: Comparing the U.S. with East Asian

countries in TIMSS. Paper presented at annual meeting of AERA, San Diego, CA.

*Legewie, J., & DiPrete, T. A. (2012). High school environments, stem orientations, and the gender gap in

science and engineering degrees. Retrieved from http://ssrn.com/abstract=2008733

*Little, A. J., & Leon de la Barra, B. A. (2009). Attracting girls to science, engineering, and technology, An

Australian perspective. European Journal of Engineering Education, 34(5), 439-445.

*Lou, S. J., Liu, Y. H., Shih, R. C., & Tseng, K. H. (2011). The senior high school students’ learning behavioral

model of STEM in PBL. International Journal of Technology and Design Education, 21(2), 161-183.

*Lou, S. J., Shih, R. C., Diez, C. R., & Tseng, K. H. (2011). The impact of problem-based learning strategies on

STEM knowledge integration and attitudes: an exploratory study among female Taiwanese senior high

school students. International Journal of Technology and Design Education, 21(2), 195-215.

*Lynch, M. G. (2009). Educating America's talent in science, technology, engineering, and mathematics: an

analysis of the effects of parental and high school factors on females' and males' decisions to enter

STEM fields of study. (Master’s thesis). Georgetown University, Washington, DC.

*Maltese, A. V. (2008). Persistence in STEM: An investigation of the relationship between high school

experiences in science and mathematics and college degree completion in STEM fields. (Doctoral

dissertation). Indiana University.

*Marle, P. D., Decker, L. L., Kuehler, C. P., & Khaliqi, D. H. (2012). Science learning in everyday life: An

evaluation of a scenario-based summer STEM camp for middle and high school students. Manuscript

submitted for publication.

Marzano, R. J. (2011). Art and science of teaching: What teachers gain from deliberate practice. The Effective

Educator, 68(4), 82-85.

*Menzemer, C. (2007). Enhancing access and fostering science, technology, engineering and math (STEM)

using civil engineering material applications for special learning disabilities middle school students.

Proc., 37th ASEE/IEEE Frontiers in Education Conf., Institute of Electrical and Electronics Engineers,

Washington, DC, T2B-3–T2B-7.


*Miller, J. L., Ward, R. B., Sienkiewicz, F., & Antonucci, P. (2011). ITEAMS: An out-of-school time project to

promote gain in fundamental science content and enhance interest in STEM careers for middle school

students. Systematics, Cybernetics and Informatics, 9(4), 76-80.

*Mji, A., & Makgato, A. (2006). Factors associated with high school learners' poor performance: a spotlight on

mathematics and physical science. South African Journal of Education, 26(2), 253-266.

*Mosina, N., Belkharraz, A., & Chebanov, D. (2012, March). LYCEE: A Pathway for high school students

towards STEM majors. Paper presented at the Integrated STEM Education Conference, Ewing, NJ.

*Moskal, B. M., Skokan, C., Kosbar, L., Dean, A., Westland, C., Barker, H., & Nguyen, Q.N. (2007). K-12

Outreach: Identifying the broader impacts of four outreach projects. Journal of Engineering Education,

96(3), 173-189.

*Nag, S., Katz, J. G., & Saenz-Otero, A. (2013). Collaborative gaming and competition for CS-STEM

education using SPHERES Zero Robotics. Acta Astronautica, 83, 145-174.

Nathan, M. J., Tran, N. A., Atwood, A. K., Prevost, A., & Phelps, L. A. (2010). Beliefs and expectations about

engineering preparation exhibited by high school STEM teachers. Journal of Engineering Education,

99(4), 409-426.

National Research Council. (1996). National science education standards. Washington, DC: National Academy

Press.

National Science Board (NSB). (2010). Science and engineering indicators. Arlington, VA: National Science

Foundation.

* Nourbakhsh, I. R., Crowley, K., Bhave, A., Hamner, E., Hsiu, T., Perez-Bergquist, A., Richards, S., &

Wilkinson, K. (2005). The robotic autonomy mobile robotics course: Robot design, curriculum design

and educational assessment. Autonomous Robots, 18(1), 103-127.

*Nugent, G., Barker, B., White, A., & Grandgenett, N. (2011). The impact of robotics competitions on youth

STEM learning, attitudes, and 21st century skills. World Conference on Educational Multimedia,

Hypermedia and Telecommunications (EDMEDIA), Lisbon, Portugal.

*Olivarez, N. (2012). The impact of a STEM program on academic achievement of eighth grade students in a

south Texas middle school (Unpublished doctoral dissertation). MPH, Loma Linda University, Loma

Linda, California

*Ornstein, A. (2006). The frequency of hands-on experimentation and student attitudes toward science: A

statistically significant relation. Journal of Science Education and Technology, 15(3-4), 285-297.

*Prins, R. J., MacDonald, S., Leech, J., Brumfield, J., Ellis, M., Smith, L., & Shaeffer, J. (2010, April).

Techfacturing: A summer day camp designed to promote STEM interest in middle school students

through exposure to local manufacturing facilities. ASEE Southeast Section Conference. Virginia Tech

University, Blacksburg, VA.

Raphael, T. E., & Au, K. H. (2005). QAR: Enhancing comprehension and test taking across grades and content

areas. The Reading Teacher, 59(3), 206-221.

*Ragusa, G. (2012). Teacher training and student inquiry and science literacy: Linking teacher intervention to

students’ outcomes in STEM courses in middle and high school classes. Annual conference for

American Society for Engineering Education, San Antonio, TX.

*Ricks, M. M. (2006). A study of the impact of an informal science education program on middle school

students’ science knowledge, science attitude, STEM high school and college course selections, and

career decisions. Doctoral dissertation submitted to the University of Texas, Austin.

Sanders, M. (2009). STEM, STEM education, STEMmania. The Technology Teacher,68(4), 20-26.

Shanahan, T., & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking content area

literacy. Harvard Education Review, 78(1), 40-59.


Shymansky, J. A., Kyle, W. C., & Alport, J. M. (1983). The effects of new science curricula on student

performance. Journal of Research in Science Teaching, 20(5), 387-404.

*Silverstein, S. C., Dubner, J., Miller, J., Glied, S., & Loike, J. D. (2009). Teachers' participation in research

programs improves their students' achievement in science. Science, 326, 440-442.

*Stephen, M. L., Bracey, G., & Locke, S. (2012, April). Teaching and learning in a 21st century, technology-

rich STEM classroom. Paper presented at the International Society for Technology in Education 2012

Conference, San Diego.

*Strautmann, M. A. (2011). Educational robotics: using the Lego Mindstorms NXT platform for increasing high

school STEM education. (Master’s thesis). Missouri University of Science and Technology, Rolla, MO.

*Sullins, A. C. (2010). Factors related to student persistence in a new residential STEM high school: The case

of the Tennessee governor’s academy for mathematics and science (Doctoral dissertation). University of

Tennessee.Knoxville, TN. Retrieved from

http://trace.tennessee.edu/utk_graddiss/753

*Sumners, C., Annette, S., Handron, K., & Jacobson, J. (2012, March). Immersive interactive learning labs for

STEM education. Presented at the Society for Information Technology & Teacher Education

International Conference 2012, 2012(1), 4894–4898.

*Tan, J., Ngo, B. K., Chandrasekaran, I., & Cai, Y. (2013). Cultivating STEM education through 3D

sabbaticals. In Y. Cai (Ed.), 3D immersive and interactive learning (pp. 71-77). Singapore, SGP:

Springer.

*Tyson, W., Lee, R., Borman, K.M., & Hanson, M. A. (2007). Science, Technology, Engineering, and

Mathematics (STEM) pathways: High school science and math coursework and postsecondary degree

attainment. Journal of Education for Students Placed at Risk, 12(3), 243–270.

*Welch, A. G. (2010). Using the TOSRA to assess high school students' attitudes toward science after

competing in the FIRST Robotics Competition: An exploratory study. Eurasia Journal of Mathematics,

Science & Technology Education, 6(3), 187-197.

*Williams, D. C., Ma, Y., Prejean, L., Ford, M. J., & Lai, G. (2007). Acquisition of physics content knowledge

and scientific inquiry skills in a robotics summer. Journal of Research on Technology in Education,

40(2), 201-216.

*Wimpey, A., Wade, C., & Benson, L. (2011, April). Linking pedagogy to practice: Improving student

motivation and academic performance in STEM courses through inquiry-based instruction. Paper

presented at the National Association for Research in Science Teaching, Orlando, FL.

Wright, K. B., Kandel-Cisco, B., Hodges, T. S., Metoyer, S., Boriack, A. W., Stillisano, J. R., & Waxman, H. C.

(2013, December). Developing and assessing students’ collaboration in the IB programme. College

Station, TX: Education Research Center at Texas A&M University. Submitted to the International

Baccalaureate Organization (IBO).

*Wyss, V. L., Heulskamp, D., & Siebert, C. J. (2012). Increasing middle school student interest in STEM

careers with videos of scientists. International Journal of Environmental and Science Education, 7(4),

501–522.

*Yoon, S. A., & Liu, L. (2010, June). Complexity, robustness, and trade-offs in evaluating large STEM

Education programs. Presented at the International Conference of the Learning Sciences, Chicago, IL.

*Zhe, J., Doverspike, D., Zhao, J., Lam, P., & Menzemer, C. (2010). High school bridge program: A

multidisciplinary STEM research program. Journal of STEM Education: Innovations and Research,

11(1), 61-68.


Table 1

Attributes of Reform-based Teaching and Learning Strategies in Artifacts Reviewed (n=25)

Reference Reform-Based

Strategy CK Skills

Technology

Use

Majors/

careers

Perceptions

and attitudes

Learning

environment

Brown et al.

(2010) Project ✓ ✓ Duran &

Sendag (2012) Problem ✓ Duran et al.

(2013) Inquiry ✓ ✓ ✓ ✓ ✓ Heggen,

Omokaro, &

Payton (2012) Inquiry ✓ ✓ ✓ ✓ Hudson,

English, &

Dawes (2012)

Engineering-

Design ✓

Hylton et al.

(2012) Inquiry ✓ ✓ Kampe &

Oppliger

(2011) Project ✓ ✓ ✓ ✓ Kanter &

Schreck

(2007) Legacy Cycle ✓ Ketelhut

(2006) Inquiry ✓ ✓

Kim (2011) Inquiry ✓ Klahr, Triona,

& Williams

(2007)

Engineering-

Design ✓ ✓ Klein &

Sherwood

(2005) Legacy Cycle ✓ Little & Leon

de la Barra

(2009) Inquiry ✓ ✓ ✓ Lou, Liu, Shih,

& Tseng

(2011) Project ✓ ✓ ✓ Lou, Shih,

Diez, & Tseng

(2011) Problem ✓ ✓ ✓ Mehalik,

Doppelt, &

Schuun (2008)

Engineering-

Design ✓


Menzemer

(2007) Hands-on ✓ ✓ Mosina,

Belkharraz, &

Chebanov

(2012) Hands-on ✓ Olivarez

(2012) Project ✓ Richards,

Hallock, &

Schnittka

(2007)

Engineering-

Design ✓

Ricks (2006) Inquiry ✓ ✓ ✓ Schnittka

(2012)

Engineering-

Design ✓ ✓ Williams et al.

(2007) Inquiry ✓ ✓ Wimpey,

Wade, &

Benson (2011) Inquiry ✓ Zhe et al.

(2010) Problem ✓

Note: CK = Content Knowledge; Skills; ✓ = present Question: What kind of skills – technology? Or others

like problem-solving, collaboration, etc.


Table 2

Attributes of Informal Education Artifacts Reviewed (n=23)

Reference Participants Venue Length Selection Pedagogies Focus

Adamchuk et

al. (2009) 147 MS SC & AS

40-80

hours R, Inquiry, PS,

GIS Attitudes, CK

Duran et al.

(2013) 77 HS

Summer &

AS 18 months

Special needs,

F

T, Collaboration,

Inquiry

CK, Careers,

Perceptions,

Attitudes

Heggen et al.

(2012) 21 MS AS 10 weeks

Minority, low

SES T, Problem-based TS, Careers

Hirsh et al.

(2006) 36 T & S

Summer

PD 2 weeks Career Awareness Attitudes, CK

Hoyles et al.

(2011) AS UK Collaboration Attitudes

Hubelbank et

al. (2007) 129 SC 2 weeks Lottery

PS, role models,

hands-on

S-E, Careers,

Courses

Hylton et al.

(2012) SC 1 month

High STEM

interest

Inquiry, PS,

enrichment

Courses,

Confidence

Javidi et al.

(2010) 87 (MS)

SC &

Saturdays 3 years

Low SES,

rural, urban R, Gaming, CP

Attitudes,

Interest, Careers

Johnson et al.

(2013) 133 SC 2 weeks Talented

Research, field

trip, scientists CK, Courses

Kim et al.

(2011) 100 Summer 1 week

Under-

represented

Inquiry, hands-

on, modules Attitudes

Marle et al.

(2012) 32 SC 4 days Average

Real life exposure

to science careers Confidence, CK

Menzemer et

al. (2007) 26 (11 LD)

Summer &

AS Varied

Special pop./

LD

Hands-on,

technology

Attitudes,

Careers, CK

Miller et al.

(2011) 9 T & 84 S

Summer

PD 2 days

Low SES,

minority CK

Mosina et al.

(2010) 239 (HS)

Summer &

AS 10 weeks

Low SES,

minority

Projects, mentors,

research, exhibits Courses

Nugent et al.

(2010) 72 (MS) Clubs Episodic

M, white,

urban

R, Collaboration,

PS

21st CS, CK, S-

E

Nugent et al.

(2010) 147 SC 40 hrs

Urban, rural,

diverse

R, LEGOs,

hands-on

Nugent et al.

(2010) 141 One event 3 hrs

Mixed

abilities Stations

attitude,

motivation

Nourbakhsh

et al. (2005) 28 (HS) SC 7 weeks

Application

process

R, challenge-

based, hands-on

CK, PS, CP,

collaboration

Prins et al.

(2010) 48 (MS) SC 3 days

Projects, mentors,

exposure to

careers

Career

Ricks (2006) 50 SC 4 weeks High STEM

interest

Hands-on, PS,

field trips, inquiry

CK, Attitudes,

courses


Welsh (2009) 58 AS 6 weeks Existing

members R, Competition Attitudes

Williams et

al. (2007) 21 SC 2 weeks Average

Inquiry, hands on,

discussions Content

Zhe et al.

(2010) 33 (HS) SC 10 weeks

High STEM

interest

Problem-based,

collaboration,

hands-on

Confidence,

Career

Note. MS = middle school; HS = high school; T = teacher; S = Student; SC = summer camp; AS = after school;

PD = professional development; F = female; M = male; SES = socioeconomic status; UK = United Kingdom;

STEM = science, technology, engineering, & math; LD = learning disabled; R = robotics; PS = problem

solving; GIS = geographic information system; CP = computer programming; CK = content knowledge; TS =

technology skills; S-E = self-efficacy; 21st CS = 21st Century Skills.


Table 3

Attributes of Teacher Factor Artifacts Reviewed (n=12)

Reference

Teacher

CK

STEM

teaching

practices Technology

Collaborative

learning

Teacher

self-

efficacy

Deliberate

instructional

practice

Duran et al. (2013) ✓ ✓

Finson, Pederson, & Thomas

(2006)

✓

Hotaling et al. (2012) ✓ ✓

✓

Hoyles, Reiss, & Tough

(2011)

✓ ✓ ✓

Lambert (2007) ✓

Moskal et al. (2007) ✓ ✓ ✓

Nag et al. (2013) ✓ ✓ ✓

Nourbakhsh et al. (2005) ✓ ✓

Ragusa (2012) ✓

✓ ✓

Silverstein et al. (2009) ✓ ✓ ✓

Yoon & Liu (2010) ✓ ✓ ✓ ✓

Zhe, Zhao, & Menzemer

(2010) ✓

Note CK = content knowledge; ✓ = present


Table 4

Attributes of Technology Related Artifacts Reviewed (n=27)

Reference

Technology

used

C

K

Technology

skills

Teaching

strategy

Majors/

careers

Perceptions

and

attitudes

21st Century

skills

Adamchuk et al. (2009A &

2009B) R, HH, CP Inquiry ⱡ PS

Brown et al. (2010) Simulations ✓ Project-based

✓

Chapman (2012) Simulations S ✓ PS

Duran & Sendag (2012) R, CP ✓

Problem-

based ✓

Duran et al. (2013) R, CP S ✓ Inquiry ✓ ✓ Collaboration

Heggen, Omokaro, & Payton

(2012) HH

✓ Inquiry ✓

Hotaling et al. (2012) HH, CP T,

S Problem-

based ⱡ PS

Huang, Liu, & Shiu (2008) CAI S Cognitive

Conflict

Javidi & Sheybani (2010) R, CP ✓ Games ✓ ✓

Johnson-Glenberg et al. (2011) Simulations Problem-

based ⱡ,

Games

✓

Kampe & Oppliger (2011) R, HH ✓ Project-based ✓ ✓ PS

Kay, Zucker, & Staudt (2013) Simulations S

Kim et al. (2011) CP Inquiry

Klahr, Triona, & Williams

(2007) Simulations S Engineering

Design

✓

Lawless, Brown, & Boyer

(2011) Simulations S ✓

Lou et al. (2011) Solar Trolley ✓ Project-based ✓ ✓ PS

Moskal et al. (2007) R, HH, CP T,

S ✓ Hands-On ⱡ

✓ Collaboration

Nag, Katz, & Saenz-Otero

(2013) R, Simulations S PS,

Collaboration

Nourbakhsh et al. (2005) R, CP ✓ PS,

Collaboration

Nugent et al. (2010) R, CP ✓

✓ PS,

Collaboration

Stephen, Bracey, & Locke

(2012) R, HH

✓ Collaboration

Strautmann (2011) R ✓

✓ ✓


Note: ⱡ = The match between Table 4 (technology) and Table 1 (reform-based artifacts) may not be exact. Pedagogical

practices are listed in this table only when the artifact reviewed discussed student gains in technology skills, attitudes about

technology, frequency of technology use, or technology content knowledge in relation to the pedagogical practice. R =

robotics; HH = handheld technology device; CP = computer programming; S = science; T = technology; ✓ = present; PS =

problem solving.

Sumners, Handron, & Jacobson

(2012) Simulations S ✓

Inquiry ⱡ,

Games

Tan et al (2013) Simulations ✓

Welch (2010) R Competition ✓

Wyss, Heulskamp, & Siebert

(2012) Recordings

✓

Yoon & Liu (2010) Simulations T


Table 5

Attributes of School Factors Artifacts Reviewed (n=10)

Reference

Sample

Size Type of Study Level

Type of

Intervention Curriculum Survey PD Standard

Cantrell et al.

(2009) 130 Convenient HS

Seminars for 5

years (16 hours

per year)

Professors

presented their

research.

Career

Interest

English et al.

(2011) 122 Convenient MS

PBL Type

lessons

Bridges and

Boats Interest

Goodwin et al.

(2013) 554 Whole School HS

1 lesson spanning

many grade

levels

Engineering

by Design -

Green Houses

Interest

Survey ✓

Standards

for Tech

Literacy

Hudson et al.

(2012)

2 focus

groups Convenient MS

4- 45 minute

lessons Catapult

Legewie et al.

(2012)

NELS

88-92

Nationally

Representative HS National Data set

Lynch (2009) 8774 NELS data set MS/HS

Mentor from

college

representative

Maltese (2008) Varies

from

NELS 88 data

set; Logistic

regression

MS/HS National Data set

Ornstein (2006) 705 Convenience MS/HS Inquiry, hands-on Attitude

Sullins et al.

(2010) 41 Whole school HS Measuring

persistence

Tyson et al.

(2010) 94,078

Large data set-

logistic reg. HS

Course taking

math & science

Note. NELS = National Education Longitudinal Study; HS = high school; MS = middle school; PBL = project-based

learning; Tech = technology


Documents

Promising Practices in STEM Teaching and Learning: A Meta ...aggiestem.tamu.edu/wp-content/uploads/2016/12/... · STEM literacy includes and integrates literacy in all four subject