1

Proposing an Evaluation Framework for Interventions: focusing on Students’ Behaviours in interactive science exhibitions

Nils Petter Hauan a, Jennifer DeWitt b& Stein Dankert Kolstø c

a University of Bergen, and VilVite, Bergen Science Centreb King’s College Londonc University of Bergen

Corresponding author: Nils Petter Hauan, Email: nph@vilvite.noWord count: 7948

AbstractMaterial designed for self-guided experiences such as worksheets and digital applications are widely used as tools to enable interactive science exhibitions to support students' progress towards conceptual understanding. However, there is a need to find expedient ways to evaluate the quality of educational experiences resulting from use of such tools. Towards this end the approach of the current study is to focus on students’ behaviours and relate identified behaviour categories to learning theory. An intervention developed as case for this study is presented. The intervention design aimed at setting up a learning environment where bodily, text-based, verbal and social experiences are embedded to facilitate progress towards conceptual understanding via the use of group assignments where students experience phenomena that correspond to focal concepts. To this end, six tasks were designed, five customized for energy-related exhibits and one which gave teachers a role to support students in pulling together the concepts they had encountered. Video recordings were transcribed and analysis investigated the quality of the intervention based on both verbal and non-verbal behaviours during the six tasks. Two overarching learning-related behavioural categories are identified: one reflecting general overall engagement in the learning environment, and a second designated as multi-modal discussions which is indicative of deeper engagement and, in turn, possibility of conceptual learning outcomes. More broadly, this research implies that frameworks based on students’ behaviours can contribute to design of valuable learning experiences and evaluation focusing on relevance for concept learning.

2

IntroductionMaterial designed for self-guided experiences has been the subject of several studies of their effectiveness as learning resources for school trips. Some museum-related research projects aim to measure learning outcomes resulting from the use of worksheets (Krombaβ & Harms, 2008) or electronic guidebooks (Sung, Hou, Liu, & Chang, 2010). Other work (e.g. Kisiel, 2003, 2007) has investigated the influence of worksheet design from teachers’ perspective and others have looked into how worksheet design influences students’ engagement in task completion (Griffin & Symington, 1997; Rix & McSorley, 1999; Stavrova & Urhahne, 2010). Still others have studied the significance of teacher involvement in students' work with worksheets (Choo, Rotgans, Yew, & Schmidt, 2011; Nyamupangedengu & Lelliott, 2012). Further research projects have investigated how particular resources may scaffold exploration to enhance learning from single exhibits (Watson, 2010; Yoon, Elinich, Wang, Schooneveld, & Anderson, 2013). These studies provide significant insight, however, in line with Rennie, Feher, Dierking, and Falk (2003) we believe that there is a need for further research on how designed tasks interact with or relate to the particular learning environments for which they have been developed by looking closely at students’ verbal and nonverbal behaviour during task completion and how these behaviours are related to learning. We also assert, in line with Author 1 and Author 3 (2014) that this approach enables evaluation of educational quality of interventions by studying their capacity to engage students in fruitful learning processes.

As a case for our study of intervention-generated behaviours, we developed an educational intervention based on field investigations, theories of conceptual development, and previous empirical research (Anderson, Lucas, Ginns, Dierking, 2000; Author 2 and colleague, 2007; Mortensen & Smart, 2007; Rix & McSorley, 1999; Stavrova & Urhahne, 2010).

Research Questions By observing students’ behaviour during the visit as well as drawing on guidance from existing research we aim to address the following research questions:

1) What categories of verbal and non-verbal behaviours are generated by the educational material provided to students during their visit to the science centre?

2) Are these behaviours consistent with behaviours that are recognised as supporting development of conceptual understanding?

To address the first question we use observations to identify and categorise types of behaviours. To address the second question we discuss the identified behaviour categories in relation to learning theory. In particular, we utilise concepts of scaffolding, joint productive activity, interthinking and preparation for future learning as lenses through which to consider whether the observed behaviours may lead to learning. Scaffolding (Wood, Bruner and Ross, 1976) is used as a reference to discuss the quality of the assigned task as tool to guide students’ exploration. Group work with characteristics of Joint Productive Activity (JPA) (e.g. facilitation of cooperation, teacher involvement in group work) also has the potential to result in learning outcomes (Dalton & Tharp, 2002). Observed group behaviours are therefore discussed in relation to the list of indicators of JPA (Dalton & Tharp, 2002). Interthinking is a term used by Mercer (2000) to describe how people cooperate to make sense of an experience and to create knowledge. As will be presented subsequently, we found interactions generated in the learning environment which we designated as multi-modal discussions to be consistent with deep engagement. We also discuss relationships between multi-modal discussions and interthinking, and consider whether these identified behaviours may be indicators of learning. Finally, we see scaffolding, JPA and interthinking as closely related to engagement, and we

3

also discuss engagement in relation to the concept of Preparation for Future Learning (Bransford & Schwartz, 1999).

Intervention developed as case for this studyTo explore the potential of focusing on learning-related behaviour when constructing frameworks for evaluation of science exhibitions, we designed an intervention with the aim of generating a rich variation of fruitful interactions with the learning environment.

Background for intervention design

Griffin and Symington (1997) found that school visits to museums can be categorised as either being "task oriented" (p. 768), instructing the students in what to see and what to do, or "learning oriented" (p. 768), focusing the students towards a predefined learning outcome and, to a lesser extent, emphasising how to achieve this outcome. These categories are consistent with teacher agendas described by Kisiel (2003) as "a survey agenda and a concept agenda" (p. 14), which he found reflected in worksheets teachers created for their visits. Worksheets are commonly used to structure school visits and to enhance the educational outcome of students' interactions with exhibits (Kisiel, 2003). Worksheets are frequently seen by teachers and students alike as necessary for learning in exhibitions to occur (Griffin & Symington, 1997; Kisiel, 2003). However, worksheets can also reduce motivation (Griffin & Symington, 1997), decrease exploration and even be annoying (Rix & McSorley, 1999). Nevertheless, well-designed worksheets can facilitate content-related discussions (Mortensen & Smart, 2007) and enhance the educational experience by providing an appropriate level of structure for the visit (Durbin, 1999).

Bearing these different worksheet possibilities in mind, Kisiel (2003) calls on museums and researchers to take an active role in developing worksheets that are more student-centred and emphasise a concept agenda. Moreover, Choo et al., (2011) and Nyamupangedengu and Lelliott (2012) remind us of the importance of giving the teacher a role. With the design of the intervention in this study, we aim to answer to these calls. In addition to ending the visit with a teacher-led review, the intervention designed for this study involves activities beyond just writing answers on worksheets. Instead, we sought to activate interactions involving peers and exhibits. Consequently, we have chosen to use the term task-sheets to describe the material used rather than worksheets.

Conceptual framework for intervention design

To help students develop meaningful understanding of selected concepts and the relationships between them, we aimed to help them “make links between the domain of objects and observables and the domain of ideas” (Millar, 2004, p.12). Put simply, the intervention was intended to support understanding of scientific concepts by letting students experience phenomena (in the hands-on exhibits) and link those phenomena to their corresponding scientific terms and descriptions (in the material provided). Our objective is therefore to provide a range of experiences that support linking between the real world and abstract ideas. By providing embodied experiences we utilise connections between the mind and our senses and muscles (Dewey, 1938). Reading or listening to text which presents scientific concepts and being challenged to make choices between different statements provides an experience which involves making meaning of terms which refer to the real world (Sutton, 1992). To work with peers to solve a shared learning task provides an experience of using words and bodily expression as tools to communicate and negotiate meaning in a social context (Givry & Roth, 2006). By embedding bodily/kinaesthetic, verbal (reading and writing) and social experiences we aim to create a learning environment which offers multiple entry points to

4

conceptual understanding (Gardner, 2006). In helping students to make meaning of the new experience we also aim to make links to their previous experiences (Ausubel, 2000).

The importance of multiple entry points and links to previous experience informs the design of the learning environment in this study, which therefore reflects four principal elements:

Exploration of phenomena. The focal exhibits are designed to provide bodily experiences with the phenomena. Some exhibits provide for direct sensory experience of the phenomena (e.g. Sousa, 2011). Other phenomena are experienced indirectly by observing others’ experiences (e.g. Noë & O’Regan, 2002).

Reading involving focal scientific concepts. Concepts are presented in a meaningful context by the text in task-sheets provided to the students. By reading the text or listening to others reading, students are challenged to make meaning of the concepts in order to solve the given tasks (Wellington & Osborne, 2001).

Group work. Students within the groups are encouraged to organise their work and cooperate to solve tasks, e.g. by creating a need to establish consensual views and share responsibility for task completion. This may, in turn, enhance learning outcomes (Johnson, Johnson, Stanne, & Garibaldi, 1990).

Anchorage. Illustrations that present how the phenomena are present in real-world settings are included in the task-sheets in order to activate prior knowledge and to facilitate anchorage in students’ existing cognitive structures (Novak, 2002).

Overall, the task-sheets given to the students are designed to provide them with a guided exploratory learning experience (Author 1 & Author 3, 2014). The aim of the guidance is to encourage students to utilise or draw upon their prior knowledge, the exhibits, and the available multimodal texts as resources in activities that involve developing, expressing and testing their understanding within the social context of group work.

MethodThe overall research project can be considered a design experiment (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003). More specifically, the research studies an intervention which has been designed for and applied in a specific context, namely the science centre exhibition, and the findings from the study have potential influence on the future evaluation and design of educational interventions. Also, the process of analysis aims to identify and classify the behaviours generated and thereby to contribute to or “develop theories” (Cobb et al., 2003, p. 9). In the remainder of this section, we describe the intervention used as case in the study, as well as data collection and analysis.

The InterventionThe subject matter covered in this intervention concerns energy-related concepts such as how electricity is generated and the sources of this energy. Such concepts are relevant for progression in the science ‘pipeline’ (as part of the curriculum), as well as for scientific literacy. Up to 30 students worked in groups of three to six on tasks corresponding to five separate exhibits. In the first phase the groups carry out five activities in the exhibition guided by their task-sheets. The second phase is conducted in a nearby room where teachers are encouraged to support their students in a task that aims to pull together the concepts that the exhibits represent into a cohesive story.

Since the overall aim is to help students to link concrete experiences to abstract ideas, the set of task-sheets attempted to guide students’ focus to relevant observations and scaffold their thinking (Millar, 2004). There are two main categories of task-sheets: Exhibit Interaction Sheets (EIS) (one for each exhibit) and a summarizing sheet which we refer to as the Concept Flow Chart (CFC). Students are also given a customised exhibition map to help them locate the relevant exhibits.

5

During school trips teachers are often preoccupied by logistics and other practicalities (Tal, Bamberger, & Morag, 2005) and cannot realistically directly support all students in their exploration. Staff is also a limited recourse at this particular science centre (as in many venues of this kind). The EISs were therefore designed as the main tool for guiding the students and scaffolding their interaction and thinking. The design of the EISs was also influenced by previous teacher feedback recommending that the amount of text should be minimised and that the layout should be informed by the design of written material with which students are familiar. In addition, the intervention was designed to fit to the curriculum and textbooks of grades six and seven in primary school (11-13-year-olds, who were the study participants). Finally, the EISs and CFC were influenced by piloting (not described here due to length considerations) which revealed issues with how students used the focal exhibits, as well as misunderstandings and misconceptions about the concepts involved.1

Each EIS has the same main elements. There is a small picture of the exhibit, identical to the one in the map, to support identification of the exhibit. The heading tells the student what the exhibit is about (e.g. ‘Energy from light’) and is aimed to guide focus towards the intended learning outcome of the exploration. An illustration presents how the phenomena might be encountered in the real world (e.g. Generator being rotated by a turbine which is turned by flowing water, see appendix). The aim of this illustration is to activate prior knowledge (Ausubel, 2000). Three different types of EISs were designed (concept cartoons, multiple choice questions, video reporting) to provide variation and to customize the tasks for different types of exhibits. Variation of task was also motivated by research indicating that variety and a degree of openness is preferable (Mortensen & Smart, 2007; Stavrova & Urhahne, 2010). Additionally, four of the five focal exhibits are “Planned Discovery” (PD) exhibits (Humphrey & Gutwill, 2005) which present phenomena in a way intended to make them easily observable for the students. The EISs for these exhibits were designed to encourage content-related talk (Mercer, 2000). Type of task was also chosen based on whether any Identified Misconception (IM) had been expressed by students during piloting. The fifth exhibit is designed for Complex Exploration (CE) with several variables, however, the information it presents is also significant for the overall story about the different ways electricity we use in our houses is produced. In addition to guiding the observations, this CE-related EIS asks the students to make a video report with the goal of giving a doable and open but defined task. The exhibits and their corresponding EIS types are presented in Table 1.

---- Insert Table 1 about here ----The Concept Flow Chart (CFC) is inspired by Collaborative Concept Mapping

(Wellington & Osborne, 2001) and is designed as a puzzle where words representing focal concepts are fixed to magnet bands and the puzzle-sheet is fixed to a magnet board. The puzzle-sheet presents a flow chart of words constituting sentences which describe different ways which energy from solar radiation is converted to electric power in houses. The task is to position focal concepts correctly in the CFC puzzle. The teachers are encouraged to support the students in this work. Illustrations from the EISs were included in the puzzle-sheet to help students to link concepts.

Data Collection and AnalysisInvitations to participate in the study were sent to schools in the region of the science

centre. The four classes in the main study came from four different state schools situated in four different boroughs in [small city], Norway. Four classes from different schools were included in the study to provide a somewhat varied sample. This is relevant to minimize effects caused by “outliers”. In order to enable in-depth coding, and as video represents very rich data which is time-consuming to analyse, it was decided to limit analysis to these four groups. Due to population homogeneity in this part of Norway, analysis by social class and/or

6

ethnicity is not feasible. (That is, students in this location are similar in social background and ethnicity.) Moreover, exploring demographic variation was beyond the scope of the current research. Most of the participating students had visited [science centre] previously. The [science centre] normally provides pre- visit material, but it was not done in this study to avoid contributing to variations in the how well the students were prepared. The teachers were only informed that energy would be the subject and that the study would take place in the exhibition.

Out of the 148 students visiting (across the four classes), two students in one group from each class were equipped with head- or chest-mounted video cameras (8 students altogether). The video recordings which showed students’ activities most clearly were used as data for the analyses. Data analyses drew upon Interaction Analysis (Jordan & Henderson, 1995). Transcription and coding for analyses were done by the first author, using Nvivo 10 software.

The purpose of the analysis was to identify student behaviours that have relevance for investigating the quality of the intervention. This process implies a need for awareness regarding theoretical sensitivity (Strauss & Corbin, 1990) since the intervention we are using as a case is based on our conceptual framework (described previously). To ensure quality of categories developed we followed the principles of Grounded Theory (Strauss & Corbin, 1990). All seemingly relevant behaviours of the students, teacher, researcher or guides, i.e. activities related to exhibitions, task-sheets or scientific concepts were transcribed in this open minded way. Next, development of the coding structure for the transcripts followed guidelines for “Open Coding” (Strauss & Corbin, 1990, p .61). Discrete observable verbal and non-verbal behaviours by single individuals were chosen as units of analysis with the purpose of having distinct units to facilitate accurate coding and allow for inclusion of all relevant contributions by the students. First description labels were created for discrete observed behaviours, and then sub-labels were assigned to some of the description labels to capture greater detail. Then, after development of these labels (describing behaviour), they were grouped into larger categories of behaviours corresponding to the four principal elements underpinning the design: Exploration of scientific phenomena, Reading text incorporating scientific concepts, Group organisation to facilitate cooperation, and Anchorage involving comments related to everyday experiences and prior knowledge. Initial analyses revealed that students’ behaviours could also be grouped into two overarching classes: one covering engagement with the elements in the conceptual framework (Exploration of phenomena, Reading about scientific concepts, etc.), summarised in Table 2, below, and one encompassing what we have designated as multi-modal discussions to incorporate both verbal and non-verbal (e.g. gestures) interactions resulting from engagement with the learning material (task-sheets and exhibits see Table 3).

The four transcripts from video recording of the groups were coded using the codes in Tables 2 and 3. To check for inter-coder reliability 17% of the transcripts, representing students’ work with different task types, were coded by a PhD candidate not involved in this study. Inter-coder reliability was found to be 0.783 (Cohen’s kappa) which is considered acceptable (Di Eugenio, 2000).

FindingsIn this section we present our categorisation of student behaviours, making reference

to transcripts of video recordings. Two broad classes of behaviours which we identify as learning-related are included: one reflecting general overall engagement in the learning environment (Table 2), and a second indicating deeper engagement (Table 3). Numbers of identified behaviours from the class of deeper engagement indicators, which constitute what

7

we have designated as multi-modal discussions, are presented graphically for the four groups (Figure 1).

Behaviours Related to Design Elements of the Conceptual FrameworkTable 2 presents the categories of learning-related behaviours which were initially

developed to code the video transcripts of students' activity while using the task-sheets. ----- Insert Table 2 about here ----

Analyses of student behaviour reflected that there was little variation among the student groups in the behaviours corresponding to the categories Exploration of phenomena and Reading text involving scientific concepts. Typical behaviour for all four groups involved most students exploring the exhibits with a few of their peers standing close by, observing their exploration. Thus, it appears that phenomena to be observed are presented for all the students either bodily (when they interact with the exhibits themselves) or via observation of their classmates’ interactions with the exhibits. Watson (2010) found both types of engagement to have equivalent effects on learning. A similar pattern of behaviours was noted with regard to the category of text reading, with each student either reading the text themselves or listening carefully to others. This pattern implies that students encounter text involving scientific concepts and the related phenomena themselves within the same timeframe.2

Contrasting with the previous two categories, analyses of behaviours in the category Group work highlighted variation among the groups in how they organised themselves to complete the tasks. In general, while all groups had at least one person who took responsibility for organising task completion, the details of how they did so and the responses and behaviour of other group members varied. All students participated in completing all tasks but variations in their focus were seemingly caused by social dynamics within the group, loss of motivation due to a stubborn or strict group organizer, or being limited in exploration. There were few instances of negative behaviours or indicators of boredom, generally off-task behaviour involved free exploration of unrelated exhibits.

By including illustrations involving applications of focal phenomena in a real world setting we aimed to activate prior knowledge. We were not able to identify talk involving prior knowledge or behaviours that revealed how students perceived the illustrations, i.e. whether they facilitated Anchorage.

There was very little variation in the role of the teachers of the four classes. They were mostly occupied with practical issues while their students were in the exhibition and seldom involved themselves in students’ task completion. They did, however, assume responsibility for supporting students in their work completing the CFC puzzle.

Indication of Deep EngagementAlthough identifying behaviours related to the conceptual framework for design was

informative about students’ activity overall, we felt that the framework was not fully capturing the richness of the interactions generated by task completion that we were seeing and their potential to support learning. Consequently, we performed a second analysis which focused on verbal and non-verbal behaviours consistent with deeper engagement within the learning environment. The categories emerging from this analysis, and corresponding behaviours and sub-behaviours are presented in Table 3. We use the term multi-modal discussions to cover all of the codes in this analysis since the interactions can be regarded as elements in a discussion which is aimed at solving a shared task.

---- Insert Table 3 about here ----

To illustrate some of the behaviours in Table 3 we will present two transcript excerpts from one group’s interactions at the Generator exhibit (Tables 4 & 5), and one from the last

8

part of these students’ work with the CFC (Table 6). A more detailed description of how the codes were applied follows the excerpts. Assigned category codes are presented to the right of the transcripts. All names in the transcripts are pseudonyms. Transcripts are translated from Norwegian.

---- Insert Table 4 about here -------- Insert Table 5 about here -------- Insert Table 6 about here ----

We now turn to illustrate how the behaviour codes in Table 3 were developed from the

data. (We did not include data corresponding to all behaviour categories in Table 3 due to length considerations.) Line 5 (Table 4) fell into the category of Testing or repeating, and the sub-category testing an idea by handling the exhibit because it reflects non-verbal behaviour indicative of efforts to understand a phenomenon by focused handling and testing. Line 13 is given the code Bodily expression, and the sub-code other gestures (within Bodily Expression) indicating that understandings or ideas were presented non-verbally. Lines 2, 10, 12, 15, 20, 22, 23, 29, 32, 43, 44, 45 and 47 are labelled as Suggestions (related to EIS or CFC). This category does not distinguish between guessing and presenting a carefully-thought through idea (as it was not possible to make this distinction from the data). Lines 4, 8, 26, 33, 35, 36, 40 and 46 are coded as Expressing thinking (related to EIS or CFC) because they indicate the use of words to present or articulate thoughts to others. Lines 17, 21, 37 and 42 are coded as Asking questions (related to EIS or CFC) which indicate a wish for input from others thinking. Lines 16, 34 and 41 are labelled as Commenting on other students’ talk or actions (within Feedback on others’ thinking) which can be seen as comparable to a reply in a dialogue.

The behaviours presented in Tables 2 and 3 are identified from a descriptive perspective. That is, the analysis thus far has focused primarily on what verbal and non-verbal behaviours were observed. In the next stage of analysis, we grouped selected behaviours into categories which we consider to be informative regarding the quality of the multi-modal discussions. Behaviours such as expressing understanding, providing feedback to others and testing one’s own understanding are the kinds of behaviours that previous research (e.g. Beard & Wilson, 2013 Dewey, 1938; Mercer, 2000) have found to be consistent with or supportive of learning. Table 7 below reflects the behaviours themselves and their associated categories. To illustrate the extent and content of multi-modal discussions within each group we have presented counts of observed behaviours within each related category (See Figure 1). Talk involving students’ prior knowledge, i.e. indication of anchoring, is not included as this was not detected in the groups studied.

---- Insert Table 7 about here -------- Insert Figure 1 about here ----

As presented in Figure 1, all categories of learning behaviours which are characteristic of multi-modal discussions were present in each group. This outcome, together with previously presented findings related to engagement in the learning environment, suggests that the identified categories (Tables 2 & 3) are informative as to the extent of learning-related behaviours resulting from students’ work with a given exhibition-related task set. Consequently, numbers of learning related behaviours within the categories of table 7 and presented in Figure 1 might be regarded as an indicator of the quality of the designed intervention.

As we assert that behaviours listed in Table 7 are indicative of deep engagement, we further argue that the difference in occurrence suggests that students’ engagement in the learning environment and quality of the ensuing multi-modal discussions vary among the groups. There does not seem to be a correlation between number of group members and the

9

quantity or quality of multi-modal discussions. A seemingly more likely cause of the variation is the way the groups are organized, and how that influenced students’ focus. Other causes are also plausible, such as familiarity with the concepts, or level of comfort with the setting or task. Such influencing factors would be interesting to explore in future quantitative and qualitative studies.

DiscussionIn this section, we link our findings to the research questions, particularly the second

question concerning whether the observed behaviours are consistent with behaviours that learning theory and previous research consider to be supportive of the development of conceptual understanding. We then situate the findings relative to wider theoretical perspectives (e.g. Preparation for Future Learning) and previous learning research (e.g. around exploration).

Transcripts of video recordings of students engaged in activities during their visit provided data to address the first research question which dealt with identification and categorisation of students’ behaviour. Two classes of (verbal and non-verbal) behaviours were developed to describe student activity: one class corresponding to engagement in relation to design elements (Table 2) and a second incorporating behaviours characteristic of multi-modal discussions (which we see as indicating deeper engagement in the learning environment).

The second research question focused on whether the behaviours identified in the analyses had the potential to support learning. As noted previously, we use the concepts of scaffolding (Wood et al., 1976), Joint Productive Activity (Dalton & Tharp, 2002) and Interthinking (Mercer, 2000) as references to consider whether the identified behaviours may indeed support learning.

One of the challenges of utilising a science centre exhibition to support learning is the lack of guidance provided by staff or teachers (Kisiel, 2003; Tal et al., 2005). Therefore, educational resources provided should be designed to guide students’ exploration and task completion, ideally supported by their peers and their teachers. Scaffolding as described by Wood et al. (1976) and further developed by others (e.g. Quintana et al., 2004) can function as one design guideline by emphasising the need to support students’ work by focusing their attention, clarifying tasks, highlighting critical features, and offering help in task completion. Brush and Saye (2002) describe two categories of scaffolding, hard and soft, which we believe are both provided by the intervention presented in this study. Hard since it is prepared and structures students' exploration, and soft since it, in line with Choo et al., (2011) and Nyamupangedengu and Lelliott (2012), assigns teachers a role in providing scaffolding during the summarizing task based on their acquired insight into students’ understanding. As presented above, the students performed the activities as intended, and the teachers supported students in solving the CFC. Thus, it appears that the set of tasks encountered by the students during their visit was likely to have scaffolded their exploration.

Another guideline for how the tasks were designed is Joint Productive Activity (JPA), described by Dalton and Tharp (2002) as one way for teachers to build […] “reflection, practice, interaction, constructive feedback, and continuous learning about local conditions and students' personal situations” (p. 3). JPA implies designing for collaboration, providing appropriate time for tasks, facilitating group work and talk, teachers’ participating in group-work, focusing on interaction when designating the groups, involving the students in planning, providing appropriate equipment, and monitoring and supporting students in positive ways. Based on the behaviour observed in the data we argue that the composition of the students, the exhibits, the tasks, and the teacher together seem to generate activity that can be designated as JPA. However, there is one essential exception. Designing for JPA implies

10

providing students with an opportunity to influence what to do and how to do it ([Author 2 and colleague], 2007). One could conclude that there is a conflict between what we, in this case, have seen as a necessary level of scaffolding and an aim to provide for openness in the design. We consider it a goal to provide for both scaffolding and openness, and suggest that this would be an important area for further research.

The EISs (Table 1) were designed to generate verbal discussions between students as advocated by several researchers (e.g. Wells, 1999). However, we found few instances of purely verbal discussions, in terms of exchanging and responding to ideas and thoughts by use of words alone. What we did find was that all of the assignments including the CFC puzzle generated a number of explorative behaviours which we have labelled as expressing understanding, inviting others, feedback to others, testing individually, and testing socially (Table 7, Figure 1). To capture the multimodality of these interactions generated by the learning environment as well as the cooperative nature of students’ endeavour, we chose to use the term multi-modal discussions. This notion is also informed by Mercer’s (2000) use of the term interthinking to describe “[…] co-ordinated intellectual activity which people regularly accomplish using language” (p. 16). Mercer (2000) uses dancing as an analogy to illustrate this term. By doing so, he points towards an extension of what communication is by describing individuals dancing together as joint cooperative activity where the brain is used to accomplish a common goal. In line with Mercer we employ the term multi-modal discussions to describe “co-ordinated intellectual activity” in which the students are engaged in work with the given assignments, but this work involves more than simply verbal activity. Moreover, and in agreement with Wertsch (1991), we argue that exhibits, assignments, and the other students which together with each student’s cognitive structure define the learning environment, can all be seen as elements in a tool kit. These elements, by generating engagement and multi-modal discussions, mediate the content which the intervention is designed to convey and thereby can potentially result in conceptual learning.

The analysis, as presented in Table 7 and Figure 1, indicates engagement and presence of learning-related behaviour in general and multi-modal discussions in particular. This is an indication that the conceptual framework guiding our design, i.e. exploration of phenomena, reading involving focal scientific concepts, group work and illustrations which link to real world settings, merits further exploration.

Thus far, we have looked at student behaviours and considered whether they have the potential to support learning, particularly from the perspectives of scaffolding, JPA and interthinking. We now turn to consider and situate these findings in relation to wider theoretical perspectives and previous research on learning.

Engagement and Preparation for Future LearningAnalyses of the behaviours presented in Table 2 provide insight into students’

engagement in the learning environment. Put simply, these analyses inform us as to whether or not exhibits were used, thus whether the phenomena were encountered (either directly or indirectly), if related words were read and shared with peers and how the group worked together. As described earlier, we found that the scaffolding design of the task-sheets did engage students in using their minds and senses in their cooperative endeavour with the given tasks. These findings are similar to those of Watson (2010), who argued that engagement in learning experiences which included both body and mind can function as “preparation for future learning” (PFL) (Bransford & Schwartz, 1999). We consider the intervention presented in this paper to be consistent with the type of interventions described by Watson (2010) and, based on our findings related to engagement, that the intervention has the potential to support students’ learning by preparing them for future encounters with the focal electricity-related concepts.

11

Evaluating Deeper Engagement and Ongoing LearningRennie et al. (2003) call for new approaches for evaluating the quality of educational

experiences at out-of-school settings such as science centres. Based on reviewing empirical studies within this field [Author 1 and Author 3 (2014)] argue for evaluating quality by studying the processes visitors are engaged in. A study taking this perspective (Barriault & Pearson, 2010) presents a tool for evaluating single exhibits based on their ability to engage visitors in learning related activities. Investigations of families’ learning outcomes from exhibit interactions (Gutwill & Allen, 2010; Jant, Haden, Uttal, & Babcock, 2014) suggest that interventions which increase particular kinds of both verbal and nonverbal behaviour can result in subsequent increases in exploratory behaviour and in content-related knowledge. These studies support the hypothesis that there is a correlation between the extent of multi-modal discussions and learning outcomes, however, specifics of the context of these studies sets limits on the generalizability of the findings.

To elucidate a discussion of whether extent and quality of multi-modal discussions inform of potential learning outcomes we turn to Ausubel (2000) who emphasizes that meaningful learning requires “that the particular learner’s cognitive structure contains relevant anchoring ideas to which the new material can be related.” (p. 1). This implies that since the studied intervention did not generate observable behaviours related to anchorage we cannot state that our findings provide evidence for learning. However, students might in principle have done such anchoring in their individual thinking. Moreover, examinations of students’ multi-modal discussions inform us of the extent and quality of their physical and mental activities, many of which have also been advocated by learning theorists as promoting learning (e.g. Dewey, 1938; Gardner, 2006; Piaget, 1935 & 1965; Wertsch, 1991). Consequently, we argue that these discussions can be used as strong indicators of learning.

Figure 1 shows that all categories of behaviours indicative of quality of multi-modal discussions were present for all four groups, albeit to varying extents. This indicates deep engagement in the learning environment and ongoing learning for all groups. In future efforts to enhance the quality of the intervention of this case study we would, guided by our current findings, focus particularly on increasing behaviours within the Testing individually category since those behaviours refer to individuals’ mental activity (Ausubel, 2000) and their use of the exhibits as tools for embodied experiences (e.g. Dewey 1938; Piaget, 1935, 1965). This quality enhancement could involve a redesign of the task sheets to encourage more testing using multiple senses. It should also involve activating concept-relevant prior knowledge and thereby facilitating anchorage by presenting e.g. “orienting questions” (Osman & Hannafin, 1994, p. 5) related to the illustrations of real world settings. We consider that effect of task designs that facilitate anchorage by activating prior knowledge to be an interesting area for further research.

Findings related to Group work also inform us of the influence which students who take on the role as organizers have on other students’ behaviours, including their participation in multi-modal discussions. This finding could be discussed with teachers with the purpose of enhancing the quality of group work as advocated by Johnson et al. (1990).

The occurrence of learning related behaviours identified from well-known research on learning, indicates that it is possible to use the categories presented in Table 7 together with categories related to engagement in the learning environment presented in Table 2, as a framework for evaluation of educational quality of interventions designed for interactive science exhibitions. As this study is based on a restricted number of groups and only one intervention, it needs to be taken as a tentative proposal. Further research might elucidate further indications of deep engagement, e.g. related to anchoring. Also, as the proposed evaluation framework is based on interaction analysis, it is limited by its reliance on the social

12

context as it does not probe into “the heads of individuals” (Jordan & Henderson, 1995, p. 41). Nevertheless, utilising visitor behaviours to analyse learning processes and progress seems to be promising in terms of both discussing the educational potential of interventions (and school trips more broadly), and identifying measures to improve interventions. We believe that this will be an important area for future research.

ConclusionsWe have investigated the quality of learning material based on students’ behaviour rather than learning outcomes. Categories of verbal and non-verbal behaviours were used as indicators of preparation for future encounters with focal concepts, and as indicators of ongoing learning. Extent and quality of multi-modal discussions inform us of the tasks’ ability to generate mental and embodied activities in a social context that are recognized as resulting in learning. The intervention design aimed to set up a learning environment where the exhibits, multimodal texts, and individual students’ existing cognitive structures are used as resources to engender particular learning activities which could, in turn, be included as resources within the social context of group discussion. Our findings indicate that the designed tasks enabled students to take advantage of the learning environment by using it to develop and test propositions related to the subject matter, thus supporting their progress towards conceptual understanding. Conceptual learning is a complex process that normally extends over a long period of time. Measuring the impact of a visit on this process demands methods and resources that normally are beyond the capacity of science centres. We consider that the evaluation framework presented in this paper contributes to the work of developing methods that make quality evaluation feasible for science centre staff and researchers. Investigation of learning-related behaviour generated during visits is also informative about how a given design works (or not). Such insights can contribute to knowledge building within the science centre field about designs that promote conceptual understanding.

ReferencesAuthor 1 & Author 3. (2014). Nordic Studies in Science Education.Author 2 & colleague. (2007). International Journal of Science Education.Author 2 & colleague. (2008). Visitor Studies.Anderson, D., Lucas, K. B., Ginns, I. S., & Dierking, L. D. (2000). Development of knowledge about

electricity and magnetism during a visit to a science museum and related post-visit activities. Science Education, 84(5), 658-679.

Ausubel, D. P. (2000). The acquisition and retention of knowledge: A cognitive view. Dordrect; Boston: Kluwer Academic Publishers.

Barriault, C. & Pearson, D. (2010). Assessing exhibits for learning in science centers: a practical tool. Visitor Studies, 13(1), 90-106.

Beard, C. & Wilson, J. P. (2013). Experiential learning: A Handbook for Education, Training and Coaching. London: Kogan Page Publishers.

Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, Vol. 24, (1999), pp. 61-100

Brush, T. A., & Saye, J. W. (2002). A summary of research exploring hard and soft scaffolding for teachers and students using a multimedia supported learning environment. The Journal of Interactive Online Learning, 1(2), 1-12.

Choo, S. S., Rotgans, J. I., Yew, E. H., & Schmidt, H. G. (2011). Effect of worksheet scaffolds on student learning in problem-based learning. Advances in health sciences education, 16(4), 517-528.

Cobb, P., Confrey, J., A. diSessa, Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

13

Dalton, S. S., & Tharp, R. G. (2002). Standards for pedagogy: Research, theory and practice. In G. W. G. Claxton (Ed.), Learning for life in the 21st century (pp. 181–194). Cambridge: Blackwell Publishers.

Dewey, J. (1938). Experience and education. New York: The Macmillan company.Di Eugenio, B. (2000, May). On the Usage of Kappa to Evaluate Agreement on Coding Tasks. In

LREC (Vol. 102, p. 114).Durbin, G. (1999). Improving worksheets. In E. Hooper-Greenhill (Ed.), The Educational role of the

museum (2nd ed., pp. 92-98). London: Routledge.Gardner, H. (2006). Multiple intelligences new horizons (Completely rev. and updated ed.). New

York: BasicBooks.Givry, D., & Roth, M. W. (2006). Toward a new conception of conceptions: Interplay of talk,

gestures, and structures in the setting. Journal of Research in Science Teaching, 43(10), 1086-1109.

Griffin, J., & Symington, D. (1997). Moving from task-oriented to learning-oriented strategies on school excursions to museums. Science Education, 81(6), 763-779.

Gutwill, J. P., & Allen, S. (2010). Facilitating family group inquiry at science museum exhibits. Science Education, 94(4), 710-742.

Humphrey, T., Gutwill, J., & Team, E. A. (2005). Fostering active prolonged engagement. San Francisco, CA: The Exploratorium.

Jant, E. A., Haden, C. A., Uttal, D. H., & Babcock, E. (2014). Conversation and object manipulation influence children's learning in a museum. Child Development.

Johnson, D. W., Johnson, R. T., Stanne, M. B., & Garibaldi, A. (1990). Impact of group processing on achievement in cooperative groups. The Journal of Social Psychology, 130(4), 507-516.

Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1), 39-103.

Kahr-Højland, A. (2010). Hands on, mobiles on: The use of a digital narrative as a scaffolding remedy in a classical science centre. MedieKultur. Journal of media and communication research, 27(50), 66-83

Keogh, B., & Naylor, S. (1999). Concept cartoons, teaching and learning in science: an evaluation. International Journal of Science Education, 21(4), 431-446.

Kisiel, J. F. (2003). Teachers, museums and worksheets: A closer look at a learning experience. Journal of Science Teacher Education, 14(1), 3-21.

Kisiel, J. F. (2007). Examining Teacher Choices for Science Museum Worksheets Journal of Science Teacher Education, 18, 29-43.

Krombaβ, A., & Harms, U. (2008). Acquiring knowledge about biodiversity in a museum—are worksheets effective? Journal of Biological Education, 42(4), 157-163.

Mercer, N. (2000). Words and minds: How we use language to think together. London: Routledge.Millar, R. (2004, June 3). The role of practical work in the teaching and learning of science. Paper

presented at the “High School Science Laboratories: Role and Vision” Meeting, Board on Science Education, National Academy of Sciences. Washington DC.

Mortensen, M. F., & Smart, K. (2007). Free-choice worksheets increase students' exposure to curriculum during museum visits. Journal of Research in Science Teaching, 44(9), 1389-1414.

Noë, A. and O’Regan, K.: 2002, ‘On the Brain-basis of Visual Consciousness: A Sensorimotor Account’, in A. Noë and E. Thompson (eds), Vision and Mind: Selected Readings in the Philosophy of Perception (pp. 567-598) MIT Press, Cambridge, MA.

Novak, J. D. (2002). Meaningful Learning: The Essential Factor for Conceptual Change in Limited or Inappropriate Propositional Hierarchies Leading to Empowerment of Learners. Science Education, 86, 548–571.

14

Nyamupangedengu, E., & Lelliott, A. (2012). An exploration of learners' use of worksheets during a science museum visit. African Journal of Research in Mathematics, Science and Technology Education, 16(1), 82-99.

Osman, M. E., & Hannafin, M. J. (1994). Effects of advance questioning and prior knowledge on science learning. The Journal of Educational Research,88(1), 5-13.

Piaget, J. (1935 and 1965). Science of Education and the Psychology of the Child. In H. E. Gruber & J. J. Voneche (Eds.) (1977), The essential Piaget (pp. 695-725). Northvale, NJ: Basic Books.

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson, D, & Soloway, E. (2004). A scaffolding design framework for software to support science inquiry. The Journal of the Learning Sciences, 13(3), 337-386.

Rennie, L. J., Feher, E., Dierking, L. D., & Falk, J. H. (2003). Toward an agenda for advancing research on science learning in out‐of‐school settings. Journal of Research in Science Teaching, 40(2), 112-120.

Rix, C., & McSorley, J. (1999). An investigation into the role that school-based interactive science centres may play in the education of primary-aged children. International Journal of Science Education, 21(6), 577-593.

Sousa, D. A. (2011). How the brain learns. Thousand Oaks, CA: Corwin Press.Stavrova, O., & Urhahne, D. (2010). Modification of a school Programme in the Deutsches Museum

to enhance students' attitudes and understanding. International Journal of Science Education, 32(17), 2291-2310.

Strauss, A., & Corbin, J. M. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA, Sage Publications, Inc.

Sung, Y. T., Hou, H. T., Liu, C. K., & Chang, K. E. (2010). Mobile guide system using problem‐solving strategy for museum learning: a sequential learning behavioural pattern analysis. Journal of Computer Assisted Learning, 26(2), 106-115.

Sutton, C. (1992). Words, science and learning. Buckingham, Open University Press Tal, R., Bamberger, Y., & Morag, O. (2005). Guided school visits to natural history museums in

Israel: Teachers' roles. Science Education, 89(6), 920-935. Watson, W. A. (2010). Middle school students’ experiences on a science museum field trip as

preparation for future learning (Unpublished doctoral dissertation.) The Graduate School of Education and Human Development of the George Washington University, Washington, DC.

Wellington, J., & Osborne, J. (2001). Language and literacy in science education. Buckingham, Open University Press.

Wells, G. (1999). Dialogic inquiry: Towards a socio-cultural practice and theory of education. Cambridge, Cambridge University Press.

Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, Mass.: Harvard University Press.

Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100.

Yoon, S. A., Elinich, K., Wang, J., Schooneveld, J. B., & Anderson, E. (2013). Scaffolding informal learning in science museums: How much is too much? Science Education, 97(6), 848-877.

15

Notes1. In some cases the intended focus of observations was simply outside of students’ field

of vision. This was caused by trivial physical reasons such as students’ height (e.g. not observing light in a bulb because it mainly shine upwards), or interaction design which resulted in exhibit-generated responses which caught their attention and thereby narrowed their focus of vision (e.g. observation of the hydro-power exhibit was drawn towards a screen with graphical presentation of data and not towards the physical phenomena). The pilot studies also confirmed that students in this age group have prior misconceptions which can be reinforced by exhibit interaction (e.g. magnets in the generator transferred their energy in the same way as batteries since they made the bulb shine). We also found that the phenomena presented by an exhibit can be misunderstood (e.g. a hydropower turbine draws water out of the nozzles).

2. Overall students’ observations were helped by the guidance provided by the EISs, and in their discussions they used the words that could be found in the presented texts.

16

APPENDIX

Exhibit Interaction Sheet (EIS) for the exhibit Generator (translated from Norwegian)

17

Concept Flow Chart (CFC)