Upload
harding-mercer
View
22
Download
0
Embed Size (px)
DESCRIPTION
21-Tech Formative Evaluation Phase III Results Presentation. CECILIA GARIBAY, PRINCIPAL JANE SCHAEFER, PROJECT DIRECTOR. Overview - PowerPoint PPT Presentation
Citation preview
21-Tech Formative Evaluation Phase III Results PresentationCECILIA GARIBAY, PRINCIPAL
JANE SCHAEFER, PROJECT DIRECTOR
1
Overview21-Tech is an IMLS-funded project focused on developing and broadly sharing methods for increasing the capacity of exhibit facilitators to use personal mobile technologies (PMTs) to engage museum visitors in learning. This presentation document provides key results from Phase III of formative evaluation, which focused on testing facilitator training.
Formative Evaluation Formative evaluation for 21-Tech involved three phases. Phase I involved an exploratory phase where project partners tested various strategies for PMT facilitation at exhibits. Phase II focused on understanding facilitator-visitor interactions and visitor perspectives of the experience.
Phase III formative testing—the focus of this document—assessed the extent to which training successfully prepared facilitators to facilitate PMT-based experiences with visitors. Results were used to improve the training. Phase III MethodsData were collected from June to September 2012. Methods included facilitator surveys, peer observations, and virtual focus groups.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 20132
Phase III Evaluation Questions
• To what extent does training increase participating staff’s knowledge of successfully facilitating PMT-based visitor experiences?
• To what extent does the training influence participants’ confidence in and motivation toward facilitating PMT-based visitor experiences?
• What components of the training seem to be most successful? What aspects are not as successful and how could they be improved?
Observation data were collected at each partner organization to determine how facilitators were applying concepts covered in the training. Observations consisted of peer-to-peer sessions in which facilitators paired up to observe and provide each other feedback. Twenty-five observations were completed. (See Appendix A breakout of observations by site.)
Facilitators completed an on-line survey after having been trained in PMT-based facilitation and had engaged with visitors on the museum floor. The survey was conducted to learn facilitators’ perceptions of and experiences with the training program. Twenty staff members completed surveys at partner sites. (See Appendix A for details by site.)
Overview, cont’d.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 3
Garibay Group staff also conducted four focus groups (one with facilitators at each partner museum) via video conferencing to collect more in-depth data, probe more deeply into survey findings, and gain increased insight into facilitators’ experiences.
Garibay Group staff developed evaluation protocols and instruments, worked with partners throughout data collection, and analyzed the data.
Facilitation staff at the four partner institutions conducted peer observations, recorded their observations using a form consisting of close-ended and open-ended questions, and submitted completed data to the evaluators.
Data AnalysisGaribay Group, as the external evaluator, conducted all data analysis.
Survey data were analyzed through descriptive statistics. Summary statistics were calculated and summarized in histograms and cross-tabulations.
Descriptive observation data and open-ended survey questions were qualitatively analyzed and coded using an emergent coding strategy (Altheide, Coyle, DeVriese, and Schneider, 2008). In emergent coding, categories are established after an examination of the data.
LimitationsOne major limitation of this study involved observation data. While data were intended to provide insight on the ways and extent to which facilitators applied skills and ideas learned during training, there was not always enough description of facilitator-visitor interaction to draw clear conclusions. Thus, observation data were primarily used to triangulate findings garnered from the survey data.
Garibay Group originally gave this presentation to the 21-Tech partners. This overview section was added to provide context for the reader.
Participating Facilitator Background
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 4
Facilitator BackgroundMost facilitators participating in this study were part-time paid staff or volunteers.
Intern Volunteer Part-time paid staff Full-time paid staff0
2
4
6
8
10
12
2
7
11
0
Your role at your science center or museum
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 5
Facilitator Background, cont’d. Only a quarter of facilitators personally owned a tablet. Thus, most facilitators’ experiences with a tablet came primarily through use on the job.
Own an iPad Own an Android tablet Do not own any tablet0
2
4
6
8
10
12
14
16
32
15
Do you personally own a tablet?
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 6
Facilitator Background, cont’d. Most facilitators had little work experience in science centers or museums and, in general, little facilitation experience. As a result, less experienced respondents could not deeply reflect on their facilitation practice; most were unable to compare facilitation with PMTs to facilitation without them. Less experienced facilitators focused on technical and practical aspects—for example, readily accessing apps, carrying the tablet, or mastering content. They were not as focused on how PMTs could inform their approach to facilitation and their interactions with visitors.
1 Year or less 2-4 Years 5-7 Years 8-10 Years 11+ Years0
2
4
6
8
10
12
1412
5
3
0 0
Facilitator experience working in the science center/museum field
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 7
Facilitator Background, cont’d. A slightly higher number of facilitators rated themselves as competent or expert in facilitating visitor experiences than in using PMTs as tools for facilitation.
Your experience facilitating visitor experience at exhibits.
Using personal mobile tech-nologies as a tool to facilitate visitor experience at exhibits.
0
2
4
6
8
10
12
01
45
8
11
8
3
Rate your knowledge/experience in the following:
Novice Beginner Competent Expert
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 8
Facilitator Feedback
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 9
ExpectationsThe majority (85%) of facilitators reported that the training met or exceeded their expectations. Fewer than a third, however, reported that their expectations were exceeded.
Missed Nearly met Met Exceeded0
2
4
6
8
10
12
0
3
11
6
Overall, how well did this trainingmeet your expectations?
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 10
Satisfaction with TrainingFacilitators were generally satisfied with most aspects of the training; in most areas, 80% or more indicated being “satisfied” or “very satisfied”. Facilitators, however, provided lower satisfaction ratings of the training module explaining how apps related to exhibit content.
Please rate your satisfaction with various aspects of the training:
Not satisfied
Somewhat satisfied Satisfied
Very satisfied
Overview of the 21-Tech project 0 2 9 9
Introduction to the tablet 0 3 6 11
Overview of apps 0 4 8 8
Content connections between app and exhibits 2 4 4 10
Facilitating visitor experiences with the tablet 0 4 10 6
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 11
Usefulness of TrainingAt least eighty-five percent of facilitators rated the four training activities as “moderately” or “very” useful. Based on interviews and survey comments, the role-playing activities may have been less useful because not enough time was devoted to role-playing during training.
How useful were the following:Not
usefulSomewhat
usefulModerately
usefulVery
useful
Individual time to explore the device and apps0 2 4 14
Group discussions as part of training1 1 5 13
Roleplaying activities1 2 7 10
Time spent in the galleries practicing skills with actual visitors
0 1 5 13
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 12
LearningAll facilitators responding to the survey reported increased knowledge in all aspects covered by the training. Facilitators especially noted learning to approach and engage visitors with the tablet and to successfully facilitate interactions at exhibits using the PMT. Facilitators reported lower ratings in the training’s impact on their knowledge of the tablet and apps. It is not clear, from facilitator comments, why this category received the lowest ratings; facilitators may have already known enough about these areas that the training had minor impact. We will investigate this in the next project phase.
How much did the training increase:
Did not increase
Increased slightly
Increased moderately
Increased significantly
Your knowledge of the tablet 0 6 5 9
Your knowledge of apps 0 3 9 8
Your understanding of how to approach and engage visitors with the tablet 0 0 7 13
Your knowledge of how to successfully facilitate visitor interactions at exhibits using the tablet 0 4 4 12
13Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013
Most Useful Skills or Ideas LearnedWhen asked “What three skills or ideas that you learned from the training were the most useful in facilitating visitor experiences?”, facilitators most mentioned these topics:
• Integrating the iPad into an explanation or activity with an visitor"Using the applications made explaining certain things much easier since some ideas are easily understandable with visual aids provided by 21-Tech. I was also more likely to engage visitors of more diverse age groups with different levels of understanding since the applications worked with the majority of visitors.”
• Approaching visitors with an iPad“[I learned to] Be as specific as possible when approaching visitors. Be short and sweet. Have your approach be tailored to the age of the visitor.”
• Making connections between an exhibit and an app “I learned how to make connections between the apps and the [tablet]. For example, with the Sciencenter colored shadows exhibit and the tablet we were trained well and this helped me facilitate with success. The more specific training on content, the more confident I felt approaching visitors.”
• Learning which apps fit a specific exhibit or situation“[I learned] All the different apps that we can use for specific exhibits.”
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 14
ConfidenceAfter the training, 95% of facilitators reported increased confidence in facilitating visitor experiences using PMTs at exhibits. Only one facilitator rated his or her confidence a “3” both before and after.
1 (low) 2 3 4 (high)0
2
4
6
8
10
12 11
6
3
0
BEFORE training
My confidence in successfully facilitating visitor experiences using the tablet at exhibits:
1 (low) 2 3 4 (high)0
2
4
6
8
10
12
0 0
9
11
AFTER training
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 15
Equipped to Facilitate with PMTsThree-quarters of facilitators reported feeling “adequately” or “very” equipped after training to successfully facilitate visitor experiences at exhibits using the tablet.
Not at all equipped Somewhat equipped Adequately equipped Very equipped0
2
4
6
8
10
12
0
4
11
5
How well equipped do you now feel to successfully facilitate visitor experiences at exhibits using the tablet?
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 16
Facilitator Understanding of PMT RoleAll facilitators agreed that PMTs could enhance visitors’ learning, with 80% providing “strongly agree” ratings. It is intriguing to find such high agreement by facilitators on this issue. In the next project phase, we will further explore this area to learn how facilitators view PMTs as enhancing visitor learning.
Disagree strongly Disagree somewhat Agree somewhat Strongly agree0
2
4
6
8
10
12
14
16
18
0 0
4
16
Please rate your level of agreement with this statement: “I clearly see the role that Personal Mobile Technologies, like the tablet,
can play in enhancing visitors’ learning.”
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 17
Motivation to use PMTsTraining appeared to successfully motivate facilitators to use PMTs with visitors. Nearly all reported being motivated “a lot” (45% of respondents) or a “great deal” (50%) to use PMTs in facilitating visitor experiences.
Not much Some A lot A great deal0
2
4
6
8
10
12
01
910
Right now, how motivated are you to use PMTs to facilitate visitor experiences?
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 18
Value ProfessionallyAll facilitators agreed that PMT-facilitation training was professionally valuable. Responses were almost evenly split, with 45% reporting the training to be “moderately valuable” while 50% called it “very valuable.” This split suggests that the training could be improved.
Not valuable Slightly valuable Moderately valuable
Very valuable0
2
4
6
8
10
12
01
910
How valuable, professionally, was PMT-facilitation training for you?
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 19
ImprovementsFacilitators suggestions to improve the training included providing more examples, more time to role-play, and more time on the floor to practice using the tablets.
More opportunities for role-playing
Levels of training based upon facilitators’ previous experience with tablets
More examples of successful interactions (e.g., videotaped interactions)
More time to practice
More examples of ways to link exhibits to apps
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 20
Challenges Identified
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 21
Fit between app and exhibit
Facilitators reported challenges in making apps fit the exhibits. For example, they may have been given an app they thought to be only tangentially related to an exhibit.
Children competing for table
Visitors, especially children, sometimes struggled to share tablets. Facilitators sometimes found themselves having to mediate between siblings all vying for time on the tablet.
Crowded or loud exhibits
Not surprisingly, facilitation in crowded or loud exhibits was more difficult than in less crowded or quieter exhibits. In one overly crowded exhibit, for example, a facilitator said, children competed to grab the tablet.
Feeling confident with exhibit content or app
It takes time for facilitators to feel confident with exhibit content or an app. While the trainings were comprehensive, the main issue was often ensuring that each module was long enough, particularly role-playing practice time and time on the floor with visitors.
Beginning and ending interactions
Some facilitators found the most challenging aspect of facilitation to be either approaching visitors to begin an interaction or ending an interaction comfortably. Sometimes visitors liked the tablet so much, they wanted to stay and play on it. Other times, a child might have felt she was “done” with the interaction before the facilitator was able to end it effectively.
Integrating certain exhibits with apps on the tablet was difficult, but the applications themselves demonstrate abstract ideas that may be challenging for a visitor to visualize.
Some apps only had slight relevance to the exhibit, and it would require taking explanations in a specific directions to have the app be helpful.
There really needs to be more time exploring and getting comfortable with ALL of the apps….Also, some exhibits did not even have apps, or the quality of apps was so low that they would never—and will never—be used.
Feeling confident with the content [is important]! It would be useful to have more background information on the content.
The more completely training can simulate actual use of the tablet with visitors, the better it will be!
[I wish] we had more time to experiment with exhibits, and to train on specific apps, or even if we had more apps.
Recommendations
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 22
Recommendations
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 23
Main Training TopicsOne key finding that emerged is that training must address five main topics:
• General facilitation skills: for instance, how to talk to visitors and how to engage with questions
• Exhibit content including the science underlying the exhibit
• Tablet hardware
• Specific apps, including how the apps are organized on the tablet and—most importantly—how the apps relate to the exhibit
• Specific facilitation skills for tablet interactions; for example, how to approach visitors with the tablet, how to help visitors share a tablet, and how to end an interaction. This training needs to include role-playing with other facilitators or staff before trying the techniques with visitors.
Training PathsOne consideration in training is to understand the relative experience of the facilitator and provide a training track appropriate to the individual’s experience.
Based on the data, we identified three different training paths based on the three types of facilitators:
1. Novice Facilitator
2. Tech-Savvy Facilitator
3. Experienced Facilitator
The graphics on the following pages illustrate each of the three paths.
Structure and Time of TrainingThe 21-Tech training can be divided into three types: 1) classroom time, i.e., traditional training in which the content is presented to facilitators; 2) time on the floor to practice what was learned; and 3) reflection time for facilitators to look back and critically consider their interactions with visitors.
Based on the data, facilitators felt they needed more time to practice on-the-floor facilitation with tablets with other facilitators as well as visitors. It also appeared that, in most cases, adequate reflection time was not built into the training structure. Because of these two factors, facilitator training should not be structured as a one-time training, but as a process with several cycles.
Facilitator Training: 3 Tracks
General Facilitation
Exhibit Content
iPad Hardware
Specific Apps
Facilitation with iPad
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 24
Tech-savvy Facilitator May not need general training on the iPad hardware.
General Facilitation
Exhibit Content
iPad Hardware
Specific Apps
Facilitation with iPad
Novice Facilitator General facilitation training is essential.
Experienced facilitatorDespite experience, needs time to role-play and practice facilitation with the iPad.
General Facilitation
Exhibit Content
iPad Hardware
Specific Apps
Facilitation with iPad
Reflective PracticePhase III findings also have implications for facilitators’ ongoing reflective practice. Ideally, following training, facilitators would use with visitors what they learned on the floor then discuss their experience with other facilitators; collaboratively, they would refine their approaches, choose new apps for exhibitions, and/or choose new exhibitions for which to add facilitation. They would then test new facilitations through role-playing before using them with visitors, re-starting the cycle.
Facilitate with Visitors
Discuss Experience with
Other Facilitators
Refine Approach/ Explore New Apps/
Apply to Other Exhibits
Role Play Using New Approach, App, or Exhibit
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 25
References and Appendices
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 201326
ReferencesAltheide, D., Coyle, M., DeVriese, K., &
Schneider, C. (2008). Emergent Qualitative Document Analysis. In Handbook of Emergent Methods. Hesse-Beiber, S.N. and Levy, P., eds., NY, NY: Guilford Press.
Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis (2nd ed.). London: Sage Publications.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 27
Appendix A: Data Sources by SiteThe table below details data sources by institution.
Facilitator Training Evaluation CMH NYSCI OMSI Sciencenter
Total
Peer observations 4 10 7 4 25
Facilitator surveys 4 6 8 2 20
Virtual focus groups (# of facilitators)
1 (5) 1 (4) 1 (1) 1 (2) 4 (12)
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 28
Appendix B: Training Plans
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 29
CMHIntroductory Session (2 hours minimum) • Discussion of what 21-Tech is, who our partners are, goals of the program, and the vision for CMH• Introduction to the iPad: describing features, tips, tricks, how to use basics• Explanation and examples of the different categories of apps (content construction, skill development, and experimentation/exploration) and
the app-exhibit connections for each. • Individualized exploration of apps, getting used to iPads, and Q/A regarding iPads and apps• Discuss how to use apps to lengthen, extend and deepen visitor experiences. Also discuss how NOT to approach visitors with an iPad. • Individual time to explore apps and discover the app-exhibit connections. • Visit Kidropolis and demonstrate (or role-play) how to make the app-exhibit connections when interacting with a visitor. Visit the Vet Clinic,
Diner, and/or Bank. • App choices: iWashDogs, iPizza, Big Fork Little Fork, BillsCoinsDrills, Jungle Math• Feedback from trainer, discussion and reflection
Content Connections and Skill Development Training (1.5 hours minimum)• Discuss new apps facilitators have discovered/used since the last training. • Allow facilitators time to explore the Invention Convention apps: Paper Plane Project, World Record Airplanes, and Wind Tunnel. Discuss the
approach for using these apps. Allow facilitators to suggest different approaches/uses. • Visit Invention convention and practice making paper airplanes. Also, practice using the Wind Tunnel app in the gallery. • Feedback from trainer, discussion and reflection
Experimentation/Exploration Facilitation Training (1.5 hours minimum)• Review the apps which act as tools to help kids explore and experiment in various exhibits. • Allow facilitators time to explore the iMotion HD app, and how stop motion works. Allow facilitators time to experiment with the app. When
done, discuss where/how the app may be used. Discuss other apps which work as tools, such as the Giant Timer and Camera. • Practicum time facilitating visitor experiences, using iMotion HD, Giant Timer and/or Camera. • Feedback from trainer, discussion and reflection**If needed, schedule follow-up training (1.5hrs) to provide facilitators more time to practice interacting with visitors using the iPad.
Final Practicum (1.5 hours minimum)“Solo” experience. Assess ability to utilize the facets of the iPad to enhance visitor experience. Determination if training is completed..
Appendix B: Training Plans, cont’d.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 30
NYSCI Day 1GoalsIntroduce Explainers to the 21-Tech ProjectExpose Explainers to the various digital tools used in 21-Tech
ObjectivesExplainers will be able to navigate and operate through the iPod, IPad2, and the Motorola Xoom
• Intro to 21-Tech Project• How to use the Xoom• How to use the iPad/iPod (iOS)• Scavenger Hunt (Two groups. Each group takes a device to complete)
Day 2• Scavenger Hunt 2 (Same groups as previous day. Each group use a different device than previous day)• Peer Training• Take turns explaining the exhibit to each other to gain a deeper understanding.• App review• Spend time going through each app
Day 3• Peer Training (Explainers explain exhibits they were assigned, incorporating PMTs into their facilitation).• Reflect on the day/training
Appendix B: Training Plans, cont’d.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 31
OMSIThis training doesn’t give them everything they need to use iPads on the floor, but it does get them familiar with the iPad, and how we expect it to be used. Next steps after training include checking in with educators in their work area, and becoming familiar with area specific apps.
DEMO Objective/ Bower Birds 20 minutes• The training opens up with the instructors facilitating two iPad demos as examples.
INTRODUCTION Welcome 10 minutes• Individual introductions. Name, work area, familiarity with iPads.• Introduce the 21-Tech grant; Why we have the iPads, how we intend to use them.• The goals and limitations of this training.
INSTRUCTOR LEAD ACTIVITY Meet the iPad 45 minutes• Introduction to the iPad where volunteers will be introduced to the basic functions and features
INSTRUCTOR LEAD ACTIVITY Switch iPads 20 minutes• Volunteers switch iPads and repeat all of the functions so they become familiar with how to use them, even if iPads are organized differently.
INDIVIDUAL ACTIVITY Break/Free explore 20 minutes
INSTRUCTOR LEAD ACTIVITY App Races 10 minutes• This is time to apply what they’ve learned. Give the group a challenge and see how long it takes for all of them to meet that challenge.
DEMO Proscope 5 minutes
GROUP DISCUSSION Rules of the Road and Informal vs. Formal Education 15 minutes• How is what we do here at OMSI (informal education) different than formal education? • How should we hold/ present the iPad?• How do you decide if it’s an appropriate time to use the iPad?• When should the iPad be used on the floor (with visitors)?• For what should the iPad be used?
INSTRUCTOR LEAD ACTIVITY Next Steps 5-10 minutesDiscuss successes and learning’s
INDIVIDUAL ACTIVITY Demo Design Plan 20 minutesDiscuss different ways the iPad may be incorporated into work areas or explore apps. Participants should leave with an idea of a specific exhibit or demo they want to work with.
Appendix B: Training Plans, cont’d.
Garibay Group | 21-Tech | Formative Evaluation Phase III Results Presentation | Winter 2013 32
SciencenterIntro to 21 Tech (30 min)• Review 21-Tech project• 21-Tech Factsheet• Review Sciencenter’s tips for interacting with visitors and facilitation strategies for 21-Tech exhibits:• Training Tips• Five-Step Process• Maddie’s YouTube video• Learn to use each of the 21 Tech tools: iPad and iTouch• Free exploration with technology and tools
Exhibit Training (30 min)• Introduce and review background materials for exhibit components and discuss science content(ex. Colored Shadows):• Colored Shadows Exhibit Guide• Colored Shadows Prompts• iPad diagrams, images, and photos• Introduce and practice using the apps: Bobo and Color Uncovered• Practice facilitation with another volunteer or staff person Reflect on the Experience• Discuss what works and what doesn’t• How do you initiate an interaction (what is the hook)?• How do you incorporate PMTs into the interaction?• How do you transition from exhibit to tool and back to the exhibit?• How do you connect content to visitors’ lives?• How do you use with younger audiences or older audiences?• What are the differences in working with families and school groups?• Recommendations and suggestions