21-Tech Formative Evaluation Phase I Results Presentation

21-Tech Formative Evaluation Phase I Results Presentation

21-Tech PARTNERS’ MEETING

BALTIMORE• OCTOBER 17, 2011

CECILIA GARIBAY, PRINCIPAL

JANE SCHAEFER, PROJECT DIRECTOR

Overview21-Tech is an IMLS-funded project focused on developing and broadly sharing methods for increasing the capacity of exhibit facilitators to use personal mobile technologies (PMT) to engage visitors in learning at their museums. A major aspect of the project focuses on testing strategies for using PMTs as part of facilitation.

This presentation was given to the 21-Tech partners by Garibay Group in order to generate team discussion and arrive at consensus about the Phase II testing. The overview section was added to provide context for the reader. Appendices were added as well.

Phase I TestingStaff from the four partner museums (Children’s Museum of Houston, New York Hall of Science, Oregon Museum of Science and Industry, and Sciencenter) conducted Phase I Testing in July and August, 2011. This early testing phase was exploratory in nature and was designed to give staff the opportunity to pursue a range of questions about how PMTs might be used by facilitators to positively engage visitors in various exhibits.

Garibay Group worked with the project team in the development of testing protocols and instruments and analyzed data collected.

The overarching question selected by the project partners for Phase I was:

How can gallery facilitators in my organization use PMTs to lengthen, deepen, and extend visitor interactions?

Partners, however, also wanted to be able to pursue more specific questions of interest to their institutions and the 21-Tech project. Therefore, the more specific questions pursued during each round of testing varied both within and between partner institutions. (See sample questions in the blue box on the right.)

2 Garibay Group | 21-Tech | Formative Evaluation Phase I Results Presentation | October 2011

Phase I Testing Questions

The following are a sample of questions that emerged over the course of testing:

• What are some strategies to work around distractions (e.g., high noise level, crowds, etc.)?

• What kinds of exhibits interactions work the best, for instance, observing, hands-on, etc.?

• How can we use the PMT in ways that are different from other extension activities? What about the PMT makes for a special or unique kind of interaction?

Overview, cont’d.Visitor groups were recruited from those visiting the partner museums on the data collection days. (See Appendix A for a summary of data collection rounds and visitor information.)

Data DocumentsDocuments analyzed included:• 27 testing debriefs completed

after each round of testing;• Testing results synthesis

summaries;• Interview documents from one-on-

one partner interviews;• Monthly team meeting notes

During the testing period, the external evaluator interviewed each partner institution’s team members to assess the progress of the testing process and further discuss findings that were emerging from the testing.

After Phase I testing was complete, each institution created a synthesis


Phase I Testing consisted of 21-Tech partner museum staff prototyping facilitation strategies with PMTs in one of more of their exhibits. Each partner site tested PMTs with visitors using a range of content (applications, games, videos, photographs, etc.). Each partner also made their own choices about what to try and in which exhibits to test PMTs.

In most cases, testing took the form of peer-to-peer observations where one facilitator tested ascenario with visitor groups and another team member took notes. In some cases, a facilitator tested a scenario with visitors and then was interviewed about the test afterwards by another team member. Staff then completed a debrief where they recorded data about visitor groups, interactions, observations, and findings.

summary of their top three to five findings with supporting observations.

Throughout Phase I, the project team leads and coordinators and evaluators met monthly by phone to share progress, discuss findings, and reflect on their process.

Data AnalysisGaribay Group, as the external evaluator, conducted an independent analysis of Phase I testing data.

The testing debriefs were coded using and emergent coding strategy (Altheide, Coyle, DeVriese, and Schneider, 2008). With emergent coding, categories are established after an examination of the data. Instead of preconceived themes or categories, researchers analyze data (in this case the debrief documents) to identify themes that recur.

Overview, cont’d.As patterns and themes were identified, researchers used a constant comparison method to tease out the strength of these patterns and themes (Miles and Huberman, 1994).

Coded data were then clustered and analyzed to look for interrelation-ships between categories, whichthen allowed us to more specifically refine broad themes and sub-categories. (See Appendix B for a sample coding scheme developed from 21-Tech data analysis.)

A major limitation in analyzing the debriefs was that there was notalways enough description of the interactions to independently verify the conclusions stated in the debrief document. Researchers were able to address this limitation, to some extent, by examining summaries, partner interviews, and team meeting notes. These documents


.helped supplement and flesh out the debrief data and served to clarify areas where data were not always clear or detailed enough and generally helped triangulate findings.

Using PMTs to Engage VisitorsBased on testing data, we identified four categories that influenced the interaction between a visitor and facilitator, which ultimately impacted both visitor and facilitator when PMTs were used. Though these categories may overlap to some extent, separating them allowed us to more systematically organize findings and more easily discuss them.

* Includes applications, games, videos, photographs, pdf’s, etc.

Garibay Group | 21-Tech | Formative Evaluation Phase I Results Presentation | October 2011 5

Facilitation Skills

In order to successfully use and engage visitors with PMTs, facilitators need comfort with and knowledge of the following:

• The content of the exhibit or component

• The apps that will be used and how they relate to the exhibit or component

• The device, including how to access apps on it

• Ways to integrate the PMTs into their facilitation with the visitor

Implication: Targeted training is needed that addresses all four areas.


Devices

Tablets, such as the iPad, tended to work better during interactions than smaller devices, such as the iTouch, because they were:

• Big enough to be shared easily by more than one person

• Easy to see what is displayed on them

Implication: From a research standpoint, do smaller devices need more testing or does the 21-Tech team feel that tablets are the ideal PMT for facilitation?


Apps

Based on the data from all partners, the three categories for apps defined by the Children’s Museum of Houston seemed to hold:

1. Concept Enrichment (e.g, Richochet, the Elements, bee videos) that help explain something in more depth;

2. Skill Building (e.g., Launch Cage Paper Airplanes, Steady Hand) that allow a visitor to practice how to do something;

3. Exploration and Experimentation Tools (e.g., Widget Wall, Giant Timer, iSeismometer) that help visitors quantify phenomena.

Implication: These categories might be a useful in helping identify ways to enrich experiences. Categories may be revised and refined as data are gathered during Phase II testing.


Apps

Apps that worked well during interactions have the following characteristics:

• A tight relationship to the physical exhibit or component that is easy for the visitor to understand

• Easy for visitors to understand how to use

• Not so compelling as a standalone app that it takes away from the exhibit experience

• Work for multi-aged social groups (i.e, apps are “juicy” enough for adults and children to use)

• Age-appropriateness needs to be considered. The target age range for the 21-Tech project is 5 to 13 years old.

Implication: Selection of appropriate apps takes careful consideration, and 21-Tech partners have spent significant time finding appropriate apps.


Exhibit

Types of exhibits where facilitation using PMTs worked well:

• Those where PMT apps could show visitors something they could not see in the exhibit (e.g., Bees, Sports Challenge)

• Those where adding a challenge could make it more engaging (e.g., Mt. Boulder)

• Those that provoked questions that could be answered using on-the-floor, on-demand references (e.g., Chemistry Lab)

• Those in which visitors have to wait for turns (e.g., Facepainting)

• Those that were not too loud or crowded

Implication: An outstanding question is can all exhibits benefit from PMT-based facilitation?


Visitor Impact

Based on an analysis of debriefs, documentation of supporting observation about visitor engagement were not detailed enough to make definitive conclusions. The following statements were made regarding interactions in which facilitators used PMTs:

• Interactions were richer when PMTs were used

• Interactions lasted longer with PMTs

• Visitors were more excited about the interactions with PMTs

• PMTs made visitors more comfortable with facilitator

Implication: Focus on documenting and understanding how facilitation using PMTs impacts the quality of visitor interactions.


Facilitator Impact

Based on an analysis of debriefs, documentation of supporting observation about facilitator impact were not detailed enough to make definitive conclusions. The following statements were made regarding facilitators:

• PMT use was increasing facilitators’ knowledge and skill sets

• PMT use was making facilitators more confident about facilitation

Implication: Focus on documenting and understanding how using PMTs impacts facilitators.


Questions that Emerged from the Data

• What are the characteristics of a tight relationship between exhibit and app?

• Does every exhibit need PMT-based facilitation?• What are the characteristics of successful engagements for

a visitor group?• Do smaller devices need more testing or does the 21-Tech

team feel that tablets are the ideal PMT for facilitation?


Potential Directions for Next Phase

Based on Phase I research, the 21-Tech team might consider the following:

• Narrow the research question(s)• Describe the nature of the visitor-facilitator interaction• Document how facilitation using PMTs impacts the

quality of visitor interactions• Document how using PMTs impacts facilitators• Work on more standardized data collection


ReferencesAltheide, D., Coyle, M., DeVriese, K., & Schneider, C.

(2008). Emergent Qualitative Document Analysis. In Handbook of Emergent Methods. Hesse-Beiber, S.N. and Levy, P., eds., NY, NY: Guilford Press.

Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis (2nd ed.). London: Sage.


.

Appendix A: Data Collection Rounds


Children's Museum of Houston Phase I Testing

Institution Day Male Female Unspecified Male Female

CMH 7/5/2011 1 0 10, 11, 11 3 3

CMH 7/5/2011 2 1 1 2 9 7 2 4

CMH 7/8/2011 3 1 1 7.5, 7.5, 7.5 7.5 4 5

CMH 7/8/2011 4 0 9, 7 2 2

CMH 7/12/2011 1 1 1 7 1 2

CMH 7/12/2011 2 2 2 8, 10 12 3 5

CMH 7/14/2011 3 0 12 1 1

CMH 7/14/2011 4 0 10 1 1

CMH 7/26/2011 1 1 1 6 1 2

CMH 7/26/2011 2 0 8 1 1

CMH 7/28/2011 1 0 8 1 1

CMH 7/28/2011 2 1 1 5, 10 2 3

CMH 8/2/2011 1 0 12 1 1

CMH 8/2/2011 2 0 11 1 1

CMH 8/2/2011 3 0 8 1 1

CMH 8/2/2011 4 0 6 1 1

CMH 8/2/2011 5 0 11, 12 2 2

CMH 8/11/2011 1 0 7, 8 2 2

CMH 8/11/2011 2 0 5, 11 2 2

CMH 8/11/2011 3 0 11 1 1

CMH 8/18/2011 1 0 6, 7 2 2

CMH 8/18/2011 2 0 5 1 1

CMH 8/18/2011 3 0 7 1 1

CMH 8/18/2011 4 0 6, 8 2 2

CMH 8/18/2011 5 0 2, 8 2 2

CMH 8/18/2011 6 0 5 1 1

CMH 8/18/2011 7 0 6, 7 2 2

CMH 8/23/2011 1 1 1 2, 4 2 3

CMH 8/23/2011 2 0 4 1 1

CMH 8/25/2011 3 1 1 4 1 2

CMH 8/25/2011 4 1 1 2 4 1 3

Totals 4 7 1 12 32 17 49 61Total # of Groups: 31

Avg. Age: 7.3

Avg. Age: 7.0

Total # of Visitors

Adults Child Age and Gender

Group

Testing Date# of

Adults# of

Children

Appendix A : Data Collection Rounds, cont’d.


New York Hall of Science Phase I Testing

Institution Day Male Female Unspecified Male Female Unspecified

NYSCI 8/8/2011 1 0 7 5 6, 7 4 4

NYSCI 8/8/2011 2 0 10 1 1

NYSCI 8/8/2011 3 1 1 2 11 10 6.5 3 5

NYSCI 8/11/2011 1 1 1 4 1 2

NYSCI 8/11/2011 2 0 12 1 1

NYSCI 8/15/2011 1 0 6.5 1 1

NYSCI 8/15/2011 2 0 8.5, 5.5, 5.5, 5.5, 5.5, 5.5 6 6

NYSCI 8/15/2011 3 0 7.5 1 1

NYSCI 8/16/2011 1 1 1 6,7 2 3

NYSCI 8/16/2011 2 1 1 6 1 2

NYSCI 8/16/2011 3 0 6.5 1 1

NYSCI 8/16/2011 4 1 1 9 1 2

NYSCI 8/16/2011 5 1 2 3 8, 8, 12 3 6

NYSCI 8/17/2011 1 1 1 7 1 2

NYSCI 8/17/2011 2 1 1 5.5 1 2

NYSCI 8/17/2011 3 1 1 6, 6, 6, 6, 6 5 6

NYSCI 8/17/2011 4 0 6, 6 2 2

5 6 1 12 10 16 9 35 47

Total # of Groups: 17

Average Age: 8.0

Average Age: 6.9

Testing Date

Group

Adults Child Age and Gender # of Children

Total # of Visitors

# of Adults

Totals:



Oregon Museum of Science and Industry Phase I Testing

Institution Day Male Female Unspecified Male Female Unspecified

OMSI 7/1/2011*

OMSI 7/1/2011*

OMSI 7/24/2011 1 1 1 7, 10 2 3

OMSI 8/1/2011*

OMSI 8/1/2011*

OMSI 8/1/2011*

OMSI 8/19-8/26/11 1 1 1 2 8 1 3

OMSI 8/19-8/26/11 2 1 1 0 1

OMSI 8/19-8/26/11 3 1 1 15 1 24 2 1 2 5 0 0 4 4 9

Total # of Children

Total # of Visitors

*Due to the style of data collection on these days, it was not possible to discern the numbers and ages of visitors engaging with PMTs. Due to the limited amount of data, an analysis of child age frequency is not respresentative.

Testing Date

Group

Adults Child Age and GenderTotal # of Adults:

Totals:



Sciencenter Phase I Testing


SCI 7/1/2011 1 1 1 6, 8 2 3

SCI 7/1/2011 2 1 1 9 1 2

SCI 7/1/2011 3 1 1 5, 7, 8, 9, 10 5 6

SCI 7/1/2011 4 1 1 7 1 2

SCI 7/1/2011 5 0 10, 10 2 2

SCI 7/1/2011 61 1

10, 10, 10, 10, 10 5 6

SCI 7/1/2011 7 1 1 2 3 1 3

SCI 7/1/2011 8 0 11 1 1

SCI 7/1/2011 9 1 2 3 0 3

SCI 7/1/2011 10 1 1 5 1 2

SCI 7/1/2011 110

10, 10, 10, 10, 11, 11, 11, 11

10, 10, 10, 11, 11, 11, 11 15 15

SCI 7/7-7/8/2011 1 1 1 10 1 2

SCI 7/7-7/8/2011 2 1 1 11, 11 2 3

SCI 7/7-7/8/2011 3 0 16 1 1

SCI 7/7-7/8/2011 4 1 1 4 1 2

SCI 7/7-7/8/2011 5 1 1 7, 11 3 3 4

SCI 7/7-7/8/2011 6 2 2 3, 3 2 4

SCI 7/7-7/8/2011 7 2 2 5 1 3

SCI 7/7-7/8/2011 8 0 10 1 1

SCI 7/7-7/8/2011 9 1 1 11 15 2 3

SCI 7/7-7/8/2011 10 1 1 9, 9 2 3

SCI 7/7-7/8/2011 11 0 9 9, 9 3 3

SCI 7/7-7/8/2011 12 0 8, 8 2 2

SCI 7/7-7/8/2011 13 0 10 1 1

SCI 7/7-7/8/2011 14 2 2 3 1 3

SCI 7/7-7/8/2011 15 0 7 1 1

SCI 7/7-7/8/2011 16 0 7, 7, 8 3 3

SCI 7/14-7/15/11 1 1 1 3 1 2

Total # of Visitors

Child Age and GenderTesting Date

Group

Adults # of Adults

# of Children



Sciencenter Phase I Testing, cont'd


SCI 7/14-7/15/11 2 2 2 4 1 3

SCI 7/14-7/15/11 3 1 1 2 5, 7 2 4

SCI 7/14-7/15/11 4 1 1 8 1 2

SCI 7/14-7/15/11 5 0 9 1 1

SCI 7/14-7/15/11 6 1 1 5 1 2

SCI 7/14-7/15/11 7 1 1 6 7 2 3

SCI 7/14-7/15/11 8 1 1 2, 5 2 3

SCI 7/14-7/15/11 9 1 1 2 1 2

SCI 7/14-7/15/11 10 2 2 6, 6 2 4

SCI 7/14-7/15/11 11 1 1 7, 7 2 3

SCI 7/14-7/15/11 12 0 12 1 1

SCI 7/14-7/15/11 13 1 1 4 1 2

SCI 7/21/11 1 0 4, 6 2 2

SCI 7/21/11 2 1 1 4, 6 2 3

SCI 7/21/11 3 0 12, 14 2 2

SCI 7/21/11 4 0 7 16 2 2

SCI 7/21/11 5 1 1 0 1

SCI 7/21/11 6 0 8, 8, 8 3 3

SCI 7/21/11 7 0 8, 8, 9, 9 4 4

SCI 7/22/2011 1 1 1 4, 7, 7 3 4

SCI 7/22/2011 2 0 7 14 2 2

SCI 7/22/2011 3 1 1 4, 7 2 3

SCI 7/22/2011 4 1 1 3 1 2

SCI 7/22/2011 5 1 1 2 8 1 3

SCI 7/22/2011 6 1 1 8 7 2 3

SCI 7/29/2011 1 1 1 2 3, 5, 7 3 5

SCI 7/29/2011 2 1 1 7 4 2 3

SCI 7/29/2011 3 1 1 12 1 2

SCI 7/29/2011 4 0 10 1 1

SCI 7/29/2011 5 0 7 8 2 2

Total # of Visitors

Testing Date

Group

Adults # of Adults

Child Age and Gender # of Children





SCI 7/29/2011 6 0 4, 4 2 2

SCI 7/29/2011 7 1 1 4 1 2

SCI 7/29/2011 8 1 1 8 1 2

SCI 7/29/2011 9 1 1 5 1 2

SCI 7/29/2011 10 1 1 8 7 2 3

SCI 7/29/2011 11 1 1 3, 4 2 3

SCI 7/29/2011 12 1 1 3 1 2

SCI 7/29/2011 13 0 10 9 2 2

SCI 7/29/2011 14 1 1 7, 7 2 3

SCI 7/29/2011 15 0 11, 12, 12, 13 4 4

SCI 8/4-8/5/11 1 0 10 1 1

SCI 8/4-8/5/11 2 1 1 14 1 2

SCI 8/4-8/5/11 3 1 1 7 5 2 3

SCI 8/4-8/5/11 4 1 1 7 1 2

SCI 8/4-8/5/11 5 0 10 1 1

SCI 8/4-8/5/11 6 2 2 9 1 3

SCI 8/4-8/5/11 7 0 8 6, 11 3 3

SCI 8/4-8/5/11 8 0 6 1 1

SCI 8/4-8/5/11 9 0 10 1 1

SCI 8/4-8/5/11 10 1 1 5, 2 2 3

SCI 8/4-8/5/11 11 1 1 0 1

SCI 8/4-8/5/11 12 1 1 5 5 2 3

SCI 8/4-8/5/11 13 1 1 0 1

SCI 8/4-8/5/11 14 1 1 0 1

SCI 8/4-8/5/11 15 1 1 6, 10 2 3

SCI 8/4-8/5/11 16 1 1 3 7 2 3

SCI 8/18/2011 1 2 2 0 2

SCI 8/18/2011 2 1 1 5 1 2

Testing Date

Group

Adults # of Adults

Child Age and Gender # of Children

Total # of Visitors





SCI8/18/2011

31 1 2 3 1 3

SCI 8/18/2011 4 1 1 7 1 2

SCI8/18/2011

52 2 4 6 2 4

SCI8/18/2011

61 1 6 8 2 3

SCI 8/18/2011 7 1 1 2 8 1 3

SCI 8/24/2011 1 1 1 7 5 2 3

SCI 8/24/2011 2 1 1 2, 3 2 3

SCI 8/24/2011 3 1 1 2 7 1 3

SCI 8/24/2011 4 1 1 7 4 2 3

SCI 8/24/2011 5 1 1 0 1

SCI 8/24/2011 6 2 2 0 2

24 52 11 87 83 84 167 254Total # of

Groups: 97Avg. Age of Boys:

Avg. Age of Girls:

7.4 7.5

# of Children

Total # of Visitors

Testing Date

Group

Adults # of Adults

Child Age and Gender

Totals:



Age distribution could not be calculated for the Oregon Museum Science and Industry.

Appendix B: Sample Coding Scheme


Theme Code Subcode Description

Apps used during successful interactions

Tight relationship to physical Addresses same concept as physical exhibit

For concept enrichment apps, the app addressed the same concept as the physical exhibit or supplemented the content of the exhibit.

Visitor could see connection Visitor could understand how the app related to the physical exhibit after a short explanation from the facilitator.

Easy for visitors to understand how to use

The facilitator does not have to spend a lot of time explaining to the visitor how to use the app.

Not a compelling standalone app

The apps is not so compelling that visitor wanted to engage with in instead of engaging with the physical exhibit.

Worked for multi-aged social group

Adult engaged with app Adult used the app

Adult asked question Adult asked a question of the facilitator

Adult encouraged child Adult encouraged child to engage with the app.

Multiple children engaged with app Multiple children in a group used the app.

Age-appropriate Age-appropriate The visitor was able to use the app.

Not age-appropriate The visitor was not able to use the app because it was not appropriate developmentally.