Field Study Results

Field Study Results

www.glasslabgames.org© 2015 GlassLab, Inc. All rights reserved. Last updated: 1/8/2015

Learning Impact: Students who play Words With Friends EDU can experience statistically significant gains in use of academic words within the first 6 gameplay sessions.

Key FindingsOverview

In this report, the GlassLab research team presents �ndings from the �eld study of Words With Friends EDU, a digital game developed to improve students’ understanding and use of academic words. In the process, Words With Friends EDU also assesses student strengths and weaknesses in this critical area of English-Language Arts pro�ciency.

Words With Friends EDU is an educational game created to address gaps in vocabulary knowledge associated with lower reading ability (Carlo et al., 2004), the game meets Common Core English-Language Arts standards related to vocabulary understanding and use (i.e., through turn-based interactions, immedi-ate feedback, and hints, as well as through real-time formative assessment and reporting built into the game experience, students gain agency and facility using academic terms). Words With Friends EDU accomplishes all of this while leveraging fun, accessible gameplay to motivate students to engage with challenging content over extended periods of time.

Research Question

After playing Words With Friends EDU, do students make statistically signi�cant gains in academic word use?

The Innovation

Words With Friends EDU is a browser-based educational version of the commercial mobile game Words With Friends, originally developed by Zynga.

Played with letter tiles, the two-player game is won by achieving a higher score than the opponent, in part, by using as many high-value letters as possible. As in the commercial version of the game, students use reading and writing skills - along with creativity - to form words vertically (up and down, from top to bottom) and horizontally (across, from left to right) on the game board, taking turns with another player in their class. Once the �rst word is played, following words must connect to a

word or words that have already been played on the game board. The game ends when all of the letters in a player’s tile rack are gone and there are no more letters in the tile bag.

The educational feature set supports and enables competency development by incentivizing and rewarding the play of high-value academic words, or “Power Words”. The proprietary list of Power Words was developed in partnership with WestEd, a nonpartisan research, development, and service agency with deep experience developing Common Core assessment materials. WestEd focused on high-value academic words that can help students at the target grade level succeed in school, and were chosen based on their prevalence in assessments, textbooks, and required reading.

In addition to earning bonus points for using Power Words, students also earn additional bonus points by choosing the Power Word’s correct usage from a three-choice bonus menu. Students are further connected to the meaning of the Power Words through De�nition Hints, which show students the meaning of the

highest scoring Power Word they can play on a given turn, and through increased surfacing of and access to the in-game Dictionary.


To drive engagement with the game for this young demographic, Words With Friends EDU comes with an Avatar and Badging system that rewards interaction with the educational features. The more Badges students earn, the more faces, backgrounds, and accessories they can unlock to customize their Avatar. Personalization creates attachment to the product and also motivates students to compete with classmates over who can earn and design the most creative avatar.

The game’s educator dashboard is powered by the GlassLab Analytics Engine, which enables systematic connection of in-game events to evidence of learning and e�ective visualization that is easy to use in any learning environment. It supports the alignment of learning design with system-wide data structures that enable powerful learning insights that can enhance the instructional experience (Owen et al., 2015).

Pilot Participants and Implementation

Students from across the country participated in the Words With Friends EDU �eld study, drawn from classes taught by 16 elementary and middle school educators. Zynga.org reached out to participants through the GlassLab Teacher Network, San Francisco Uni�ed School District, KIPP, WestEd, and other organizations. Over 300 students, representing a range of socioeconomic backgrounds, in grades 4 through 8 participated. A large proportion (n=270) completed an average of 6 game sessions over a period of approximately 7.5 weeks. The results of those play sessions are the focus of this report.

The Words With Friends EDU experience itself combines gameplay with whole class and small group activities. The associated instructional supports provide teachers with lesson plans and materials for classroom activities, which prepare students for a productive gameplay experience, and reinforce key concepts through supplementary activities. Teachers in the �eld study implemented Words With Friends EDU using up to 6 lesson plans aligned to Common Core State Standards, often 10-20 minutes per classroom session.

Analytical Techniques & Results

Research Question: Learning Gains

Game log data were �rst analyzed for improvements in students’ ability to use academic words. “In-game academic word use” is de�ned as the total number of academic words played within each Words With Friends EDU game. The number of academic words played in the �rst game played are counted, and then compared to the student’s individual best game, in which they played their highest number of academic words. By comparing the �rst game with the best, and �nding a consistent pattern in increased usage, we show the ways in which the game has motivated students to increase their exposure to and use of key vocabulary.

The associated �eld study presented here was conducted by a data scientist from GlassLab Games to apply core principles of Evidence-Centered Design (ECD) and best practices in educational data mining, including several iterations of exploratory and con�rmatory analysis and statistical modeling (Mislevy et al., 2014).

The di�erence between the �rst game and best game in academic word use was determined through a paired-samples t-test. Students’ use of academic words increased from an average of 0.2 words in the �rst game played to 0.87 in the best game played. Field study results show that, on average, students showed a signi�cant increase in academic word use between the �rst game played and the best game played for each student within the �rst six games played (t(270)=9.95, p<.001).


Words With Friends EDU combines high quality educational experiences, formative assessment, and game design to drive measurable learning gains and engagement in an area of English Language Arts that is known to challenge many students. In doing so, this game was also designed to meet elementary and middle school Common Core ELA standards in a short timeframe (i.e., less than 1 hour). In fact, students who play Words With Friends EDU can experience statistically significant gains in use of academic words within the first 6 gameplay sessions. Information gathered on the game’s effectiveness supports its use as a high-quality formative assessment that can aid teachers’ understanding of what they students know and can do.

Conclusion

ReferencesCarlo, M.S., August, D., McLaughlin, B. Snow, C.E., Dessler, C., Lippman, D.N., Lively, T., & White, C.E. (2004). Closing the gap: Addressing the vocabulary needs of English-language learners in bilingual and mainstream classrooms. Reading Research Quarterly, 39, 188-215.

Mislevy, R.J., Oranje, A., Bauer, M.I., Von Davier, A., Hao, J., Corrigan, S., & John, M. (2014). Psychometric considerations in game-based assessment. GlassLab Report.

Owen, V.E., Corrigan, S., Escuadra, P., Frenz, M., Hoffman, E., Riconscente, M. M., Rushton, E. & Todorovic, M. (2015, August). Towards Surfacing Salient Learning Game Data: The GlassLab Analytics Engine. Poster presented at the CRESST Conference 2015, Redondo Beach, CA.

Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer games and instruction, 55(2), 503-524.

Figure 1 shows the distribution of the frequency and evidences the positive trend. Moreover, 40% of students showed improvement in academic word use. This evidence suggests that Words With Friends EDU is an e�ective tool to deepen academic word use.

About Us

GlassLab empowers youth to claim their path to 21st century success through high-impact digital games. It is made possible through the generous support of The Bill and Melinda Gates Foundation and The John D. and Catherine T. Macarthur Foundation.

Words With Friends EDU is an educational version of Words With Friends built for the classroom. Students learn new words as they play in a safe environment with their classmates, and teachers and parents track students’ progress using custom teacher dashboards. Words With Friends EDU is available on GlasslabGames.org and at WordsWithFriendsEDU.com

Figure 1: Frequency of Academic Word Use (First Game v. Best Game)

250

200

150

100

50

00 Words 1 Word 2 Words 3 Words 4 Words 5 Words 6 Words

First Game

Best GameN

umbe

r of S

tude

nts

Documents

Field Study Results