Embed Size (px)
Citation preview
USABILITY & EFFECTIVE LEARNING EVAL 0
U sability a
DUOLINGO EVALUATION Usability & Learning Effectiveness
Lisa Marie DiFranco, Warren Watkins & Mario Torres California State University Monterey Bay
Abstract Based on the results of a usability and learning effectiveness test designed by our
research team. This report aims to prove that after interacting with Duolingo, participants will show improvement in language retention.
USABILITY & EFFECTIVE LEARNING EVAL 1
Contents I. Introduction .................................................................................................................................. 2
II. Methodology ............................................................................................................................... 2
Prototype ..................................................................................................................................... 2
Learner ........................................................................................................................................ 3
Try Out ......................................................................................................................................... 3
Expected Outcomes .................................................................................................................... 4
III. Results ....................................................................................................................................... 5
Entry Conditions .......................................................................................................................... 5
Instruction .................................................................................................................................... 5
Outcomes .................................................................................................................................... 6
Table 1. .................................................................................................................................... 7
Table 2. .................................................................................................................................... 7
Table 3. .................................................................................................................................... 8
Table 4. .................................................................................................................................... 9
Recommendations .................................................................................................................... 10
IV. Summary ................................................................................................................................. 11
V. Appendices ............................................................................................................................... 12
Appendix A: Demographic Survey ............................................................................................ 12
Appendix B: Demographic Survey Results ............................................................................... 14
Appendix C: Pretest/Posttest .................................................................................................... 17
Appendix D: Observation Checklist .......................................................................................... 20
Appendix E: Participant Survey ................................................................................................ 21
Appendix F: Participant Survey Results ................................................................................... 24
USABILITY & EFFECTIVE LEARNING EVAL 2
I. Introduction
Duolingo is a free science-based language education platform that prides itself on
personalized education by analyzing how millions of people learn. Inspired by the concept of
making learning fun, Duolingo claims to currently be one of the most effective educational
language systems (Settles, 2016). Through examining the results of a usability and learning
effectiveness test designed by our research team, this report aims to prove that after 10 randomly
selected participants complete two introduction language modules of Esperanto in Duolingo,
they will show improvement in language retention and feel more compelled/motivated to use the
software again.
II. Methodology
Prototype
The prototype created to measure the usability and learning effectiveness of Duolingo is
an informational and interactive website titled Duolingo Evaluation
(https://sites.google.com/csumb.edu/t7duolingo/home). The research team created and designed
the website with Google Sites. The purposes of this prototype is guide participants through the
Duolingo evaluation process while seamlessly collecting all their response data. Data collection
was conducted through a series of surveys and tests created in Google Forms. A second
informational page titled “Usability” (https://sites.google.com/csumb.edu/t7duolingo/usability)
was created for the 5 out 10 participants selected to be observed by the research team. The only
difference between these two informational pages is that the Usability page contains a 5th step
titled “Participant Survey”. Having the participants complete this survey will allow the research
USABILITY & EFFECTIVE LEARNING EVAL 3
team to compare the participant’s experience to their observation notes. Once a survey or test is
completed, all participant responses are saved/documented in real time through Google Forms.
Google Forms also generated most of the analytical graphs and tables provided in this report.
Learner
Duolingo’s interface was designed to be universally accessible and easy to access with
little to no explanation. Our hope was to get a wide variation of test takers who brought with
them various attitudes and experience for testing Duolingo. To achieve this, the research team
decided to administer learning effectiveness test on a group of 10 randomly selected adults. To
tests Duolingo’s accessibility and functionality, we randomly selected 5 out of the 10 users to be
observed and surveyed on usability. The research team decided to put almost no restriction or
prerequisite on our test subjects. The only mandatory requirement the test subjects had to meet
was that they had never spoke a word of the test language Esperanto. Test takers were not
required to have any prior knowledge of the Duolingo nor did we require any computer
competency skills. Selected at random, the 10-person test group ended up consisting of an equal
ratio of male to female adults. Surprisingly they all have completed some level of higher
education (see Appendix B: Demographic Survey Results).
Try Out
The Duolingo evaluation process began with a scripted introduction to insure all
participants received the same information. All 10 participants were assigned a link to the
appropriate Duolingo Evaluation webpage. In-person usability observations were conducted on 5
out of the 10 test takers. Two researchers were picked to conduct these observations. An
Observation Procedure and Checklist document was created to uniformly guide the observer’s
interaction with the participant as well as provide a space for observers to additional notes.
USABILITY & EFFECTIVE LEARNING EVAL 4
After test takers are granted access the Duolingo Evaluation webpage, their first step is to
complete a Demographics Survey. The Demographics Survey allows researcher to identify and
compare trend in the data collected from each test taker .Once completed, test takers are
prompted to return to the Evaluation webpage and proceed to Step 2: Pre-Test. Pretest and
Posttest were designed to test users on the learning content from Duolingo’s first two Esperanto
language lesson. These tests contain multiple choice questions pertaining to the vocabulary
presented in Esperanto learning module 1 and 2. Upon completing the Pretest participants are
once again asked to proceed back to the Evaluation webpage and asked to proceed to Step3:
Duolingo Lesson in Esperanto. Once here participants are given direction on how to access
Duolingo’s learning content as well as a hyperlink to Duolingo’s homepage. Participants are then
advised to complete the first two basic Esperanto lesson recommended by Duolingo. Half the
participants will complete the Duolingo lessons on their own, while the other half complete their
lesson will being observed by one of the researchers. After lesson completion, participants are
prompted to return to the Evaluation webpage to complete Step 4: Post-test. Participants take the
same test administered prior to the completion of their language lesson. The results from posttest
will then be compared to the results of pretest. These results should tell the research team
whether or not learning actually took place after the completion of the first two Esperanto
modules in Duolingo. At this point, the Evaluation study has ended for half the test group. The
other five participants are asked to complete Step 5: Participant Survey. Upon completion, test
takers are thanked for their time and the Duolingo Evaluation Test is complete.
Expected Outcomes
Upon completing the two lessons or spending around 20 minutes in Duolingo, test takers
will be expected to understand how to speak or translate basic gender word such as Girl, Boy,
Man, & Women in Esperanto. Measuring learning effectiveness will require researchers to
USABILITY & EFFECTIVE LEARNING EVAL 5
compare the means of the pre and posttest scores. Participants’ scores from the pretest and
posttest will show a statistically significant improvement in language retention.
Based on Duolingo’s testimonials the research team hypothesis that a majority of our
participants well have a positive experience while using Duolingo. Measuring usability will
require the research team to examine the result of our Observation checklist as well as the
Participant Surveys.
III. Results
Entry Conditions
There was little difference between the intended and observed conditions of participants
in the tryout. Participants were all adults with between the ages of 26 and 51 and have graduated
from high school or have earned the equivalent to one. Only one of the participants had used
Duolingo in the past and no participants had used another app to learn a language. No
participants had fluency in the language they would learn in our modules, Esperanto.
Instruction
There were some gaps between intended and observed instruction. The first issue was
with the steps our team created on how to use Duolingo. The tryout participants encountered
prompts by Duolingo that we didn’t address in our instructions. Before starting the first module
participants were asked to select a daily goal. After the completion of the first modules
participants were asked to register to save their progress. Our instructions should have told
participants that they would encounter these screens and described how to respond.
Another issue with our team’s instruction occurred during the pretest. Some participants
noted that the some questions on the pretest clued or helped to give the answer to another
question on the pretest. The difference between the pretest and posttest may be show less
USABILITY & EFFECTIVE LEARNING EVAL 6
learning than what actually occurred because some participants got a higher score on the pretest
because they are savvy test takers.
Some participants got stuck on portions of Duolingo. In some tasks, the user hears a word
or phrase in English and are asked to translate the word into the language they are studying. The
user sees an empty box and below the box are letters unique to the language they are studying.
Users thought they had to respond with the letter icons on the screen but not all letters needed for
the word were there. Eventually participants tried to type in the response and that is what they
were expected to do. Another issue users noted with Duolingo was that prompts on how to
respond were not given if you entered the module a second time. Participants who took the
modules shortly after another participant ran into this issue. This could have been avoided if the
evaluator had closed the web browser and opened a new one for the next participant.
Outcomes
Ten participants completed the two modules and the pre and posttest. The pretest and
posttest contained 13 items. These tests were given and graded through Google forms. The
responses were exported to excel. The pretest mean score was 7.5 or 57.6% and the posttest
mean was 11.2 or 86.1% (Tables 1, 2, and 3). Results were analyzed through a paired sample t-
test. The absolute value of the t-stat (4.384) was greater than the critical value (1.83) see Table 4.
Analysis of the pre and posttest show that learners made significant games as a result of
completing two modules of Esperanto.
USABILITY & EFFECTIVE LEARNING EVAL 7
Table 1.
Pre-Test Scores Post-Test Scores
9 11
10 12
8 13
9 11
4 11
10 9
4 11
7 9
8 13
6 12
Table 2.
USABILITY & EFFECTIVE LEARNING EVAL 8
Table 3.
USABILITY & EFFECTIVE LEARNING EVAL 9
Table 4.
Overall the participants had a positive reaction to using Duolingo. On a scale of 1 to 5,
with 5 being very easy the average rating was among participants was a 4.6 when asked about
the ease beginning a lesson in Duolingo. Using the same scale the average rating on the tasks
they were asked to complete in Duolingo was 4.2. 80% of participants felt hardly any frustration
in using Duolingo and noted that the modules increased their knowledge of Esperanto
significantly or very significantly. The majority of the participants will not continue to their
study of Esperanto but they would recommend Duolingo to a friend wishing to study a foreign
language. The most common feedback regarding what the participants liked about Duolingo was
that the lessons were short and that they provided instant feedback. Negative feedback on
Duolingo was related to instructions and inconsistency. For example, some tasks gave explicit
instructions to the users while others did not and words were not always pronounced by the
USABILITY & EFFECTIVE LEARNING EVAL 10
program (see Appendix F: Participant Survey Results).
Recommendations
Based on the findings, the following recommendations are provided:
1. Instructions should be explicit: Most tasks the participants completed in the modules
were simple to figure out but others were not. Instructions for all tasks should be stated so
the user knows how they are expected to perform. This occurred mostly when
participants were asked to write a translation of a word in English to Esperanto. The user
saw an empty box with icons of letters specific to Esperanto below. The word
participants needed to spell did not require those letter so they were unsure how to
respond. Eventually the users tried to type in the response using the keyboard, which was
how they were required to respond. It should not be assumed that a user will see a blank
box and know how to respond. Duolingo allows users to hover over words for
translations a similar feature could be added to guide the user to what they are supposed
to do for specific. It is possible this exists but none of the participants in this survey found
this or used this feature.
2. Modules should be consistent: Duolingo allows users to retake modules to strengthen
their language skills. However when a module is entered for a second time instructions
are no longer provided for some of the tasks. While this issue arose as result of an error
made by an evaluator, it could be frustrating for a user and my prompt some to not
continue to study a foreign language through Duolingo. Another inconsistency was in
regard to when the program spoke to the user. The words in Esperanto were not always
spoken aloud. This may be a common tactic when teaching someone a foreign language,
but the writers of this report are not foreign language educators.
USABILITY & EFFECTIVE LEARNING EVAL 11
IV. Summary
Analysis of the data showed that people learned through Duolingo and they enjoyed
using the program. Duolingo could be made easier for user by providing more instructions and
being more consistent in the task such as allowing the users to hear the word in the language they
are speaking in all tasks. However, based on the evaluation Duolingo proved to keep users
motivated and increased their willingness to learn a foreign language.
This study has provided valuable insight to the writers of this report. All steps in the
process should be completed prior to the actual testing. Typos and other errors of that nature
should not have been seen by the participants. Steps that confused the participants, such as what
to select for a daily goal, could have been avoided by going through the test as participant or
having someone complete the tasks before starting the formal evaluation.
Overall the writers of this report believe that this exercise was a successful and thorough
evaluation of the Duolingo application, which provided valuable insight into its effectiveness.
USABILITY & EFFECTIVE LEARNING EVAL 12
V. Appendices
Appendix A: Demographic Survey
Duolingo Evaluation Demographic SurveyDemographic survey for participants of the Duolingo evaluation
* Required
1. Age? *
Mark only one oval.
18 25
26 40
41 55
56 65
2. Gender *
Mark only one oval.
Female
Male
3. Highest Level of Education Attained *
Mark only one oval.
High School Graduate/GED or equivalent
Some College
Associate's Degree
Bachelor’s Degree
Master’s Degree Completion
Doctoral Degree
4. How would you describe your experience with languages other than English? *
Mark only one oval.
I am fluent in another language(s).
I am somewhat fluent in another language(s).
I have little experience with another language.
I have no experience with another language.
5. Have you used Duolingo before? *
Mark only one oval.
Yes
No
USABILITY & EFFECTIVE LEARNING EVAL 13
Powered by
6. Have you used another web based program to learn a language? *
Mark only one oval.
Yes
No
7. If you choose "Yes", please provide the nameof the program:
USABILITY & EFFECTIVE LEARNING EVAL 14
Appendix B: Demographic Survey Results
USABILITY & EFFECTIVE LEARNING EVAL 15
USABILITY & EFFECTIVE LEARNING EVAL 16
USABILITY & EFFECTIVE LEARNING EVAL 17
Appendix C: Pretest/Posttest
Duolingo Evaluation PreTestPreTest for participants in the Duolingo Evaluation. This is taken to test the participants knowledge of Esperanto.
* Required
1. What is the correct translation of "man" in Esperanto? *
Mark only one oval.
Viro
Virino
Knabino
2. How do you say "no" in Esperanto? *
Mark only one oval.
Na
Ne
No
3. What is the translation of "Mi estas knabo"? *
Mark only one oval.
I am a boy.
We are boys.
I see a boy.
They are boys.
4. How do you say "you" in Esperanto? *
Mark only one oval.
Mi
Tu
Vi
5. How do you say "the man" in Esperanto? *
Mark only one oval.
Lo vira
La viri
La viro
USABILITY & EFFECTIVE LEARNING EVAL 18
USABILITY & EFFECTIVE LEARNING EVAL 19
USABILITY & EFFECTIVE LEARNING EVAL 20
Appendix D: Observation Checklist
USABILITY & EFFECTIVE LEARNING EVAL 21
Appendix E: Participant Survey
Duolingo Evaluation Participants SurveySurvey for participants in the Duolingo Evaluation. This should be completed after taking the PostTest.
* Required
1. How would you describe the clarity of the steps to start a lesson in Duolingo? *
Mark only one oval.
Much Worse than Expected
Worse than Expected
As Expected
Better than Expected
Much Better than Expected
2. How would you rate the ease of use of the Duolingo application? *
Mark only one oval.
1 2 3 4 5
Extremely Difficult Very Easy
3. What level of frustration did you experience while using Duolingo? *
Mark only one oval.
Lots
Some
Undecided
Hardly Any Frustration
Absolutely no frustration
4. Did completing a Duolingo lesson motivate you to continue? *
Mark only one oval.
Yes
No
5. How would you rate the difficulty of the tasks required in Lesson 1? *
Mark only one oval.
1 2 3 4 5
Extremely Difficult Very Easy
USABILITY & EFFECTIVE LEARNING EVAL 22
USABILITY & EFFECTIVE LEARNING EVAL 23
USABILITY & EFFECTIVE LEARNING EVAL 24
Appendix F: Participant Survey Results
USABILITY & EFFECTIVE LEARNING EVAL 25
USABILITY & EFFECTIVE LEARNING EVAL 26
USABILITY & EFFECTIVE LEARNING EVAL 27
USABILITY & EFFECTIVE LEARNING EVAL 28
USABILITY & EFFECTIVE LEARNING EVAL 29
USABILITY & EFFECTIVE LEARNING EVAL 30
