Round 2 UX Research Cycle

I.Design: Making ID Real A User Experience Design Challenge

EDIT 752

Spring 2018 Dr. Bannan

Jordana Anderson • Youliduzi Niyazi • Mary-Leigh Phillips • Brett Sparrgrove • Sonya

Wein

Background 2

Goals 2

Updates in Prototype for Round Two Testing 3

Methodology 5 Research Questions 5 Participants 5 Think Aloud Approach and UX Inspection 6 Brief Questionnaire 7

Round Two Data Collection 7 Think-Aloud Results 7 Questionnaire Results 8

Analysis 10

Conclusions 11

Appendix I: Round Two Questionnaire Detailed Responses 13 Participant 1. 13 Participant 2. 15 Participant 3. 17

Appendix II: Round Two Testing User Instructions 19 I.Design: Making ID Real 19 Recording 20 Informed Consent 20 Testing Process 20 Informed Consent Form 21 Think Aloud Overview 23 Series of Tasks 24

Appendix III: Round Two Think-Aloud Data 25

1

Background The group designed a technology resource that enhances the learning experience of instructional design and technology (IDT) students who are pursuing degrees and certificates at George Mason University (GMU).

Specifically, the technology resource curates concise, accurate, and accepted IDT information. It also provides practice scenarios that 1) show how IDT principles are applied in real-life design projects and 2) poses design problems for users to reflect on/solve.

Goals The goal of the research and testing plan is to determine how best to refine, enhance, and re-design the I.Design application. This is the second round of testing the app. Each round will include task-based and user experience inspection methods. The prototype has been redesigned to incorporate key user feedback (see Table 1. Updates in Prototype Round 2). The research and testing process will answer the following overarching questions:

● Are efficiency and flexibility balanced within the application workflows? ● What is the emotional impact of the application? ● Is the design intuitive and consistent? What is the impact of the design on the

user’s memory load?

Our objective is to improve the application in each of these areas. The main difference in this round from round two is the new users’ background experience. These participants are not currently enrolled in the George Mason Instructional Design program and two did not have any UX or Instructional Design expertise. The following section outlines additional research questions falling under the overarching themes and objective.

2

Updates in Prototype for Round Two Testing After Round 1 of Testing there were several upgrades, developments and improvements that took place due to user feedback. The table below highlights those updates:

Table 1. Updates in Prototype for Round Two

Category Round 1 Testing Feedback Updates to Prototype Screenshot

Login Page & Tutorial

1. Be consistent with the placement of the tutorial boxes

Boxes are more “static” now and less distracting as user moves through them

2. Clicking create account should take users directly to create account page

This functionality has been added

3. Be concise and ensure all typos and spelling is correct

Fix language issues for clarity

4. Allow user to login (or create account) directly from home page

Home page adjusted to include login fields and create account button

3

5. Remove tutorial assistance that is too simple and basic (like “click the house to go home”)

Tutorial has been simplified to exclude the obvious

Settings 6. Need a place to collect user feedback and suggestions

Added “Suggestion box”

Practice page

7. Bookmarks icon functionality confusing

Reworked bookmarks functionality and removed bookmarks button.

8. Too many navigation buttons add confusion and can overwhelm

Removed bookmarks button and additional resources button. These functions have been better integrated into i.design in more logical places.

9. Too many navigation buttons Moved favorites, which will be less utilized than the other buttons, to the lower corner of the screen

4

Methodology Research Questions Building from our overarching goals for the test and evaluation phase, additional research questions can be grouped into the following categories, 1) task-based and 2) user experience inspection. These questions are listed below.

Task- Based

● Do the users understand all that is available to them in the mobile application from the first page?

● Where might there be redundancy (e.g. unnecessary clicking, superfluous content)? Where do we need to build in more support?

● Does the user understand the program’s feedback and understand what to do with mentor interaction in the scenarios or how to favorite a page?

UX Inspection

● Are the favorites features (i.e. playlist) organized efficiently for effective UX?

● Does a user find making an account helpful in order to use the app as it was intended to be used?

● Can the user easily navigate the scenarios and find all its features useful?

Our task-based instructions for the think-aloud data collection were derived from these questions.

Participants Due to low rates of detection of design flaw rates by individual inspectors, we selected 3 individuals. We did not want to select more than 5 participants as the standard operation model in the UX field shows that more than 5 dual experts will produce diminishing returns. Participants in round 2 were not involved in the George Mason Instructional Design program although one was an expert in UX design and another was a practicing instructional designer. One participant lived in Lebanon. The participants not familiar with UX or Instructional Design held careers that worked closely with people and intra-personal communication such as clergy and international relations. This allowed us to see how non-students and non-designers viewed the usefulness and simplicity of the app’s features.

5

Think Aloud Approach and UX Inspection The participants inspected the application using heuristic evaluation and corresponding forms to complete key tasks outlined in Appendix II. The inspection also included a think-aloud component and captured participants’ performance using Zoom Meeting (www.zoom.us), a web application that allows for screen capture. Appendix III Thematic Map Data includes a table of the participants’ qualitative think-aloud feedback as well as their heuristic rating forms including, tone of voice, potential frustrations, and emotional impact. The Think-Aloud participant tasks (see Appendix II) led each participant through the working features of the prototype version two looking at the following features:

● Home Screen: ○ Tutorial ○ Login/account ○ Search

● Glossary: ○ Glossary video ○ Favorite feature

● Practice: ○ Filter for scenarios ○ Selecting a Scenario ○ Scenario Navigation and Icons

■ Scaffolding ■ Mentor Feature

Due to the updates to the prototype, users did not need to test the bookmark feature. Instead they had the opportunity to react to a pop-up menu mentioning its use (see Table 1. Updates to Prototype for Round 2). Throughout testing, the participants did not comment on every feature listed above. The lack of comments seemed to indicate that the feature was clear, there was ease of use, and the feature met user expectation. The following are features that fall into this category:

● Playing the glossary video ● Selecting a scenario

6

Brief Questionnaire Following the heuristic inspection sessions, the team sent out a brief questionnaire to the participants to capture more data on the application’s emotional impact, expectations and further suggestions.

Round Two Data Collection Think-Aloud Results After watching the recordings of the Think-Aloud and screen-share recordings we coded the qualitative data based on recurring themes identified in round 1 and round 2. The themes that followed from this coding analysis include:

● Cognitive Overload - Although one of the main goals of our application was to prevent cognitive overload that many students experience when reading dense academic research, the format of our prototype and the fact that it was not fully built out, caused its own type of overload. Some participants became overly preoccupied by certain icons looking like they should be links, or the tutorial being too long. In further testing, we will revise the tutorial and move to an application development environment with fully functional links and additional content.

● Clarity of the Objectives - not having a full scenario with content linked to ID 101 and Glossary in an application format made the objectives unclear for one users in this round. Despite the limitations of the tool, much of the feedback from the other users was that the layout was intuitive, clear, and helpful. Further testing in an application environment is necessary to determine any areas which need additional clarity.

● Aesthetically Pleasing Content Formats - In both round 1 and round 2 testing phases, our users found the colors and layout aesthetically pleasing. The consistency in color and iconography throughout the application was appreciated. The use of multimedia and video received a great deal of praise. We will not change the color and look and feel when developing in a mobile application environment, but rather focus on small specific changes to details in the application mentioned by the testers.

7

Questionnaire Results Following the Think-Aloud, participants further evaluated their experience by answering ten questions. The heuristic evaluation forms were aggregated below: Table 2.

Question Participant 1 Participant 2 Participant 3

1. Ease of Navigation

3 4 There were some elements that weren't easily found, but otherwise the amount of instruction and steps were helpful

4 Clear how to go through the app, but some icons are confusing (i.e. scaffolding, stop and think). Also the opening tutorial blocked the words that it was describing

2. Usefulness 3 Nice concept, but the navigation, layout, and labeling could get a bit in the way of its usefulness.

4 I'm not an instructional designer, but I'm sure it's useful for designers once all the elements of the app are functional

5 I think the combination of video, text, connecting to others, and design principles laid out in a logical way would be very helpful to a student.

3. Overall Aesthetic 4 4 Potentially fix where the cursors allows for clicking. Sometimes it was difficult to tell where told [sic] hold the cursor to be able to click on certain elements.

5 Very colorful, friendly, not too cluttered, and kind of fun

4. Amount of Multimedia

Yes Yes I think once all of the videos for the app are fully functional it will be enough.

Yes - Those things catch the attention of learners but it was not overwhelming.

5. Interactivity Depends. There was something called "feedback" which made me think that this was interactive with a live mentor. If so, that's great. If not, then it has some good videos, but not much interactivity.

Yes, but again not everything was fully functional when I tested the app.

Yes - I could choose where to go and what to learn about. It seemed to have lots of cool, useful features that helped with interaction (i.e. favorites)

8

6. What do you Like?

Looks like it's a good digital manual of instructional design.

I liked the pop up walk through in the beginning, I liked the aesthetics of the app

The simplicity, usefulness, color, friendliness, fun, visually appealing, and user choice.

7. What was confusing or difficult?

Navigation at times. The log in function was bit confusing, and some of the testing instructions could've been a little clearer.

The opening shpiel blocked the words. I think most people access apps like this on their phone in which case the videos might be small, unless there's a way to go full screen. Some of the icons were confusing, as I mentioned.

8. Is it helpful for school/work?

I don't do instructional design for a living. If I did, perhaps I would reference this if I was a novice. This feels a bit like an online course, without the assignments (which would make it most useful).

I think it could be a quick go to resource for instructional designers.

Because you find answers to your questions specifically and it's tailored for individual learning.

9. Suggestions See recording. Change the cursors, make the create account button a little larger, put volume controls closer to videos

See above. Cite sources for videos and include info about where it comes from.

10. How does it compare to other resources?

Like Wikipedia with videos.

This is my first time testing an instructional design app, but I think it was good.

Much more fun and interactive than a textbook, for example.

The survey results above were consistent with the feedback in the video. Overall these comments seemed positive especially with respect to the overall aesthetic. It was clear that after round 1 and round 2 initial testing the application was ready to be further built out. Participants in this round noted that some features were difficult to test because of the lack of buildout and the application being built in Storyline instead of an ios or android environment. Many of the suggestions were features that I.Design will eventually include but due to engineering resources and time constraints and the

9

balance between investment in build-out and prototype testing our team was unable to include them at this stage.

Analysis In addition to the major themes coming out of our recorded task-based think aloud sessions, some detailed suggestions were made. First, there were a few comments regarding the tutorial. The tutorial is meant to use signaling to direct the user’s attention to the icons and their corresponding uses, however it was also blocking the words and icons. This led to a universal frustration that the text and images were not contiguous. This can be remedied in a full application development. We were constrained with the Storyline software. Second, the tasks our protocol asked users to complete confused participants at times. The task instructions were more confusion than the actual application. Some of the participants did not follow the tasks in order and found it challenging to complete the action item. They may have had a challenging time due to the wording on the task list more than navigating through the app. One participant even mentioned that the tasks were unclear. This should not be a reflection on the app itself. In future rounds, with a further build out in terms of instructional design content, the team can use a free-form think aloud to understand how users would organically get to know the application. This will also help us determine areas of the application which are underutilized and may not be necessary. The tutorial needs further testing and development. It caused confusion with users and could be broken down into much smaller chunks. It was necessary given the level of content we were able to engineer into the prototype at this stage as well as the Storyline software used to build the prototype. Once the application moves to an ios or android development environment and includes more content, help or initial use pop-ups with instructions can be designed to be more streamlined. Eventually a tutorial could become obsolete and replaced with a clear description in the application store. Lastly, although the overall aesthetics were praised by users, the iconography for scaffolding was confusing. Both the term and the icon for scaffolding could have been made more clear. Further testing may allow for a better representation for this ID term. With more time, it could be tested using card sorting and we could consider replacing it with a different icon. Extensive research of application iconography could help us find a better term to use along with a more recognizable icon.

10

Conclusions Overall, round two testing successfully highlighted functions of the application that met user expectation as well as areas that needed to be altered. Our research sought to answer several questions. Our first question was, “Are efficiency and flexibility balanced within the application workflows?” Overall, our users found the application efficient although we did get some comments that the application seemed more like an e-learning module than a quick-reference, bite-sized learning application. This is due to the limitation of our development software, Storyline and incompletely built out content. The testers, especially those in an IDT graduate program currently, appreciated the multimedia nature and design of resources to help them learn. Many of them praised the design and stated that they would absolutely use it as a supplement for their studies. Our second research question was, “what is the emotional impact of the application?” Our analysis showed that users generally were pleased and in many cases impressed with the design prototype’s layout and strategy for making ID content more digestible. Most of the frustration seemed to pertain more to the testing instructions and limitations within Storyline than with the application design. Our third research question was, “is the design intuitive and consistent?” With the exception of some specific icons and links the overall feedback showed that the design was intuitive and very consistent. There are expectations that users have for mobile applications such as hidden passwords when one writes in his or her password or a double table feature for a favorite; we chose not to focus on these because we focused on the area of our application that was unique to our project which was the practice scenarios feature. The last question we hoped to answer was, “what is the impact of the design on the user’s memory load?” This area needs further testing once the application has been populated with more practice scenario content and is in a mobile application environment and either the task based instructions are revised or a free-form think aloud is performed. We were challenged to balance the results from participants with little or no background in instructional design and those who are well versed even as new students. We concluded that our app is mired in a context in that it is designed for ID students. Our original intent was to give a first-year student who was working on their homework a supplement to their studies and we confirmed this was the appropriate audience. That

11

said, some specific attention to various terms and icons, e.g. scaffolding, will be addressed in additional iterations of the application. We were successful in meeting the users’ expectation of connectivity and sharing information through this app, in fact, this goal was not an original research question of ours but it turned out to be an important feature that we will certainly prioritize in future build-outs. Some of the comments were features that we couldn’t realize using Storyline tool for prototyping. If we do develop it and put it in the app store there will need to be a description that will establish the users’ expectation. For people who have used similar tools before, they have expectations for common iconography and vernacular across applications. While the questionnaire provided rich description, actually observing users use the application via the recordings gave us thick, rich, data and cognitive insights we wouldn’t have realized given our developer bias and own frequent use of the prototype. Ultimately, the iterative process made our product better.

12

Appendix I: Round Two Questionnaire Detailed Responses Participant 1.

13

14

Participant 2.

15

16

Participant 3.

17

18

Appendix II: Round Two Testing User Instructions  

 

I.Design: Making ID Real Thank you for helping us test our prototype! We are very interested in your feedback. We have created this information packet to clarify the testing procedures for you. Please review it before we meet, and do not hesitate to contact us if you have any questions!  

Jordana Anderson • Youliduzi Niyazi • Mary-Leigh Phillips • Brett Sparrgrove • Sonya Wein  

 

19

Recording You can use whatever recording software you have available to capture and send us the test. If you would like one of our designers to manage the recording, he/she will use Zoom. Zoom is a video conferencing platform that allows participants to see and hear each other and share what’s on their computer screens. In order to use Zoom, you will need to sign up for a free account at https://zoom.us/.  Informed Consent Because we will be recording the testing session, we request your permission. Please review and sign the informed consent form on page 3.  Testing Process 

1. If you chose to use Zoom, first, you will receive an email with a Zoom link, which will take you to the virtual space wherein you will do the pilot testing. One of our designers will email you this link once you agree on a day and time for the test. 

2. Once in the virtual space, the designer will greet you and begin recording the session.  

3. The pilot testing involves exploring our app by trying to accomplish a series of tasks. As you move through the app, we would like you to vocalize your thoughts. This is called a think aloud. The designer will run through both the tasks and the think aloud protocol with you before beginning the pilot testing. You can also review them on pages 5 and 6 of this packet. 

4. The designer will provide you with a link to the I. Design prototype. 

5. In Zoom, you will share your screen with the designer so he or she can see what you see. Then, you will explore the app and think aloud. 

6. At the conclusion of the pilot testing, the designer will thank you for your participation and provide you with a follow-up questionnaire link. Please complete the questionnaire as soon as possible following the pilot testing. 

20

 

Informed Consent Form I.Design Learning Application

RESEARCH PROCEDURES This research is being conducted to test to usability, functionality, and effectiveness of the I.Design Mobile Learning Application prototype. If you agree to participate, you will be asked to test the application by completing specific tasks, exploring the application on your own, and providing feedback. RISKS There are no foreseeable risks for participating in this research. Participants are requested to wear glasses or contact lenses as they ordinarily would while using a laptop or tablet to prevent eye strain. BENEFITS There are no benefits to you as a participant other than to further research into the usefulness of a learning application for instructional designers. CONFIDENTIALITY The data in this study will be confidential. Names and other identifiers will not be placed on surveys or other research data. This study involves the audio or video recording of your interview with George Mason students. Neither your name nor any other identifying information will be associated with the audio or video transcript. Only the George Mason university students in 752-002 will be able to listen (view) to the video recordings. Transcripts of your interview may be reproduced in whole or in part for use in presentations or written products that result from this study. PARTICIPATION Your participation is voluntary, and you may withdraw from the study at any time and for any reason. If you decide not to participate or if you withdraw from the study, there is no

21

penalty or loss of benefits to which you are otherwise entitled. There are no costs to you or any other party. CONTACT This research is being conducted by as a part of a course on research and design methods, EDIT 752-002 at George Mason University. The research team may be reached by contacting Mary-Leigh Phillips by email at maryleighp@gmail.com for questions or to report a research-related problem. This research is being conducted under the guidance of Dr. Brenda Bannan, bbannan@gmu.edu. You may contact the George Mason University Office of Research Integrity & Assurance at 703-993-4121 if you have questions or comments regarding your rights as a participant in the research. CONSENT I have read this form, all my questions have been answered by the research staff, and I agree to participate in this study. _______ I agree to audio (video-) recording. _______ I do not agree to audio (video-) taping.

__________________________

Name

__________________________

Signature

__________________________

Date 

22

Think Aloud Overview 

 

Thank you for participating in this test! While you use I.Design, we would like you to vocalize your thoughts so that we can better understand your experience. This is called a think aloud. The following video demonstrates the think aloud process: https://www.youtube.com/watch?v=g34tOmyKaMM.  

WHAT DO YOU THINK ALOUD ABOUT? 

Think aloud as you’re wondering or trying figure something out, for instance. Talk about what catches your eye or confuses you. It is important for you to say what you’re thinking not describe what you’re doing.  

WHAT IF I DON’T KNOW WHAT TO SAY? 

We might prompt you to think aloud by asking, “What are you thinking?”, “What do you think you should do next?”, or “What do you/did you expect will/would happen?” 

SHOULD I FILTER MYSELF? 

Be honest in your thoughts; we’re not judging you, and you won’t hurt our feelings! 

   

23

Series of Tasks 

Please complete the following tasks as you move through the app. Remember to think aloud as you navigate. You can also leave written comments on the right side of the screen. 

1. Navigate the pop-up tutorial. 

2. Login/Create an account (You don’t actually have to enter any info. Just click 

create an account.) 

3. Return to the home screen. 

4. Access the glossary and perform a search for the term “Multimedia Principle”. 

a. Play the video. 

b. Favorite the page. 

5. Return to the home screen. 

6. Go to practice. 

a. Explore the variety of filter options (ADDIE Process & Skill level). 

b. Expand the details of the scenario ‘campus safety’. 

i. Select this scenario to experience 

7. Located the ADDIE menu. Under the ‘analysis’ part of the ADDIE process, select 

‘interviewing audience’. 

a. Use scaffolding icon to support your learning. 

i. Exit out of the scaffolding assistance. 

b. Select the stop and think icon.  

i. View the mentor’s feedback. 

ii. Exit stop and think. 

c. Navigate to the next screen “Identifying Learner Traits” 

d. Return to the home screen. 

24

Appendix III: Round Two Think-Aloud Data

Issues Tutorial

cognitive dissonance, Contiguity principle

Tried clicking the icons in each pop-up ("Those don't do anything") "Words in blue seem like links but don't do anything" "the icons are not directly associated with their titles in the tutorial" ""[Would] Be nice to have the logos next to the text" "tries but recognizes not working buttons" "tutorial is helpful but it would be less confusing if the tutorial was not covering the features it described"

cognitive overload

"I kind of just ignored most of what I just saw." "gets to the last popup and reads to self as if skimming" "It seems like by the time the user gets to the "i" page they do not end up reading it" "Practice, glossary instructions seem obvious and unnecessary" "also skims last box"

having a tutorial is helpful and exciting

Tutorial brought awareness about the features of the app like favorites and connect. "Favorites seems like a nice feature" "Connect is a good feature - it meets expectation that technology is about connectivity" "I like this description of what the different buttons are doing." "favorites and shareable "playlists" cool- ooh this tool will allow for networking and collaboration" "connect- cool!" "everything is clear so far"

spelling error "Spelling Error: Practice contains a number "of" real world instead of "or"' "practice box has a typo"

Order of use

"user assumed that the tutorial came after creating an account" "Login seems pretty intuitive at first but then confusing .Make create account button bigger or switch so that create account comes before login"

Delighted that you can personalize

"seems interested in personalizing with a photo" "seems excited to create a personal account"

need additional info suggests the need for a FAQ for the app other than tutorial

needs security feature- not yet built out Password - expectation is to be hidden for security but it can be seen

order of use Expectation is that user would create an account upon initial use and then login for other use when working with a mobile app.

intuitive create an account was intuitive

Favorite Feature/Page

misleading "Looks like it's already favorited."

tag feature unclear Is the heart the favorite? Or is it a like? What's the button underneath? "Maybe instead of saying 'saved' you say 'favorited'"

intuitive He observes favorites on his own thinks it is good

delight create a playlist is very nice

25

Practice

misleading: coherency principle

"What to do with the underlying. Does it mean you could do things on the keyboard?"

need multiple ways to get to it

"Clicked campus safety; nothing happened (had to click the dropdown arrow)"

unintuitive

"I'm not sure what I'm looking at here" "Tried to move ahead in tasks but still stuck on filter page" "the page does not seem like a search at first, but he figures it out soon after" "it seems there are too many choices to filter the scenario" "Complexity and Skill Level sounds similar and unclear, perhaps rephrase"

navigation unclear, need a back button

"I don't see a way to get back to the overview" "How do you get back to the questions?" "ADDIE is unfamiliar, perhaps you could have a hover definition"

coherency principle

"This looks like it's going to take me away from here because of how it's positioned on the page. Makes me think it will take me outside of the 'A' [in ADDIE]"

unclear: suggests feature follow video

Couldn't identify it at first. "You can make it so that it says stop and think right after the videos over"

Unclear of objective

"Why do i have all of these tabs under A? do they have to complete a task? is it a case study challenge to complete? is this a learning tool?" "it seems that there needs to be a task in order to learn something. what are they supposed to do with the scenario" "what do you want the user to do with these questions? will they using it alone or with a group?"

SITE YOUR SOURCES!!!! FOR CREDIBLE RESOURCES

"likes to have videos but it would be helpful to have the videos cited to confirm that it is a credible source with a short information"

clear, completed task successfully and intuitively

"Finds filter dropdown menu" "navigates to scenario intuitively using Go button" "found the back button to the practice icon but prefers a clearer option" "intuitively explored the practice features, stop and think, mentor, bookmark and return to home screen without being prompted" "navigates through A menu easily" "recognizes the universal icon to expand the information" "Navigates throught screens using arrows easily to search for desired page" " sees information and is able to identify that there is more to read and uses the scroll bar intuitively"

"Oh, that's nice."

Spelling error "Beginner" in filter options is a spelling error

helpful and pleasant design "ADDIE acronym is defined in the filter menu which is helpful" "the icon interactivity (hovering feature) is eye friendly"

delighted to have multimedia (videos)

"expresses excitement to watch the videos" "Is cool- recognizes that is a pre-recorded video to watch" "questions for reflection-that's good" "The bookmark feature is very good- a feature that he did not realize he needed" "like the bookmarking functionality" "filter is explored easily" "it is clear that

26

there are case studies listed" ""Brilliant idea to filter" "it's not too cluttered"

unclear: design issue

"instead of using back arrow, he uses the A dropdown. this may be due to the faded grey color instead of a hover color for the back button" "to expand scenario to see details about it is not intuitive"

scaffolding

"the term scaffolding is an unclear term for those outside of education or new to education" "He recognizes the scaffolding icon but believes it not to be intuitive or clear as it stands alone. It was chosen as in a process of elimination"

suggestion to make it clearer

"hand indicates a high-five or a stop. Suggests a stop sign or a person with a thinking with a thought bubble. it is not as clear as is" "finding the stop and think button was frustrating" "stop and think button is unclear because it was not identified, it would be helpful to have a hover text to identify it"

27