HCDE 517 Winter 2014 | Usability Study Test Kit Caroline ...michaelgelon.com/wp-content/uploads/2015/01/The... · Please note we are only looking for HR professionals & recruiters

Refrek.comUsability Study Test Kit

HCDE 517 Winter 2014 | Caroline Colón, Michael Gelon, Bill Grimmer, Irina Litvin

Includes:

Screening Questionnaire

Participant Confirmation Email& Directions

Consent Form & CompensationReceipt Form

Logistics Checklist & Schedule

Task Flow Overview

Participant Task Handouts

Moderator Script

- Pre-Test Questions

- Post-Task Questions

- Post-Test Questions

p. 2 - 3

p. 4 - 8

p. 9 - 10

p. 11

p. 12

p. 25 -35

p. 13 - 24

Participant ScreenerThis is the questionnaire used to select participants for this study. It is administered through Google forms:https://docs.google.com/forms/d/13QsrYvsTZZ3B2nsL1tJnzyAR7mZqz_ZTfUDvDiVPEwI/viewform

UW HR Usability Study QuestionnaireWe are a group of University of Washington graduate students working on a usability study with a motivatedclient. We are looking for HR and recruiting professionals to participate in a 45 minute usability study oncampus.

* Required

Contact InfoWe will use this info exclusively for contacting you about further participation.

First Name *__________________

Last Name *__________________

Email *__________________

Phone *__________________(xxx)xxxxxxx

What is your gender? * Male Female Prefer not to say

What is your age? * [drop down menu] < 24 25 34 35 44 45 54 55 64 65 75 > 75

What is your current job title? *__________________

(continues on next page )

HCDE 517 Winter 2014 | C. Colón, M. Gelon, B. Grimmer, I. Litvin_________________________________________________________________________________________________________2 of 35

https://docs.google.com/forms/d/13QsrYvsTZZ3B2nsL1tJnzyAR7mZqz_ZTfUDvDiVPEwI/viewform

How long have you been working as a Human Resources professional or recruiter? *Please note we are only looking for HR professionals & recruiters for this study.

Under 1 year 1 3 years 3 5 years 5 10 years 10+ years

With respect to hiring technologies used by your company (i.e. job posting websites, applicanttracking systems) which applies to you? *

I am responsible for the selection / recommendation of hiring technologies. I am responsible for the approval of hiring technologies. I am not involved in hiring technology decisions for my company.

Which webbased communication applications have you used for hiring processes? *Select all that apply.

Email Skype Google Hangout WebEx Lync None of the above Other:

How comfortable are you using the internet? *Select a value from a range of 1 (Not very comfortable) to 5 (Very comfortable)

Not very comfortable 1 2 3 4 5 Very Comfortable

Please select your preferred session date and time: *This is for a 45 minute session on the University of Washington Campus in Seattle.

Thursday, February 20th, 2pm 6pm Sunday, February 23rd, 10am 2pm I am available for either date above I am interested, but not available at the listed times. Please contact me about other available

timeslots


Participant Confirmation Email and DirectionsHello [NAME],

This email confirms your participation in our upcoming study. We look forward to having you joinus and appreciate your time. Please respond to this email to confirm your attendance.

Date: [DATE OF SESSION]Time: [TIME OF SESSION] ***Please arrive 10 minutes early to complete paperwork***Location: The User Testing lab, 3rd Floor, Sieg Hall, University of Washington CampusParking: We have included some suggestions, but there are many parking options. Please call[Greeter Name] if you need help.

On the day of the study we’ll have [INSERT GREETER NAME] available as your primary point ofcontact. They will meet you at the entrance to Sieg Hall to walk you to the study lab. Please call[phone number] upon your arrival and we will coordinate meeting you at the building entrance youare at.

Please note that this is a oneonone session and our researcher is expecting you atyour scheduled date and time. If, for any reason, you are not able to keep this appointment,please let us know as soon as possible by calling [phone number] or emailing us [email protected].

A few notes:

You will be asked to sign a standard usability testing consent form. If you haveany questions about this document, please let me know prior to your session.

If you wear glasses, please bring them with you. You may have to read printeddocuments.

Please expect to spend about 4560 minutes with us once you arrive. This will be a videotaped session. We use the recording for review and internal

presentations. Your name will not be associated with these recordings.

Your $50 Amazon.com gift card will be provided at the completion of the study before you leavethe campus. We will also reimburse you for parking fees.

All the best,[email protected]

PS We have attached a PDF with a campus map and directions for you.


CONTENT TO ADD WHEN CUSTOMIZING:Names and phone numbers:Caroline Colon: 2062409942Michael Gelon: 2062767674Bill Grimmer: 2063305435Irina Litvin: 8189877361

[email protected]


Getting to the User Testing Lab at the UWWe look forward to seeing you for the HR & Recruiting usability study. Here are directions tohelp you find us on the UW campus in Seattle.

We are located in Sieg Hall on the 3rd floor. This building is circled onthe map below. Parking options are the red buildings in the map below.

When you arrive at Sieg Hall, please call us at [INSERT GREETERPHONE] We will come down to the first floor to find you and bring youto the 3rd floor.

UW Map (with interactive parking options.): http://www.washington.edu/maps/


http://www.google.com/url?q=http%3A%2F%2Fwww.washington.edu%2Fmaps%2F&sa=D&sntz=1&usg=AFQjCNFfaaSY_oL9iAudubvWvexJzqhduw

http://www.google.com/url?q=http%3A%2F%2Fwww.washington.edu%2Fmaps%2F&sa=D&sntz=1&usg=AFQjCNFfaaSY_oL9iAudubvWvexJzqhduw

If you are driving and enter from the Main Gatehouse on 45thStreet, the closest parking lot is C17. See the map below fordriving to lot c17. This will be next to the building.


If you park in the Central Plaza Parking Garage here is a map withwalking directions to Sieg Hall:


Participant Consent & Recording Release Form

By signing this release and accepting the incentive offered to me as payment for attending, I hereby agree toparticipate in this study conducted and recorded by University of Washington Human Centered Design &Engineering Department graduate students (UW HCDE).

I understand that participation in this usability study is voluntary.

I give permission for audio and/or video to be recorded during this session. I understand that these materialswill be used for research purposes only, and that the final data will not be associated with my name. Irelinquish all rights to the recording and understand the recording may be copied and used by UW HCDEwithout further permission.

I have read and agreed to this release, understand all its terms, and execute it voluntarily.

Date: _____________

Please print your name: ____________________________________________________

Please sign your name: ____________________________________________________


Receipt Form: Refrek Usability Test Compensation

Please sign below to indicate that you have received the promised compensation for your participation intoday’s usability study.

Date: _____________

Please print your name: ____________________________________________________

Please sign your name: ____________________________________________________

Thank you!We appreciate your participation.


Logistics Checklist and ScheduleStudy Checklist

24 hours before study Print consent forms Print copy of script/notes form Gather compensation & compensation confirmation form Check Morae project file setup Buy chocolates & water for test room Make sure we have two PC laptops running Morae software to supplement test room computer

(one for task manager & one for observing team members)

1 hour before study Check calibration for Morae & Tobii software Prepare room with bottled water & chocolate for participant Set out notes sheets and participant handouts in the room

After study Organize result files and export from Morae Unplug the Tobii eye tracker Compile Notes from team Clean up food and other leftover items

Scheduling

2/15 | Pilot Test Participant Name Participant ID

2:30 PM Mike M. P001

Thursday 2/20 | Session 1 Participant Name Participant ID

2:30 PM - 3:30 PM TBD A001

4:00 PM - 5:00 PM TBD A002

5:30 PM - 6:30 PM TBD A003

Sunday 2/23 | Session 2 Participant Name Participant ID

10:00 AM - 11:00 AM TBD B001

11:30 AM - 12:30 PM TBD B002

1:00 PM - 2:00 PM TBD B003


Task FlowTask 1: Getting Started Upon visiting Refrek.com for the first time, describe what services you think

the site provides.

Task 2: Signing Up Sign up for a corporate account as you would at work.

Task 3: Simple Interview Now that you are logged in -a) Let’s create a simple interview questionnaireb) Fill out the form as if you were hiring for an office manager in a Seattle office.c) Edit a questiond) Delete a questione) Add a new questionf) Enter 4 candidate names and email addressesg) Send the interview questionnaire to the users

Task 4: Advanced Interview You are still logged in. You want to create a new interview with avariety of question types.a) Name and describe the interviewb) Choose a type [should choose “Create from Scratch” option]c) Create a questionnaire name & select a categoryd) Give your initial impressions of this page [page after task 4.c]e) Asked, “How would you create a new question?”f) Use the “Add A Question” button to add a multiple choice question & save the question.g) Next, add a video response question.h) Use the “Select From Existing” option to add a question from the existing databasei) The survey is complete. Now send it out.

Task 5: Send Advanced Interview & Debriefa) Enter candidate information for two people, complete the form, and send it out.b) How do you feel about the screen that comes up after completion?


Moderator Script and NotesParticipant ID Moderator Name Session Start Time Session Date

[Before entering the lab, the greeter will show the participant around, including the restroom andthe waiting area. The greeter will also have the participant sign consent forms.]

Hi, I am ______. Thanks for coming in today to help us with our study.

[Brief small talk, offer the participant water and snacks, tell them to make themselves comfortable]

Again, my name is ___. I will be your study moderator today, and will be leading you through the study.

In order to be consistent across all participants, I will be reading from a script. Before we begin, I’ll go overthe basics of what we will be doing during our session. Feel free to ask any questions as they come up.

Today, we will be looking at a web based videointerviewing platform called Refrek.com. I will lead youthrough a series of tasks designed to evaluate the effectiveness of this website’s features. As we go along,your actions will be recorded in several ways. This is for reference, and the final data will not be associatedwith your name. I will be taking notes, and we also have ___________ here to help with the logistics.

For the accuracy of our results, we will need to calibrate the eye tracker before we begin.[Calibrate the Tobii Eye Tracker]

Some things to keep in mind:

The website is being tested, not you. This is not a race, so please do not feel pressured to rush through it. Ididn’t design or build anything that you will see today, so nothing you say will offend me. Although I will notbe able to directly help you through the tasks, let me know if you are confused or are unsure how tocomplete a task, as that information is very helpful to us. I may periodically ask you to describe what justhappened, or what you expect to happen.

At any point, feel free to ask me any questions. Do you have any questions before we begin?

During the study, I encourage you to verbalize your thoughts as you work through the tasks. Everything youare thinking is important here today, and I’m interested in any comments, positive or negative that you mayhave. Don’t worry, I’ll keep reminding you about this, since it usually doesn't come naturally to people.

We’ll start out with an exercise to practice thinking out loud. Let’s have you walk me through the front doorof your home describe the windows, as you walk past them...

[Let them describe, ask a follow up question as appropriate]

Ok, that was good. Let’s continue.


PRETEST QUESTIONS:As we get started, I have a few general questions about your daily routine.

1.) How often do you use a PC or Mac at home? not very often 1 2 3 4 5 very often

2.) How often do you use a PC or Mac at home? not very often 1 2 3 4 5 very often

3.) Do you have and use a LinkedIn account? [ Y / N ]

5.) How often do you use the internet for professional networking? not very often 1 2 3 4 5 very often

4.) Do you consider yourself tech savvy? [ Y / N ]

6.) Do you like to try new and emerging technologies? [ Y / N ]

We want to understand your thoughts about online video interviewing.

1.) I can see myself using an online video interviewing service for recruiting purposes.[ ] yes; why? _________________________________________________________________________________________

[ ] no; why not? _______________________________________________________________________________________

2.) Before today, have you ever heard of webbased video interviewing services? [ Y / N ]

2b.) If ‘yes’, please list services you have heard of:__________________________________________________________________________________________________

3.) Have you ever used a webbased video interviewing service? [ Y / N ]

3b.) If ‘yes’, please list services you have used:

_________________________________________________

Before we start exploring the site, we would like to know more about your professional roles.1.) Does your company utilize any online video interviewing technologies?

[ Y / N ]

2.) Have you ever been involved in the implementation of a new hiring technology at your company?[give example of Applicant Tracking Software, if they need a prompt]

[ Y / N ] If so, what was your specific role? _________________________


TASK 1 Viewing Homepage:

[The moderator brings up Refrek.com]Review the Homepage. What services do you think this website provides?

1.) Based on this page, what services do you think Refrek.com provides? [get 36]Video interviewingcandidate screeningLinkedIn

POST TASK QUESTIONS [1]:

1.) Based on your first impression, how does this site compare to similar sites you have seen?Refrek.com is:[ ] much higher quality[ ] higher quality[ ] similar quality[ ] lower quality[ ] much lower quality[ ] I have never seen this kind of service before

2.) Does Refrek.com look like a site you might use?[ ] yes; why?

[ ] no; why not?

3.) How easily do you expect to navigate Refrek.com?not very easily 1 2 3 4 5 very easily


TASK 2 Setting up an Account

You’re interested in using this service to screen candidates.To get started, you’ll want to create a corporateaccount.

Pre Task Question How do you expect this task to go?not very easy 1 2 3 4 5 very easy

Go ahead create an account now. Here is the information you will need for this task.*Provide printout of credentials for participant. Make sure second tab is open to Gmail with participant’semail panel open. Handout = “Setting up an Account”*Participant success = signs up successfully as Employer.

Path A: Top Navigation bar Sign In

Path B: Candidate Screening Platform Link

Path C: Refrek In

Path D: 1click video interview


1.) How would rate this task compared to what you expected?not very easy 1 2 3 4 5 very easy

2.) Why did you give it that rating?


TASK 3 Participant is logged in. Starts at INTERVIEW page[https://refrek.com/NewCorpUser/InterviewList.aspx]

A. You need to setup a quick and easy online interview for hiring a Seattle Office Manager.

Before we start:How do you expect this task to go?not very easy 1 2 3 4 5 very easy

Go ahead and start. *Participant success (A) = chooses Simple option

[User is now on the simple interview questions page]Can you describe this page? What can tell me about your thoughts on the content and layout of this page?

B. Again, this is for a Seattle Office Manager role and is a first round of screening interviews. Let’s proceedwith setting up this questionnaire. Go ahead and starting filling it out.

C D. Let’s say there are too many questions how do you fix that?


https://www.google.com/url?q=https%3A%2F%2Frefrek.com%2FNewCorpUser%2FInterviewList.aspx&sa=D&sntz=1&usg=AFQjCNFJM-djuVediZdvuwduOcvtQ6iuaw

E. Lets say there is one question missing, about a candidate’s job history. How would you create a newquestion?Handout = “Seattle Office Manager Job History question”*Participant success (BE) = correctly enters content and edits question fields. Delete, Edit, Add.

E.1 Of the questions that are listed, what format of response do you think you’ll get from candidates?(Because this is a multimedia interviewing platform, there are several types of question formats: interview,survey, fill in the blank, etc.)

F G. There are four candidates you’re considering. Please enter their contact information and forward thisto the candidates.*Provide printout of contact information to participant Handout = “Seattle Office Manager CandidateInformation”*Participant success (F G) = correctly enters Name and Email content, and pushes SENDQUESTIONNAIRE button.

You’ve completed the first series of tasks. I’ve got some questions about your experience so far I’d like youto answer before we move on to the next tasks.


1.) How would rate this task compared to what you expected?not very easy 1 2 3 4 5 very easy

2.) Why did you give it that rating?


TASK 4 [Participant is logged in. Starts at INTERVIEW page]

A. [Choose ‘advanced’ option]You need to set up a more complex interview template with your own variety of questions, to hire aRegional Manager for a customer service position.

Before we start:How do you expect this task to go?not very easy 1 2 3 4 5 very easy

Ok, go ahead and start.*Participant success (A) = chooses Advance option.

[Participant is viewing Add Interview Title and Description popup]Again, this is for a Regional Manager, for a customer service position and is a first round ofscreening interviews. [if the user needs prompting, “Go ahead and proceed”]*Participant success (A) = Title and Description added. Pushes NEXT button

B. [Participant is viewing Select Interview Questionnaire popup with three options; create fromscratch / select from existing / my interview questionnaire]

Will you describe what you think each feature does?

Create From Scratch:

Select From Existing:

My Interview Questionnaire:

We want to create an interview with our own questions, so let’s pick the option for FROMSCRATCH.


*Participant success (B) = selects Create from ScratchC. [Participant is viewing Add Interview Questionnaire popup]

Describe what you think this screen is asking for.How many categories do you see to select from? [if the user needs prompting, ”Please pick the most relevant for a customer service position.”]*Participant success (C) = Enters Questionnaire Name, and selects from dropdown ofQuestionnaire Categories. Target in dropdown is ‘customer service’, which may require participantto scroll down. Pushes CONTINUE button

D E. [Participant is viewing Manage Interview Questionnaire]Can you describe this page?What options do you have on this page?How would you go about adding a question?*Participant success (D) = Participant thinks aloud and answers moderator questions.

F. [Participant is viewing Add New Question popup]You would like to add a multiplechoice question. Here is a sample question, or you can create yourown.[Go ahead and click Add A Question]*When participant gets to popup, provide printout of question and answers. (Ex. Are you willing tocommute to any of these cities? Bellevue Woodinville Everett Tacoma) Handout=”Multiple choicequestion and answer”*Participant success (F) = Participant successfully adds multiple choice question to questionnaire.


G. [Adding video question]You want to add a question and receive a video response from your candidate. How can you dothis?*When participant gets to popup, provide printout of question. (Ex. Describe what the three mostimportant values you demonstrate as a leader?) Handout=”Add a video question”*Participant success (G) = Participant successfully adds video question to questionnaire.

H. [Prewritten question]You want to add a question that is prewritten in the system. How would you do this?*Participant success (H) = Participant successfully clicks Select from Existing button.

H.2 Participant is viewing Select from Existing Interview Questions popupCan you describe this page?What options do you have on this page?How would you go about adding a preexisting question?Again, this is for a Regional Manager, for a customer service position.*Participant success (H) = Participant thinks aloud and answers moderator questions. Participantchooses Category (customer service) and Questionnaire (job type = any) fields from dropdownmenus. Participant clicks checkmarks to select questions and Add Selected Questions button.

H.3 Participant is viewing Manage Interview Questionnaire complete with selected questions.Can you describe this page, now that your questions are entered? Do you have any questions orobservations?*Participant success (H) = Participant thinks aloud and answers moderator questions.

I. [Sending]How would you forward this to your candidates?*Participant success (I) = Participant clicks on “Continue”


TASK 5 Starts at Send Interview Questionnaire page:

A. [Sending advanced]There are two candidates you’re considering. Please enter their contact information and forward thisto the candidates.Provide printout of contact information to participant Handout= “Regional Manager Candidateinfo”*Participant success (I) = Participant successfully adds two candidates, adds their company name,and clicks Send Questionnaire button.

B. Participant is viewing Interviews pageNow that we have sent the interview, can you describe this page we’ve landed on?*Participant success (H) = Participant thinks aloud and answers moderator questions.

[wrap up]You’ve completed the second series of tasks. I’ve got some questions about your experience so far I’d likeyou to answer before we wrap up.


1.How would rate this task compared to what you expected?not very easy 1 2 3 4 5 very easy

2. Why did you give it that rating?


PostTest Questionnaire

The System Usability Scale

These questions will be administered via the Morae software and will come up on the screen forthe user at the conclusion of our study.

The questions will all be answered via this 5 point Likert scale below:

1. I think that I would like to use this web site frequently.2. I found the website unnecessarily complex.3. I thought the website was easy to use.4. I think that I would need the support of a technical person to be able to use this

website.5. I found the various functions in this website were well integrated.6. I thought there was too much inconsistency in this website.7. I would imagine that most people would learn to use this website very quickly.8. I found the website very cumbersome to use.9. I felt very confident using the website.10. I needed to learn a lot of things before I could get going with this website.


Thank you for participating in today’s study. Please answer the following final questionsregarding your experience today.

1. Do you feel like you have a good idea of the scope services this site offers after yourinteraction with the homepage and after doing the tasks?

2. What task(s) were comfortable for you?

3. What task(s) were more difficult for you?

4. Would you have any suggestions for improving this website?

6. Would you use this type of service why or why not?

7. Would you recommend this website to your colleagues why or why not?

Session End Time

[Give the participant the compensation form and the gift card. Thank them profusely fortheir participation. Hand the participant to the greeter to direct them out.]


Participant Handouts

Setting Up An Account

Email: [email protected]: userofrefrek

















Seattle Office Manager Job History Question

Tell me about your previous work experience for small companies


Seattle Office Manager Candidate Information

Bob Jones [email protected]

Jane Smith [email protected]

Sam Miller [email protected]

Sue Shore [email protected]


mailto:[email protected]



Regional Manager multiple choice question and answer

Are you willing to commute to any of these cities?BellevueWoodinvilleEverettTacoma


Regional Manager Add a video question

Describe the three most important values you demonstrate as a leader?


Regional Manager Candidate Information

Sam Miller [email protected]

Sue Shore [email protected]



Documents

HCDE 517 Winter 2014 | Usability Study Test Kit Caroline ...michaelgelon.com/wp-content/uploads/2015/01/The... · Please note we are only looking for HR professionals & recruiters