Usability Test Report for InnoVenture.com Date of Report: March 6, 2014 Date of Test: February 08, 2014 – February 18, 2014 Location of Test: Usability Testing Facility, Clemson University, Clemson, SC Prepared for: John Warner, CEO, InnoVenture, LLC. Phone Number: 864-653-8888 Email: johnwarner@innoventure.com Prepared by: Brian Gaines Phone Number: 619-240-6322 Email: blgaine@g.clemson.edu

  2  

Table of Contents

Usability Test Report for InnoVenture.com ........................................................................ 1  Table of Contents ........................................................................................................................ 2  Executive Summary .................................................................................................................... 3  Methodology ................................................................................................................................ 4  

What happened during the usability test  ...........................................................................................  4  Who we tested  ................................................................................................................................................  4  What participants did  ...................................................................................................................................  5  Where we tested  ............................................................................................................................................  6  

Introductory Questions & Tasks .................................................................................................. 7  What participants did  ...................................................................................................................................  7  

Findings & Recommendations .................................................................................................... 9 Conclusion ................................................................................................................................. 15

    Appendices ................................................................................................................................ 16  

  3  

Executive Summary Ten users participated in this study, six females and four males. The average age of the participants was 28, two of the participants were from foreign countries, all of the participants held a bachelor’s degree, two were Ph.D. students, two were Ph.D. candidates, and one held a doctorate. All of the participants identified as having an interest in entrepreneurship and innovation, all considered themselves Internet savvy, three of the participants identified as having a working knowledge of HTML, CSS, and JavaScript, and one participant had recently built a website. Prior to the usability study, a survey was administered to 385 current users of InnoVenture.com, with 75 reporting. The questions featured a variety of open-response and Likert-type questions. The study used a think-aloud protocol methodology where the participants were given tasks that related to the innoventure.com web interface. The participants were video recorded as they performed the tasks, talking aloud about the tasks and saying what they were thinking as they worked. Approximately 10 hours of video was recorded, and the participants’ behaviors were subsequently coded and analyzed. The accuracy of their performance was also assessed during the task analysis. After completion of the think-aloud protocol methodology, a System Usability Scale (SUS) survey was administered to the participants to gage the innoventure.com web interface. Finally, an informal interview was conducted with each participant to allow the participant to further elaborate on any positive or negative experiences they may have experienced with the interface. No compensation or incentive was given to any participants for their voluntary participation in the study. The study found that users tended to find the InnoVenture social media interface to be a great idea, and that overall the site appeared to be credible. Four of the users felt the color choices, icons, and layout was aesthetically pleasing, and that these choices were strong indicators of the site’s credibility. Five of the participants stated they would most likely recommend the site to others who have an expressed interest in entrepreneurship and innovation, and three said they would recommend the site if changes to the usability of the site were implemented. However, the ease of use concerning navigation, information architecture, intuitiveness (ease of use), and information design was lower than desirable. On a 4-point scale where 4=completed with ease and 1=failed to complete, the participants averaged a score of 2.81, which placed them between “completed with great difficulty” and “completed with some difficulty” in regards to tasks associated with the interface. The SUS survey, which is scored on a 100- point scale with a score of 68 being “usable”, the participants’ averaged 28.0.

  4  

The coding of the video collected and an examination of the patterns which emerged revealed that the participants experienced difficulties in navigating to specific areas of the site, experienced difficulties in locating specific information, and concern over the functionality of certain features. Three of the participants in the study left the InnoVenture interface to locate information that pertained specifically to the interface. All of the participants stated that the interface contained too much text for a Web interface, and that the abundance of text was a great distraction. Six of the participants felt the text was too large on certain pages within the site, and five of the participants felt that there was no cohesiveness to the information design of the pages, and that this further increased the distraction created by the amount of text. The remainder of this report provides more details concerning the methodology and findings of this study, as well as recommendations based on these findings. Methodology What happened during the usability test

The usability evaluation of the InnoVeture.com social media interface was conducted by Brian Gaines at Clemson University’s Usability Testing Facility in Clemson, SC, between February 8, 2014 and February 18, 2014. During the usability evaluation, 10 participants, matching the user profile(s), were asked to spend 30-45 minutes with the site. During this time, participants:

§ Completed the appropriate Consent Forms (Appendix A) § Answered questions concerning qualifications for the study § Performed real-world tasks on the site while thinking aloud § Completed a SUS survey (Appendix B) and answered questions about

their overall satisfaction

Who we tested Ten participants, having the following profile characteristics, evaluated

InnoVenture.com. • Expressed an interest in entrepreneurship and/or innovation. • Held an advanced degree(s) or be matriculated into a graduate course of

study. • Had at least a working knowledge of how to use the Internet.

  5  

Participant Age Gender Educational

Level Uses

Internet Interested in

Entrepreneurship/Innovation? Heard of

InnoVenture.com?

1 28 F Ph.D. Student Yes Yes No

2 22 F Master’s Student Yes Yes No

3 25 M Master’s Student Yes Yes No

4 32 M Ph.D. Candidate Yes Yes No

5 27 F Ph.D. Candidate Yes Yes No

6 33 M Master’s Student Yes Yes No

7 28 F Ph.D. Yes Yes No

8 32 M Ph.D. Student Yes Yes No

9 24 F Master’s Student Yes Yes No

10 25 F Master’s Student Yes Yes No

What participants did During the usability evaluation, participants were asked to complete 10

scenarios or “real-life” tasks on the site. The tasks were presented in sequential order, as listed below and the participants were aware that no assistance of any kind would be offered.

The participants performed the following tasks, with prompting from the usability software.

  6  

# Task 1 [Navigate to the URL “ http://www.innoventure.com”.

2 From the innoventure.com homepage, navigate to the “About Us” section and identify

who the founder is.

3 Locate the contact information for innoventure.com

4 Locate how you would join the InnoVenture community.

5 Navigate to “Networks” and locate the “Clemson Innovation” network. Within this

network, locate Adam Hoover’s “Big Idea” and view his presentation. 6 [Locate and identify the number of projects that the “Clemson Innovation” network is

currently promoting.

7 Navigate to the screen where you would learn to promote your “Big Idea”. Determine how many ways in which you may promote your “Big Idea”.

8 Navigate to the screen where you would most likely upload any contacts you may want to promote your “Big Idea” to.

9 Locate the search bar and perform a search for “Drew Felty”. Once you have located Mr. Felty, describe his “Big Idea”.

10 Log out of “http://www.innoventure.com”.

Where we tested Following is a summary of the participants’ computing environment,

located within Clemson University’s Usability Testing Facility.

URL of tested website: http://www.innoventure.com Computer platforms: Dell OptiPlex 960 with dual 18”

displays Browser tested: Google Chrome, Version 33.0.1750.146

Screen resolution: 1024 X 768 Operating system: Windows XP Connection speed: Shared T1

  7  

The following software and hardware was also used to document the participants

§ Techsmith Morae Usability Testing Software § LogiTech Web Camera

Within the Usability Testing Facility (UTF), only one participant and the researcher were present at any given time. The UTF remains locked at all times, and entrance to the facility is only granted through the permission of the director, Tharon W. Howard, Ph.D. All participants and visitors to the facility must be accompanied by either the director or authorized personnel. Introductory Questions & Tasks  

At the beginning of each session, we asked participants 4 questions:

§ Do you have an interest in entrepreneurship and/or innovation? § Have you ever heard of or used InnoVenture.com? § Who do you think this site is designed for? Why? § Would you be willing to try the site out and give me feedback

concerning your experiences with it?

Upon answering “yes” to the following questions, the participants disclosed personal information regarding educational status and computer skills. After the requirements for the study were met, the participants completed consent forms allowing them to be recorded and to give the researcher permission to conduct the usability test. Prior to the actual usability test, the participants were asked to assemble a ballpoint pen and to describe their actions to build a sense of rapport and to prepare for the usability test.

After assembling the ballpoint pen, the participant then sat positioned in front of the computer monitor and LogiTech Web camera used to document the test. The researcher, seated across the room and separated by a small partition, monitored and coded the session via Morae Observer, a component within the TechSmith Morae Usability Testing Software. From this station, the researcher was able to view the screen the participant was using, as well as being able to view the participant via a picture-in-picture (PIP) display. The researcher took notes concerning the completion or failure of the tasks at hand, and was able to view the SUS survey.

What participants did During the usability evaluation, participants were asked to complete 10

scenarios or “real-life” tasks on the site. The tasks were presented in sequential

  8  

order, as listed below and the participants were aware that no assistance of any kind would be offered. The participants performed the following tasks, with prompting from the usability software.

# Task 1 [Navigate to the URL “ http://www.innoventure.com”.

2 From the innoventure.com homepage, navigate to the “About Us” section and identify

who the founder is.

3 Locate the contact information for innoventure.com

4 Locate how you would join the InnoVenture community.

5 Navigate to “Networks” and locate the “Clemson Innovation” network. Within this

network, locate Adam Hoover’s “Big Idea” and view his presentation. 6 [Locate and identify the number of projects that the “Clemson Innovation” network is

currently promoting.

7 Navigate to the screen where you would learn to promote your “Big Idea”. Determine how many ways in which you may promote your “Big Idea”.

8 Navigate to the screen where you would most likely upload any contacts you may want to promote your “Big Idea” to.

9 Locate the search bar and perform a search for “Drew Felty”. Once you have located

Mr. Felty, describe his “Big Idea”.

10 Log out of “http://www.innoventure.com”.

Numerical assignments were given to the ability to complete tasks, with one (1) being “Failure to Complete task”, two (2) “Completed Task with Difficulty, three (3) “Completed Task with Some Difficulty, and four (4) being “Completed Task With Ease”. The data collected from these tasks were scored and averaged.

  9  

Findings & Recommendations  

A survey was sent to 385 InnoVenture users, with 75 users responding. The survey consisted of a combination of open response questions and a five-point Likert-type survey. The questions were geared towards the value of connections made on the InnoVenture interface, as well as some of the features, such as uploading 250 contacts at once.

When asked if any valuable connections had been made on InnoVenture.com, 29% of the respondents indicated that, yes, valuable connections had been made. 47% of those who responded indicated that they had not made a valuable connection. 19% of the respondents were unsure, and five percent of the survey participants did not respond. When asked if they were aware of how to make valuable connections, 61% indicated yes, 11% were not aware, 24% were not sure, and 4% did not respond. In regards to being aware of uploading up to 250 contacts at a time, 56% of the respondents indicated yes, 23% indicated no, 15% were unsure, and four percent did not respond. When asked about valuable connections that were made as a result of regularly updating progress on a Big Idea, the respondents indicated that 19% of them did in fact make valuable connections, 60% did not, 20% were unsure, and one percent did not answer. The open response section of the survey varied among the 75 respondents, with the majority giving support for Warner and the site, or promoting their own Big Idea. A few respondents suggested linking to other social media platforms, such as LinkedIn, Facebook, and Twitter.

The participants selected for the study took part in a task analysis and think-aloud protocol, completed a SUS survey, and agreed to take part in an exit interview for the study. Through this process, many positive and negative insights came to light.

The task analysis featured 10 tasks, ranging from very simple to complex.

Numerical assignments were given to the ability to complete tasks, with one (1) being “Failure to Complete task”, two (2) “Completed Task with Difficulty, three (3) “Completed Task with Some Difficulty, and four (4) being “Completed Task With Ease”. The data collected from these tasks were scored and averaged (Figure 1.). The range of the scores was1.8 to 2.8, and the mean score among the 10 participants to complete tasks was 2.41. Based on these scores, the participants experienced difficulty in performing most of the tasks in the interface’s current iteration (Figure 1.).

  10  

(Figure 1. ) Graphical depiction of Task Analysis. On a scale of 1 (failed to complete) to 4 (completed with ease), the mean score among 10 participants was 2.8.

At the conclusion of the task analysis, a System Usability Scale (SUS) was administered to the participants. The usability of a system, as defined by the ISO standard ISO 9241 Part 11, can be measured only by taking into account the context of use of the system — i.e., who is using the system, what they are using it for, and the environment in which they are using it. Furthermore, measurements of usability have several different aspects:

• effectiveness (can users successfully achieve their objectives?) • efficiency (how much effort and resource is expended in achieving those

objectives?) • satisfaction (was the experience satisfactory?)

The SUS scores were determined by calculating each item's score

contribution range, from 0 to 4. For items 1,3,5,7,and 9 the score contribution was the scale position minus 1. For items 2,4,6,8 and 10, the contribution was 5 minus the scale position. Multiplying the score’s sums by 2.5 yielded the system usability score (Brooke, 1996). The scores were then averaged to determine the mean of 28.0.

0  

1  

2  

3  

4  

Average  Task  Errors  

Tasks  

Average  Task  Errors  

  11  

While SUS scores are not exact percentages, it is generally accepted that

scores in the 68 and above range indicate a system is usable, and scores below 50 generally indicate poor usability. The InnoVenture.com interface had a range of 0.00 to 60.0, and a mean score of 28.0 among the 10 participants (Figure 2.).

(Figure 2.) System Usability Scale (SUS) scores for participants. The mean system usability of InnoVenture.com is 28.0 on a scale of 0-100.

The participants felt there were several positive attributes within the InnoVenture social media interface. 100% of the participants thought that a social media platform focused on connecting individuals focused on innovation and entrepreneurship with those with valuable resources or connections was a great idea. Sixty percent felt the interface was credible, owing to the clean design and the color palette. Additionally, 60% stated they would recommend the site to others. Ten percent described the interface as “looking the way a social media site should.”

Unfortunately, the participants felt the negative aspects of the

InnoVenture.com web interface overwhelmingly outweighed the positives. All of the participants stated that there was simply too much text, with over half stating they would not read all of it. Eighty percent described the interface as frustrating, particularly concerning the inoperability of the search bars, uploading, and navigation features of the site. Ninety percent all of the participants expressed concern over the scarcity of information concerning how to use the site, the

0  

10  

20  

30  

40  

50  

60  

70  

p1   p2   p3   p4   p5   p6   p7   p8   p9   p10  

SUS  Score  

  12  

difficulty in navigating to the Frequently Asked Questions (FAQ) areas, the size of the font, lack of cohesion in menu buttons, and the lack of intuitiveness of the overall site. One particularly colorful participant described the interface as a “Podunk ass website” and further stated that they would not recommend this site to anyone under any circumstance. Of the 50% of the participants who elected to join the Innoventure.com, 80% of those voiced concern over having to verify their membership via a two-step process involving email, as well as whether or not InnoVenture would send unsolicited bulk advertisement via email (spam), and how frequently.

The introductory video on the home page was viewed by 30% of the

participants, who described it as “cheesy”, “unprofessional”, and “stupid.” One participant identified the background music as the instrument track to popular song “Gangnam Style”, and wondered aloud if InnoVenture had secured the rights to use the music.

The participants were also questioned during the informal interview as to what they would feel would make InnoVenture.com a more suitable social media web interface. All of the participants thought that less text and more imagery would greatly enhance the interface (Figure 3.), as well as prominently displaying FAQs, working functionalities, more explicit instructions on how to get started, and a more intuitive interface. Sixty percent of the participants suggested uniformity of the menu buttons (Figure 4.), 40% suggested stating the benefit of being a part of the InnoVenture social network, and 30% suggested a messenger or chat feature, in addition to having the option to group Presenters and Followers based on the interests of the users themselves. One participant suggested a more customizable home page, where a particular user’s settings would make the interface more intuitive based on personal preference.

Despite the numerous negative associations by the participants, 80%

were hopeful that the problems associated with the interface could be easily resolved. For example, one participant suggested outright that making changes to the size and amount of text would present a more aesthetically pleasing layout that would make it more legible. Making minor changes to the interface, such as the amount of text, font size, improved functionality of buttons and search bars, and a more prominent FAQ would greatly enhance the overall user experience.

  13  

(Figure 3.) Screen Capture and Recommendations for Home Page.

  14  

(Figure 4.) Screen Capture and Recommendations for Changes to Fonts and Text.

 

  15  

Recommendations at a Glance  

• Establish uniformity among all buttons in the navigation menu.  • Correct search bar issue.  • Make buttons relating to uploading contacts clickable.  • Make Frequently Asked Questions (FAQ) section more prominent.  • Make information about how to join less scarce.  • Reduce the amount of text on information pages.  • Add more imagery to site as a whole.  • Make font size more uniform in body of text.  • Eliminate email verification for joining InnoVenture.  • Consider revising introductory video.  • Create more explicit directions for joining Innoventure.com.  • Create customizable features for Dashboard.  • Create a messenger or chat feature.  

   

Conclusion

The usability testing for InnoVenture.com has exposed several minor issues affecting the overall usability of the interface. While in themselves minor, the unique combination of these usability issues has been shown through a task analysis and SUS survey to present major issues to new users of the interface. While potential new users to the interface agree that it is a great idea, problem areas concerning navigation, searching, functionality of the site, amount of text, and a multimedia presentation that could be considered below the intellectual mindset of a user present serious challenges. Through implementing these recommendations, InnoVenture.com can create a positive user experience for both established and new users of the site.

   

  16  

Appendices Appendix A. Informed Consent Form

Consent Form for Participation in a Research Study

(Innoventure.com Usability Study) Description of the research and your participation: You are invited to participate in a research study conducted by Brian Gaines under the supervision of Dr. Tharon Howard. The purpose of this research is to study usability of the InnoVenture.com web interface. Your participation will involve completing a series of tasks pertaining to the InnoVenture.com web interface, as well as answering interview questions. You will be asked to complete these tasks to the best of your ability and to discuss what steps or actions you are performing according to multiple scenarios (approximately 10 total) that will be read to you prior to each task. Once your permission to record the performing of the tasks has been given, we will begin taping the data. Also, a portion of the study involves audio recording your answers, and taping will only begin when your permission has been given at that time. The amount of time required for your participation will be approximately 30 - 45 minutes. Risks and discomforts: There are certain risks or discomforts associated with this research. While all precautions to minimize identity will be implemented, your recorded image may appear in a video that will be viewed by the MAPC Thesis Committee. Potential benefits: By participating in this study, you will help to further the research on website usability. This will help academics and practitioners understand how to better design websites for a more enjoyable user experience. Protection of confidentiality: Once we have been given your permission to do so, we will begin recording using Techsmith Morae software. We will use this data, and the data collected from other people, to determine how usable the InnoVenture.com web interface is. The data, audio,

  17  

and video collected will be compiled and the results shared with the MAPC Thesis committee as part of fulfillment of the requirements of the Master of Art in Professional Communication degree.. Various practitioners and academics may potentially use this information for their own research purposes. However, your name will not appear in any of the written documents, or the presentation, and we will do everything possible to protect your privacy. All video recordings, audio recordings, notes, and other materials pertaining to this research will be kept in a locked cabinet in the Clemson Usability Testing Facility where they are only accessible to members of the Usability Testing Facility Staff. The information will be kept for no longer than 5 years before it will be destroyed. Also, no identifying information, such as locations, addresses, actual names or demographic data that could be used to identify participants will be used in any written document or presentation resulting from this study. Voluntary participation: Your participation in this research study is voluntary. You may choose not to participate and you may withdraw your consent to participate at any time. You will not be penalized in any way should you decide not to participate or to withdraw from the study. Compensation: No compensation will be offered for your participation in this study. Contact information: If you have any questions or concerns about this study, or if any problems arise, please contact Brian Gaines at 619-240-6322. If you have any questions or concerns about your rights as a research participant, please contact the Clemson University Institutional Review Board at 864-656-6460. Consent: I have read this consent form and have been given the opportunity to ask questions. I give my consent to participate in this study. Participant’s signature: ________________________________ Date: _______________ A copy of this consent form should be given to you.

Appendix A (cont.). Video Release Form

  18  

Audio / Video Release Form The signature below indicates my permission for Dr. Tharon Howard and Brian Gaines to use video and audio recordings that have been collected during the research study conducted for: InnoVenture.com Usability Study on _________________(date) in which I served as participant. My name will not be reported in association with session results nor will my name be included in any of the data collection results. The video and / or audio recordings may be used for the following purposes:

• Analysis of research and reporting results • Thesis write-up and presentation • Thesis defense presentation • Conference presentations • Educational presentations • Informational presentations

I will be consulted about the use of the video recording for any purpose other than those listed above. There is no time limit on the validity of this release nor is there any geographic specification of where these materials may be distributed. This release applies to recordings collected as part of the research session listed on this document only. I have been given a blank copy of this release form for my records. Name (please print): _____________________________________ Date: ____________ Signature: _______________________________________________________________ Address: ________________________________________________________________ _______________________________________________________________________ Phone: _________________ Email: __________________________________________

  19  

Appendix B. System Usability Scale

  Strongly  Disagree  

Disagree   Neither  Agree  nor  Disagree  

Agree   Strongly  Agree  

1. I think that I would like to use this

system frequently.

 

         

2. I found the system unnecessarily

complex.

 

         

3. I thought the system was easy to use.

 

         

4. I think that I would need the support of

a technical person to be able to use

this system.

 

         

6.I found the various functions in this

system were well integrated.

 

         

7. thought there was too much inconsistency in this system.  

         

I found the system very cumbersome to

use.

 

             

9. I felt very confident using the system.

 

             

10. I needed to learn a lot of things before I

could get going with this system.

 

             

  20