GUERILLA HCIAnd Customer Based Design

Like a boss.

Quentin ChristensenQuentin.Christensen@live.com

Agenda• Customer Activities• Research Methods• Shipping Activities

Special thanks to Jakob Nielsen, his 1994 Guerilla HCI article is the basis for this presentation• http://en.wikipedia.org/wiki/Comparison_of_usability_evaluation_methods• http://www.useit.com/papers/guerrilla_hci.html• http://www.useit.com/alertbox/discount-usability.html

Cost savings in a medium scale software project

Original usability cost estimate by [Mantei and Teorey 1988] $128,330Scenario developed as paper mockup instead of on videotape - $2,160Prototyping done with free hypertext package - $16,000All user testing done with 3 subjects instead of 5 - $11,520Thinking aloud studies analyzed by taking notes instead of by video taping - $5,520

Special video laboratory not needed - $17,600Only 2 focus groups instead of 3 for market research - $2,000Only 1 focus group instead of 3 for accept analysis - $4,000Questionnaires only used in feedback phase, not after prototype testing - $7,200

Usability expert brought in for heuristic evaluation + $3,000Cost for "discount usability engineering" project $65,330

Source: Jakob Nielsen 1994 http://www.useit.com/papers/guerrilla_hci.html

Customer Activities Research Methods• Focus Groups• Customer Focused Design• Customer Visits• Councils

• Customer• Partner• Marketing/Sales

• Customer Buddies• Internal Partners• Partner Relationships

• Concept Box Testing• Usability Lab Testing• Discount Usability Testing

• Scenario• Customer design review• Simplified think aloud• Pluralistic walkthroughs• Heuristic evaluation

• UX/UI/String Reviews• Polish SWAT• Real world projects• Scenario testing & health• White papers• Early customer deployment• Dogfood

Shipping Activities

Customer Activities

• Focus Groups• Customer Focused Design• Customer Visits• Councils

• Customer• Partner• Marketing/Sales

• Customer Buddies• Internal Partners• Partner Relationships

What should I build?

Focus Groups / Customer Focused Design

• Focus Groups• Create a topic• Get several people in a room• Ask them questions• Record their responses

• Customer Focused Design• Same as focus group• List out the design ideas• Have the room rate the design ideas• Confirm the priorities with the group

Customer Visits• Travel to the customers location• Spend a day or two meeting various users• Use one on one interviews and focus groups• Try to collect artifacts if possible• Take detailed notes

Councils• 3-20 people that are experts in the area of the problem you

are solving• 3 Types:

• Customers – people that will use your product• Partners – developers that will extend your product• Marketing/Sales – the people that will sell your product

• Create the council when you begin planning the product• Meet with the council throughout development• Quarterly meetings

• Collect feedback• Present early designs Product demos• Discuss product plans• Show UI

Customer Buddies• 1-3 users that you have a deep ongoing relationships with• Frequent meetings and emails• Conduct UI walkthroughs during iteration, early & often• Discuss major design problems and design alternatives• In depth design discussion• Clarification of requirements

Internal Partners• People at your organization that will be “users” of your

product• Easy to setup a meeting and discuss issues!• Help find usability study participants• In depth discussions• UI walkthroughs• Dogfood preparation

Partner Relationships• Relationships with developers that will build on or extend

your product• Get their feedback as experts that understand the market• Design a platform that supports their needs• Help them go to market earlier and increase adoption• Two types:

• Independent Software Vendor (ISV) – sell and markets an app that uses your platform

• Systems Integrator (SI) – sells your app and helps customers deploy and integrate with other software

Research Methods

• Concept Box Testing• Usability Lab Testing• Discount Usability Testing

• Scenario• Customer design review• Simplified think aloud• Pluralistic walkthroughs• Heuristic evaluation

How should I build it?

Testing With 3-5 Users

http://www.useit.com/alertbox/20000319.htm

Scenario• Explain in a paragraph• Reduce to the bare minimum• Small and easy to change• Use low cost prototypes to

describe and test on users

Try this with:• Your team members• UX• Individual users

When:• You first start designing

Example: Jim is authoring a document on hurricane patterns in the Gulf of Mexico. He shares the document with Armando, who works at a different university, for review. Armando adds several comments and Jim is delightfully surprised that he can see the comments right in the document and easily make the updates.

Customer Design Review• Meet with a small group of users• Present a design / prototype• Explain the design and functionality• Lead the discussion• Ask probing questions• Focus on uncovering requirements

Try this with:• Individual users • Groups of customers

When:• You have a decent prototype• Early and late in the design

Simplified Think Aloud• Test one user at a time• Have a list of tasks / use cases• Show a prototype• Users perform the tasks• Detailed design, but low fidelity• Take notes and measure performance• Try testing multiple designs

Try this with:• Individual users• UX• Your team members

When:• Nearly final design• After coding + some polish

I have no idea

what to do next.

Cognitive / Pluralistic Walkthroughs• Walk through the design with people from various backgrounds

together • Ask everyone to write on paper how they would accomplish each task

Discuss amongst the group• Present the correct answer if the group goes too far off course• Don’t lead or try and explain everything to users• Have a list of tasks and a prototype

Try this with:• Customers• UX• Your team members

When:• Nearly final design• After coding + some polish

Heuristic Evaluation• 3-5 evaluators• Have a prototype or actual implementation• Walk through the design• Rate the design based on heuristic standards

Try this with:• Your team members• UX

When:• Any time

Heuristic Evaluation1. Visibility of system status The system should always keep users informed about what is going on, through

appropriate feedback within reasonable time. 2. Match between system and the real world The system should speak the users' language, with words,

phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

3. User control and freedom Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

4. Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

5. Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.

6. Recognition rather than recall Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

7. Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

8. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

9. Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

10. Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

http://www.useit.com/papers/heuristic/heuristic_evaluation.html

Microsoft eDiscovery Examples• User tasks• Score sheet• Scenario Health• Design for eDiscovery Preservation

User Tasks1. A legal event has occurred that requires you to hold content

1. Identify the mailbox for Dan Jump and Quentin Christensen, document center SharePoint site, and file share

2. Get an estimate of how many items there are3. Reduce the amount of content using some keywords4. Hold the content to make sure it cannot be deleted5. Gather content to ensure it is held

2. You know more about the case and need to limit the content so you can export it, but you don’t want to affect how it is preserved

1. Use keywords to limit the data set2. Get the content to the correct set of data you need3. Close your work so you can access it another time

3. You now need to gather the data set you have specified 1. Select the content you want to gather2. Send it to a location where you can easily access it3. Verify you have the correct content and are complete

4. You need to print a report to prove you adequately held content 1. Create a report that shows the dates of all the content you preserved

Heuristic Score Sheet

http://office/15/teams/exchange/SEC/FindMyVault/FederatedFind/Shared Documents/eDiscovery Scenario Status.xlsx

Scenario HealthStatus Scenario Task Bugs/Notes

  Create a case  

  Create a discovery set  

  Add a mailbox  

  Add a SharePoint site  

  Get the number of items  

  Enter a filter and view the stats  

  Preserve and save the discovery set  

  Check the preservation in Exchange    Delete an email - notice it is kept in the dumpster    Check the SharePoint site see it is preserved  

  Edit a doc - it is copied to the preservation library  

  Delete a doc -  it is copied to the preservation library  

  View auditing report of when content was preserved    Create a new query    Specify a query with a wildcard and proximity  

  Narrow down the query scope  

  View results from Exchange, SharePoint, and file shares  

  Show the hovercard with a preview of the message   Open the message    Select multiple refinements for SharePoint  

  View the hovercard for SharePoint items  

  Save the query  

  Export the query  

Design for eDiscovery Preservation

3:00PM1/23/2008

Search SaveEstimate Hits

eDiscovery

Document Name – Windows Internet Explorer

Favorites | Website 1 Website 2 Website 3

http://mysite/ediscovery Search

HomeCustodiansLocationsSearchesExportsReports

All of these words:The exact phrase:

Any of these words:

None of these words:

Scope

Date Range: to

Email Property Restrictions

File Property RestrictionsLanguage Restrictions

All Mailboxes

Mailboxes

All Locations

Locations

Result Type

Keywords Keywords:

Add Mailboxes Add Locations

Add Mailbox

CancelSave

Custodian: Lionel Penuchot

Add To Case Name: Luka AbrusPreserve MailboxNotify

Place Associated Locations on Hold

Associated Locations

><

Import Locations

Lukas DesktopFileshare BTeam SiteProject Team Site

3:00PM1/23/2008

eDiscovery

Document Name – Windows Internet Explorer

Favorites | Website 1 Website 2 Website 3

http://mysite/ediscovery Search

Case Manager Home

Cases

Holds

Custodians

Content Sources

Content Sources

Add Content Source

Name Type Items Size

Data Wiz Site SharePoint Site 3,460 1.384GB

Data Wiz Files

Fileshare 3,403 1.323GB

3:00PM1/23/2008

eDiscovery

Datum v. Contoso – Windows Internet Explorer

Favorites | Website 1 Website 2 Website 3

http://portals/sites/ediscovery/contoso Search

HomeIdentify & Preserve Custodians LocationsSearch and Process Searches ExportsReportsCase Settings

Case Owners

Status

Preservation Settings

Tom Perham;

Active

Preserve all content Select preservation type per

mailbox Filter preserved contentDate Range: toQuery:

Add Additional Filters

Mailboxes

Preserve all content Select preservation type per

location Filter preserved content

Locations

Preview

3:00PM1/23/2008

eDiscovery

Document Name – Windows Internet Explorer

Favorites | Website 1 Website 2 Website 3

http://mysite/ediscovery Search

Case Manager HomeCasesCollections

Create a New Case Datum v. ContosoName:

Tom PerhamCase Owner:

CancelSave

Collections

Add Collection

None

Collections

CancelSave

Name Date Created Managed By On Hold TypeWidget F Files 9/26/2010 Tom Perham Yes Mailbox; SharePoint

Steve Den 3/7/2010 Tom Perham Yes Mailbox

Widget D 2/4/2010 Tom Perham No SharePointCreative Files 6/2/2009 Luka Abrus Yes File Share;

SharePointController Files 3/6/2009 Tom Perham Yes Search Results

Active Collections | Closed Collections | Create New Collection

Add Collection

Filter:

Preview

Date Range: to

Preserve

Add Sources

Sources

Preservation Settings

Preserve this content

Save and Apply ChangesCancel

Update Statistics

Filter (optional)

Add a filter to reduce the content (e.g. date>10/10/2010, images)

Collection Statistics

Source Item Count

Size Cull Rate

Total 0 0 0%

Name: Name your collection

Preservation Type Statistics

Content Type Statistics

Add a collection

Queries

CASE HOMECOLLECTIONSSEARCHESEXPORTSSOURCESDOCUMENTSREPORTS

Filter (optional)

Add a filter to reduce the content (e.g. date>10/10/2010, images)

Name: Name your collection

Date: to:Sources

Name Source Type Quentin Christensen Exchange

Mailbox http://contoso/teams/sales SharePoint Site //back/scratch/quentin File Share

Add sources

Searches

Name Modified Items Size

Add new item

Preservation Type StatisticsContent Type Statistics

Statistics

Source Items Size

Total

Preservation Settings Preserve this content Collect content

Save CancelUpdate Statistics

Preview Results

HOMECOLLECTIONSQUERIESSOURCESEXPORTSDOCUMENTS

Apply Filter

Save CancelPreview Results

Name* Name your collection

Sources

Source Source Type Preservation Status Items Size

TotalManage sources

Filter

Add a keyword to filter content

Start Date: End Date:Keywords:

Preservation

Certain repositories such as SharePoint and Exchange support in place preservation. Selecting preserve this content will protect it in its original location.

Preserve

Author:

There are no sources in this collection. To add sources, click “Manage Sources”.

Initial Design

Initial Build

Redesign

Beta1

Shipping Activities• UX/UI/String Reviews• Polish SWAT• Real world projects• Scenario testing & health• White papers• Early customer deployment• Dogfood

UX/UI/String Reviews• Get team members, UX, design in a room• Walkthrough the experience• Testers write down bugs and file later

• UX/UI – focus on the overall experience, flow, interactions, look and feel. Get into the details on spacing, color, font, etc.

• String – Focus on the text that is displayed to users, be sure to cover all error cases• Get a list of all text in areas you own• Try to view it in context, but be sure to review everything• Get recommendations on how to change the string

Polish SWAT

No, this Polish SWAT1. Identify page/area/experience to polish2. File one bug to track the SWAT3. Identify existing bugs and track them4. Do a UI review, identify all issues5. Track issues in a shared OneNote6. Review with your team7. Create a design with all changes 8. Fix all issues9. Dev sends link to changes10. Review in person, identify remaining

issues, fix, repeat11. Your done:

1. Everyone agrees its ready2. Polished – no remaining UI bugs

Members:• Dev / Test• UX/Design• SMEs

Polish SWAT Tips• Double check CSS – make it consistent (IE F12 Developer

Tools)• Get it right - but don’t stress UI and CSS you don’t control, go

for consistency not the way you want it to look• Meet – with the developer, make real time changes• Focus – pick one page or task• Schedule a week – the SWAT is a top priority, meet ad hoc

not on a schedule • Agility – move to each step as fast as possible, if the

developer completes the update meet immediately Agreement – its done when you all agree its done

• Its Done – the page looks ready for customers (aside from any issues you don’t control)

Example: Microsoft eDiscoveryBefore SWAT

After SWAT23 initial issues identified10 new issues identified during iterationLooks way better after 1 week

Spec

Real World Projects• Build real world applications using the latest build• Build a real thing, use real data:• Team up – get 3-4 people• Dedicate a week – it should be the majority of your time for

that week• File as many bugs as possible! 50-100 for a team or even

one person is doable• Can you complete the scenario?

Example: Microsoft Sales Portal• Rebuild the Microsoft Sales Portal site on the new version

of the product (Alpha Build)• 1 week, 4 people full time,112 bugs filed

Scenario Testing & Health• Identify core tasks for your scenario• Conduct regular test passes• Rate each task

• Green – no issues or very minor issues• Yellow – minor visual issues, not great

experience• Red – major visual or functional issues• Black – cannot complete task

• Mark bad bugs scenario blocking• Track usability & experience issues

ExampleID Assigned To Sub Status Title290742 ramansom Paging doesnt work for exchange sources319499 boxinli Bug 

UnderstoodWhen an update panel refreshes search no longer works

325512 carloa No stats on the saved searches list view; need Items and Size328714 srivr Auth: The realm in the token is hardcoded to 'microsoft.com'333406 shiyao Sum of # of documents displayed in refiners is greater than the total 

number of documents returned in the query results (for very simple result sets)

350069 boxinli Need a way to cancel a query which has been issued to the SDP360106 carloa Case Manager: When failing to talk to search, the seaching window in the 

query edit form keeps searching forever

361198 carloa Creating a new Query should search all sources in the case with a blank filter

373183 ramansom Search fail to get token when searching using OAuth376625 grahamc Local Fix Exchange SourceFilter includes Path property; Exchange exports don't 

work

2295591

sveinar Local Fix Thread.CurrentPrincipal in the flow is not correct

2314397

carloa The hover panel is gone; Object required in searchui.js line 1 col 9342

2342688

carloa Refiners fall back to non Discovery refiners if you specify a specific collection as the query scope

2345525

carloa Loading advanced query options dialog gives a 404

2354380

carloa Automation Failure: Search returned results from the SharePoint Source [ACTUAL: 0, EXPECTED: Not 0] (Method:

2358810

jessals Investigating We need a way to resolve a web title from search results

2376343

carloa Query for the top-level of the query fragment web part doesn't include hiddenconstraints

2376391

carloa Expanding a query fragment in the query fragment web part first queries with HiddenConstraints and then without

2386198

grahamc Local Fix Creating an export fails; The given key was not present in the dictionary. at Microsoft.Office.Server.Discovery.Export.GetExportContent()

2387170

grahamc Local Fix SourceFilter in export.xml for an export includes properties that shouldn't have values; export fails

Scenario Health Scenario Blocking Bugs

Status Scenario Task  Create a case Create a discovery set Add a mailbox Add a SharePoint site Get the number of items  Enter a filter and view the stats  Preserve and save the discovery set  Preservation works in Exchange Preservation works in SharePoint View auditing report of when content was preserved Create a  query  Specify a query with a wildcard and proximity Narrow down the query scope View results from Exchange, SharePoint, and file shares  Show the hovercard with a preview of the message Open the message  Select multiple refinements for SharePoint View the hovercard for SharePoint items  Save the query  Export the query

White Papers• A big doc of how to do “it”• Guidance, config, setup, best

practices• Write a draft early (before

Beta1)• Give to early customers• Get feedback – hard to

setup? Are the requirements reasonable?

• Add to it when you make a decision that customers should know about

Early Customer Deployments• Beta users• Adoption Program

• High touch:• Weekly synch ups• Prototype testing – Beta 1• Go live – Beta2 – RTM

• Ongoing feedback• Bug finding• Lots of support from engineering• Prove features and fix gaps

Dogfood• Internal use of your features

before official release• Important source of validation and

bug finding• Every product feature should have

a dogfood plan

Secret Guerilla Tactics & Tips• If you don’t have at least one user to talk to – get one• At least do one test with 3 people before release!• Don’t lead users - give tasks, observe, and take notes• Ask users to think aloud• Know your user, you are not the user• When giving tasks – don’t use the phrases in the UI• Don’t take everything you hear at face value• Don’t be afraid to show early designs to real customers• Do projects to validate your features• Use real world data and examples• Measure your health and readiness to ship