Design Critiquing Design Critiquing SystemsSystems

Jason E. Robbins

Information and Computer Science Dept. University of California at Irvine

2

OverviewOverview• IntroductionIntroduction

– What are Design Critics?What are Design Critics?– Why Use Design Critics?Why Use Design Critics?

• The ADAIR Design Critiquing ProcessThe ADAIR Design Critiquing Process

• Context: Context: Intelligent User InterfacesIntelligent User Interfaces

• Focus: Focus: Design Critiquing SystemsDesign Critiquing Systems

• ConclusionsConclusions– State of the ArtState of the Art– New Research DirectionsNew Research Directions

3

What are Design Critics?What are Design Critics?• A A design criticdesign critic analyzes a design and provides analyzes a design and provides

feedback to help the designer improve the feedback to help the designer improve the designdesign..– Critics are a type of Intelligent User Interface (IUI). IUI’s Critics are a type of Intelligent User Interface (IUI). IUI’s

are user interfaces that use “intelligence” to adapt to are user interfaces that use “intelligence” to adapt to the user. More on this later.the user. More on this later.

– Critics should be embedded in design tools so that they Critics should be embedded in design tools so that they act while the designer is in the mental context of act while the designer is in the mental context of decision making. decision making.

– Improvement opportunities are errors, omissions, Improvement opportunities are errors, omissions, violations of guidelines, sub-optimal choices, awkward violations of guidelines, sub-optimal choices, awkward use of notation, etc.use of notation, etc.

– Design improvements are changes to the design artifact, Design improvements are changes to the design artifact, goals, or the designer’s understanding of the design.goals, or the designer’s understanding of the design.

4

Why Use Critiquing?Why Use Critiquing?

• Three main reasons given in literature:Three main reasons given in literature:

– Acceptable user interface for expert systems Acceptable user interface for expert systems e.g., ONCOCIN, ATTENDING, TraumaTIQe.g., ONCOCIN, ATTENDING, TraumaTIQ

– Avoidance of specific errorsAvoidance of specific errorse.g., CLEER, SEDARe.g., CLEER, SEDAR

– Support for designers’ cognitive needsSupport for designers’ cognitive needse.g., Janus, Argoe.g., Janus, Argo

5

Argo/UML ExampleArgo/UML Example

6

Our Definition of Design Our Definition of Design CriticsCritics• A A design criticdesign critic is an intelligent user is an intelligent user

interface mechanism embedded in a interface mechanism embedded in a design tool that analyzes a design in the design tool that analyzes a design in the context of decision-making and provides context of decision-making and provides feedback to help the designer improve feedback to help the designer improve the design.the design.

• Things to noticeThings to notice– Critics are a type of IUICritics are a type of IUI– Critics should be embedded in design toolsCritics should be embedded in design tools– Purpose of feedback is primarily to prompt and Purpose of feedback is primarily to prompt and

guide improvementsguide improvements

7

Why Use a Critiquing Why Use a Critiquing Process?Process?

• Critiquing systems are interactive systemsCritiquing systems are interactive systems

• Evaluation criteria must be grouped. There Evaluation criteria must be grouped. There is a temporal ordering between the groups.is a temporal ordering between the groups.– Helps explain the criteria groupsHelps explain the criteria groups– Helps in imaging how new support features Helps in imaging how new support features

could be added to each systemcould be added to each system

• Several critiquing papers use process Several critiquing papers use process models to describe individual systemsmodels to describe individual systems– Making a more generic process model helps to Making a more generic process model helps to

summarize previous thinking on how designers summarize previous thinking on how designers should interact with criticsshould interact with critics

8

Previous Work on Critiquing Previous Work on Critiquing ProcessesProcesses

Designer Critiquing SystemGenerates Design Observes

Analyzes Design Analyzes Design

Experiences Breakdowns and Reflects on Critics

Signals PotentialProblems

May Request Rationale

Delivers RationaleReflects on Rationale

Modifies Design or Rationale

[Sumner,... 1997]

9

The ADAIR Critiquing The ADAIR Critiquing ProcessProcess

• This process describes the activity of This process describes the activity of critics and designers at a high levelcritics and designers at a high level

• Phases may be active concurrently and Phases may be active concurrently and multiple instances of the process can multiple instances of the process can exist in multiple stages of completionexist in multiple stages of completion

• The approaches and systems reviewed The approaches and systems reviewed are evaluated in terms of their support for are evaluated in terms of their support for ADAIRADAIR

Activate Detect Advise Improve Record

10

The ADAIR Critiquing The ADAIR Critiquing ProcessProcess

Activate Detect Advise Improve Record

Selects timely and relevant critics from all in knowledge base

• User models • Goal models • Task models

Increasing support increases relevance of feedback to the designer

Each active critics detects assistance opportunities

• Comparative or • Analytic

Increasing support increases the amount of knowledge available to the designer

System presents feedback in a useful and usable form

• Dialog boxes • Message logs • Visual indicators • “To do” list

Increasing support makes feedback more useful and manageable

Designer carries out suggested design improvements

• Instructive text• Automation • Task specific UI

Increasing support reduces knowledge and effort needed to make improvements

System records resolution of criticism

• User models • Argumentation • Design history

Increasing support increases design context, helps maintain critics

What

How

Why

11

-Phase One: Activate-Phase One: Activate

• An appropriate subset of all available An appropriate subset of all available critics is selected for activation. critics is selected for activation.

• Critics that are relevant and timely to the Critics that are relevant and timely to the designer’s current decisions should be designer’s current decisions should be activated so as to support those decisions. activated so as to support those decisions.

• Increasing support for activation tends to Increasing support for activation tends to make the advice provided by the system make the advice provided by the system more useful to designers and reduces the more useful to designers and reduces the amount of feedback presented that is not amount of feedback presented that is not useful.useful.

Activate Detect Advise Improve Record

12

-Phase Two: Detect-Phase Two: Detect

• Active critics detect assistance Active critics detect assistance opportunities and generate feedback opportunities and generate feedback items. items.

• The most common type of assistance The most common type of assistance opportunity is the identification of a opportunity is the identification of a syntactic or simple semantic error.syntactic or simple semantic error.

• Increasing support for detection by Increasing support for detection by adding new types of detection adding new types of detection mechanisms broadens the range of mechanisms broadens the range of advice that the system can offer.advice that the system can offer.

Activate Detect Advise Improve Record

13

-Phase Three: Advise-Phase Three: Advise

• Design feedback items are presented to Design feedback items are presented to advise the designer of the problem and advise the designer of the problem and possible improvements. possible improvements.

• Feedback may take the form of messages in Feedback may take the form of messages in a dialog box or feedback pane, or it may be a dialog box or feedback pane, or it may be a visual indication in the design document a visual indication in the design document itself (e.g., a wavy, red underline). itself (e.g., a wavy, red underline).

• Increasing support for the presentation of Increasing support for the presentation of advice helps designers make more effective advice helps designers make more effective use of the feedback that they are given. use of the feedback that they are given.

Activate Detect Advise Improve Record

14

-Phase Four: Improve-Phase Four: Improve

• If the designer agrees that a change is If the designer agrees that a change is prudent, he or she makes changes to improve prudent, he or she makes changes to improve the design and resolve identified problems. the design and resolve identified problems.

• Tools can provide suggestions for Tools can provide suggestions for improvements or corrective automations that improvements or corrective automations that fix the identified problem fix the identified problem (semi-)automatically. (semi-)automatically.

• Increasing support for the Improve phase Increasing support for the Improve phase reduces the procedural knowledge and effort reduces the procedural knowledge and effort needed and increases the likelihood that needed and increases the likelihood that designers will follow through on suggestions.designers will follow through on suggestions.

Activate Detect Advise Improve Record

15

-Phase Five: Record-Phase Five: Record

• The resolution of each feedback item is The resolution of each feedback item is recorded to inform future decisions. recorded to inform future decisions. – Important because design decisions interactImportant because design decisions interact– Critics prompt designers to explain their decisions Critics prompt designers to explain their decisions

• Increasing support for the Record phase Increasing support for the Record phase provides designers with more design provides designers with more design context and improves the usefulness of context and improves the usefulness of feedback over time. feedback over time.

• Furthermore, recording outcomes and Furthermore, recording outcomes and collecting metrics is key to effective collecting metrics is key to effective maintenance of critiquing systems.maintenance of critiquing systems.

Activate Detect Advise Improve Record

16

Scope of this SurveyScope of this SurveyContext: IUI ApproachesContext: IUI Approaches

TutorsTutors

CoachesCoachesUI AgentsUI Agents

WordWordCorrectorsCorrectors

WizardsWizards

FocusFocus: : CriticsCritics

Each dot is a system reviewedEach dot is a system reviewed

17

-Tutoring Systems-Tutoring Systems

• Tutoring systems are used in classrooms Tutoring systems are used in classrooms by students trying to master the material by students trying to master the material in a lesson. in a lesson. – Students solve problems that test their Students solve problems that test their

understanding of the material. understanding of the material.

– The tutor tracks the student’s progress, The tutor tracks the student’s progress, provides explanations, and selects new provides explanations, and selects new problems.problems.

A A I RD

18

-Critiquing Systems-Critiquing Systems

• Critiquing systems are design tools Critiquing systems are design tools intended for use by professional intended for use by professional designers in their normal practice, designers in their normal practice, however they may also be useful in however they may also be useful in classroom or “on the job” training. classroom or “on the job” training. – A typical session with a critiquing system A typical session with a critiquing system

consists primarily of the designer working with consists primarily of the designer working with a design tool to construct or modify a design. a design tool to construct or modify a design.

– Only when specific improvement Only when specific improvement opportunities, for example design errors, are opportunities, for example design errors, are detected do critics provide feedback.detected do critics provide feedback.

A A I RD

19

-Coaching Systems-Coaching Systems

• Coaching systems assist users by watching Coaching systems assist users by watching their actions and suggesting help topics. their actions and suggesting help topics. Coaching systems can be deployed in Coaching systems can be deployed in desktop applications and used by a broad desktop applications and used by a broad class of users. class of users. – Users proceed to use the tool as normal, until a Users proceed to use the tool as normal, until a

the system detects a specific assistance the system detects a specific assistance opportunity, for example misuse of one of the opportunity, for example misuse of one of the application’s features. application’s features.

– At that time the coach suggests a better way to At that time the coach suggests a better way to use the tool.use the tool.

• This type of IUI could complement criticsThis type of IUI could complement critics– Most critics do not provide tool-use knowledgeMost critics do not provide tool-use knowledge

A A I RD

20

-Wizards-Wizards

• Wizards are task-specific user interfaces that Wizards are task-specific user interfaces that contain procedural knowledge used to guide contain procedural knowledge used to guide users through a series of steps or decisions.users through a series of steps or decisions.– Each step provides some explanation of what infor-Each step provides some explanation of what infor-

mation is being requested or what action is to be mation is being requested or what action is to be donedone

– Branching decision logic can be encoded in the Branching decision logic can be encoded in the wizardwizard

– Main strength is in automationMain strength is in automation

• This type of IUI could complement criticsThis type of IUI could complement critics– Most critics do not provide procedural knowledgeMost critics do not provide procedural knowledge

A A I RD

21

-Word Correcting Systems-Word Correcting Systems

• We consider two types of intelligent user We consider two types of intelligent user interface for automatic word correction: interface for automatic word correction: dynamic revision and spell checking. dynamic revision and spell checking. – E.g., “teh” becomes “the”E.g., “teh” becomes “the”

• Dictionary look-up certainly does not require Dictionary look-up certainly does not require intelligence, however the best spelling intelligence, however the best spelling checkers consider common causes of checkers consider common causes of spelling errors, knowledge of word spelling errors, knowledge of word frequency in business writing, and the frequency in business writing, and the relationship of the suspect word to the relationship of the suspect word to the surrounding text.surrounding text.

A A I RD

22

-User Interface Agents-User Interface Agents

• User interface agents assist users primarily User interface agents assist users primarily by continuously retrieving or filtering by continuously retrieving or filtering information. information. – For example, interface agents can categorize For example, interface agents can categorize

and prioritize one’s email, filter news streams, and prioritize one’s email, filter news streams, or recommend entertainment [16, 17, 18]. or recommend entertainment [16, 17, 18].

• Agents are directable: the user specifies a Agents are directable: the user specifies a set of goals for the agent, and the agent set of goals for the agent, and the agent tries to fulfill those goals over a period of tries to fulfill those goals over a period of time, with a minimum of further interaction time, with a minimum of further interaction with the user. with the user.

A A I RD

23

IUI Cross-Pollination IUI Cross-Pollination ADAIR Critiquing Process Phase

IUI Approach Activate Detect Advise Improve Record

Tutoring Systems

Critiquing Systems

Coaching Systems

Wizards

Word Correctors

UI Agents

24

Focus: Critiquing SystemsFocus: Critiquing Systems

• Systems reviewed in this survey:Systems reviewed in this survey:• ONCOCIN ONCOCIN • TraumaTIQ• TraumaTIQ• ATTENDING FamilyATTENDING Family • AIDA• AIDA• Janus FamilyJanus Family • UIDA• UIDA• FramerFramer • SEDAR• SEDAR• KRI/AGKRI/AG • Argo Family• Argo Family• CLEERCLEER • ICADS• ICADS• VDDEVDDE

25

ONCOCINONCOCIN

• Advises doctors on cancer treatment Advises doctors on cancer treatment plansplans

• Originally implemented as an expert Originally implemented as an expert system, then converted to the critiquing system, then converted to the critiquing approachapproach

• Emphasis on explanationEmphasis on explanation– Used user solution to guide explanation of Used user solution to guide explanation of

system solutionsystem solution

– Natural language text generationNatural language text generation

Comp. A A I RD

26

ATTENDING FamilyATTENDING Family

• Advises doctors in anesthesia, Advises doctors in anesthesia, hypertension, or jaundicehypertension, or jaundice

• Seen as alternative to expert system user Seen as alternative to expert system user interfaceinterface

• Comparative critiquing using well-known Comparative critiquing using well-known solutions, and some analytic rulessolutions, and some analytic rules

• Natural language text generationNatural language text generation

Both A A I RD

27

Janus FamilyJanus Family

• Kitchen design environments: Crack, Janus, Hydra, KIDKitchen design environments: Crack, Janus, Hydra, KID

• Critics embedded in design tool, supply brief Critics embedded in design tool, supply brief descriptions of problems and links into argumentative descriptions of problems and links into argumentative hypermediahypermedia

• Specification sheets prompt designers to specify goalsSpecification sheets prompt designers to specify goals

• Hydra supports groups of critics called Hydra supports groups of critics called critiquing critiquing perspectivesperspectives

• Research motivated by exploration of cognitive Research motivated by exploration of cognitive theories, namely theories, namely reflection-in-actionreflection-in-action

Analytic A A I RD

28

FramerFramer

• Helps with user interface window layout Helps with user interface window layout (CLIM)(CLIM)

• Includes a checklist process modelIncludes a checklist process model

• Activates critics appropriate to current Activates critics appropriate to current itemitem

• Brief error messages, phrased Brief error messages, phrased constructivelyconstructively

• Offers to automatically fix some identified Offers to automatically fix some identified problemsproblems

Analytic A A I RD

29

-KRI/AG-KRI/AG

• Checks user interfaces for conformance Checks user interfaces for conformance to published UI style guidelinesto published UI style guidelines

• Batch system, not embedded in a design Batch system, not embedded in a design tooltool

• Feedback includes brief descriptions of Feedback includes brief descriptions of the problem and references to published the problem and references to published guidelinesguidelines

Analytic A A I RD

30

-CLEER-CLEER

• CAD system for antenna on military shipsCAD system for antenna on military ships

• Critics check for mechanical and Critics check for mechanical and electromagnetic interference electromagnetic interference

• Offers brief, negative feedback when Offers brief, negative feedback when users press the “Evaluate” buttonusers press the “Evaluate” button

Analytic A A I RD

31

-VDDE-VDDE

• Voice Dialog Design Environment helps Voice Dialog Design Environment helps designers make telephone menu systemsdesigners make telephone menu systems

• Critics check for inconsistency within a Critics check for inconsistency within a design or between a design and standard design or between a design and standard design guidelinesdesign guidelines

• Multiple sets of critics are explicitly Multiple sets of critics are explicitly activated by the useractivated by the user

• Brief error messages with links to Brief error messages with links to hypertext argumentshypertext arguments

Analytic A A I RD

32

TraumaTIQTraumaTIQ

• Checks ER doctor’s treatment plansChecks ER doctor’s treatment plans

• Criticism must be informative and briefCriticism must be informative and brief

• Infers doctor’s goals from proposed planInfers doctor’s goals from proposed plan

• Only substantial differences between the Only substantial differences between the system’s plan and the doctor’s plan system’s plan and the doctor’s plan generate feedbackgenerate feedback

• Can combine one or more feedback items Can combine one or more feedback items into brief natural language outputinto brief natural language output

Comp. A A I RD

33

-AIDA-AIDA

• Used by hospital laboratory technicians to Used by hospital laboratory technicians to identify antibodies in blood samples to be identify antibodies in blood samples to be used in transfusionsused in transfusions

• The system can generate its own The system can generate its own solution, but has limited competencesolution, but has limited competence

• Critics detect errors and differences at Critics detect errors and differences at several distinct points in the taskseveral distinct points in the task– after any changeafter any change– when the technician changes screenswhen the technician changes screens– when the technician declares that the task is when the technician declares that the task is

donedone

Comp. A A I RD

34

-UIDA-UIDA

• Command-line tool to check user Command-line tool to check user interface window layouts for interface window layouts for inconsistency with style rules or each inconsistency with style rules or each otherother

• Tool prompts user to confirm repairsTool prompts user to confirm repairs– Not much information in feedbackNot much information in feedback

– Automates some design improvementsAutomates some design improvements

Both A A I RD

35

SEDARSEDAR

• Critiquing system for roof design in Critiquing system for roof design in AutoCADAutoCAD

• Checks for mechanical interference in Checks for mechanical interference in placement of drains, walkways, A/C units, placement of drains, walkways, A/C units, etc.etc.

• Includes a task model and a goal modelIncludes a task model and a goal model– Uses them to activate critics when appropriateUses them to activate critics when appropriate

• Feedback can visually suggest placing a Feedback can visually suggest placing a new element in the design diagramnew element in the design diagram

Analytic A A I RD

36

Argo FamilyArgo Family

• Three systems for software architecture, Three systems for software architecture, state-based requirements, and object-state-based requirements, and object-oriented designoriented design

• Wide range of criticsWide range of critics

• Task and goal models and used to control Task and goal models and used to control criticscritics

• ““To do” list feedback managementTo do” list feedback management

• Some support for automated improvementSome support for automated improvement

• Some support for recording design historySome support for recording design history

Analytic A A I RD

37

-ICADS-ICADS

• Batch system for critiquing CAD models Batch system for critiquing CAD models of residential apartmentsof residential apartments

• One critiquing module checks building One critiquing module checks building codes, a second module uses heuristic codes, a second module uses heuristic rules about style and comfortrules about style and comfort

• Heuristics are only applied after all Heuristics are only applied after all building codes have been satisfiedbuilding codes have been satisfied

• Feedback includes suggestions for Feedback includes suggestions for improvementimprovement

Analytic A A I RD

38

Critiquing System Critiquing System SummarySummary

Phases of the ADAIR Critiquing ProcessSystem

Activate Detect Advise Improve Record

ONCOCIN ComparativeATTENDING Both Janus Analytic Framer Analytic KRI/AG Analytic CLEER Analytic VDDE Analytic TraumaTIQ Comparative AIDA Both UIDA Both SEDAR Analytic Argo Analytic ICADS Analytic

39

-Rubric-Rubric• ActivateActivate

– Having any strategyHaving any strategy – Multiple strategies– Multiple strategies– Process modelProcess model – Goal model– Goal model

• Detect: Comparative or Analytic or BothDetect: Comparative or Analytic or Both• AdviseAdvise

– ExplanationsExplanations – Browsing– Browsing– Visual indicationsVisual indications – Design context– Design context– Natural language textNatural language text

• ImproveImprove– step-wise instructionsstep-wise instructions – Visual cues– Visual cues– Corrective automationsCorrective automations

• RecordRecord– Capture rationalCapture rational – Record critique – Record critique

resolutionsresolutions– Update user models, goal models, task modelsUpdate user models, goal models, task models

40

State of the Art in State of the Art in Critiquing SystemsCritiquing Systems• All of the reviewed critiquing systems All of the reviewed critiquing systems

were research prototypeswere research prototypes– Each addresses some aspects and not othersEach addresses some aspects and not others– Generally few critics, limited knowledgeGenerally few critics, limited knowledge– Very few usersVery few users– Little emphasis on software engineering issuesLittle emphasis on software engineering issues

• In comparison, some of the other In comparison, some of the other intelligent user interface approaches intelligent user interface approaches played a role in widely used commercial played a role in widely used commercial productsproducts

41

Challenges & New Challenges & New DirectionsDirections

1. Combine With Other IUI Approaches1. Combine With Other IUI Approaches– Critics+Coaches, Critics+WizardsCritics+Coaches, Critics+Wizards

2. Organization-Specific Critics2. Organization-Specific Critics– Disseminate company policies, stakeholder concernsDisseminate company policies, stakeholder concerns

3. Reusable Critiquing Infrastructure3. Reusable Critiquing Infrastructure– Stop re-inventing basics, start exploring finer pointsStop re-inventing basics, start exploring finer points

4. Critic Development Life-cycle4. Critic Development Life-cycle– Where do critics come from? How are they maintained?Where do critics come from? How are they maintained?

5. Critic Implementation Language5. Critic Implementation Language– Useable by practicing designers; who does work vs. Useable by practicing designers; who does work vs. benefitsbenefits

6. Limited Role for Machine Learning6. Limited Role for Machine Learning– Sort and prioritize feedback, programming by Sort and prioritize feedback, programming by

demonstrationdemonstration

42

-New Direction:-New Direction:1.1. Combine With Other IUI Combine With Other IUI ApproachesApproaches• Critics + CoachesCritics + Coaches: Provide advice about : Provide advice about

design decisions and design tool usagedesign decisions and design tool usage

• Critics + WizardsCritics + Wizards: Provide knowledge to : Provide knowledge to identify errors, explain them, and identify errors, explain them, and procedural knowledge to (semi-) procedural knowledge to (semi-) automatically correct themautomatically correct them

43

-New Direction:-New Direction:2. Organization-Specific Critics2. Organization-Specific Critics

• Engineering involves both natural laws Engineering involves both natural laws and human interactions, often human and human interactions, often human interactions dominateinteractions dominate

• Many organizations disseminate policy Many organizations disseminate policy information using memos, meetings, and information using memos, meetings, and procedure manuals, these are out of the procedure manuals, these are out of the context of particular design decisionscontext of particular design decisions

• Design critics may be a better way to Design critics may be a better way to disseminate organizational design policiesdisseminate organizational design policies

• SEDAR developers also thought of thisSEDAR developers also thought of this

44

-New Direction:-New Direction:3. Reusable Critiquing Infrastructure3. Reusable Critiquing Infrastructure

• Most of the reviewed critiquing systems Most of the reviewed critiquing systems were implemented from scratch: high were implemented from scratch: high development effort, forces small scopedevelopment effort, forces small scope

• Each system gives incomplete supportEach system gives incomplete support

• A scalable, reusable critiquing A scalable, reusable critiquing infrastructure based on ADAIR would infrastructure based on ADAIR would reduce effort needed to build a new reduce effort needed to build a new system, allow larger scopes, and provide system, allow larger scopes, and provide more complete supportmore complete support

45

-New Direction:-New Direction:4. Critic Development Life-cycle4. Critic Development Life-cycle

• Most systems reviewed have been prototypes that Most systems reviewed have been prototypes that never reached the maintenance phasenever reached the maintenance phase

• Sustained usage and maintenance of critiquing Sustained usage and maintenance of critiquing systems has not been explored much (one systems has not been explored much (one exception is Fischer et al. “Seeding” paper)exception is Fischer et al. “Seeding” paper)

• Ideas for critics will arise and be refined in useIdeas for critics will arise and be refined in use

• Effectiveness of individual critics should be Effectiveness of individual critics should be monitoredmonitored

• Argo allows designers to send email to critic Argo allows designers to send email to critic maintainersmaintainers

• Possible interactions with design review meeting Possible interactions with design review meeting procedures and organizational memories procedures and organizational memories

46

-New Direction:-New Direction:5. Critic Implementation Language5. Critic Implementation Language

• Many of the reviewed systems use standard Many of the reviewed systems use standard languages (OPS5, PROLOG, Lisp, Java) to languages (OPS5, PROLOG, Lisp, Java) to implement critics, others use their own critic implement critics, others use their own critic language (e.g., KRI/AG)language (e.g., KRI/AG)

• None of the reviewed systems proposes a None of the reviewed systems proposes a standard, reusable critiquing languagestandard, reusable critiquing language

• Argo’s reusable class framework for critiquingArgo’s reusable class framework for critiquing

• We emphasize ability to express corrective We emphasize ability to express corrective actions, readability by designers, and ease of actions, readability by designers, and ease of incremental changesincremental changes

47

-New Direction:-New Direction:6. Limited Role for Machine Learning6. Limited Role for Machine Learning

• Machine learning probably should not play a Machine learning probably should not play a central role in the near future, the problem is central role in the near future, the problem is dissemination of well-known design knowledgedissemination of well-known design knowledge

• Machine learning could play a minor role in Machine learning could play a minor role in making the critiquing approach more effectivemaking the critiquing approach more effective– Sorting and prioritizing feedbackSorting and prioritizing feedback

– Choosing to automatically apply defined corrective Choosing to automatically apply defined corrective actionsactions

– Programming by demonstration author new Programming by demonstration author new corrective actionscorrective actions

48

ConclusionsConclusions• A design critics analyzes a design and helps A design critics analyzes a design and helps

a designer make improvementsa designer make improvements

• The ADAIR process extends previous work The ADAIR process extends previous work on critiquing process models and serves as on critiquing process models and serves as a feature checklist and framework for a feature checklist and framework for comparisoncomparison

• Research on critiquing has been done for 15 Research on critiquing has been done for 15 years, but the field has not yet maturedyears, but the field has not yet matured– No reusable infrastructure or S.E. methodologiesNo reusable infrastructure or S.E. methodologies– No ownership by domain practitioners No ownership by domain practitioners – Weak community ties, no shared forumWeak community ties, no shared forum

49

-State of Critiquing -State of Critiquing ResearchResearch• The critiquing community is not cohesiveThe critiquing community is not cohesive

– Most researchers are mainly doing something elseMost researchers are mainly doing something else• Medicine, engineering, cognitive psychology, CHI, AI, Medicine, engineering, cognitive psychology, CHI, AI,

management, etc.management, etc.

– They share only surface-level conceptsThey share only surface-level concepts– Many researchers have entered and exited this fieldMany researchers have entered and exited this field

• Many student projectsMany student projects• Only two have really made a career of itOnly two have really made a career of it

– There is no central forumThere is no central forum– Researchers in this field don’t cite each other muchResearchers in this field don’t cite each other much– Doing a citation graph really clarified thisDoing a citation graph really clarified this– Solution: www.critiquing.org ?Solution: www.critiquing.org ?