Download pdf - Design and the Analytic Method: The Future of Human Computer Interaction (HCI)

8/14/2019 Design and the Analytic Method: The Future of Human Computer Interaction (HCI)

1/12

Design and the Analytic Method: The Future of HumanComputer Interaction (HCI)

Laurian C. Hobby

Center for HCI, Computer ScienceVirginia Tech660 McBryde

Blacksburg, Virginia [email protected]

Abstract. When designing or choosing an analytic method, the designer shouldchoose a method based on the criteria of having a good cost benefit trade-off,ease of use, and reproducibility of results. Even these criteria can lead a designmethod astray if it does not have the right focus of attention. HCIs future indesign depends on shifting the focus away from simply the user, to add work context which incorporates the user and the work being done.Technomethodology is only one of many possible design methods that attemptsto meet these criteria and has the proper focus. This paper explores these issuesand how a merging of social and tangible computing can create a superior

analytic design methodology that will guide HCI into the future of design.

1 Introduction

The world and its systems are complex. To understand them correctly requires theinquisitive mind to analyze their structure and content in detail. To be analytic is tothink in terms of elemental parts or basic principles [10], or to be a logician of sorts. However, the analytical process is much more than that - it is to come to theright conclusion by breaking the problem into manageable parts in a structured andmethodical way. To further understand how to think analytically, it is beneficial tothink of its counterpart, how to think empirically, and what the two say about eachother. Empiricism, to define it, is the use of metrics and measures to accept or reject ahypothesis. In comparing the two approaches, it may seem that an empirical method issterile. This is because the rules of analytic methods can often be ill-defined, whereasthe rules of empirical methods are more likely rigorous and sturdy. A typical examplewould be to think of the evaluation of software: to use a checklist of neededcharacteristics would be an analytic method, whereas to take a measurement of how


2/12

2 Laurian C. Hobby

many people were able to complete a task with a piece of software would be anempirical method. There are benefits and costs of using an analytic method for design,therefore, the process used to select and develop an analytic method should becarefully selected. How this method compares to general design methods, and how itcan help to initiate or inspire formal design methods, should also be explored. The restof this paper will investigate these ideas in depth and come to a set of conclusions.

2 What Makes a Good Analytic Method?

When it comes to the creation of a software system the criteria for which analyticmethod to use is unclear. It is easy to apply general rules to software systems design,such as generalizability, precision, and realism [ 21]; but for selecting an analyticmethod for the assessment, the criteria need s to be more specific. Typical examples of analytic methods to choose from, (although they mostly , if not only, focus on onephase of design , e.g., evaluation, ) are usability inspections [ 23, 24], model-basedanalysis [ 17, 25], and claims analysis [ 32, 7] (for more information on all three see[29]). When to use which method , and what criteria to apply in order to create a goodanalytic method , are still unclear in the current literature. However, t he primarycriteria for what makes a good analytic method are listed and discussed in the nextthree subparagraphs in order to help clarify and ground what is a superior analytic

method.

2.1 Criteri on 1: The Cost Benefit Tradeoff

The cost benefit tradeoff, when viewed at a high level, is similar to asking whether theanalytic method is cost effective , i.e., a re the benefits gained from using this methodequal or better to the cost of using it? The costs and the benefits are computed andcompared . As long as the comparison is in favor of the benefits, then th is analyticmethod may be considered as one that is good. An example of a software designproject where the cost benefit ratio was in favor of performing the analytic method

was the Goals, Operators, Methods, and Selection Rules project ( GOMS ). BonnieJohn and her colleagues worked on a project using the GOMS methodology where inthe acti vity levels of phone operators were desired to be faster /quicker . A newsoftware system was proposed to replace the original system. John was able toaccurately predict that the new computing system was going to cause the phoneoperator to be slower with the proposed system than with the original system . She wasable to do this with a minimal cost of time and effort because she was able to break


3/12

Design and the Analytic Method: The Future of Human Computer Interaction (HCI) 3

down the tasks that the phone operators would be doing with the new system with ashort amount of effort and time [17] . Thus, the costs were low when comparing themto the costs of the creation of bad software. This is an example where the cost of performing the cost benefit analysis was low; however, the benefits of understandingthat the new system would not improve the situation were high . When the cost of applying an analytic method results in significant benefits, then it fits the criteri on forbeing a superior analytic method.

2.2 Criteri on 2: The Ease of Use

Put simply, the method should be easy to use. One of the major criticisms of someanalytic method s are that t hey can produce variable results which are dependent onthe quality /ability of the person (the designer) using them, if they are not easy to use .Analytic methods, in essence, are based or grounded on a set of theoretical constructsthat need to be interpreted by the person during their application . This interpretationand the results thereof , are directly dependent on the expertise and training level of thedesigner. This creates a situation where given the same problem to be solved , andusing the same analytic method, two designers of different expertise levels have a highprobability of generating widely different outcomes if the method is difficult to apply .For example, in claims analysis, designers create claims that stipulate the upsides anddownsides of using a particular interface object such as a menu. The difficulty withusing this method is that expert designers create much better claims than novicedesigners if there is no guided tool support. Specifically, novice designers have a hardtime creating accurate downsides for claims about design artifacts [ 18]. As can beseen from this example, the expertise of the designer can be a factor in design, eventhough similar results should be obtained no matter what the knowledge level.Therefore, the ease of use of the method in producing good results should be similarno matter what the designers background knowledge and level of expertise . Inempirical design, as a comparison, the results of such methods are more repeatableirrespective of the expertise level of the designer. (For more information see [ 29].)This is partially due to the ambiguous nature of analytic method s and the products

produced; empirical methods may take a certain level of expertise for the results to beinterpret ed, but the same interpretation will be arrived at each time. Nevertheless,even though analytic products may be nebulous, the designers level of backgroundexpertise should not be a factor in the success of a method.


4/12

4 Laurian C. Hobby

2.3 Criteri on 3: Reproducible

The last and perhaps the most important criteri on for a superior analytic method is forthe process to produce repeatable results. This is similar to having ecological validity[21] in that the method should be rigorous enough to be used by different people overdifferent periods of time in different settings and to still be able to produce reliablerepeatable results. Although this seems to be the holy grail of design, it is specificallythe most important one for an analytic method because of the high variability withinthe current methods. This may be due to the fact that designers naturally bring in theirown biases and expectations into analytic design. For example, usability analysis isthe analytic method where the operation of a created computer system is tested andproblems with the software (example: the user is not able to find a box to click) arereported and fixed. Rolf Molich, Meghan Ede, Klaus Kaasgaard, and BarbaraKaryunkin showed that evaluating the same software at nine different organizationsresulted in 310 different usability problems being reported; however, only two of thoseproblems were the same [Molich]. Although this example is an extreme case , it showsthe high variability that can exist in using analytic methods , and because of this, ananalytic method that is able to produce re peatable results will be a superior one. (Formore information on variability see [15, 16].)

3 The Evolution of Analytic and Design Methods

In comparison to the centuries of electrical engineering development, softwareengineering is a relatively new science, and Human Computer Interaction (HCI) iseven newer. The question What makes a good software design? has been recognizedas the million-dollar question, and with the rising interest in the Science of Designthere has been a lot of interest and work trying to answer which is the bestmethodology (or possibly methodologies) for designers to select for their task of taking a set of specifications and creating a software design. As with most voyages of discovery there are phases of discovery, each building on the previous work. Therehave been many novel design methods proposed in the past years that meet one or twoof the three criteria discussed in Section 2 above, but very few that meet all three. (Tocite a few: Activity Theory [1], Cognitive Dimensions [3], Usability Engineering [11],and Patterns [4]). There have been many theories that have subsequently becomeaccepted as laws (e.g., Fitts Law [20]) and some that are so fundamental to the studyof HCI that they are no longer looked at as stand-alone study items as they havebecome standard text book material (e.g., Information Theory [27]). This knowledgehas led to the generation of questions such as: What design method is the right one to


5/12


choose? Is the use of an analytic model inherently what makes a design method good?Clearly the random selection of the chosen design method will not be correct becausethe design methodology will likely have the wrong focus. To explore all these issues, alook into the past is the best place to investigate how these questions and distinctionshave arisen and what answers it can lead us to for the future of design an analyticmethod that meets the criteria and has the right focus.

3.1 The History of the Analytic Method and Design

All design methods are based on an analytic method. Although design methods may

have analytic components to them, as seen in the examples below, it is not necessary.For example, the Commercial Off-The-Shelf (COTS) method of design is consideredto be an empirical design method [2], but it should be recognized that the COTSdesign was likely founded on an analytic method. A second example would be toconsider an analytic method such as Pattern Languages [4]. Pattern languages aresimilar to COTS in that the designer can create a design using patterns that have beendefined before; however, to create those patterns in the first place an analysis wouldhave to have been performed to quantify the patterns in existence. Thus, it seems, alldesign methods being employed today, whether empirical, analytical, or a mixture of both, can trace their lineage to one or more analytic methods. It must also berecognized that this traceability does not automatically mean that the designmethod(s) will produce good design results. The focus, or the paradigm, upon whichthe design method was originally established is the main criterion for whether or notthe design method will be successful in todays computing environment.

The first design method to be widely accepted within the computing industry was theWaterfall Model. During this period of design evolution the hardware platform wasconsidered to be augmented by the software. The software merely helped thehardware get its job done. So this model was based upon the idea that the computerhardware was the locus of attention and computation, instead of the human being.Within this method, the designer was guided through the steps of how to gather

requirements, derive requirements, and then construct software in an iterative fashion.Although this method was widely adopted, it later went on to take significantcriticism due to its design limitations and resulted in an eventual paradigm shiftwithin the software engineering community. Nevertheless, the Waterfall Model isimportant as it was the first to incorporate an analytic phase into the design process.In the analysis phase the designer would create a requirements document that wouldspecify the needs and costs of creating a piece of software. Even with theincorporation of an analytic phase however, the indiscriminate use of a method that


6/12

6 Laurian C. Hobby

may have an incorrect focus on where the analysis should be taking place, is one of the main factors that resulted in the failure of this model to be applicable to solve allclasses/types of software design problems.

Hence, a second paradigm shift arose where the focus of the resulting design was onthe human user and not computer hardware. It is generally agreed that this shift wasmostly due to the influx of personal computers into the academic and businessenvironments. This shift resulted in design methodologies such as the Spiral Modeland Evolutionary Prototyping. What these new designs had in common was that theyincorporated the input/output methods of the human users more closely as the

computer systems themselves were designed. There has also been a gradualincorporation of involving the user more and more into the design process.

It is important to note that this paradigm may also have started to fail. It has beenpostulated by Alan Crabtree [8, 9] that the reason for this failure is its reliance onformal methodologies to guide the designer: The problem, then, is that theunderstanding of work generated through the use of formal design methods tend to bevery abstract , focusing more on what should be done rather than on what is actuallyneeded to be done in a particular work setting. Formal methods of design [22] wererevolutionary in their time because there was a need for accurate, shareable, andtraceable information in design teams [Randall]. Here, however, history may be

repeating itself. By not using an analytic method properly, as earlier with the WaterfallModel, the design methods from the second paradigm were, and are still, failing toproduce good repeatable results. Evidence of this can be seen with the increase in newdesign methods, especially in HCI, such as those listed above, an example beingUsability Engineering. Specific methodologies such as participatory design [6] arestarting to become used more frequently in design. This has led many designers, suchas Crabtree, Button, and Dourish [5, 8] to conjecture that we are on the cusp of a thirdparadigm shift within the HCI design field.

In summary, software design methods have shifted their focus from the computerhardware, to the human user, and are still failing to produce good software designsolutions for all types of design problems. The proposed new paradigm will befocused on two changes: namely the work context and secondly a much larger use of an analytic model for design.


7/12


3.2 A Third Paradigm Shift

The beginning of the third paradigm shift started with the publishing of LucySuchmans work on plans and situated action [30]. She proposed a reinvention of theunderstanding of plans and human-problem solving to one that involved situatedaction: i.e., how plans are not executed into actions, but instead, actions are guidedmoment-by-moment responses to the rational accountability of action [31]. Sheshowed how the sterile previous understanding of how plans are actually executed wasunrealistic when compared to the reality of chaotic and sinuous circumstances thatusers of computer systems really work within. With this new understanding, alongwith a change in technology to focus on social and tangible computing, the field of design began to focus more upon process instead of product (to use Andy Crabtreesterms [8]). Other terms used in this third paradigm are work versus system, whereprocess and work are analytic methods for design in which the focus is on the work context (i.e. the user and the work) instead of only on the user. (For more informationon the above see [13]).

Crabtree argues that there are two forms of design: first, the old (product) and secondparadigm methods; and secondly, the new (process) paradigm methods. Crabtree alsoargues that although product cannot be disregarded, it should not be the rulingparadigm for how to design. The product oriented view to design, in essence, is rooted

in the belief that software is a product that stands on its own. Meaning, that thecontext of use is well understood and that all requirements can be elicited before thecreation of the software: The product-oriented perspective then, ignores or isinsensitive to, the interactional dynamics of analysis and design. That is, to the humanprocess of learning and communication through which design solutions are producedin the collaborations and cooperative work of the parties to system development [8].Table 1 highlights some important factors to distinguish the two orientations todesign. Here Crabtree users the term referent system to mean the work setting thatincludes the people and the work being done there.

Table 1. The Product vs. Process-oriented view of how to design for referent systems.

Product-Oriented View Process-Oriented View

1. The referent system is chosen with aview towards developing the softwaresystem.

1. The referent system is chosen with aview towards designing the work processsupported by the software system

2. The referent system is described interms of information processing.

2. The referent system is described interms of work and its processes.


8/12

8 Laurian C. Hobby

3. The software developer is outsidethe referent system.

3. The software developer becomes a partof the referent system in the process of coming to understand the use context.

4. The interaction between softwareand it's environment is predefined aspart of the development.

4. The interaction between software andits environment is tailored by users to theactual needs arising from their work.

The new paradigm emerges with a process-oriented view that focuses on how to work with, and within, the referent system [14]. A product-oriented view cannot bediscarded completely however due to the fact that some level of design decomposition

must occur in order to understand the referent system. This means that an indivisiblecombination of the two should be used, with the process-oriented view being theleading view to the design approach. This call for a new design methodology is onethat seeks to merge the analytic method with work context/referent system [26]; i.e., tomerge technical design with social understandings. Dourish proposes that the mostfruitful place to forge these relationships is at a foundational level, one that attempts totake sociological insights into the process and fabric of design [13]. Wendy Mackayand Anne-Laure Fayard propose a triangulation across disciplines that merges naturaland social science with engineering, design, and fine arts. By using pieces from allthree Mackay and Fayard believe that the best possible design method will be created[19]. It should be noted that this background of social and tangible computing

provided thus far is not enough to guide us successfully with this merging, becauseeven with computing involved in every physical medium, the systems that are createdwill remain reductionist until the design principles for internal representation arechanged. The next section will discuss one proposed design methodology thatattempts to focus on work context and fits the above criteria.

4 Technomethodology

Technomethodology is one attempt to merge the social with the technical. It is basedon the theory of ethnomethodology and resulted from a call to the HCI community toturn towards the area of the social sciences [5]. Ethnomethodology, in references toTechnomethodology, is the method of detailing what actions and interactions occurduring the work process. It is accomplished by doing field work investigations, wherethe designers move into the work setting to understand the organization from theinside, and/or by gaining insight into the organization of activities and interactionswithin the workplace by other means. By incorporating ethnomethodology into thedesign of technological systems, a hybrid design methodology has arisen:


9/12


technomethodology. Technomethodology is able to address Suchmans and Dourishscall for a change in the conceptual model and internal representations within design. Itis able to do this because, as Dourish says, it is an approach that deals not so muchwith this technology or that form of work, but rather more generally with interactivetechnology per se and the generally operative social process that underpins anysociological account of behavior. Thus, by moving away from the rules that guideformal methods, Technomethodology is able to ground itself fully into the work context in two ways: (1) by not relying only on observations from the work setting, butinstead by using ethnomethodologys ability to understand that an organization of action are improvised, spontaneous, and natural; and (2) it relates the understandings

gained in (1) to fundamental principles in design like abstraction, function,substitution, identity, and representation instead of a specific system or setting. [12, 9]Thus, with technomethodology, sociological understanding has been forged into thecore of software design principles. Technomethodology generally meets the criterialisted in Section Two. It scores well in the cost benefit ratio area due to that almostany design emanating from it should have the correct focus, and thus inherently be abetter design. However, since it is still a new analytic method, the users and the resultsof technomethodology have not yet been widely tested. Therefore, it is prematuretoday to predict whether or not it specifically will be easy to use for any level of design. Ethnomethodology, on the other hand, has been fairly successful at being ableto reproduce good results (see examples in [13)] and [5]). As Technomethodology is

founded on ethnomethodology and it is an extension on how to refocus designmethods, it can be easily argued that criteria two and three are currently being met byethnomethodology and can thus be extended to technomethodology. To summarize,Technomethodology is one attempt at merging the social sciences with design byunderstanding moment-by-moment human actions/reactions and by relating thoseunderstandings to the foundational principles of design. Therefore, with the correctfocus to design, the work context, and by meeting the criteria from Section Two,Technomethodology is a good first step at attempting to create a better and superiordesign.

5 Conclusions

The analytic and design methods are the areas of HCI research that are continuing toevolve at a rapid rate. By remodeling design with the criteria proposed in Section Oneand with the focus proposed in Section Two, a whole new era of design is starting toemerge. Technomethodology is one attempt at promoting this evolution, and althoughit has its critics, it provides a strong starting position. The important point of design


10/12

10 Laurian C. Hobby

methodologies is to continuously question what makes a good design and to try andfind the method that matches those criteria. As computing changes and evolves it isimportant to reevaluate what is the focus of the design method. It has evolved from thecomputer, to the user, and hopefully, now to the work context. We should expect moreparadigm shifts as the work is performed. What is needed is the ability to change andto evolve existing methodologies into more useful and usable theories. This has beendone recently by continuously involving the analytic method into the design process tothe point where in some cases the analytic method has become synonymous with thedesign method. The power of the analytic method is its ability to inspire and initiatedesign methods and with the right focus there is no foreseeable end to the science of

design.

References

1. Bertelsen, O. W., Bdker, S.: Activity Theory. In: HCI Models, Theories, and Frameworks.Carroll, J. M. (Eds). Morgan Kaufmann Publishers, San Francisco (2003) 291 324

2. Bianchi, A., Caivano, D., Conradi, R., Jaccheri, L., Torchiano, J., Visaggio, G.: Chapter 13:COTS Products Characterization: Propose and Empirical Assessment. In: EmpiricalMethods and Studies in Software Engineering Experiences from ESERNET Project,Conradi, R., Inge, A. (Eds). Spring Verlag LNCS, (2003) 20

3. Blackwell, A., Green, T.: Notational Systems The Cognitive Dimensions of NotationsFramework. In: HCI Models, Theories, and Frameworks, Carroll, J. M. (Eds). MorganKaufmann Publishers, San Francisco (2003) 103 133

4. Borchers, J.: A Pattern Approach to Interaction Design. In: ACM DIS 2002 DesigningInteractive Systems. ACM, (2002) 369 378

5. Button, G., Dourish, P.: Technomethodology: Paradoxes and Possibilities. In: Proceedings of the 1996 Conference on Human Factors in Computing Systems. ACM, Vancouver Canada(1996) 19-26

6. Carroll, J. M.: Dimensions of Participation: Elaborating Herbert H. Simons Science of Design. In: Les Sciences de la Conception (Science of Design), The InternationalConference in Honour of Herbert Simon, Jacques Perrin (Eds). INSA de Lyon, (2002)

7. Carroll, J. M: Making Use: Scenario-based design of human-computer interactions. MITPress (2000)

8. Crabtree, A.: Designing Collaborative Systems: A Practical Guide to Ethnography. Spring-Verlag, London (2003)

9. Crabtree, A.: Taking Technomethodology Seriously: Hybrid Change in theEthnomethodology-Design Relationship. In: European Journal of Information Systems, Vol.(13). (2004) 195 - 209

10. Dictionary.com: analytic. January 5 th, 2004


11/12


11. Dix, A: Upside-Down As and Algorithms Computational Formalism and Theory. In: HCIModels, Theories, and Frameworks. Carroll, J. M. (Eds). Morgan Kaufmann Publishers, SanFrancisco (2003) 381 430

12. Dourish, P., Button, G., On Technomethodology: Foundational Relationships betweenEthnomethodology and System Design. In: Human Computer Interaction. (1998) 13(4) 395- 432

13. Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. MIT Press,Cambridge Massachusetts (2001)

14. Grudin, J., Grinter, R.: Ethnography and Design. In: Computer-Supported CooperativeWork. (1995) 3(1) 55 - 59

15. Hartson, R., Andre, T., Williges, R.: Criteria for Evaluating Usability Evaluation Methods.In: International Journal of Human-Computer Interaction. Lawrence Erlbaum Associates,(2001) 13(4) 373 - 410

16. Hix, D., Hartson, R.: Developing User Interfaces: Ensuring Usability Through Product &Process. Wiley, USA (1993)

17. John, B. E., Kieras, D. E.: Using GOMS for User Interface Design and Evaluation: WhichTechnique? In: ACM Transactions on Computer Human Interaction. ACM, 3(4) 287 319

18. Keith, S., Blandford, A., Fields, R., Theng, Y. L.: An Investigation into the application of Claims Analysis to Evaluate Usability of a Digital Library Interface. In: Proc. JCDLWorkshop on Usability, Blandford, A., Buchanan G. (Eds). (2002)

19. MacKay, W. E., Fayard, A. L.: HCI, Natural Science and Design: A Framework forTriangulation Across Disciplines. In: ACM DIS97Proceedings of the Conference onDesigning Interactive Systems. ACM, (1997) 223-234

20. MacKenzie, S.: Motor Behavior Models for Human-Computer Interaction. In: HCI Models,Theories, and Frameworks. Carroll, J. M. (Eds). Morgan-Kaufmann Publishers, SanFrancisco (2003) 27-54

21. McGrath, J.: Methodology Matters: Doing Research in the Behavioral and Social Sciences.In: Readings in Human-Computer Interaction: Towards the Year 2000, Beaker, R., Grudin,J., Buxton, W., Greenberg, S. (Eds.). Morgan-Kaufman Publishers, San Francisco (1994)152-169

22. Monk, A., Gilbert, N.: Chapter 2 Formal Methods. In: Perspectives on HCI: DiverseApproaches. Academic Press, London (1995) 9-43

23. Nielsen, J.: Usability Inspection Methods. In: Conference Companion CHI94 (Boston,MA, 24-28 April). ACM, (1994) 413 - 414

24. Nielsen, J., Molich, R.: Heuristic Evaluation of User Inerfaces. In: Proc. ACM CHI90 Conf (Seattle, WA, 1-5 April). ACM, (1990) 249 256

25. Payne, S., Green, T.: Task-action Grammars: A Model of the Mental Representation of Task Languages. Human-Computer Interaction, (1986) 2(2) 93 133

26. Pejtersen, A. M., Hertzum, M., Andersen, P. B., Bdker, S.: Bridging the Gab between FieldStudies and Design. In: NordiCHI 2002. ACM, Aarhus Denmark (2002)

27. Pierce, J. R.: A Introduction to Information Theory: Symbols, Signals and Noise. DoverPublications, New York (1980)

28. Randall, D., Rouncefield M.: Tutorial 7 Fieldwork for System Design. Computer SupportedCollaborative Work ACM, Chicago (2004)


12/12

12 Laurian C. Hobby

29. Rosson, M. B., Carroll, J. M.: Usability Engineering. Morgan Kaufmann Publishers, SanFrancisco (2002)

30. Suchman, L: Plans and Situated Actions: The problems of Human-MachineCommunication. Cambridge University Press, Cambridge (1987)

31. Suchman, L: Writing and Reading: A Response to Comments on Plans and Situated Actions . In: The Journal of the Learning Sciences. Lawrence Earlbaum Associates, (2003)12(2) 299-306

32. Sutcliffe, A. G., Carroll, J. M.: Designing Claims for reuse in Interactive systems design.Int. J. Human-Computer Stud., 50(3) (1999) 213 - 241

33. Winograd, T.: From Programming Environments to environments for designing. In:Communications of the ACM. ACM, (1995) 38(6) 65-74