Usability in the Organisational Context: a Semiotic-Participatory Approach

Rodrigo Bonacin1, M. Cecília C. Baranauskas2

1 CESET - State University of Campinas Rua Paschoal Marmo, 1888

13484 370 Limeira, SP - Brazil +55 19 32958178

rodrigo@ceset.unicamp.br 2 Institute of Computing - State University of Campinas

Caixa Postal 6176 13083 970 Campinas, SP - Brazil

+55 19 37885870 cecilia@ic.unicamp.br

Abstract. Usability engineering represents a general framework for thinking about the design of usable systems. Nevertheless, the usability of a product is strongly determined by the social and cultural contexts of prospective users. In this paper we draw upon Organisational Semiotics and Participatory Design to promote the usability of systems designed for the context of work in organisa-tions. Our approach takes into account the semantic, pragmatic and social as-pects involved in system design and evaluation. We illustrate the proposal with results of applying the Semiotic Conference to a participatory context of de-signing Pokayoke: a Computer-Supported Cooperative Work (CSCW) designed to support problem solving in the context of a manufacturing organisation.

1 Introduction

While usability engineering represents a general framework for thinking about the design of usable systems, and has been widely adopted and documented, literature has also shown some of its pitfalls. The overemphasis on usability testing method-ologies without corresponding emphasis on the process of setting usability goals is one of them [40]. A usability test is a procedure for determining whether the goals defined in the usability specification have been achieved; if so, it assumes that one has defined the correct usability goals. Nevertheless, choosing attributes that charac-terize the usability of a system requires an understanding of users and their work. This is especially true when we are talking about designing and developing systems for the specific context of work organisations.

Usability of a product has been defined by the ISO working group on human-system interaction as “the extent to which the product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [16, p. 691]. From this definition it is clear that the usability of a product is not an attribute of the product alone, but it is determined by the context of

use, with all social and cultural aspects brought with it. Thus, the full experience of the usability of a system is hardly captured in a set of metrics. The more varied the users and environments, the more difficult to arrive at a meaningful usability goal specification.

The Organisational Semiotics [30] community has studied the use of systems in organisations. Stamper’s semiotic framework has been used as an analytic tool to understand information systems at different abstraction levels and some recent papers have proposed ways of considering aspects of user interface analysis based on it [8, 3]. The Information Systems Actability Theory (ISAT) has been discussed in the OS community bringing light to aspects of the use of IT systems by considering social action and speech act theories [15]. This theory has also been used as theoretical background to be elaborated on user interfaces of Information Systems regarding its socio-pragmatic and communicative aspects. Sjöström and Goldkuhl [33] propose a socio-pragmatic analysis of user interfaces in which “the user should be conceived as taking part in business communication with other human actors …”.

While the question of how to evaluate the usability of a system remains important, it is necessary to situate it within the process of designing the system. While we ac-knowledge Sjöström and Goldkuhl’s perspective on the use of IT systems, in this paper we argue that design and evaluation are part of a dialectical process involving designers and prospective users of the system considered in their social and cultural contexts. Thus, in order to “inform” the interface design, some evaluative techniques and activities are necessary primarily to engage the group in a process of constructing meaning for the elements involved in the system design. In this paper we draw on concepts from the Organisational Semiotics (OS) and Participatory Design (PD) to discuss usability taking into account the semantic, pragmatic and social aspects in-volved in system design and evaluation. Our approach intends to promote usability by including the dimensions of participation and signification.

The design of computer applications that enhance not only the quality of products, but also mainly the quality of work practices has been one of the main focuses of the Participatory Design approach. The PD practitioners have long advocated active cooperation between users and designers, and many researches have been conducted in establishing meaningful and productive interactions among those directly in charge of technology design and use [38]. Work has fundamentally a social nature, and critical aspects of it are often poorly addressed in traditional system development [18]. Therefore, we start with the view that to design a computer system that is part of human working environment, we must understand the social context of the organi-sation in which the work is carried out. Signification is constructed as a result of cooperation between designers and workers (prospective users of the technology).

Although participatory techniques are useful instruments to capture the social con-text of the workers by their active participation, it is equally important that designers and users could share a model representation for the domain of work to be supported with the prospective system. Methods and formalisms of OS could be useful in rep-resenting the signification constructed from analysis of work practices to the design of the system. On the other hand, PD techniques are rich in fostering participation needed to promote direct and continued interaction between professional designers and those who use technology in their work, ultimately the arbiters of system ade-quacy.

We illustrate our approach with results of applying the Semiotic Conference [4] to a participatory context of designing Pokayoke: a CSCW system aimed at supporting problem solving and decision-making in the context of a manufacturing organisation. Results from this approach for evaluation are illustrated to highlight contributions. The paper is organised in the following way: Section 2 presents the theoretical back-ground based on fundamentals of PD and OS methods. Section 3 presents our ap-proach to usability considering the participation and signification dimensions. Sec-tion 4 presents a case study applying the approach to the context of evaluating while designing Pokayoke, a CSCW system created for supporting problem solving in an automotive industry. Section 5 concludes.

2 Theoretical Foundation

In this section we present the two main research areas and background for this work: the Participatory Design (PD) and the Organisational Semiotics (OS) more specifi-cally two of the MEASUR [34,35] methods: the Semantic and Norm analysis.

2. 1 Participatory Design

Participatory Design, an approach developed initially in northern Europe, uses a variety of techniques to carry design with the user, rather than for the user. The Par-ticipatory Design approach stresses the importance of democracy in the workplace to improve the work methods, the efficiency in the design processes (with the users’ background and feedback), to improve the quality of the systems, and “to carry on” formative activities [26]. In Participatory Design, the users have direct interaction with designers in the overall cycle of development, and some control over design decisions.

The activities carried out during Participatory Design concern the development of a common view of the technology and of the organisation, exploring new organisa-tional structures, systems requirements, and prototypes of new systems. Therefore the technology is understood as a tool to improve the organisation and the workers are taken as the people who really know about the context and the objectives of the or-ganisation.

The collaboration of the users during the whole process of design provides infor-mation and feedback to the designers. Furthermore, PD activities improve the quality of the resulting system through a better understanding of the user’s work and the combination of a practitioner’s different background during the design process [2].

Participatory design raises questions of democracy, power, and control at the workplace. According to Ehn [11] the democratic ideal is a beautiful human inven-tion: Every human should have the right to participate equally in decisions concern-ing his or her life. In practice this freedom has always been limited; particularity the democracy at the work is limited by constraints imposed by the market economy and the power of capital.

In Scandinavia, trade unions have served as the vehicles for industrial democracy by advancing the interest of the workers' collective. Some early projects of PD em-phasized the strategy of including the trade unions into the design process. Kensing and Blomberg point out some results of the early projects, such as the DEMOS [10] project: “The results of these efforts included increased bargaining power due to better informed shop stewards, strengthened co-determination agreements, and na-tionals laws guaranteed, to those who claimed their rights, information about man-agement’s plans for new technology” [17, p. 170]. Those projects had also leaded an international community of researchers focusing on the interface between technology and the workplace.

Müller et al. [26] consider three convergent motivations for participatory ap-proaches, in addition to the democracy: � Efficiency, Expertise and Quality. They are improved by the direct user participa-

tion during the design, since no single person or discipline has all the knowledge that is needed for system design;

� Commitment and Buy-In. The participation in PD activities can improve the sys-tem acceptance. Some of the recent PD projects concentrate in the promotion of participation of the

workers in the system design . PD researchers have been constantly proposing new alternatives to promote the workers participation in countries with different socio-economic contexts and in the new Scandinavia and Europe contexts with the decrease in the bargaining power of unions. The conditions to promote the workers participa-tion [7], the strategy to promote it, the definition of who participate in the design (often manager participation is restricted), and the level of involvement of the worker in the design tasks and decisions are examples of issues frequently address by the PD researchers.

PD researchers have also developed techniques that explore different approaches to promote productive worker-designer co-operation. These techniques have the aim of providing designers and workers with a way of connecting current and future work practices with envisioned new technologies [17].

A participatory design technique does not presuppose to be a straightforward se-quence of well-understood steps that produce a guaranteed outcome, but a scaffold or an infrastructure for a complex group process. We present below a synthesis of some PD techniques explored in the case study detailed in this paper (from Müller et al. [26]): � Search Conference or Starting Conference. Participants from multiple, interrelated

organizations, at multiple levels of management and power, meet together to ana-lyse current working relationships, future opportunities and how to move from the current situation to the future. Participants at different levels of power are partially protected from risks due to exposing their ideas or perspectives to their own or-ganizations [31];

� Ethnographic practices. Ethnographic practices have been highly influential in Participatory Design. However, many practitioners have stated quite clearly that these are not simple techniques that can be picked up and used by anyone. Rather, ethnography requires extensive training in both specific practices, and more impor-tantly, in the perspectives and disciplines that underlie those practices. Many tech-

niques based on ethnography [32] have been widely used in PD with the aim of designing information systems. These techniques include video recording, immer-sion and other mechanisms for observing users in their real-life contexts;

� HOOTD(Hierarchical Object-Oriented Task Decomposition). Participants decom-pose a task description into the information objects acted upon and the actions taken on them, and assign groups of those objects to interface windows;

� Icon Design Game. One participant (the sketcher) draws informal icons while other participants attempt to guess at the concept that the sketcher is trying to ex-press. The drawings become first drafts (not finished artwork) from the develop-ment of icons. The game can be played co-operatively (with a single team) or competitively (with multiple teams) [24];

� Artifact Walkthrough. Users utilize artefacts from their work environment to re-construct and review a specific example of their work process [39];

� Prototyping. Prototyping has been used in many ways in participatory activities. PD methods such as Storyboard Prototyping and CISP(Cooperative Interative Storyboard Prototyping) were used in this work. In Storyboard Prototyping users and others evaluate and use a prototype that exist only as a storyboard – a series of images. This type of prototype is often faster and cheaper to create than prototypes created with traditional programming languages, so iterations of design and evaluation are faster. Some versions of the method, such CISP involve users co-developing prototypes instead of just evaluating a version of it; so other people can go off on their own to create the next version [1];

� Participatory Heuristic Evaluation (PHE). Inspectors use an extended set of heu-ristics – some product-oriented, some process-oriented to assess potential problems of design, prototype, or system, in terms of both usability and appropriateness to the end-users’ work. [25,27].

The choice made of these techniques was based on their adequacy to different

phases of the system development, the facility of applying them in the work context, and their potential for stimulating reflection and collaboration during design. Accord-ing to the flexible nature of the PD techniques, some of the methods were adapted to facilitate their use with the OS concepts, but their main features are preserved. In addition to these techniques we have also proposed the Semiotic Conference (session 3.1).

2.2 Semantic and Norm Analysis

Complementary to the PD techniques, we need methods, which are inline with the social understanding of an organisation, to model it. For this we are drawing on Or-ganisational Semiotics, in particular two of the MEASUR methods [34,35]: (1) the Semantic Analysis Method (SAM), which delineates the area of concern of an organi-sation and identifies the basic patterns of agents’ behaviour and (2) the Norm Analy-sis Method (NAM) which describes how an agent can judge a situation and take ac-tions.

According to Liu [21, p.61] “If one wishes to model a business system one has to focus on the level of patterns of behaviour. The method of Semantic Analysis pro-

vides a means for these purpose by devising a canonical formalism with focus on the responsible agent and his repertoire of behaviour.” Some basic concepts of SAM adopted in this paper are based in Liu [21]: � “The world” is socially constructed by the actions of agents, on the basis of what

is offered by the physical world itself; � “Affordance”, the concept introduced by Gibson [14] can be used to express the

behaviour of an organism made available by some combined structure of the or-ganism and its environment. In Semantic Analysis, affordances [21] are social con-structs in a certain social context;

� “Agent” can be defined as something that has responsible behaviour. An agent can be an individual person, a cultural group, a language community, a society, etc. (an employee, a department, an organisation, etc.);

� “An ontological dependency” is formed when an affordance is possible only if certain other affordances are available. The affordance “A” is ontological depend-ent of the affordance “B” meaning that “A” is only possible when “B” is also pos-sible;

� “Determiners” are properties which are invariants of quality and quantity that differentiate one instance from another;

The concepts of semantic analysis are represented by means of ontology charts.

Figure 1 shows the graphic representation used in the examples presented during this paper and the correspondent concepts of the semantic analysis. This graphical repre-sentation is based on NORMA, which is a knowledge language representation that is part of the MEASUR methods [34,35,36].

Fig. 1. Graphical representation of some Semantic Analysis concepts

In addition to the Semantic Analysis (that models the static aspects of the agents’ behaviour), it is also necessary to model their dynamic aspects. Through Norm Analysis we can represent the agents’ behaviour in the pragmatic and social levels of the semiotic framework.

At the pragmatic level, the Norm Analysis (NA) describes the relationships be-tween an intentional use of signs and the resulting behaviour of responsible agents in a social context. At the social level it captures beliefs, expectations, commitments, contract, law, culture, as well as business. Norms corresponds at the social level to the idea of an affordance at the individual level [21]. The norms are a result of affor-dances of agents in a society and they control and coordinate the agents’ actions in a social context. They also can be seen as collective affordances of complex agents (e.g. organizations, governments, departments, cultural groups, etc.) at the social level. The norms are not necessarily obeyed by all agents in all circumstances; they are social convention (laws, roles and informal conventions) that should be obeyed. For example: a norm specify that the agents are obliged to pay a tax; if an agent have no money it will not pay, but usually there is a cost when an agent does not obey the norms.

Norms are based on Deontic Logic, and the following format is suitable for speci-fication of behavioural norms [20]: <Norm>::= whenever <condition> if <state> then <agent> is <D> to do <Action>

Where <D> is a deontic operator that specifies that the action is obligatory, permit-ted or prohibited.

A norm analysis is usually carried out on the basis of the result of the semantic analysis [21]. The semantic model delineates the area of concern of an organization and identifies the basic patterns of behaviour (affordances) of the agents, while norms specify the condition and constraints on the behaviour [37]. The norms can be cou-pled with affordances in the ontology chart, during the Semiotic Conference (session 3.1).

3 Usability in the Socio-Organisational Context: a Semiotic-Participatory Approach

The traditional approaches to Information Technology (IT) development present a strict separation between design, implementation and use of the system. These ap-proaches assume that there is a common conceptual model of the domain, shared by all practitioners and the only problem is to identify this model and codify it. How-ever, several authors acknowledge the fact that shared domain models do not de facto exist but instead are socially constructed over time by communities of practice [12, 18, 19].

PD implies that designers learn about the application domain and practitioners learn about the new possibilities brought by technology. We claim that the interac-tion between the participants could be understood in terms of semiotic relations estab-lished among them, as they develop and share a common sign system. We intend to enhance the concept of usability by focusing on participation and signification; we hope to contribute with ideas on what evaluation in the workplace could be and what roles it could play in the system design.

An approach rooted in participatory practices and semiotic basis is proposed as a way of supporting users and designers in applying their knowledge and experience to

evaluate while designing the system. Figure 2 shows SPaM (Semiotic Participatory Method) used in the integration of PD and OS techniques, discussed in [5].

Organisational Semiotics

Participatory Evaluation

Organisational Context Design and Review

System Requirements Elicitation and High level Design

Detailed Design Prototype

Construction

Participatory Design

Fig. 2. PD and OS Integration approach

The circular arrow shows the software development model as a cyclic process composed of 4 main phases: organisational context design and review, system re-quirements elicitation and high level design, detailed design and prototype construc-tion. PD, SO and evaluation activities are made during all these phases, in all itera-tions, enabling the integration of these techniques in the whole design lifecycle. The evaluation is not a special phase but it is included as something that happens during the whole development cycle in order to engage the group in a process of significa-tion about the elements involved in whole system design.

The four SPaM phases are briefly described below: 1. Organisational Context Design and Review. During this phase designers with the

users understand and model the organisational context, and propose a model of a new context (using ontology charts and norms descriptions) with the opportunities brought by technology. After the first prototyping cycle, the users and designers also evaluate the organisational models (ontology charts and associated norms) and prototypes constructed during the previous iteration. The plan constructed to achieve the desired organisational context from the actual context is also reviewed at each cycle;

2. System Requirements Elicitation and High level Design. During this phase, design-ers and workers identify the affordances that the system should support, based on the organisational model constructed during the phase one. Alternatives to support the work-practices are discussed and evaluated during this phase. The designers and users construct or change sketches of the interface with the purpose of clarify-ing the problem and also to discuss the design options. After that the OS diagrams are translated to the technological models; a procedure to translate ontology charts to initial design class diagrams are described at Bonacin et al. [6];

3. Detailed Design. It concerns the design of the computational system based on the technology used to construct it (e.g. Object Oriented Design). The initial models constructed during the last phase are expanded and detailed in order to construct a complete design model. The user participation is promoted through the use of PD

techniques. The users evaluate the design options and participate in the interface design decisions;

4. Prototype Construction. It refers to the implementation of the prototype. The par-ticipation of the users during this phase (through the application of PD techniques) is not restricted to the evaluation of the system; they should participate in the tech-nical decisions that affect in some way the use of the system and consequently their work practices. They are also stimulated to participate in the deployment of the system in the organisational context.

In SPaM, PD complements OS as a way of eliciting/capturing the signs that consti-

tute the organisational context. PD offers systematic methods to promote the direct participation of users in the system design. Their participation and feedback are im-portant to make explicit the way communication and interpretation of signs affect work context. The OS, in its turn, results in models that are used back in other PD activities as an artefact to explore opportunities to improve the current work practices.

The next section presents the participatory technique designed to support the ap-

proach under discussion.

3.1 The Semiotic Conference Participatory Technique

We describe this technique in terms of six attributes proposed by Müller [27] for Participatory Design techniques: (1) the object model that describes the material used in the practice, (2) the process model that describes how people communicate with one another, how they make decisions and what they do with the material, (3) the participation model that describes who is involved in the work and the specific roles needed for the practice to work well, (4) expected results that describe the tangible benefit and how the result is used, (5) position of the technique in the whole life cycle of the product and (6) the group size. The Semiotic Conference attributes are de-scribed below:

� Object Model: Copies of Organisational Semiotic Models: ontology charts (seman-

tic analysis) and norm analysis (pragmatic and social level analysis), prototype screen shots, and pens are the material used for running the participatory practice.

� Process Model: In the format of a Conference, the designer shows concepts of the workplace modelled in ontology charts that include the affordances available or/and that would be available by the system use in a new organisational context. This model contains the concepts compiled by the designers during the initial defi-nition (the diagram must contain only the terms used by the users) of the system obtained from application of other PD techniques or from the last model of the former prototype. Copies of the models are distributed to all practitioners and, if necessary, the designer clarifies the notations and concepts described by the dia-gram. After that the designer read the ontology chart for the group. During the reading, for each concept quoted in the model that any person of the group judges important, the practitioners discuss the semantic dependencies with other concepts and the formal and informal norms associated. Members of the group, as a result

of discussions, propose changes in the semiotic models. Copies of the prototype in-terface should be distributed in order to discuss how the model’s changes could re-sult in interface changes. Alternative solutions are raised on prototype screen shots to reflect the conceptual changes in the model. The changes in the prototype can be directly drawn in the prototype screen shots or, if necessary, other PD tech-niques can be used to define them.Participation Model: Members from different levels and functions in the organisation together with the design team participate in the Conference. If necessary facilitators mediate the interaction among the group.

� Results: Semantic and Norm models constructed or reviewed during the confer-ence with people from the organisational context are produced. The establishment of social norms that will be applied to the workplace considered with the prospec-tive application is also produced as a result of the technique. The user interface is created, evaluated and reviewed through the “mutual learning” promoted by the technique.

� Position in the product life cycle: Into the SPaM this technique is applied many times during the Organisational Context Design and Review and System Require-ments Elicitation and High level Design phases. If applied in a traditional devel-opment process it can contribute for the following phases: Problem Identification & Clarification, Requirements & Analysis, High-Level Design, Evaluation and Re-Design (after the creation of the first prototype).

� Group Size: 4-10 (the number of designers should not be higher than the number of users) During the Semiotic Conference the ontology charts and norms are used as arte-

facts to discuss the work context, assuming that there is a conceptual and design de-pendency between the interface and the work practices; changes in the OS models should reflect into prototyping changes. The changes can be done directly in the pro-totype screen shots or if necessary other PD techniques should be applied in order to clarify the relations between the work practices and the interface. The next section presents results of applying the Semiotic Conference technique within the SPaM to a real context of design and evaluation.

4 Pokayoke: A Case Study

We illustrate our discussion with results of applying our approach to the participatory context of designing Pokayoke: a CSCW system aimed at supporting problem solving and decision-making in the context of a manufacturing organisation.

4.1 The Organisational Context for the System

The context of discussion is a project we are conducting [28, 29], involving a multi-national company that manufactures components for automotive systems. This or-ganisation applies the “lean” production philosophy to manufacturing [9]. Differently from “mass” production, the main characteristics of the lean production includes the

distribution of decision-power among the personnel, the incentive to teamwork and a more dynamic role for the shop floor workers. Together, these ideas propose that people from different organisational levels including shop floor workers should dis-cuss and create solutions for factory routine problems in a collaborative way. This project establishes a cooperative relationship between a research group at the Univer-sity and workers in the Factory, towards the design of technological artefacts to sup-port collaborative discussion and problem solving. The project allows the group to experience mutual learning and to put into practice theoretical and conceptual ideas about design and evaluation in a real context of an organisation.

The recognized value in this active participation of the worker in the approach proposed in this paper is totally in line with the role the employees play in “lean pro-duction” organisations. The former predominant values of product quality and work productivity are enhanced by broadened participation and skill development, under the premises that these values are closely related.

The approach we present here is also inline with the practice of lean production methodologies, as it aims at facilitating and stimulating collaboration among different categories of workers in the organisation (engineers, operators, administrators), in problem solving and decision-making. The collaboration of the users during the whole process of design provides information and feedback for the designers.

Pokayoke is a system proposed to support problem solving and decision-making in that context . It includes the major problem solving tools that were used previously by the Department of Quality in a paper-based format. The design and development of Pokayoke was conducted exploring this proposed approach.

Pokayoke is based on a procedure carried on the factory to analyse and implement corrective, preventive, security, and health actions, known as “The Five Steps”. The objective of the Five Steps procedure is to define a systematic method for dealing with problems in the everyday production routine. Every time that an unconformity is identified, an action must be taken to correct it and to make a new occurrence diffi-cult. Also, every time that a situation of potential unconformity is indicated, an error proofing (Poka Yoke) action should be carried out. The steps are: (1) the description of the unconformity found; (2) the immediate action to be taken; (3) the determina-tion of the source cause for the problem; (4) the corrective/preventive/security plan established, and (5) the analysis of the implantation and efficacy of the action.

During the application of some PD practices, the practitioners stressed the impor-tance of using the artefacts at the problem solving process in a co-operative way in all departments and levels of the organisation. The use of these artefacts in paper-based forms was previously restricted to the personnel in the Department of Quality and some people from the Engineering Department. The four main reasons emphasised by the workers to broaden the use of the tools (artefacts) were: 1. The paper-based format of the tools makes communication difficult among the

workers. Many times the workers need to leave their workplace to engage in ac-tivities such as recording the problem, contacting the team that will work on the problem and obtaining permission to execute corrective actions;

2. A long meeting is necessary in order to use some paper-based tools, but the work-ers (mainly the floor workers) have difficulty in leaving their workplaces for long periods at a time to participate in the meetings;

3. Training is necessary to enable the workers to use the tools according to the or-ganisation’s guidelines;

4. Each person keeps his/her own “five steps” forms. This practice makes it difficult to reuse the solutions on new problems.

4.2 Situating the Proposed Approach in the Lifecycle of Pokayoke

The Pokayoke system was developed in 14 months distributed in five prototype cycles. In each prototyping cycle we applied some PD techniques that have produced relevant results to all SPaM phases. These techniques were applied during 15 visits to the organisation.

Figure 3 shows the use of PD techniques during the Pokayoke design. In Figure 3 the arrows from top to bottom represent the use of PD techniques (linked by doted arrows) during the five Pokayoke prototype cycles. According to Figure 3: � Starting Conference was applied during the first, third, forth and fifth prototyping

cycles; � Ethnographic practices were applied during the first and forth prototyping cycles; � HOOTD was applied during the first prototyping cycle; � Icon Design Game was applied during the first prototyping cycle; � Artefact Walkthrough was applied during the first, third and forth prototyping

cycles; � Prototyping was applied during the five-prototyping cycles. � Participatory Heuristic Evaluation was applied during the third, forth and fifth

cycles. � Semiotic Conference was used during first, second and fifth prototyping cycles;

Fig. 3. The use of PD techniques during the Pokayoke Design

Regarding the application of the PHE technique, it was postponed to the third pro-totype cycle because only after that stage did we have a more expressive material to evaluate. The PHE was applied to a group of 7 users and 5 interface experts, based on the guidelines presented by Muller et al. [25,27], including the product-oriented and process-oriented heuristics. The 15 heuristics proposed by Muller [27] include: (1) System Status, (2) Task Sequencing, (3) Emergency Exists, (4) Flexibility and Effi-ciency of Use, (5) Match Between System and the Real World, (6) Consistency and Standards, (7) Recognition Rather than Recall, (8) Aesthetic and Minimalist Design, (9) Help and Documentation, (10) Help Users Recognize, Diagnose, and Recover from Errors, (11) Error Prevention, (12) Skills, (13) Pleasurable and Respectful Inter-action with the User, (14) Quality Work and (15) Privacy.

Fig. 4. Example of proposed changes during the semiotic conference

The Semiotic Conference was applied to iterations one, two and five of the proto-type cycles during the Pokayoke design: at the beginning of each iteration, in order to review the concepts of the former prototype and to redirect the design of the next prototype, and at other moments when it was necessary to review concepts. Figure 4 shows an example of using the Semiotic Conference during the Pokayoke design. It

refers to concepts in one of the steps of the problem solving process (Step IV). Figure 4 (top) presents part of the ontology chart for the second iteration of the prototype. It shows the drawing made by a practitioner suggesting changes in the semantic model (He proposed to include Brainstorming in this step). Figure 4 (bottom) shows the modification in the prototype interface proposed during the application of the Semi-otic Conference.

Figure 5 (top) shows the ontology chart of the third prototype including the changes proposed by the practitioner. The affordances marked with circles in Figure 5 (top) are associated to interface constructions in Figure 5 (bottom). Figures 4 and 5 show affordances related to the registration of the solution ideas into the system; although the solution ideas exist even after a brainstorming session, at this specific context they are dependent of the brainstorming because the register of solution ideas should not exist without a brainstorming associated to it. This dependence is trans-lated to the interface in Figure 5 (bottom), where the ideas are always attached to the brainstorming form.

Fig. 5. The resulted ontology chat and interface

The ontological dependencies of the objects are represented by the conditions of enabling or disabling functions in the software system; for example, it is necessary to have a Multifunctional Team to have the Brainstorming functionality enabled. A

Brainstorm

Solution Ideas

Actions

norm interpretation package was developed to interpret formal norms (organisation rules) that specify the responsibility, duties and permission in the problem solving method. These norms specify the permission to access some Pokayoke functions, for example: any user is allowed to include an idea in a brainstorming session but she/he is not allowed to specify actions to correct the problem. These norms also specify who could be responsible for which task at different phases of the problem solving processes. Changes in the norms specifications proposed during the Semiotic Confer-ence are reflected in attributes configured by the norm interpretation module.

Computational tools in the Pokayoke system support the human affordances ac-cording to the norms associated to them. The main ontology chart generated five more detailed semantic diagrams, which model the domain of each tasks that are supported by the Pokayoke system at each problem resolution step. Figure 6 shows a snapshot of the main screen of Pokayoke system prototype, in which the problems being discussed and their respective stages in “Five Steps” process are shown to the workers.

On the top of Figure 6, the label “problem state” highlights four panels with the following functionalities: (1) visualize the problems in resolution, (2) execute the corrective action manager, (3) visualize the solved problem and (4) execute a follow up tool. On the bottom of Figure 6, the label “step” highlights buttons for access to the steps of the resolution process and a progress bar that shows the steps concluded in the resolution of the selected problem.

Tools to support the problem solving process are distributed in different phases of the process, for example: Ishikawa Diagrams are used at step three, brainstorming at step two and three, and 5-why at step three. These tools are embedded in the system and are combined with asynchronous communication artefacts.

Fig. 6. The Pokayoke main Screen Snapsho

4.3 Some Outcomes of the Approach

The problems detected during the application of PHE and SC can be classified into: (1) product-oriented which focuses on the software artefact and (2) process-oriented which focuses on the human work processes that the computer artefact is intended for support [13, 23, 25, 27]. Although is out of scope in this paper to compare the find-ings of SC and PHE, we could briefly say that the two techniques have different na-tures and complement each other. While PHE focuses on the user interface, high-lighting more product-oriented problems, the SC findings are more process-oriented, pointing out problems and opportunities for improvement in the work organisation through the system.

In the SC analysis we have considered all the changes proposed for a previous model or prototype during the whole meeting. The total number of non-redundant problems identified is 39. Nineteen of them have the main focus on process-oriented issues; they were discovered when the practitioners were discussing alternative solu-tions on prototype screen shots to reflect the conceptual changes in the model. The total problems that have the main focus on the process-oriented issues was 20.

We have verified the identification of a special group of process-oriented prob-lems that is strongly connected to the signification process, in accordance to the idea that in order to “inform” the interface design, some evaluative activities are necessary primarily to engage the group in a process of signification about the elements in-volved in the system design.

The main characteristic of these problems is that they result in possibilities of im-provement in the organisational context identified during the system design, or result from the designer’s misunderstanding of the work practices. In the first case they represent changes in the system or organisational settings, proposed by the users in order to review and create new work practices. This is part of the signification proc-ess by which users learn about the opportunities brought by technology through the evaluation of the system models and prototypes. In the second case, the detection of misunderstandings of the work practice is also part of the signification process by which designers learn about the work context. To illustrate, we present some identi-fied issues that resulted in further discussions about the organisational context and system design. � “Who should trigger the Five Steps procedure?” Only the Quality Department staff

used to be in charge of starting a Five Steps procedure. After the discussion, staff from the Quality Department consented that all departments’ staff should be al-lowed to start their Five Steps. A strategy for disseminating the culture of system-atic problem solving with Five Steps was also discussed.

� “The evaluation form is not always filled out (a form that presents details about the problem at the time it occurs).” The designers had understood that the evaluation form should always be filled out, but in specific conditions ordinary workers do not fill it out. A discussion about the organisational context clarified the designers’ misunderstandings of these problems.

� “The Ishikawa Diagrams, Five Why and Brainstorming tools are used only during some specific steps.” In previous models and prototypes, these tools were not as-sociated with specific steps of the problem solving process. The models and the

system were changed to support the idea that those tools should occur only during specific steps. That discussion has corrected designers’ misunderstandings.

� “We have to define a role for the person who opens and the person who coordi-nates the problem solving process.” During the SC, the workers had realised, that they did not have well defined roles regarding responsibilities during the five-step procedure. They have emphasised the necessity of having it. All practitioners have discussed the responsibility roles during some meetings. They have established names for each responsible role; someone should be “responsible” for opening the Five Steps and for carrying out steps one and five, and someone else should be re-sponsible for the correction phases (steps II, III and IV). New practices were adopted in the factory as a result, even before the system was used in large scale.

� “The person in charge of the five step procedure is not responsible for the actions of correction.” Again, another discussion about responsibilities, but now focuses on who should be responsible for the actions. The “responsible for actions” con-cept was clarified. New practices were adopted regarding this fact, even before the use of the system.

� “We have different Five Steps for each part of the factory.” The organisation, with which we have worked, has four different products oriented sites. For each one they had a slightly different five-step procedure; many differences have appeared during the SC. A common pattern was defined for all the sites.

� “After an Ishikawa Diagram we want to do a Five Why.” This change in the sys-tem model has resulted from a discussion about the “Five Why” tool needed and resulted in a better understanding of its use by the system designers.

� “The person in charge of the Five Step identification and not the person in charge of the Five Step procedure should choose the workers who would work in the multifunctional team.” Another discussion about responsibility was carried out in order to establish the roles of two workers during the Five Steps.

� “The definition of what should be implemented to correct the problem is in the fourth step” Another designer misunderstanding about the work practice was de-tected during the SC. A discussion about what should be done in the third and the fourth steps was carried out, clarifying the work practices.

� “The IV step should be finished only after conclusion or cancellation of the ac-tions.” Some workers used to finish step IV before the conclusion of some actions. A mechanism to avoid this practice, which is considered a bad one, was embedded in the system.

� “The quality manager should verify all the Five Steps.” The practitioners have engaged in a discussion about the manager’s role, after detecting that s/he should be the responsible for verifying all the Five Steps.

� “How to audit this new context.” The workers had realised that a role for the audit tasks was missing in the system model. The workers discussed how to do this task using the system and new possibilities of making this work easier were carried out.

Discussion raised by SC practice in the proposed approach fostered a shared un-

derstanding of the signs in the work practices and their discussion contributed to a mutual learning about those practices and responsibilities when considered through the system under development. In general, the problems identified by SC led to deeper discussions about the future organisational context, and aspects of usability

close to the organisational context, but PHE and SC should be seen as complementary techniques to achieve the identification of diversified aspects of the interface includ-ing process oriented and systems oriented aspects. The main drawback of the pro-posed approach is the difficulty to explain the semantic analysis concepts to the users, but after the second application the users were able to contribute directly to the mod-elling without the necessity of detailed explanations about the concepts.

5 Conclusion

Despite the tradition and best practices brought by the traditional concept of usability, it has also been accepted that usability is not an attribute of a product alone. The full experience of system usability shall take into account the semantic, pragmatic and social aspects of the prospective system usage. In this paper we have situated the concept of usability in a context of developing application for organisations, propos-ing a semiotic-participatory approach to system design and evaluation.

The proposed approach combines the perspectives of Organisational Semiotics and Participatory Design aiming at promoting system usability by considering the dimen-sions of participation and signification of end-users for elements of the system model. Results of the application of the approach were illustrated in the Pokayoke design, a CSCW system created for an automotive manufacturing organisation. Findings show the nature of the problems raised during evaluation, mostly related to work signs, their interdependency, and the definition of roles and responsibilities, highlighting opportunities for improvement in the work practice of the organisation, considered through the system.

Acknowledgments

This work was partially supported by grants from Brazilian Research Council (CAPES BEX2214/02-4, CNPq 301656/84-3 and FAPESP 2000/05460-0). The au-thors also thank Delphi Automotive Systems in Jaguariúna, Brazil, and the members of the AIS Lab (University of Reading) for their collaboration and partnership.

References

1. Andriole, S. J. Rapid application prototyping: the storyboard approach to user requirements analysis (2nd ed.) Boston: QED (1992).

2. Braa, K. Influencing qualities of informantion systems – Future challenges for participatory design in PDC´96 Proceedings of the Participatory Design Conference, 163-172.

3. Baranauskas, M. C. C., Liu, K., Chong, S. (2002) Website Interfaces as Representamen of Organisational Behaviour, In Proceedings of 5th International Workshop on Organisational Semiotics. (1996).

4. Bonacin, R. and Baranauskas, M. C. C., Semiotic Conference: Work Signs and Participatory Design, in Proceedings of 10th International Conference on Human - Computer Interac-tion, (2003), 38-43.

5. Bonacin, R. and Baranauskas, M. C. C., Designing Towards Supporting and Improving co-operative Organisational Work Practices, in Proceedings of 5th International Conference on Enterprise Information Systems, (2003), 233-238.

6. Bonacin R., Baranauskas, M. C .C and Liu, K. From Ontology Charts to Class Diagrams: semantic analysis aiding systems design. in 6th International Conference on Enterprise In-formation Systems, Porto – Portugal, (2004).

7. Clement, A. e Van den Besselar A Retrospective Look at PD Projects. In M. Müller e S. Kuhn (eds.). Participatory Design: Special Issue of the Communications of the ACM, vol. 36, no. 4, (1993) pp.29-39.

8. Connolly, J.H., Phillips, I.W. User-System Interface Design: an Organisational Semiotic Perspective. Proceedings of IFIP WG8.1 Working Conference on Organisational Semiotics, Montreal, QC, (2001).

9. Deming, W. E. Out of Crisis. The MIT press, (1992). 10.Ehn, P., e Sanberg, A. Management Control and Wage Earner Power (Foretagsstyrning och

Lontagarmakt). Falkoping: Prisma, (1979). 11.Ehn, P. Scandinavian Design: On Participation and Skill in Schuler, Douglas and Aki,

Namioka (Eds.) Participatory Design: Principles and Practices, Hillsdale, NJ: Erlbaum, (1993), 41-77.

12.Fischer, G. , Lindstaedt, S., Ostwald, J., Stolze, M., Sumner, T., Zimmermann, B. From Domain Modeling to Collaborative Domain Construction. In ACM Proceedings of DIS’95 Symposium on Designing Interactive Systems: Processes, Practices, Methods & Tech-niques, (1995), 75-85.

13.Floyd, C. Outline of a paradigm change in software engineering. In G. Bjerkners, P. Ehn, and M. Kyng (Eds), Computers and democracy: A Scandinavian challenge. Brookfield VT: Gower, (1987).

14.Gibson,J. J. The Ecological Approach to Visual Perception. Houghton Miffin Company, Boston, Massachusetts, (1968).

15.Goldkuhl, G., and Agerfalk, P. J. Actability: A way to understand information systems pragmatics. In Liu, K et al (eds.) Coordination and Communication Using Signs: Studies in Organisational Semiotics - 2, Kluwer Academic Publishers, Boston, (2002).

16.Karat, J. User-Centered Software Evaluation Methodologies. . In Handbook of Human-Computer Interaction, second completely revised edition. M. Helander, T.K. Landauer, P. Prabhu (eds.) Elsevier Science B.V., pp. 689-704

17.Kensing F., e Blomberg J. (1998) Participapory design: issues and concerns. Computer Supported Cooperative Work, Special issue on Participatory Design, Jeanette Blomberg e Finn Kensing (eds.), 7, (1997), 167-185.

18.Kyng, M. Designing for Cooperatin: Cooperating in Design. Commun. ACM 34,12, (1991), 65-73.

19.Lave, J. Wenger, E. Situated Learning: Legitimate Peripheral Participation. IRL report 90-0013, Palo Alto, Calif.: Institute for Research on Learning, (1990).

20.Liu, K. e Dix, A. Norm governed agents in CSCW. The First International Workshop on Computational Semiotics. Paris. University of De Vince, (1997).

21.Liu K. Semiotics in information systems engineering, Cambridge University Press, (2000). 22.Mazzone, J. O Sistema Enxuto e a Educacao no Brasil in Computadores e Conhecimento:

Repensando a Educacao, J.A.Valente (ed.), Gráfica Central da Unicamp, (1993), 247-312. 23.Mahemoff, M. J. and Johnston, L. J. Principles for a Usability-Oriented Pattern Language In

Calder, P. and Thomas, B. (Eds.), OZCHI '98 Proceedings , 132-139. Los Alamitos, CA, (1998).

24.Müller, M. J., Wildman, D. M., e White, E. A. Participatory design through games and other exercises. Tutorial at CHI’94 conference, Boston, (1994).

25.Muller, M. J., McClard, A., Bell B., Dooley S., Meiskey L., Meskill J. A., Sparks R., and Tellam, D. Validating an Extension to Participatory Heuristic Evaluation: Quality of Work and Quality of Work Life, In Proceedings of CHI ´95 Conference companion, (1995), 115-116.

26.Müller, M. J., Haslwanter, J. H., and Dayton, T. Participatory Pratices in the Software Life-cycle in Handbook of Human-Computer Interaction, M. Helander, T. K. Landauer, P. Prabhu (eds.), Elsevier Science, 2 ed., (1997), 255-297.

27.Müller, J. M., Matheson, L., Page, C. & Gallup, R. Participatory Heuristic Evaluation. Interactions. september + october 1998. V. 5, Issue 5, ACM Press, (1998), 13-18.

28.Nied, Revitalizing Training and Learning in Industries in Brazil, Nied Núcleo de Informática Aplicada à Educação, Unicamp, internal document, (1997).

29.Nied, O Uso da Internet na Formação Colaborativa e Descentralizada de Funcionários de Fábricas Enxutas, Nied Núcleo de Informática Aplicada à Educação, Unicamp, internal document, (Projeto Fapesp2000/05460-0), (2000).

30.OSW. The circulation document in the Organisational Semiotic Workshop, Enschede, (1995).

31.Palshaugen, O. Method of designing a starting conference. Oslo: Work Research Institute, (1986).

32.Rose, A., Shneiderman, B., and Plaisant, C. An applied ethnographic method for redesign-ing user interfaces in ACM proceedings of DIS’95 Symposium on Designing Interactive Systems: Processes, Practices, Methods & Techniques, (1995), 115-122

33.Sjöström J., and Goldkuhl, G. The semiotics of user interfaces - a socio-pragmatic perspec-tive. In Proceedings of 6th International Workshop on Organisational Semiotics, (2003), 57-72.

34.Stamper, R. K., Althaus, K., Backhouse, J., MEASUR: Method for Eliciting, Analyzing and Specifying User Requirements. In Olle, T. W., Verrijn-Stuart, A. A. e Bhabuts, L. (eds.) Computerized Assistance During the Information Systems Life Cycle. Elsevier Science, Am-sterdam, (1988), 67-116.

35.Stamper, R. K., Language and computer in organized behaviour in Riet, R. P. v .d. and Meersman, R. A., (eds.), Linguistic Instruments in Knowledge Engineering, Elsevier Sci-ence, (1992), 143-163.

36.Stamper, R., Social Norms in Requirements Analysis – an outline of MEASUR. In Jirotka, M., Goguen, J. and Bickerton, M. (eds.), Requirements Engineering, Technical and Social Aspects, Academic Press, New York, (1993).

37.Stamper, R., Liu, K., Hafkamp, M. and Ades, Y., Understanding the roles of signs and norms in organizations – a semiotic approach to information system design, Behaviour & Information Technology, 19, 1, 15-27, (2000).

38.Schuler, D., and Namioka, A. Participatory design: Principles and Practices. Lawrence Erlbaum Associates, Inc., Publishers, New Jersey, USA, (1993).

39.Wixon, D. R., e Raven, M. B. Contextual inquiry: Grounding your design in user work. Tutorial at CHI’94 conference, Boston, (1994).

40.Wixon, D., and Wilson, C. The Usability Engineering Framework for Product Design and Evaluation. In Handbook of Human-Computer Interaction, second completely revised edi-tion. M. Helander, T.K. Landauer, P. Prabhu (eds.) Elsevier Science B.V. , (1997) , 653-688