An Approach to the Evaluation of Usefulness as a Social Construct using Technological Frames Jose Abdelnour Nocera and Lynne Dunckley Thames Valley University

Jose Abdelnour Nocera and Lynne DunckleyThames Valley University - Institute for Information

Technology

An Approach to the Evaluation of Usefulness as a Social Construct using An Approach to the Evaluation of Usefulness as a Social Construct using Technological FramesTechnological Frames

An Approach to the Evaluation of Usefulness as a An Approach to the Evaluation of Usefulness as a Social Construct using Technological FramesSocial Construct using Technological Frames

Jose Abdelnour Nocera Jose Abdelnour Nocera

[email protected]

Lynne DunckleyLynne Dunckley

[email protected]

Institute for Information TechnologyInstitute for Information Technology

Thames Valley UniversityThames Valley University

EnglandEngland


Technology


Relevance:Rapid prototyping allows designers to obtain quick feedback with a minimum investment in the early stages of design; however, no valuable information regarding its usefulness can be collected until a broader adoption of the technology.(De Paula, 2003)


Technology


Key objectives:

1. The use of concept of technological frames to explore how systems acquire their usefulness once deployed in their actual contexts of use.

2. Bringing the sociology of technology closer to HCI, CSCW and IS research.


Technology


Key characteristics:

An alternative epistemology towards the study of technology and the groups related to it that questions:

•The idea of systems having a fixed useful character.

•The boundaries between producers and users of IT/ between settings of design and use.


Technology


How many bicycles can you see?

•A Unsafe bicycle

• A Macho bicycle


Technology


The “working” and “nonworking” of an artifact are socially constructed assessments, rather than intrinsic properties of the artifact. One artifact (in the old sense) comprises different socially constructed artefacts, some of which may be “working” while others are “nonworking”.(Bijker, 1995)


Technology


The Metaphor of Technology as Text...

“just as the meaning of a written text is not a property of the text, so too the character of technology is not determined by its technical structure” (Mackay,2000)

Interpretative flexibility of Technology:


Technology


Soc. of Tech. offers an epistemology congruent with the study of usefulness of IT as social proxies

Traditional HCI Technology as Social Proxies

Prototype based

Subject Screening

Lab Settings

Extrinsic Motivation

Task Orientation

Reliable Product

Critical Mass

Real World Settings

Intrinsic Motivation

Real Problems and Needs


Technology


Previous work on studying the usefulness of systems:Information Systems:Technology Acceptance Model (TAM) Davis (1989)-Perceived Usefulness: reduced to idea of performance

-Perceived Ease of UseLimitations of Quantitative Methods

• Relation to developers responsiveness Gefen, D., & Keil, M. (1998)• Relation to User Interface Characteristics Calisir, F., & Calisir, F. (2004)


Technology


HCI: Nielsen’s view on usefulness

Sociology of Technology


Technology


• Focused on Meaning (Constructionist Paradigm) instead of Perceived Performance (Positivistic Paradigm)

• Interpretative flexibility of Technology (Sociology of Technology)

Usefulness as a social construct:


Technology


The technological usefulness of software artefacts remains unknown, or known only indirectly, and relies upon improvisations that remain invisible to professional design (Suchman, 2002).

Users ‘construct’ technology; they do this both symbolically, in their reading of artefacts, and literally, in the articulation work that is essential before a generic system-product can be used as an artefact supporting day-to-day practices. (Hales, 1993)

Usefulness as a social construct:


Technology


The core set of assumptions, expectations and knowledge of technology collectively held by a group or group. (Orlikowski and Gash, 1994, p.199)

A technological frame comprises all elements that influence the interactions within relevant social groups and lead to the attribution of meaning to technical artifacts- and thus to constituting technology.(Bijker, 1995, p. 123)

Technological Frame:


Technology


This approach integrates Technological Frames with the concept of Breakdowns:

A breakdown is not a negative situation to be avoided, but a situation of non obviousness, in which the recognition that something is missing leads to revealing (generating through our declarations) some aspect of the network of tools that we are engaged in using.

Winograd and Flores (1986, p.165-166)

(Similar ideas in Situated Action –Suchmann, AT, Hermeneutics)


Technology


• Goals Societal and practical

short and long term

• Problems (breakdowns)

• Elements of interpretationAssumptions, knowledge (e.g. problems solving strategies, praxis) and expectations: reflected by people or in discourse

• Elements of practiceAny action, practice leading to or responding to the attribution of meaning to technology

Revised Conception of Technological Frames:


Technology


EMPIRICAL WORK• ELSOFT: European based software vendor of standard ERP software (BIZWARE)

for small and medium sized organisations, sold around the world

• ETNOGRAPHY and QUALITATIVE CASE STUDIES:– RESEARCHER ADOPTED THE ROLE OF “USABILITY CONSULTANT”

• Informed Consent issues• Conflicts of roles and “going native”.

– ANALYSIS OF THE PROCESS OF USERS’ ROLE IN DEVELOPMENT – APPROPIATON OF BIZWARE IN CUSTOMER SITES IN INDONESIA,

HONG KONG, SPAIN AND UNITED KINGDOM.

• DATA :– INTERVIEWS, – INTRANET DOCUMENTS, – E-MAILS AND FAXES– PARTICIPANT OBSERVATION


Technology


Why Enterprise Resource Planning Software for SME’s?

•Used to support tangible work activities•Standard design used in different cultures•Producers presented it as embodying “best business practice”• Access


Technology


Natural?

Usefulness

Utility

Usability

Stakeholders’ Technological Frames

SoftwarePractical?

breakdownCoping

Strategies

Coping Strategies


Technology


Usefulness of BIZWARE defined with meanings (themes) related to…

• Security

• Efficiency

• Customer Centredness

• Situated Usability

• Openness

• Completeness

• Quality of Automation


Technology


Technological Frames Identified in the Social Construction of the usefulness of Bizware

Sales & marketing Developers Users Supporters Various User Groups

Goal

Promote and Sell Bizware. Create a fast and efficient business data processing technology.

Solve all the queries and local needs of users

Support and automate their business activities wit the least effort.

Problems

Persuade a very diverse market with a standard system.

Keep their software and database ‘bug’ free.

Support different user needs with a standard system and a limited power to change it.

Having an unsuitable tool to support their work.

Elements of Interpretation

Universality of TechnologyTechnical Centredness

Universality of TechnologyTechnical CentrednessUser awareness

User awareness Previous Experience with TechnologyUnderstanding of Information System GenresPerception of Foreign OriginOrganizational Models and Culture.

Elements of Practice

Sales & marketing Strategies Technological Inventory and workspaceDevelopment practicesContact with UsersCoping Strategies:Problem Locus Construction

Technological Inventory and workspaceContact with UsersCoping Strategies:Suggestions of WorkaroundsProblem Locus Construction

Local Working PracticesCoping Strategies:WorkaroundsProblem Locus ConstructionSubscription to Technology


Technology


CASE 1: INDONESIA Technological Frames

Themes User Community IND Developers at ELSOFT

Security

Users’ interpretation of the security he Bizware offers for their activities

In the current functionality of [Bizware] the cost price of the good can be seen by the user who runs this function. (…) we do not want warehouse people to know the values of the goods for security reasons.(App. 5.2.1) During the evaluation of your product neither was it brought to your attention and thus it was overlooked by assuming that it is similar to our former software. .(…) I know I may not be a big customer to your company but I really need the attention on this BUG issue. (App. 5.2.2)

Developer:Unfortunately this is not possible. When somebody has the rights to enter receipts, he also can have a look at the purchase order and the prices of the purchase order.(App. 5.2.6)


Previous Experience with technology

Customers based their definition of the problem in relation to their previous experience with similar software

Organisational Model and Culture

The breakdown raised by IND reflects an aspect of organisational culture that requires a more refined handling of users’ rights to information than offered by the Bizware. However, developers at Elsoft do not see this as a critical limitation.


Technology

An Approach to the Evaluation of Usefulness as a Social Construct using An Approach to the Evaluation of Usefulness as a Social Construct using Technological FramesTechnological FramesCASE 1: INDONESIA

Elements of practice

User Community IND

Workarounds IND altered some of their procedures to cope with what they construe as a security vulnerability: To survive of these problem, the customer is having to have somebody else (eg. finance people assigned at sales order entry, or somebody who the management can trust better) to do the job for the time being.(App. 5.2.4)

These are actions created by users with the intention of solving a breakdown in the use of the software.

Problem Locus Construction

IND users claimed that the absence of a more refined control of information in the warehouse operation is a limitation of the Bizware. For them this should be a natural condition of the software. This breakdown is located by them in the software and this confirmed when they refer to it as a “bug”.Developers claim the opposite and indicate that this feature is an option and a special requirement of IND working practices. Thus, the problem is located in located by them in the users.The exchange of letters between IND managers and staff of Elsoft in Indonesia and Europe reflects this difference.


Technology


CASE 2: SPAIN Technological Frames

Themes User Community ELCA ELSOFT

Openness

How flexible the software is in allowing users to configure it to suit their activities, i.e., multiple early payment discounts.

In this case “closed” means that the Bizware is not able to handle multiple early payment discounts in invoices: Traject (one of the ERP competitors) was eliminated because we saw it very limited and closed, and afterwards [after the Bizware was implemented] we noticed that also with this software.(Ramon, interview excerpt, 27/07/2003)

The main problem is that this is a closed program. It will never be enough for an enterprise. (Ricardo, interview excerpt 27/07/2003)

I’m telling you this is not because is an intrinsic problem of this company all the food companies at least here in Spain work in that way [with multiple early discount]. (Ricardo, interview excerpt 27/07/2003)

Elsoft depicts it software as appropriate for ELCA’ needs:Sales Manager for Spain:In the same way we consider that the Bizware is a program perfectly appropriate for your needs, because it is precisely a live, modern, and Standard product with continuous functional improvements. (App. 5.3.2)Development manager for Elsoft in Asia:This method is not supported and not planned as a functional change either. (…) We have to be very critical in what we accept as a functional change or not.(App.5.3.4)


Software Genres Users at ELCA compare the Bizware with other software of the same genre (ERP). Based on this comparison they interpret the software as “closed”. Elsoft presents its software as “appropriate, line and modern”.

Organisational Models and Models

One of ELCA’s managers emphasises the idea that the Bizware does no support their “commercial management” by limiting their ability to handle multiple level of discounts. Developers at Elsoft does not see as critical and affirm that the software is “appropriate” their needs. This is an incongruence between the business model inscribed in the software and that of ELCA.


Technology


CASE 2 SPAIN: Elements of practice

Workarounds User Community ELCA ELSOFT

These are actions created by users with the intention of solving a breakdown in the use of the software. In this case workarounds were characterised by:-Extra Work -Creative use of Interface Elements and Data fields

Users had increase manual operations to partially address the discount problem. In ELCA multiple discounts are included before the invoice is created: In order to solve some of the issues we have to perform extra work (…)About 20% of our client has Prepayment discount. For them the final amount is not Correct and that is why they do not pay us.. At the end of the month we make a global discount.(App.5.3.3)

“When errors have appeared a thing that was usually done in 30 minutes is completed in three hours. This has created a very stressful situation among our staff. Is this going to work? (…).This is not normal.(Ramon, interview excerpt 27/07/2003)

To solve this we have had to assign different codes to the same client, that is, creating multiple identical clients on the system but with different discount levels. This is not practical. (Ricardo, interview excerpt, 27/07/2003)

Alternative use of Internet Elements:Since we started working with the Bizware we are able to see the expiry date of the product because this can be seen in the sales order reference number. In this way you could see if the customer could stop an order because an expiry date issue. (Marta, interview excerpt, 26/07/2003)

In the fields tagged as “extra”, I create each article (item) card and I use a lot of the extra fields to include cost information per boxes, per pallet, net weight, etc. (Marta, interview excerpt, 26/07/2003)In some order lines some clarifying notes or in ‘your reference’ field we put some references to our order , I don’t need that field so I use it to know, for instance, what’s inside that order, e.g., marcador butter. (Ale, Interview Excerpt, 29/07/2003)

Development staff at Elsoft suggested a workaround using “phantom items” in the invoice. However, the local consultant in Spain was not too convinced since it was a partial solution:

Please do not reject after all. In job 07.440.257 some improvements have been made. However, the customers still need to be able to record somewhere in the master data that a debtor gets an invoice discount. And this info needs to automatically be used when entering an order/invoice. (App.5.3.4)


Technology


CASE 2 SPAIN: Elements of practice

Problem Locus Construction

User Community IND ELSOFT

The General Manager for Elsoft in Spain writes about the discount problem as a fundamental limitation of the Bizware:This is a legal requirement, this is business crucial (App.5.3.4)

The development manager in Elsoft’s headquarters questions the necessity for that feature in the software. This is evident when he says that a “functional change” is not planned. This means that this feature is not considered as fundamental for the software:This method is not supported and not planned as a functional change either. (…) We have to be very critical in what we accept as a functional change or not.(App.5.3.4)


Technology

An Approach to the Evaluation of Usefulness as a Social Construct using An Approach to the Evaluation of Usefulness as a Social Construct using Technological FramesTechnological FramesConclusions:

• The definitions of how useful a system is and of the problems it is deemed to solve are not static but negotiable, fluid and subject to the sociocultural perspectives and practices of both producers and users.

Contributions and Further Work:

• TF offer qualitative sociological framework that explains how the usefulness of systems is socially constructed.

• Analysis framework that can be integrated with novel UCD context-based techniques such as Bill Gaver’s cultural probes.

• Reflective approach that can equally consider producers’ and users’ perspectives for the identification of conflicts, e.g. elements of interpretation and practice in RAD methods such as XP and DSDM.

Documents

An Approach to the Evaluation of Usefulness as a Social Construct using Technological Frames Jose Abdelnour Nocera and Lynne Dunckley Thames Valley University