©Brian Whitworth 2006, [email protected] THE WEB OF SYSTEM PERFORMANCE (WOSP) Excellence requires Balance

©Brian Whitworth 2006, [email protected]

1

THE WEB OF SYSTEM PERFORMANCE (WOSP)

Excellence requires Balance


2

Overview

I. System performance issues

II. The Web of System Performance (WOSP)

III. System design implications

IV. Validation experiments

V. Summary


3

Past Predictions– Leisure society (4 day week)- cf a 6 day week!– Paperless office (Toffler) – cf more paper than ever before!– No more programmers (Martin)- programmers still in demand– AI will replace people – we still prefer human telephone help – Star-Trek’s video-phone – we have the technology, but it didn’t happen

– Video-conferencing – millions $ invested, but didn’t happen – Virtual-reality (headset) games – cf MMORGs & WADs – A multi-media Internet - the Internet is still mainly text, though

people download MM files– E-commerce will dominate trade – still < 10%

People who invested in predictions, like Internet bandwidth and video-conferencing, lost money


4

Bleeding Edge Theory• Media Richness Theory

– Rich medium rich message?

– A “richness dimension?

– “Lean” e-mail, chat & text-messaging are the success stories!

• Technology Acceptance Model (TAM)– That Usefulness+Usability define end-user acceptance?

– Yet Security, Privacy and Reliability are key issues!• Microsoft spent millions on NT to make windows more reliable

– “Mr Clippy” (Word’s Assistant) was a friendly animation, but such a failure, its exclusion was an XP selling point!


5

Leading Edge Practice• Practitioners develop “unpredicted” products like:

– E-mail: Considered a too simple text messaging system– Chat rooms: Not multi-media, so not predicted– WWW: Pundits argued it needed centralized control– HTML: Experts felt it was a too-simple tag language– Cell phones: Originally thought to be a yuppy toy– Google: Was a simple search engine (unlike multi-mediaYahoo)– Text messaging: Why text with a multi-media phone? Because!– Blogs, wikis, chat, reputation systems etc are not multi-media!

IS Practice is ignoring IS theory


6

We need theory!• Imagine if we ran an atomic or nuclear program

by trial and error. • Imagine if we ran the space shuttle program by

trial and error • Are we developing the technical infrastructure

of the future World Society by trial and error? – Theory distills experience, and lets us avoid costly

mistakes. It prevents predictable errors.

IS Needs Theory!


7

What is performance?

• Propose all the problems above arise from a too limited view of system performance.

• So: What is system performance?– Is it single or multi-dimensional?– Is it absolute or relative?– Is there a “best” performance?– How does one produce high performance systems?


8

System performance is?

How well a system interacts with its environment

• An environment can offer any system: – Opportunity – environment gives benefits– And/Or– Risk – environment gives damage or loss

• General Systems Theory approach (Bertalanffy)


9

Alexander & Synthesis• Alexander’s “Notes on the Synthesis of Form” raised

the idea of “design tension” over 40 years ago. IS developers now use it as “Pattern Theory”. It argues that system performance is always a balancing act.

• A vacuum cleaner is a design “form” in the following problem design space:

Reliability

Usability

Functionality

Cost


10

Evolution & Performance

• Successful biological systems range from simple viruses to powerful predators– Not just the strong are “fit”– Balance performance goal tensions differently– One sided “excellence” can cause extinction

• Information systems illustrate a virtual “evolution”, where success has many forms, e.g.– Mobile laptops vs powerful desktops– “Light” background utilities vs mainline applications– One purpose add-ins vs all purpose suites– Open source public vs closed private systems


11

System Levels

1. Social: norms, culture, laws, zeitgeist, sanctions, roles (Sociology)

2. Personal: semantics, attitudes, beliefs, opinions, ideas (Psychology)

3. Software: programs, data, bandwidth, memory, processing (Computing)

4. Hardware: computers, wires, printer, keyboard, mouse (Engineering)

The model proposes four levels of “system” :


12

Emergence• Levels are different “views”, not a different systems!

(Alter, Grudin, Kuutti, 1996)

1. Information emerges from MechanicsPhysical acts create information (see Shannon & Weaver)

2. Meaning emerges from Information: Information + cognitive processing creates human meaning

3. Culture emerges from Meaning: Personal meanings common across a group that continue over time create a culture

• Each level “emerges” from the previous• Higher levels have higher requirements and higher

performance benefitse.g. the benefits of a cooperative society require a just society


13

Social-Technical Systems• Hardware Systems involve physical exchanges• Technology Systems are hardware + information

data flows• HCI Systems are technology plus people • Social-technical Systems (STS) are social

systems, built on HCI systems, built on software systems, built on hardware systems, e.g. email

• i.e. involve all four system levels • The Web of System Performance (WOSP) model

applies to any level of a social-technical system


14

People vs Technology?• Machines that become like us or replace us:

the Terminator, IBM's Deep Blue, AI the movie.• Machines that control us:

From the industrial dark ages to the Matrix. • Machines that join with us and take over:

Star Trek's Borgs, Star War’s Darth Vader. • Technology has evolved for <100 years but people

have evolved for 3 million years– WE ARE THE SENIOR PARTNER

• For human/machine to successfully work together, technology must support human processes


15

• Define the human process, then design the technology– Human Process Technology Support– 1:1 conversation Email – Many sense processing Multi-media system– Trading Ebay type systems– Human foraging Browser design– Group conversations Chat– Associative memory Hypertext– Feedback learning The wonderful back button– Normative behavior Reputation systems

• Likewise social needs like justice can define email architectures

Technology and human processes


16

Part II

The Web of System Performance


17

System elements

• Assume four common system elements:– Boundary: monitors system entry/exit– Internal structure: supports and controls – Effectors: generate output effects– Receptors: process input signals

• Examples:– People: skin, brain and organs, muscles and

senses– Computers: case, mother-board architecture,

peripheral output and input


18

Environment Responses

• Assume two desirable system outcomes:– Gain Opportunity: The environment responds

with a system benefit– Avoid Risk: The environment responds with a

system loss


19

WOSP Goals• Each system element can be designed to gain value

(opportunity) and/or avoid loss (risk):– Boundary

• To enable useful entry (extendibility).• To deny harmful entry (security).

– Internal structure• To accommodate external change (flexibility).• To accommodate internal change (reliability).

– Effector• To maximize external effects (functionality).• To minimize internal effort (usability).

– Receptor• To enable meaning exchange (connectivity).• To limit meaning exchange (privacy)


20

A System is not “high performance” if:

1. Ineffectual – it cannot do the job.2. Unusable – you cannot make it work.3. Unreliable – it breaks down often.4. Insecure – it succumbs to viruses.5. Inflexible – it fails when things change.6. Incompatible – it cannot import standard

plug-ins or data. 7. Disconnected – it cannot communicate.8. Indiscreet – it reveals private information.


21

Functionality

Usability

Connectivity

Security

Privacy

Reliability

Flexibility

Extendibility


22

• Goal: To act on the environment• What can the system do?• Also called: functionality,

capability, power, usefulness, effectiveness

• Examples:– A car that goes fast, stops well,

turns well etc

– Software that does what the user wants

Functionality


23

• Multi-functional• All purpose (e.g. Office Suites)• Feature driven (“bloatware”)• Can always “get the job done”• Complicated - large menus• Most people only use 50% or less

of their potential • Hard to use

Functional systemsTend to be:


24

• Goal: To operate efficiently or easily, to minimize the resource costs of action

• How easy is it to operate/run?• Also called: easy to use, simple,

user friendly, efficient• Examples:

– A car that is easy to drive, with simple controls

– People can use usable software without a lot of training or help

– HTML is usable, email is usable

Usability


25

• Easy to use, learn and operate (intuitive)

• Require little training, help or documentation

• Provide only what is necessary now (contextual screen information)

• Simple, focus on essentials, remove unnecessary “bells or whistles”

• Less powerful

Usable systemsTend to be:


26

• Goal: To continue operating despite internal data or component failure (errors)

• How often can it perform?

• Also called: Dependability, stability, trustworthy, ruggedized, MTBF, robustness, durability, maintainability, recoverability.

• Examples:– A reliable car starts every day

– Reliable software doesn’t “hang” with internal “bugs”, or data errors

– Windows reliability has increased over the years - XP was sold on this

Reliability


27

• Be long lasting (so lifetime warrantees are economic)

• Be predictable (so users can plan)

• Have Redundant parts - if one part fails another takes over

• Be minimally coupled - one error does not cause another• Recover quickly from breakdown (module or data recovery)

• Have Undo/Back operations to reverse errors

• Be hard to change

Reliable systemsTend to:


28

• Goal: To still work in new environments, e.g. by changing its operation

• Where can it perform?• Also called: Adaptability, tailorability,

customizability, portability, platform independence, plasticity, agility, modifiability, pervasive

• Examples:– An all-terrain vehicle works anywhere– Mobile/pervasive computing lets people use

software in any environment

Flexibility


29

• Easily customized to individuals (e.g. control panel)

• Easily “fitted” to different situations (e.g. hardware independent software, O/S independent programming languages)

• Able to learn usage trends (e.g. cache prediction algorithms)

• Demand driven (e.g. CSMA/CD Ethernet vs polling networks)

• Interconnected, holistic - one change can affect all • Error prone (e.g. changing Windows 95 screen display could be

catastrophic! - until an Undo was provided)

Flexible systemsTend to be:


30

• Goal: To use outside components/data• With what can the system perform?• Also called: Tool use, openness, scalability,

interoperability, standards, permeability, compatibility

• Examples:– Cars can fit add-ons (e.g. towbar, spoiler, rack ..)

(if towbar ball is standard size)– Open software, like Netscape, accepts “plug-ins”– Clip and paste between applications allows one

system to use another’s data easily

Extendibility


31

• Be able to accept third-party “add-ons” (Open IBM PC vs McIntosh sealed system)

• Be open source (system code openly displayed)• Support general standards • Be easily added to - Open Systems Architecture

“Plug & Play”• Be easily scalable (like the World Wide Web)• Have decentralized control – object orientated • Not be as secure

Extendible systemsTend to:


32

• Goal: To resist or repel unauthorized entry, misuse or take over

• Who controls the system?• Also called: Anti-virus, firewall,

protectiveness, defense, integrity, safety, integrity

• Examples:– A secure car’s lock cannot be picked– Secure software (and data) cannot be

used or modified by hackers or viruses

Security


33

• Centrally controlled, bureaucratic• Strong on boundary checks (logon codes,

passwords)• Resistant to external attack or entry• Internally idiosyncratic• Risk avoiding rather than opportunity

enhancing • Hard to add on to, not opportunistic

Secure systemsTend to be:


34

• Goal: To communicate with other similar systems

• Who can we talk to?• Also called: Communication,

information exchange, networking, sociabilty, interactivity

• Examples:– A connected car could sense other cars

and avoid a crash– Connected software uses the Internet:

email, chat rooms and groupware

Connectivity


35

• Be informed by other systems, and “group aware”

• Be up to date by download/upload • Have many channeled (multi-media), two-

way duplex (reciprocal), high linkage (many-to-many) communication

• Have a low “cycle time” (user complaints, software patches)

• Experience information overload/distraction

Connected systemsTend to:


36

• Goal: To control internal information release and disclosure

• Who sees us?

• Also called: Tempest proof, opaqueness, privacy, autonomy, camouflage, stealth confidentiality, secrecy

• Examples:– Car with tinted windows, a radar scrambler

– Privacy software controls interactions with others (e.g. Black Ice, Zone Alarm)

Privacy


37

• Be regenerative (“packing” a file needs privacy)

• Restrict information use (e.g. digital signatures copy protection)

• Be information encrypted • Respect rights to own information, where one

needs permission before data access • Protect personal information (documents, credit cards) • Hard to connect to

Private systemsTend to:


38

Performance is a Balance


39

Part III

System design implications


40

WOSP Area = Performance• The WOSP area best estimates “performance”• Performance (the area) needs all the WOSP dimensions• ‘Experts” predict based based on recent advance(s)• If the most recent advance is the strongest dimension• It also has the greatest “tension” with other factors• So it soon gives diminishing returns• Then, IS progress moves “unexpectedly” to a dimension where the tension is less• The gaming development of the last decade was not graphics but

social gaming (MMORPGs massively multiplayer online role-playing games)


41

WOSP Lines = Tensions • WOSP lines are cross-cutting tensions• Increasing any aspect of performance means any other

can “bite back” (Edward Tenner)

– “Open” systems can increase security risks, and more security makes systems less open eg US.

– More functions can mean more for users to learn– Flexible control panel can reduce reliability, when users

change the wrong things

• Yet one can have security and openness, ease of use and functionality, connectivity and privacy, etc

• As systems advance, progress needs tension resolution


42

Innovation can resolve tensions

• Find innovative ways to reconcile“opposites”– Functionality + Usability = elegance – Extendibility + Security = discrimination– Reliability + Flexibility = autonomy – Connectivity + Privacy = legitimacy

• Avoid one-dimensional progress (it bites back)

• Find progress at the points of least tension

System design is an art as well as a science


43

Expanding the Web• In 1992, Apple CEO John Sculley introduced the hand held Newton, saying portability (flexibility) was the wave of the future – he was right

• But the Newton’s portability reduced its data input usability, and in 1998 Apple dropped the line.

• When Palm’s Graffiti language improved handwriting recognition, the PDA market took off.

• Innovations may need combination advances


44

Why are “killer apps” functionally simple?

• A basic email system can be written in a weekend, yet email is a killer app

• When systems begin, the web is “slack”. As they evolve, tensions arise

• A simple new application allows WOSP performance expansion

• A powerful new function, like video e-mail, is much more difficult to support all-roundMinimize functionality to increase your system

performance!


45

WOSP shape/profile and Environment “Fit”

• Performance has no “perfect” form• In opportunity environments right action gives benefit,

favoring the four success creating goals:– Functionality, Reliability, Extendibility and Connectivity

• In risk environments, wrong action gives great loss, favoring the four failure avoiding goals:– Security, Privacy, Usability, Flexibility

• Different environments, organizations and applications favor different performance “shapes”


46

Analyze the performance profile

• What % of your project should be spent on: – Security Resist outside attack/take-over?– Extendibility Use outside components/data?– Functionality Required functionality?– Usability Conserve user/system effort?– Reliability Avoid/recover from internal failure?– Flexibility Predict/adapt to external changes?– Connectivity Communicate with other systems?– Privacy Control self-disclosure?

• Don’t spend 90% of project effort on what will be only 50% of system performance


47

Convert WOSP goals to performance requirements

• For a browser– Privacy

• Any sensitive information I give the browser, like logon passwords, is encrypted, so others can’t see it.

• Password information always shows as asterisks, so others cannot look over my shoulder to see them

• It stops web sites from getting my name or email from my computer’s data.

– Security• It can detect and prevent spyware from installing• It can detect and prevent popup ads


48

System design “layers”• Specialist specifications/teams for:

– Functionality: Traditional specs (main module)

– Usability: User costs (HCI) (interface, help, wizards)

– Reliability: Error analysis (error & recovery code)

– Flexibility: Portability analysis (control panel, preferences)

– Extendibility: Compatibility analysis (plug-in manager, import/export data, clip and paste)

– Security: Threat analysis (logon/registration module)

– Connectivity: Internet/network analysis (comms module)

– Privacy: Legitimacy analysis (rights control module)


49

Design Integration

• For projects to produce known and established systems, whose goals are well known and well defined, specialization and specialists maximize results (they give more local output per local criteria)

• For projects to produce new systems, whose goals are not well known or defined, requires integration as well as specialization, to increase synergy and decrease cross-cutting conflicts


50

Increasing Integration

• Cross-disciplinary leader or project “guru”– Hard to find

• Increase cross-discipline communication: Regular cross-specialist meetings under multi-discipline chairs – hard to run

• Combine specialty teams. e. g. for: actions (functionality + usability), interactions (security + extendibility), contingencies (reliability + flexibility) and sociability (connectivity + privacy).


51

Extreme Programming

• For small teams producing innovative/new projects– Everyone contributes to every system part– All team members involved in every aspect of

system design– Common entry point for all system changes

• Maximizes system integration (goal synergies)

• Minimizes design conflicts


52

Part IV

Validation experiments


53

Validation – Conjoint Analysis

• Subjects were told they were company managers who had to choose a new common browser for their company

• They were given 30 browsers to choose from, each with different ratings on the eight WOSP factors

• Then asked to rate and rank the browsers


54

Results: Criteria Weights

Performance Factor

Avg. Importance

Std Dev. 99% Conf % > 12.5

Security 22.78 12.78 16.07-29.50 70.83 Privacy 15.47 9.19 20.30-10.64 58.33

Usability 14.16 9.88 19.36-8.97 50.00 Functionality 12.02 8.21 16.33-7.70 29.17

Reliability 11.64 8.15 15.93-7.36 33.33 Connectivity 9.24 6.54 12.68-5.80 33.33 Extendibility 7.69 4.56 10.09-5.30 16.67

Flexibility 6.99 6.46 10.39-3.59 16.67

Security, Privacy & Usability came before Functionality!


55

Analytical Hierarchical Processing Study

• TAM proposes perceived usefulness and ease of use affect technology acceptance (2 factors)

• Subjects used both TAM and WOSP criteria

• Subjects rated 3 browsers using AHP pair-wise comparisons

• They then compared the two methods


56

Results: TAM vs WOSP

Effect N TAM WOSP df MS F p

Confidence 20 3.7 2.4 1 16.9 12.8 .002*

Accuracy 20 3.6 2.45 1 13.2 12.4 .002*

Completeness 19 4.21 2.42 1 30.4 16.8 .001*

Outcome Satisfaction 18 3.14 2.35 1 5.64 9.63 .006*


57

CA vs AHP WOSP Ranks

CRITERIA AHP WEIGHTS RANK

LOCAL GLOBAL AHP CA

Extendibility 0.21 0.04 8 7

Security 0.79 0.17 2 1

Flexibility 0.43 0.09 7 8

Reliability 0.57 0.12 4 5

Usability 0.58 0.09 3 3

Functionality 0.42 0.12 6 4

Privacy 0.64 0.20 1 2

Connectivity 0.36 0.11 5 6

Has the Web become a threat environment?


58

Part V

Summary


59

What Users Want• Functionality and Usability

• Reliability and Flexibility

• Security and Extendibility

• Connectivity and Privacy

• They want it all!

• Why?

• We ourselves are such a balanced system, so we expect no less of the systems we work with


60

Design Tips Summary• Define the human process, then design the technology• Less (functionality) may mean more performance (avoid

the Version 2 flop problem)

• Match project effort to your app’s performance profile• For new applications, increase design integration by

cross-disciplinary staff, or more/better communication

• Expect cross-cutting conflicts• Convert WOSP goals to technology requirements• Check your web of system performance - success

needs many causes, but system failure needs only one


61

Design Questions

• What is the human process/task your technology will support?

• What is your system level? (S/W,HCI, STS)• What is your application’s performance profile?

(e.g. your top 3-4 criterion goals are?)• What are specific requirements of each goal?• Do you have cross-cutting design issues? • What designs satisfy your “performance space”?


62

Summary

• One-dimensional progress bites back• Progress is a train on many tracks at once• Advanced system performance has many goals • Reconciling these goals requires innovation• Innovation finds the points of least tension• System design is an art as well as a science,

needing integration as well as specialization• The “killer” advances of the last decade (email,

browsers, chat, blogs) succeeded by balance as well as excellence


63

ConclusionExcellence requires Balance

See brianwhitworth.com for: 1. Whitworth, B., Fjermestad, J., Mahinda, E., 2006, The Web

of System Performance: A multi-goal model of information system performance, Communications of the ACM, May, Vol 49, No 5, p93-99. http://brianwhitworth.com/wosp06.pdf

2. Whitworth, B., & Zaic, M. (2003). The WOSP Model: Balanced Information System Design And Evaluation. Communications of the Association for Information Systems, 12, 258-282. http://brianwhitworth.com/WOSP03.pdf


64

Derived vs Observed

• WOSP goals were derived from the general nature of systems

• Yet match observed system requirements quite well

• Current system requirements are very confused and confounded

• E.g. are privacy and reliability part of security?

Documents

©Brian Whitworth 2006, [email protected] THE WEB OF SYSTEM PERFORMANCE (WOSP) Excellence requires Balance