Measuring Information Architecture Quality The Nebulous Science Nick Ragouzis, Enosis Group [email protected] ACM SIGCHI, CHI2001Seattle, Washington April,

Measuring Information Architecture

Quality

The Nebulous Science

Nick Ragouzis, Enosis Group

[email protected]

ACM SIGCHI, CHI2001 Seattle, Washington April, 2001 (Extended version)

CHI2001, Measuring IA Quality, Extended version April 2001, Nick Ragouzis, [email protected] 2

Why are we asking about measuring IA quality?

• Basis: – Not: about the research/science domain– Rather: about commercial/applied domain

• In significant, challenging, environments

• Important, relevant?– What do we mean? To what extent?– Assumptions, Causes, Consequences?

• Or is it merely symptomatic… of IA’s lost place?


Information will have an Architecture

• Quality is an attribute of value

• An IA is not intrinsically valuable

• Value, in terms of– Creation, continuity– Users; time, often money– Relative to competing forces


Just Wasted Time?

• A sea of assumptions– High dimensionality– High volatility– … in aspects, degrees, time frames, etc.

• Quality of one IA over another– Just another design job– Good-enough: Just favor one of several IA

• Assumptions more directly addressed by increasing and sifting focus to innovation


The IA, Defined

• Information Architect [1996, noun, archaic]:– The individual who organizes … the patterns

inherent in data … making the complex clear

– A person who creates … structure or map … allows others … personal path to knowledge

(Editorialized from Richard Saul Wurman)

– The 21st century professional occupation addressing needs of the age [, who is] focused on

• clarity, • human understanding and • the science of the organization of information


Information Architect’s Main JobIntegrator’s

Craft [circa 1996]

Clarifies -mission and vision, balancing an IA

Determines-content, functionality an IA

Specifies-org., nav, labeling, searching an IA

Maps-change, growth an IA

Pride(Editorialized from Rosenfeld & Morville)


IA, Re-Defined

• Information Architect [2001, verb]:– A facet of the capabilities and responsibilities

of the professionals participating in ainterdisciplinary collaborative design system

– Executive committee, Marketing, Sales, Strategic planning, OD/Comm, Finance, Independent divisions and departments, Product management, IS management, Project management, interaction designers, experience designers, graphics designers, programmers …

– … and all other interested parties, constituents, and users directly or by convenient proxy and surrogate.


Information Architect’s Main Job?Integrator’s

Craft [circa 1996]

Embedded in Org’s Design

Systems [circa 2001]

Clarifies -mission and vision, balancing an IA

BOD, Executive committee Marketing, Org. comm., Finance, Divisions, Competition…

Determines-content, functionality an IA

Executive committee Product marketing, Competition, IS, …

Specifies-org., nav, labeling, searching an IA

Marketing, IS, Product management, designers, programmers, …

Maps-change, growth an IA Executive committee

Marketing, Finance, …

Pride EBIT, NCF


Learn what a [website] customer wants and direct them to it

1996• Only an Information

Architect could know what’s best

2001• Just another design job• IA-in-a-box

–Automatic; Prescription

• Heard throughout the interdisciplinary team:

“I’ll do an info architecture on that by Friday.”


Meanwhile …

Parker, Day, The Dawn of Man, Burlington, 1992 wf.carleton.ca/Museum/man/evnman.html

Irrelevant Self-Congratulatory Technical Legerdemain

User; as submissive Grudging conciliation

User as computer HCI playbook found; cooperative execution declared

User as customer Assistance through anticipation

User as people Full-fledged design challenge in the social/applied psychology domain

Our “Users” evolve:


Users as People

• Social-Psychological, Arousal– Reeves & Nass

• Persuasion– Captology, BJ Fogg et al– CHI2001 Website Credibility

• Trust– CHI2001 Relational Agents

• Choice– Surfing Regularities, Information Scent (CHI2001)

• Huberman, Pirolli, Card, et al,


The Main Job of an Interdisciplinary,

Collaborative, Design System

Innovate, or follow strategically


Innovation is?

• Which of these flashy, money-hogging, schedule-overrunning, customer-frustrating features would you like to add to your website project today?

• Perhaps you also have some of your own to suggest? I’d prefer we spend the

money on a sweepstakes…


Innovation is?

• Do?, When?; Which combination?• Both in basic navigation & interaction structure,

and so-called value-adding services e.g., personalization, chat, real-time svcs

• Risks in investment, and not• Wasted time and money on one, another, or more?

• Missed something crucial today, next week?

Back button as opportunity?


Innovating Strategically

• Deliver verifiable improvements in the customer’s perceived value

That’s for more than just the website or whatever, right?


Following Strategically

• Choose, in order to maximize:

– Differentiation from most serious rivals

– Affinities with competitors and partners having similar differentiation requirements

– Competitive power of the explicit or tacit market group to which the follower belongs

– De facto considerations, where differentiation isn’t relatively valuable

– Follower-specific pathologies


A call from the past …


How much innovation?

“In this age companies have lost

the privilege of standing still.”†

relative to rivals

† 1962, Philip Kotler, Marketing Management


Create customers† more efficiently‡

than your important rivals

† 1962, Peter Drucker, “The first task of a company … create customers.” although my meaning in this presentation is not limited to commercial companies and their consumers. Neither do I interpret Drucker as intending such limitation.

‡ Efficiency in its fully-strategic implications


Customer-Oriented Focus †

Benefits to us:1. Customer needs are more basic than particular

offerings

2. Attention to customer needs helps us spot new opportunities more quickly

3. In return we are more effective at satisfying customers

4. In return we bring our own interests into greater harmony with society’s interests.

† 1957, Wendell Smith


In other words…

We must apply out creativeness more intelligently to people, and their wants and needs,

than to products. †

† 1959, Charles Mortimer, italics in original


Basic metric of innovation

• EBIT, NCF/CAO

• Return on reinvestment of customer’s investment (their time, operations on their financial gains) matched to customer’s ability to further benefit

A Change?Simple Growth?

Compound rates of growth…early, and sustained


Efficiently creating customersRelative advantage to rivals

Requires sustainedabove-average performance

Not So Good Much Better


The sounds of average performance• Parity, more or less

• Do what 90% of other companies do

• Implement de facto practices

• Follow 75%-likely recommendations

• Improved conversion rates on launch

and…

• The quality of an information architecture

The Average Surfer?


Choosing: a wealth of opportunities• Profile of

– Customers perceived value mechanisms and – Value perception history; – Including vis-à-vis important rivals

• Real-options-based valuation– Achievable rates of growth in perceived value– Achievable relative advantage to rivals– Volatility: in opportunities, actions by rivals, own

organization’s ability to perform– Information returns to organization’s systems

• Including in (re)designing the design systems


Choosing; a wealth of opportunities• Synergy Effects, Interdependence

• Extended View of Opportunities– Flexibility in delay– Multi-staged, mixed– Multi-pronged, with late run-off– Wait-and-see; emphasis on See– Flexibility to abandon– Preemption; establishing, responding– Restructuring rivalries; switching barriers– Unilateral alliances


Summary• Justifying IA? It’s the wrong question

– Times have changed

• What was IA is, rather, a design job– For the organization’s entire design system

• Deliver verifiable improvements in the customer’s perceived value

• Innovate or follow, strategically– Do so persistently; rates of growth, compounded

• The fittest wins– Beware short-term, false, maxima!


Thank You

Nick Ragouzis

[email protected]


----Further Notes----


But IA is important!• Measurements we would have seen:

– Depletion/decay of a info architecture’s integrity• Average onset time• Rate of undone effort, investments• Remediation, costs, effort, lost opportunity

– Rival-comparison of an architecture’s effectiveness

– Consistent maps establishing dependency of results on features, activities, or responsibilities

– Half-life of an information architect’s efforts

– Factor for valuation and volatility for IA’s efforts in overall project valuations


But we need the name!

• Fine– In some domains

• But in commercial applied domain– It will inevitably fall into use as a verb– There, it’s superfluous

• Interdisciplinary, collaborative, “zone” approach

• Project managers have more influence on an IA

Is the name the problem?


Inspecting the false parallel of “Architect” in IA

Structure, function, utility vs Experience

Design & usability analysis vs Perceived value analysis

Construction engineers vs Architects

Information architects vs Experience/Interaction designers

Return to: We Need The Name!


The IA as Bridge From Research to Applied Domain

• Shaping the Design System

• Empirically disconfirming propositions

• Challenging the status quo

• Harvesting information gain– Options on further innovation– Options on organization change

Design Research Reconciliation


Aspects of an IA’s Role as Bridge†

• Problem Driven

• Client Centered

• Challenges the Status Quo

• Presents Empirically-Disconfirmable Propositions

• Systems Theory-based

• Usable in Everyday Life

† Loosely Adapted from Action Science: Lewin, Susman, Argryis, …


Damaged Merchandise?

• Teaming the academic and the industrial

• Addressing the limits of academic peer review in cross-disciplinary environments

• Addressing the poor quality of methodology coupled with the naïve claims of validity

• Improving the ability of those working throughout the design continuum to make valid inferences

Gray and Salzman, HCI 1998, 13(3) Return to IA as Bridge


Center for Design Research Reconciliation

“Research on such real-world problems is difficult, and this is probably an ideal problem to be tackled by academic-

industrial partnerships”Olson, Moran, introductory editorial to HCI 1998 13(3)



• Ground other’s past and newly published work in other’s past and current (and future) media and design disciplines

• To relate the published research to business agenda and personal experience agenda

• To reconcile conflicting or disjoint work

• To help raise the level of understanding of the qualities in research that is most helpful and productive

• To help raise understanding in organizations regarding what they can do to incorporate such research into their design systems



• Enlist sponsors: Corporations (clients, systems houses, developers, researchers, etc.) academe, associations, and foundations. From many disciplines and having many interests.

• Guide a grant process following a community-developed agenda• Integrate researchers (basic and applied) and practitioners into

the effort• Hold in appropriate escrow the “sacred” data• Explore and evolve standards for the work• NOT doing the resulting peer-review publishing; CDRR-

supported research published as grantees desire plus appropriate CDRR-supported releases.

Return to IA as Bridge


Backtracking as Success

• Strength of the “prior” page relative to followers

• Power of exploratory enticement

• User confidence in space, in performance

• Strength of “home base”– Who has investigated user-set “checkpoints” in

browser history path pruning services

Return to Innovation is?


Crucial, Yet Rarely Incorporated

… in design… in applying, extending results… in selecting subjects

Return to Average Performance

Strong Regularities in WWW Surfing, Huberman et al, Science V280, 3 April 1998, pp. 95-97; Case: AOL

Mean: 2.98 clicks; σ = 2.06…but by then almost 2/3 are gone

Mode: P(1.5 clicks) = .3…so most common experience is 1 click


The unexpected may happen:

NASDAQ Composite Index

Documents

Measuring Information Architecture Quality The Nebulous Science Nick Ragouzis, Enosis Group [email protected] ACM SIGCHI, CHI2001Seattle, Washington April,