Designing Evidence-based Disclosures: A Case Study of Financial Privacy Notices

204 THE JOURNAL OF CONSUMER AFFAIRS

LORETTA GARRISON, MANOJ HASTAK, JEANNE M. HOGARTH,SUSAN KLEIMANN, AND ALAN S. LEVY

Designing Evidence-based Disclosures: A Case Studyof Financial Privacy Notices

Disclosure is a key component of consumer protection policy. Byinforming consumers about a product or service, disclosures can helpconsumers understand product features and shop among productsand providers to find the combination of features and price that bestmeets their needs. For example, the Gramm-Leach-Bliley Act (GLBA,15 U.S.C. 6801-6809) provides for disclosures of information-sharingpractices of financial institutions and, in some cases, requires that theseinstitutions offer consumers the opportunity to limit some of this shar-ing. Using these disclosures as a case study, this paper explores howresearch can help policymakers shift from a perspective of develop-ing disclosures that are in technical compliance with the law to oneof developing disclosures that consumers pay attention to, understandand use in their decision making.

In an ideal marketplace, if complete information was available atno cost (financial or search costs) to all participants, fully informedconsumers would make decisions that are optimal for their financialsituations and lifestyles and, at the same time, enable markets to functionefficiently. In the absence of complete information, models of informationsearch in economics posit that consumers will seek product informationand comparison shop as long as they perceive a marginal benefit fromthese activities; benefits can take the form of lower prices or better value,as each consumer defines it (Ippolito 1988; Stigler 1961). Informationsearch, however, is not ubiquitous. Consumers’ perception that search

Loretta Garrison ([email protected]) is a Washington attorney and, until September 2011,was a Senior Attorney at the Federal Trade Commission. The views expressed do not necessarilyrepresent the official views of the Federal Trade Commission or of any individual Commissioner.Manoj Hastak ([email protected]) is a Professor in the Kogod School of Business at AmericanUniversity. Jeanne M. Hogarth ([email protected]) is a Manager at Federal Reserve Board.The analysis and conclusions set forth in this presentation represent the work of the authors and donot indicate concurrence of the Federal Reserve Board, the Federal Reserve Banks, or their staff.Mention or display of a trademark, proprietary product, or firm by the authors does not constitutean endorsement or criticism by the Federal Reserve System, and does not imply approval to theexclusion of other suitable products or firms. Susan Kleimann ([email protected]) is aPresident at Kleimann Communications Group. Alan S. Levy ([email protected]) is a SeniorScientist at the Food and Drug Administration.

The Journal of Consumer Affairs, Summer 2012: 204–234DOI: 10.1111/j.1745-6606.2012.01226.x

Copyright 2012 by The American Council on Consumer Interests

SUMMER 2012 VOLUME 46, NUMBER 2 205

costs (effort, time, difficulty of finding information) are too high canoutweigh any perceived benefits, especially marginal ones.

Disclosures that make product pricing and features more transparentcan reduce search costs, potentially improving outcomes for those con-sumers who shop. Although many financial service firms provide productinformation in the absence of mandatory disclosure requirements, manda-tory disclosures impose common standards of terminology, presentationand calculation of relevant figures that can aid consumers in makingcomparisons between products and providers. For this reason, disclosureis a fundamental component of consumer protection policy in financialservices (Durkin and Elliehausen 2011).

Consumer decisions are affected not only by the broad context ofthe consumer’s economic, social, cultural and political environment, butalso by more personal psychological and socioeconomic factors. Forexample, research on consumer search for financial products shows thatfinding ways to motivate consumers to shop, seek out and pay attentionto information disclosures is a significant challenge (Lee and Hogarth1999a). And there is evidence that consumers have problems processinginformation in financial markets just as they do in other markets (e.g., thenutrition and health care markets; Hibbard, Slovic, and Jewett 1997; Leeand Hogarth 1999b; Levy, Fein, and Schucker 1996; Viswanathan andHastak 2002). In addition, individual differences in experience, expertiseand self-confidence affect how consumers process information (Loiblet al. 2009; Payne, Bettman, and Johnson 1993).

Research in psychology, marketing and behavioral economics hashighlighted not only the possible limitations of disclosure, but also newopportunities for its use. For example, research shows that the way inwhich choices are structured or presented can change the likelihoodthat consumers will select a particular option (Madrian and Shea 2001).One implication is that as policymakers confront decisions about theproper presentation of disclosure, they may face choices about whetherdisclosure should be “neutral” in presenting product options or shouldpromote or discourage a consumer’s selection of certain products orfeatures, a particularly thorny issue since the desirability of some featuresdepends on their fit with a particular consumer’s personal situation.

While these behavioral insights are increasingly a part of the pol-icy discussion and debate, the statutory disclosure requirements and thelongstanding goals of increased transparency and competition in marketsalso continue to motivate policymakers to improve the effectiveness ofdisclosures. The task for policymakers and regulators is to implement dis-closure standards that have the potential for being timely, comprehensible


and useful in consumer decision making. This paper demonstrates howresearch can be used to develop disclosures that consumers pay attentionto, understand and use in decision making, and how policymakers canshift from a compliance and enforcement perspective to a user perfor-mance and effectiveness perspective.

In 1999, Congress passed the Gramm-Leach-Bliley Act (GLBA, 15U.S.C. 6801-6809) requiring for the first time that financial institutionsprovide a privacy notice to all their customers describing how theyused customers’ personal information and what rights, if any, consumershad to “opt out” or to limit certain types of information sharing.The notices were sent by a variety of financial institutions—banks,credit unions, insurance companies, securities firms, automobile dealers,tax preparers, mortgage brokers and non-bank finance companies. Theregulation implementing the law provided some sample clauses that firmscould use in their disclosures, with language that focused on compliancewith law, not on consumer comprehension.

In the spring of 2001, privacy notices from financial institutionsbegan appearing in consumers’ mailboxes. They were lengthy, confusing,written in a highly legalistic style and generally incomprehensible.Consumers were not expecting them and did not know what to dowith these new notices. The notices varied considerably, not only informat, presentation, language, length, style and tone, but also, wherefirms shared information more broadly, in how they informed consumersof their rights to limit certain sharing of their personal information.

When these notices first appeared, there was a wide public outcry offrustration from the media, advocacy groups, individual consumers andmembers of Congress. In response, the eight federal agencies responsiblefor regulating these privacy notices decided to explore ways to makethese notices more easily understandable and usable for consumers.These eight agencies are the Commodity Futures Trading Commission(CFTC), Federal Deposit Insurance Corporation (FDIC), Federal ReserveBoard (Board), Federal Trade Commission (FTC), National Credit UnionAdministration (NCUA), Office of the Comptroller of the Currency(OCC), Office of Thrift Supervision (OTS) and Securities and ExchangeCommission (SEC).

In 2004, six of these agencies1 launched a consumer research projectwith the goals of identifying barriers to consumer understanding of

1. The six agencies that initially sponsored the project were the Board, FDIC, FTC, NCUA, OCCand SEC. The OTS joined the project for the quantitative testing phase. The CFTC participated inthe rulemaking and outreach aspects of the project.


privacy notices and developing an alternative notice that consumers couldmore easily understand and use. Staff from the agencies included mid-level staff attorneys and research staff with backgrounds in economics,marketing and sociology. The project had two research phases: first,qualitative testing to iteratively develop a new prototype notice; andsecond, a quantitative study to test the prototype notice against otherprivacy notices.

The project presented special challenges: How do you take severalvery complex laws and translate them into a format that consumerscan quickly and easily understand so that they can make a decisionbased on that information? How do you balance the tension between thelegal requirements for consumer disclosures and the legal intent for thedisclosures to be effective, i.e., understood and used by consumers? Howdo you communicate information-sharing concepts defined by the lawthat are largely unfamiliar to consumers? Further, how do you make thenotice neutral—so that it allows consumers to express their preferenceswithout being biased by the information presentation?

This paper follows the development, via consensus and collaboration,of the notice. After a brief review of related literature, we provideinformation on the testing protocol, present results and offer some lessonslearned, discussion and conclusions.

PREVIOUS RESEARCH

Several large streams of research fed into this project, and a com-plete treatment of previous literature would be a paper in itself. Forthe sake of brevity, we focus on three particular areas. One stream flowsfrom practitioners trying to design privacy notices, including a substantialbody of work from the information technology field on online privacy.Not surprisingly, many of these studies find that consumers do not paymuch attention to online privacy notices (Jensen and Potts 2004; Jensen,Potts, and Jensen 2005). Furthermore, consumers spend only about one-fifth of the estimated required reading time when they do read notices(Grossklags and Good 2007; Kay and Terry 2009). Shorter summariesof information or layered notices get mixed results (Good et al. 2005;McDonald et al. 2009; Proctor, Ali, and Vu 2008). Standardization mayhelp consumers transfer learning and make comparisons (McDonald et al.2009); research on a standardized privacy “label” showed improvementsin consumers finding and understanding information and easily compar-ing policies (Kelley et al. 2009a, 2009b). Finally, risk perception andfactors such as consumers’ perceived self-efficacy, age, perceived threat,


overall concern about privacy and general overall readability of the noticeare related to the probability that consumers will read online privacynotices (McDonald et al. 2009; Milne, Culnan, and Greene 2006; Milne,Labrecque, and Cromer 2009; Milne, Rohm, and Bahl 2004).

A second stream comes from the field of market research that exploresconsumers’ use of disclosures and labels, their information-processingcapabilities and their decision heuristics. The level and quality ofinformation acquired and used by consumers depend on the availability ofinformation and the costs (in time and resources) of acquiring it (Brownand Dimsdale 1973; Ippolito 1986, 1988; Mandell 1973; Stigler 1961).Disclosure is valuable because it makes costly information more available(Kirsch 2002; Lacko and Pappalardo 2007; Macro International 2007).Consumers may choose the option that seems the easiest (Novemskyet al. 2007) or the most unique (Simonson and Nowlis 2000). They mayinvoke a set of screening rules (Gilbride and Allenby 2004) or rely onrecommendation agents, such as Consumer Reports (Swaminathan 2003).They may make choices depending on the availability and salience ofkey information (DellaVigna 2009) or they may feel so overwhelmed thatthey make no decision at all. In the specific case of privacy notices, theoption of “doing nothing” results in some financial companies sharingconsumers’ information with a potential set of affiliates and non-affiliatedthird parties—an outcome that consumers may not prefer.

A third stream of research includes work from the field of informationdesign, specifically developing readable and usable consumer notices(Culnan and Milne 2001; Heroux, Larouche, and McGown 1988; Kellerand Staelin 1987; Moorman 1996; Payne, Bettman, and Johnson 1994;Roe, Levy, and Derby 1999; Russo 1988; Wogalter, DeJoy, and Laughery1999). The fields of information design and decision research sharea similar perspective on how consumers develop decision strategies.Underlying good document design decisions are an array of integratedprinciples, but three categories deserve particular attention: motivatingconsumers to decide to read (Anderson and Pearson 1984; Durkin 2006;Haviland and Clark 1974; Song and Swarz 2010), getting consumersto see critical information (Few 2004; Kirsch et al. 1993; KleimannCommunication Group 2008; Tufte 1990) and getting consumers tounderstand and comprehend the information (Bloom 1956; Kimble2006; National Assessment of Adult Literacy 2007; Redish 1989; Saw2000).

Comprehension often is linked with readability and literacy, butcomprehension can take on a range of levels from the ability tocorrectly decode and remember conveyed information to the ability to


synthesize and apply information to a task (Bloom 1956). For example,the National Assessment of Adult Literacy (2007) expanded the definitionand measurement of literacy beyond readability formulas to includedocuments such as bus schedules. The goal for the privacy noticeproject was to move consumers beyond reading to analysis, synthesisand evaluation—all higher levels of cognitive processing.

A document’s ability to facilitate informed decision making isparamount to true comprehension. To ensure that a decision is“informed,” all decision elements and their attributes should be present.Furthermore, when consumers see all the elements (the whole), they bet-ter understand the parts, including the interdependencies between theparts (Saw 2000). If informed decision making assumes a comparison ofkey attributes, then these attributes must be arranged so that comparisonis easy.

USING CONSUMER TESTING TO DEVELOP THE NOTICE

Overview

Although the GLBA established the content of the required privacynotice, consumer testing allowed the agencies to take the requiredelements and determine what was more important to consumers, as wellas what was misunderstood or misinterpreted. The agencies and theircontracted researcher, Kleimann Communication Group (Kleimann),were primarily interested in the performance of the whole document,while making sure that the parts of the document worked for consumers(Kleimann Communication Group 2006). The team of the agencies’staff and Kleimann based their definition of “worked for consumers”not merely in terms of information conveyed, but also in behavioralobjectives such as making choices consistent with preferences. Inaddition, the team used Bloom’s scale of seven levels of cognitiveprocessing—from simple recall of details (knowledge) to assessmentof the value of information (evaluation)—to gauge the comprehensionlevel of participants in the qualitative research test sessions (Bloom1956).

The agencies set certain design considerations and constraints on thenotice. The notice had to be objective and neutral: it needed to informconsumers about privacy laws and financial institutions’ sharing practicesin an unbiased, factual way. The notice had to fully comply with all legalrequirements. Additionally, to minimize testing variables, the agencieslimited the design to black and white on 8.5 × 11 inch paper, with alarge, readable font.


FIGURE 1Kleimann Communication Group’s Information Design Model

Plan Develop Assess/Revise Test Deliver Evaluate

Research Design and Method

The agencies and Kleimann relied upon a six-stage consumer researchdesign model to gather data and revise the prototypes (Figure 1)(Kleimann Communication Group 2006).2 In the Plan stage, the agen-cies’ staff attorneys worked with the researchers to identify the legalrequirements to go into the notice. These included the different wayspersonal information could be shared and whether consumers had a rightto limit or opt out of certain sharing.

During the Develop stage, Kleimann used two focus groups (twentypeople) to obtain baseline information on consumers’ impressions,attitudes, likes and dislikes about the initial test documents. Also in theDevelop stage, one session of preference testing (seven people) usedin-depth, one-on-one interviews to explore consumers’ preferences forvocabulary, headings, notice components and ordering of the informationas well as varied ways to present the information.

In the Assess/Test/Revise stage, Kleimann conducted a pretest and aseries of five interview sessions to iteratively test the usability and tofine tune the prototype notice. Each session consisted of six to eight one-on-one interviews (according to Virzi (1992), 80% of usability problemswere uncovered with five participants and 90% after ten). Thirty-nineparticipants from Baltimore (pretest), San Francisco, Richmond, Austin,Boston and St. Louis were tested. Each iteration of testing sought toanswer three key questions: (1) Do participants understand the context,that is, why they are getting the notice and the implied motivation toread it? (2) Do participants understand the purpose and the main points?(3) Do participants understand the differences in sharing practices? Each

2. This model also has been used by the Internal Revenue Service, the Food and Nutrition Service,and the Department of Housing and Urban Development in developing some of their documents andcommunications (see Kleimann, Enlow, and Kleimann 2003; Kleimann, Kingsley, and Kleimann2004; Kleimann and Spears 2008).


session included multiple versions of a notice from fictional banks. Inthe Pretest there were two styles (prose and graphic) each with fourdesign variations (i.e., different arrangements of the information); inSessions 1 and 2 only one design was tested per session, but each designshowed three levels of information sharing by the banks; in Sessions 3,4 and 5 there were two design versions, each with three levels of sharing(Kleimann Communication Group 2006).

The testing looked at how an individual worked with the notice toaccomplish the task of choosing a bank based solely on the notice. Theinterview consisted of three parts. In the first part, participants did a“think-aloud” as they read through the notice, articulating their “innermonologue” and their many small reactions as they read the document(Fonteyn, Kuipers, and Grobe 1993). The moderator asked no questionsso as not to bias the participant by drawing attention to any aspect of thenotice. This approach allowed researchers to see what consumers paidattention to, what information they ignored, what questions they hadand what they understood and misunderstood about the notice. In thesecond part of the interview, the researcher asked a series of structuredquestions to probe comprehension and design issues. In the third partof the interview, the researcher gave the participant two additionalnotices for banks with different sharing practices and asked a seriesof questions about comparisons among the three. The researcher thenasked participants to choose a bank and explain their rationale for thechoice.

Researchers did an immediate debriefing after each session of inter-views, and then transcribed each session, coding comments usinggrounded theory (Glaser and Strauss 1967). Researchers listened toconsumer interpretations and unexpected reactions, and linked these toinformation design and processing theories. After each session of testing,Kleimann staff met with the agencies to report findings, discuss consensusdecisions and ensure the accuracy of the disclosures; then the notice con-tent and design were revised for further testing. Several times during thisstage the Kleimann staff and the agency staff met with senior-level stafffrom across the agencies to update them on the progress of the project.The data were compelling—watching the form evolve and “hearing” thewords of consumers were essential to maintain the commitment of allinvolved to the research process and the results.

In the Deliver stage, the form was delivered to the agencies for sub-sequent Evaluation (stage six) of the form’s effectiveness. The agenciesused a mall-intercept survey method to evaluate the effectiveness of thenew notice relative to three alternative formats; 1,032 consumers were


surveyed at locations in California, Massachusetts, Maryland, Michiganand Texas.

RESULTS

Develop

At the outset of the project, the typical privacy notice lookedsomething like the form in Figure 2. Both the focus groups and preferencetesting showed that consumers had little prior understanding of thevarieties of information-sharing practices of financial institutions. It wasclear that the notice needed an educational component to successfullyengage consumers and explain the relevance and significance of theinformation.

The key challenge was how to select and organize the content ofthe notice. For example, to motivate participants to pay attention toand read the notice, the researchers tested various titles. The use of thephrase “Privacy Policy” or “Privacy Notice” led consumers to assumethat all policies were the same and that they would never have any choiceabout the sharing practices, nor would these notices require any actionon the consumer’s part. Consumers who saw these titles admitted thatthey would not even bother to read the notice.

Assess, Revise and Test

Testing revealed that people needed to understand what “sharingof information” really meant; in other words, there needed to bean educational component to support understanding. The researchersworked with examples of types of personal information that might beshared and examples of everyday business practices that might requiresharing of personal information. The goal was to provide a frame ofreference to emphasize the relevance and significance of the disclosuretable. Placement of this information evolved to the top of the form.The headings to mark the different kinds of background informationbegan as complete sentences and evolved to the familiar words of“why,” “what” and “how” to stimulate a willingness to read the notice(Table 1).

Plain language was important but consumers’ reactions to the wordingwere more so. For example, the term “normal business purposes”(included in the regulation) was not as meaningful to consumers as“everyday business purposes.” The title was changed from a statement toa question in order to motivate consumers to read the document. The fact


FIGURE 2Current Notice (also Test Notices for Quantitative Study—Current Notice)

that federal law required these notices to be sent was important to theagencies but less so to consumers, so the statements about federal lawmoved from the top to a line within the text. Eventually, information suchas definitions and additional information required by law, which testingshowed was helpful but not essential to consumers’ understanding, wasplaced on page two (see Figure 3).


FIGURE 2Continued

To address the key information about sharing practices, a varietyof designs were developed and tested to meet the goal of comparisonacross notices. Ultimately a table was used to structure the description ofinformation-sharing practices. The table also evolved during the courseof testing, with the labels of the columns changing the most. In thelast three sessions of testing, the table version consistently outperformeda non-tabular prose version of the notice in terms of ease of use,comprehension and consumers’ ability to compare information-sharingpractices.

By the final session of testing, consumers could identify and distin-guish among the sharing practices of the various fictional banks. Theycould process the information using page one alone, although they appre-ciated the supplemental information on page two.


TAB

LE

1E

xam

ples

ofC

hang

esM

ade

Dur

ing

Iter

ativ

eTe

stin

g

Sess

ion

Titl

eL

ayou

tFr

ame

Plai

nL

angu

age

Exa

mpl

e“F

eder

alL

awR

equi

res”

Exa

mpl

e

Pret

est

Why

we

are

send

ing

this

notic

eG

raph

icim

ages

Four

disc

losu

res:

Wha

tpe

rson

alin

form

atio

nw

eco

llect

;H

oww

epr

otec

tin

form

atio

n;H

oww

esh

are

info

rmat

ion;

How

you

can

limit

wha

tis

done

with

your

info

rmat

ion

“To

dobu

sine

ss”

Top,

afte

rtit

le

1Fa

cts:

Use

ofyo

urpe

rson

alin

form

atio

nK

eyfr

ame

ques

tions

with

pros

ete

xtW

hat?

How

?W

hy?

Who

?W

hen?

“Nor

mal

busi

ness

purp

oses

”To

p,af

ter

title

2Fa

cts:

Use

ofyo

urpe

rson

alin

form

atio

nK

eyfr

ame

ques

tions

with

pros

ete

xtW

hat?

How

?W

hy?

Whe

n?W

ho?

“Nor

mal

busi

ness

purp

oses

”W

ithin

text

3Fa

cts:

Nep

tune

Ban

k’s

use

ofyo

urpe

rson

alin

form

atio

nK

eyfr

ame

ques

tions

with

tabl

eW

hy?

Wha

t?W

ho?

“Nor

mal

busi

ness

purp

oses

”W

ithin

bulle

tlis

t

4Fa

cts:

Wha

tdo

esN

eptu

neB

ank

dow

ithyo

urpe

rson

alin

form

atio

n?

Key

fram

equ

estio

nsw

ithta

ble

Why

?W

hat?

How

?“N

orm

albu

sine

sspu

rpos

es”

Firs

tse

nten

cew

ithin

text

5Fa

cts:

Wha

tdo

esN

eptu

neB

ank

dow

ithyo

urpe

rson

alin

form

atio

n?

Key

fram

equ

estio

nsw

ithta

ble

Why

?W

hat?

How

?“E

very

day

busi

ness

purp

oses

”Se

cond

sent

ence

with

inte

xt


FIGURE 3Example of Final Notice Form

Deliver

The final, two-page prototype notice that emerged from this processhad four main components—the title, the frame, the disclosure tableand, where needed, the opt-out information (Figure 3). After pretestingand five sessions of usability testing, the agencies could see that thenew notice was working effectively on all measures—comprehension,


FIGURE 3Continued

usability and comparability. In March 2006, the prototype form wasmade public, along with the research findings from the qualitativephase (Kleimann Communication Group 2006). The agencies briefedCongressional staff on the research project and the prototype form, andin October 2006, Congress explicitly directed the agencies to developand propose “a model form,” based on the agencies’ research, forvoluntary use by financial companies (see section 728 of the Financial


Services Regulatory Relief Act of 2006, Public Law No. 109-351,120 Stat.1966 (2006), amending GLBA). On March 21, 2007, theagencies jointly proposed for public comment the prototype notice as amodel form. The notice was ready for further evaluation by quantitativetesting.

Evaluate

The objective of the evaluation stage was to assess the effectivenessof the new notice against three alternative notices. The study plan andquestionnaire were developed by Levy and Hastak (2009), which wasimplemented as a mall-intercept study using one-on-one interviews. Thestudy sought to compare the notices in terms of their ability to helpconsumers compare banks’ information collection and sharing practices,evaluate available opt-out choices described in the notices, and makeinformed and reasoned choices between banks.

Macro International collected the data using computer-assisted per-sonal interviews in shopping malls in five locations across the UnitedStates during March and April 2008 (Macro International 2008). Respon-dents were screened to ensure that they were eighteen years of age orover, fluent in reading and speaking English and not an employee ofa bank or other financial institution. The sample of 1,032 respondentsbroadly reflected the U.S. population based on age and gender, as wellas education, race/ethnicity and income.

Four different privacy notice styles were evaluated in the study (seeTable 2; Figures 2, 4, 5 and 6). The Kleimann Table Notice (Figure 4)reflected the prototype notice developed by the project team. TheKleimann Prose Notice (Figure 5) was identical to the Kleimann TableNotice, except it replaced the table on page one with a bulleted listthat contained the same information. The Current Notice (Figure 2) wascreated to represent typical notices provided to consumers in 2008. Thisnotice included a strong opening statement assuring consumers that thebank secures and protects consumers’ personal information; “highlights”or summary of the bank’s policy; and details of the bank’s informationcollection and sharing, and, where needed, opt-out choices. The SampleClause Notice (Figure 6) was composed of the sample clauses thatappeared in the agencies’ original regulations implementing the GLBArequirements.

Privacy notices for three fictional banks—Mars, Mercury and Neptune,each with a different level of sharing—were created for each of thefour notice types, for a total of twelve privacy notices (three levels of


TAB

LE

2L

evel

sof

Shar

ing

Test

edin

Qua

ntit

ativ

eE

valu

atio

n

Mar

saM

ercu

ryN

eptu

ne

Rea

sons

We

Can

Shar

eY

our

Pers

onal

Info

rmat

ion

Doe

sM

ars

Ban

kSh

are?

Can

You

Lim

itT

his

Shar

ing?

Doe

sM

ercu

ryB

ank

Shar

e?

Can

You

Lim

itT

his

Shar

ing?

Doe

sN

eptu

neB

ank

Shar

e?

Can

You

Lim

itT

his

Shar

ing?

For

our

ever

yday

busi

ness

purp

oses

—to

proc

ess

your

tran

sact

ions

,m

aint

ain

your

acco

unt,

and

repo

rtto

cred

itbu

reau

s

Yes

No

Yes

No

Yes

No

For

our

mar

keti

ngpu

rpos

es—

toof

fer

our

prod

ucts

and

serv

ices

toyo

uY

esN

oY

esN

oY

esN

o

For

join

tm

arke

ting

wit

hot

her

finan

cial

com

pani

esN

oW

edo

n’t

shar

eN

oW

edo

n’t

shar

eY

esN

o

For

our

affil

iate

s’ev

eryd

aybu

sine

sspu

rpos

es—

info

rmat

ion

abou

tyo

urtr

ansa

ctio

nsan

dex

peri

ence

s

No

We

don’

tsh

are

Yes

No

Yes

No

For

our

affil

iate

s’ev

eryd

aybu

sine

sspu

rpos

es—

info

rmat

ion

abou

tyo

urcr

editw

orth

ines

s

No

We

don’

tsh

are

Yes

Yes

Yes

Yes

For

our

affil

iate

sto

mar

ket

toyo

uN

oW

edo

n’t

shar

eN

oW

edo

n’t

shar

eY

esY

esF

orno

naffi

liate

sto

mar

ket

toyo

uN

oW

edo

n’t

shar

eN

oW

edo

n’t

shar

eY

esY

es

a Mar

sB

ank

shar

edth

ele

ast,

Nep

tune

Ban

kth

em

ost,

and

Mer

cury

was

inbe

twee

n.If

cons

umer

sex

erci

sed

all

the

opt

outs

offe

red

byM

ercu

ryan

dN

eptu

ne,

this

rank

orde

rin

leve

lof

shar

ing

wou

ldst

illre

mai

n.


FIGURE 4Test Notices for Quantitative Study—Kleimann Table Notice

sharing for each of the four notice styles; see Table 3).3 Mars Bankshared the least, Neptune Bank the most and Mercury was in between.If consumers exercised all the opt-outs offered by Mercury and Neptune,this rank order in level of sharing would still remain. The four noticeformats were assessed using measures of judgment quality and perceptualaccuracy.

3. Respondents were randomly assigned to see one of the four notice styles; they saw noticesfor all three banks (all three levels of sharing) during the interview sequence. The sequence of thenotices was rotated among the participants (additional details on the research method and procedurescan be found in Levy and Hastak 2009).


FIGURE 4Continued

To assess judgment quality, the consistency of choices made byrespondents (i.e., whether they chose a lower-sharing or a higher-sharing bank) and the reasons provided for that choice were examined.Respondents who saw the Kleimann Table Notice were significantlymore likely than respondents who saw the other notice styles to givecorrect fact-based reasons for selecting the bank they did (Levy andHastak 2009). For example in Table 4, 41% of respondents exposed tothe Kleimann Table Notice chose the lower-sharing bank and gave areason that was a true statement about which bank shared less, whilethe corresponding percentages for the other three notice styles were


FIGURE 5Test Notices for Quantitative Study—Kleimann Prose Notice

significantly lower (21% for the Kleimann Prose Notice, 12% for theCurrent Notice and 26% for the Sample Clause Notice). Respondentswho saw the Kleimann Table Notice were also significantly less likelythan respondents who saw other notice styles to give reasons based onsubjective evaluations of the notice rather than on bank sharing practices(10% vs. 16%–25%, respectively). In contrast, respondents who saw theCurrent Notice were significantly more likely than respondents who sawother notice styles to give reasons not based on any information in thenotice (30% vs. 15%–23%, respectively), and they were also significantlyless likely to give correct fact-based reasons for their choice.4

4. Additional detailed results and statistical analysis are available in Levy and Hastak 2009.


FIGURE 5Continued

To assess perceptual accuracy, four questions covered: (1) the infor-mation collection practices, (2) information-sharing practices, (3) opt-outchoices provided by the banks and (4) level of sharing after consumersopted out. Overall, the Kleimann Table Notice was significantly betterthan the Current Notice on three measures (collecting, sharing and shar-ing after opt out), significantly better than the Kleimann Prose Notice ontwo of the three measures, and significantly better than the Sample ClauseNotice on one of the three measures. For example, one question askedrespondents if one of the two banks collected more personal informationor if both banks collected the same amount of information. The accuracyscore for this question was significantly higher for respondents who sawthe Kleimann Table Notice than for those who saw the Kleimann Prose


FIGURE 6Test Notices for Quantitative Study—Sample Clause Notice

Notice and the Current Notice; there was no statistical difference withthe Sample Clause Notice. The Kleimann Table Notice outperformed allthe other notices on the most difficult question that asked respondents tocompare the sharing level of the banks after all available opt outs hadbeen exercised.


TAB

LE

3D

escr

ipti

onof

Not

ices

Test

ed(E

xam

ples

ofE

lem

ents

inth

eN

otic

es)

Not

ice

Styl

eT

itle

Fram

eD

iscl

osur

eIn

form

atio

nE

xerc

isin

gO

ptO

ut

KC

Gta

ble

notic

eFa

cts:

Wha

tdo

es[t

heba

nk]

dow

ithyo

urpe

rson

alin

form

atio

n?

Why

?W

hat?

How

?Ta

ble

Opt

-out

box

KC

Gpr

ose

notic

eFa

cts:

Wha

tdo

es[t

heba

nk]

dow

ithyo

urpe

rson

alin

form

atio

n?

Why

?W

hat?

How

?C

anyo

ulim

it?B

ulle

ted

list

with

pros

eO

pt-o

utbo

x

Cur

rent

notic

ePr

ivac

ypo

licy

• Str

ong

open

ing

stat

emen

tab

out

trus

tan

dre

liabi

lity

• Hig

hlig

hts

• How

we

use

your

info

rmat

ion

• Pro

tect

ing

your

info

rmat

ion

• You

have

ach

oice

Pros

ePr

ose

Sam

ple

clau

seno

tice

Priv

acy

polic

y• I

nfor

mat

ion

that

we

colle

ctfr

omyo

uPr

ose

Pros

e

• Inf

orm

atio

nw

edi

sclo

se• O

pt-o

utch

oice

s• P

rote

ctin

gno

npub

licpe

rson

alin

form

atio

n


TAB

LE

4D

istr

ibut

ion

ofR

easo

nsfo

rC

hoic

eby

Not

ice

Styl

e(f

rom

Tabl

e1

inL

evy

and

Has

tak

2009

)

Not

ice

Styl

eT

rue

Low

-Sha

ring

Tru

eO

ptO

utIn

vent

edN

otic

eFa

lse

Shar

ing

Fals

eO

ptO

utN

oPr

efer

ence

KC

Gta

ble

notic

e40

.6%

a15.

1%c

16.5

%ab

10.2

%a

3.5%

a4.

3%b

18.9

%ab

2

KC

Gpr

ose

notic

e21

.1b

10.7

ab22

.6b

15.7

b3.

8a3.

8b21

.1ab

Cur

rent

notic

e12

.0c

7.8b

c29

.8c

16.7

b2.

3a1.

2b24

.8b

Sam

ple

clau

seno

tice

25.9

b12

.7a

15.1

a24

.7c

4.2a

0.8a

15.1

a

Not

es:

True

low

-sha

ring

.A

true

stat

emen

tab

out

the

rela

tive

shar

ing

char

acte

rist

ics

ofth

etw

oba

nks

whe

nch

oosi

ngth

elo

wer

-sha

ring

bank

.Tr

ueop

tou

t.A

true

stat

emen

tab

out

whi

chba

nkof

fere

dm

ore

opt

outs

asa

reas

onto

choo

seth

ehi

gher

-sha

ring

bank

.In

vent

ed.

Ast

atem

ent

abou

tth

ere

lativ

ech

arac

teri

stic

sof

the

chos

enba

nkth

atw

asno

tba

sed

onan

ythi

ngin

the

priv

acy

notic

e(e

.g.,

Ban

kX

offe

rslo

wer

inte

rest

rate

s).

Not

ice.

An

opin

ion

abou

tth

epr

ivac

yno

tice

with

nore

latio

nshi

pto

bank

shar

ing

char

acte

rist

ics

(e.g

.,I

thou

ght

Ban

kX

priv

acy

notic

ew

ascl

eare

r).

Fal

sesh

arin

g.A

fals

est

atem

ent

abou

tth

ere

lativ

esh

arin

gch

arac

teri

stic

sof

the

two

bank

s(e

.g.,

Ich

ose

Ban

kX

,th

ehi

gher

-sha

ring

bank

,be

caus

eit

shar

esle

ssth

anB

ank

Y).

Fal

seop

tou

t.A

fals

est

atem

ent

abou

tw

hich

bank

offe

rsm

ore

opt

outs

.N

oP

refe

renc

e.N

opr

efer

ence

expr

esse

d.1C

ells

with

ina

colu

mn

that

dono

tsh

are

the

sam

ele

tter

are

sign

ifica

ntly

diff

eren

tfr

omon

ean

othe

r,p

<.0

5.2Pe

rcen

tage

sac

ross

aro

wm

ayno

tsu

mto

100%

beca

use

som

ere

spon

dent

sga

veno

reas

onfo

rth

eir

choi

ceor

are

ason

that

was

code

d“o

ther

.”


The key feature responsible for improved performance in the KleimannTable Notice seems to be the table that arrayed the several possiblekinds of information sharing that can take place and identified the onesengaged in by the particular institution. Providing a fuller context forthe disclosure of information sharing seems to help consumers focuson information sharing as an important and differentiating feature offinancial institutions.

Feedback and Supplemental Testing

Upon review of the quantitative research findings and public commentssubmitted on the proposed notice, the agencies decided to move theinformation on how to opt out to page one, thereby consolidating allthe information in the notice on the front and back sides of a singlepiece of paper. The agencies contracted with Kleimann to conductadditional testing of this revision to ensure that the notice wouldcontinue to rate high on comprehension. Kleimann used the sametesting approach as for the initial prototype, but with a more targetedfocus, testing the placement and content of the opt-out information(Kleimann Communication Group 2009). In this phase of testing, it wasclear that placing the opt-out information on the front of the noticewas an improvement for consumers. Results confirmed that the finalnotice provided clear, accurate information in a way that consumers canunderstand and use. The final privacy notice was unveiled in December2009 (74 FR 62890, December 1, 2009; see Figure 3).

DISCUSSION AND CONCLUSIONS

Rigorous consumer testing requires a solid research design and specificgoals for consumers: what do policymakers want consumers to know ordo as a result of reading the disclosure? Is the disclosure simply to informconsumers or to encourage them to take action as a result of reading theinformation?

Our objective for this project was to design a notice that satisfiesboth the spirit and the letter of the intended disclosure—one thatis understandable and useful to consumers and that complies withthe information requirements of the law. The process of combiningpolicy-minded researchers at Kleimann Communication Group andresearch-minded policymakers at the agencies demonstrated the potentialinherent in this kind of effort: a well-designed disclosure that improvesconsumer comprehension levels was accepted by the policy community


both in the agencies and in Congress, and was accepted by financialinstitutions that recognize the importance and merit of research-basedpolicy recommendations.

Lessons Learned

As the research team worked with consumers across the variousiterations of the notice, it was very easy to tell what parts of the noticewere comprehensible and where there continued to be problems. Forexample, plain language is important, and simple words and phrasesgenerally are good. But it is also important to gauge consumers’ reactionsto those words to make sure they are not misunderstood, misinterpretedor misleading. In this project, policymakers had to “let go” of some ofthe technically precise, but confusing, language of the law in favor oflanguage that consumers found more accessible and understandable. Ifthe information in the notice or disclosure is not commonly known orunderstood by consumers, then the notice needs to provide a context—aneducational component—to motivate consumers to read the notice andhelp them understand what it is about, why it is important and how touse the information in their decision making.

Principles of information design, plain language and the right structuralformat can have a significant impact on consumers’ comprehension andform usability. In this particular project, presenting information aboutsharing practices in a table was a key element of success. Tables havebeen used in a variety of other consumer information situations such asnutrition labeling, credit card disclosures and mortgage loans. However, atable does not ensure comprehension; poorly designed tables may createa visual complexity that impedes rather than enhances comprehension.Furthermore, tables are not always the right design tool (Keitel 2010). Infact, contextual framing, visual hierarchy, information sequencing, fontchoice, white space, and many other design features are critical factorsin the success of consumer disclosures.

Layering of information needs to be done carefully and thoughtfully.Researchers and policymakers need to identify the key pieces ofinformation that are essential to consumer understanding and usabilityand make sure those pieces are located together in a highly visible placeand format. Consumers in this study tended to use the information onpage one, with minimal references to the information on page two.

One challenge in determining the real-world impact of changes indisclosures is the problem of observing and evaluating decisions. In acontrolled environment, a researcher can control the options presented.


But, in practice, it is easier to obtain information about the choice aconsumer made ex post than it is to know the details of all the exante options they considered. It also can be difficult to distinguish a“good” decision from a “bad” one, especially if that evaluation dependson consumer-specific preferences or circumstances. A product or choicemay be a good fit for one consumer but a bad fit for another.

Being Consumer-CentricA key feature of the development process was a sustained commitment

to create notices that would enable consumers to make more informedchoices. One of the key principles in education is that effective educationneeds to be learner-driven (Marceau 2003); similarly, disclosures need tobe consumer-driven. It is important to understand and involve consumersin the process. It is also important to be clear about the goal of thedisclosure—whether it is simply to inform or to encourage recipients totake action as a result of reading the information.

Although some of the problems uncovered by consumer testing canbe resolved by design or language changes, others may remain unre-solved. In these cases, policymakers may make judgments informed bythe research thus far, or may determine that policy tools other than dis-closures are needed. Given the inherent limitations of disclosure, such asspace and design constraints, there is a role for supplementary strategiessuch as educational materials or substantive regulations (Hogarth andMerry 2011).

Along with being consumer-centric, policymakers need to be mindfulabout theoretical approaches to their issues—an understanding of theoriesof consumer motivation and behavior can help inform how informationis framed and how context can be set. It is important to appreciate thatdocuments such as privacy notices will not be studied by consumersmotivated to learn for the sake of learning, but instead will be read—ormore likely skimmed—by consumers who have other demands on theirtime and attention. The conception of the reader as one who wants to seea purpose for reading before making an effort to read and the decisionmaker as one who wants to accomplish a task with a minimum of effortand attention are fruitful starting points for designing more effectivedisclosures.

Involving Responsible Policy StaffInvolving consumers in the process of designing and developing

notices and disclosures has gained ground among both policymakersand law makers. Similarly, involving key staff with the research was


also valuable because it counteracted a common tendency among policyand subject-matter experts to rely too heavily on their own commonsense understanding of what consumers know, understand and need.Without direct exposure to actual consumers trying to understand anduse the disclosures, it is all too easy to let self-confidence in one’sown communication skills undercut the design of effective consumercommunications.

All decision making was by consensus among the agencies involved;thus, the research briefings that were part of the process were exception-ally important to informing the staff and senior management involvedand in guiding the ensuing discussions.

Using Multiple MethodsBy using multiple methods, including focus groups, preference testing,

cognitive interviews, usability testing and surveys, policymakers whosupervised the project were exposed to the consistency of findingsacross the various techniques and witnessed firsthand the cumulativeimprovements made in the notice. The quantitative research focusingon performance measures validated the improvements made in thequalitative phase and provided a convincing demonstration of thesuperiority of the new privacy notice compared with existing forms andother possible alternatives. The process created a sustained commitmentto thinking about how to improve notices and disclosures. The multipleskill sets brought to the table included communication, design, writing,qualitative analysis, research design and quantitative analysis; theseinsured that policymakers were exposed to the full set of issues involvedin designing more effective notices. Furthermore, the robustness of theresults was an effective counterweight to the usual tendency to focuson the legal rather than the communication requirements of the privacynotices.

Exploiting SynergiesAs we had to ensure that the model form was fully compliant with

fairly complex laws, the final form contained a lot of information.Through the consumer-focused research and testing process, we wereable to prioritize the information for comprehension and usability that ismost critical and place that information in an accessible format on thefirst page of the notice. However, the agencies and contractor still neededto operate within the confines of existing statutes and rules.

Legislative and regulatory processes raise and solve problems withsignificant behavioral content. Often, however, legislators and regulators


do not seek guidance from research to address common sense behavioralissues with empirically driven, evidence-based research. Behavioralconcepts, such as “informed financial decisions” and “understanding,”demand clarification and research-based measurement. Furthermore, lawsand regulations often over-specify content, and this content choice canbe unrelated to what matters to consumers or what they find significant.Too frequently, “research” takes place entirely after the first round oflegislative and regulatory processes have been completed.

In retrospect, our results are also consistent with the sets of principlesfor summary disclosures and full disclosures more recently outlined bythe Office of Management and Budget (Sunstein 2010). These include:(1) identify the goals for the disclosure, (2) be simple and specific andavoid undue detail or excessive complexity, (3) be accurate and inplain language, (4) be properly placed and timed, (5) use meaningfulratings and scales where applicable, (6) test the effects of disclosuresand monitor effects over time and (7) identify and consider the costsand benefits of disclosure requirements. Subsequently, Congress hasinstitutionalized consumer testing for consumer financial disclosures as aregular part of the rulemaking process within the Consumer FinancialProtection Bureau (Dodd-Frank Wall Street Reform and ConsumerProtection Act, PL 111-203).

Coordinating the goals and expertise of policymakers and researchersis hard work, but essential to achieving better policy outcomes. Thisstudy reinforces this point by highlighting a case study and showing itcan be done.

REFERENCES

Anderson, Richard C., and P. David Pearson. 1984. A Schema-Theoretic View of Basic Processesin Reading Comprehension. In Handbook of Reading Research, edited by P. David Pearson,Rebecca Barr, Michael L. Kamil, and Peter B. Mosenthal, vol. 1 (255–291). Mahwah, NJ:Erlbaum.

Bloom, Benjamin S. 1956. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain,New York: David McKay Co., Inc.

Brown, Stephen W., and Parks B. Dimsdale Jr. 1973. Consumer Information: Toward an Approachfor Effective Knowledge Dissemination. Journal of Consumer Affairs, 7 (1): 55–60.

Culnan, Mary J., and George R. Milne. 2001. The Culnan-Milne Survey on Consumers & OnlinePrivacy Notices: Summary of Responses. Presentation at Get Noticed: Effective Financial PrivacyNotices Interagency Public Workshop. http://www.ftc.gov/bcp/workshops/glb/supporting/culnan-milne.pdf.

DellaVigna, Stephano. 2009. Psychology and Economics: Evidence from the Field. Journal ofEconomic Literature, 47 (2): 315–372.

Durkin, Thomas A. 2006. Credit Card Disclosures, Solicitations, and Privacy Notices: Sur-vey Results of Consumer Knowledge and Behavior. Federal Reserve Bulletin, 92(August):A109–A121.


Durkin, Thomas A., and Gregory Elliehausen. 2011. Truth in Lending: Theory, History, and a WayForward. New York: Oxford University Press.

Few, Stephen. 2004. Show Me the Numbers: Designing Tables and Graphics to Enlighten. Oakland,CA: Analytics Press.

Fonteyn, Marsha E., Benjamin Kuipers, and Susan J. Grobe. 1993. A Description of Think AloudMethod and Protocol Analysis. Qualitative Health Research, 3 (November): 430–441,

Gilbride, Timothy J., and Greg M. Allenby. 2004. A Choice Model with Conjunctive, Disjunctive,and Compensatory Screening Rules. Marketing Science, 23 (3): 391–406.

Glaser, Barney, and Anselm Strauss. 1967. The Discovery of Grounded Theory: Strategies forQualitative Research. Chicago: Aldine Publishing Company.

Good, Nathaniel, Rachna Dhamiga, Jens Grossklags, David Thaw, Steven Aronowitz, Deirdre Mul-ligan, and Joseph Konstan. 2005. Stopping Spyware at the Gate: A User Study of Privacy, Noticeand Spyware. Proceedings of the Symposium on Useable Privacy and Security, Pittsburgh, PA.

Grossklags, Jens, and Nathaniel Good. 2007. Empirical Studies on Software Notices to Inform Poli-cymakers and Usability Designers. Proceedings of Usable Security. Scarborough, Trinidad/Tobago.

Haviland, Susan E., and Herbert H. Clark. 1974. What’s New? Acquiring New Information as aProcess in Comprehension. Journal of Verbal Learning and Verbal Behavior, 13 (5): 512–521.

Heroux, Lisa, Michel Larouche, and K. Lee McGown. 1988. Consumer Product Label InformationProcessing: An Experiment Involving Time Pressure and Distraction. Journal of EconomicPsychology, 9: 195–214.

Hibbard, Judith H., Paul Slovic, and Jacqueline J. Jewett. 1997. Informing Consumer Decisionsin Health Care: Implications from Decision-Making Research. The Milbank Quarterly, 75 (3):395–414.

Hogarth, Jeanne M., and Ellen A. Merry. 2011. Designing Disclosures to Inform Consumer Finan-cial Decisionmaking: Lessons Learned from Consumer Testing. Federal Reserve Bulletin, 97(3). http://www.federalreserve.gov/pubs/bulletin/2011/pdf/designingdisclosures2011.pdf.

Ippolito, Pauline M. 1986. Consumer Protection Economics: A Selective Survey. In ConsumerProtection Economics, edited by Pauline M. Ippolito and David T. Scheffman. Washington DC:Federal Trade Commission.

———. 1988. The Economics of Information in Consumer Markets: What Do We Know? What DoWe Need to Know? In The Frontiers of Research in the Consumer Interest, edited by E. ScottMaynes (235–263). Columbia, MO: American Council on Consumer Interests.

Jensen, Carlos, and Colin Potts. 2004. Privacy Policies as Decision Making Tools: An Evaluation ofOnline Privacy Notices. Proceedings of the Conference on Human Factors in Computing Systems.Vienna, Austria: Association for Computing Machinery.

Jensen, Carlos, Colin Potts, and Christian Jensen. 2005. Privacy Practices of Internet Users: Self-reports versus Observed Behavior. International Journal of Human-Computer Studies, 63 (1–2):203–227.

Kay, Matthew, and Michael Terry. 2009. Textured Agreements: Re-envisioning Electronic Consent(Technical Report CS-2009-19). David R. Cheriton School of Computer Science, University ofWaterloo.

Keitel, Philip. 2010. Consumer Testing Informs Policy: Overdraft Regulation as a Case Study.Philadelphia Federal Reserve Bank Payment Cards Center Discussion Paper DP 10-03.http://www.philadelphiafed.org/payment-cards-center/publications/discussion-papers/2010/D-2010-June-Overdrafts.pdf.

Keller, Kevin L., and Richard Staelin. 1987. Effects of Quality and Quantity of Information onDecision Effectiveness. Journal of Consumer Research, 14 (2): 200–213.

Kelley, Patrick G., Joanna Bresee, Lorrie Faith Cranor, and Robert W. Reeder. 2009a. A “NutritionLabel” for Privacy. Proceedings of the Symposium on Usable Privacy and Security. MountainView, CA.

Kelley, Patrick G., Lucian Cesca, Joanna Bresee, and Lorrie Faith Cranor. 2009b. StandardizingPrivacy Notices: An Online Study of the Nutrition Label Approach. Carnegie Mellon UniversityCyLab Paper 09-014. http://www.cylab.cmu.edu/files/pdfs/tech_reports/CMUCyLab09014.pdf.


Kimble, Joseph. 2006. Lifting the Fog of Legalese. Durham, NC: Carolina Academic Press.Kirsch, Larry. 2002. Do Product Disclosures Informs and Safeguard Insurance Policyholders?

Journal of Insurance Regulation, 20 (3): 271–296.Kirsch, Irwin, Ann Jungeblut, Lynn Jenkins, and Andrew Kolstad. 1993. Adult Literacy in America:

A First Look at the Results of the National Adult Literacy Survey. Washington, DC: NationalCenter for Educational Statistics.

Kleimann Communication Group, Inc. 2006. Evolution of a Prototype Financial Privacy Notice: AReport on the Form Development Project. http://www.ftc.gov/privacy/privacyinitiatives/ftcfinalreport060228.pdf.

. 2008. Putting Theory into Practice: A Best Practices Approach to Developing TaxInstructions. Washington, DC: Internal Revenue Service.

. 2009. Financial Privacy Notice: A Report on Validation Testing Results. http://www.ftc.gov/privacy/privacyinitiatives/validation.pdf.

Kleimann, Susan, and Eric Spears. 2008. Summary Report: Consumer Testing of the Good FaithEstimate (GFE). http://www.huduser.org/publications/pdf/Summary_Report_GFE.pdf.

Kleimann, Susan, Barbra Enlow, and Kristin Kleimann. 2003. Guide to Assessing Food StampApplications. Washington, DC: U.S. Department of Agriculture and Food and Nutrition Service.

Kleimann, Susan, Barbra Kingsley, and Kristin Kleimann. 2004. Voice of the American Taxpayer:Results of Testing on the CP2000 for the Internal Revenue Service. Washington, DC: InternalRevenue Service.

Lacko, James M., and Janice K. Pappalardo. 2007. Improving Consumer Mortgage Disclosures: AnEmpirical Assessment of Current and Prototype Disclosure Forms. http://www.ftc.gov/os/2007/06/P025505MortgageDisclosureReport.pdf.

Lee, Jinkook, and Jeanne M. Hogarth. 1999a. Returns to Information Search: Consumer MortgageShopping Decisions. Journal of Financial Counseling and Planning, 10 (1): 49–66.

. 1999b. The Price of Money: Consumers’ Understanding of APRs and Contract InterestRates. Journal of Public Policy and Marketing, 18 (Spring): 66–76.

Levy, Alan S., Sara B. Fein, and Raymond E. Schucker. 1996. Performance Characteristics of SevenNutrition Label Formats, Journal of Public Policy and Marketing, 15 (Spring): 1–15.

Levy, Alan S., and Manoj Hastak. 2009 Consumer Comprehension of Financial Privacy Notices: AReport on the Results of the Quantitative Testing. http://www.ftc.gov/privacy/privacyinitiatives/Levy-Hastak-Report.pdf.

Loibl, Cazilia, Soo Jyun Cho, Flirian Diekmann, and Marvin T. Batte. 2009. Consumer Self-Confidence in Searching for Information. Journal of Consumer Affairs, 43 (1): 26–55.

Macro International. 2007. Design and Testing of Effective Truth in Lending Disclosures.http://www.federalreserve.gov/dcca/regulationz/20070523/Execsummary.pdf.

. Mall Intercept Study of Consumer Understanding of Financial Privacy Notices: Method-ological Report. http://www.ftc.gov/privacy/privacyinitiatives/Macro-Report-on-Privacy-Notice-Study.pdf.

Madrian, Brigitte C., and Dennis F. Shea. 2001. The Power of Suggestion: Inertia in 401(k)Participation and Savings Behavior. Quarterly Journal of Economics, 116: 1149–1187.

Mandell, Lewis. 1973. Consumer Knowledge and Understanding of Consumer Credit. Journal ofConsumer Affairs, 7 (1): 23–36.

Marceau, Georges. 2003. Professional Development in Adult Basic Education. New Directions forAdult and Continuing Education. 2003 (98): 67–74.

McDonald, Alecia M., Robert W. Reeder, Patrick G. Kelley, and Lorrie Faith Cranor. 2009. AComparative Study of Online Privacy Policies and Formats. Proceedings of the PrivacyEnhancing Technologies Symposium. Seattle, WA.

Milne, George R., Mary J. Culnan, and Henry Greene. 2006. A Longitudinal Assessment of OnlinePrivacy Notice Readability. Journal of Public Policy and Marketing, 25 (2): 238–249.

Milne, George R., Lauren I. Labrecque, and Cory Cromer. 2009. Toward an Understanding of theOnline Consumer’s Risky Behavior and Protection Practices. Journal of Consumer Affairs, 43(3):449–473.


Milne, George R., Andrew J. Rohm, and Shalini Bahl. 2004. Consumers’ Protection of OnlinePrivacy and Identity. Journal of Consumer Affairs, 38 (2): 217–232.

Moorman, Christine. 1996. A Quasi Experiment to Assess the Consumer and Information Determi-nants of Nutrition Information Processing Activities: The Case of the Nutrition Labeling andEducation Act. Journal of Public Policy and Marketing, 15 (1): 28–44.

National Assessment of Adult Literacy. 2007. Literacy in Everyday Life: Results from the2003 National Assessment of Adult Literacy. http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007480.

Novemsky, Nathan, Ravi Dhar, Norbert Schwarz, and Itamar Simonson. 2007. Preference Fluencyin Choice. Journal of Marketing Research, 44: 347–356.

Payne, John, James R. Bettman, and Eric J. Johnson. 1994. The Adaptive Decision Maker.Cambridge, UK: Cambridge University Press.

Proctor, Robert W., M. Athar Ali, and Kim-Phuong L. Vu 2008. Examining Usability of WebPrivacy Policies. International Journal of Human-Computer Interaction, 24 (3): 307–328.

Redish, Janice C. 1989. Reading to Learn to Do. IEEE Transactions on Professional Communication,32 (4): 289–293.

Roe, Brian, Alan S. Levy, and Brenda M. Derby. 1999. The Impact of Health Claims on ConsumerSearch and Product Evaluation Outcomes: Results from FDA Experimental Data. Journal ofPublic Policy and Marketing, 18 (1): 89–105.

Russo, Jay E. 1988. Information Processing from the Consumer’s Perspective. In The Frontiersof Research in the Consumer Interest, edited by E. Scott Maynes (185–218). Columbia, MO:American Council on Consumer Interests.

Saw, James T. 2000. Gestalt. http://daphne.palomar.edu/design/gestalt.html.Simonson, Itamar, and Steven M. Nowlis. 2000. The Role of Explanations and Need for Uniqueness

in Consumer Decision Making: Unconventional Choices Based on Reasons. Journal of ConsumerResearch, 27: 49–68.

Song, Hyunjin, and Norbert Schwarz. 2008. Processing Fluency Affects Effort Prediction andMotivation. Psychological Science, 19: 986–988.

Stigler, George J. 1961. The Economics of Information. The Journal of Political Economy, 69 (3):213–225.

Sunstein, Cass. 2010. Memorandum on Disclosure and Simplification as Regulatory Tools.Washington, DC: Office of Management and Budget. http://www.whitehouse.gov/sites/default/files/omb/assets/inforeg/disclosure_principles.pdf.

Swaminathan, Vanitha. 2003. The Impact of Recommendation Agents on Consumer Evaluation andChoice: The Moderating Role of Category Risk, Product Complexity, and Consumer Knowledge.Journal of Consumer Psychology, 13 (1 and 2): 93–101.

Tufte, Edward R. 1990. Envisioning Information. Cheshire, CT: Graphics Press.Virzi, Robert. 1992. Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?

Human Factors, 34: 457–486.Viswanathan, Madhubalan, and Manoj Hastak. 2002. The Rose of Summary Information in

Facilitating Consumers’ Comprehension of Nutrition Information. Journal of Public Policy &Marketing, 21, 2 (Fall): 305–318.

Wogalter, Michael S., David M. DeJoy, and Kenneth R. Laughery, eds. 1999. Warnings and RiskCommunication. Padstow, UK: T.J. International Ltd.

Documents

Designing Evidence-based Disclosures: A Case Study of Financial Privacy Notices