Technological Innovations and Tools to Technological Innovations and Tools to Enable Public Participation and Manage Enable Public Participation and Manage

Public CommentsPublic Comments

Jamie Callan Jamie Callan —— Carnegie Mellon UniversityCarnegie Mellon UniversityEduard Eduard HovyHovy —— USC/Information Sciences InstituteUSC/Information Sciences Institute

Stuart Shulman Stuart Shulman —— Drake University Drake University Stephen Zavestoski Stephen Zavestoski —— University of San Francisco University of San Francisco

May 28, 2004May 28, 2004

AcknowledgementsAcknowledgementsThis research is supported by exploratory grants from This research is supported by exploratory grants from the National Science Foundationthe National Science Foundation

EIAEIA--0089892 0089892 SGER Citizen AgendaSGER Citizen Agenda--Setting in the Regulatory Process: Electronic Setting in the Regulatory Process: Electronic Collection and Synthesis of Public CommentaryCollection and Synthesis of Public Commentary

EIA 0327979,EIA 0327979, 0328175, 0328914 & 03286180328175, 0328914 & 0328618SGER Collaborative: A Testbed for eRulemaking DataSGER Collaborative: A Testbed for eRulemaking Data

Any opinions, findings and conclusions or Any opinions, findings and conclusions or recommendations expressed in this material are those recommendations expressed in this material are those of the authors and do not necessarily reflect those of of the authors and do not necessarily reflect those of the National Science Foundationthe National Science Foundation

Scientific Research ObjectivesScientific Research Objectives

Applied Research ObjectivesApplied Research ObjectivesHelp agencies handle the information flood Help agencies handle the information flood Measure efficiency & quality improvementsMeasure efficiency & quality improvements

Basic Research ObjectivesBasic Research ObjectivesAdvance NLP & IR Advance NLP & IR

Analyze and categorize floods of text according to several Analyze and categorize floods of text according to several novel dimensions: stakeholders, opinions, argumentsnovel dimensions: stakeholders, opinions, arguments

Advance social science methods for measuring the Advance social science methods for measuring the impact of IT on democracyimpact of IT on democracy

Develop metrics for assessing the quality and content of the Develop metrics for assessing the quality and content of the public commentpublic commentDevelop a public comment database that facilitates greater Develop a public comment database that facilitates greater empirical social science researchempirical social science research

May 2001

Council for Excellence in Government

June 2002

National Defense University

Washington Workshops Build Partnerships withPractitioners and Developers in Federal Agencies

Accomplishments: Text TestbedAccomplishments: Text Testbed

USDA National Organic ProgramUSDA National Organic Program20,936 public comments (ASCII text)20,936 public comments (ASCII text)

USDOT HoursUSDOT Hours--ofof--Service RegulationsService Regulations22,954 public comments, 356 related documents, and 2 docket 22,954 public comments, 356 related documents, and 2 docket descriptor files (OCR PDF)descriptor files (OCR PDF)

USDOT Corporate Average Fuel Economy StandardsUSDOT Corporate Average Fuel Economy Standards18,407 public comments, 42 related documents, and 2 docket 18,407 public comments, 42 related documents, and 2 docket descriptor files (OCR PDF)descriptor files (OCR PDF)

More datasets are available & in the pipelineMore datasets are available & in the pipelineAll available to the Digital Government research communityAll available to the Digital Government research community

FullFull--text search & bitext search & bi--gram counts are upgram counts are upExpanded functionality over timeExpanded functionality over time

http://http://hartford.lti.cs.cmu.edu/ERulemaking/Data.htmlhartford.lti.cs.cmu.edu/ERulemaking/Data.html

Technology: An Overview Technology: An Overview Our objectivesOur objectives

Help agencies handle the Help agencies handle the information floodinformation floodSince you can’t stop the tide of electronic commentary!Since you can’t stop the tide of electronic commentary!

Help produce more Help produce more defensible regulationsdefensible regulationsUse information technology to weave regulations and Use information technology to weave regulations and public comments more tightlypublic comments more tightly

What followsWhat follows……A series of proposed functions/modules: A series of proposed functions/modules:

easy (= almost ready now) easy (= almost ready now) →→ hard (= needs research) hard (= needs research) Thoughts on integrating & displaying them Thoughts on integrating & displaying them

Language TechnologiesLanguage TechnologiesComputers can sometimes do useful things with Computers can sometimes do useful things with human language …even though they don’t human language …even though they don’t understand the language like we do: understand the language like we do:

Web search enginesWeb search enginesSimple voice recognition on cell phonesSimple voice recognition on cell phonesMonitoring newswires for “interesting” eventsMonitoring newswires for “interesting” eventsCreating searchable databases from online job postingsCreating searchable databases from online job postings

The systems that do these tasks are not perfectThe systems that do these tasks are not perfectBut accuracy is often comparable to ordinary peopleBut accuracy is often comparable to ordinary peopleThey enable people to work with far more information They enable people to work with far more information

E.g., carpenters and power sawsE.g., carpenters and power saws

Simple Statistical TechniquesSimple Statistical TechniquesRecognize words and phrasesRecognize words and phrases

Handle morphology (e.g., plurals, past & present tense, etc)Handle morphology (e.g., plurals, past & present tense, etc)Recognize parts of speech (e.g., nouns and verbs)Recognize parts of speech (e.g., nouns and verbs)

Count how often things occurCount how often things occurIn generalIn generalOr as compared to a reference model (e.g., general English)Or as compared to a reference model (e.g., general English)

Count how often two things occur togetherCount how often two things occur togetherIn general, or as compared to a reference modelIn general, or as compared to a reference model

Measure similarity of documents / paragraphs / sentencesMeasure similarity of documents / paragraphs / sentencesBased on the words they have in commonBased on the words they have in common

Sort things into categories (based on similarity)Sort things into categories (based on similarity)Recognize simple syntactic relationships and patternsRecognize simple syntactic relationships and patterns

An Analyst’s WorkbenchAn Analyst’s WorkbenchFile Tools Help

OpenUSDA 02-032-2 (Solid Wood Packing Material)USDA TMD-94-00-2 (National Organic Program)USDOT 2350 (Hours of Service)USDOT 11419 (Corporate Average Fuel Economy)

New Ctrl+NOpen Ctrl+OCloseSave Ctrl+SSave AsPrint Ctrl+PExit

FileBasic StatisticsDuplicate DetectionFrequent ConceptsStakeholders

Tools

Basic StatisticsFirst comment date: 12/15/1997Last comment date: 05/01/1998Number of comments: 20,936 Average comment length: 176 words

12/15/1997 05/01/1998By Week

Submission Pattern

Public Comment: w0000050Public Comment: w0000050received date = 12/16/1997received date = 12/16/1997name = *******, ******name = *******, ******company = company = state = GAstate = GAzip = 30295zip = 30295country = United Statescountry = United Statescategory = Consumercategory = Consumertopic = Applicabilitytopic = Applicabilitysection = 205.3section = 205.3comment = The term organic should never include comment = The term organic should never include

genetic engineering or genetic engineering or irridiationirridiation. Please change . Please change your rules to make this absolutely clear. These your rules to make this absolutely clear. These are totally unacceptable practices.are totally unacceptable practices.

USDA National Organic Program, Docket Number: TMD-94-00-2

Problem: Duplicate DetectionProblem: Duplicate DetectionMany public comments are form letters or edited form lettersMany public comments are form letters or edited form letters

(Real grassroots or (Real grassroots or astroturfastroturf created by interest groups, lobbies)created by interest groups, lobbies)

http://www.pcworld.com/news/article/0,aid,110305,00.asp

Duplicate Detection SolutionsDuplicate Detection SolutionsDuplicate detection algorithmsDuplicate detection algorithms

Generate summary countsGenerate summary countsIdentify the reference copyIdentify the reference copySummarize differences from reference copySummarize differences from reference copy

NearNear--duplicate detection techniquesduplicate detection techniquesUse cosine correlation to identify similar documentsUse cosine correlation to identify similar documentsIdentify nearIdentify near--duplicates using document “fingerprints”duplicates using document “fingerprints”

Sequences of words that match in each documentSequences of words that match in each document

OutputOutputA reliable and easy count of duplicatesA reliable and easy count of duplicatesUnique passages isolated and displayed/clusteredUnique passages isolated and displayed/clustered

Near Duplicate Detection ExampleNear Duplicate Detection Example

Reference comment Reference comment w0011812w001181241 exact duplicates41 exact duplicates11 near duplicates (nothing deleted, additional 11 near duplicates (nothing deleted, additional comments appended)comments appended)

ww00122810012281: “Please don't mess up good food!!”: “Please don't mess up good food!!”W0015812: “… Municipal W0015812: “… Municipal biosolidsbiosolids can contain heavy metals can contain heavy metals and toxins. I spend extra money to buy organic food because I and toxins. I spend extra money to buy organic food because I want to get food that is healthy for my family. I do not considewant to get food that is healthy for my family. I do not consider r heavy metals and toxins to be healthy. …”heavy metals and toxins to be healthy. …”W0017150: “I work frequently with our municipal, scientific anW0017150: “I work frequently with our municipal, scientific and d small business communities. I believe it is economically and small business communities. I believe it is economically and scientifically practical to use NONscientifically practical to use NON--municipal municipal biosolidsbiosolids for soil for soil fertility and crop nutrient management.”fertility and crop nutrient management.”w0017259: “… Municipal w0017259: “… Municipal biosolidsbiosolids (sledge) are currently (sledge) are currently banned in San Joaquin County where this ranch is located. …”banned in San Joaquin County where this ranch is located. …”

Public Comment Example (cont.)Public Comment Example (cont.)

received date = 12/16/1997received date = 12/16/1997name = *******, ******name = *******, ******company = company = state = GAstate = GAzip = 30295zip = 30295country = United Statescountry = United Statescategory = Consumercategory = Consumertopic = Applicabilitytopic = Applicabilitysection = 205.3section = 205.3comment = The term organic should never include comment = The term organic should never include

genetic engineering or genetic engineering or irridiationirridiation. Please change . Please change your rules to make this absolutely clear. These are your rules to make this absolutely clear. These are totally unacceptable practices.totally unacceptable practices.

An Analyst’s WorkbenchAn Analyst’s WorkbenchFile Tools HelpBasic StatisticsDuplicate DetectionFrequent ConceptsStakeholders

Tools

Frequent Concepts (Noun Phrases)

364animal products581food industry365animal cannibalism598organic food365factory farming759sewer sludge366growth hormones803national organic standards369crop production817organic food production act417livestock production837organic foods production act421livestock feed851nosb recommendations432united states909national organic program522dear ms945food supply524national organic production rule1,015food products535federal register1,187food irradiation554organic foods1,897food production564term effects2,855national list568genetic engineering4,044national organic standards board572docket #tmd5,590sewage sludge

CountPhraseCountPhrase

Public Comment: w0010435Public Comment: w0010435received date = 04/03/1998received date = 04/03/1998name = *****, *****name = *****, *****company =company =state = NJstate = NJzip = 07739zip = 07739country = United Statescountry = United Statescategory = Consumercategory = Consumertopic = Generaltopic = Generalcomment = As a mother and consumer of organic comment = As a mother and consumer of organic

products, I strongly oppose the proposed regulations.products, I strongly oppose the proposed regulations.My family purchases organic food so that we know what My family purchases organic food so that we know what

we are eating. I understand the current standards that we are eating. I understand the current standards that have been selfhave been self--imposed by the organic food imposed by the organic food industry.…industry.…

Public Comment: w0006200Public Comment: w0006200received date = 03/06/1998received date = 03/06/1998name = ******, *****name = ******, *****company = Columbia College company = Columbia College ChicagooChicagoostate = ILstate = ILzip = 60626zip = 60626country = United Statescountry = United Statescategory = Consumercategory = Consumertopic = Generaltopic = Generalcomment = Thank you for providing me with the opportunity to comment = Thank you for providing me with the opportunity to

make these remarks concerning the Proposed Rule (docket make these remarks concerning the Proposed Rule (docket number TMDnumber TMD--9494--0000--2).2).

I am a doctor of engineering, a writer, and a consumer of organiI am a doctor of engineering, a writer, and a consumer of organic c products.products.

(1) I am concerned that the Proposed Rule does not take the (1) I am concerned that the Proposed Rule does not take the previous usage of agricultural land into consideration in the previous usage of agricultural land into consideration in the certification process. This seems unrealistic, given the lingericertification process. This seems unrealistic, given the lingering ng character of many industrial pollutants.character of many industrial pollutants.

Automatic Stakeholder IDAutomatic Stakeholder IDA stakeholder is anyone who has an interest in a topicA stakeholder is anyone who has an interest in a topic

Useful to know which stakeholder groups are represented in the Useful to know which stakeholder groups are represented in the comments, and in what proportionscomments, and in what proportions

People often identify which stakeholder groups they People often identify which stakeholder groups they belong tobelong to

“As a family farmer, I believe….”“As a family farmer, I believe….”“As a long“As a long--time consumer of organic foods, I am horrified…”time consumer of organic foods, I am horrified…”

Simple patterns can recognize common stakeholdersSimple patterns can recognize common stakeholdersThey won’t get everything, but that’s okayThey won’t get everything, but that’s okay

Very simple example ruleVery simple example rule“As a” + <token> + punctuation“As a” + <token> + punctuation

Token must be a noun, adjective, article, etcToken must be a noun, adjective, article, etc

Stakeholder ID ExperimentStakeholder ID ExperimentAbout 22,000 public comments analyzed from About 22,000 public comments analyzed from US Department of Agriculture’s 1997 National US Department of Agriculture’s 1997 National Organic ProgramOrganic Program2484 stakeholder strings identified2484 stakeholder strings identified

1006 unique strings1006 unique stringsOnly 153 occurred more than onceOnly 153 occurred more than once

Many of the stakeholder strings that occurred Many of the stakeholder strings that occurred just once were meaningful, but too detailedjust once were meaningful, but too detailed

“active research scientist using genetic engineering “active research scientist using genetic engineering for clinical applications”for clinical applications”“active supporter of the existing organic standards”“active supporter of the existing organic standards”

A generalization method is neededA generalization method is needed

An Analyst’s WorkbenchAn Analyst’s WorkbenchFile Tools HelpBasic StatisticsDuplicate DetectionFrequent ConceptsStakeholders

Tools

consumer (195)organicorganic food/foods (109)organic food products (7)organic goods (8)organic products (8)organic produce (37)

: : : :mother (61)

homemaker / at home (3)concerned (2)working (5)new (2)of a baby/child/… (3)

<profession> (82)scientist (13)biologist (8)farmer (7)grower (8)organic gardener (21)

: : :citizen (58)

American (9)taxpayer (13)concerned (16)

Candidate Stakeholders (Hierarchical Display)

Capturing the Public’s Comments Capturing the Public’s Comments Help public Help public formulate formulate comments comments

Par 2.2(a):I am for thisI am against thisbecause…

This is what should be done (please be specific):

pollution: _____________ (see similar views)expense: ______________ (see similar views)unfit for elderly: ____________ (see similar views)unfit for children: ___________ (see similar views)add a new reason: ___________ (explore all views)

relax requirement c: _____________ (similar ideas)phase in alternative: _____________ (similar ideas)add a new suggestion: _______________________

_________________ (explore all suggestions)

Xxx xx xxx xx x xxx x xxx Xx xxxx x xxx x xxxxxxx xXxxxx x xx xxxx x X xx xxxx x xxxx x xxxXxxx xxxx x xxxxxxxx xxxXxx xxx xxxxxxx x xxx xx xXx xx xxxx x

Xxx xx xxx xx x xxx x xxx Xx xxxx x xxx x xxxxxxx xXxxxx x xx xxxx x X xx xxxx x xxxx x xxxXxxx xxxx x xxxxxxxx xxxXxx xxx xxxxxxx x xxx xx xXx xx xxxx x

Link to existing Link to existing commentary, commentary, regulation draft, regulation draft, other materialother material

Clustering By Opinion Clustering By Opinion

For each (sub)topicFor each (sub)topicGroup together all Yes, No & InGroup together all Yes, No & In--betweensbetweens

Create these categories manually and Create these categories manually and have the system learn to duplicate that have the system learn to duplicate that Extract reasons/motivations/authorities Extract reasons/motivations/authorities using characteristic phrases: using characteristic phrases:

“…against it because I think X”“…against it because I think X”“Y will have such a beneficial effect…”“Y will have such a beneficial effect…”

An Analyst’s WorkbenchAn Analyst’s WorkbenchFile Tools HelpInterpretationsOpinions

Tools

pro (19,566)“pollution” (11,003)”energy-efficient” (9,812)“safe” or ”safety” (534)

: : : : : :

guarded pro (4,661)“energy” (3,652)“safe” or “safety” (1,202)

: : : : : :

anti (8,002)“cost”, “expense”, “expensive” (7,314)

: : : : :

guarded anti (758)“cost” (500)“difficult” (105)

: : : : :

Main Opinions

Display IdeasDisplay Ideas

Grouped by opinion Grouped by opinion and writer type and writer type

mothers

cons

umers

Sierra

Club

lawye

rs

Car co

mpanie

slaw

yers

cons

umers

Con Pro25,000

20,000

15,000

10,000

5,000

Par 2.2(a1)Par 2.2(a1)Con:Con:

150, 818: “impossible to maintain”150, 818: “impossible to maintain”272: “too expensive for elderly”272: “too expensive for elderly”

Pro: Pro: 169, 213, 391, 392, 394: “already 169, 213, 391, 392, 394: “already being done in Alaska”being done in Alaska”18: “extend to children”18: “extend to children”

Xxx xx xxx xx x xxx x xxx Xx xxxx x xxx x xxxxxxx xXxxxx x xx xxxx x X xx xxxx x xxxx x xxxXxxx xxxx x xxxxxxxx xxxXxx xxx xxxxxxx x xxx xx xXx xx xxxx x

Xxx xx xxx xx x xxx x xxx Xx xxxx x xxx x xxxxxxx xXxxxx x xx xxxx x X xx xxxx x xxxx x xxxXxxx xxxx x xxxxxxxx xxxXxx xxx xxxxxxx x xxx xx xXx xx xxxx x

Xxx xx xxx xx x xxx x xxx Xx xxxx x xxx x xxxxxxx xXxxxx x xx xxxx x X xx xxxx x xxxx x xxxXxxx xxxx x xxxxxxxx xxxXxx xxx xxxxxxx x xxx xx xXx xx xxxx x

Grouped by topic Grouped by topic and crossand cross--correlatedcorrelated

Automated ‘Summarization’ Automated ‘Summarization’ For single comment or cluster of similar For single comment or cluster of similar comments, automatically extract central comments, automatically extract central characteristic(s) characteristic(s)

‘Headline’ for cluster, extract of doc, etc. ‘Headline’ for cluster, extract of doc, etc. Regulation writer can focus summarizer on Regulation writer can focus summarizer on aspects of interest aspects of interest Uses: Uses:

Quickly navigateQuickly navigate through comments / clustersthrough comments / clustersJumpJump--start writing responsestart writing response to comments to comments pointpoint--byby--point point

Putting It All Together Putting It All Together Assemble tools in a Assemble tools in a RegReg--Writer WorkbenchWriter WorkbenchWith interface/display, With interface/display, regreg--writer is able to: writer is able to:

switch modules on or off switch modules on or off move smoothly from one display to anothermove smoothly from one display to another

Ex., from comments to regulation to clusterEx., from comments to regulation to cluster

drag information easily into the responsedrag information easily into the response--writer writer window window produce integrated regulatory text, response, produce integrated regulatory text, response, comments, etc.comments, etc.

Broader ImpactBroader ImpactTechnical developments Technical developments

New dimensions of text classification (CS research)New dimensions of text classification (CS research)Results annually evaluated in user studies (SS research)Results annually evaluated in user studies (SS research)

Integration of social and computer sciencesIntegration of social and computer sciencesWorking out difficult research design issuesWorking out difficult research design issuesBridging methodological gapsBridging methodological gapsDefining new integrative research domainsDefining new integrative research domains

Informed crossInformed cross--agency dialogueagency dialogueBuilding social capital at the federal levelBuilding social capital at the federal levelLinking agency personnel, academics and stakeholdersLinking agency personnel, academics and stakeholders

Calling attention to the consequences of ITCalling attention to the consequences of ITAvoid the unreflective roll out of applications and architectureAvoid the unreflective roll out of applications and architectures that s that no one likes or usesno one likes or usesEncouraging public involvement & transparencyEncouraging public involvement & transparency