Natural Language and Dialogue Systems Lab Midterm Feb 23 rd. 7 to 9PM

Natural Language and Dialogue Systems Lab

Midterm Feb 23rd. 7 to 9PM

NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB

UC SANTA CRUZ

Announcements Midterm Review: Wednesday, 3:30-4:40 pm JBE 165, Midterm Review: Thursday, 2-3 pm, Soc Sci II 179 NO CLASS NEXT MONDAY. PRESIDENT’S DAY. Final AIMA homework covering AIMA chaps 10 thru

12 and online ontologies. LIBRARY has 2 copies of 3rd Edn.

Project presentations: March 7th & 9th, 12 mins each => 192 minutes. Need to be READY. Need slides in PDF uploaded somewhere we can all use the same machine.

FINAL: Wed March 16th. All team members present Round robin evaluation of all projects Final Report Due


Question Answering: IBM Watson on Jeopardy. TONIGHT!


UC SANTA CRUZ

Can’t seem to get it live.

http://www.nytimes.com/interactive/2010/06/16/magazine/watson-trivia-game.html?ref=magazine

PLAY VIDEO


UC SANTA CRUZ

Roots of Question Answering

Information Retrieval (IR) Information Extraction (IE)


UC SANTA CRUZ

How long have people been working on it? TREC = Text REtrieval Conferences

Series of annual evaluations, started in 1992 Organized into “tracks”

Test collections are formed by “pooling” Gather results from all participants Corpus/topics/judgments can be reused

TREC has had a QA Track since 1999. http://trec.nist.gov/data/qa.html http://trec.nist.gov/data/qa/T8_QAdata/

development.qa


UC SANTA CRUZ

Information Retrieval (IR)

Can substitute “document” for “information”

IR systems Use statistical methods Rely on frequency of words in query, document,

collection Retrieve complete documents Return ranked lists of “hits” based on relevance

Limitations Answers questions indirectly Does not attempt to understand the “meaning”

of user’s query or documents in the collection


UC SANTA CRUZ

The Information Retrieval Cycle

SourceSelection

Search

Query

Selection

Ranked List

Examination

Documents

Delivery

Documents

QueryFormulation

Resource

query reformulation,vocabulary learning,relevance feedback

source reselection


UC SANTA CRUZ

Supporting the Search Process

SourceSelection

Search

Query

Selection

Ranked List

Examination

Documents

Delivery

Documents

QueryFormulation

Resource

Indexing Index

Acquisition Collection


UC SANTA CRUZ

Information Extraction (IE) IE systems (usually… but recent advances)

Identify documents of a specific type Extract information according to pre-defined

templates Place the information into frame-like database

records

Templates = pre-defined questions Extracted information = answers Limitations

Templates are domain dependent and not easily portable

Weather disaster: TypeDateLocation

DamageDeaths...


UC SANTA CRUZ

Types of Question Answering http://trec.nist.gov/data/qa/T8_QAdata/

development.qa Factoid

Who discovered oxygen? When did Hawaii become a state? Where is Ayer’s Rock? What team won the World Series in 1992?

List What countries export oil? Name U.S. cities that have a “Shubert” theater.

Definition Who is Aaron Copland?

What is a quasar?


UC SANTA CRUZ

Central Idea of Factoid QA

Determine the semantic type of the expected answer

Retrieve documents that have keywords from the question

Look for named-entities of the proper type near keywords

“Who won the Nobel Peace Prize in 1991?” is looking for a PERSON

Retrieve documents that have the keywords “won”, “Nobel Peace Prize”, and “1991”

Look for a PERSON near the keywords “won”, “Nobel Peace Prize”, and “1991”


UC SANTA CRUZ

An Example

But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.The military junta took power in 1988 as pro-democracy

demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.The regime, which is also engaged in a battle with

insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.

Who won the Nobel Peace Prize in 1991?


UC SANTA CRUZ

Generic QA Architecture

Question Analyzer

Document Retriever

Passage Retriever

Answer Extractor

NL question

IR Query

Documents

Passages

Answers

Answer Type


UC SANTA CRUZ

Question analysis

Question word cues Who person, organization, location (e.g.,

city) When date Where location What/Why/How ??

Head noun cues What city, which country, what year... Which astronaut, what blues band, ...

Scalar adjective cues How long, how fast, how far, how old, ...


UC SANTA CRUZ

Using WordNet

wingspan

length

diameter radius altitude

ceiling

What is the service ceiling of an U-2?

NUMBER


UC SANTA CRUZ

Extracting Named Entities

Person: Mr. Hubert J. Smith, Adm. McInnes, Grace Chan

Title: Chairman, Vice President of Technology, Secretary of State

Country: USSR, France, Haiti, Haitian Republic

City: New York, Rome, Paris, Birmingham, Seneca Falls

Province: Kansas, Yorkshire, Uttar Pradesh

Business: GTE Corporation, FreeMarkets Inc., Acme

University: Bryn Mawr College, University of Iowa

Organization: Red Cross, Boys and Girls Club


UC SANTA CRUZ

More Named Entities

Currency: 400 yen, $100, DM 450,000

Linear: 10 feet, 100 miles, 15 centimeters

Area: a square foot, 15 acres

Volume: 6 cubic feet, 100 gallons

Weight: 10 pounds, half a ton, 100 kilos

Duration: 10 day, five minutes, 3 years, a millennium

Frequency: daily, biannually, 5 times, 3 times a day

Speed: 6 miles per hour, 15 feet per second, 5 kph

Age: 3 weeks old, 10-year-old, 50 years of age


UC SANTA CRUZ

How do we extract NEs?

Heuristics and patterns Fixed-lists (gazetteers) Machine learning approaches Combinations of Wordnet, Wikipedia like

YAGO DBPedia

Has anyone made my Wikipedia entry?


UC SANTA CRUZ

Answer Type Hierarchy


UC SANTA CRUZ

Let’s try it

http://www.nytimes.com/interactive/2010/06/16/magazine/watson-trivia-game.html?ref=magazine


UC SANTA CRUZ

Limitations?

Hard to tell the limits of the IBM Watson so far.


UC SANTA CRUZ

Conclusion: QA

Question answering is an exciting research area! Lies at the intersection of information retrieval

and natural language processing A real-world application of NLP technologies

The dream: a vast repository of knowledge we can “talk to”

Grew out of IR/IE and been supported by DARPA and other military funders for ‘information analysts’, potentially many commercial applications.


UC SANTA CRUZ


Project Presentations, Evaluation and Report


UC SANTA CRUZ

Presentations (thinking about random assignment of slots)

12 minutes. All team members present Introduction What your aims were for intelligence Short demonstration and/or sample

interaction Any experimental results so far

(comparisons) Future work


UC SANTA CRUZ

PROJECT EVALUATION

USER EVALUATIONS During Finals Slot, Wed March 16th 7 to 10 PM Each person evaluates 5 to 10 systems Carefully and COMPLETELY Fills Out Evaluation

Forms (part of HW4) Submits them electronically (do on paper while

evaluating system them enter into form on website)

PROJECT WRITEUP DUE AT FINALS SLOT 8 PAGES ACL FORMAT USE WINE SELECTOR AND ANNA AS A MODEL


UC SANTA CRUZ

PROJECT EVALUATION

I WILL PROVIDE A USER SATISFACTION SURVEY EACH TEAM:

PREPARE 2-4 SCENARIOS FOR PEOPLE TO TRY WITH YOUR SYSTEM

WRITE THEM UP AND HAVE AVAILABLE NEXT TO YOUR SYSTEM FOR USERS TO READ

MAKE SURE YOU CAN LOG ALL RELEVANT INFORMATION FOR EACH CONVERSATION

EACH DIALOG SHOULD HAVE ITS OWN ID (DATE/TIME IS A GOOD WAY TO MAKE IT), TELL USER THE DIALOG ID.

TURN IN THE LOGS, UPLOAD THEM TO MOODLE with other project material.


UC SANTA CRUZ

USER SATISFACTION SURVEY

State on a scale of 1 to 5 your agreement with the following statements:

I was able to complete the task I tried to do. I would be interested in using this system in the future The system seemed to understand what I was saying The system’s output was well matched to its task The dialog manager utilized well designed dialog

strategies The system was smart in some way. OPEN TEXT FIELD: Please enter any general comments with respect to

smartness, good points or bad points about this system.


UC SANTA CRUZ

PROJECT EVALUATIONS

EACH SYSTEM SHOULD END UP WITH ABOUT 27 USERS

69 people in class X 6 system evaluations each / 15 projects = 27.6 users per system

EACH USER EVALUATION CONSISTS OF: 1-2 SCENARIO INTERACTIONS WITH USER

SATISFACTION SURVEY FILLED OUT AFTER EACH CONVERSATION, IDENTIFY DIALOG ID.

1 FREE FORM DIALOG OF WHATEVER YOU WANT (AFTER SCENARIO CONVERSATION) with its own separate


Classical Planning & Planning Graphs


UC SANTA CRUZ

Where does planning get used?

Path planning: games, robotics Dialogue management: dialog systems Scheduling


UC SANTA CRUZ

Planning Domain Definition Language A planning domain:

Initial state Actions available in a state Result of applying an action The goal test


UC SANTA CRUZ

States

State: conjunction of fluents, ground functionless atoms Fluent: predicate with time stamp, specifies

aspects of the world that can change At (Truck1, UCSC) ∧ At (Truck2, SJC)

Factored Representation: A vector of state variables like in the Wumpus world example

Database Semantics: closed world, unique names


UC SANTA CRUZ

Actions

Action schemas for Actions(s); Result (s,a)

Action name + list of variables Precondition: a conjunction of literals Effect: a conjunction of literals, Delete list (negative

fluents), Add list (positive fluents) An action a can be executed in state s iff S =>

precond (a) Applicable: an action is applicable in state s if

preconds are satisfied Frame problem: results of actions are what changes,

not what doesn’t change.


UC SANTA CRUZ

Dialogue Act Examples

Inform (Allen & Perrault, 1980)

Persuade (Moore & Paris, 1993)


UC SANTA CRUZ

Air Cargo Transport Problem

Figure 10.1p. 369 R&N 3e

a = airportCi = cargop = plane


UC SANTA CRUZ

Figure 10.2p. 370R&N 3e

Spare Tire Planning Example


UC SANTA CRUZ

Two Approaches to Searching for a Plan Forward (Progression Search) vs.

Backward (Regression Search)


UC SANTA CRUZ

Forward Search Planning

Start in the initial state At each step apply actions available to

generate successor states Keep going

until you get to GOAL Termination condition on length of search path


UC SANTA CRUZ

Backward Search Planning

Start at Goal G’ = (G – (Add (a)) U Precond (a)

ie effects that were added by the action may not have been true, but preconditions must have held

Del(a) not included. Don’t know whether they were true before or not.

Use the INVERSE of the actions to search backward

Add states for preconditions that need to satisfy

Constrain to searching thru RELEVANT states, i.e. ones related to the goal.


UC SANTA CRUZ

Forward Search vs. Backward Search Forward Search prone to exploring

irrelevant actions Consider Task of Buying a Book given ISBN

number ISBNs are 10 digits Forward Search starts hypothesizing which

10

=> Need heuristics for forward search


UC SANTA CRUZ

Planning: What makes good Heuristics Chapter 3: Admissible Heuristic

H(S) estimates the distance from a state S to a goal An admissable heuristic never overestimates An admissable heuristic can be derived from RELAXED

PROBLEM that is easier to solve. The cost of the solution to the relaxed problem = H

Search: Nodes are states and edges are actions Find a path connecting initial state to goal state RELAX: add more edges, combine states to make

fewer Planning uses a FACTORED representation for

states and action schemas => domain-independent H


UC SANTA CRUZ

Planning HEURISTICS

Factored Representations: set of conjuncts, can just drop one (or more)

Ignore Preconditions => every action applicable in every state, then any goal can be achieved in one step

H1: number of steps is number of unsatisfied goals BUT Some actions may achieve multiple goals BUT Some actions may undo effects of other actions

Ignore possibility of undoing => H2: Count actions required to achieve all the

literals in the goal


UC SANTA CRUZ

Planning HEURISTICS

FACTORED REPRESENTATIONS IGNORE DELETE LISTS => Make

monotonic progress towards the goal STATE ABSTRACTIONS => IGNORE SOME

FLUENTS This is equivalent to ‘relaxing some

constraints’


UC SANTA CRUZ

Planning HEURISTICS

DECOMPOSITION: Divide a problem P into parts P1.. Pn Solve each subproblem independently, then

combine

SUBGOAL Independence Assumption: Cost of solving a conjunction of subgoals is sum of costs of solving each subgoal independently. NOT Admissible when subplans contain redundant

actions

Max (Cost Pi) is admissable (but too low?) Show independence then Cost(Pi) + Cost (Pj)

is admissable


UC SANTA CRUZ

Graph Planning: Cake Problem


UC SANTA CRUZ

Graph Planning: Graphplan Algorithm

Objective: Better HeuristicsNeed: structure that clarifies problemSignificance: faster convergence, more

manageable branch factorUse Graphical Language of Constraints,

ActionsNotation

Operators (real actions): large rectanglesPersistence actions (for each literal): small

squares, denote non-changeGray links: mutual exclusion (mutex)


UC SANTA CRUZ

MUTEX

A mutex relation holds between two actions at a given level if any of the following holds: Inconsistent effects: One action negates an effect of

the other. For example Eat(cake) and the persistence of Have(Cake) have inconsistent effects because they disagree on the effect of Have(Cake)

Interference: one of the effects of one action is the negation of a precondition of the other, e.g. Eat(Cake) interferes with the persistence of Have(Cake) by negating its precondition

Competing Needs: one of the preconditions of one action is mutually exclusive with a precondition of the other. For example, Bake(Cake) and Eat(Cake) are mutex because they compete on the value of the Have(Cake) precondition.


UC SANTA CRUZ

Mutex for literals

Two literals are mutex at the same level if One is the negation of the other Each possible pair of actions that could achive

the two literals is mutually exclusive. For example Have(Cake) and Eaten(Cake) are mutex in S1 because the oly way of achieving Have(Cake), the persistenc action, is mutex with the only way of achieving Eaten(Cake), namely Eat(Cake).

In S2 the two literals are not mutex, because there are new ways of achieving them, such as Bake(Cake) and the persistence of Eaten(Cake) are not mutex.


UC SANTA CRUZ

GRAPH PLAN: STATES S AND ACTIONS A A DIRECTED GRAPH WITH LEVELS Plan has been propositionalized (see p. 368) LEVEL S0 = INITIAL STATE HAS NODES FOR

EVERY FLUENT THAT HOLDS IN S0 LEVEL A0 = NODES FOR EACH GROUND

ACTION THAT MIGHT BE APPLICABLE IN S0 ALTERNATING LEVELS Si followed by Ai until

we reach termination condition Si = literals that COULD hold at at time i, if

either P or ~P could hold both represented Ai = all actions that COULD have preconds

satisfied at time i


UC SANTA CRUZ

Graph Planning: Cake Problem


UC SANTA CRUZ

Graph Planning: Graphplan Algorithm

• Operators (real actions): large rectangles

• Persistence actions (for each literal): small squares, denote non-change

• Gray links: mutual exclusion (mutex)

• NOTE:

• If any goal fails to appear in the final level then the problem is unsolvable

• Estimate cost of Goal G from state S as the level that G first appears in the plan graph = LEVEL COST heuristic


Stopped around here on 2/14


UC SANTA CRUZ

GraphPlan Algorithm

Alternating Steps Solution extraction

Expansion

Extract-Solution: Goal-Based (Regression)

Expand-Graph: Adds


UC SANTA CRUZ

Graph Planning:Spare Tire Example ( Levels off at S2)


UC SANTA CRUZ

GRAPH Planning: Extract Solution

Backward Search Problem Initial state is last level of planning graph Sn

along with the set of goals of the problem The actions available in a state at level Si are

to select any conflict free subset of Actions in Ai-1 whose effects cover the goals in the state. The resulting state has level i-1 and has as its set of goals the preconditions for the selected set of actions.

The goal is to reach a state at level S0 such tht all the goals are satisfied

Cost of each Action is 1.


UC SANTA CRUZ

GRAPH Planning

Interference: Remove (Flat, Axle) is mutex with LeaveOvernight because one has the precondition At(Flat, Axle) and the other has its negation as an effect

Competing Needs: Puton(Spare,Axle) is mutex with Remove(Flat,Axle) because one has AT(Flat,Axle) as a precondition and the other has its negation

Inconsistent Support: At(Spare,Axle) is mutex with At(Flat,Axle) in S2 because the only way of achieving At(Spare,Axle) is by PutOn(Spare,Axle) and that is mutex with the persistence action that is the only way of achieving At(Flat,Axle)


UC SANTA CRUZ

Graph Planning:Spare Tire Example ( Levels off at S2)


UC SANTA CRUZ

Graph Planning; OTHER HEURISTICS

• Conjunction of Goals:

• MAX LEVEL heuristic: Max level cost of any of the goals, admissible but possibly not accurate

• LEVEL Sum heuristic: Add levels for each subgoal. Subgoal independence assumption, inadmissible but can work well in practice for decomposable problems

• Set level heuristic: level at which all the literals in the conjunctive goal appear without being mutex.

• Admissible, Dominates max-level, works well when interaction among subplans


UC SANTA CRUZ

GRAPH PLAN: TERMINATION

Read 10.3.3. page 385


UC SANTA CRUZ

Planning Domain Definition Language A planning domain:

Initial state Actions available in a state Result of applying an action The goal test


UC SANTA CRUZ

PDDL for putting on a pair of shoes


Does it matter which order the socks go on?


UC SANTA CRUZ

Partial Order Planning Efficiency


UC SANTA CRUZ

Partial Order Planning

Order of some actions often doesn’t matter


UC SANTA CRUZ

Partial Order planning


UC SANTA CRUZ


Start with goal state, detect flaws & fix, until start


UC SANTA CRUZ

Planning


UC SANTA CRUZ


Detect violations (flaws)


UC SANTA CRUZ

Summary

Planning for deterministic, fully observable, static environments

Planning systems are problem solving algorithms that operate on explicit propositional or relational representations of states and actions

PDDL. Planning domain definition language

State space search: forward (progression) or backward (regression)

Planning graphs efficient approach.


Chapter 11. Real World Planning


UC SANTA CRUZ

Planning with abstract actions. HTN, HLA


UC SANTA CRUZ

Hierarchical Planning


UC SANTA CRUZ

Will finish Monday after the Midterm, Feb 28th


UC SANTA CRUZ

Angelic Search: Agent gets to choose


UC SANTA CRUZ


UC SANTA CRUZ

Doubles Tennis


UC SANTA CRUZ


UC SANTA CRUZ