Speech & NLP (Fall 2014): Field Study of Narrative Maps & Automated Extraction of Geo-Spatial Data from NL Route Directions

Speech & NLP

A Field Study of Narrative Maps &

Automated Extraction of Geo-Spatial Data

from Route Directions

www.vkedco.blogspot.com

Vladimir Kulyukin

http://www.vkedco.blogspot.com/

Outline

A Field Study of Narrative Maps

Extraction of Geo-Spatial Data from Route

Descriptions: A Knowledge Representation Lab

A Field Study of Narrative Maps

Narrative Map

Narrative map consists of two verbal descriptions:

verbal route descriptions and verbal surveys of a given

environment

The origins of narrative maps vary from blogs,

forums, books, O&M instructors

Question: To what extent can narrative maps be

mined/parsed to extract various geo-spatial

knowledge (e.g., places, paths, SSH levels)?

Background

We investigated the utility of narrative maps in a

longitudinal study of blind shopping in supermarkets

In our system, called ShopTalk, verbal route

directions were generated from a manually

constructed topological map (inspired by the

topological level of the SSH) of the supermarket’s

locomotor space

ShopTalk 1.0

http://www.youtube.com/watch?v=A7UsazVCE7Y



Field Study Results

Ten visually impaired participants were able to

detect environmental cues needed to make sense of

the generated verbal instructions (provided at

beginning of experiment)

The participants used their O&M skills to localize and

orient themselves in the store, without any wearable

or environment-embedded sensors

Field Study Results

A key finding was that verbal route directions were

sufficient for our sample of independent travelers to

navigate this supermarket reliably

The more they used the system, the less they

requested verbal route directions

This finding suggests that the stores (and other

dynamic and complex environments) may not need to

be instrumented with any external sensors, such as

RFID tags, Wi-Fi routers, IR transmitters, etc.

Extraction of Geo-Spatial Data

from

Route Descriptions

Methodology

Take a route description

Go through it instruction by instruction

For each instruction, identify objects, predicates,

functions, relations (i.e., formalize a

conceptualization)

Run a syntactic parser (e.g., Stanford Parser) on

each instruction and come up with computable

techniques to automatically extract

conceptualizations

Sample Route Description

We will do both syntactic and semantic analysis

of a route description

We will use Stanford Parser as for syntactic and

dependency analysis

We will also attempt to write down a

conceptualization of the first 10 instructions from

the route description

Our conceptualization will be loosely based on

Kuipers’ SSH

Baruch College Route Description

1 Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.

2 Walk to 24th street in 80 feet.

3 This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.

4 Cross 24th, and continue 225 feet to 25th street.

5 Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.

6 Turn left and cross 3rd Ave.

7 This is a signalized, 6-lane crossing with 1-way Northbound traffic only.

8 Continue to the The Newman library, which is mid-block.

9 Follow the right side building edge.

10 This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches, trash, etc.

11 At 220 feet, a prominent right side landmark is the round metal revolving door of the library.

12 This is easily detected by a cane.

13 To the right of this door is the accessible entry door.

14 Enter here.

source http://www.clickandgomaps.com

http://www.clickandgomaps.com/



Route Instruction 1: Parser Output

(ROOT

(S

(NP

(NP (NN Exit) (JJ northbound) (NN bus))

(SBAR

(S

(NP

(NP (CD 101) (, ,) (CD 102) (, ,)

(CC or)

(CD 103))

(PP (IN at)

(NP (DT the) (JJ 24th) (NN street))))

(VP (VB stop)

(CC and)

(VB turn)))))

(VP (VBD left))

(. .)))

nn(bus-3, Exit-1)

amod(bus-3, northbound-2)

dobj(stop-14, bus-3)

nsubj(left-17, bus-3)

nsubj(stop-14, 101-4)

nsubj(turn-16, 101-4)

num(101-4, 102-6)

conj_or(101-4, 103-9)

nsubj(stop-14, 103-9)

det(street-13, the-11)

amod(street-13, 24th-12)

prep_at(101-4, street-13)

rcmod(bus-3, stop-14)

rcmod(bus-3, turn-16)

conj_and(stop-14, turn-16)

root(ROOT-0, left-17)

Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.

Corrected Route Instruction 1: Parser Output

(ROOT

(S

(NP (PRP You))

(VP

(VP (VBZ exit)

(NP (JJ northbound) (NN bus))

(NP

(NP (CD 101) (, ,) (CD 102) (, ,)

(CC or)

(CD 103))

(PP (IN at)

(NP (DT the) (JJ 24th) (NN street) (NN

stop)))))

(CC and)

(VP (VBP turn)

(ADVP (RB left))))

(. .)))

nsubj(exit-2, You-1)

nsubj(turn-17, You-1)

root(ROOT-0, exit-2)

amod(bus-4, northbound-3)

iobj(exit-2, bus-4)

dobj(exit-2, 101-5)

num(101-5, 102-7)

dobj(exit-2, 103-10)

conj_or(101-5, 103-10)

det(stop-15, the-12)

amod(stop-15, 24th-13)

nn(stop-15, street-14)

prep_at(101-5, stop-15)

conj_and(exit-2, turn-17)

advmod(turn-17, left-18)

You exit northbound bus 101, 102, or 103 at the 24th street stop and turn

left.

Route Instruction 1: Conceptualization

Exit northbound bus 101, 102, or 103 at the 24th street stop and turn left.

Objects NorthBoundBus101, NorthBoundBus102, NorthBoundBus103, 24thStreet, 24thStreetStop, Agent

Predicates Place(NorthBoundBus101), Place(NorthBoundBus102), Place(NorthBoundBus103),

Place(24thStreetStop), Path(24thStreet), On(24thStreetStop, 24thStreet), AgentLocation(Agent,

NorthBoundBus101) V AgentLocation(Agent, NorthBoundBus102) V AgentLocation(Agent,

NorthBoundBus103)

Actions <AgentLocation(Agent, NorthBoundBus101), Travel, AgentLocation(Agent, 24thStreetStop>;

<AgentLocation(Agent, NorthBoundBus102), Travel, AgentLocation(Agent, 24thStreetStop>;

<AgentLocation(Agent, NorthBoundBus103), Travel, AgentLocation(Agent, 24thStreetStop>;

<AgentLocation(Agent, 24thStreetStop), Turn(Left), AgentLocation(Agent, 24thStreetStop>;


(ROOT

(S

(VP (VBP Walk)

(PP (TO to)

(NP

(NP (JJ 24th) (NN street))

(PP (IN in)

(NP (CD 80) (NNS feet))))))

(. .)))

root(ROOT-0, Walk-1)


prep_to(Walk-1, street-4)

num(feet-7, 80-6)

prep_in(street-4, feet-7)

Walk to 24th street in 80 feet.


Walk to 24th street in 80 feet.

Objects 24thStreet

Predicates AgentLocation(Agent, 24thStreetStop)

Actions <AgentLocation(Agent, 24thStreetStop), Travel(80 feet), AgentLocation(Agent, 24thStreet>


(ROOT

(S

(NP (DT This))

(VP (VBZ is)

(NP (DT a) (JJ signalized) (, ,) (JJ 2-lane) (NN crossing))

(PP (IN with)

(NP (JJ 1-way) (NNP Eastbound) (NN traffic)))

(ADVP (RB only)))

(. .)))

nsubj(crossing-7, This-1)

cop(crossing-7, is-2)

det(crossing-7, a-3)

amod(crossing-7, signalized-4)

amod(crossing-7, 2-lane-6)

root(ROOT-0, crossing-7)

amod(traffic-11, 1-way-9)

nn(traffic-11, Eastbound-10)

prep_with(crossing-7, traffic-11)

advmod(crossing-7, only-12)

This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.


This is a signalized, 2-lane crossing with 1-way Eastbound traffic only.

Objects Crossing01

Predicates Crossing(Crossing01); SignalizedCrossing(Crossing01); NumberOfLanesCrossing01, 2);

DirectionOfTraffic(Crossing01, East); OneWayCrossing(Crossing01); Path(Crossing01);

CrossingOf(Crossing01, 24thStreet);

Actions No Actions


(ROOT

(S

(NP (NNP Cross))

(VP

(VP (VBZ 24th))

(, ,)

(CC and)

(VP (VBP continue)

(NP (CD 225) (NNS feet))

(PP (TO to)

(NP (JJ 25th) (NN street)))))

(. .)))

nsubj(24th-2, Cross-1)

nsubj(continue-5, Cross-1)

root(ROOT-0, 24th-2)

conj_and(24th-2, continue-5)

num(feet-7, 225-6)

dobj(continue-5, feet-7)


prep_to(continue-5, street-10)

Cross 24th, and continue 225 feet to 25th street.


Cross 24th, and continue 225 feet to 25th street.

Objects 24thStreet, 25thStreet

Predicates Path(24thStreet); Path(25thStreet)

Actions <AgentLocation(Agent, Crossing01), Travel, AgentLocation(Agent, Crossing01)>;

<AgentLocation(Agent, Crossing01), Travel(225 feet), AgentLocation(Agent, 25thStreet)>;


(ROOT

(NP

(NP (NNP Cross) (NNP 25th))

(, ,)

(NP

(NP (DT a) (JJ 2-lane) (JJ signalized) (NN street))

(PP (IN with)

(NP (JJ 1-way) (NNP Westbound) (NN traffic))))

(. .)))

nn(25th-2, Cross-1)

root(ROOT-0, 25th-2)

det(street-7, a-4)

amod(street-7, 2-lane-5)

amod(street-7, signalized-6)

appos(25th-2, street-7)


nn(traffic-11, Westbound-10)

prep_with(street-7, traffic-11)

Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.


Cross 25th, a 2-lane signalized street with 1-way Westbound traffic.

Objects 25thStreet

Predicates Path(25thStreet); SignalizedStreet(25thStreet); DirectionOfTraffic(West); OneWayStreet(25thStreet);

Crossing(Crossing02); CrossingOf(Crossing02, 25thStreet); NumberOfLanes(25thStreet, 2);

Actions <AgentLocation(Agent, Crossing02), Travel, AgentLocation(Agent, Crossing02)>;


(ROOT

(S

(VP

(VP (VB Turn)

(ADVP (RB left)))

(CC and)

(VP (VB cross)

(NP (JJ 3rd) (NNS Ave.))))))

root(ROOT-0, Turn-1)

advmod(Turn-1, left-2)

conj_and(Turn-1, cross-4)

amod(Ave.-6, 3rd-5)

dobj(cross-4, Ave.-6)

Turn left and cross 3rd Ave.


Turn left and cross 3rd Ave.

Objects 3rdAve, Crossing03

Predicates Path(3rdAve); Crossing(Crossing03); CrossingOf(Crossing03, 3rdAve);

Actions <AgentLocation(Agent, Crossing02), Turn(Left), AgentLocation(Agent, Crossing02)>;

<AgentLocation(Agent, Crossing02), Travel, AgentLocation(Agent, Crossing03)>;

<AgentLocation(Agent, Crossing03>, Travel, AgentLocation(Crossing03)>;


(ROOT

(S

(NP (DT This))

(VP (VBZ is)

(NP (DT a) (JJ signalized) (, ,) (JJ 6-lane) (NN crossing))

(PP (IN with)

(NP (JJ 1-way) (NNP Northbound) (NN traffic)))

(ADVP (RB only)))

(. .)))

nsubj(crossing-7, This-1)

cop(crossing-7, is-2)

det(crossing-7, a-3)

amod(crossing-7, signalized-4)

amod(crossing-7, 6-lane-6)

root(ROOT-0, crossing-7)


nn(traffic-11, Northbound-10)

prep_with(crossing-7, traffic-11)

advmod(crossing-7, only-12)

This is a signalized, 6-lane crossing with 1-way Northbound traffic only.


This is a signalized, 6-lane crossing with 1-way Northbound traffic only.

Objects Crossing03

Predicates Crossing(Crossing03); NumberOfLanes(Crossing03, 6); SignalizedCrossing(Crossing03);

OneWayCrossing(Crossing03); DirectionOfTraffic(North);

Actions No actions


(ROOT

(S

(VP (VB Continue)

(PP (TO to)

(NP

(NP (DT the) (NNP The) (NNP Newman) (NN library))

(, ,)

(SBAR

(WHNP (WDT which))

(S

(VP (VBZ is)

(ADJP (JJ mid-block))))))))

(. .)))

root(ROOT-0, Continue-1)

det(library-6, the-3)

nn(library-6, The-4)

nn(library-6, Newman-5)

prep_to(Continue-1, library-6)

nsubj(mid-block-10, library-6)

cop(mid-block-10, is-9)

rcmod(library-6, mid-block-10)

Continue to the The Newman library, which is mid-block.


Continue to the The Newman library, which is mid-block.

Objects TheNewmanLibrary

Predicates Place(TheNumanLibrary); Block(Block01); On(Block01, 3rdAve); MiddleOf(Block01,

TheNewmanLibrary);

Actions <AgentLocation(Agent, Crossing03), Travel, AgentLocation(Agent, TheNewmanLibrary)>


(ROOT

(S

(VP (VB Follow)

(NP (DT the) (JJ right) (NN side) (NN building) (NN edge)))

(. .)))

root(ROOT-0, Follow-1)

det(edge-6, the-2)

amod(edge-6, right-3)

nn(edge-6, side-4)

nn(edge-6, building-5)

dobj(Follow-1, edge-6)

Follow the right side building edge.


Follow the right side building edge.

Objects TheRightSideBuildingEdge

Predicates Edge(TheRightSideBuildingEdge);

Actions <AwareOf(Agent, TheRIghtSideBuildingEdge), Follow, AwareOf(Agent, TheRightSideBuildingEdge)>


(ROOT

(S

(NP (DT This) (NN sidewalk))

(VP (VBZ is)

(ADJP

(NP (CD 15) (NNS feet))

(JJ wide))

(, ,)

(PP (IN with)

(NP

(NP (JJ left) (NN curbside) (NNS obstacles))

(PP (JJ such) (IN as)

(NP (NNS poles) (, ,) (NNS benches) (, ,) (NNS trash) (, ,) (FW

etc.))))))

(. .)))

det(sidewalk-2, This-1)

nsubj(wide-6, sidewalk-2)

cop(wide-6, is-3)

num(feet-5, 15-4)

npadvmod(wide-6, feet-5)

root(ROOT-0, wide-6)

amod(obstacles-11, left-9)

nn(obstacles-11, curbside-10)

prep_with(wide-6, obstacles-11)

nn(trash-18, poles-14)

dep(trash-18, benches-16)

prep_such_as(obstacles-11, trash-18)

dep(trash-18, etc.-20)

Follow the right side building edge. This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches,

trash, etc.


This sidewalk is 15 feet wide, with left curbside obstacles such as poles, benches,

trash, etc.

Objects Sidewalk01

Predicates Path(Sidewalk01), WidthOf(Sidewalk01, 15 feet);

Actions <AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOf(Agent, X) & Pole(X) &

Obstacle(X)>;

<AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOf(Agent, X), & Bench(X) &

Obstacle(X)>;

<AgentLocation(Agent, LeftCurbOf(SideWalk01), Sense, AwareOfAgent, X) & Trash(X) &

Obstacle(X)>;

References

B. Kuipers. (2000). “The Spatial Semantic Hierarchy.” Artificial Intelligence 119, pp. 191-

233.

R. Schank, C. Riesbeck W. A. (1981) Inside Computer Understanding. Lawrence Erlbaum &

Associates.

J. Nicholson, V. Kulyukin, D. Coster. (2009). “ShopTalk: Independent Blind Shopping Through

Verbal Route Directions and Barcode Scans.” The Open Rehabilitation Journal, ISSN: 1874-

9437 Volume 2, 2009, DOI 10.2174/1874943700902010011.

Kulyukin, V. and Tammineni, T. Digital Labeling and Narrative Mapping in Mobile Remote

Audio Signage. Lap Lambert Academic Publishing, May 2014, ISBN-13: 978-3-659-52863-7,

ISBN-10: 3659528633, EAN: 9783659528637.

Kulyukin, V. and Reddy, T. Narrative Map Augmentation with Automated Landmark

Extraction and Path Inference. In K. Miesenberger et al. (Eds.): Proceedings of the 14th

International Conference on Computers Helping People with Special Needs (ICCHP 2014),

Part II, LNCS 8548, pp. 50-53, 2014. Springer International Publishing Switzerland 2014.

https://www.lap-publishing.com/catalog/details/store/gb/book/978-3-659-52863-7/digital-labeling-and-narrative-mapping-in-mobile-remote-audio-signage



















http://www.icchp.org/



Science

Speech & NLP (Fall 2014): Field Study of Narrative Maps & Automated Extraction of Geo-Spatial Data from NL Route Directions