MAGIC Seen from the Perspective of RAGS

MAGIC Seen from the Perspective of RAGS

Kathleen R. McKeownKathleen R. McKeown

Department of Computer ScienceDepartment of Computer Science

Columbia UniversityColumbia University

MAGICMAGIC

Multimedia Abstract Generation of Intensive Multimedia Abstract Generation of Intensive Care dataCare data

Collaborators:Collaborators:

Steven Feiner, Desmond JordanSteven Feiner, Desmond Jordan

Shimei Pan, James Shaw, Michelle ZhouShimei Pan, James Shaw, Michelle ZhouKris Concepcion, Liz Chen, Jeanne Fromer Kris Concepcion, Liz Chen, Jeanne Fromer

ScenarioScenario

Goal: provide post-operative information on Goal: provide post-operative information on bypass patients (CABG)bypass patients (CABG)

Prior to completion of surgery and before Prior to completion of surgery and before transport to Cardiac Intensive Care Unit transport to Cardiac Intensive Care Unit (ICU)(ICU)

Status needed for ICU nurse, cardiologistStatus needed for ICU nurse, cardiologist Time criticalTime critical

Issues for Language GenerationIssues for Language Generation

ConcisenessConciseness: Coordinated speech and text that : Coordinated speech and text that is brief but unambiguousis brief but unambiguous

Coordination with other mediaCoordination with other media: Modify : Modify wording and speech to coordinate references wording and speech to coordinate references with graphical highlightingwith graphical highlighting

Media specific tailoringMedia specific tailoring: : Produce wording appropriate for spoken languageProduce wording appropriate for spoken language Use information from language generation to improve Use information from language generation to improve

quality of synthesized speechquality of synthesized speech

StatusStatus

Implemented prototype showing Implemented prototype showing coordination between media for limited inputcoordination between media for limited input

Text output for large numbers of input casesText output for large numbers of input cases Undergoing evaluation *now* in ICUUndergoing evaluation *now* in ICU Runs on live data on a daily basisRuns on live data on a daily basis 5-10% error rate5-10% error rate

Continuing research on effects of LG Continuing research on effects of LG information on prosody, partial resultsinformation on prosody, partial results

PrinciplesPrinciples

Early processes produce media independent Early processes produce media independent representationsrepresentations

Representations use partial orderings in order to Representations use partial orderings in order to make early commitments where possible and make early commitments where possible and retain flexibilityretain flexibility

Both the speech and graphics content planner may Both the speech and graphics content planner may add content and ordering constraintsadd content and ordering constraints

Constraints on later decisions may be added early Constraints on later decisions may be added early on (e.g., lexical choice)on (e.g., lexical choice)

Data Server and Filter (conceptual)Data Server and Filter (conceptual)

InputInput 18:25 18:25 <drug><drug> DripsDrips NorepinephrineNorepinephrine 18:2718:27 <drug><drug> DripsDrips NorepinephrineNorepinephrine 18:2918:29 <drug><drug> Misc.Misc. Magnesium SulfateMagnesium Sulfate 18:2918:29 <surgery><surgery> CardiacCardiac Defibrillated by surgeonDefibrillated by surgeon

18:33:1118:33:11 100 (BP)100 (BP) 51 (HR)51 (HR) 18:34:0118:34:01 9696 5252

OutputOutput C-inanimate entity -> C-drug -> C-operating-room-medication ->C-Drip C-inanimate entity -> C-drug -> C-operating-room-medication ->C-Drip

-> C-Norepinephrine-> C-Norepinephrine Top-level categoriesTop-level categories

C-state, C-event, C-entity (abstract, physical, organization, math)C-state, C-event, C-entity (abstract, physical, organization, math) InferencesInferences

Hypotension: time, duration, drugs givenHypotension: time, duration, drugs given

General Content Planner - SOAP(Rhetorical, semantic, conceptual)General Content Planner - SOAP(Rhetorical, semantic, conceptual)

OverviewOverview DemographicsDemographics

Name, Age, MRN, Gender, Doctor, OperationName, Age, MRN, Gender, Doctor, Operation Medical historyMedical history LinesLines TherapyTherapy DevicesDevices

Detail ViewDetail View Drips (on leaving)Drips (on leaving) Induction infoInduction info DevicesDevices Lab reportLab report

TimelineTimeline InferencesInferences

End valuesEnd values ConclusionsConclusions

Speech Content Planner - Satisfying ConcisenessSpeech Content Planner - Satisfying Conciseness

Speech content planner Speech content planner groups information into groups information into sentencessentences Ms. Jones is an 80 year old, hypertensive diabetic female Ms. Jones is an 80 year old, hypertensive diabetic female

patient of Dr. Smith undergoing CABG.patient of Dr. Smith undergoing CABG. Ms. Jones is an 80 year old, female patient of Dr. Smith Ms. Jones is an 80 year old, female patient of Dr. Smith

undergoing CABG. She has a history of diabetes and undergoing CABG. She has a history of diabetes and hypertension.hypertension.

To satisfy communicative goal to be concise, To satisfy communicative goal to be concise, selects adjectives, prepositional phrases when selects adjectives, prepositional phrases when possible.possible.

Input to speech content planner -semantic propositionsInput to speech content planner -semantic propositions

X is-a patientX is-a patient X has-property X has-property last name = Joneslast name = Jones X has-property X has-property age = 80 yearsage = 80 years X has-property X has-property history = hypertensionhistory = hypertension X has-propertyX has-property history = diabetes history = diabetes X has-property X has-property gender = femalegender = female X has-property X has-property surgery = CABGsurgery = CABG X has-property X has-property doctor = Ydoctor = Y Y has-property Y has-property last name = Smithlast name = Smith

Forming Sentence Structure(Rhetorical, semantic, lexical, syntactic)Forming Sentence Structure(Rhetorical, semantic, lexical, syntactic)

((relation is-a)((relation is-a) (arg1 ((item ((class name) (arg1 ((item ((class name) (last-name “Jones”))))) (last-name “Jones”))))) (arg2 ((item ((class patient)))))) (arg2 ((item ((class patient))))))

((relation is-a)((relation is-a) (arg1 ((item ((class name) (arg1 ((item ((class name) (last-name “Jones”))))) (last-name “Jones”))))) (arg2 ((item ((class patient)) (arg2 ((item ((class patient)) ( (premod ((history hypertensionpremod ((history hypertension))))))))))))

3 Types of Aggregation3 Types of Aggregation

Hypotactic aggregationHypotactic aggregation: Given a set of : Given a set of propositions, can one be realized as a modifier?propositions, can one be realized as a modifier?

Semantic aggregationSemantic aggregation: if a patient is on : if a patient is on multiple drips and all devices, a patient has multiple drips and all devices, a patient has received received massive cardiotonic therapymassive cardiotonic therapy

Paratactic aggregationParatactic aggregation: Combine related : Combine related propositions using conjunction and appositionpropositions using conjunction and apposition

Coordination across mediaCoordination across media

Temporal media Temporal media

Coordinate spoken references with Coordinate spoken references with highlighting of graphical referenceshighlighting of graphical references

Requires negotiation of ordering and Requires negotiation of ordering and duration of media actionsduration of media actions

Negotiating OrderingNegotiating Ordering

Spoken language generator has grammatical Spoken language generator has grammatical constraints on linear orderingconstraints on linear ordering

Graphics generator has spatial constraints Graphics generator has spatial constraints on layouton layout

Individual accounts of these constraints Individual accounts of these constraints may result in an incoherent presentationmay result in an incoherent presentation

Ms. Jones is an 80 year old, diabetic, Ms. Jones is an 80 year old, diabetic, hypertensive female patienthypertensive female patient

of Dr. Smith undergoing CABG.of Dr. Smith undergoing CABG.

Problems for Language Generation: OrderingProblems for Language Generation: Ordering

When to provide an ordering over references?When to provide an ordering over references? produce a partial ordering after word choiceproduce a partial ordering after word choice

How to select an ordering compatible with How to select an ordering compatible with graphics?graphics?

produce several possibilities ordered by preferenceproduce several possibilities ordered by preference

How to communicate orderings with graphics?How to communicate orderings with graphics? maintain a mapping between strings and semantic objectsmaintain a mapping between strings and semantic objects

Media Negotiation(Conceptual, Semantic, Document)Media Negotiation(Conceptual, Semantic, Document)

Speech componentsSpeech components produce produce candidate candidate partial orderspartial orders1.(< name age (* diabetes hypertension) gender surgeon 1.(< name age (* diabetes hypertension) gender surgeon

operation) operation) 1010

2. (< name age gender surgeon operation (* diabetes 2. (< name age gender surgeon operation (* diabetes hypertension) hypertension) 5 5

3. (< name age gender (* diabetes hypertension) surgeon 3. (< name age gender (* diabetes hypertension) surgeon operation) operation) 44

Media NegotiationMedia Negotiation

Graphics components Graphics components produceproduce candidate candidate partial orderspartial orders11. (di (highlight demographics) ((<m) (subhighlight (mrn . (di (highlight demographics) ((<m) (subhighlight (mrn

age gender))(subhighlight (medhistory))(subhighlight age gender))(subhighlight (medhistory))(subhighlight (surgeon operation)))(surgeon operation))) 1010

2. (di (highlight demographics)(* (subhighlight (mrn age 2. (di (highlight demographics)(* (subhighlight (mrn age gender))(subhighlight (medhistory))(subhighlight gender))(subhighlight (medhistory))(subhighlight (surgeon operation)))(surgeon operation))) 7 7

CTS Architecture CTS Architecture

Prosody model Speech Corpus

NLGSystem

ProsodyRealizer

TTS

MachineLearning

Input

OtherSource

Text +Text +

StructureStructure

ProsodicProsodic

AnnotatedAnnotated

TextText

SoundSound

RulesRules

Focus of Research(Rhetorical, Semantic, Syntactic, Prosodic)Focus of Research(Rhetorical, Semantic, Syntactic, Prosodic)

Build a prosody model for CTS using Build a prosody model for CTS using

prosodic features (based on ToBI):prosodic features (based on ToBI): pitch accent, phrase accent, boundary tone, break pitch accent, phrase accent, boundary tone, break

index.index.

Features produced by LGFeatures produced by LG Syntactic structure, POS tags, Semantic boundaries, ConceptSyntactic structure, POS tags, Semantic boundaries, Concept Informativeness, predictability (statistical models)Informativeness, predictability (statistical models) Abnormality, unexpectedness, sequential rhetorical relationAbnormality, unexpectedness, sequential rhetorical relation

Mapping to RAGSMapping to RAGS

Data filter - Data filter - conceptualconceptual

General Content Planner -General Content Planner - rhetoricalrhetorical, , semanticsemantic, , conceptualconceptual

Speech Content Planner - Speech Content Planner - rhetoricalrhetorical, , semanticsemantic plus plus constraints on constraints on lexicalizationlexicalization, , syntaxsyntax

Lexical Chooser - Lexical Chooser - semanticsemantic, , lexicallexical, , syntacticsyntactic

Media Coordination - Media Coordination - semanticsemantic, , conceptualconceptual, , documentdocument

Syntactic Realization - Syntactic Realization - semanticsemantic, , syntacticsyntactic

Prosody Realization -Prosody Realization -rhetoricalrhetorical, , semanticsemantic, , syntacticsyntactic, , prosodicprosodic

AcknowledgmentsAcknowledgments

This work was funded in part byThis work was funded in part by

DARPADARPA NSFNSF ONRONR New York State Center for Advanced New York State Center for Advanced

TechnologyTechnology NLMNLM