Teiresias: Applications of meta-level knowledge

Sam Holmes

History

Stanford University1970’s

Task

Expert AssistantPerformance

program

Task

Expert TeiresiasPerformance

program

Goal

“... make possible the interactive transfer of expertise from a human expert to the knowledge base of a high performance program, in a dialog conducted in a restricted subset of the natural language.”

Architecture

DomainExpert

Explanation

KnowledgeAcquisition

InferenceEngine

KnowledgeBase

Teiresias

Expert’s Job

“Test” the system Ignorance Stupidity Incompetence Formalism inadequacy

Teiresias Functions

ExplanationKnowledge acquisition

Starting point

My therapy recommendation will be based on the following probable infection(s) and potential causative organism(s):

INFECTION-1 is PRIMARY-BACTEREMIA<Item 1> E.COLI [ORGANISM-1]<Item 2> PSEUDOMONAS-AERUGINOSA [ORGANISM-1]

Are all these identities correct?>> NAre there any identities missing?>> YWhich one(s)?>> KLEBSIELLA-NEUMONIAE PROTEUS-NON-MIRABILISAre there any identities that shouldn’t have appeared?>> YWhich one(s)?>> PSEUDOMONAS-AERUGINOSA

Starting point

Do you want to1] track down the problem yourself, or2] allow me to guide the debugging, or3] ignore it for now and fix it another time?

Type 1, 2 or 3>> 1

Explanation

Assumptions: Program recap is acceptable Necessary framework

Methodology: Determine primitives Record primitives Select framework (and/or tree) Program explains trace to user

Explanation

Why?

How?

Explanation

...Then it is definite (1.0) that the site of the culture is a locus of infection in

the patient.>> WHY[i.e., Why is it important to determine a locus of infection in PATIENT-1?][3.0] ... In order to find out about an organism of class of organisms for

which therapy should cover.It has already been established that:[3.1] the site of the culture is blood, and[3.2] the stain of the organism is grampos, and[3.3] the portal of entry of the organism is GI,therefore, if:[3.4] [A – abdomen is a locus of infection in the patient, or, B – pelvis is a locus of infection in the patient]then:There is strongly suggestive evidence (.9) that enterobacteriaceae is a class of organisms for which therapy should cover.

Explanation

...There is strongly suggestice evidence (.9) that enterobacteriaceae is a

class of organisms for which therapy should cover.>> HOW[i.e., How were you determining an organism or class of organisms for

which therapy should cover?][4.0] At that point RULE095 was being used. In addition,

[4.1] RULE162 was used to conclude that bacteroides is an organism for which treatment should cover, and[4.2] RULE115 will be tried later, as well.

Explanation

WHY 1-10 Certainty Special purpose responses Shortcomings Single framework No general / factual explanation Implementation dependent

Starting point II

Do you want to1] track down the problem yourself, or2] allow me to guide the debugging, or3] ignore it for now and fix it another time?

Type 1, 2 or 3>> 2

Rule Acquisition

1. Tracking the error2. Text to rule3. Checking results4. Second guessing5. Final check6. Bookkeeping7. Re-run initial consultation

Tracking the error

We’ll start with the PSEUDOMONAS-AERUGINOSA first.The following rules were used in deducing that the identity of

ORGANISM-1 is pseudomonas-aeruginosa.RULE184 Since:

[1.1] the category of ORGANISM-1 is not known[1.2] the gram stain of ORGANISM-1 is gramneg[1.3] the morphology of ORGANISM-1 is rod[1.4] the aerobicity of ORGANISM-1 is facultative

There is weakly suggestive evidence (.3) that the identity of ORGANISM-1 is pseudomonas-aeruginosa

Is this rule correct?>> Y

Tracking the error

Should it’s premise have failed for this case?>> YWhich clause of the premise should have been false [give #]>> 1No conclusions were reached concerning the category of

ORGANISM-1Should it have been possible to conclude a category?>> YWhich one?>> ENTEROBACTERIACEAEThere were no rules successfully invoked in the attempt to

deduce that the category of ORGANISM-1 is enterobacteriaceae

Tracking the error

The following (potentially applicable) rules were not executed because the starred [**] clause failed:RULE116

....RULE050

....Are these rules correct?>> YShould any of them have been successfully invoked?>> NIs there a rule missing?>> Y

Tracking the error

1. Determine specific error2. Show expert rules used to form

conclusion3. Handle error cases

Rule expansion narrowed by categoryExpectations narrowed as well

Text to rule

Ok, then I need a rule that would allow me to deduce that the category of ORGANISM-1 is enterobacteriaceae.

Would you like to give me such a rule now?>> YGood...The new rule will be called RULE383If: 1 – The patient’s infection is primary-bacteremia

and 2 – the site is one of the sterile sitesand 3 –

Then: 1 – There is evidence (.8) that the category is enterobacteriaceaeand 2 –

This may take a few moments...

Text to rule

This is my understanding of your rule:RULE383IF: 1) Primary-bacteremia is an associated infection for which therapy would cover, and

2) The infection with the organism was acquired while the patient was hospitalized, and

3) The culture was taken from a sterile source, and4) The organism is not normally found at this site

Then:There is strongly suggestive evidence (.8) that the

category of the organism is enterobacteriaceae

Text to rule

Bottom up and top downRule models Examples Description More general More specific

Simple templates

Checking results

This is my understanding of your rule....

Okay? (Yes or No)>> NChange premise?>> YWhat clause should be changed in premise?>> CHANGE 1Suppose (1) were changed to read:

1 – The infection is primary-bacteremiaWould that be ok?>> Y

Checking results

Then you’ll probably want to delete:2) The infection with the organism was acquired while the patient was hospitalized

Correct?>> Y<Delete / Undelete example><Additional Teiresias-suggested wording change>>> OKChange action?>> N

Checking results

This is my understanding of your rule:RULE383IF: 1) The infection is primary-bacteremia, and

2) The site of the culture is one of: those sites that are normally sterileTHEN:

There is strongly suggestive evidence (.8) that the category of the organism is enterobacteriaceae

Okay?>> Y

Checking results

Translation testAlternatives presentedRequires some knowledge to be most effective

Second guessing

I hate to criticize, Dr. Davis, but did you know that most rules about what the category of a organism might be, that mention –

the site of a culturethe infection

Also mention – [A] – the portal of entry of a organism

Shall I try to account for [A]?>> YHow about –

[A] – The portal of entry of the organism is GIOk?>> Y

Second guessing

Also, most rules about what the category of a organism may beAlso conclude about –

[A] – the identity of a organismShall I try to write a clause to account for [A]?>> N

Second guessing

The rule now reads:RULE383IF: 1) The infection is primary-bacteremia, and

2) The site of the culture is one of: those sites that are normally sterile, and

3) The portal of entry of the organism is GITHEN:

There is strongly suggestive evidence (.8) that the category of the organism is enterobacteriaceae

Second guessing

Rule model

Bookkeeping

Please describe briefly why you added (or changed) this rule. Type an empty line when done.

>> The system failed to realize it could conclude >> category, and this allowed rule184 to incorrectly >> conclude identity>>RULE383 has now been added to the knowledge base

Re-run initial consultation

I will now rerun the consultation to check the effectiveness of your new rule. Get comfortable, since this may take a bit.

Also, new questions may be asked as a result of this new rule. If a question is printed, please answer it with reference to this current patient.

<MYCIN OUTPUT>Well, it appears that things are straightened out.Are all these values correct now?>> YGood, then he problem’s been fixed.

Adding rules

Context Meta-knowledge Shortcomings Rule models Natural language Knowledge base impact

Adding concepts

<expert is entering a new rule>IF: ....THEN:

1 – The organism is likely (.4) to be an E.COLISorry but I don’t understand

THE ORGANISM IS LIKELY (.4) TO BE AN E.COLIBecause these words are unknown: E.COLIWould you care to try rephrasing that?>> Y(please use *’s to mark what you think might be a new attribute

or a new value of a attribute)1 – The organism is likely (.4) to be an *E.COLI*

Adding concepts

It looks as thoughE.COLI

Refers to a new value of an attribute, correct?>> Y... Is it a new value of the identity of a organism?>> YNow please tell me a few things about it...Please give the full, formal name for “E.COLI”>> ESCHERICEA COLINow please give all synonyms or abbreviations for ESCHERICEA

COLI which you would like the system to accept:>> E.COLI>>

Adding concepts

Please give a short description of E.COLI as a identity of a organism.

>> ?[e.g., for PSEUDOMONAS, the description is “an organism

commonly found in the hospital, quite often a source of secondary infections in patients whose immune response is weakened”]

>> Normal flora in the lower intestine, but a cause of septicemia when transmitted to the blood stream

What is the aeobicity of E.COLI?The answer should be in the form of 1 or more of the following –

<aerobicity> <certainty factor>>> FACULATATICE .8>> AEROBIC .2>>

Adding concepts

What is the gramstain for a E.COLI?The answer should be in the form of a

<gramstain>>> RODSorry, but the following are invalid

ROD is not a recognized <gramstain>Please enter it again [use the same answer if you really meant

it.]>> GRAMNEG

<morphology example>

[Adding E.COLI to ORGANISMS]Ok, done with E.COLI now...Back to the rule...

Adding concepts

Meta-knowledge systemSimilar to rule modelsAddition by descent

Adding concepts thoughts

Factual knowledge.Is a system necessary?Benefits?Useful?

Conclusions

ExplanationAdding rulesAdding concepts