Incorporating Prior Information in Causal Discovery Rodney O'Donnell, Jahangir Alam, Bin Han,...

Incorporating Prior Information in Causal Discovery

Rodney O'Donnell, Jahangir Alam,

Bin Han, Kevin Korb and Ann Nicholson

Outline• Methods for learning causal models

– Data mining, Elicitation, Hybrid approach

• Algorithms for learning causal models– Constraint based– Metric based (including our CaMML)

• Incorporating priors into CaMML– 5 different types of priors

• Experimental Design

• Experimental Results

Learning Causal Bayesian Networks

Elicitation Data mining

Requires domain knowledge Requires large dataset

Expensive and time-consuming

Sometimes, the algorithms are “stupid”

(no prior knowledge →no common sense)

Partial knowledge may be insufficient

Data only tells part of the story

A hybrid approach

• Combine the domain knowledge and the facts learned from data

• Minimize the expert’s effort in domain knowledge elicitation

Elicitation Data Mining

Causal BN

• Enhance the efficiency of the learning process– Reduce / bias the search space

Objectives

• Generate different prior specification methods

• Comparatively study the influences of priors on the BN structural learning

• Future: apply the methods to the Heart Disease modeling project

Causal learning algorithms

• Constraint based– Pearl & Verma’s algorithm, PC

• Metric based– MML, MDL, BIC, BDe, K2,

K2+MWST,GES,CaMML

• Priors on structure– Optional vs. Required – Hard vs. Soft

Priors on structure

Required Optional Hard Soft

K2 (BNT) yes yes

K2+MWST

(BNT)yes yes

GES

(Tetrad)yes yes

PC (Tetrad ) yes yes

CaMML yes yes yes

CaMML• MML metric based• MML vs. MDL

– MML can be derived from Bayes’ Theorem (Wallace)– MDL is a non-Bayesian method

• Search: MCMC sampling through TOM space– TOM = DAG + total ordering – TOM is finer than DAG

A

B C

Two TOMs: ABC, ACB

A B C

One Tom: ABC

Priors in CaMML: arcs

Experts may provide priors on pairwise relations:1. Directed arcs:

– e.g. {A→B 0.7} (soft)– e.g. {A→D 1.0} (hard)

2. Undirected arcs– E.g. {A─C 0.6} (soft)

3. {A→B 0.7; B→A 0.8; A─C 0.6}– Represented by 2 adjacency matrices

0.7

0.8

A B C

A

B

C

Directed arcs

0.6

0.6

A B C

A

B

C

Undirected arcs

Priors in CaMML: arcs (continued)

• MML cost for each pair

AB: log(0.7) + log(1-0.8)

AC: log(1-0.6)

BC: log( default arc prior)

expert specified network

A0.7

0.8

0.6

B C

One candidate network

A

B C

Priors in CaMML: Tiers

• Expert can provide prior on an additional pairwise relation

• Tier: Temporal ordering of variables

E.g., Tier {A>>C 0.6;B>>C 0.8}

IMML(h)=log(0.6)+log(1-0.8)

A

C

B

One possible TOM

Priors in CaMML: edPrior

• Expert specifies single network, plus a confidence– e.g. EdConf=0.7

• Prior is based on edit distance from this network

A

B

Expert specified network

C

IMML(h)=-2*(log0.7-log(1-0.7))

One candidate network :ED=2

A

B C

Priors in CaMML: KTPrior

• Again, expert specifies single network, plus a confidence– e.g. KTConf = 0.7

• Prior is based on Kendall-Tau Edit distance from this network – KTEditDist = KT + undirected ED

A B C

Expert specified dagTOM: ABC

A

B C

A candidate TOM: ACB

IMML(h)=-3*(log0.7-log(1-0.7))

• B-C order in expert TOM disagrees with candidate TOM • KTEditDist = KT(1) + Undirected ED (2) = 3

Experiment 1: Design• Prior

– weak, strong– correct, incorrect

• Size of dataset– 100,1000,10k and 100k– For each size we randomly generate 30 datasets

• Algorithms: – CaMML– K2 (BNT)– K2+MWST (BNT)– GES (TETRAD)– PC (TETRAD)

• Models: AsiaNet, “Model6”(An artificial model)

Models: AsiaNet and “Model6”

Experimental Design

Priors

Algorithms

Sample Size

Experiment Design: Evaluation

• ED: Difference between Structures

• KL: Difference between distributions

Model6 (1000 samples)

Model6 (10k samples)

AsiaNet (1000 Samples)

Experiment 1: Results

• With default priors: CaMML is comparable to or outperforms other algorithms

• With full tiers: – There is no statistically significant differences

between CaMML and K2– GES is slightly behind, PC performs poorly.

• CaMML is the only method allowing soft priors: – with the prior 0.7, CaMML is comparable to other

algorithms with full tiers– With stronger prior, CaMML performs better

• CaMML performs significantly better with expert’s priors than with uniform priors

Expertiment 2:Is CaMML well calibrated?

• Biased prior– Expert’s confidence may not be consistent

with the expert’s skill

e.g, expert 0.99 sure but wrong about a connection

– Biased hard prior– Soft prior and data will eventually overcome

the bad prior

Is CaMML well calibrated?

• Question: Does CaMML reward well calibrated experts?

• Experimental design– Objective measure: How good is a proposed

structure?: • ED: 0-14

– Subjective measure: Expert’s confidence• 0.5 to 0.9999

– How good is the learned structure?• KL distance

Effect of expert skill and confidence on quality of learned model

Better ← Expert Skill → Worse

Overconfidence penalized

Justified confidence rewarded

Unconfident expert

Experiment 2: Results

• CaMML improves the elicited structure and approaches the true structure

• CaMML improves when the expert confidence matches with the expert skill

Conclusions

• CaMML is comparable to other algorithms when given equivalent prior knowledge

• CaMML can incorporate more flexible prior knowledge

• CaMML’s results improve when expert is skillful or well calibrated

Thanks