Graphs and Conditional Independence - Oxford Statisticssteffen/teaching/gm11/intro.pdfp1544 p1567 p1146 p1043 p892 p443p447 p227 p269 p960 p1218 p962 p961 p632 p268 p751 p1419 p1418

ExamplesConditional independence

General independence

Graphs and Conditional Independence

Steffen Lauritzen, University of Oxford

Graphical Models, Lecture 1, Michaelmas Term 2011

October 10, 2011

Steffen Lauritzen, University of Oxford Graphs and Conditional Independence



A directed graphical model

Directed graphical model (Bayesian network) showing relationsbetween risk factors, diseases, and symptoms.




A pedigree

Graphical model for a pedigree from study of Werner’s syndrome.Each node is itself a graphical model.




A large pedigree

p1379p1380 p1381 p1382 p1383p1384p1385p1386p1387

p1129p1128 p1133p1134p1135 p1136p1137

p1536p1535

p1131p1408p1407p1406p1405p1404

p1126

p1396 p1395p1394 p1393p1392 p1391p1390p1389p1388

p1127

p1426 p1425p1424p1423

p1130

p1565p1564p1530

p1132

p1073

p1457p1456p1455 p1454

p1078

p1349

p1079

p1226p1225 p1224p1223 p1222p1221p1220

p1076

p1242 p1241p1240p1239 p1238p1237

p1561

p1236

p1575

p1235

p1499

p1234

p1480p1511 p1534

p1233

p1219

p1074

p1376p1375 p1374 p1373p1372 p1371p1370 p1369p1368

p1077

p1232 p1231p1230 p1229p1228p1227

p1072

p1438p1437p1436 p1435 p1434 p1439

p1080

p1362p1361 p1360

p1190

p1449p1450p1451 p1452p1453

p1187

p1427 p1428p1429 p1430p1431 p1432p1433

p1184 p1189p1188p1186p1185

p1075

p927p928p929

p331

p544p543p542

p328

p1585 p657p659p660

p1556p1555 p1557

p767

p1514

p1568

p765

p875p876p877

p766

p661p662

p745

p842

p1367 p1366p1365 p1364p1363

p746p747

p658

p506

p744p742p743

p512

p712p709p711

p784 p783p782 p781p780p779p778

p704

p864p843

p706

p856p855

p710

p869 p868p870

p708

p1465p1443p1442

p707

p1422

p705

p511

p760p701p702

p1576

p773p879

p772 p774p775 p776p777

p1572 p1571

p840

p703

p507

p716

p509p510p508

p505

p1170

p1573

p1167

p1169p1168 p1166p1165p1163p1162 p1161

p1554p1553

p1164

p964

p1463p1464

p963

p965

p644p645

p504

p571 p572

p1488 p1487p1486

p1173

p1569

p1174 p1171p1175p1176p1177 p1172

p1099

p1183

p1096

p1110 p1109p1108 p1107

p1547p1533

p1296 p1295p1297 p1298p1299 p1300p1301p1302 p1303

p1105

p1350 p1351p1352

p1102p1106p1104 p1103

p1097 p1098p1100p1101

p930 p931

p1591

p1603 p1602p1601 p1600p1599 p1598p1597p1596p1595 p1594 p1604

p1590 p1589

p1586

p1592

p1587

p932p933

p503

p327

p541

p1460p1377

p1581p1580

p1461 p1378

p1081p1082

p1044p1045

p540p937 p621p622

p750 p749

p625

p624 p759

p1093 p1091p1090p1089

p1322p1321p1320p1319 p1318 p1317p1316p1315 p1314p1313p1323

p1092

p623p626

p1083

p934 p935p936 p606

p538

p576

p539

p329p330

p70p51

p34

p1070

p1459 p1094

p1462

p1095

p1071

p1119p1118

p1479

p1117 p1116p1115p1113 p1112

p1359p1358p1357p1356p1355 p1354p1353

p1114

p1532p1542

p1120

p1032

p1159p1158 p1157

p1475p1503p1502 p1501p1500

p1155

p1515 p1543p1485

p1154

p1513p1469

p1153

p1545p1546

p1156p1152 p1151p1160

p1033p1034

p901

p959 p958 p957p956p955

p1069

p1584

p1068

p1468

p1067

p1328

p1063p1066

p1440 p1441

p1065p1064 p1061p1060

p1348 p1347 p1346p1345 p1344 p1343p1342 p1341p1340 p1339

p1062

p954

p1562

p1247

p1484

p1245

p1560 p1559

p1246 p1248p1249p1250 p1251p1252p1253 p1254

p1087p1084 p1085

p1255p1256p1257 p1258 p1259p1260 p1261 p1262 p1263p1264

p1086 p1088

p949 p953 p952p951

p1327 p1324p1325p1326

p1001p1002 p999

p1286

p1507 p1531

p1287p1288p1289 p1290p1291p1292p1293p1294

p998 p997

p1312

p1548p1549

p1304

p1582

p1306p1311 p1310p1309p1308 p1307 p1305

p1000

p1409p1410p1411 p1412 p1413p1414p1415 p1416 p1467

p1148

p1150p1403 p1400 p1401p1402

p1149

p950

p902

p1040 p1031p1041

p903p904p906

p1039p1038 p1037p1036p1035

p905

p263

p198 p26

p440 p441

p647 p646

p442

p231

p228

p446

p730

p513

p1211p1212

p1570p1574

p1210 p1213 p1214 p1215p1216p1217

p519

p518 p517 p516p515p514

p445

p600

p599

p748

p601

p602

p444

p1042

p1147

p1397 p1398p1399

p1141p1142

p1498p1497 p1496p1466

p1143

p1506 p1524p1523p1522

p1144

p1541p1505p1504 p1478p1474

p1145

p1567p1544

p1146

p1043

p892

p443p447

p227

p269

p960

p1218

p961p962

p268p632

p751

p1420p1419 p1418 p1417p1421

p752

p633

p634

p264

p620

p1593

p757

p756

p796 p795p794

p753

p854p848

p754

p755

p619p618

p267 p266

p582

p265

p226

p966p968

p1284p1283 p1282p1281p1280p1279 p1278p1277p1276p1275 p1274 p1273p1285

p969

p970 p971

p972

p973

p1209 p1208p1207 p1206p1205p1204p1203

p1566

p1202

p1470p1471p1472

p1200p1198

p1540 p1539p1538p1537

p1199

p1516p1517 p1518p1519

p1201

p967

p888

p229

p230

p5p6

p907 p908p909p910p911 p912

p560 p562

p570p569

p561p563

p106 p340

p181

p439

p438

p1550 p1551 p1552

p736

p731

p732 p846 p803p802 p801 p800 p799p798p797

p733p734p735p737

p536p594

p847

p652

p651

p650

p648 p649

p810 p809p808p807p806 p805p804

p653

p593

p1003

p764p763p762

p861p862

p761

p729

p595 p596p597p598

p437

p881 p880

p436

p526

p434

p1028

p1448p1447 p1446p1445p1444

p1029p1030

p476

p700p699 p698p697

p849

p696

p818p819p820 p821p822p823 p824 p825

p695

p865p872p878

p694 p693p689 p690 p692

p811p812 p813p814 p815p816p817

p691

p683 p1022p1024 p1025

p1528p1527p1526p1525 p1529

p1179

p1583 p1563

p1180p1181

p1178

p1023

p1243 p1244

p1473

p1140 p1138

p1489p1476

p1139

p1026p1027

p919p921 p920p887 p886

p435

p284

p581p580p579p578

p341 p342

p49 p183p182

p73 p346

p52 p185

p283

p259

p564

p260

p395

p399

p677

p758

p676

p397

p720 p722p721p719 p718 p717

p398

p396

p549

p257

p258

p394

p845

p672p667 p668 p671

p669 p670p866p857

p673 p674

p675

p391

p389p1111

p390p393p392

p261

p262

p27p28

p252

p474

p640 p643

p826p827p828 p829p830 p831 p832p839

p642p641 p639

p256

p410

p400

p412

p401

p414p413 p411

p402p403p404 p405 p406p407p408 p409

p387p386 p385

p489p490

p383 p384

p1059p1057 p1058

p388

p251 p254 p255

p863

p841p792 p791p790p789p788p787 p786p785

p738

p853p852 p851 p850

p739

p873 p874p867

p740

p502

p1579

p1056p1050

p1048 p1049 p1051

p1495

p1052p1053p1054

p1578 p1577

p1055

p501

p678

p793

p682

p681

p680p679

p500

p498p499

p253

p241

p475

p242

p943 p974p975p976p977 p978p979

p890

p1182p980p981p982 p983p984 p985

p889 p473

p1007p1008 p1009 p1010 p1011 p1012

p915p917 p918

p1021

p916

p461

p1013

p891

p243

p15 p16

p535

p306

p532

p713

p556p714

p715

p531p530p529p528p527

p304

p728p726

p1477p1490

p1193

p1491p1492 p1493p1494

p1192

p1191 p741p1194

p727

p555

p305

p36 p37

p148

p525

p3

p431

p885p884

p429

p883p882

p430

p638

p1046 p1047

p637 p635p636

p428

p1020 p1018p1016

p1512

p1458 p1197

p1017

p1019

p432

p433

p274p275

p84p85

p497

p858p844

p836

p496

p838

p688 p687 p686 p685p684

p491

p723

p494p495 p493 p492

p272

p419p416 p418p417

p420

p273

p8

p1014 p1015

p893

p995 p994

p1483p1482 p1481

p1196p1195

p996

p894

p895 p896

p232 p449

p603 p605

p604

p448

p382

p768 p769p770p771

p381

p380

p379

p378 p630p629p628 p627

p376

p617 p612p614

p1329 p1330p1331p1332 p1333 p1334p1335p1336p1337 p1338

p613

p871

p616

p615

p611

p377

p233

p450

p2

p270p271

p427

p860 p859

p837

p725 p724

p533 p534

p426

p554p553

p415

p278

p41

p577 p458p456 p454p455p453 p460p459 p457

p336p337

p42p56

p551p550 p552

p50p314

p315

p98p99

p900 p899 p897p898

p573

p942

p574 p575

p566

p941

p944

p939p940

p565p567 p568

p296p297

p92 p93

p343

p352p351 p350

p344

p104 p105

p76

p246 p349 p472p471p470p469

p468

p466

p609

p610

p607p608

p467 p465

p244

p425 p421

p423 p424 p422

p245p247

p17 p18

p1

p520

p656 p655

p664

p663

p665

p835p834 p833

p666

p654

p521

p631p922p923

p546p548 p547 p545 p926 p925p924

p522

p279

p79 p80

p299

p39p40

p477p478

p479

p480

p287

p312

p353

p311

p332 p333p523 p524

p310

p288

p89 p90

p43

p187 p55 p67

p174

p69

p1004

p1005

p1006

p913

p1125

p1124 p1123p1122

p1121

p914

p145 p463

p464

p462

p236 p234p235

p237

p61 p62

p11 p7

p83

p298

p33p178

p206

p250

p22p249p248

p19 p20

p286

p176 p285

p483 p482p481

p291 p307

p308

p309

p29 p97

p150

p159

p127

p177

p25

p301p300

p94p95

p316

p317 p318

p100

p488p487 p486p485

p484

p335p334

p292 p293p294p295

p68 p91

p31

p320p345 p451

p452

p10p282

p59 p60

p96

p125

p363

p141 p155p24

p23

p146

p120p110 p132p130

p4p12

p47

p240

p339p290 p289

p238p239

p13p14

p180p38p321

p48 p175

p144

p211 p375

p374

p369 p588

p1520p1521

p586

p1508p1509 p1510

p584

p1558

p945

p585

p368

p151 p152

p195

p372

p592

p591p589

p590

p373

p170p171

p168p364

p142

p172 p202

p131p163

p169

p118

p107 p122p119

p989

p1272p1271 p1270p1269 p1268p1267 p1266 p1265

p986 p987

p990

p991p992p993

p1614p1613 p1612p1611 p1610p1609p1608p1607p1606 p1605

p988

p938

p143 p537

p46p280 p281

p87 p88

p276

p277

p86 p72

p203

p116 p123p109

p138

p164

p200

p225

p192

p947 p1588

p946p948

p139p587

p193 p365

p367

p366

p153p154

p194

p189

p147

p124p219

p191

p126p128 p583

p362

p190p137

p188

p165 p157

p173

p114 p160

p149 p207 p112 p224

p319

p35

p9

p121

p204 p201

p216 p166

p133

p208

p220p217 p371

p370

p134 p135

p108

p117

p162

p111

p221

p115

p158

p205

p167

p215p213

p196

p197

p223

p214p209

p179

p113p212 p222p129 p210p161

p184

p103

p156

p218

p557p558 p559

p313

p44 p199

p53

p140

p82

p303 p302

p21 p30

p45 p136

p322p323p324

p63 p64

p347

p325p326

p101 p102

p74p338

p65p66

p348

p75p186

p359p360p361

p354p355 p356p357p358

p77p78

p81

Family relationship of 1641 members of Greenland Eskimopopulation.




Basic ideas and pitfallsFormal definitionFundamental properties

Independence

We recall that two random variables X and Y are independent if

P(X ∈ A |Y = y) = P(X ∈ A)

or, equivalently, if

P{(X ∈ A) ∩ (Y ∈ B)} = P(X ∈ A)P(Y ∈ B).

For discrete variables this is equivalent to

pij = pi+p+j

where pij = P(X = i ,Y = j) and pi+ =∑

j pij etc., whereas forcontinuous variables the requirement is a factorization of the jointdensity:

fXY (x , y) = fX (x)fY (y).

When X and Y are independent we write X ⊥⊥Y .Steffen Lauritzen, University of Oxford Graphs and Conditional Independence




Admissions to Berkeley by department

Here are three variables A: Admitted?, S : Sex, and D:Department.

Department Sex Whether admittedYes No

I Male 512 313Female 89 19

II Male 353 207Female 17 8

III Male 120 205Female 202 391

IV Male 138 279Female 131 244

V Male 53 138Female 94 299

VI Male 22 351Female 24 317

When dealing with complex systems of many random variables, wemust have a concept which is more sophisticated, but equallyfundamental: that of conditional independence.





For three variables it is of interest to see whether independenceholds for fixed value of one of them, e.g. is the admissionindependent of sex for every department separately?

We denote this as A⊥⊥ S |D and display it graphically asu u uA D S

Algebraically, this corresponds to the relations

pijk = pi+|k p+j |k p++k =pi+k p+jk

p++k.





Marginal and conditional independence

Note that there the two conditions

A⊥⊥ S , A⊥⊥ S |D

are very different and will typically not both hold unless we eitherhave A⊥⊥ (D,S) or (A,D)⊥⊥ S , i.e. if one of the variables arecompletely independent of both of the others.

This fact is a simple form of what is known as Yule–Simpsonparadox.It can be much worse than this: A positive conditional associationcan turn into a negative marginal association and vice-versa.





Admissions revisited

Admissions to Berkeley

Sex Whether admittedYes No

Male 1198 1493Female 557 1278

Note this marginal table shows much lower admission rates forfemales.Considering the departments separately, there is only a differencefor department I, and it is the other way around...





Admissions to Berkeley by department

Department Sex Whether admittedYes No

I Male 512 313Female 89 19

II Male 353 207Female 17 8

III Male 120 205Female 202 391

IV Male 138 279Female 131 244

V Male 53 138Female 94 299

VI Male 22 351Female 24 317

Apart from Department I, it holds that A⊥⊥ S |D. In DepartmentI, a higher proportion of females are admitted!





Florida murderers

Sentences in 4863 murder cases in Florida over the six years1973-78

SentenceMurderer Death Other

Black 59 2547White 72 2185

The table shows a greater proportion of white murderers receivingdeath sentence than black (3.2% vs. 2.3%), although thedifference is not big, the picture seems clear.





Controlling for colour of victim

SentenceVictim Murderer Death Other

Black Black 11 2309White 0 111

White Black 48 238White 72 2074

Now the table for given colour of victim shows a very differentpicture. In particular, note that 111 white murderers killed blackvictims and none were sentenced to death.





Formal definition

Random variables X and Y are conditionally independent given therandom variable Z if

L(X |Y ,Z ) = L(X |Z ).

We then write X ⊥⊥Y |Z (or X ⊥⊥P Y |Z )Intuitively:Knowing Z renders Y irrelevant for predicting X .Factorisation of densities:

X ⊥⊥Y |Z ⇐⇒ f (x , y , z)f (z) = f (x , z)f (y , z)⇐⇒ ∃a, b : f (x , y , z) = a(x , z)b(y , z).





Undirected graphical models

3 6

1 5 7

2 4

u uu u u

u u@@@

��

@@@

@@@

@@@

��

��

For several variables, complex systems of conditional independencecan for example be described by undirected graphs.Then a set of variables A is conditionally independent of set B,given the values of a set of variables C if C separates A from B.

For example in picture above

1⊥⊥{4, 7} | {2, 3}, {1, 2}⊥⊥ 7 | {4, 5, 6}.





Directed graphical models

Directed graphs are also natural models for conditionalindpendence:

3 6

1 5 7

2 4

u uu u u

u u

-

@@@R

��

@@@R

-

@@@R

@@@R

��

��

-

Any node is conditional independent of its non-descendants, givenits immediate parents. So, for example, in the above picture wehave

5⊥⊥{1, 4} | {2, 3}, 6⊥⊥{1, 2, 4} | {3, 5}.





Proof of (C5)

We have

X ⊥⊥Y | (Z ,W ) ⇒ f (x , y , z ,w) = a(x , z ,w)b(y , z ,w).Similarly

X ⊥⊥Z | (Y ,W ) ⇒ f (x , y , z ,w) = g(x , y ,w)h(y , z ,w).If f (x , y , z ,w) > 0 for all (x , y , z ,w) it thus follows that

g(x , y ,w) = a(x , z ,w)b(y , z ,w)/h(y , z ,w).

The left-hand side does not depend on z so let z = z0 be fixed.Then we have

g(x , y ,w) = ã(x ,w)b̃(y ,w).

Insert this into the second expression for f to get

f (x , y , z ,w) = ã(x ,w)b̃(y ,w)h(y , z ,w) = a∗(x ,w)b∗(y , z ,w)

which shows X ⊥⊥ (Y ,Z ) |W .Steffen Lauritzen, University of Oxford Graphs and Conditional Independence



Graphoids and independence modelsExamples

Conditional independence can be seen as encoding abstractirrelevance. With the interpretation: Knowing C , A is irrelevant forlearning B, (C1)–(C4) translate into:

(I1) If, knowing C , learning A is irrelevant for learning B,then B is irrelevant for learning A;

(I2) If, knowing C , learning A is irrelevant for learning B,then A is irrelevant for learning any part D of B;

(I3) If, knowing C , learning A is irrelevant for learning B,it remains irrelevant having learnt any part D of B;

(I4) If, knowing C , learning A is irrelevant for learning Band, having also learnt A, D remains irrelevant forlearning B, then both of A and D are irrelevant forlearning B.

The property analogous to (C5) is slightly more subtle and notgenerally obvious. Also the symmetry (C1) is a special property ofprobabilistic conditional independence, rather than of generalirrelevance.





Separation in undirected graphs

Let G = (V ,E ) be finite and simple undirected graph (noself-loops, no multiple edges).

For subsets A,B,S of V , let A⊥G B | S denote that S separates Afrom B in G, i.e. that all paths from A to B intersect S .Fact: The relation ⊥G on subsets of V is a compositionalgraphoid.

This fact is the reason for choosing the name ‘graphoid’ for suchindependence model.





Systems of random variables

For a system V of labeled random variables Xv , v ∈ V , we use theshorthand

A⊥⊥B |C ⇐⇒ XA⊥⊥XB |XC ,

where XA = (Xv , v ∈ A) denotes the variables with labels in A.The properties (C1)–(C4) imply that ⊥⊥ satisfies thesemi-graphoid axioms for such a system, and the

An independence model of this kind is said to be probabilistic.

Probabilistic independence models are often not compositional.


ExamplesConditional independenceBasic ideas and pitfallsFormal definitionFundamental properties

General independenceGraphoids and independence modelsExamples

Documents

Graphs and Conditional Independence - Oxford Statisticssteffen/teaching/gm11/intro.pdfp1544 p1567 p1146 p1043 p892 p443p447 p227 p269 p960 p1218 p962 p961 p632 p268 p751 p1419 p1418