EXPLAINABLE AI (AND RELATED CONCEPTS)

A QUICK TOUR

AI Present and future

Jacques Fleuriot

STATE OF PLAY• Generally, machine learning models are black boxes

• Not intuitive

• Difficult for humans to understand, often even by their designers do not fully understand the decision-making procedures.

• Yet they are being widely deployed in ways that affect our daily lives

• Bad press when things go wrong

• We’ll look at a few areas/case studies quickly but this is an active, fast growing research field and there is much, much more going on.

SOME OF THE ISSUES• Fairness/Bias

• e.g. Amazon ML tool for recruitment was found to be biased against women (Reuters, Oct 2018)

• Even what one might consider benign e.g. Netflix serving artworks for movies that many thought were based on racial profiling (Guardian, Oct 2018)

• We know that “bad” training data can result in biases and unfairness

• Challenge: How does one define “fairness” in a rigorous, concrete way (in order to model it)?

• Trust

• One survey: 9% of respondents said they trusted AI with their financials, and only 4% trusted it for hiring processes (Can We Solve AI’s ‘Trust Problem’? MIT Sloan Management Review, November 2018)

• Safety

• Can one be sure that a self-driving car will not behave dangerously in situations never encountered before?

• Ethics (see previous lectures)

EXPLAINABLE AI• Not a new topic

• Rule-based systems are generally viewed as explainable (but are not scalable, able to deal with data, etc.)

• Example: MYCIN expert system (with ~500 rules) for diagnosing patients based on reported symptoms (1970s)

• It could be asked to explain the reasoning leading to the diagnosis and recommendation

• It operated at the same level of competence as a specialist and better than a GP

• Poor interface and relatively limited compute power at the time

• Ethical and legal issues related to the use of computers in medicine were raised (even) at the time

• Are there ways of marrying powerful blackbox/ML approaches with (higher-level) symbolic reasoning?

DARPA’S XAI: ONE POSSIBLE VISION

Learning Process

Video & Social Media

Learned Function

Output

Today

This incident is a violation

(p = .93)

Video & Social Media

New Learning Process

Explainable Model

Explanation Interface

Tomorrow •  I understand why •  I understand the

evidence for this recommendation •  This is clearly one to

investigate

What should I report?

What should I report?

Detecting Ceasefire ViolationsIncident

Detecting Ceasefire ViolationsIncident

These events occur before tweet reports

This is a violation:

•  Why do you say that? •  What is the evidence? •  Could it have been an

accident? •  I don’t know if this

should be reported or not

Source: DARPA XAI

PERFORMANCE VS EXPLAINABILITY

Lear

ning

Per

form

ance

Explainability

Neural Nets

Statistical Models

Ensemble Methods

Decision Trees

Deep Learning

SVMs

AOGs

Bayesian Belief Nets

Markov Models

HBNs

MLNs

SRL CRFs

Random Forests

Graphical Models

Source: DARPA XAI

Explainability

Neural Nets

Statistical Models

Ensemble Methods

Decision Trees

Deep Learning

SVMs

AOGs

Bayesian Belief Nets

Markov Models

HBNs

MLNs

Model Induction Techniques to infer an explainable model from any model as a black

box

Deep Explanation Modified deep learning

techniques to learn explainable features

SRL

Interpretable Models Techniques to learn more

structured, interpretable, causal models

CRFs Random Forests

Graphical Models

Lear

ning

Per

form

ance

PERFORMANCE VS EXPLAINABILITY

Source: DARPA XAI

INTERPRETABLE MODELS• Example: Human-level concept learning through probabilistic

program induction (Lake et al. 2015, Science)

• “Model represents concepts as simple programs that best explain observed examples under a Bayesian criterion”

• Bayesian Program Learning: Learn visual concepts from just a single example and genralise in a human-like fashion — one-shot learning

• Key ideas: compositionality, causality and learning to learn

• Note: Interpretable and explainable are not necessarily the same

INTERPRETABLE MODEL

Source: DARPA XAI

DATASET

• Dataset of 1623 characters from 50 writing systems

• Images and pen strokes collected

• Good for comparing humans and machine learning:

• cognitively natural and are used as benchmarks for ML algorithm

GENERATIVE MODEL

BAYESIAN PROGRAM LEARNING

• Lake et al. showed that it is possible to perform one-shot learning at human-level accuracy

• Most judges couldn’t distinguish between the machine- and human-generated characters in (“visual Turing”) tests.

• However, BPL still sees less structure in visual concepts than humans

• Also what’s the relationship with explainability?

MODEL INDUCTION• Example: Bayesian Rule Lists (Letham et al., Ann. of Applied

Statistics, 2015)

• Aim: Predictive models that are not only accurate, but are also interpretable to human experts.

• Models are decision lists, which consist of a series of if...then...statements

• Approach: Produce a posterior distribution over permutations of if...then... rules, starting from a large, pre-mined set of possible rules

• Used to develop interpretable patient-level predictive models using massive observational medical data

BAYESIAN RULE LISTS

• BRLs can discretise a high-dimensional, multivariate feature space into a series of interpretable decision statements e.g. 4146 unique medications and condition for above + age and gender

• Experiments showed that BRLs have predictive accuracy on par with the top ML algorithms at the time (approx. 85- 90% as effective) but with models that are much more interpretable

• For technical details of underlying generative model, MCMC sampling, etc. (see paper).

USING VISUALISATION• Example: Interpretable Learning for Self-Driving Cars by Visualizing

Causal Attention (Kim and Canny, ICCV 2017)

• If deep neural perception and control networks are to be a key component of self-driving vehicles, these models need to be explainable

• Visual explanations in the form of real-time highlighted regions of an image that causally influence the car steering control (i.e. the deep network output)

USING VISUALISATION

• Method highlights regions that causally influence deep neural perception and control networks for self-driving cars.

• The visual attention model is augmented with an additional layer of causal filtering.

• Does this correspond to where a driver would gaze though?

USING VISUALISATION AND TEXT

• Video-to-text language model to produce textual rationales that justify the model’s decision

• The explanation generator uses a spatio-temporal attention mechanism, which is encouraged to match the controller’s attention Show, Attend, Control, and Justify: Interpretable Learning for Self-Driving Cars.

Kim et al, Interpretable ML Symposium (NIPS 2017)

COMBINING DEEP LEARNING AND SYMBOLIC/LOGICAL REASONING

• Example: DeepProbLog: Neural Probabilistic Logic Programming, (Manhaeve et al., 2018)

• Deep learning + ProbLog = DeepProbLog !

• Approach that aims to “integrate low-level perception with high-level reasoning”

• Incorporate DL into probabilistic LP such that

• probabilistic/logical modelling and reasoning is possible

• general purpose NNs are possible

• end-to-end training is possible

(Just when you thought you were done with logic programming)

DEEP PROBLOG

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

DEEP PROBLOG

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

EXAMPLE

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

DEEP PROBLOG• Relies on the fact that there is a differentiable version of ProbLog that allows for parameters

update of the logic program using gradient descent

• Seemless integration with NN training (which uses backprop) is thus possible

• Allows for a combination of “symbolic and sub-symbolic reasoning and learning, program induction and probabilistic reasoning”

• For technical details, consult the paper by Manhaeve et al.

DEEP + SYMBOLIC• Recent exciting work (aside from one involving ProbLog!):

• Harnessing Deep Neural Networks with Logic Rules (Hu et al., 2016)

• End-to-End Differentiable Proving (Rocktäschel and Riedel, 2017)

• Logical Rule Induction and Theory Learning Using Neural Theorem Proving (Campero et al., 2018)

• Planning Chemical Syntheses with Deep Neural Networks and Symbolic AI (Segler et al., 2018)

• and many more…

CONCLUSION• Current ML models tend to be opaque and there’s an urgent need for interpretability/

explainability to ensure fairness, trust, safety, etc.

• Rapidly moving research area that is still wide open because (among many other issues):

• It’s unclear to many ML/DL researchers how their models actually achieve their decisions (yet alone come up with explanation or interpretation, or both)

• What is an explanation anyway?

• DARPA XAI is a step in the right direction (but there are other initiatives and views on the topic)

• Logical/symbolic representations and inference provide high-level (abstract) means of reasoning, these are usually explainable too

• Combining probabilistic, symbolic and sub-symbolic reasoning and learning seems promising

• Finally, as this course tried to emphasise, AI ≠ ML (or DL)