Agency

Mark ZangariCEO Quantellia

Agency

Agency’s aims:

1. Extend using machine learning into a new domain of problems.

2. The beginnings of a formal technique for linking data science to business levers.

3. Instead of answering “Given input data A, what output data B does my model predict?”, Agency answers “If I do A to complex dynamical system S, with the intention of achieving objective B, what is the probability that S’s outputs are closer to B?”

Agency

The four pillars for effective machine learning:

1. Availability of training data

2. System stability over characterstic times(a) training data period (past)(b) prediction period (future)

3. Availability of input data at the time the prediction is to be made.

4. Sufficient signal-to-noise in data for pattern recognition to be possible.

Agency

The four pillars for effective machine learning:

What can we do if one or more of the four pillars are not present?

Agency

The four pillars for effective machine learning:

What can we do if one or more of the four pillars are not present?

Example from finance:Commercial lending vs. consumer lending.

Agency

Consumer Credit:1. Ample, accessible data2. Large, consistent, classifiable population 3. Relevant variables easily measured4. Strong correlations between input variables and credit risk.

Consumer

Credit Transactions

Data Warehouse

Rating System Credit Score

Bill Fair & Earl Isaac

Contacted 58 of the nation’s top lending institutions in 1958 offering to show them how using data would help them make better credit decisions...

Only one responded.

Image composed from stock images and portraits of Fair and Isaac at http://www.fico.com/en/about-us#our_history

Agency

Consumer Credit:1. Ample, accessible data2. Large, consistent, classifiable population 3. Relevant variables easily measured4. Strong correlations between input variables and credit risk.

Commercial Credit for Small/Medium Businesses:1. Little data available, hard to obtain2. Each business different, many are very new with little history.3. Complex inter-entity relationships affect credit risk.4. Many correlations , hard to isolate those that are good predictors

of credit risk.

Agency

Fully automated credit rating is rarely used for scoring Small/Medium Business Loans.

The process commonly used is a good example of Agency in action…

Agency

Type of Business

Loan Amount Probability of Default

Typical Model Decomposition for Commercial Loan Decision

Grant this Loan?

Profitable Loan Book

Desired ObjectiveBusiness Lever

ExternalInputs

Agency

Agency Principle One:

Learning and other analytics are designed to discriminate actions that will increase the probability of the objectives being met, from actions that do not.

So…

What is an “objective?”

Agency

Objective:

Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes.

Probability of default

Des

ired

Lo

an F

req

uen

cy

Agency

Objective:

Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes.

Probability of default

Des

ired

Lo

an F

req

uen

cy

Agency

Type of Business

Loan Amount Probability of Default

Typical Model Decomposition for Commercial Loan Decision

Grant this Loan?

Profitable Loan Book

Desired ObjectiveBusiness Levers

ExternalInputs

Measure Measure Learn

Interest Rate

Agency

Agency Principle Two:

If a system does not ideally support machine learning because it does not satisfy the four pillars, decompose it using a dynamic system model until each link between each node either:(a) Satisfies the four pillars, or(b) Can be described using known domain-specific

relationships (formulas, rules, approximations, etc.)

Agency

Financial Spreading

Values

Market & Industry Data

Management Team Bios & Experience

Relationships to other Entities

Key Ratios

Aggregate Statistics

Scoring

Probability of Default

Known mathematical relationships

Machine Learning(intangibles)

Rating

Machine Learning

Transform Tables

Typical Model Decomposition for Business Credit Rating

Feature Engineering

Agency

Management Team

Experience

Agency gives us insight into decisions via the model levers.

E.g. What interest rate do we charge and how does thisaffect our success?

Interest Rate 1

Management Team

ExperienceInterest Rate 2

Interest Rate 1 Interest Rate 2

Agency

Summary of characteristics where Agency becomes highly relevant and useful:

• Sparse data

• Complex emergent dynamics, including feedback loops, phase changes, and other non-linear effects.

• Provides a way of including intangibles with few measureables in the model

• Can utilize relationships that the training data does not make apparent.

Agency

A (brief) mathematical representation of Agency:

We draw analogy from the information measure:

The “Agency” A of a lever L, which maps some part of the output space to favorable outcomes, f and some other part to unfavorable outcomes, with measure M(f) and M(u) is given by:

Agency

Agency:

• Helps business users connect real-world decisionswith support from data.

• Provides a architecture for using machine learning in situations that are not typically well suited to ML solutions.

• Allows ML results to be extrapolated beyond information contained in the data, by integrating knowledge of system dynamics.