ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30
Hershey Lodge ▪ Hershey, Pennsylvania
Using the Power of Predictive
Analytics for Case Outcomes
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Participants
Presenter – Edward V. Lehman, Jr.
Director, Case Processing & Data Management
Philadelphia Family Court – Domestic Relations
Division
Presenter – Steven J. Golightly, Ph.D.
Los Angeles County Child Support
Moderator – Joyce Match
Business Analyst Manager
Pennsylvania Bureau of Child Support
Enforcement
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 Hershey Lodge ▪ Hershey, Pennsylvania
Predictive Analytics in Action Edward V. Lehmann, Jr.
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Access to the right information is extremely important
What is the requirement?
Increase Visible Results
Reduce Information
Overload
Work the Right Cases
Effective Case Worker Management
Determine Next Appropriate
Action Proactively
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Understanding Past, Measuring Present, Predicting the Future
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Predictive Analytics in Child Support
• Better estimates of delinquencies, child support collections • Fewer child support cases in arrears • Model results can be used to quantify historical process inefficiencies • Significant marketing value – “We Know Our Citizens” • More reliable payments for children
• Shift from reactive enforcement to early intervention • Learning component for new case management approaches • Assign high risk cases to case workers sooner • Different approaches for different regions and different case types • Automate certain research processes
• Reduce user guesswork and “cherry picking” • Prioritize workload based on high impact • Incorporate historical experience to drive future activities • Eliminate a one size fits all approach • Efficiency of customer service interactions • Over time reduce future workload
Improved Workforce
Effectiveness
Increased Resource Allocation Efficiency
Improved Outcomes
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Predictive Modeling for Child Support
o Child support enforcement has traditionally been a reactive process.
o What if we had a tool that could help us predict which NCPs are most likely to become in-arrears in the near future?
o We could use such a tool to: o Prevent arrears o Decrease custodial parent complaints o Take the right action on a case at the right time o Assign the right case workers to the right cases o Gather information for potential policy changes
Use analytics to make smarter decisions and do more with less.
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
How do Models Work?
Statistical models just refine what we do naturally all the time.
For example: which of these NCP’s is most/least likely to pay next month?
Gene: 35 years old $2000 in arrears No other children on case No Salary attachment History of family violence Case is a IV-A assistance
Tom: 28 years old $100 in arrears 1 other child on case Salary attachment No history of violence Case is not an assistance case
Rick: 39 years old $8000 in arrears No other children on case No salary attachment No history of violence NCP has other child support cases
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
How do Models Work?
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Pennsylvania’s Predictive Analytics solution included updates to the existing Performance Improvement Module (PIM) solution and a new application to calculate a predictive analytics “score.”
• Allows a worker to calculate a score for a case based on 20
variables • Score is the likelihood of the defendant to pay 80% towards
current support obligation in the next three months • Information required to generate the score is easily accessed
from the system at the time the support order is created or modified
• Provides a list of upcoming child support establishment conferences with the cases assigned to the worker to allow the score creation prior to the conference.
Pennsylvania’s Solution
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
PIM/Predictive Analytics
o Targeted case lists o Suggested case actions o Predictive score and
reasons o Performance metrics o Projects for targeted,
custom outreach including text messages
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Payment Score Calculator Objectives
• Establish a good
payment pattern: Increased quantity and frequency of collections
• More effective meetings with defendants
• Make available new methods of reaching out
• Identifying opportunities for proactive enforcement activities
• Maximize
performance metrics
• Minimize costs related to enforcement
• Improve cost/effectiveness ratios
• Effective case
assignment based on scores
• Establishing consistent payment patterns based on successful business process actions for various scores
• Provide the right services at the right time to encourage compliance with the order
• Let the defendant know the case is being monitored
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Variable Selection Final model contains 20 variables
36 variables considered for modeling after Exploratory Data Analysis [EDA] Phase
64 candidate variables created during data scrubbing / brainstorming phase
Over 400 data elements collected and considered in variable creation phase
The case for which a score is being calculated is compared against the historical behavior of thousands of cases with similar characteristics
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Example Predictive Variables Number Predictive Variables
1 Collection Indicator 2 High Number of Enforcement Activities Indicator 3 Number of Cases 4 Number of Enforcement Activities 5 Balance of Arrears 6 Defendant Net Income 7 Active Income Attachment Indicator 8 Number of Defendant Member Addresses on MADD 9 Number of Defendant Employers
10 Distance between the Defendant and Plaintiff
These are just a few examples of predictive variables used in
Pennsylvania. Variables used in each State are different.
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Predictive Modeling Scores Probability of NCP paying 80 percent on current support in the three months after support order issued/modified.
0 – 30% 31 – 50%
51 – 79% 80% +
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Confidence in the Score
Payment Score
Average Percent of Current Support
1 51% 2 66% 3 77% 4 87%
To validate the accuracy of the payment score, a study of 5,000 PACSES cases was completed
For each payment score, the average FYTD percent of current support collected was calculated.
A direct relationship between payment score and percent of current paid was found (i.e., the
higher the payment score the higher the percent of current paid.
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Planned enhancements to the existing functionality of the Payment Score Calculator • Development of a mechanism for calculating a Payment
Score for all open cases • Quarterly refresh the Payment Score Calculations on all
open cases • Allow worker to complete a guideline calculation if the
PSC has been updated within the last three months • Modification of PIM to add several pre-defined filters
which use the Payment Score as one of the factors for consideration
• Implementation planned for June, 2015
PSC Enhancements
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
After PIM was implemented, Pennsylvania became the first State to achieve above 80% in both percent of current and percent of arrears
PIM/Predictive Analytics Effect in Pennsylvania
PA has been ranked #1 in the United States for both percent of current and percent of arrears, both of which have increased significantly since
the implementation of the Performance Improvement Module
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Revenue Impact: Exceeding 80% of collections targets
Accuracy: “Likely payers” 4x more likely to be above 50% paid than “very unlikely payers”
Timeliness: Allows taking action at time of support order, before payments are missed
Actionability: Caseworker gets clear guidance on recommended steps
Policy Insights: Ability to better segment NCPs to better understand policy impacts
Workforce effectiveness:
• Shift from reactive enforcement to pro-active education
• Ability to prioritize workload based on high impact
Efficiency: Increased effectiveness of retention outreach campaign, audits
Time to production: Scored 17,000 cases within first three months
Transparency: Unified view across all regions and business processes
Marketing value: Helps build positive relationships with our clients. Everyone wins when
payments are made
Predictive Analytics Impact in Child Support
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Edward V. Lehmann, Jr. Director, Case Processing and Data Management Philadelphia Family Court Domestic Relations Section [email protected]
Feel free to contact us.
Questions? Need More Information?
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30
Hershey Lodge ▪ Hershey, Pennsylvania
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
WHERE DO WE START? Starting with the Why
Simon Sinek’s Golden Circle
2
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
WHY
3
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
TAILORING SERVICES USING
PREDICTIVE ANALYTICS
4
Apply a Predictive Analytics lens to Case Management
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
PREDICTIVE ANALYTICS – CLASSIFICATION MODEL
Forecast (predict) an outcome using multiple predictors
TARGET: Predict which cases we can expect a payment of at
least 70% of Current Support Amount in subsequent month
Built a Classification Model
• started with 100+ programmatic, demographic, and economic
variables
• narrowed down to 20 statistically significant variables
5
Model accurately
predicts target
variable for
90% of cases.
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
CASE SCORING
6
0%
10%
20%
30%
40%
50%
0 1 2 3 4 5 6 7 8 9
Payment Status
(Prior Month)
Prediction
(Next Month)
Confidence Score Score
(Propensity
Score)
1 Non-Payer Non-Payer .91 1
2 Non-Payer Non-Payer .73 3
3 Payer Payer .91 9
4 Payer Payer .73 7
5 Payer Non-Payer .55 5
6 Non-Payer Payer .55 5
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
PREDICTIVE ANALYTICS - SEGMENTATION
Built a Segmentation Model
• Started with same variables as in classification model
• Clustering Analysis identified natural groupings based on payment history
Group cases using one or more input fields
7
TARGET: Create cluster of cases based on case variables to
identify natural grouping of cases
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
EARLY SUCCESSES
Arrears Only Project - January 2014
9,486 cases with an Arrears balance, no minor children, no current
support, and no payment within last federal fiscal year
Strategy – Focused Caseload, Locate Experts
Results – $2 million in 12 months!
8
47.96%
55.99%
Department Arrears Only
FPM 4
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
PREDICTIVE ANALYTICS FINDINGS
Arrears Only Project Success
• Personalized the client’s experience
• Case Worker a subject matter expert for
case type 9
• Predictive Analytic Model identifies cases that
perform similarly
• PA Model predicts, with a high level of confidence,
how a case will perform
• PA Model identifies characteristics and motivations
of different customer types
One-Size Fits One
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
HOW
CREATE NEW BUSINESS MODEL Case Ownership Case Segmentation
EST
Zero Order
Non-Paying
Occasional
Paying
Arrears Only
Other
Estab.
Zero Order
Paying
Occasional Non-
Paying
Arrears Only
IGR
10
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
SEGMENTATION MODEL
Intake
Establishment
Arrears Only IGR
Customer Response
11
ENFORCEMENT
Div. 1 Div. 3 Div. 4 Div. 5 Div. 6
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
ENFORCEMENT CASELOAD
Definition –
Case has an order for $0
OR Case has a reserved or
pending supplemental
order OR Case has a Medical Only
order OR Case does not have an
active order, but case
has an arrears balance
and child is a minor
Occasional Paying
Zero Order Non-Paying
12
ZERO ORDER
QUADRANT
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
ENFORCEMENT CASELOAD
Definition –
Case has an order AND
Current Support Amount
is a dollar amount greater
than $0 AND
NCP pays regularly AND
NCP pays over 70% of
CS amount
Occasional Paying
Zero Order Non-Paying
13
PAYING QUADRANT
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
ENFORCEMENT CASELOAD
Definition –
Case has an order
AND
Current Support
Amount is a dollar
amount greater than
$0 AND
NCP pays
irregularly OR
NCP pays less than
70% of CS amount
Occasional Paying
Zero Order Non-Paying
14
OCCASIONAL PAY
QUADRANT
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
ENFORCEMENT CASELOAD
Definition –
Case has an order
AND
Current Support
Amount is a dollar
amount greater than
$0 AND
NCP has not made a
payment during the
current Federal Fiscal
Year
Occasional Paying
Zero Order Non-Paying
15
NON-PAY QUADRANT
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
Occasional
Send letters of Introduction to new case manager
UIB/DIB/ Mod review (CMT Sort)
Review for SLMS suspension – Manual submission
for NCP’s with valid license
Incarcerated Special Mod/closing Project
CMT High TMSO paying less than 90% sort
Review EFO case function cases for order for
subsequent child
Work LC005 task for IWOs
Review CMT for SSI benefits
Paying
Send letters of Introduction to new case manager
Low Order with reported Earnings Mod Project
Admin IWO Project
Incarcerated Special Mod Project
CMT High TMSO paying less than 90% sort
UIB/DIB Mod review (CMT Sort)
Review EFO case function cases for order for
subsequent child
New Order clerical contact of NCP (EI)
Work LC005 task for IWOs
Zero Order
Send letters of Introduction to new case manager
Zero Mod Project
Review EFO case function cases for order for
subsequent child
Review Paid in Full cases to confirm emancipated
DP’s updated for auto closures.
Non-Paying
Send letters of Introduction to new case
manager
Review for SLMS suspension – Manual
submission for NCP’s with valid license
Incarcerated Special Mod Project
Review EFO case function cases for order
for subsequent child
Work LC005 task for IWOs
Review CMT for SSI benefits
STRATEGIES BY QUADRANT
16
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
NEW CSTATS
17
Old CSTATS:
• Performance across the
division
• Compares branch-to-branch
New CSTATS:
• Performance across
the division
• And performance by
quadrants
• Compares branches
by quad
• Month, YTD
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
IMPLEMENTATION
18
Phased Roll-Out
Roadshows
Feedback/Workgroups
New Performance Measures
Reset every FFY
ERICSA 52nd Annual Training Conference & Exposition ▪ April 26 – 30 ▪ Hershey Lodge ▪ Hershey, Pennsylvania
QUESTIONS
19