Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Acquisition scorecard:Reject inference
methodologies
An EXL whitepaper
Written by
Nupur VermaDecision [email protected]
1 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
Reject inference methods address this
issue by using actual performance of
approved applications and derived
performance of rejected applications. This
makes the model more accurate, robust
and unbiased. If applied correctly, reject
inference methods help in increasing the
approval rate, without compromising the
credit quality. This paper outlines steps
involved in some reject inference methods
which can be used in the development
of an acquisition scorecard. The methods
have been empirically tested and the best
suitable method for different scenarios has
been suggested.
IntroductionIn any statistical model, the key assumption
made is that the characteristics of the
development sample are similar to the
population on which the model will
be applied. This is true for collection
models because they are built on
approved applications and applied on the
same type of accounts - the approved
applications. Acquisition models are
built on approved applications, but have
to be used for the entire through-the-
door population – approved as well as
rejected. Here, the development sample
is biased because it is not representative
of the entire population. Reject Inference
methodologies are a way to account for
and correct this sample bias.
One of the basic problems a bank faces is that an application scorecard has to be applied on the through-the-door population, but the models are built only on the approved applications because the performance of only the approved applications is available with the lender.
Performance of rejected applications is not readily available, and hence the development of a model based on both rejected and approved applications is a challenge.
2 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
Canadian Imperial Bank of Commerce,
Toronto, Canada, discusses manual
estimation of method of reject inference,
when the modeler uses preexisting
knowledge to manually simulate
performance on the declined population.
The paper also discusses the augmentation
and extrapolation techniques. A paper
has also been written by Dennis Ash
and Steve Meester, May 2002, which
discusses Group Parceling, Heckman’s
bias correction and Supplemental Bureau
Data methods of reject inference. John
Banasik and Jonathan Crook in their paper
“Reject inference, augmentation, and
sample selection”, June 2006, discusses
augmentation and banded data techniques
of reject inference.
MethodologyVarious methods of RI are available to
include the performance of rejected
applications and make the model
applicable on the entire through-the-door
(TTD) population. There are two broad
categories of RI:
i. When the performance of rejected
applications is available – in-house or with
some other lender through credit bureau
ii. When the performance of rejected
applications is not available to the lender
When the performance of the rejected
applications is not available with a
competitor or in-house their performances
Reject Inference (RI) becomes a very
important tool when the creditor wants to
relax the approval criteria to improve the
approval rate. By inferring the performance
of the rejected applications, the impact
of such a decision can be studied. If the
approval rate is low then RI serves as a
very important tool. If the approval rate is
already very high and there is limited scope
of increasing the approval rate, RI may not
be very helpful.
Literature ReviewThere have been a number of studies to
solve for the sample bias in an application
scorecard using reject inference methods.
“Reject Inference Methodologies in Credit
Risk Modeling” by Derek Montrichard,
3 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
have to be mathematically derived, or
some assumptions have to be made on
their performance. This paper focuses on 4
methods of RI when bureau performance
of rejected applications is not available.
These are:
• Hard cutoff method
• Single weighted approach
• Double weighted approach:
• Augmentation
For all the 4 methods above, logistic
regression models were built on real data
for a personal loan portfolio of a MNC bank.
The data was carefully sanitized to ensure
security of confidential information. The
approval rate for this portfolio is 40%, with
a sample size of 90,000. The modeling
dataset had only application data and no
bureau data for accepts or rejects. Logistic
regression model results were studied.
These methods are discussed below.
1. Hard cutoff method
This is one of the simplest reject inference
methods. All rejects above a score cutoff
are tagged as “bads” and a model is then
built on actual performance of accepts and
inferred performance of rejects.
i. A logistic regression model on approved
population is built using their internal
performance to predict the probability
of default. Rejects are scored using this
model.
ii. Approved applications are ordered and
divided in 10 equal bands. An expected
bad rate level is set above which rejected
applications are tagged “bad”; all rejected
applications below this level are classified
as “good.” This cutoff score can be
determined through an iterative process.
iii. Inferred goods and bads for the rejected
applications are then added to the actual
good and actual bad of the approved
applications. A final logistic regression
model is built on this dataset to predict the
probability of default. This model can be
used in the new underwriting strategy.
4 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
In the figure 1 (right), approved applications
are distributed in 10 scorebands of equal
size. From scoreband 1 to scoreband 9 the
actual bad rate is increasing at a steady
pace. The bad rate jumps from 5.5% to
11% in the 10th bin. Here, 1.99% has been
taken as a cut-off. Rejected applications
with scores less than 1.99% are classified as
good and those with score more than 1.99%
are classified as bad. Actual performace
for the approved applications is already
available, hence there is no need to apply
this cutoff rule for approved applications.
Note that as high as 47% of the rejected
applications would be classified as bad
in this example. This is intuitive because
rejected applications would have a higher
bad rate than approved applications,
otherwise they would not have been
rejected.
In this method, the final modeling dataset
was heavily concentrated with the bads
of the rejected applications. The method
performed well in terms of Gini, KS and
concordance.
Approved %Rejected
Band Min Score Max Score Actual Bad Rate %Approved
1 0.00% 0.06% 0.2% 10% 0%
2 0.06% 0.12% 0.3% 10% 1%
3 0.12% 0.18% 0.4% 10% 2%
4 0.18% 0.25% 1.0% 10% 2%
5 0.25% 0.35% 1.9% 10% 2%
6 0.35% 0.50% 2.8% 10% 12%
7 0.50% 0.77% 3.7% 10% 9%
8 0.77% 1.21% 4.6% 10% 13%
9 1.21% 1.99% 5.5% 10% 13%
10 1.99% 40.00% 11.0% 10% 47%
Figure 1
5 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
2. Single weighted approach
This is a two-step modeling approach.
Here, instead of classifying a reject into
good or bad, each reject is classified as a
partial good and partial bad, with a weight
assigned to it. The score of the known good
bad model is used as a weight for rejects in
the final model. The steps in this approach
are outlined below:
i. A logistic regression model is built on
approved applications to estimate the
probability of default – P(default).
ii. All rejected applications are scored on
this model to get P(default).
iii. A weight is assigned for approved
and rejected applications. Weight for
approved application=1. Weight for each
reject is applied such that there are 2
observations for each rejected application:
Wt_1=P(default), Wt_2=1-P(default).
Figure 2 shows a sample of the modeling
dataset.
iv. A final logistic regression model is built
with approved appearing once in the
modeling dataset and rejects appearing
twice in the modeling dataset.
In this method there is no ambiguity in
assigning classes to the rejects as in the
hard cutoff method. Each reject is once
counted as a bad with a weight assigned
to it wt_1. The reject enters the modeling
dataset again as a good, with a weight
assigned to it wt_2. This method assumes
that all rejected applications have same
probability of getting approved, which
might not be the case.
Application_id Decision PD Weight Bad_Var Weight_description
1 Approve 0.01 1.00 1 =1
2 Approve 0.02 1.00 0 =1
3 Reject 0.20 0.20 1 =PD
3 Reject 0.20 0.80 0 =1-PD
4 Reject 0.30 0.30 1 =PD
4 Reject 0.30 0.70 0 =1-PD
5 Reject 0.12 0.12 1 =PD
5 Reject 0.12 0.88 0 =1-PD
Figure 2
6 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
3. Double weighted approach:
This method is similar to simple weighted
approach. This, however, is a 3 step
modeling approach. The approach for this
technique is as follows:
i. A logistic regression model is first built
on approved applications to estimate
the probability of default – P(default). All
rejected applications are scored on this
model to get P(default).
ii. An approve/reject model is built on
approved as well as rejected applications.
This will give P(approved).
iii. Weight is assigned to all applications.
For approved applications weight=1. Each
rejected application appears twice in the
modeling dataset: once as a bad with a
weight wt_1 and then as a good with a
weight wt_2.
Weights for reject are calculated as below:
• Wt_1=P(default)*P(Approved)
• Wt_2=1-P(default )*P(Approved)
A sample of the modeling dataset is shown
below:
iv. A final model is built on approved and
rejected applications. Actual performance
of approved application is used. Inferred
performance of rejected applications is
used.
Application_id Decision PD P_Approved Weight Bad_Var Weight_description
1 Approve 0.01 0.80 1.00 1 =1
2 Approve 0.02 0.70 1.00 0 =1
3 Reject 0.20 0.01 0.00 1 =P_Approved*PD
3 Reject 0.20 0.01 0.01 0 =P_Approved*(1-PD)
4 Reject 0.30 002 0.01 1 =P_Approved*PD
4 Reject 0.30 0.02 0.01 0 =P_Approved*(1-PD)
5 Reject 0.12 0.04 0.00 1 =P_Approved*PD
5 Reject 0.12 0.04 0.04 0 =P_Approved*(1-PD)
7 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
This method incorporates not just the
probability of a default for a reject, but also
the probability of getting approved. The
method is more complicated than other
methods because a set of three models
have to be built, which is a cumbersome
exercise.
4. Augmentation
In this method, rejects are not tagged
as goods or bads. Instead, approved
applications are reweighted or ‘weighted
up’ to represent the rejected applications
with similar chances of getting approved.
Methodology for this is mentioned below:
i. An approve/reject model is built to
predict the probability of an application
getting approved
ii. The scored dataset from the above step
is divided into 10 equal bands. Approved
applications of each band are assigned
a weight which is inversely proportional
to the average approval score of that
band. Instead of assigning weights on the
basis of bins, each approved applications
can also be assigned a different weight=
1/p(approval)
iii. Final KGB model is built on this
augmented dataset with weight assigned
to each accepted application.
Scorebands with low p(approval) are
assigned a higher weight and accepts in
these score bands have characteristics
similar to the rejected population. The
augmented dataset is expected to
represent the behavior of both approved
and declined applications. In this method,
performance of the model on the rejects
cannot be gauged and hence makes
it difficult to be compared with other
methods of RI.
8 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
Results:
A preferred method of RI should was
expected to satisfy the following criteria:
• Bad rate should be higher for rejects than
accepts
• Method should be able to increase the
approval rate without compromising on
the credit quality of the portfolio
• Method with higher gini, KS and AUC
would be considered to be more
predictive than others
• Method with higher accuracy would be
preferred
Model discriminatory powerThe performance of the 4 RI methods
in terms of separation power between
the goods and bads is shown in the table
below:
Method Population Actual %Bad Predicted %Bad Gini KS AUC
Hard cut-off
Overall 40% 40% 85% 73% 92.5%
Accepts 2% 6% 50% 38% 75.0%
Rejects 69% 65% 89% 81% 94.5%
Single
Weighted
Overall 4% 4% 55% 41% 77.5%
Accepts 2% 2% 59% 47% 79.5%
Rejects 5% 5% 44% 32% 72.0%
Double
Weighted
Overall 3% 3% 60% 47% 80.0%
Accepts 2% 2% 61% 47% 80.5%
Rejects 4% 4% 46% 34% 73.0%
Augmentation
Overall - * 4% - * - * - *
Accepts 2% 2% 63% 50% 81.5%
Rejects - * 6% - * - * - *
*Inferred performance of rejects is not available for the augmentation technique.
At overall level (accepts + rejects) the
hard cutoff approach gives a very good
separation power in terms of gini, KS and
AUC.
The separation is good for the rejects, but
not so good for the accepted applications.
9 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
Accuracy:
Table 2 shows how accurately bads and
goods are being classified in the 4 methods
at score cutoff of 2%, 3%, 4%, 5% and 6%.
Table 2 shows accuracy for accepts and
rejects taken together, while table 3 shows
the accuracy for accepts alone.
From table 2 we can see that the accuracy
at overall level is highest for double
weighted approach at all score cutoffs.
Accuracy for accepts alone is lowest
for the hard cutoff approach at all score
cutoffs. Accuracy for Accepts and Rejects
Score-cutoff Hard Cutoff Single Weighted Double Weighted Augmentation
600 74% 76% 88% -
500 71% 64% 83% -
400 70% 60% 80% -
300 69% 55% 76% -
200 68% 49% 73% -
Accuracy for Accepts
Score-cutoff Hard Cutoff Single Weighted Double Weighted Augmentation
600 70% 91% 90% 91%
500 68% 88% 87% 88%
400 65% 86% 85% 85%
300 64% 79% 81% 80%
200 56% 75% 79% 75%
Table 2
Table 3
10 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
Approval Rate
Table 4 shows the approval rate for all the
four methods of RI at various score cutoffs.
At any given level of score cutoff, double
weighted approach gives highest approval
rate, and hard cutoff technique gives
lowest approval rate. This implies that by
keeping the bad rate constant, the bank will
be able to approve more applications than
other methods.
In summary, hard cutoff approach performs
well in discriminating goods from bad, but
does not give better accuracy and higher
approval rate than the other methods.
The augmentation technique has highest
separation power after hard cutoff
approach and also gives high accuracy,
Approval Rate
Score-cutoff Hard Cutoff Single Weighted Double Weighted Augmentation
600 36% 76% 84% 45%
500 33% 65% 72% 42%
400 31% 60% 66% 40%
300 30% 53% 60% 37%
200 28% 48% 56% 34%
Table 4
but the approval rate is very low. Double
weighted approach performs better than
other methods in terms of accuracy and
approval rate, but its discriminatory power
is lower than the cutoff and augmentation
techniques.
11 © 2016 ExlService Holdings, Inc.
[ Acquisition scorecard : Reject inference methodologies ]
ConclusionAs discussed earlier, two key objectives
of RI are to correct for sample bias and
identify rejects with good performance
which should have otherwise been
approved. All the four methods of RI
correct for sample bias by including
the performance of rejects in the final
modeling population. Double weighted
and single weighted approaches are
able to increase the approval rate. If the
business objective is to increase the
approval rate without compromising the
credit quality then the double weighted
approach should be used. This method
will significantly increase the approval rate,
and will accurately predict both rejects and
accepts. If the approval rate is already very
high and the objective is to identify bads
References • Derek Montrichard, Reject Inference
Methodologies in Credit Risk Modeling
• Naeem Siddiqi, Credit Risk Scorecards,
Chapter 6
• Reject Inference, augmentation and
sample selection, John Banasik,
Johnathan Crook, 25th Jan 2007
• Reject Inference Techniques
Implemented in Credit Scoring for SAS®
Enterprise Miner™, Billie Anderson, Susan
Haller, and Naeem Siddiqi, SAS Institute,
Cary, NC
AcknowledgementI would like to thank Alok Rustagi and
Varun Aggrawal for their valuable inputs
which greatly helped in making this paper
more informative.
among the through-the-door population,
then hard cutoff method should be used.
GLOBAL HEADQUARTERS280 Park Avenue, 38th Floor, New York, NY 10017
T: +1.212.277.7100 • F: +1.212.277.7111
United States • United Kingdom • Czech Republic • Romania • Bulgaria • India • Philippines • Colombia • South Africa
Email us: [email protected] On the web: EXLservice.com
EXL (NASDAQ: EXLS) is a leading operations management and analytics company that helps
businesses enhance growth and profitability in the face of relentless competition and continuous
disruption. Using our proprietary, award-winning Business EXLerator Framework®, which
integrates analytics, automation, benchmarking, BPO, consulting, industry best practices and
technology platforms, we look deeper to help companies improve global operations, enhance
data-driven insights, increase customer satisfaction, and manage risk and compliance. EXL
serves the insurance, healthcare, banking and financial services, utilities, travel, transportation
and logistics industries. Headquartered in New York, EXL has more than 24,000 professionals in
locations throughout the United States, Europe, Asia, Latin America, Australia and South Africa.
© 2016 ExlService Holdings, Inc. All Rights Reserved.
For more information, see www.exlservice.com/legal-disclaimer