Comments on Validation Methods for Two Group Classification Models Widely Accepted in Credit Scoring or Response Analysis

Abstract

The two group classification methods are popular approaches for the separation of one group from the other. For these purposes either parametric or non-parametric classification approaches are used. In many cases a scoring algorithm is derived and the score distribution serves as a basis of the decision making. Generally, validation of a model is to assure the model has reasonable separation power when it is applied to a different data set not used for the development of the model, i.e., holdout data set. In the credit scoring case, Regulation B of Equal Credit Opportunity Act requires the scoring algorithm be revalidated frequently enough to ensure that it continues to meet statistical standards. In addition, in case of comparison of more than one model, it is necessary to quantify model performance in some way. Two sample Kolmogorov Smirnov test statistic, Kullback-Leibler Number, and Mahalanobis Distance, etc. are popular ways of quantifying model performance. In this study, such popular methods are discussed along with the advantages and disadvantages of each method using a simulated data set and a suggestion of an improved, intuitive, and simple quantifying method for model performance is made