I got confused about AUCs for the multiple-predictors-case (i.e., when there is a model with many input variables and one output variable). Because then there are several (continuous) variables, instead of one.

**ROC example**

The true positive rate (or sensitivity) is regressed on (or plotted against) the false positive rate (or 1-specificity). We want the former to be as close to 1 as possible, and the latter to be as close to 0 as possible. Therefore, we want the ROC curve to be as far to the upper left corner of the graph, as possible.

To derive the sensitivity and specificity, the values of a single continuous variables are varied. In the bivariate case, this is just the originial continuous variable. In the multivariate case, this is the prediction of a multivariate model. For example, in logistic regression, the model derived (predicted) probabilities of belonging to the positive class. The R code for an example showing the calculation of the AUC for the bivariate and multivariate case is provided below.

**Interpretation of AUC**

The AUC has straightfoward interpretation: "When using normalized units, the area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative')." (source: Wikipedia)

**Problems with AUC**

However, there are some problems with the AUC curve. Hand (2009) noted that the the AUC uses different misclassification cost distributions for different classifiers. This is problematic, because the relative severity of misclassifications should not depend on the classifier, but on the classification problem. Furthermore, Smits (2010) has noted that using Youden's index (sum of sensitivity and specificity minus one) to select the cut-off value for classification is problematic, as it makes the relative cost of a false negatives and false positives dependent on the prevalence.

**R code example**

# generate predictor variables: self-esteem (questionnaire), number of adverse life events and presence of a depressed parent:

set.seed(25)

self.esteem <- round(rnorm(250, 20, 5))

tmp <- round(rnorm(1000, 0, 5))

adverse.events <- sample(tmp[tmp>=0], 250)

depressed.parent <- round(runif(250, 0, 1))

# generate error:

error <- round(rnorm(250,0,3))

# continuous variable 'depression' is a function of above variables

depression <- -5 + 1.8*self.esteem + 1.5*adverse.events + 2.3*depressed.parent + 1.0*error

# dichotomize continuous depression variable (i.e., depression diagnosis)

diagnosis <- depression>=37

library(pROC)

**# bivariate ROC analysis: relationship between self-esteem and depression diagnosis**

plot(roc(diagnosis~self.esteem))

coords(roc(diagnosis~self.esteem), "best")

# in this example, self-esteem can be used for predicting depression diagnosis, as the AUC > .5. A cut-off value of 18.5 provides best sensitivity and specificity

**# illustration of bivariate ROC analysis: dichotomized variable has AUC of 1.0 if predicted with originial continuous variable:**

plot(roc(diagnosis~depression))

coords(roc(diagnosis~depression), "best")

# when a continuous variable can discriminate perfectly between two non-overlapping classes, the AUC is 1.0. As expected, the cut-off value providing best sensitivity and specificity is the one originally used for dichotomizing: 37.

**# multivariate relationship between self-esteem, adverse life events and presence of a depressed parent**

my.glm <- glm(diagnosis ~ self.esteem + adverse.events + depressed.parent, family="binomial")

predicted.probs <- predict(my.glm, type="response")

plot(roc(diagnosis~predicted.probs))

coords(roc(diagnosis~predicted.probs), "best")

# using self-esteem, no. of adverse events and presence of a depressed parent as predictors, improves predictive accuracy, compared to the bivariate model in which only self-esteem was used for prediction. The cut-off value for the model-derived probability of being in the positive class, providing the best sensitivity and specificity, is 0.4004744.

**References**

Hand, D. J., & Till, R. J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems.

*Machine Learning*,

*45*(2), 171-186.

Smits, N. (2010). A note on Youden's J and its cost ratio.

*BMC medical research methodology*,

*10*(1), 89.

## No comments:

## Post a Comment