Improving Recall in Diabetes Risk Prediction: The Effects of Class Imbalance and Feature Importance in BRFSS Health Data

Undiagnosed diabetes and prediabetes can be harmful because of their associated consequences, but machine learning models can contribute to diagnosing the two illnesses by detecting patterns in measurable health indicators. However, class imbalance could lead to an inability to detect positive examples in the dataset. The current study has evaluated the impact of balancing training data and adjusting classification threshold on recall of prediction of the combined prediabetes/diabetes class on 2015 Behavioral Risk Factor Surveillance System Diabetes Health Indicators Dataset. Accuracy, precision, recall, F2 score, receiver operating characteristic area under the curve, average precision, and confusion matrices have been used for evaluating performance of Logistic Regression and Random Forest classifiers. The Random Forest classifier trained on the balanced dataset and validated with a classification threshold of 0.44 has demonstrated accuracy of 0.674, precision of 0.279, recall of 0.846, F2 score of 0.602, ROC-AUC of 0.822, and average precision of 0.409. The model has correctly predicted 5,983 instances out of 7,069 of the combined class. Permutation importance based on decrease in recall suggested that general health, BMI, age, high blood pressure, and high cholesterol are the most significant predictors. Whereas balance and threshold modification helped improve the level of recall, it also led to an increase in the number of false positives. Such results show that the use of recall as an assessment method helps to identify clinically significant differences that cannot be seen using overall accuracy.