First classifiers and how to score them

Sign in<br>Subscribe

In Post 10 we learned how to split data into training, validation, and test sets, and in Posts 3 and 4 we built our first regression models and baselines. Now we take the next step: training our first real classifiers on the UCI Adult Census Income dataset, where we predict whether someone earns more than 50K a year. By the end of this post we will have a pruned decision tree that reaches an area under the ROC curve (AUC) of 0.903 on the test set, and we will understand why accuracy alone would have misled us at every turn. The through-line of this post is the difference between ranking and deciding: classifiers can order people by income likelihood, but turning that order into a yes or no prediction requires a threshold, and every threshold tells a different story.<br>We build three families of classifiers on the Adult data, score them with metrics that expose different failure modes, and then calibrate their probabilities so the numbers we output mean what they claim. The dataset gives us about 48,842 rows and 14 columns, a clean binary problem with numeric and categorical features, missing values, and a class imbalance we cannot ignore.<br>The data<br>Before any model, we inspect the raw data. The base rate of high earners, those making over 50K, sits at 24 percent, which means a model that predicts everyone earns less would be right 76 percent of the time. That number becomes our floor: any classifier must beat it to earn its place. Missing values are confined to three categorical columns, workclass, occupation, and native_country, and we impute them with the mode in the pipeline. We drop fnlwgt, which is a sampling weight rather than a predictive feature, and we drop education because education_num carries the same information as a number.<br>The class imbalance is visible in Figure 1.<br>The numeric features show the expected skews. capital_gain and capital_loss are zero-inflated, with most people having no capital activity at all, and hours_per_week has over 13,000 values beyond 1.5 times the interquartile range from the quartiles. We leave them unclipped; the tree-based and distance-based models we are about to train can handle the shape. The categorical columns have long tails, so we one-hot encode with handle_unknown='ignore', which is enough at this data size.<br>The histograms in Figure 2 make those skews concrete.<br>The preprocessing pipeline combines a median imputer and standard scaler for the five numeric features, and a mode imputer with one-hot encoding for the seven categorical ones. After fitting on the training split, we end up with 88 encoded features, and we hold out 12,198 rows for the final test, 10,978 for validation, and 25,614 for training.<br>Starter models<br>With the encoding done, the first model we try is k-Nearest Neighbors, the lazy classifier: it stores the training rows and assigns a class by majority vote among the nearest neighbors. Distance metrics define what nearby means, so we try both Euclidean and Manhattan distance on a 10,000-row subset stratified to preserve the 76/24 class split, because a full 25,614 by 88 distance search is too slow for a laptop demo.<br># kNN votes by majority among the k closest training rows<br>knn_euclidean = KNeighborsClassifier(n_neighbors=5, metric='euclidean').fit(X_knn_enc, y_knn)<br>knn_manhattan = KNeighborsClassifier(n_neighbors=5, metric='manhattan').fit(X_knn_enc, y_knn)<br>Both beat the baseline. Euclidean reaches 83.58 percent validation accuracy and Manhattan 83.38 percent, a small gap that tells us the geometry of the feature space is not particularly sensitive to the distance definition. The perceptron, our first linear classifier, does worse at 70.87 percent, below the 76 percent baseline; class_weight='balanced' keeps it from predicting the majority class for everyone, but that does not beat the constant predictor.<br>Polynomial regression serves as a reminder that linear models can be made flexible with polynomial features, but its raw scores are not probabilities. On validation, the degree-2 polynomial pipeline produces scores ranging from negative 0.443 to 1.881, and thresholding at 0.5 gives 81.75 percent accuracy. The sigmoid link function maps a real score to the interval between zero and one, which is why we will need it later for calibration.<br>Figure 3 shows the shape of that mapping.<br>The perceptron is fast and linear, but it only returns decision scores, not probabilities. That limitation will matter when we compare models with probability-based metrics, and it sets up the calibration section at the end.<br>Decision trees<br>The perceptron showed that linear scores are not enough; a decision tree loosens that assumption by splitting rows with yes or no rules. The CART algorithm grows one binary split at a time by minimizing impurity. Gini impurity, the chance a random row in a node is mislabeled if labeled by that node's class proportions, is the default measure; information gain is the...