Modeling and Tuning

Loaded: modeling-and-tuning.md — build, tune, compare, and choose a final model.

Covers spine sections 8–12. The metrics helper comes from the resolved fork file; define it once before the first model.

Algorithm scope — ask once, lock it in

Ask at the start of this section which algorithms are in scope, then don't re-ask per model. If there's no stated preference, offer a default and say why:

• Classification: Logistic Regression as an interpretable baseline, then Decision Tree, then Random Forest.
• Regression: Linear Regression as baseline, then Decision Tree, then Random Forest.

A baseline that's deliberately simple is worth its cell — it establishes what the complex models have to beat, and occasionally it wins, which is itself worth knowing.

If the task comes from a course or has a stated syllabus, scope follows that, and adding an algorithm outside it should be flagged rather than slipped in.

Per-model loop

For each algorithm, in order:

1. A short "what is X / how does it work" explainer, pitched at the person's apparent familiarity. Two or three sentences, not a lecture.
2. Instantiate with defaults, fixed random_state. Fit. Batchable — a bare .fit() produces nothing to interpret.
3. Evaluate on train and test through the shared helper.
4. Observations naming over/underfitting from the gap, not from the two numbers side by side. An unconstrained tree hitting near-100% on train and much lower on test is memorising; say that.
5. For classification, the confusion matrix here, interpreted in counts and business terms.

Baseline comparison before tuning

Concatenate every untuned model's test results into one table before touching hyperparameters. It'll be superseded, and it's still worth doing — it shows whether tuning is fixing a real problem or polishing something already adequate, and it's the only place the relative strengths of the raw algorithms are visible.

Compute budget rule — size before running

Before proposing any GridSearchCV or RandomizedSearchCV, compute and state out loud:

n_combinations × cv_folds × (n_estimators, if an ensemble) = total fits

alongside the training row count. For scale: the LOS reference notebook's grid was 3×3×2 = 18 combinations × 5 folds = 90 Random Forest fits on 400,000 rows, which will likely exhaust free-tier Colab. A comparable grid on 3,000 rows finishes in a couple of minutes.

If the estimate looks likely to exceed a few minutes, offer three options and let the person choose: RandomizedSearchCV with n_iter, a reduced grid, or a subsample of training rows. Never propose a fit that hasn't been sized.

Set n_jobs=-1 to use available cores.

Tuning

Score on the metric agreed in the fork file, not the default. GridSearchCV(scoring="recall") and GridSearchCV() optimise different things, and the difference shows up in the selected hyperparameters.

Skip tuning a deliberately simple baseline (plain Logistic or Linear Regression) unless regularisation strength is genuinely in question — say that's why rather than leaving it looking like an omission.

After tuning: re-evaluate on train and test, and in the Observations name (a) the change per metric against that model's own baseline, and (b) whether the train/test gap shrank. The second is the point of tuning; the first is the visible result.

Feature importance

Per tuned model, where the algorithm supports it: tree-based feature_importances_, or coefficients for linear models — noting for coefficients that magnitude is only comparable across features if they're on comparable scales.

Observations should:

• Name the top features with their actual values.
• Name what the importance is spread across. A single dominant feature and a smooth gradual decline mean different things about model behaviour.
• Reconcile against EDA explicitly. A variable with a large raw bivariate gap that lands near zero importance is a genuine finding, not a discrepancy to smooth over — it usually means the raw gap rested on a small sample, or that another feature carries the same signal more reliably. Say which, if it's determinable. The reverse — low bivariate signal, high importance — usually means an interaction the univariate view couldn't show.
• Note where one-hot encoding split a variable across several dummies, since importance is then divided among them and each individual dummy understates the parent variable.

Choosing the final model

One comparison table, all models, tuned results where tuning happened.

Recommend one, naming the tradeoff rather than just the top row. Two rules that stop this from being automatic:

The 1% rule. If the tuned model's gain over its own baseline is under roughly 1%, say so explicitly and ask whether it justifies the compute, rather than defaulting to "tuned wins." The LOS reference notebook took a 0.5% RMSE gain and declared the tuned model final — defensible, but a choice, not an automatic consequence.

Metric conflict. When one model wins on the metric chosen for tuning and another wins overall, say so and recommend on the stated business priority rather than quietly switching to whichever metric supports the preferred answer. A 0.2-point recall difference against a 4-point precision difference is not a tie, and the reasoning should be visible.