Overview and EDA

Loaded: overview-and-eda.md — Data Overview, univariate, and bivariate analysis.

Covers spine sections 3 and 4. The plotting functions referenced here live in assets/plot_helpers.py — paste them verbatim rather than retyping from memory.

Data Overview

Run in this order. Each numbered item is one step under the core loop unless marked batchable.

1. .head() — confirms the data loaded and matches the dictionary.
2. .shape — one-line markdown stating the counts.
3. .info() — Observations on dtypes and any non-null mismatch. Missing values are named here, treated later (post-split, in prep-and-leakage.md). Also name any column whose dtype is defensible but whose semantics need a later decision — an ordinal categorical read as object, a numeric code that's really a category.
4. Identifier columns — .nunique() and the group-leakage check below. Then drop. Batchable with step 5.
5. .duplicated().sum().
6. .describe().T — Observations naming outlier suspicion by comparing max against the 75th percentile, and flagging any suspicious minimum (a 0 where 0 is implausible). Named, not treated.
7. Loop over categoricals printing .value_counts(normalize=True) — Observations on balance and dominance. Name any category under about 5% of rows explicitly; conclusions drawn about it later will rest on a thin sample, and that caveat should be established here so it can be referenced rather than re-argued.

Group leakage check — run before dropping the ID

If an ID column has fewer unique values than rows, the same entity appears more than once. A random split then puts the same entity in both train and test, inflating every metric. In the LOS reference notebook this went unflagged: 126,399 unique patientid across 500,000 rows, split randomly, R² = 0.973.

Check before dropping — once the column is gone the signal is gone. If found, flag it and offer GroupShuffleSplit or GroupKFold on the ID as the alternative split strategy, then carry that decision into prep-and-leakage.md.

Exploratory Data Analysis

Univariate

Define histogram_boxplot() once (from assets/plot_helpers.py), then one call per numeric variable — each its own step, each with its own Observations covering:

• Range, and whether it's plausible for the variable's meaning.
• Skew direction, read from the mean/median relationship, not asserted.
• Outliers: present or absent, and roughly where they start. "No outliers" is a real finding — the boxplot whiskers reaching min and max is worth stating.
• Modality — a second peak is a finding worth naming, since it often signals two behaviours mixed in one column.

Define labeled_barplot() once, then one call per categorical variable. Observations name the dominant category with its percentage, and flag thin categories again where relevant.

Include the target in this pass. For classification, its class balance determines whether accuracy will mislead later; state the no-information baseline explicitly ("a model predicting the majority class every time would be right X% of the time"), because it's the number every later accuracy figure has to beat to mean anything.

Bivariate

Order matters — heatmap first, then target relationships.

1. Correlation heatmap for all numeric variables. Observations name the strongest and weakest pairs by actual value, flag multicollinearity risk, and note each predictor's correlation with the target where the target is numeric or binary.
2. Numeric vs. target: boxplot per numeric variable, split by a categorical target; scatter or binned means for a continuous target. Observations state the actual gap in real numbers. "Little to no difference" carries the same weight as a strong finding and should be stated with the same confidence — a variable that doesn't separate is genuinely informative about what to expect from feature importance later.
3. Categorical vs. target: stacked_barplot() — prints the crosstab counts, plots the normalised crosstab. Observations compute the per-category rate from the printed counts and compare it against the overall base rate, rather than describing bar heights.

If the problem statement poses specific business questions, give each one its own heading — ### Question 1: How does current occupation affect lead status? — followed by the plot that answers it and then its ### Observations: cell. Answer them in the problem statement's own order.

Headings rather than a passing mention, because the question then appears in the report's contents beside the finding that answers it, and a reader can check coverage without reading the section. Where the statement poses no explicit questions, use plain subject headings instead; don't invent questions to fill the pattern.

Close the section with a summary block: strong predictors, weak or null relationships, each with its number. This becomes the backbone of the final insights section and stops those insights from being re-derived (or quietly re-invented) later.

Observations pattern

Every Observations block is its own markdown cell, headed ### Observations: <subject> — never a bold line inside a cell that also carries other prose. Bullets, real values bolded inline, no hedging language that carries no information. Where a plot raises an obvious follow-up, end with the one-line hypothesis the next plot tests.

Never describe a chart you haven't seen. If the person hasn't sent the output back yet, don't write the Observations block speculatively and ask them to confirm it — that inverts the whole loop and invites a plausible-sounding fabrication into the notebook.