Data Analysis

The section structure, code idioms, and voice here derive from two sources: one Great Learning reference notebook (Hospital LOS Prediction, regression, style reference only — it contains at least two methodological flaws) and one classification project built end to end with a user (ExtraaLearn lead conversion). Both are supervised tabular problems on datasets under 500k rows.

That means: the regression and classification paths are grounded in real completed work. The time-series path is not — it's reasoned from principle, not validated against a finished project. Say so if a time-series task arrives, and treat that fork as a first draft rather than a proven spine.

Do not describe this structure as an industry standard or "the correct way." It's a house style with a small sample behind it.

This skill is deliberately thin. The spine does not contain enough detail to run a section from memory. Before starting any section, read its reference file. Attempting a section from the spine's one-line summary will silently produce weaker work with no error and no warning — that is the main failure mode of this design, and avoiding it is a hard requirement, not a preference.

Each reference file opens with a load-confirmation line. Say it once when the file is read, so it's visible in conversation whether routing actually happened.

1. Name the section and the question this step answers, in plain language ("Let's check whether...").
2. Propose exactly one unit of work.
3. Stop. Wait for the person to run it and report the real output.
4. Write the Observations: interpretation, grounded only in what was actually reported. Never invent a number, percentage, range, or plot shape. If pasted output or a screenshot is ambiguous, ask.
5. Propose the next step.

Mechanical steps may be batched, since they produce nothing to interpret: imports, data loading, .copy(), dropping identifiers, helper-function definitions, the split plus a shape check. Anything producing a number, table, or plot gets its own step and its own Observations.

Where the gate sits when code is executed directly. Step 3 has no equivalent — there is nothing to wait for, because you ran it yourself. The gate does not disappear, it moves: run and interpret step by step, then stop at the section boundary and wait for a go before starting the next section.

The section boundaries are the right place because that is where decisions with downstream consequences get taken — the split strategy, the encoding, the algorithm scope, the metric that drives tuning. Running past one commits the person to a choice they were never shown.

Two things this does not license:

• Batching turns is not batching interpretation. Every output still gets its own Observations block. Four charts in one turn means four Observations, not one paragraph covering all four.
• It is not a default to apply silently. Name the cadence in the first reply — one line, e.g. "Running each step and interpreting as I go, stopping at each section boundary" — so the person can ask for tighter or looser gating. Choosing it without saying so is the same omission the rule below covers, just earlier in the conversation.

The person can switch this off. If they say "just give me the section" or "I'm short on time" — give it, prefixed with one line naming what's being skipped (e.g. "Breaking the loop — full bivariate block below; you won't have seen the heatmap before the bar charts were chosen"). One line, no lecture. Resume gating at the next section unless told otherwise.

If a later output contradicts an earlier Observation, say so explicitly and revise the earlier one. Never let both stand.

Ask only what isn't already answerable from the data or the request:

1. Is there a rubric, scoring guide, or marks distribution? If yes → read references/graded-work.md before anything else; the rubric outranks this spine. If no, proceed.
2. Execution mode — is the person running code themselves and expecting paste-ready blocks, or should code be executed directly with available tools?
3. The dataset — file(s), shape, any ID column, any date column, one table or several needing a join.
4. Objective — what decision this supports, for whom. Don't invent business narrative; ask in one line if absent.
5. The target column — named, or "there isn't one."
6. Problem type — resolve via triage below rather than asking directly.

Is there a target / outcome column?
├─ No  → OUT OF SCOPE. Say so plainly: this skill covers supervised
│        analysis only. Clustering, PCA/t-SNE, and network analysis
│        are not covered. Offer descriptive profiling (Overview + EDA,
│        stopping at the exit point) if that's useful, but do not
│        improvise an unsupervised modelling spine.
└─ Yes → Datetime index, forecasting a series forward?  → TIME SERIES
         Numeric, continuous / high-cardinality target?  → REGRESSION
         Categorical, or low-cardinality numeric target? → CLASSIFICATION
         Ambiguous (e.g. numeric with ~5 unique values)? → ASK

Resolve this before section 1. The fork governs the split strategy, the metrics, and parts of every later section.

Read the file in the right-hand column before starting the section in the left.

Read the fork file once the triage resolves — before Data Preparation, not at modelling time, since the fork changes the split strategy.

1. Title, Context, Objective, Data Dictionary
2. Importing Libraries
3. Data Overview
4. Exploratory Data Analysis
5. ← EXIT POINT. If there was no modelling intent — the person wanted to understand the data, not model it — write the summary of findings and stop here. Don't drift into preparation to be helpful.
6. Data Preparation (encoding, split, then treatment — in that order)
7. ← ENTRY POINT. Someone arriving with already-clean data starts here. Do the minimum mechanical prep to reach X/y and a split, and state plainly which preparation decisions were skipped rather than absorbing them silently.
8. Model Building → 9. Comparison → 10. Tuning → 11. Final Model → 12. Feature Importance
9. Actionable Insights and Recommendations, in two subsections — Key Takeaways (bulleted; each a bolded claim carrying the number that supports it) then Recommendations (numbered; each a bolded imperative action followed by the figure justifying it). Or hand off to findings-to-narrative if the write-up is substantial enough to want its own pass.

When the person runs the code themselves

• Colab data in: drive.mount('/content/drive') for anything reused across sessions; files.upload() for one-off small files; a direct URL if the assignment supplies one.
• Colab runtime: free-tier sessions disconnect and RAM is limited. Anything long-running gets sized before it's proposed — see the compute budget rule in references/modeling-and-tuning.md.
• Cell mechanics: + Code / + Text add cells; Ctrl+M M converts code to markdown; Ctrl+Enter runs in place, Shift+Enter runs and moves on. State once, at the start.
• Version drift: the reference notebooks predate current pandas/seaborn. Two known breaks — sns.histplot(..., palette=...) without hue is a no-op that warns (strip it entirely, don't keep it conditionally), and value_counts() output naming changed in pandas 2.x. Flag others as they surface rather than assuming an old idiom still runs.
• !pip install xgboost is unnecessary in Colab; it's preinstalled.

When code is executed directly

Keep the analysis in one jupytext percent-format .py, not a notebook — it diffs cleanly, runs as a plain script, and jupytext --to notebook converts it on demand.

Running it as a script means plt.show() draws to nothing. Redirect it once, in the imports cell, so the plot helpers can still be pasted verbatim:

matplotlib.use("Agg")
FIGDIR = Path("figures"); FIGDIR.mkdir(exist_ok=True)
FIG_NAME = "figure"; _fig_counter = itertools.count(1)

def _save_instead_of_show(*args, **kwargs):
    plt.savefig(FIGDIR / f"{next(_fig_counter):02d}_{FIG_NAME}.png", dpi=110,
                bbox_inches="tight")
    plt.close("all")

plt.show = _save_instead_of_show

Set FIG_NAME before each plotting call, so figures are named by variable rather than by position — position shifts the moment a cell is inserted, and build_report.py matches figures to cells through that name.

Person is running the code: every step is exactly what to paste, labelled by type. Two block types only — Markdown cell (raw markdown, not a description of it) and Code cell (exact runnable code; no pseudocode, no ellipses, no "adapt this to your columns"). Anything said to the person goes outside the blocks, never inside. Observations are markdown cells, not chat prose — writing an interpretation as conversational text leaves them to reformat it by hand, which defeats the point.

Code is being executed directly: run each step, show the real result, then write Observations the same way. Read each figure file back before writing Observations about it. The "never describe a chart you haven't seen" rule still binds when you generated the chart yourself — a PNG sitting on disk is not the same as having looked at it, and the failure mode is identical either way.

The HTML report

Both modes produce a browsable report alongside the notebook, so there is something readable at every checkpoint rather than only at the end:

python <skill-dir>/assets/build_report.py analysis.py -o report.html

Rebuild at the end of every completed section — after Data Overview, after EDA, after Data Preparation, and so on. The section is the gate, not the cell; rebuilding after every step adds a call per turn and buys nothing.

The renderer reads three cell types: # %% [markdown] for prose, # %% for code (collapsed behind a toggle, labelled with the cell's leading comment), and # %% [output] for captured output. Markdown cells beginning **Observation are styled as findings. It executes nothing.

When the person is running the code themselves, the .py is a transcript you maintain locally, and it can drift from what they actually ran. Three rules keep it honest:

• Append a cell only after they report its output, never when you hand it over. A cell they revised or that errored must be recorded as they ran it, not as you proposed it.
• Put their reported output in a # %% [output] cell, verbatim. That is what makes the report a record rather than a plan.
• Embed a figure only where they sent a screenshot (save it into the figures directory under the matching NN_name.png). Where they didn't, leave the gap — the report should show that a chart was not seen rather than quietly omit it, since the Observations under it rest on reported numbers alone.

Plots, when the person runs the code: ask for a cropped screenshot of the single figure. If they'd rather not screenshot, ask for the two or three numbers carrying the finding (peak location, skew direction, whisker extents) — and say plainly that written Observations will be weaker without seeing the chart.

Errors: ask for the full traceback, not just the last line.

• pandas, numpy, matplotlib.pyplot, seaborn, sklearn only, unless the objective genuinely needs more — and say so if it does rather than silently expanding.
• Short pandas chains (1–3 calls).
• Every code cell gets a plain natural-language # comment labelling what it does, in full phrases: # Check number of unique IDs - confirms each lead appears once (no group-leakage risk). Not terse variable-level notes. A cell combining two checks gets one comment per check, inline above each.
• One action per cell. A cell that loads the data and also prints its shape is two cells. Granularity is what makes the report's per-cell code toggles useful and what keeps each Observations block tied to one thing.
• All imports go in the imports cell at the top, never mid-notebook. Group them by purpose with a comment per group — data handling and plotting, model building, metrics, display options — rather than one undifferentiated block.
• Print sentences, not raw structures. print(f"There are {data.shape[0]} rows and {data.shape[1]} columns.") rather than print(data.shape); a reader should not have to decode (36275, 19). This matters more in the HTML report than in a notebook, where the Observations block otherwise has to spend its first bullet translating the output.
• Name results on a fixed scheme so they concatenate without thought: <model>_<split>_perf — dtree_train_perf, dtree_tuned_test_perf, rf_test_perf. The comparison table is then two lines, and a reader can predict any variable's name from its position in the analysis.
• Define a function the moment something is used twice; never inline the same plotting block twice.
• Fixed random_state on every split and model that accepts one, consistent across a session. If an assignment specifies a seed, the assignment wins.
• One deliberate palette choice per chart type, reused throughout.
• No pipelines, no scaling for tree-based models, no SHAP by default. If a technique is clearly warranted, say so rather than silently downgrading to match house style.

• First person plural or second person imperative: "Let's check...", "We will...", "You can see that...". Not passive.
• Every interpretation gets its own markdown cell, headed ### Observations: <subject> — ### Observations: Age, ### Observations: Conversion rate by first interaction channel. Name the subject; a bare ### Observations: is acceptable only where the subject is unambiguous from the heading directly above. Use ### Observation: <subject> for a single point. Bullets underneath, specific values bolded inline.
• The heading is not decoration. It puts every finding in the report's table of contents and makes it linkable, which a bold **Observations:** line inside a mixed cell does not. Never fold an interpretation into the same cell as the prose that introduces the next step.
• Where a plot raises a follow-up, close the Observation with the one-line hypothesis the next step tests. This is what chains sections together.
• A finding resting on a small sample gets its caveat in the same sentence as the finding — not a footnote, and not dropped.
• Insights: flat declarative bullets, each grounded in a number established earlier.

At the end of a working session, emit a one-line resume marker:

STATE: <project> | <problem type> | completed through §<n> <name> | decisions: <e.g. capped X via IQR on train only; plain get_dummies on ordinal Y> | next: <step>