Reference

Frameworks: hypothesis-driven problem solving and Analysis of Competing Hypotheses

Part of the Hypothesis Stress Testing skill · loaded on demand from SKILL.md

Contents

  1. Why two traditions, and what they share
  2. McKinsey hypothesis-driven problem solving (Rasiel 1999; Rasiel & Friga 2001; Conn & McLean 2018)
  3. Constructing a hypothesis tree from an issue tree
  4. The quick-and-dirty test: refutation conditions per hypothesis
  5. Heuer's Analysis of Competing Hypotheses (1999), the eight steps with reasoning
  6. Diagnosticity in depth, with an illustration
  7. The sensitivity check in depth
  8. Strong Inference (Platt 1964), falsifiability (Popper), and the six elements of a testable hypothesis
  9. Common misreadings of ACH
  10. Adapting ACH from intelligence to business problems
  11. Known limitations and how the skill compensates

1. Why two traditions, and what they share

The consulting tradition and the intelligence-analysis tradition arrived independently at the same discipline: state an answer early, then organise the work around trying to break it. The consulting version optimises for speed under deadline; the intelligence version optimises for avoiding a particular cognitive failure. Together they give a method that is both fast and hard to fool.

What they share is a rejection of the "gather everything, then see what it says" approach. Both hold that facts do not organise themselves, that an analyst without a hypothesis collects facts that happen to be available rather than facts that discriminate, and that the human default, once a plausible story is in mind, is to notice what fits it. The consulting version answers the first problem (no focus); Heuer answers the second (false focus). This skill needs both because a consulting team with a strong initial hypothesis and no ACH discipline is precisely the analyst Heuer is worried about.

2. McKinsey hypothesis-driven problem solving

Sources and what each contributes

Rasiel, "The McKinsey Way" (1999). Describes the three pillars of the firm's problem-solving approach as fact-based, rigidly structured, and hypothesis-driven. On the last: form an initial hypothesis at the outset, with the aim of solving the problem at the first meeting, then test it with fact-based analysis. The initial hypothesis is a road map for the work, structured by drawing the main problem, the three or four issues on which it turns, and the sub-issues under each. Rasiel's warning, in the same passage, is that the initial hypothesis is a starting point, not the final answer, and that the team must be ready to discard it when the facts go against it.

Rasiel and Friga, "The McKinsey Mind" (2001). Extends the practice with the quick-and-dirty test (QDT): before investing in analysis, ask what assumptions would need to be true for the hypothesis to be true, and check the ones that are cheap to check. If a cheap check fails, the hypothesis is abandoned before the expensive work starts. Also restates the 80/20 discipline ("don't boil the ocean"): analysis should be aimed at the key drivers, not at completeness.

Conn and McLean, "Bulletproof Problem Solving" (2018). Formalises a seven-step cycle: define the problem, disaggregate it with logic trees, prioritise (prune to the critical path), build a workplan, analyse, synthesise, communicate. The "one-day answer" is their term for the early best-guess answer, revisited and rewritten as the cycle iterates. They distinguish several tree types, including hypothesis trees, in which each branch is a claim to be tested rather than a question to be answered, and they recommend starting with simple heuristics before heavy analytical tools.

The mechanism

An explicit hypothesis converts an open question into a finite list of checks. "Why has margin fallen?" admits infinite investigation; "margin has fallen because national-account mix has shifted" admits about four checks. The workplan is therefore shorter, the team knows what data to request on day one, and progress is measurable (checks passed or failed) instead of felt (pages of analysis produced). The second mechanism is discardability: a hypothesis that is wrong should be cheap to abandon, because the checks that would show it wrong were the first things done. This only holds if the checks were designed to expose the hypothesis rather than to illustrate it, which is where Heuer becomes necessary.

The failure modes it prevents and the one it creates

Prevents: the ocean-boiling diagnostic, where a team spends six weeks describing the business and presents findings without an answer. Prevents: analysis that cannot be scoped or staffed because nobody knows what "enough" looks like.

Creates: anchoring. A hypothesis formed in the first meeting is usually the senior person's prior. Once stated, the team works to support it. Rasiel names this risk; the consulting literature does not supply a procedure for managing it beyond "be willing to change your mind". Heuer supplies the procedure.

Hypothesis tree versus issue tree

An issue tree decomposes a question into sub-questions (MECE at each level). A hypothesis tree decomposes an answer into the sub-claims that must all hold for it to be true. The two are convertible: each prioritised branch of an issue tree can be rewritten as a claim. The hypothesis form is more useful for planning analysis because each leaf names the observation that would confirm or refute it, and the tree as a whole shows which leaf, if it fails, brings down the answer. In this skill the issue tree arrives from problem-framing and the hypothesis tree is built in stage 2. When several rival top-level answers exist, each gets its own tree, and the leaves become the refutation conditions in stage 7.

3. Constructing a hypothesis tree from an issue tree

Start from the issue tree's prioritised branches. For the Northgate margin question (see worked-example.md), the issue tree from problem-framing had four branches: is the decline real; is it customer mix; is it price; is it cost. Each branch becomes one or more claims. The conversion rule is that every node must be a declarative sentence that could be false.

Issue tree (question form):

  • Why has gross margin fallen 230 bps?
    • Is the reported decline real?
    • Has the customer mix changed?
    • Has realised price fallen relative to list?
    • Has cost of goods risen relative to price?
      • Vendor cost increases?
      • Pass-through to list?
      • Pass-through timing?

Hypothesis tree for H3 (claim form), each leaf with its killing observation:

  • H3: Pass-through lag on list-priced core accounts explains at least 50 bps blended.
    • Vendor costs on the affected lines rose faster than list prices (dies if within 1 point).
    • The affected lines are a large share of core revenue (dies if under 30%).
    • The lag affects list-priced customers only (dies if national margin also fell).
    • The lag lengthened during the period (dies if the update cycle is unchanged).
    • The arithmetic of the above reaches 50 bps blended (dies if under 30 bps).

Three rules for the conversion. First, the leaves must be jointly sufficient: if every leaf holds, the top claim holds by arithmetic or logic, not by further argument. Second, each leaf must be independently checkable, with its own data source. Third, the tree should reveal which leaf is cheapest to check and which leaf is most likely to fail; those two properties determine the order of analysis in stage 7. A tree in which every leaf is expensive and none is likely to fail has been built to be confirmed, and should be rebuilt.

When rival hypotheses share a leaf (H1 and H3 both depend on national-account margin being stable), that leaf is checked once and the result enters the matrix as a single evidence row rated against both. When rival hypotheses make opposite predictions on a leaf (H1 predicts core margin flat; H3 predicts core margin down), that leaf is the crucial experiment and goes first.

4. The quick-and-dirty test: refutation conditions per hypothesis

Rasiel and Friga (2001) describe the quick-and-dirty test (QDT): before investing in analysis, list the assumptions that must hold for the hypothesis to be true and check the cheap ones first. This skill calls the output of that step refutation conditions (or QDT conditions), because each is written with the observation that would show it false. One historical note, kept here because it is the one place the phrase legitimately appears in this skill: the QDT is the analytical ancestor of Lafley and Martin's "what would have to be true" (Playing to Win, 2013), which applies the same logic to strategic options and asks under what conditions an option would be a winning choice. That strategy form belongs to strategic-choice and keeps the phrase; this skill uses the analytical form and does not. The two must not be confused in a client conversation, because the strategy form asks "under what conditions would we want to do this" and the analytical form asks "under what conditions is this claim true".

The procedure for writing and checking refutation conditions:

  1. Write the hypothesis in falsifiable form.
  2. List the conditions that must all hold for it to be true. Each condition should be an observable fact, not a restatement of the hypothesis. For "pass-through lag explains 66 bps": vendor costs on the affected lines rose faster than list prices; the affected lines are a large enough share of COGS; the lag affects list-priced customers and not cost-plus customers; the lag lengthened during the period.
  3. Mark each condition known-true, known-false, or unknown, with the source.
  4. For each unknown, specify the check, the data, the effort, and the result that would show it false.
  5. Order the checks by refutation power per day of effort. The cheapest check with the highest chance of killing the hypothesis goes first.
  6. Run until a condition fails (hypothesis dead or must be split) or all conditions hold (hypothesis survives this round; go to the ACH matrix).

The reason to prefer disproof is asymmetry: a hypothesis with ten conditions is confirmed only when all ten hold, but refuted when any one fails. Checking the shakiest condition first is the fastest route to a decision either way.

5. Heuer's Analysis of Competing Hypotheses: the eight steps with reasoning

Source: Richards J. Heuer, Jr., "Psychology of Intelligence Analysis", Center for the Study of Intelligence, Central Intelligence Agency, 1999, chapter 8. The chapter's illustration concerns whether Iraq would retaliate for the 1993 US strike on its intelligence headquarters, with three hypotheses ranging from no retaliation to a major attack.

Heuer's diagnosis of the default method: analysts identify what appears to be the most likely hypothesis and then assemble evidence to support it. He calls this satisficing. Its flaw is that the supporting evidence is usually consistent with the alternative hypotheses too, and those alternatives are never examined, so the analyst's confidence rests on evidence that does not actually discriminate. ACH is designed so that the alternatives cannot be ignored.

Step 1. Identify the possible hypotheses to be considered

Heuer's text: use a group of analysts with different perspectives to brainstorm the possibilities. Reasoning: the matrix can only choose among the hypotheses it contains, so an omitted hypothesis is an invisible error. Different perspectives are the cheapest way to widen the set. He distinguishes generating hypotheses (be inclusive) from evaluating them (be selective), and warns against discarding a hypothesis at this stage merely because there is no evidence for it yet: "unproven" is not "disproven". In consulting practice this is where the team's day-one answer meets the CFO's, the sales head's and the auditor's answers.

Step 2. Make a list of significant evidence and arguments for and against each hypothesis

Reasoning: "evidence" is broad on purpose. It includes facts, but also the analyst's own assumptions and logical deductions, because those drive judgment whether or not they are written down, and writing them down exposes them to challenge. It also includes absence: if a hypothesis predicts something that should have been observed and it was not, that absence is evidence against it. Heuer's illustration is Sherlock Holmes's dog that did not bark. The procedure at this step is to ask, for each hypothesis, what one would expect to see if it were true, and then to check whether it is seen.

Step 3. Prepare a matrix with hypotheses across the top and evidence down the side; analyse the diagnosticity of the evidence

Reasoning: the matrix forces every item of evidence to be considered against every hypothesis, which the satisficing analyst never does. Each cell answers: if this hypothesis were true, how likely would I be to see this evidence? Heuer uses C (consistent), I (inconsistent) and N/A, or plus and minus signs with double signs for strong cases. The most important instruction in the chapter is to work across rows (one item of evidence against all hypotheses) rather than down columns, because the point is comparison. Diagnosticity is assessed at this step: evidence that is consistent with all hypotheses has no diagnostic value, whatever its intrinsic interest.

Step 4. Refine the matrix

Reasoning: after step 3 the analyst usually sees that some hypotheses should be reworded, combined or split, and that some evidence is doing no work. Delete non-diagnostic evidence from the inference (Heuer suggests keeping a record of it). Add hypotheses that the evidence has suggested. The matrix is a working document, not a final one.

Step 5. Draw tentative conclusions about the relative likelihood of each hypothesis; proceed by trying to disprove hypotheses rather than prove them

Reasoning: this is the inversion at the heart of the method. Heuer's argument is that the scientific method tests hypotheses by seeking refutation, and that the hypothesis with the least evidence against it, not the most for it, is probably the best. His stated reason: a hypothesis can accumulate any amount of consistent evidence without being true, because the same evidence is consistent with rivals; but a single solid inconsistent item may be sufficient to reject a hypothesis. So the ranking is by inconsistency count weighted by reliability. He adds two cautions: the tentative conclusion should be the analyst's own judgment informed by the matrix, not a mechanical count; and if the analyst disagrees with the matrix, the disagreement should be examined, because it usually means an assumption is missing from the evidence list.

Step 6. Analyse how sensitive your conclusion is to a few critical items of evidence

Reasoning: in step 5 the analyst will notice that the ranking depends on a small number of cells, typically the I and II ratings against the leading hypotheses. Those are the linchpins. For each, ask whether the evidence could be wrong, incomplete, deceptive or interpretable another way, and what the conclusion would be if it were. Heuer notes that in intelligence the possibility of deliberate deception must be considered; in business the equivalents are mis-tagged data, restated accounts, and interested testimony. The output of this step is a list of the evidence on which the conclusion rests and what would happen if each fell.

Step 7. Report conclusions; discuss the relative likelihood of all the hypotheses, not just the most likely one

Reasoning: a report that presents only the winner deprives the reader of the ability to judge how close the race was and to react correctly when new evidence arrives. Reporting the standing of all hypotheses also makes explicit what was refuted and by what, which is the part a sceptical reader most wants.

Step 8. Identify milestones for future observation that may indicate events are taking a different course than expected

Reasoning: the conclusion is tentative and the world will provide more evidence. Specifying in advance which observations would indicate that a rejected hypothesis is reasserting itself, or that the leader is failing, converts the analysis into a monitoring plan and protects against the tendency to explain away later disconfirmation.

Heuer's own assessment of the method

He is careful about what ACH guarantees. It does not guarantee the right answer, because every cell rating is still a judgment made on incomplete information. What it guarantees is a process: all hypotheses considered, all evidence considered against all hypotheses, non-diagnostic evidence identified, the basis for the conclusion made visible and therefore auditable. He also notes that ACH is more effort than the default method, and recommends it for the questions where being wrong is expensive.

6. Diagnosticity in depth, with an illustration

Diagnosticity is the property of an item of evidence that it changes the relative likelihood of hypotheses. It is a property of a row, not of a cell. Three consequences:

First, it is independent of how striking the evidence is. "Margin fell in every region" is a dramatic fact and is consistent with mix shift, pricing, cost pass-through, product mix and reclassification alike. It has almost no diagnosticity.

Second, it depends on the hypothesis set. A row that reads C, C, C, C, II is highly diagnostic for the fifth hypothesis and uninformative among the first four. If the fifth is dropped after refinement, the row loses its value.

Third, magnitude creates diagnosticity where direction does not. Five hypotheses may all predict a margin decline; only some predict a 160 bps decline concentrated in core accounts. Quantitative evidence should therefore be rated on size, timing and scope, not just sign.

A practical rating: High when at least one cell is II, or the row contains both CC and I; Medium when the row contains at least one I, or a CC alongside N cells (it separates hypotheses by what they predict without refuting any); Low when every cell is C or N. Low rows are removed from the inference at step 4 and retained in an appendix.

Illustration. Three rows from the Northgate example, rated against the same five hypotheses (mix, discount, pass-through, product mix, artefact):

RowH1H2H3H4H5DiagnosticityWhy
All six regions declinedCCCCCLowEvery hypothesis predicts a broad decline; the row cannot separate them
National contracts are cost-plusCNCCNNMediumPredicted by H3, compatible with H1, silent on the rest; separates by prediction, refutes nothing
Core-account margin down 160 bpsIICCCCIHighH1 predicted flat; H5 cannot produce a like-for-like within-segment move; H3 predicted exactly this

The first row is the kind of fact that fills diagnostic decks and decides nothing. The third row is the one that ended the day-one answer. A useful habit when reviewing a matrix is to count the High rows: if there are fewer than three, the analysis plan has not yet produced discriminating evidence and the ranking should be labelled provisional.

A second subtlety: diagnosticity is not the same as reliability. A row can be highly diagnostic and unreliable (a single interview that flatly contradicts one hypothesis) or reliable and useless (an audited figure consistent with everything). The sensitivity check in section 7 exists because the conclusion rests on rows that are both diagnostic and, one hopes, reliable; the check tests the second property for the rows that have the first.

7. The sensitivity check in depth

The check has three parts. Identify: list every I and II cell against the top two hypotheses; the rows involved are the candidates. Interrogate: for each, name the way it could be wrong (source error, definition change, sampling, interested party, alternative reading) and the cheapest confirmation. Quantify: where the row is a number, compute how far it would have to move to change the ranking, and compare that with the plausible error of the source. A linchpin that would need to move by 300% to change the ranking is safe; one that would need to move by 15% when the source is a manual tag is not. The output is a short table: row, vulnerability, confirmation check, distance to flip, status. It is reported with the conclusion, not hidden in an appendix, because it is the honest content of the confidence statement.

8. Strong Inference, falsifiability, and the six elements of a testable hypothesis

Platt, "Strong Inference" (Science, vol. 146, 1964), described the practice of fast-moving scientific fields as a repeated cycle: devise alternative hypotheses; devise a crucial experiment whose possible outcomes will exclude one or more of them; carry it out cleanly; recycle with the survivors. His observation was that fields which did this advanced faster than fields which accumulated confirmations. Applied to analysis planning, it means preferring analyses that can distinguish between two live hypotheses over analyses that can only support one.

Behind both Platt and Heuer stands Popper's criterion (The Logic of Scientific Discovery, 1934; English edition 1959): a statement is scientific to the extent it forbids something, and the more it forbids the more it says. A hypothesis that is compatible with every observable outcome has no content. This is the reason stage 2 requires a killing observation in every hypothesis.

The six elements of a testable hypothesis, and why each is there

ElementWhat it fixesWhat goes wrong without it
SubjectThe specific quantity or state claimed"Profitability" cannot be measured; "gross margin in core accounts" can
Direction and magnitudeHow much, in the decision-maker's unitDirection-only claims are consistent with almost any evidence; magnitude is what rows can refute
MechanismThe causal pathWithout a mechanism the hypothesis predicts nothing beyond the fact it was built to explain, so no second row can test it
Time windowWhen the effect began or appliesLets timing evidence (a step change at a known date) count for or against
ScopeWhere it should and should not appearTurns "should not appear in cost-plus accounts" into a killing observation
Killing observationThe measurement that refutes itWithout this, the hypothesis is a theme; the team will find it "broadly supported" whatever the data

The mechanism element deserves emphasis. A hypothesis with a mechanism makes collateral predictions: pass-through lag predicts that the lag lengthened, that list-priced customers were hit and cost-plus customers were not, and that branches would report "we cannot raise prices". Each collateral prediction is a potential evidence row. A hypothesis without a mechanism ("margin fell because of pricing") makes only the prediction it was built from, and so can only ever be rated against one row.

Two tests to apply to each draft hypothesis. The negation test: write the negation and ask whether it is also a plausible hypothesis; if the negation is absurd ("margin did not fall because of anything to do with price"), the original was too broad. The two-analyst test: give the hypothesis and the data to two analysts separately and ask whether they would reach the same verdict on whether it survived; if the wording leaves room for disagreement, tighten the magnitude and the killing observation.

9. Common misreadings of ACH

Treating the matrix as a vote. Counting C cells per column and declaring the column with the most the winner. This is the satisficing strategy dressed in a table. The ranking is by evidence against.

Treating N as "unknown". N means the hypothesis makes no prediction about this evidence. Unknown evidence is not in the matrix at all; it is in the analysis plan.

Rating the hypothesis instead of the cell. The cell question is "if H were true, would I expect to see E?", not "does E make me believe H?". The second question smuggles in the prior. The two diverge most for evidence that is consistent with H but equally consistent with rivals: the first question gives C, the second gives a feeling of support.

Assigning numerical probabilities to cells and multiplying. Heuer's method is deliberately ordinal. Probabilities invite false precision, and the multiplication treats rows as independent when they usually are not (several rows from one source). If a quantitative treatment is wanted, keep it in the sensitivity check, where the question is how far one number would have to move.

Using the matrix once. The matrix is built early with the evidence in hand, then refilled as the refutation plan returns results. A matrix built only at the end, from evidence gathered under a single hypothesis, inherits the bias the method was meant to remove.

Confusing "no evidence against" with "proven". The leader is the hypothesis that has survived the evidence so far. Steps 6 and 8 exist because that is a provisional status.

Dropping refuted hypotheses from the report. Step 7 requires reporting the standing of all hypotheses. The decision-maker's confidence in the winner depends on seeing what lost and why; and the refuted hypotheses are usually the ones the decision-maker held.

Conflating this method with the pre-mortem. ACH asks what is true; the pre-mortem asks how a chosen plan fails. A team that finishes the matrix and then starts listing implementation risks has changed skill without noticing. Hand off to a pre-mortem.

10. Adapting ACH from intelligence to business problems

Four adaptations are needed.

Decomposition rather than a single winner. Intelligence questions are often "which of these mutually exclusive things is happening". Business performance questions are often "how much of the effect does each cause explain". The matrix still works, with hypotheses phrased as "H explains most of the effect" so that magnitude evidence can refute them, and the conclusion is a reconciled decomposition. The sum must reconcile to the observed total, and the residual must be named.

Addressability as a second ranking. The largest cause and the most fixable cause are different questions. Report both, because the decision-maker needs the second and will assume the first.

Testimony handling. Executives will supply confident explanations. Enter them as evidence about beliefs, rate them on diagnosticity like any other row, and expect them to rate Low. The value of the interview is the mechanism it suggests, which then becomes a check.

Time budget. Intelligence analysts may spend weeks on a matrix; consulting teams have days. The compensation is the quick-and-dirty test in stage 7: run the cheapest killing checks first, and treat the matrix as a living document that fills in as results return.

11. Known limitations and how the skill compensates

The ratings are judgments. Two analysts will rate some cells differently. Compensation: write the reasoning for every I and II cell, and have a second person rate the linchpin rows blind.

The set may omit the true hypothesis. Compensation: the artefact and null hypotheses are mandatory; the refinement step adds hypotheses the evidence suggests; the milestones step watches for the return of rejected ones.

Evidence rows are not independent. Several rows may derive from the same source, so a source error would flip them together. Compensation: note the source of each row, and in the sensitivity check treat rows from one source as one linchpin.

The method can be gamed by rewording. A hypothesis can be weakened until nothing contradicts it. Compensation: the refinement step records wording changes, and the falsifiability test rejects any hypothesis that has lost its killing observation.