Manager Crumble (a fictional villain) runs the cookie counter at the Y. He swears the average Terp eats only 40 cookies a semester, so he can cut the cookie order and pocket the savings. Students smell a fib. You are the judge. The evidence: one random group of students. The question: is the evidence strong enough to throw out his claim? And because our Terps are simulated, we also have the security tape, so we can check whether each verdict was actually right.
Previously on The Cookie Question
One interview, two jobs: estimate the truth, or put a claim on trial
🍪
Last time we didn’t know how many cookies a Terp eats, so we used one random interview to estimate it: a single best guess x̄ (the point estimate) plus a confidence interval showing how far off that guess might be. Nobody had made a claim; we just wanted the number.
Now Manager Crumble shows up with a number of his own. We use the very same kind of interview, but its job changes: instead of “what is μ?”, we ask “does his number survive what we saw?” The interview becomes evidence in a trial.
Last time · estimation
“About how many cookies does a Terp eat?”
Start from the data. x̄ is our best guess for μ, and the CI is the range of values for μ that fit the data.
🍪same interview, same numbers
This time · hypothesis test
“Is Crumble’s claim believable, given what we saw?”
Start from the claim. Pretend μ = μ₀ and ask how surprising our x̄ would be. x̄ is now the evidence we weigh.
Is it fair to say the interview “stops being an estimate and becomes an observation”?
Almost. x̄ is the same number either way, and it is always both an estimate of μ and a random observation. What changes is the question we ask of it:
Estimation asks what x̄ tells us about μ, and reports x̄ ± z*·SE.
Testing fixes a hypothesized value μ₀ and treats x̄ as the observed value of a random variable, then asks how unusual it is under H₀: z = (x̄ − μ₀)/SE and its p-value.
Both use the same sampling distribution and the same SE, which is why, for a mean, a two-sided test at level α and a (1 − α) CI always agree (Step 3 shows this). For a proportion the two use different SEs and can disagree in borderline cases (Step 3, the ⚠️ box).
Step 1 · The accusation
Manager Crumble’s claim goes on trial
⚖️
In this courtroom the claim is presumed honest until the evidence proves otherwise, just like “innocent until proven guilty.” We don’t get to convict him because we dislike him; we need evidence that would be hard to explain if he were telling the truth.
So the burden of proof is on the accusation. That’s why there are only two possible verdicts: guilty (the evidence knocks his claim down) or not guilty (it doesn’t). There is no verdict called “proven honest.”
The charge sheet
H₀ is Crumble’s claim (presumed honest). Hₐ is the accusation: what we conclude if the evidence is strong enough.
Statistician’s wording
H₀ (null hypothesis) is assumed true for the purpose of the test; we look for evidence against it. Hₐ (alternative hypothesis) is supported only if the data provide convincing evidence against H₀. The possible decisions are reject H₀ or fail to reject H₀.
📼 The security tape (God’s view)
We built all 10,000 students ourselves, so we know the real answer: μ = cookies (σ = ).
Real courts never get this tape. Turn it off in the bar below to see the trial the way a real researcher would.
Court words ⇄ statistics words
In the courtroom
In statistics
What it means here
Presumed honest
H₀ assumed true
We start by assuming Crumble’s number is right.
The accusation
Hₐ
Terps eat a different number (or more, or fewer) than he says.
The evidence
one random sample, x̄
The average cookie count of the students we question.
“Beyond reasonable doubt”
significance level α
How rare the evidence must be (if he were honest) before we convict. Set before the trial.
How damning is it?
p-value
If he were honest, how often would luck alone produce evidence at least this damning?
Guilty
reject H₀
The evidence is convincing; his claim is thrown out.
Not guilty
fail to reject H₀
Not enough evidence. It does not mean he was honest.
Convicting an honest manager
Type I error
Rejecting a true H₀. Happens about α of the time when he’s honest.
A lying manager walks free
Type II error
Failing to reject a false H₀.
The security tape
“God’s view”
The true μ. Only a simulation has it.
40
n
α
Hₐ
Step 2 · The evidence
We question a random group of Terps. How far is their average from his claim?
🔎
We pick students at random and count their cookies. Their average is our evidence. Even if Crumble were honest, a random group would never land exactly on his number: some groups happen to include big eaters, some don’t. So the real question is not “is there a gap?” but “is the gap bigger than luck usually produces?”
To judge that, we measure the gap in standard errors (SE), the typical size of a luck-of-the-draw gap for a group this size. A gap of 1 SE is everyday luck; a gap of 3 SE is a lot to blame on luck. Press “New interview” to question a different group.
sample mean x̄
sample SD s
SE = s/√n
gap x̄ − μ₀
z = gap / SE
p-value
What the p-value means, in one question
If Crumble were telling the truth, how often would a random group give evidence at least as damning as ours, just by luck?
1Give him the benefit of the doubt
Imagine a world where he’s honest: the true average really is cookies. There, the true gap is zero, and any gap a group shows is pure luck of the draw.
2Measure how damning our evidence is
“Damning” (statisticians say extreme) means far from his claim, counted in standard errors, the typical size of a lucky gap: This is the same z-scale as the charts in Step 3.
3Ask: how often would luck do this?
In the honest-manager world, the share of groups that land :
Don’t trust the formula? Build the honest-manager world.
We make Crumble honest by adding cookies to every one of our 10,000 students, so their average is exactly his claim (same skew, same smuggler). Then we question 1,000 random groups of the same size and see how often luck alone produces a gap as damning as ours.
Statistician’s wording
The p-value is the probability, computed assuming H₀ is true, of obtaining a test statistic at least as extreme as the one observed, where “extreme” is in the direction(s) specified by Hₐ.
What the p-value is not:
Step 3 · Three ways to weigh the evidence
p-value, critical value, confidence interval: three judges, one verdict
👩⚖️
Imagine three judges looking at the same evidence. The first two start from “suppose Crumble is honest” and ask whether our group looks too far off. One measures it as a rarity (how often luck does this), the other as a distance (is it past the line we drew in advance). The third judge starts from the evidence instead and asks: “which claims could an honest manager have made, given what we saw?” For a mean they always reach the same verdict, and this step shows why. Mean only For a proportion, the third judge can occasionally disagree in borderline cases; the ⚠️ box near the end of this step shows why.
All charts share the same cookie axis. Methods 1 and 2 are the same test: both draw the curve H₀ predicts for x̄ (centered at the claim μ₀) and ask whether our x̄ is too far out. Method 1 judges that by an area, Method 2 by a position. Method 3 flips the viewpoint and centers the curve on our x̄. The gold dotted line is the true μ, from God’s view.
Test view · Methods 1 and 2 both assume Crumble is honest (H₀ true)Same curve: if μ really were μ₀, x̄ ~ N(μ₀, SE). Two ways to read it.
Method 1 · compare areas
p-value
If the claim were right, how often would x̄ land at least this many SEs away? That share is the shaded area. Reject when it’s ≤ α.
⇔same curve same verdict
Method 2 · compare positions
Critical value
Mark the cutoff z* that leaves area α outside. Reject when x̄ (or z) is past it.
Put them on one curve: area ⇔ position
The red region starts at z* and has area α. The blue region starts at our z and has area p. Each position on the ruler has its own tail area, and the further out you start, the smaller the area.
Method 1 asks
Method 2 asks
CI view · start from the evidenceCentered on our x̄: which claims are believable?
Method 3 · CI view
Confidence interval
Build the interval of plausible μ’s around x̄. Reject when μ₀ falls outside it.
p-value arearejection region (area α)confidence intervalx̄ (observed)μ₀ (claim)true μ
Same distance, two viewpoints: why the test and the CI can’t disagree Mean only
⚠️
Mean only. Both rows below use the same ruler, SE = s/√n, so the bars have identical widths and the verdicts flip at exactly the same gap. For a proportion the test’s ruler is √(p₀(1 − p₀)/n) and the CI’s is √(p̂(1 − p̂)/n): two different widths, so near the cutoff they can disagree. See the proportion box below.
📏
The test stands at Crumble’s claim and asks: “is the evidence within z* standard errors of me?” The CI stands at our evidence and asks: “is the claim within z* standard errors of me?” The distance from your house to school is the same as from school to your house, so the two questions always get the same answer. Drag the distance, or press sweep, and watch both rows flip at the same moment.
Coverage: what “95% confident” is really counting Mean only
Now put Crumble in the honest-manager world (his claim is the truth) and hold 50 trials. Each row is one random group: the dot is its average, the bar is its CI, and the shaded column is the claim’s “ordinary” zone. A trial’s CI misses the true claim exactly when its dot falls outside the zone. Same distance, same verdict, every row.
Statistician’s wording
⚠️ Proportions: the two viewpoints use different rulers
📐
New charge: Crumble claims only 10% of Terps are “big eaters.” We count big eaters in a random group. Now the yardstick depends on the proportion itself: SE = √(p(1 − p)/n). The test stands at his claim and must use his p₀ (that’s what “assume he’s honest” means). The CI has no claim to use, so it plugs in our p̂. Two different p’s, two rulers of different lengths. The gap between p̂ and p₀ is the same from both ends, but it’s measured against different yardsticks.
Most of the time both still give the same verdict. Only when the evidence sits right near the cutoff can one say guilty while the other says not guilty. Try the presets.
Where can they disagree? Every possible count, same claim and n
Each tick is one possible number of big eaters (0 to n). Red: both say guilty. Green: both say not guilty. Gold: the two views disagree. The gold only ever shows up in thin slivers next to the cutoffs.
📋 Rule of thumb: when can the CI stand in for the test?
One line for students: for a proportion, the CI gives the same verdict as the test unless p₀ sits right at the edge of the interval. If it’s that close, run the test.
Statistician’s wording
For H₀: p = p₀, the one-proportion z-test uses SE₀ = √(p₀(1 − p₀)/n) because H₀ specifies p; the Wald CI uses SE = √(p̂(1 − p̂)/n). The two decisions can differ when p̂ is near the rejection boundary; the discrepancy shrinks as n grows. Report the test as the formal decision. The interval that exactly matches the test is the Wilson (score) interval, the set of all p₀ the z-test would not reject; in R, prop.test(x, n, p = p0, correct = FALSE) reports both, consistently.
Why do these rules work?
Three “but why?” questions every jury asks. Each answer updates with the settings bar.
1 · Why reject when p ≤ α? (and is it α or α/2?)
2 · Why is “beyond z*” the rejection region?
3 · Why reject when μ₀ falls outside the CI? Mean only
Step 4 · Reading the verdict
Guilty or not guilty: the decision rules, side by side
🔨
The bar for “guilty” (α) is set before the trial, so nobody can move it after seeing the evidence. If the evidence clears the bar, the claim is thrown out. If not, the case is dismissed, and Crumble walks out not guilty, which is not the same as proven honest.
Hₐ
p-value
Critical value: reject if…
CI: reject if…
μ ≠ μ₀
2·P(Z ≥ |z|)
|z| ≥ zα/2 (1.96 at α = .05)
μ₀ is outside the (1 − α) CI
μ > μ₀
P(Z ≥ z)
z ≥ zα (1.645 at α = .05)
μ₀ is below the one-sided lower bound x̄ − zα·SE
μ < μ₀
P(Z ≤ z)
z ≤ −zα (−1.645 at α = .05)
μ₀ is above the one-sided upper bound x̄ + zα·SE
Highlighted row = your current Hₐ. With small n, use t in place of Z.
⚠️
The CI column agrees exactly only for a mean. For a mean, the test and the CI use the same SE = s/√n, so all three methods always agree. For a proportion, the p-value and critical-value columns still always agree (same test), but the CI column uses SE from p̂ instead of p₀ and can split from them in borderline cases. Then the test is the formal verdict.
Reject H₀ · court: “guilty”
p-value ≤ α, or z in the rejection region, or μ₀ outside the CI.
Statistician’s wording: “The data provide convincing evidence (at the α level) that the average Terp eats μ₀ cookies.”
Fail to reject H₀ · court: “not guilty”
p-value > α, or z outside the rejection region, or μ₀ inside the CI.
Statistician’s wording: “The data do not provide convincing evidence (at the α level) that the average differs from μ₀.”
Never say: “we accept H₀”, “the average is μ₀”, “H₀ is true.” Step 5 shows why.
Step 5 · Why “not guilty” ≠ “innocent”
“Fail to reject” means “not ruled out,” not “proven true.” The security tape proves it.
📼
Real courts never find out whether a “not guilty” defendant actually did it. We can, because we have the security tape: the true average is 42.20 cookies. So we can watch Crumble walk free while lying, over and over, and see exactly why a statistician never says “we accept H₀.”
1One body of evidence, many claims it can’t convict Mean only
Keep the same interview and test every possible claim from 25 to 75 cookies. Green = we would fail to reject that claim; red = we would reject it.
⚠️
Mean only: here the green band equals the usual CI exactly. For a proportion, the band of claims the test can’t reject is the Wilson (score) interval, which is close to, but not the same as, p̂ ± z*·√(p̂(1 − p̂)/n). The big idea (many claims survive, so “not guilty” ≠ “innocent”) holds either way.
2An unreliable witness: what if the smuggler lands in our group?
One student in the population claims 300 cookies. Force him into our sample and watch x̄ jump, s balloon, and the CI stretch until even a wild claim can’t be rejected. The switch changes the interview everywhere on the page (Steps 2–5).
3What the verdict does and doesn’t say
“Not guilty” ≠ “innocent”A “not guilty” verdict only means the evidence wasn’t strong enough. Crumble might still be lying, and with a small group of witnesses he often gets away with it.
“Fail to reject” ≠ “H₀ is true”Our sample couldn’t rule out μ₀, and it couldn’t rule out lots of nearby values either (part 1). An unreliable witness like the smuggler can collapse the case without proving anything.
Statistician’s wording
Failing to reject H₀ is not evidence that H₀ is true. It means the data are consistent with H₀ (and with other values in the confidence interval). Conclude: “The data do not provide convincing evidence for Hₐ,” never “we accept H₀.”
4Hold 200 trials and check every verdict against the tape
200 different courts, each with its own random group of n students, each running the same test against the current claim. Try the claims below, then change n. The gold outline in the table marks where your one interview from Step 2 lands.