CompTIA Data+ DA0-002 · Free study guide

Type I vs Type II errors, p-values, and what significance actually claims

The everyday meanings of "significant" and "probability" differ from their technical meanings in hypothesis testing. This review separates those meanings and connects each term to a concrete decision.

Scope of this guide

DA0-002 V2 does not explicitly list hypothesis tests, p-values, Type I/II errors, power, ANOVA, or chi-square in its current objectives. This is optional analytics enrichment, not a current exam checklist. Use the objective 3.2 lesson for the assessed statistical scope.

The two errors, and a way to stop confusing them

Hypothesis testing starts from a null hypothesis: the default claim that there is no effect, no difference, no relationship. You then ask whether your data is surprising enough under that assumption to reject it.

Two things can go wrong.

A Type I error is a false positive. You reject a null hypothesis that was actually true — you announce an effect that does not exist. Under the null and the test's assumptions, its long-run probability is controlled by the chosen significance level α, often 0.05.

A Type II error is a false negative. You fail to reject a null hypothesis that was actually false — a real effect was there and you missed it. Its probability is called β, and power is 1 − β: the chance of detecting a real effect when one exists.

If the numbering will not stick, use the order of harm in a drug trial: the first error is approving something that does not work. The second is missing something that does.

The key relationship is that tightening α to reduce false positives raises the false-negative rate, holding the test, effect, and sample size constant. You do not get both for free. A larger sample can improve power without increasing the chosen Type I error rate.

What a p-value is, and the four things it is not

A p-value is the probability, under the null model, of obtaining a test statistic at least as extreme as the observed one. That conditional clause is the whole concept, and common misreadings drop it.

A p-value of 0.03 does not mean:

• "There is a 3% probability the null hypothesis is true." The p-value is computed assuming the null is true. It cannot also be the probability of that assumption.
• "There is a 97% probability the alternative hypothesis is true." Same error, restated.
• "The effect is large." Significance and magnitude are independent. With a large enough sample, a difference too small to matter to anyone will be statistically significant.
• "The result has been proven." Hypothesis testing never proves anything. You reject the null or you fail to reject it. You never accept the null; that tempting phrasing overstates what the evidence established.

Correct reading of p = 0.03 at α = 0.05: the observed data would be unlikely under the null, so reject the null. That is all it licenses.

Confidence intervals say the same thing more usefully

A 95% confidence interval gives a range of parameter values reasonably consistent with the data under the model. When it is constructed under the same assumptions as a matching two-sided significance test, an interval for a difference that contains zero corresponds to failure to reject zero at that level.

An interval from −1.5 to 4.2 is therefore not significant, no matter how much more of it sits above zero than below. And the midpoint carries no special probability — "there is a 95% chance the true value is 1.35" is wrong.

Intervals show magnitude and precision in a way a bare p-value does not. A wide interval signals low precision, which can result from a small sample, high variability, the design, or the model.

Underpowered studies, and why "no effect found" is often meaningless

A study with twelve participants that fails to reject the null has not, by that fact alone, shown the treatment does not work. The result may be compatible with effects the design had little power to detect. Absence of evidence is not automatically evidence of absence; examine effect estimates, intervals, design, and power.

Power depends on sample size, effect size, variability, test design, and the chosen threshold. A small sample and imprecise estimate should make a null result less conclusive, not force a claim in either direction.

Multiple comparisons — the trap that looks like a discovery

Run 40 independent tests at α = 0.05 and, under the null, you expect about two significant results from chance alone. So "we tested 40 metrics and two came back significant" is compatible with chance alone and is not, by the count alone, compelling evidence against the joint null.

The corrections worth recognizing are Bonferroni (divide α by the number of tests) and false discovery rate control. Better still is deciding the primary hypothesis before looking. Raising α to compensate is offered as a distractor and makes the problem worse.

The A/B testing version: checking results continuously and stopping the moment p dips below 0.05. A p-value fluctuates as data accumulates, so peeking repeatedly and stopping at the best moment inflates the true false-positive rate well beyond the nominal level. Fix the sample size in advance, or use a method designed for repeated looks.

Base rates and predictive value

A test with 99% sensitivity and 99% specificity for a condition affecting 1 in 10,000 people returns a positive result. How likely is it that the person has the condition?

The instinct says 99%. Work it out with 1,000,000 people:

• About 100 genuinely have the condition; 99 of them test positive.
• About 999,900 do not; roughly 1% — nearly 9,999 — test positive anyway.

So about 99 true positives sit among roughly 10,098 positives: under 1%. The test is excellent; the condition is rare; false positives swamp true ones. This is why rare-condition screening programs often use confirmatory testing and why "the test is 99% accurate so the result is 99% reliable" is wrong.

Picking the right test

The following is a simplified orientation table; assumptions, pairing, distribution, sample size, and study design still matter:

The reliable tell is the data type. Means and numeric outcomes point to t-tests and ANOVA; counts and categories point to chi-square. Running repeated pairwise t-tests instead of ANOVA is a multiple-comparisons error, which is why ANOVA exists.

What to retain from this enrichment

Be able to state, in one sentence each: what a p-value is conditional on, why you never accept the null, why significance is not magnitude, why a small sample makes a null result uninformative, and why a rare condition makes a positive test weak evidence. Then practise the test-selection table until the data type and design narrow the appropriate test rather than assuming a single name is universally correct.