From the free course Introductory Statistics on Lykke

Hypothesis Testing with One Sample

Key concepts: Null Hypothesis (H0) · Alternative Hypothesis (Ha) · P-value · Significance Level (Alpha) · Type I and II Errors

The formal process for testing claims about a population using a single sample.

Hypothesis Testing with One Sample

Hypothesis testing is the formal statistical backbone of the scientific method. It provides a rigorous framework for moving beyond mere observation into the realm of inferential statistics, where we use sample data to make probabilistic statements about an entire population. At its core, hypothesis testing is a "proof by contradiction" mechanism: we assume a default state of the world (the Null Hypothesis) and determine if our observed data is so unlikely under that assumption that we are forced to reject it in favor of a new explanation.

In a one-sample context, we are typically comparing a single sample mean ($\bar{x}$) or proportion ($\hat{p}$) against a known or hypothesized population parameter ($\mu_0$ or $p_0$). This process is essential in quality control, medical research, and A/B testing, where one must decide if a deviation from the norm is a genuine effect or simply the result of stochastic noise.

The Logical Framework: $H_0$ and $H_a$

The foundation of any test lies in the construction of two mutually exclusive and collectively exhaustive statements.

The Null Hypothesis ($H_0$)

The Null Hypothesis represents the status quo, the "no-effect" state, or the assumption that any observed difference is due to chance. In a one-sample test, $H_0$ always contains an equality ($=, \le, \text{ or } \ge$).

The Alternative Hypothesis ($H_a$)

The Alternative Hypothesis (sometimes denoted as $H_1$) is the claim the researcher hopes to support. It represents a departure from the status quo. $H_a$ never contains an equality ($<, >, \text{ or } \neq$).

Hypothesis Type Notation Logical Role Mathematical Symbols
Null $H_0$ The "Innocent" plea; the baseline assumption. $=, \leq, \geq$
Alternative $H_a$ The "Guilty" verdict; the research claim. $\neq, >, <$

The Burden of Proof: In the Frequentist tradition, the burden of proof lies entirely on the Alternative Hypothesis. We do not "accept" the Null; we merely "fail to reject" it if the evidence is insufficient. This is analogous to a court of law where a defendant is found "not guilty" rather than "innocent."

The Decision Calculus: P-values and Alpha ($\alpha$)

To decide between $H_0$ and $H_a$, we need a threshold for "unlikeliness."

Significance Level ($\alpha$)

The Significance Level ($\alpha$) is the probability threshold set by the researcher before the experiment. It defines the maximum risk of rejecting a true null hypothesis that the researcher is willing to take. Common values include 0.05, 0.01, and 0.10.

The P-value

The P-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. It is NOT the probability that the null hypothesis is true; rather, it is $P(\text{Data} | H_0)$.

Decision Rules

  • If P-value $\le \alpha$: The result is statistically significant. Reject $H_0$.
  • If P-value $> \alpha$: The result is not statistically significant. Fail to reject $H_0$.

Statistical Error Theory: Type I and Type II

Because we are working with samples rather than entire populations, our decisions are subject to two types of errors.

  1. Type I Error ($\alpha$): Rejecting $H_0$ when it is actually true (a "False Positive").
  2. Type II Error ($\beta$): Failing to reject $H_0$ when it is actually false (a "False Negative").
Reality \ Decision Fail to Reject $H_0$ Reject $H_0$
$H_0$ is True Correct Decision (Confidence: $1-\alpha$) Type I Error (Significance: $\alpha$)
$H_0$ is False Type II Error ($\beta$) Correct Decision (Power: $1-\beta$)

The Power of a Test

The Power of a statistical test ($1 - \beta$) is the probability that the test will correctly reject a false null hypothesis. Increasing the sample size ($n$) or the effect size generally increases the power of the test.

Mechanics of the One-Sample Test

The choice of test depends on whether the population standard deviation ($\sigma$) is known and whether the sample size is large enough for the Central Limit Theorem to apply.

1. The One-Sample Z-Test

Used when the population variance $\sigma^2$ is known and the population is normally distributed or the sample size $n$ is large ($n \ge 30$).

2. The One-Sample T-Test

Used when the population variance $\sigma^2$ is unknown (which is almost always the case in practice). It uses the sample standard deviation ($s$) and follows the Student's t-distribution with $n-1$ degrees of freedom ($df$).

Mathematical Derivation (LaTeX/Math)

The following represents the formal derivation of the test statistic and the probability density function for the T-distribution.

\text{The Test Statistic for a One-Sample T-Test is defined as:}
\\
t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}
\\
\text{Where:}
\begin{cases} 
\bar{x} & \text{Sample Mean} \\
\mu_0 & \text{Hypothesized Population Mean} \\
s & \text{Sample Standard Deviation} \\
n & \text{Sample Size} 
\end{cases}
\\
\text{The Probability Density Function (PDF) of the T-distribution with } \nu \text{ degrees of freedom:}
\\
f(t) = \frac{\Gamma(\frac{\nu+1}{2})}{\sqrt{\nu\pi}\,\Gamma(\frac{\nu}{2})} \left(1+\frac{t^2}{\nu}\right)^{-\frac{\nu+1}{2}}

Step-by-Step Execution of a One-Sample Test

To ensure scientific validity, a hypothesis test must follow a standardized pipeline.

Step 1: State the Hypotheses

Define $H_0$ and $H_a$ clearly. Determine if the test is one-tailed (directional, e.g., "greater than") or two-tailed (non-directional, e.g., "different from").

Step 2: Set the Criteria

Choose the significance level $\alpha$. In high-stakes fields like pharmaceuticals, $\alpha$ might be 0.01; in social sciences, 0.05 is standard.

Step 3: Verify Assumptions

  • Independence: Observations must be independent of each other.
  • Randomness: Data should be collected via a random sampling method.
  • Normality: The population should be normally distributed, or $n$ should be large enough (Central Limit Theorem).

Step 4: Calculate the Test Statistic and P-value

Compute the Z or T score and find the corresponding P-value using distribution tables or software.

Step 5: Make the Decision

Compare the P-value to $\alpha$ and state the conclusion in the context of the original problem.

Component Two-Tailed Test One-Tailed Test (Right) One-Tailed Test (Left)
$H_a$ $\mu \neq \mu_0$ $\mu > \mu_0$ $\mu < \mu_0$
Rejection Region Both tails Right tail Left tail
P-value Calc $2 \times P(T > t )$

Critical Extensions: Effect Size and Confidence Intervals

A common pitfall in hypothesis testing is over-reliance on the P-value. With a large enough sample size, even a trivial difference can become "statistically significant." To counter this, we use Effect Size.

Cohen's $d$

Cohen's $d$ measures the magnitude of the difference in terms of standard deviations. $$d = \frac{|\bar{x} - \mu_0|}{s}$$

Effect Size Cohen's $d$ Value
Small 0.2
Medium 0.5
Large 0.8

Confidence Intervals (CI)

A Confidence Interval provides a range of plausible values for the population parameter. If the hypothesized mean $\mu_0$ falls outside the $(1-\alpha)$ confidence interval, we reject $H_0$ at the $\alpha$ level.

Common Pitfalls and Misconceptions

  1. "P-hacking": Running multiple tests or adjusting the sample size until a significant P-value is found. This inflates the Type I error rate.
  2. Misinterpreting the P-value: Thinking a P-value of 0.05 means there is a 95% chance the alternative hypothesis is true.
  3. Absence of Evidence vs. Evidence of Absence: Failing to reject $H_0$ does not prove $H_0$ is true; it simply means the data is not strong enough to prove otherwise.
  4. Statistical vs. Practical Significance: A result can be statistically significant ($p < 0.05$) but practically useless if the effect size is near zero.

Summary of the Decision Matrix

When approaching a one-sample problem, use this logic flow:

  1. Is the data continuous? (If yes, proceed to Means. If no, use Proportions).
  2. Is $\sigma$ known? (If yes, use Z-test. If no, use T-test).
  3. Is $n < 30$? (If yes, the population must be normal to use the T-test).
  4. Is the claim directional? (If yes, use a one-tailed test).
Hypothesis Testing with One Sample - Introductory Statistics - image 1
Hypothesis Testing with One Sample - Introductory Statistics - image 1
Hypothesis Testing with One Sample - Introductory Statistics - diagram 1
Hypothesis Testing with One Sample - Introductory Statistics - diagram 1
Hypothesis Testing with One Sample - Introductory Statistics - diagram 2
Hypothesis Testing with One Sample - Introductory Statistics - diagram 2
Hypothesis Testing with One Sample - Introductory Statistics - diagram 3
Hypothesis Testing with One Sample - Introductory Statistics - diagram 3

Source Materials

Study Introductory Statistics on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Introductory Statistics · Browse more course pages

Hypothesis Testing with One Sample — Introductory Statistics | Lykke