From the free course Statistics and Probability on Lykke

Modeling Data Distributions

Key concepts: Z-scores · Percentiles · Linear Transformations · Normal Distribution · Empirical Rule (68-95-99.7)

Exploring the position of data points within a distribution and the properties of Normal distributions.

Modeling Data Distributions

In the preceding chapters, we focused on descriptive statistics—the art of summarizing and visualizing the data we have actually observed. However, the true power of statistics lies in modeling: the transition from describing a finite set of data points to defining a mathematical abstraction that represents the underlying process. Modeling allows us to make predictions, compare disparate datasets, and understand the probability of future occurrences.

This section explores how we locate individual values within a distribution, how we transform entire datasets to fit specific scales, and how we utilize the Normal Distribution—the "bell curve"—as a foundational tool for statistical inference.

Measuring Relative Standing: Percentiles and Z-Scores

When we look at a single data point, its raw value often tells us very little. If a student scores 85 on a test, we cannot determine if they performed well without knowing the distribution of the rest of the class. To provide context, we use measures of relative standing.

Percentiles

The $p$-th percentile of a distribution is the value such that $p$ percent of the observations fall at or below it.

Definition: If a value $x$ is at the 90th percentile, it means 90% of the data points in the set are less than or equal to $x$.

Percentiles are particularly useful for non-symmetric distributions because they are resistant to outliers. A common visualization for percentiles is the Cumulative Relative Frequency Graph (or Ogive), which plots the percentile rank on the y-axis against the data values on the x-axis.

Z-Scores (Standardized Scores)

While percentiles tell us about rank, Z-scores tell us about distance. A Z-score measures how many standard deviations a value is from the mean.

The formula for calculating a Z-score is: $$z = \frac{x - \mu}{\sigma}$$

Where:

  • $x$ is the raw value.
  • $\mu$ is the population mean.
  • $\sigma$ is the population standard deviation.
Feature Percentiles Z-Scores
Unit of Measure Percentage (0-100) Standard Deviations
Center Median is the 50th percentile Mean is always $z = 0$
Sensitivity Robust to outliers Sensitive to outliers (via $\mu$ and $\sigma$)
Comparison Best for ranking within one group Best for comparing across different groups

Linear Transformations

Often, we need to convert data from one scale to another—for example, converting Celsius to Fahrenheit or adding a curve to exam scores. These are linear transformations. A linear transformation follows the form $x_{new} = a + bx$.

The Effect of Shifting (Adding/Subtracting $a$)

When we add a constant $a$ to every value in a dataset:

  1. Measures of Center and Location (mean, median, quartiles, percentiles) increase by $a$.
  2. Measures of Spread (range, IQR, standard deviation) remain unchanged.
  3. The shape of the distribution remains unchanged.

The Effect of Scaling (Multiplying/Dividing by $b$)

When we multiply every value in a dataset by a constant $b$:

  1. Measures of Center and Location are multiplied by $b$.
  2. Measures of Spread are multiplied by $|b|$.
  3. The shape of the distribution remains unchanged.

Mathematical Derivation of Linear Transformation Effects

Let X be a random variable with mean E[X] and variance Var(X).
Define a transformed variable Y = a + bX.

1. Expectation (Mean):
   E[Y] = E[a + bX]
   E[Y] = E[a] + E[bX]
   E[Y] = a + b * E[X]

2. Variance:
   Var(Y) = Var(a + bX)
   Var(Y) = Var(bX)  (Adding a constant 'a' does not change spread)
   Var(Y) = b^2 * Var(X)

3. Standard Deviation:
   SD(Y) = sqrt(b^2 * Var(X)) = |b| * SD(X)

Density Curves

As we move from discrete data points to continuous models, we use density curves. A density curve is a mathematical model that describes the overall pattern of a distribution.

The Two Rules of Density Curves:

  1. The curve is always on or above the horizontal axis.
  2. The total area under the curve is exactly 1.

The area under the curve between two points represents the proportion of all observations that fall within that range. While the mean of a density curve is its "balance point," the median is the "equal-areas point" (the point that divides the area under the curve in half).

Distribution Shape Mean vs. Median
Symmetric Mean = Median
Skewed Right Mean > Median (mean is pulled toward the tail)
Skewed Left Mean < Median (mean is pulled toward the tail)

The Normal Distribution

The most important density curve in statistics is the Normal Distribution. It is symmetric, single-peaked, and bell-shaped. Any Normal distribution is completely described by two parameters: its mean ($\mu$) and its standard deviation ($\sigma$). We denote this as $N(\mu, \sigma)$.

Why the Normal Distribution Matters

The Normal distribution appears everywhere in nature and industry—heights, blood pressure, errors in measurement, and standardized test scores. Its ubiquity is largely explained by the Central Limit Theorem, which states that the sum (or mean) of many independent random variables tends toward a Normal distribution, regardless of the original distribution's shape.

The Probability Density Function (PDF)

The height of the Normal curve at any point $x$ is given by the Gaussian function: $$f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}$$

The Empirical Rule (68-95-99.7)

For any Normal distribution, the Empirical Rule provides a quick way to estimate the proportion of data within certain distances of the mean.

The 68-95-99.7 Rule:

  • Approximately 68% of the observations fall within $1\sigma$ of the mean $\mu$.
  • Approximately 95% of the observations fall within $2\sigma$ of the mean $\mu$.
  • Approximately 99.7% of the observations fall within $3\sigma$ of the mean $\mu$.

Worked Example: Industrial Quality Control

Suppose a factory produces steel rods with a mean diameter of 50mm and a standard deviation of 0.02mm. The distribution of diameters is Normal.

  1. What range contains 95% of the rods? According to the Empirical Rule, 95% fall within $2\sigma$. Range = $50 \pm 2(0.02) = [49.96mm, 50.04mm]$.
  2. What percent of rods are thicker than 50.04mm? Since 95% are between 49.96 and 50.04, the remaining 5% are outside that range. Because the curve is symmetric, half of that 5% (which is 2.5%) will be above 50.04mm.
Distance from Mean Area Under Curve (Cumulative) Tail Area (Outside)
$\pm 1\sigma$ 0.6827 0.3173
$\pm 2\sigma$ 0.9545 0.0455
$\pm 3\sigma$ 0.9973 0.0027

The Standard Normal Distribution

To simplify calculations, we often transform any $N(\mu, \sigma)$ distribution into the Standard Normal Distribution, denoted as $N(0, 1)$. This is achieved by converting every $x$ value into its Z-score.

Once standardized, we can use a Standard Normal Table (Z-table) or software to find the area under the curve for any Z-value.

Assessing Normality

Not every bell-shaped curve is Normal. Before applying the Empirical Rule or Z-score probabilities, we must verify the "Normality" of the data.

  1. Visual Inspection: Plot a histogram or stemplot. Look for symmetry and a single peak.
  2. Check the Empirical Rule: Calculate $\bar{x}$ and $s$. See if approximately 68%, 95%, and 99.7% of the data points fall within 1, 2, and 3 standard deviations.
  3. Normal Probability Plot (Q-Q Plot): This is the most reliable method. A Q-Q plot graphs the observed data against the "theoretical" quantiles of a Normal distribution.
    • If the points on a Normal probability plot lie close to a straight line, the data is approximately Normal.
    • Systematic deviations from a straight line indicate non-Normality (e.g., a curve indicates skewness).

Common Pitfalls and Misconceptions

  • Confusing Z-scores and Percentiles: A Z-score of 1.0 does not mean the 1st percentile; it means the value is 1 standard deviation above the mean (which corresponds to roughly the 84th percentile).
  • Assuming Normality: Many students apply the 68-95-99.7 rule to distributions that are heavily skewed. This leads to massive errors in probability estimation. Always check a Q-Q plot first.
  • Standardizing Shape: Standardizing (Z-scores) changes the scale and center of a distribution, but it never changes the shape. If the original data was skewed, the Z-scores will be equally skewed.
  • The "0.5" Error: In a Normal distribution, the probability of a single exact point (e.g., $P(X = 100)$) is technically zero because the area of a line is zero. We always calculate probabilities for intervals.
Modeling Data Distributions - Statistics and Probability - image 1
Modeling Data Distributions - Statistics and Probability - image 1
Modeling Data Distributions - Statistics and Probability - diagram 1
Modeling Data Distributions - Statistics and Probability - diagram 1
Modeling Data Distributions - Statistics and Probability - diagram 2
Modeling Data Distributions - Statistics and Probability - diagram 2

Source Materials

Study Statistics and Probability on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Statistics and Probability · Browse more course pages

Modeling Data Distributions — Statistics and Probability | Lykke