Three Levels of "Independence" for Random Variables

A logical progression from the strongest constraint to the weakest — deeply understand the nature of statistical dependence.

Key Idea: In probability and statistics, "independence" or "no relation" has multiple levels of meaning. The strictest — mutual independence — means a complete absence of association, while the weakest — linear uncorrelatedness — only rules out linear relationships. Understanding these distinctions is essential for building correct statistical models.

Logical Implication Hierarchy

Mutual Independence
⇒
Mean Independence
⇒
Uncorrelated

One-way implication; the converse does not hold.

Counterexamples:

  • • Uncorrelated ↛ Mean Independent: a nonlinear relationship may exist
  • • Mean Independent ↛ Mutually Independent: conditional variance may change
1

Mutual Independence

The values of random variables $X$ and $Y$ do not affect each other. Knowing anything about $Y$ cannot change the probability distribution of $X$, and vice versa.

Definition (joint distribution equals the product of marginals)
$$f_{X,Y}(x,y) = f_X(x)f_Y(y) \quad \forall x,y$$

Real-world Example:

Flip a fair coin ($X$) and roll a fair die ($Y$). The outcome of the coin (heads or tails) has no effect on the die's value, and vice versa.

Property: Mutual independence ⇒ $E[g(X)h(Y)] = E[g(X)]E[h(Y)]$ holds for all functions $g,h$
2

Mean Independence

Knowing the value of $Y$ does not change the expected value (first moment) of $X$, but it may affect the variance or higher-order moments of $X$.

Definition (conditional expectation equals unconditional expectation)
$$E[X|Y] = E[X] \quad \text{almost surely}$$

Real-world Example:

House area ($X$) and house price ($Y$). Conditional on price, the expected area may stay constant, but its variance may grow as price rises (heteroskedasticity).

Key Property:

Mean independence ⇔ $Cov(X, h(Y)) = 0$ for all functions $h$ (not only linear ones)

3

Uncorrelated

There is no linear relationship between $X$ and $Y$. The covariance and correlation coefficient are both zero.

Definition (covariance is zero)
$$Cov(X, Y) = E[XY] - E[X]E[Y] = 0$$

Real-world Example:

Circle radius ($X$) and circle area ($Y = \pi X^2$). They have a quadratic, not linear, relationship, so they are uncorrelated yet strongly dependent.

Note: Uncorrelatedness is the weakest notion of "no relation" — it only rules out linear relationships, yet strong nonlinear dependence may still exist.

Distribution Patterns and Dependence Verification

Observe the satisfaction of three "independence" conditions by adjusting relationship patterns

Current Pattern Description:Linearly Correlated: There is a clear linear relationship between X and Y; none of the three conditions is satisfied.

Covariance

0.00

Correlation

0.00

E[X|Y] ≈ E[X]

0.00

Sample Size

300

Mutual Independence

--

Joint distribution = Product of marginals

Mean Independence

--

E[X|Y] = E[X]

Uncorrelated

--

Cov(X,Y) = 0

Practical Applications

Econometrics

In regression analysis, requiring explanatory variables to be uncorrelated with error terms is central to the Gauss-Markov assumptions, ensuring unbiased OLS estimates.

Portfolio Theory

Uncorrelated asset returns form the theoretical basis for diversification, effectively reducing portfolio risk.

Hypothesis Testing

Independence tests (e.g., chi-squared test) determine if two categorical variables are independent, a common tool in data analysis.

Machine Learning

Correlation analysis between features helps select independent features, avoiding multicollinearity and improving model performance.