Simple Linear Regression — Formulas and Parameters
Simple Linear Regression is one of the most fundamental supervised learning algorithms in Machine Learning, used to build a linear relationship model between one independent variable and one dependent variable.
Model equation
\[ y = \beta_0 + \beta_1 x + \varepsilon \]
Least squares parameter estimation
\[ \hat{\beta}_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}, \quad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} \]
Loss function (Mean Squared Error, MSE)
\[ \text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 = \frac{1}{n}\sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2 \]
\( y \)
Dependent variable (response variable / target variable). It is the numerical value we want to predict or explain, such as house price, temperature, or sales. In simple linear regression, \( y \) is a continuous numerical value.
\( x \)
Independent variable (predictor variable / feature variable). It is the numerical value used to explain or predict \( y \), such as house area, date, or advertising spend.
\( \beta_0 \)
Intercept term (Intercept). It represents the expected value of \( y \) when \( x = 0 \), and determines the position of the regression line on the vertical axis. In the chart, it appears as the intersection of the line with the y-axis.
\( \beta_1 \)
Slope. It represents the average change in \( y \) for each one-unit change in \( x \), and determines the direction and degree of the regression line's tilt. \( \beta_1 > 0 \) indicates a positive correlation, and \( \beta_1 < 0 \) indicates a negative correlation.
\( \varepsilon \)
Random error term (Error Term). It represents the random fluctuation that the model cannot explain, originating from measurement error, omitted variables, or the randomness of the data itself. It is assumed that \( \varepsilon \) follows a normal distribution with mean 0.
\( n \)
Sample size. The total number of data points. The larger the sample size, the more accurate the parameter estimates and the more stable the model. However, in practical applications, the cost of data acquisition must be weighed.
Interactive Simulation — Adjust Parameters in Real Time
Drag the sliders below to adjust the model's slope \( \beta_1 \) and intercept \( \beta_0 \), and observe how the fitted line changes. Click the "Optimal Fit" button to automatically compute the least squares solution.
Model Parameters
Slope \( \beta_1 \) 1.00
Intercept \( \beta_0 \) 2.00
Data Settings
Number of data points \( n \) 30
Noise level \( \sigma \) 1.50
Actions
Model Evaluation
MSE Loss
--
R² Goodness of Fit
--
Current Equation
y = 1.00x + 2.00
Regression Fit Visualization 30 data points | noise σ=1.50
In-Depth Understanding of Simple Linear Regression
The following content comprehensively introduces simple linear regression from multiple dimensions to help build a complete understanding.

Standard Introduction

Simple Linear Regression is one of the most fundamental and classic supervised learning algorithms in statistics and Machine Learning. It assumes that there is a linear relationship between the dependent variable \( y \) and the independent variable \( x \), and estimates the model parameters by minimizing the residual sum of squares between observed and predicted values (i.e., Ordinary Least Squares, OLS). The model form is \( y = \beta_0 + \beta_1 x + \varepsilon \), where \( \beta_0 \) is the intercept, \( \beta_1 \) is the slope, and \( \varepsilon \) is a random error term that follows a normal distribution. Although the model structure is simple, its mathematical derivation is rigorous and its theoretical properties are complete (it is the Best Linear Unbiased Estimator, BLUE, under the Gauss-Markov assumptions), making it the cornerstone for understanding complex Machine Learning models.

Plain-Language Introduction

Imagine you are studying the relationship between "study time" and "exam scores." You collect data from some students and find that, roughly speaking, the longer the study time, the higher the score. Simple linear regression helps you draw the most appropriate straight line through these scattered data points. This line can be expressed with a simple formula: Score ≈ Intercept + Slope × Study Time. The "intercept" is the baseline score when study time is 0, and the "slope" indicates how many points the score increases on average for each additional hour of study. With this line, you can predict the score corresponding to any study time. Although real data will not fall perfectly on this line (due to noise and individual differences), this line is the best summary of the "overall trend."

What Problem Does It Solve

Simple linear regression mainly solves the following three core types of problems:

  • Prediction problems: Given the value of the independent variable \( x \), predict the expected value of the dependent variable \( y \). For example, predicting house prices given house area, or predicting sales given advertising spend.
  • Relationship quantification problems: Precisely quantify the strength and direction of the linear relationship between two variables. The sign and magnitude of the slope \( \beta_1 \) directly answer "how large is the effect of \( x \) on \( y \), and is it positive or negative."
  • Hypothesis testing problems: Through statistical tests (such as the t-test and F-test), determine whether the effect of the independent variable \( x \) on the dependent variable \( y \) is statistically significant, thereby providing a basis for scientific decision-making.

Why It Matters

The importance of simple linear regression is reflected not only in its direct application value but also in its status as a theoretical cornerstone of Machine Learning:

  • Theoretical cornerstone: It is the starting point for understanding all regression models (multiple linear regression, logistic regression, ridge regression, LASSO, etc.). Once you master core concepts such as least squares, loss functions, and gradient descent, you can easily transfer them to more complex models.
  • Strong interpretability: Unlike black-box models (such as deep neural networks), the parameters of simple linear regression have clear physical meaning and interpretability, which is crucial in fields such as healthcare and finance that require interpretability.
  • Computationally efficient: It has an analytical solution (closed-form solution) and requires no iterative optimization, with a computational complexity of O(n), so it can be solved quickly even in big data scenarios.
  • Baseline model: In practical projects, simple linear regression is often used as a baseline model to measure whether the performance improvement of a more complex model is worth the additional complexity it introduces.
  • Teaching classic: Almost all statistics and Machine Learning courses use simple linear regression as the first model, and its derivation process covers a complete methodology including parameter estimation, hypothesis testing, and model evaluation.

Application Scenarios

Simple linear regression is widely applied in various industries:

  • Economics and finance: Analyzing the relationship between GDP and employment rate, the impact of interest rates on stock prices, and the relationship between income and consumer spending.
  • Real estate: Using a single factor such as house area, house age, or geographic location to predict house prices.
  • Marketing: Quantifying the relationship between advertising spend and sales, and optimizing the allocation of advertising budgets.
  • Medicine and biostatistics: Studying the relationship between drug dosage and therapeutic response, and the relationship between age and blood pressure.
  • Environmental science: Analyzing the relationship between carbon dioxide concentration and global temperature change, and the relationship between pollutant emissions and the air quality index.
  • Education: Studying the relationship between factors such as study time and attendance rate and exam scores, providing data support for teaching strategies.
  • Engineering and manufacturing: Analyzing the impact of process parameters such as temperature and pressure on product quality indicators, and optimizing production processes.

Limitations and Drawbacks

Although simple linear regression is fundamental and important, it has the following obvious shortcomings:

  • Can only handle a single independent variable: In the real world, an outcome is often affected by multiple factors jointly. Explaining it with only one variable is often insufficient, and it needs to be extended to multiple linear regression.
  • The linear assumption is too strict: If the true relationship is curved (such as a parabola or an exponential relationship), simple linear regression will produce systematic bias and poor prediction performance.
  • Sensitive to outliers: Least squares minimizes the squared residual sum, meaning that outliers far from the line are penalized in an amplified way, severely distorting the fitting result.
  • Cannot handle categorical variables: When the independent variable is categorical (such as color or gender), it needs to be converted into dummy variables or another model must be used.
  • Many assumptions: Strictly speaking, the excellent properties of least squares estimation depend on assumptions such as linearity, error independence, homoscedasticity, and normality, which real data often cannot fully satisfy.
  • Extrapolation risk: The model is only valid within the range of observed data, and predicting outside the data range (extrapolation) may lead to serious errors.
  • Cannot automatically discover nonlinear patterns: Manual inspection of residual plots is required to determine whether polynomial terms should be introduced or variable transformations should be performed.