Regression fits a line y = b0 + b1*x to data. The least squares method minimizes the sum of squared residuals. R-squared measures the fraction of variance explained. Inference for the slope measures uncertainty about the linear association under the model’s assumptions; it does not establish causation.
How close can your line get?
Each dot is an invented observation. Move the line to make the vertical gaps small. Those gaps are residuals: observed value minus predicted value.
Predict y from x · dashed vertical lines are residuals
Sum of squared residuals: 23.00. Best possible for these points: 4.73.
Adding a point keeps your current line. Predict how the best line will change, then fit again. Axes stay fixed; parts of a trial line outside the plot are clipped.
Read predictions and residuals
x
Observed y
Predicted y
Residual
1
2
2.50
-0.50
2
4
3.00
1.00
3
3
3.50
-0.50
4
5
4.00
1.00
5
5
4.50
0.50
6
8
5.00
3.00
7
7
5.50
1.50
8
9
6.00
3.00
The line you have built
ŷ = b₀ + b₁x. The hat on y marks a prediction. b₀ is the intercept; b₁ is the slope. Your current line is ŷ = 2.00 + (0.50)x. Least squares chooses the coefficients that minimize the sum of squared residuals, ∑(y − ŷ)².
A good fit does not establish causation. Extrapolating beyond observed x values needs additional justification.
A moment to think
Least squares line
The slope b1 = r * (sy / sx) and the intercept b0 = y-bar - b1 * x-bar. These minimize the sum of squared residuals. The correlation r measures linear association; the regression line passes through (x-bar, y-bar).
# Least squares regression
xs = [1, 2, 3, 4, 5, 6, 7]
ys = [52, 58, 65, 70, 74, 80, 85]
n = len(xs)
x_bar = sum(xs) / n
y_bar = sum(ys) / n
ss_xy = sum((x - x_bar) * (y - y_bar) for x, y inzip(xs, ys))
ss_xx = sum((x - x_bar)**2for x in xs)
b1 = ss_xy / ss_xx
b0 = y_bar - b1 * x_bar
print(f"y = {b0:.2f} + {b1:.2f} * x")
print(f"Predicted y at x=5: {b0 + b1 * 5:.2f}")
Residuals
A residual is the observed value minus the predicted value: e = y - y-hat. Residuals should show no pattern when plotted against x or y-hat. Patterns indicate the linear model is missing something: curvature, heteroscedasticity, or outliers.
# Residuals
xs = [1, 2, 3, 4, 5, 6, 7]
ys = [52, 58, 65, 70, 74, 80, 85]
b0, b1 = 46.43, 5.54
residuals = [y - (b0 + b1*x) for x, y inzip(xs, ys)]
print(f"Residuals: {[round(r, 2) for r in residuals]}")
print(f"Sum: {sum(residuals):.4f}")
print(f"SSE: {sum(r**2 for r in residuals):.2f}")
R-squared
R-squared = 1 - SSE/SST, where SST is the total sum of squares. It measures the proportion of variance in y explained by x. An R-squared of 0.95 means the regression captures 95% of the variability. It equals the square of the correlation coefficient r.
To test H0: beta1 = 0 (no linear relationship), compute t = b1 / SE(b1), where SE(b1) = sqrt(MSE / SS_xx). This follows a t-distribution with n - 2 degrees of freedom. The confidence interval is b1 +/- t* * SE(b1).
# Inference for slopeimportmath
xs = [1, 2, 3, 4, 5, 6, 7]
ys = [52, 58, 65, 70, 74, 80, 85]
b0, b1 = 46.43, 5.54
n = len(xs)
residuals = [y - (b0 + b1*x) for x, y inzip(xs, ys)]
mse = sum(r**2for r in residuals) / (n - 2)
x_bar = sum(xs) / n
ss_xx = sum((x - x_bar)**2for x in xs)
se_b1 = math.sqrt(mse / ss_xx)
t = b1 / se_b1
print(f"t = {t:.3f}, SE(b1) = {se_b1:.4f}")
print(f"Reject H0? {'Yes' if abs(t) > 2.571 else 'No'}")
Prediction intervals
A confidence interval targets the mean response at x. A prediction interval targets a single new observation. The prediction interval is wider because it adds individual-level noise on top of estimation uncertainty: SE_pred = sqrt(MSE * (1 + 1/n + (x - x-bar)^2 / SS_xx)).
Simple linear regression