Multiple regression uses several predictors to model a continuous outcome. Logistic regression models a binary outcome using the sigmoid function. Both extend simple regression: one by adding predictors, the other by changing the link function.
Multiple predictors
The model y = b0 + b1*x1 + b2*x2 + ... + bp*xp uses p predictors. Each coefficient bi represents the expected change in y for a one-unit increase in xi, holding all other predictors constant. This "holding constant" interpretation is the difference from running p separate simple regressions.
For least-squares models with an intercept fitted to the same observations, adding predictors cannot decrease R-squared, even when they have no useful predictive signal. Adjusted R-squared penalizes for the number of predictors: R-adj = 1 - (SSE/(n-p-1)) / (SST/(n-1)). It increases only when the reduction in residual error outweighs this penalty; an increase alone does not establish that the predictor will help on new data.
When predictors are correlated with each other, individual coefficients become unstable: large standard errors, sign flips, sensitivity to which observations are included. The variance inflation factor (VIF) quantifies this: VIF = 1 / (1 - R^2_j), where R^2_j is from regressing predictor j on all other predictors. VIF above 5-10 signals trouble.
Scheme
; Variance Inflation Factor; VIF = 1 / (1 - R^2_j); R^2_j = how well predictor j is predicted by other predictors; If x1 and x2 have correlation r = 0.9:; R^2 of x1 ~ x2 is roughly r^2 = 0.81
(define (vif r-squared) (/ 1 (- 1 r-squared)))
(display "Correlation between predictors -> VIF:") (newline)
(display " r = 0.0: VIF = ") (display (vif 0.00)) (newline)
(display " r = 0.5: VIF = ") (display (vif 0.25)) (newline)
(display " r = 0.7: VIF = ") (display (vif 0.49)) (newline)
(display " r = 0.9: VIF = ") (display (vif 0.81)) (newline)
(display " r = 0.95: VIF = ") (display (vif 0.9025)) (newline)
(display "Rule of thumb: VIF > 5 is concerning")
Python
# VIF for different correlations
vif = lambda r_sq: 1 / (1 - r_sq)
for r in [0.0, 0.5, 0.7, 0.9, 0.95]:
print(f"r = {r}: VIF = {vif(r**2):.2f}")
Predicting a yes or a no
Imagine a model for passing a practice test. A straight line could predict a probability below zero or above one. A logistic curve keeps the prediction between those limits.
These coefficients are invented for exploration, not fitted to evidence about studying. The model is predictive; changing a slider does not establish a causal effect.
Model probability of passing
At 4 hours, predicted probability: 31.0%. Odds: 0.45 to 1.
Keep the coefficient fixed and move along the curve. One more hour always adds the same amount to log odds, but its change in probability depends on where you start.
Read the curve as a table
Hours
Probability
0
1.8%
1
3.9%
2
8.3%
3
16.8%
4
31.0%
5
50.0%
6
69.0%
7
83.2%
8
91.7%
9
96.1%
10
98.2%
Compress the curve into a model
log(p / (1 โ p)) = bโ + bโx. Here bโ = โ4 and bโ = 0.8. p / (1 โ p) is odds; log means the natural logarithm. Each extra hour multiplies the odds by ebโ = 2.23.
Equivalently, p = 1 / (1 + eโ(bโ + bโx)). Multiple logistic regression adds more predictors inside the same expression.
A moment to think
Logistic regression
For a binary outcome (0 or 1), linear regression can predict probabilities outside 0-1. Logistic regression fixes this by modeling the log-odds: log(p/(1-p)) = b0 + b1*x. Solving for p gives the sigmoid: p = 1 / (1 + e^-(b0+b1*x)). The coefficients represent changes in log-odds, not in probability.
Scheme
; Logistic regression: the sigmoid function; log(p / (1-p)) = b0 + b1*x; p = 1 / (1 + e^(-(b0 + b1*x)))
(define b0 -4)
(define b1 0.8)
(define (sigmoid z) (/ 1 (+ 1 (exp (- z)))))
(define (predict-prob x) (sigmoid (+ b0 (* b1 x))))
(display "P(y=1) at different x values:") (newline)
(define test-xs (list 02456810))
(for-each (lambda (x)
(display " x = ") (display x)
(display " -> p = ") (display (predict-prob x))
(newline))
test-xs)
; At x=5, the log-odds = b0 + b1*5 = -4 + 4 = 0; So p = 0.5: this is the decision boundary
(display "Decision boundary at x = ")
(display (/ (- b0) b1))
Python
# Logistic regressionimportmath
b0, b1 = -4, 0.8
sigmoid = lambda z: 1 / (1 + math.exp(-z))
predict = lambda x: sigmoid(b0 + b1 * x)
print("P(y=1) at different x values:")
for x in [0, 2, 4, 5, 6, 8, 10]:
print(f" x = {x} -> p = {predict(x):.4f}")
print(f"Decision boundary at x = {-b0/b1}")
Odds ratios
The odds of an event is p/(1-p). In logistic regression, e^b1 is the odds ratio: the factor by which the odds multiply for each one-unit increase in x. An odds ratio of 2.23 means the odds more than double per unit increase. This is the natural scale for interpreting logistic coefficients.
Scheme
; Odds ratios; If b1 = 0.8, then odds ratio = e^0.8
(define b1 0.8)
(define odds-ratio (exp b1))
(display "b1 = ") (display b1) (newline)
(display "Odds ratio = e^b1 = ") (display odds-ratio) (newline)
(display "Interpretation: each unit increase in x") (newline)
(display " multiplies the odds by ") (display odds-ratio) (newline)
; Example: if baseline odds = 1:3 (p=0.25)
(define baseline-odds (/ 13))
(display "Baseline odds: ") (display (* 1.0 baseline-odds)) (newline)
(define new-odds (* baseline-odds odds-ratio))
(display "After x+1: ") (display new-odds) (newline)
; Convert odds to probability
(define (odds->prob odds) (/ odds (+ 1 odds)))
(display "Baseline prob: ") (display (odds->prob baseline-odds)) (newline)
(display "New prob: ") (display (odds->prob new-odds))
Multiple linear regression