Predictive Modeling

Exam Strategies & Common Pitfalls¶

Common Pitfalls to Avoid¶

Linear Regression:

• ✗ Confusing prediction interval with confidence interval

• ✗ Using $R^2$ to compare models with different sample sizes

• ✗ Ignoring collinearity when standard errors are large

• ✗ Including interaction without main effects

• ✗ Forgetting to check residual plots

Logistic Regression:

• ✗ Interpreting $\beta_j$ as change in probability (it’s log-odds!)

• ✗ Confusing odds with probability

• ✗ Trusting high accuracy with class imbalance

• ✗ Using wrong metrics (accuracy vs F1/recall/precision)

Model Selection:

• ✗ Using forward selection as if it’s optimal (it’s greedy!)

• ✗ Confusing minimizing AIC with maximizing (BIC same)

• ✗ Forgetting to standardize before ridge/lasso

Time Series:

• ✗ Fitting ARMA to non-stationary series (must difference first!)

• ✗ Assuming MA implies non-stationary (MA is always stationary)

• ✗ Confusing ACF cutoff (MA) with PACF cutoff (AR)

• ✗ Not checking if roots satisfy stationarity condition

• ✗ Forgetting that forecasts converge to mean (long-term)

Hypothesis Testing:

• ✗ Confusing failing to reject $H_0$ with accepting $H_0$

• ✗ Using t-test when F-test is needed (multiple coefficients)

• ✗ Not reporting degrees of freedom for test statistics

• ✗ Misinterpreting p-value as probability $H_0$ is true

Calculations:

• ✗ Arithmetic errors (double-check!)

• ✗ Wrong degrees of freedom ($n-p-1$ vs $n-1$ vs $n-k$)

• ✗ Forgetting square root in RMSE or SE formulas

• ✗ Sign errors (especially with differencing)

Key Concepts Summary¶

When to Use Each Method:

Stationarity?

• Plot series, check ACF

• If trend/changing variance → transform or difference

• If seasonal → seasonal differencing or STL

AR vs MA vs ARMA?

• ACF cuts off, PACF tails → MA($q$)

• PACF cuts off, ACF tails → AR($p$)

• Both tail off → ARMA($p,q$)

Which regularization?

• Want variable selection → Lasso

• Many correlated predictors, all important → Ridge

• Not sure → Try both, compare CV error

Which interval?

• Predict mean/average → Confidence interval

• Predict single observation → Prediction interval

Key Things to Remember¶

• $R^2$ increases with more predictors, use adj-$R^2$

• F-test for overall significance, t-test for individual coefficients

• VIF $> 10$ suggests collinearity

• Cook’s $D > 0.5$ suggests influential point

• Logistic: coefficients are log-odds, not probabilities

• MA is always stationary, AR requires $|$roots$| > 1$

• ACF for MA, PACF for AR

• Difference before fitting ARIMA to non-stationary series

• Always standardize before ridge/lasso

• F1 score for imbalanced classification

Essential Formulas & Key Calculations¶

Regression Formulas¶

$R^2$: $R^2 = 1 - \dfrac{\text{RSS}}{\text{TSS}} = 1 - \dfrac{\sum (y_i - \hat y_i)^2}{\sum (y_i - \bar y)^2}$

Adjusted $R^2$: $\text{adj-}R^2 = 1 - \left[\dfrac{\text{RSS}/(n-p-1)}{\text{TSS}/(n-1)}\right]$

F-statistic: $F = \dfrac{(\text{TSS} - \text{RSS})/p}{\text{RSS}/(n-p-1)}$

Confidence interval for $\beta_j$: $\hat\beta_j \pm t_{\alpha/2,\,n-p-1} \times \operatorname{SE}(\hat\beta_j)$

VIF: $\text{VIF}_j = \dfrac{1}{1 - R_j^2}$

Saturated Model¶

Problem: $n$ observations, $n$ parameters

• Perfect fit: $\text{RSS} = 0$, all residuals $= 0$

• Estimate of $\sigma^2 = 0$

• Interpretation: Complete overfitting, no generalization

Logistic Regression Calculations¶

From $\beta$ to probability:

• Linear predictor: $\eta = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$

• Probability: $p = \dfrac{e^\eta}{1 + e^\eta} = \dfrac{1}{1 + e^{-\eta}}$

• Odds: $\text{odds} = e^\eta$

• Log-odds: $\log(\text{odds}) = \eta$

From probability to odds:

• $\text{odds} = p/(1-p)$

• Example: $p = 0.2 \Rightarrow \text{odds} = 0.2/0.8 = 0.25$ (or 1:4)

• Example: $p = 0.8 \Rightarrow \text{odds} = 0.8/0.2 = 4$ (or 4:1)

From odds to probability:

• $p = \text{odds}/(1 + \text{odds})$

• Example: $\text{odds} = 4 \Rightarrow p = 4/(1+4) = 0.8$

Log-odds (logit):

• $\operatorname{logit}(p) = \log\left(\dfrac{p}{1-p}\right)$

• Example: $p = 0.5 \Rightarrow \operatorname{logit} = \log(1) = 0$

AR Process Variance¶

AR(1): $X_n = a X_{n-1} + W_n$, $|a| < 1$

• Take variance: $\operatorname{Var}(X_n) = a^2 \operatorname{Var}(X_n) + \sigma^2$

• Solve: $\operatorname{Var}(X_n) = \sigma^2/(1 - a^2)$

Stationary mean:

• Take expectation: $E(X_n) = a E(X_n)$

• Solve: $E(X_n)(1 - a) = 0 \Rightarrow E(X_n) = 0$

Characteristic Polynomial Check¶

AR(2) Example: $X_n = 1.5 X_{n-1} - 0.9 X_{n-2} + W_n$

Characteristic polynomial: $\varphi(z) = 1 - 1.5z + 0.9 z^2 = 0$

Solve for roots: $z = \dfrac{1.5 \pm \sqrt{1.5^2 - 4(0.9)(1)}}{2 \cdot 0.9}$ $z = \dfrac{1.5 \pm \sqrt{-1.35}}{1.8}$

Check: If both $|z| > 1$ → stationary

Alternative: Rewrite as polynomial in $z^{-1}$ and check roots inside unit circle

MA(1) Autocovariance¶

Model: $X_n = W_n + b W_{n-1}$

Variance: $\gamma(0) = \operatorname{Var}(X_n) = \operatorname{Var}(W_n + b W_{n-1}) = \sigma^2 + b^2 \sigma^2 = (1 + b^2)\sigma^2$

Lag-1 Covariance: $\gamma(1) = \operatorname{Cov}(X_n, X_{n+1}) = \operatorname{Cov}(W_n + b W_{n-1}, W_{n+1} + b W_n)$ $\quad = b \cdot \operatorname{Var}(W_n) = b \sigma^2$

Lag-$h$ Covariance ($h \ge 2$): $\gamma(h) = 0$ (no overlapping terms)

ACF:

• $\rho(0) = 1$

• $\rho(1) = \gamma(1)/\gamma(0) = \dfrac{b \sigma^2}{(1 + b^2)\sigma^2} = \dfrac{b}{1 + b^2}$

• $\rho(h) = 0$ for $h \ge 2$

Linear Regression Fundamentals¶

Model Assumptions¶

The linear regression model assumes:

1. Linearity: $E(Y \mid X) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p$

2. Independence: errors $\varepsilon_i$ are independent

3. Homoscedasticity: $\operatorname{Var}(\varepsilon_i) = \sigma^2$ (constant variance)

4. Normality: $\varepsilon_i \sim N(0, \sigma^2)$ (needed for inference)

Check via residual plots!

Simple Linear Regression¶

Model: $Y = \beta_0 + \beta_1 X + \varepsilon$

• $\beta_0$: intercept (value of $Y$ when $X = 0$)

• $\beta_1$: slope (change in $Y$ for unit change in $X$)

• $\varepsilon$: error term with $E(\varepsilon) = 0$, $\operatorname{Var}(\varepsilon) = \sigma^2$

Least Squares Estimation: Minimize $\text{RSS} = \sum (y_i - \hat y_i)^2$ to find $\hat\beta_0$ and $\hat\beta_1$

Key Quantities:

• TSS (Total Sum of Squares) $= \sum (y_i - \bar y)^2$ (total variance in $Y$)

• RSS (Residual Sum of Squares) $= \sum (y_i - \hat y_i)^2$ (unexplained variance)

• $R^2$ $= 1 - \text{RSS}/\text{TSS}$ (fraction of variance explained, $0 \le R^2 \le 1$)

• RSE $= \sqrt{\text{RSS}/(n - p - 1)}$ (residual standard error, estimate of $\sigma$)

Multiple Linear Regression¶

Model: $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \varepsilon$

Hypothesis Testing:

Individual t-test: Test $H_0: \beta_j = 0$ vs $H_1: \beta_j \ne 0$

• Test statistic: $t = \hat\beta_j / \operatorname{SE}(\hat\beta_j) \sim t_{n-p-1}$

• Reject $H_0$ if $|t| > t_{1 - \alpha/2, df=n-p-1}$ or if p-value $< \alpha$

  • if $P > |t|$ is less than $\alpha$ (i.e. 0.05): The parameter is significant, reject $H_0 := \beta_j = 0$

  • else if $P > |t|$ is greater than or equal $\alpha$ (i.e. 0.05): Parameter not significant, retain $H_0 := \beta_j = 0$

Global F-test: Test $H_0: \beta_1 = \beta_2 = \dots = \beta_p = 0$ (no predictors useful)

• $F = \dfrac{(\text{TSS} - \text{RSS})/p}{\text{RSS}/(n-p-1)} \sim F_{p,\,n-p-1}$

• If $H_0$ true, expect $F \approx 1$

• If $H_1$ true (at least one $\beta_j \ne 0$), expect $F > 1$

• Large $F$ and small p-value → reject $H_0$

Partial F-test: Test if subset of $q$ coefficients are zero

• $H_0: \beta_{p-q+1} = \dots = \beta_p = 0$ ($q$ coefficients)

• $F = \dfrac{(\text{RSS}_0 - \text{RSS})/q}{\text{RSS}/(n-p-1)} \sim F_{q,\,n-p-1}$

• $\text{RSS}_0$: smaller model (without $q$ predictors)

• RSS: larger model (with all predictors)

Important Insight: If all individual t-tests fail to reject but global F-test is significant → collinearity problem!

Confidence vs. Prediction Intervals¶

Confidence Interval: For mean response $E(Y \mid X = x_0)$

• Estimates average $Y$ for given $X = x_0$

• Formula: $\hat y \pm t_{1 - \alpha/2, df=n-p-1} \times \operatorname{SE}(\hat y)$

• Narrower interval

Prediction Interval: For individual response $Y \mid X = x_0$

• Predicts single observation $Y$ for given $X = x_0$

• Formula: $\hat y \pm t_{1 - \alpha/2, df=n-p-1} \times \sqrt{\hat\sigma^2 + \operatorname{SE}^2(\hat y)}$

• Wider interval (includes both model uncertainty and $\sigma^2$)

Which to use?

• Mayor wants prognosis for his town → Prediction interval (single observation)

• Researcher wants average effect → Confidence interval (mean)

Residual Analysis & Diagnostics¶

Residual Plots¶

Tukey-Anscombe Plot (Residuals vs Fitted Values):

• What to look for: Random scatter around zero

• Patterns indicate problems:

  • Curve/non-linear pattern → non-linearity (try polynomial or transformation)

  • Funnel shape → heteroscedasticity

  • Outliers → investigate data quality

Scale-Location Plot ($\sqrt{\lvert\text{standardized residuals}\rvert}$ vs fitted):

• Purpose: Check homoscedasticity (constant variance)

• Good: Horizontal line with random scatter

• Bad: Increasing/decreasing trend → heteroscedasticity

Normal Q-Q Plot:

• Purpose: Check normality of residuals

• Good: Points follow diagonal line

• Bad: Deviations at tails → heavy-tailed/skewed distribution

Residuals vs Leverage:

• Purpose: Identify influential observations

• Shows Cook’s distance contours

• Points outside Cook’s $D = 0.5$ or 1.0 are influential

Influential Points¶

Leverage ($h_i$):

• Measures how far predictor values $x_i$ are from $\bar x$

• High leverage: $h_i > 2(p+1)/n$

• High leverage doesn’t mean influential!

Cook’s Distance ($D_i$):

• Measures influence of observation $i$ on fitted values

• Combines leverage and residual size

• Problematic: $D_i > 0.5$ or $D_i > 1$

• Influential points have high Cook’s $D$

Standardized Residuals:

• $r_i = \dfrac{e_i}{\hat\sigma \sqrt{1-h_i}}$

• Outlier if: $\lvert r_i\rvert > 2$ (or $> 3$ for stricter criterion)

Decision Tree:

• High leverage + large residual → influential (investigate!)

• High leverage + small residual → not influential (okay)

• Low leverage + large residual → outlier but not influential

• Low leverage + small residual → typical observation

Collinearity¶

Definition: Predictor variables are highly correlated

Variance Inflation Factor (VIF):

• $\text{VIF}_j = \dfrac{1}{1 - R_j^2}$

• $R_j^2$ from regression of $X_j$ on all other predictors

Interpretation:

• $\text{VIF} = 1$: no collinearity

• $\text{VIF} > 5$: moderate collinearity

• $\text{VIF} > 10$: severe collinearity (problematic!)

Effects of Collinearity:

• Inflates standard errors of coefficients

• Makes coefficients unstable (sensitive to data changes)

• Reduces statistical power (harder to reject $H_0$)

• Difficult to interpret individual coefficients

• Does NOT reduce $R^2$ or predictive accuracy!

Solutions:

• Remove correlated predictors

• Combine correlated predictors (e.g., average)

• Use ridge regression or lasso

• Increase sample size

• Variable transformation (e.g., center variables)

Qualitative Predictors & Interactions¶

Dummy Variables (Categorical Predictors)¶

Two Levels (e.g., Gender: Male/Female):

• Create one dummy variable: $X = 1$ if Female, $X = 0$ if Male

• Model: $Y = \beta_0 + \beta_1 X + \varepsilon$

• Interpretation:

  • $\beta_0$: mean for males (baseline)

  • $\beta_0 + \beta_1$: mean for females

  • $\beta_1$: difference between females and males

• Test $H_0: \beta_1 = 0$ tests if gender matters

Three Levels (e.g., Ethnicity: Asian, Caucasian, African American):

• Create two dummy variables ($k$ levels → $k-1$ dummies):

  • $X_1 = 1$ if Asian, 0 otherwise

  • $X_2 = 1$ if Caucasian, 0 otherwise

  • African American is baseline (both $X_1 = 0$, $X_2 = 0$)

• Model: $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \varepsilon$

• Interpretation:

  • $\beta_0$: mean for African Americans (baseline)

  • $\beta_0 + \beta_1$: mean for Asians

  • $\beta_0 + \beta_2$: mean for Caucasians

  • $\beta_1$: difference between Asian and African American

  • $\beta_2$: difference between Caucasian and African American

Key Rule: For $k$ categories, need $k-1$ dummy variables (one category is baseline)

Baseline Choice: Arbitrary, doesn’t affect model fit, only interpretation

Interaction Effects¶

Model with Interaction:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_1 X_2 + \varepsilon$

When $\beta_3 \ne 0$: Effect of $X_1$ on $Y$ depends on value of $X_2$ (synergy/interaction)

Rewrite to see interaction: $Y = (\beta_0 + \beta_2 X_2) + (\beta_1 + \beta_3 X_2) X_1$

• Intercept for $X_1$: $\beta_0 + \beta_2 X_2$ (changes with $X_2$)

• Slope for $X_1$: $\beta_1 + \beta_3 X_2$ (changes with $X_2$)

Example (Advertising): $\text{Sales} = \beta_0 + \beta_1 \cdot \text{TV} + \beta_2 \cdot \text{Radio} + \beta_3 \cdot (\text{TV} \times \text{Radio})$

• $\beta_3 > 0$: spending on TV increases effectiveness of Radio (and vice versa)

• Interaction means: 50/50 split between TV and Radio is better than all on one

Hierarchical Principle: If interaction $X_1 X_2$ is significant, must include both main effects $X_1$ and $X_2$ in model, even if they’re not individually significant

Polynomial Regression¶

Model: $Y = \beta_0 + \beta_1 X + \beta_2 X^2 + \dots + \beta_p X^p + \varepsilon$

Key Points:

• Used when relationship between $X$ and $Y$ is non-linear

• Still a linear model (linear in parameters $\beta$!)

• Can fit using standard least squares

• Degree $p$ determined by:

  • Residual plots (check if pattern remains)

  • Cross-validation

  • Hypothesis tests (is $\beta_p$ significant?)

Example: $\text{mpg} = \beta_0 + \beta_1 \cdot \text{horsepower} + \beta_2 \cdot \text{horsepower}^2$ captures non-linear relationship

Warning: High-degree polynomials can overfit!

Model Selection¶

Variable Selection Methods¶

Best Subset Selection:

• Fit all $2^p$ possible models (with $0,1,2,\dots,p$ predictors)

• For each size $k$, choose best model (lowest RSS or highest $R^2$)

• Select single best model using criterion (AIC, BIC, adj-$R^2$, $C_p$)

• Problem: Computationally infeasible for $p > 40$

BIC and AIC Scorers for Linear Regression Model Selection

Forward Stepwise Selection:

1. Start with $M_0$ (null model: only intercept)

2. For $k = 0,1,\dots,p-1$:

   • Add one predictor that most improves fit (lowest RSS)

   • Result: $M_{k+1}$

3. Select single best model from $M_0, M_1, \dots, M_p$ using criterion

• Advantage: Only $p(p+1)/2 + 1$ models (computationally efficient)

• Greedy algorithm (may miss best overall model)

Backward Stepwise Selection:

1. Start with $M_p$ (full model: all predictors)

2. For $k = p,p-1,\dots,1$:

   • Remove predictor whose removal least hurts fit

   • Result: $M_{k-1}$

3. Select single best model using criterion

• Advantage: Only $p(p+1)/2 + 1$ models

• Limitation: Requires $n > p$

Hybrid Stepwise Selection:

• Alternates between adding and removing variables

• Can correct for earlier mistakes

• Similar computational efficiency to forward/backward

Model Selection Criteria¶

Goal: Balance fit vs. complexity (avoid overfitting)

Adjusted $R^2$ → Select model that maximizes adj-$R^2$:

• Formula: $\text{adj-}R^2 = 1 - \dfrac{\text{RSS}/(n-p-1)}{\text{TSS}/(n-1)}$

• Regular $R^2$ always increases with more predictors

• Adjusted $R^2$ penalizes adding predictors

AIC (Akaike Information Criterion) → Select model that minimizes AIC:

• For least squares: $\text{AIC} = \dfrac{1}{n \hat\sigma^2} \text{RSS} + 2(p+1)$

• General form: $\text{AIC} = -2 \log(L) + 2q$

  • $L$: likelihood, $q$: number of parameters

• Penalty term: $2(p+1)$ increases with more predictors

BIC (Bayesian Information Criterion) → Select model that minimizes BIC:

• Tends to select smaller models than AIC

• For least squares: $\text{BIC} = \dfrac{1}{n \hat\sigma^2} \text{RSS} + \log(n)(p+1)$

• General form: $\text{BIC} = -2 \log(L) + \log(n) q$

• Penalty term: $\log(n)(p+1)$ stronger than AIC for $n > 7$

Mallows’ $C_p$ → Select model that minimizes $C_p$:

• Formula: $C_{p} = \frac{1}{n} \left( \mathrm{RSS} + 2p\hat{\sigma}^{2} \right)$

• Proportional to AIC for least squares

Key Insight:

• All criteria balance fit (RSS) vs. complexity ($p$)

• AIC: prediction-focused

• BIC: explanation-focused, stronger penalty → simpler models

• For large $n$, BIC ≈ AIC with stronger penalty

Regularization Methods¶

Ridge Regression¶

Objective Function: Minimize $\text{RSS} + \lambda \sum_{j=1}^p \beta_j^2$

Components:

• RSS: want good fit

• $\lambda \sum \beta_j^2$: $L_2$ penalty (shrinkage penalty)

• $\lambda \ge 0$: tuning parameter

Behavior:

• $\lambda = 0$ → ordinary least squares (no penalty)

• $\lambda \to \infty$ → all $\beta_j \to 0$ (maximum shrinkage)

• Intermediate $\lambda$: balance fit and shrinkage

Properties:

• Shrinks coefficients toward zero

• Does NOT set coefficients exactly to zero

• All $p$ predictors remain in final model

• Reduces variance at cost of increased bias

• Helps with collinearity (stabilizes coefficients)

Equivalent Constrained Form: Minimize RSS subject to $\sum \beta_j^2 \le s$

• Larger $s$ (or smaller $\lambda$) → less shrinkage

Why standardize?

• Ridge penalty depends on scale of predictors

• Always standardize predictors before applying ridge

Lasso Regression¶

Objective Function: Minimize $\text{RSS} + \lambda \sum_{j=1}^p |\beta_j|$

Components:

• RSS: want good fit

• $\lambda \sum |\beta_j|$: $L_1$ penalty

• $\lambda \ge 0$: tuning parameter

Properties:

• Shrinks coefficients toward zero

• Sets some coefficients exactly to zero (automatic variable selection!)

• Produces sparse models (only subset of predictors)

• More interpretable than ridge

• Also helps with collinearity

Equivalent Constrained Form: Minimize RSS subject to $\sum |\beta_j| \le s$

Why Lasso Produces Sparse Solutions:

• $L_1$ constraint region has corners (diamond shape in 2D)

• RSS contours (ellipses) often intersect constraint at corners

• At corners, some coefficients are exactly zero

• Ridge has circular constraint (no corners) → no exact zeros

Comparing Ridge and Lasso¶

Bias-Variance Trade-Off:

• Both reduce variance by introducing bias

• As $\lambda$ increases: variance ↓, bias ↑

• Optimal $\lambda$ minimizes test MSE

Selecting Tuning Parameter $\lambda$¶

Method: Cross-Validation

1. Choose grid of $\lambda$ values (e.g., 10^-2 to 10⁶)

2. For each $\lambda$:

   • Perform $k$-fold CV

   • Compute CV error (MSE for regression)

3. Select $\lambda$ that minimizes CV error

4. Refit model on full training data with chosen $\lambda$

Typical Result:

• $\lambda$ too small → overfitting (high variance, acts like OLS)

• $\lambda$ too large → underfitting (high bias, coefficients too small)

• Optimal $\lambda$ in middle

Logistic Regression¶

Simple Logistic Regression¶

Goal: Model probability $P(Y = 1 \mid X)$ where $Y$ is binary (0 or 1)

Logistic Function: $p(x) = P(Y = 1 \mid X = x) = \dfrac{e^{\beta_0 + \beta_1 x}}{1 + e^{\beta_0 + \beta_1 x}}$

Properties:

• Output always between 0 and 1 (valid probability)

• S-shaped curve

• Linear relationship in log-odds scale

Odds: $\operatorname{odds}(x) = \dfrac{p(x)}{1 - p(x)} = e^{\beta_0 + \beta_1 x}$

• Odds = 1: equal probability (50-50)

• Odds $> 1$: more likely is $p(x)$

• Odds $< 1$: more likely is $1 - p(x)$

Log-Odds (Logit): $\log\left(\dfrac{p(x)}{1 - p(x)}\right) = \beta_0 + \beta_1 x$

• Linear in $x$!

• This is why it’s called “logistic regression”

Interpretation of Coefficients:

• $\beta_1$: change in log-odds for 1-unit increase in $X$

• Unit increase in $X$ → odds multiply by $e^{\beta_1}$

• $\beta_1 > 0$: positive association ($X \uparrow \Rightarrow P(Y=1) \uparrow$)

• $\beta_1 < 0$: negative association ($X \uparrow \Rightarrow P(Y=1) \downarrow$)

Estimation:

• Use Maximum Likelihood (not least squares!)

• No closed-form solution, requires numerical optimization

Example Calculation: If $\beta_0 = -10.65$, $\beta_1 = 0.0055$, $X = 1000$:

• log-odds $= -10.65 + 0.0055 \cdot 1000 = -5.15$

• odds $= e^{-5.15} \approx 0.0058$

• $p = 0.0058/(1 + 0.0058) \approx 0.0058$

Multiple Logistic Regression¶

Model: $P(Y = 1 \mid X_1, \dots, X_p) = \dfrac{e^{\beta_0 + \beta_1 X_1 + \dots + \beta_p X_p}}{1 + e^{\beta_0 + \beta_1 X_1 + \dots + \beta_p X_p}}$

Log-odds: $\log\left(\dfrac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p$

Interpretation:

• $\beta_j$: change in log-odds when $X_j$ increases by 1, holding other $X_j$ constant

• Holding others constant, odds multiply by $e^{\beta_j}$

Classification & Model Evaluation¶

Classification Rule:

• If $p(x) \ge$ threshold (typically 0.5) → predict $Y = 1$

• If $p(x) <$ threshold → predict $Y = 0$

Confusion Matrix:

                Predicted Class
              0           1
Actual  0    TN          FP    (Total N)
        1    FN          TP    (Total P)

• TP (True Positive): correctly predicted 1

• TN (True Negative): correctly predicted 0

• FP (False Positive): predicted 1, actually 0 (Type I error)

• FN (False Negative): predicted 0, actually 1 (Type II error)

Performance Metrics:

Accuracy $= \dfrac{\text{TP} + \text{TN}}{\text{Total}}$

• Proportion of correct predictions

• Problem: Misleading with class imbalance!

Precision $= \dfrac{\text{TP}}{\text{TP} + \text{FP}}$

• Among predicted positives, what fraction are correct?

• High precision: few false alarms

Recall (Sensitivity, True Positive Rate) $= \dfrac{\text{TP}}{\text{TP} + \text{FN}}$

• Among actual positives, what fraction did we detect?

• High recall: catch most positives

Specificity (True Negative Rate) $= \dfrac{\text{TN}}{\text{TN} + \text{FP}}$

• Among actual negatives, what fraction did we correctly identify?

F1 Score $= 2 \times \dfrac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

• Harmonic mean of precision and recall

• Balances both metrics

• Good for imbalanced classes

False Positive Rate $= \dfrac{\text{FP}}{\text{FP} + \text{TN}} = 1 - \text{Specificity}$

Classification Error $= \dfrac{\text{FP} + \text{FN}}{\text{Total}} = 1 - \text{Accuracy}$

Receiver Operating Characteristic (ROC) Curve¶

Receiver Operating Characteristic (ROC) Curve:

• Plot True Positive Rate (Recall) vs False Positive Rate

• Each point corresponds to different threshold

• Ideal: Curve hugs top-left corner (high TPR, low FPR)

• Baseline: Diagonal line (random classifier)

Area Under Curve (AUC):

• AUC = 1: perfect classifier

• AUC = 0.5: random classifier

• AUC $> 0.8$: good classifier

Using ROC to Choose Threshold:

• High threshold → low FPR, low TPR (conservative)

• Low threshold → high FPR, high TPR (liberal)

• Choose threshold based on cost of FP vs FN

Class Imbalance Problem¶

Problem Example:

• Dataset: 97% class 0, 3% class 1

• Trivial classifier: always predict 0

• Accuracy = 97% (looks great!)

• But useless: never detects class 1

Why Imbalance is Bad:

• Likelihood function dominated by majority class

• Model learns to predict majority class

• Minority class ignored

Solutions:

1. Down-sampling: Randomly remove observations from majority class

   • Balanced classes

   • Loses data

2. Up-sampling: Replicate observations from minority class

   • Balanced classes

   • Can overfit

3. Adjust Threshold: Lower threshold to catch more positives

   • Simple

   • Trade-off: more false positives

4. Use Appropriate Metrics: Focus on F1, precision, recall (not accuracy)

5. Cost-Sensitive Learning: Weight observations by class

Cross-Validation for Classification¶

$k$-Fold Cross-Validation / $k$-Fold CV:

Split data into test data and training data, then split training data into k folds. Estimate the classification score by averaging across folds.

1. Split data into $k$ folds (typically $k = 5$ or 10)

2. For $i = 1$ to $k$:

   • Train on $k-1$ folds

   • Test on fold $i$

   • Compute classification error on fold $i$: $\text{Err}_i$

3. CV error $= \dfrac{1}{k} \sum \text{Err}_i$

Leave-One-Out Cross-Validation (LOOCV): $k = n$

• Less bias, more variance

• Computationally expensive for large datasets (trains $n$ models)

Purpose:

• Estimate test error

• Choose between models

• Tune hyperparameters (e.g., threshold)

Important: Stratify folds to preserve class proportions!

Time Series: Introduction & Decomposition¶

Fundamental Concepts¶

Time Series: Observations $x_1, x_2, \dots, x_n$ ordered in time

Stochastic Process: Sequence of random variables $X_1, X_2, X_3, \dots$

• Time series is one realization (sample path) of process

• Like drawing one sample from a distribution

Serial Correlation: Observations are dependent over time

• Violates independence assumption of standard regression

• Requires special methods

Goals:

1. Describe and visualize data

2. Model underlying process

3. Decompose into components (trend, seasonality, noise)

4. Forecast future values

5. Regression with time series predictors

Basic Stochastic Processes¶

White Noise:

• $W_1, W_2, \dots$ are i.i.d. with $E(W_i) = 0$, $\operatorname{Var}(W_i) = \sigma^2$

• No correlation: $\operatorname{Cov}(W_i, W_j) = 0$ for $i \ne j$

• Completely unpredictable

• ACF: $\rho(h) = 0$ for $h > 0$

Random Walk:

• $X_n = X_{n-1} + W_n$, starting $X_0 = 0$

• Cumulative sum: $X_n = W_1 + W_2 + \dots + W_n$

• With drift: $X_n = X_{n-1} + \delta + W_n$ (adds trend $\delta$)

• Non-stationary (variance grows with time)

• $E(X_n) = n\delta$ (if drift), $\operatorname{Var}(X_n) = n \sigma^2$

Moving Average (MA):

• Example (3-point): $X_n = (W_{n-1} + W_n + W_{n+1}) / 3$

• Smooths white noise

• Reduces high-frequency fluctuations

• Stationary

Autoregressive (AR):

• Example (AR(2)): $X_n = a_1 X_{n-1} + a_2 X_{n-2} + W_n$

• Current value depends on past values

• Can exhibit oscillations, trends (if non-stationary)

• Captures dependence structure

Stationarity¶

Strict Stationarity:

• Joint distribution unchanged by time shifts

• For any $h$, $(X_1, X_2, \dots, X_k)$ and $(X_{1+h}, X_{2+h}, \dots, X_{k+h})$ have same distribution

• Too strong for practice

Weak Stationarity (Second-Order Stationarity): Two conditions:

1. Constant mean: $E(X_n) = \mu$ for all $n$

2. Autocovariance depends only on lag:

   • $\operatorname{Cov}(X_n, X_{n+h}) = \gamma(h)$ for all $n$

   • Doesn’t depend on absolute time $n$, only lag $h$

Why Weak Stationarity Matters:

• Allows statistical inference from single time series

• Can estimate autocovariance from one realization

• Most models assume weak stationarity

Non-Stationary Indicators:

• Trend (increasing/decreasing mean)

• Changing variance over time

• Seasonality (periodic pattern)

• ACF decays very slowly

Stationarity for a Time Series requires three conditions:

1. Constant mean: $\mathbb{E}[X_n] = \mu$ (independent of $n$)

2. Constant variance: $\text{Var}(X_n) = \gamma_0$ (independent of $n$)

3. Autocovariance depends only on lag: $\text{Cov}(X_n, X_{n-k}) = \gamma_k$ (independent of $n$)

For ARMA processes, a simpler criterion exists: the characteristic equation (or equivalently, checking the roots of the characteristic polynomial).

Proof Method 1: Characteristic Equation (Most Direct)¶

Convention 1: Rewrite the AR(2) process in lag operator notation:

Rearrange to Standard Form (move all $X$ terms to the left side):

The characteristic equation is obtained by replacing $X_{n-k}$ with $z^{-k}$:

Multiply by $z^2$:

This is the standard form for the quadratic formula: $az^2 + bz + c = 0$,

for which: $a = 1$, $b = -\frac{1}{2}$, $c = -\frac{1}{3}$.

Both roots satisfy $|z| < 1$ (they lie inside the unit circle). Since both roots of the characteristic polynomial lie strictly inside the unit circle, the AR(2) process is stationary.

Convention 2: Rewrite the AR(2) process into its Characteristic Polynomial:

Rearrange to Standard Form (move all $X$ terms to the left side):

Introduce the Lag Operator (Backshift Operator):

Define the lag operator $L$ such that: $LX_n = X_{n-1}$, $L^2X_n = X_{n-2}$, $L^3X_n = X_{n-3}$ and rewrite the equation using $L$:

Factor out $X_n$ on the left:

This is the lag polynomial form: $\Phi(L)X_n = W_n$

where $\Phi(L) = 1 - 0.5L + 0.5L^2 + 0.1L^3$

Direct Polynomial Substitution: Replace $L$ with $x$, set equal to zero and rearrange to standard form (coefficients in descending order of power):

Stationarity condition: All roots satisfy $|x| > 1$

Solve Manually for roots using the quadratic formula: