Advanced Statistical Data Analysis

Review of Multiple Linear Regression¶

Initial Remarks¶

Regression analysis is used to model the relationship between a response variable $Y$ and one or more explanatory variables $x^{(1)}, \dots, x^{(m)}$ , where the relationship is masked by random noise.

Objectives of Regression Analysis¶

General description of data structure.
Assessment of the effect of explanatory variables on the response.
Prediction of future observations.

Error Assumptions¶

The standard assumptions for the error terms $\mathcal{E}_i$ are:

Stochastically independent.
Expectation zero and constant variance $\sigma^2$ (homoscedasticity).
Normally (Gaussian) distributed: $\mathcal{E}_i \sim \mathcal{N}(0, \sigma^2)$ .

Matrix Representation¶

To simplify notation, the regression equation Definition 1 is written in matrix form:

\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\mathcal{E}}

(2)

where:

$\mathbf{Y}$ is an $n \times 1$ vector of responses.
$\mathbf{X}$ is an $n \times p$ matrix of explanatory variables (including a column of 1s for the intercept).
$\boldsymbol{\beta}$ is a $p \times 1$ vector of unknown coefficients ( $p = m+1$ ).
$\boldsymbol{\mathcal{E}}$ is an $n \times 1$ vector of unobserved random variables.

Tukey’s First-Aid Transformations¶

Standard recommendations used to linearize relationships and stabilize variance when there is no specific domain theory to guide variable transformation. These should be applied to both explanatory variables and responses unless a valid reason exists to do otherwise:

Data Type	Recommended Transformation
Concentrations and Amounts	$\log(x)$
Count Data	$\sqrt{x}$
Counted Fractions / Shares	$\tilde{x} = \text{logit}(x) = \log\left(\frac{x + 0.005}{1.01 - x}\right)$

Model Fitting and Diagnostics¶

Least Squares Estimation¶

The coefficients $\boldsymbol{\beta}$ are estimated by minimizing the sum of squared residuals.

The OLS estimator is given by:

\hat{\boldsymbol{\beta}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{Y}

(3)

Model Adequacy (Residual Analysis)¶

Model adequacy is checked using diagnostic plots:

Tukey-Anscombe Plot: Residuals vs. Fitted values to check for non-linearity or heteroscedasticity.
Normal Q-Q Plot: To check the normality assumption of errors.
Scale-Location Plot: To check for constant variance.
Residuals vs. Leverage: To identify influential observations (Cook’s Distance).