Reweighted Estimation: GLMs, Expectiles & Robust M-Estimators

Navigation:

Every non-Gaussian scalar target is fitted by Iteratively Reweighted Penalized Least Squares (IRLS) [Nelder and Wedderburn, 1972]: repeat the P-WLS atom with a working response \(z\) and weights \(W\) that are refreshed from the current linear predictor \(\eta=\Phi\theta\) until convergence.

Exponential-family GLMs (P-IRLS)

For a link \(g\) (\(\eta=g(\mu)\)), variance \(V(\mu)\) and mean \(\mu=g^{-1}(\eta)\), the Penalized Iteratively Reweighted Least Squares (P-IRLS) updates [Wood, 2017, Nelder and Wedderburn, 1972, Green, 1984] are:

\[ z = \eta + (y-\mu)\,g'(\mu), \qquad w = \frac{1}{g'(\mu)^2\,V(\mu)}. \]

Supported families: Gaussian (identity), Poisson & Gamma (log link), Binomial (logit). Step-halving on the penalized deviance guarantees monotone descent [Wood, 2017].

Robust M-estimators

\(z=y\); the loss enters through a bounded-influence weight [Huber, 1964] against a MAD robust scale \(s\):

\[ \text{Huber: } w=\min\!\left(1,\ \tfrac{\delta}{|u|/s}\right), \qquad \text{Student-t: } w=\frac{\nu+1}{\nu + (u/s)^2}. \]

Expectiles (asymmetric least squares)

The \(\tau\)-expectile (asymmetric least squares [Newey and Powell, 1987]) weights residuals asymmetrically (\(w=\tau\) above the fit, else \(1-\tau\)); the Jones/Yao-Tong bijection [Jones, 1994, Yao and Tong, 1996] maps an expectile level back to a genuine quantile.

\[ w = \tau\,\mathbb 1\{y\ge\eta\} + (1-\tau)\,\mathbb 1\{y<\eta\}. \]