期末复习--实用回归分析
\[y=\left[\begin{array}{c} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{array}\right], X=\left[\begin{array}{cccc} 1 & x_{11} & \cdots & x_{1 p} \\ 1 & x_{21} & \cdots & x_{2 p} \\ \vdots & \vdots & & \vdots \\ 1 & x_{n 1} & \cdots & x_{n p} \end{array}\right], \epsilon=\left[\begin{array}{c} \epsilon_{1} \\ \epsilon_{2} \\ \vdots \\ \epsilon_{n} \end{array}\right], \beta=\left[\begin{array}{c} \beta_{0} \\ \beta_{1} \\ \vdots \\ \beta_{p} \end{array}\right] \] \[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon \] \[\boldsymbol{X}=\left(\mathbf{1}, \boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{p}\right)-- n \times(p+1) \] \[\varepsilon=\left(\varepsilon_{1}, \ldots, \varepsilon_{n}\right)^{\prime} \]Gauss-Markov条件:
\[\left\{\begin{array}{l} E\left(\varepsilon_{i}\right)=0, i=1, \ldots, n \\ \operatorname{Cov}\left(\varepsilon_{i}, \varepsilon_{j}\right)=0, i \neq j ; \quad \operatorname{Var}\left(\varepsilon_{i}\right)=\sigma^{2} \end{array}\right. \]正态性假设:
\[\left\{\begin{array}{l} \varepsilon_{i} \sim N\left(0, \sigma^{2}\right), i=1, \ldots, n \\ \varepsilon_{1}, \ldots, \varepsilon_{n} \quad \text { 相互独立 } \end{array}\right. \]LSE
\[Q(\boldsymbol{\beta})=(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta}) \] \[\frac{\partial Q(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\boldsymbol{X}^{\prime} 2(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})=-2 \boldsymbol{X}^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})=0 \] \[\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \] \[\hat{\boldsymbol{y}}=\boldsymbol{X} \hat{\boldsymbol{\beta}}=\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \stackrel{\text {def}}{=} \boldsymbol{H} \boldsymbol{y} \]\(H=X(X'X)^{-1}X'\rightarrow H^2=X(X'X)^{-1}X'*X(X'X)^{-1}X'=X(X'X)^{-1}X'=H\)
\((I_n-H)^2=I_n^2-2HI_n+H^2=I_n-H\)
回归系数估计的最大似然法
\(y \sim N(X\beta,\sigma^2I_n)\)
如果正态分布假设满足，
\[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon, \quad \varepsilon \sim N\left(\mathbf{0}, \sigma^{2} \boldsymbol{I}_{n}\right) \]则 \(\boldsymbol{y}\) 的概率分布为 \(: \boldsymbol{y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}_{n}\right),\) 这是似然函数为
\[L(\boldsymbol{\beta})=\left(2 \pi \sigma^{2}\right)^{-n / 2} \exp \left\{-\frac{1}{2 \sigma^{2}}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})\right\} \]对其做“-ln”变换，记为 \(\ell(\boldsymbol{\beta}),\) 可以证明 \(\boldsymbol{\beta}\) 的极大似然估计 \(\hat{\beta}_{M L E}\) 与最小二 乘估计等价。而误差项的方差 \(\sigma^{2}\) 的极大似然估计
\[\hat{\sigma}_{M L E}^{2}=\frac{1}{n} S S E=\frac{1}{n} \boldsymbol{e}^{\prime} \boldsymbol{e} \]EX:
对于多元线性回归模型: \(\mathrm{Y}=\mathrm{X} \beta+\varepsilon, \varepsilon \sim \mathrm{N}\left(0, \sigma^{2} \mathrm{I}_{n}\right)\)
(1).利用极大似然估计求出 \(\widehat{\sigma^{2}}\);
(2).求 \(\mathrm{E}\left(\widehat{\sigma^{2}}\right)\)
(1)如果正态分布假设满足，
\[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon, \quad \varepsilon \sim N\left(\mathbf{0}, \sigma^{2} \boldsymbol{I}_{n}\right) \]则 \(\boldsymbol{y}\) 的概率分布为 \(: \boldsymbol{y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}_{n}\right),\) 这是似然函数为
\[L(\boldsymbol{\beta})=\left(2 \pi \sigma^{2}\right)^{-n / 2} \exp \left\{-\frac{1}{2 \sigma^{2}}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})\right\} \]对其做“-ln”变换，记为 \(\ell(\boldsymbol{\beta}),\) 可以证明 \(\boldsymbol{\beta}\) 的极大似然估计 \(\hat{\beta}_{M L E}\) 与最小二 乘估计等价。
\[\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \]而误差项的方差 \(\sigma^{2}\) 的极大似然估计
\[\widehat{\sigma^{2}}=\frac{SSE}{n} \](2)
\[E\left(\hat{\sigma}^{2}\right)=\left(\frac{n-p-1}{n}\right) \frac{SSE}{n-p-1} \]偏决定系数
\[\begin{array}{l} \text { Model1: } y_{i}=\beta_{0}+\beta_{1} x_{1 i}+\beta_{2} x_{2 i}+\varepsilon_{i}, i=1, \ldots, n \\ \text { Model0 : } y_{i}=\beta_{0}+\beta_{2} x_{2 i}+\varepsilon_{i}, i=1, \ldots, n \end{array} \] \[r_{y 1 ; 2}^{2}=\frac{\operatorname{SSE}\left(x_{2}\right)-\operatorname{SSE}\left(x_{1}, x_{2}\right)}{\operatorname{SSE}\left(x_{2}\right)} \stackrel{?}{=} \frac{\operatorname{SSR}\left(x_{1}, x_{2}\right)-\operatorname{SSR}\left(x_{2}\right)}{\operatorname{SSE}\left(x_{2}\right)} \]当模型中已有 \(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\) 时， \(y\) 与 \(x_{j}\) 的偏决定系数为:
\[\begin{aligned} r_{y j ;-j}^{2} &=\frac{\operatorname{SSE}\left(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\right)-\operatorname{SSE}\left(x_{1}, \ldots, x_{p}\right)}{\operatorname{SSE}\left(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\right)} \\ &=\frac{S S R-\operatorname{SSR}(-j)}{\operatorname{SSE}(-j)} \end{aligned} \]