习题9.1

EM算法分为E步与M步

对于E步，计算期望。 \(\mu_j^{(i+1)} = \frac{\pi^{(i)}(p^{(i)})^{y_j}(1-p^{(i)})^{1-y_j}}{\pi^{(i)}(p^{(i)})^{y_j}(1-p^{(i)})^{1-y_j} + (1 - \pi^{(i)})(q^{(i)})^{y_j}(1-q^{(i)})^{1-y_j}}\)

对于M步，极大似然进行估计。 \(\pi^{(i+1)} = \frac{1}{n} \sum \mu_j^{(i+1)}\) ， \(p^{(i+1)} = \frac{\sum \mu_j^{(i+1)}y_j}{\sum \mu_j^{(i+1)}}\) ，\(q^{(i+1)} = \frac{\sum (1-\mu_j^{(i+1)})y_j}{\sum (1 - \mu_j^{(i+1)})}\)

第一轮迭代

E步：观测值为1的 \(\mu^{(1)} = 0.4115\) ，观测值为0的\(\mu^{(1)} = 0.5374\)

M步：\(\pi^{(1)} = 0.4619\) ，\(p^{(1)} = 0.5346\) ， \(q^{(1)} = 0.6561\)

第二轮迭代

E步：观测值为1的 \(\mu^{(2)} = 0.4116\) ，观测值为0的\(\mu^{(2)} = 0.5374\)

M步：\(\pi^{(2)} = 0.4619\) ，\(p^{(2)} = 0.5346\) ，\(q^{(2)} = 0.6561\)

EM算法两轮收敛，得到参数的极大似然估计值。

\(\hat \pi = 0.4619, \hat p = 0.5346,\hat q =0.6561\)

习题9.2

引理9.2：若 \(\tilde P_{\theta}(Z) = P(Z|Y,\theta)\) ， 则 \(F(\tilde P, \theta) = log P(Y|\theta)\)

\(F(\tilde P, \theta) = E_{\tilde P}[logP(Y,Z|\theta)] + H(\tilde P) \\ = E_{\tilde P}[logP(Y,Z|\theta)] - E_{\tilde P} log \tilde P(Z) \\ =\sum_Z \tilde P_{\theta}(Z) logP(Y,Z|\theta) -\sum_Z \tilde P(Z) log \tilde P(Z) \\ =\sum_Z P(Z|Y,\theta) logP(Y,Z|\theta) -\sum_Z P(Z|Y,\theta) log P(Z|Y,\theta) \\ =\sum_Z P(Z|Y,\theta) log \frac{P(Y,Z|\theta)}{P(Z|Y,\theta)} \\ = \sum_Z P(Z|Y,\theta) logP(Y|\theta) = logP(Y|\theta)\)

习题9.3

调用 sklearn.mixture.GaussianMixture 这个API进行训练，可得

\(\alpha_1 = 0.8668, \mu_1 = 32.9849, \sigma_1^2 = 429.4576\)

\(\alpha_2 = 1- \alpha_1 = 0.1332, \mu_2 = -57.5111, \sigma_2^2 = 90.2499\)

习题9.4

Mixture of Naive Bayes Model（NBMM，混合朴素贝叶斯模型）

NBMM的EM算法

E步：\(w_j^{(i)} = P(z^{(i)} =1|x^{(i)};\phi_z,\phi_{j|z^{(i)}=1},\phi_{j|z^{(i)}=0})\)

M步：\(\phi_{j|z^{(i)}=1} = \frac{\sum w^{(i)}I(x_j^{(i)}=1)}{\sum w^{(i)}}\) ，\(\phi_{j|z^{(i)}=0} = \frac{\sum (1-w^{(i)})I(x_j^{(i)}=1)}{\sum (1-w^{(i)})}\)， \(\phi_{z^{(i)}} = \frac{\sum w^{(i)}}{m}\)