PyTorch-Adam优化算法原理,公式,应用

    概念:Adam 是一种可以替代传统随机梯度下降过程的一阶优化算法,它能基于训练数据迭代地更新神经网络权重。Adam 最开始是由 OpenAI 的 Diederik Kingma 和多伦多大学的 Jimmy Ba 在提交到 2015 年 ICLR 论文(Adam: A Method for Stochastic Optimization)中提出的.该算法名为「Adam」,其并不是首字母缩写,也不是人名。它的名称来源于适应性矩估计(adaptive moment estimation)

  Adam(Adaptive Moment Estimation)本质上是带有动量项的RMSprop,它利用梯度的一阶矩估计和二阶矩估计动态调整每个参数的学习率。它的优点主要在于经过偏置校正后,每一次迭代学习率都有个确定范围,使得参数比较平稳。其公式如下:

  PyTorch-Adam优化算法原理,公式,应用

  其中,前两个公式分别是对梯度的一阶矩估计和二阶矩估计,可以看作是对期望E|gt|,E|gt^2|的估计; 
公式3,4是对一阶二阶矩估计的校正,这样可以近似为对期望的无偏估计。可以看出,直接对梯度的矩估计对内存没有额外的要求,而且可以根据梯度进行动态调整。最后一项前面部分是对学习率n形成的一个动态约束,而且有明确的范围。

  优点:

1、结合了Adagrad善于处理稀疏梯度和RMSprop善于处理非平稳目标的优点; 
2、对内存需求较小; 
3、为不同的参数计算不同的自适应学习率; 
4、也适用于大多非凸优化-适用于大数据集和高维空间。

  应用和源码:

  参数实例:

class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)

  参数含义:

  params(iterable):可用于迭代优化的参数或者定义参数组的dicts。

  lr (float, optional) :学习率(默认: 1e-3) betas (Tuple[float, float], optional):

  用于计算梯度的平均和平方的系数(默认: (0.9, 0.999)) eps (float, optional):

  为了提高数值稳定性而添加到分母的一个项(默认: 1e-8) weight_decay (float, optional):权重衰减(如L2惩罚)(默认: 0)

  torch.optim.adam源码:

 import math
from .optimizer import Optimizer class Adam(Optimizer):
def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8,weight_decay=0):
defaults = dict(lr=lr, betas=betas, eps=eps,weight_decay=weight_decay)
super(Adam, self).__init__(params, defaults) def step(self, closure=None):
loss = None
if closure is not None:
loss = closure() for group in self.param_groups:
for p in group['params']:
if p.grad is None:
continue
grad = p.grad.data
state = self.state[p] # State initialization
if len(state) == 0:
state['step'] = 0
# Exponential moving average of gradient values
state['exp_avg'] = grad.new().resize_as_(grad).zero_()
# Exponential moving average of squared gradient values
state['exp_avg_sq'] = grad.new().resize_as_(grad).zero_() exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
beta1, beta2 = group['betas'] state['step'] += 1 if group['weight_decay'] != 0:
grad = grad.add(group['weight_decay'], p.data) # Decay the first and second moment running average coefficient
exp_avg.mul_(beta1).add_(1 - beta1, grad)
exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) denom = exp_avg_sq.sqrt().add_(group['eps']) bias_correction1 = 1 - beta1 ** state['step']
bias_correction2 = 1 - beta2 ** state['step']
step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1 p.data.addcdiv_(-step_size, exp_avg, denom) return loss

  使用例子:

 import torch

 # N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10 # Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out) # Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
torch.nn.Linear(D_in, H),
torch.nn.ReLU(),
torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(reduction='sum') # Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use Adam; the optim package contains many other
# optimization algoriths. The first argument to the Adam constructor tells the
# optimizer which Tensors it should update.
learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for t in range(500):
# Forward pass: compute predicted y by passing x to the model.
y_pred = model(x) # Compute and print loss.
loss = loss_fn(y_pred, y)
print(t, loss.item()) # Before the backward pass, use the optimizer object to zero all of the
# gradients for the variables it will update (which are the learnable
# weights of the model). This is because by default, gradients are
# accumulated in buffers( i.e, not overwritten) whenever .backward()
# is called. Checkout docs of torch.autograd.backward for more details.
optimizer.zero_grad() # Backward pass: compute gradient of the loss with respect to model
# parameters
loss.backward() # Calling the step function on an Optimizer makes an update to its
# parameters
optimizer.step()

  到这里,相信对付绝大多数的应用是可以的了.我的目的也就基本完成了.接下来就要在应用中加深理解了.

  

参考文档:

1 https://blog.csdn.net/kgzhang/article/details/77479737

2 https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_optim.html

上一篇:MySQL数据库中的高级(进阶)语句:VIEW视图、联集和常见计算(长大之后,发现满世界对你承诺,而你不敢承诺,只能忍让,偏偏忍无可忍。所有的食物都不卫生,所有的爱情都不正直,于是你依旧是个毫无依赖的小孩。)


下一篇:【9.2校内测试】【开学祭】【exgcd】【树规(背包】【模拟】