本文算是对常用梯度图下降的算法综述，从方向导数开始得到梯度下降的原始算法，接着描述了动量梯度下降算法。 而由于超参数学习率对梯度下降的重要性，所以梯度算法就有多个自适应梯度下降算法。 主要有以下内容：

• 方向导数和梯度下降
• 梯度下降的形式，批量梯度下降，SGD以及mini-batch 梯度下降
• 梯度下降的进化，主要是几种自适应梯度下降算法：AdaGrad,RMSprop,AdaDelta,Adam

方向导数和梯度

方向导数

方向导数指的是函数z=f(x,y)在某一点P沿某一方向的变化率，其表示形式为

https://www.zealer.com/detail/1251497 https://www.zealer.com/detail/1251496 https://www.zealer.com/detail/1251495 https://www.zealer.com/detail/1251494 https://www.zealer.com/detail/1251493 https://www.zealer.com/detail/1251492 https://www.zealer.com/detail/1251491 https://www.zealer.com/detail/1251490 https://www.zealer.com/detail/1251489 https://www.zealer.com/detail/1251488 https://www.zealer.com/detail/1251487 https://www.zealer.com/detail/1251486 https://www.zealer.com/detail/1251485 https://www.zealer.com/detail/1251484 https://www.zealer.com/detail/1251483 https://www.zealer.com/detail/1251482 https://www.zealer.com/detail/1251481 https://www.zealer.com/detail/1251480 https://www.zealer.com/detail/1251479 https://www.zealer.com/detail/1251478 https://www.zealer.com/detail/1251477 https://www.zealer.com/detail/1251476 https://www.zealer.com/detail/1251475 https://www.zealer.com/detail/1251474 https://www.zealer.com/detail/1251473 https://www.zealer.com/detail/1251472 https://www.zealer.com/detail/1251471 https://www.zealer.com/detail/1251470 https://www.zealer.com/detail/1251469 https://www.zealer.com/detail/1251468 https://www.zealer.com/detail/1251467 https://www.zealer.com/detail/1251466 https://www.zealer.com/detail/1251465 https://www.zealer.com/detail/1251464 https://www.zealer.com/detail/1251463 https://www.zealer.com/detail/1251462 https://www.zealer.com/detail/1251461 https://www.zealer.com/detail/1251460 https://www.zealer.com/detail/1251459 https://www.zealer.com/detail/1251458 https://www.zealer.com/detail/1251457 https://www.zealer.com/detail/1251456 https://www.zealer.com/detail/1251455 https://www.zealer.com/detail/1251454 https://www.zealer.com/detail/1251453 https://www.zealer.com/detail/1251452 https://www.zealer.com/detail/1251451 https://www.zealer.com/detail/1251450 https://www.zealer.com/detail/1251449 https://www.zealer.com/detail/1251448 https://www.zealer.com/detail/1251447 https://www.zealer.com/detail/1251446 https://www.zealer.com/detail/1251445 https://www.zealer.com/detail/1251444 https://www.zealer.com/detail/1251443 https://www.zealer.com/detail/1251442 https://www.zealer.com/detail/1251441 https://www.zealer.com/detail/1251440 https://www.zealer.com/detail/1251439 https://www.zealer.com/detail/1251438 https://www.zealer.com/detail/1251437 https://www.zealer.com/detail/1251436 https://www.zealer.com/detail/1251435 https://www.zealer.com/detail/1251434 https://www.zealer.com/detail/1251433 https://www.zealer.com/detail/1251432 https://www.zealer.com/detail/1251431 https://www.zealer.com/detail/1251430 https://www.zealer.com/detail/1251429 https://www.zealer.com/detail/1251428 https://www.zealer.com/detail/1251427 https://www.zealer.com/detail/1251426 https://www.zealer.com/detail/1251425 https://www.zealer.com/detail/1251424 https://www.zealer.com/detail/1251423 https://www.zealer.com/detail/1251422 https://www.zealer.com/detail/1251421 https://www.zealer.com/detail/1251420 https://www.zealer.com/detail/1251419 https://www.zealer.com/detail/1251418 https://www.zealer.com/detail/1251417 https://www.zealer.com/detail/1251416 https://www.zealer.com/detail/1251415 https://www.zealer.com/detail/1251414 https://www.zealer.com/detail/1251413 https://www.zealer.com/detail/1251412 https://www.zealer.com/detail/1251411 https://www.zealer.com/detail/1251410 https://www.zealer.com/detail/1251409 https://www.zealer.com/detail/1251408 https://www.zealer.com/detail/1251407 https://www.zealer.com/detail/1251406 https://www.zealer.com/detail/1251405 https://www.zealer.com/detail/1251404 https://www.zealer.com/detail/1251403 https://www.zealer.com/detail/1251402 https://www.zealer.com/detail/1251401 https://www.zealer.com/detail/1251400 https://www.zealer.com/detail/1251399 https://www.zealer.com/detail/1251398 https://www.zealer.com/detail/1251397 https://www.zealer.com/detail/1251396 https://www.zealer.com/detail/1251395 https://www.zealer.com/detail/1251394 https://www.zealer.com/detail/1251393 https://www.zealer.com/detail/1251392 https://www.zealer.com/detail/1251391 https://www.zealer.com/detail/1251390 https://www.zealer.com/detail/1251389 https://www.zealer.com/detail/1251388 https://www.zealer.com/detail/1251387 https://www.zealer.com/detail/1251386 https://www.zealer.com/detail/1251385 https://www.zealer.com/detail/1251384 https://www.zealer.com/detail/1251383 https://www.zealer.com/detail/1251382 https://www.zealer.com/detail/1251381 https://www.zealer.com/detail/1251380 https://www.zealer.com/detail/1251379 https://www.zealer.com/detail/1251378 https://www.zealer.com/detail/1251377 https://www.zealer.com/detail/1251376 https://www.zealer.com/detail/1251375 https://www.zealer.com/detail/1251374 https://www.zealer.com/detail/1251373 https://www.zealer.com/detail/1251372 https://www.zealer.com/detail/1251371 https://www.zealer.com/detail/1251370 https://www.zealer.com/detail/1251369 https://www.zealer.com/detail/1251368 https://www.zealer.com/detail/1251367 https://www.zealer.com/detail/1251366 https://www.zealer.com/detail/1251365 https://www.zealer.com/detail/1251364 https://www.zealer.com/detail/1251363 https://www.zealer.com/detail/1251362 https://www.zealer.com/detail/1251361 https://www.zealer.com/detail/1251360 https://www.zealer.com/detail/1251359 https://www.zealer.com/detail/1251358 https://www.zealer.com/detail/1251357 https://www.zealer.com/detail/1251356 https://www.zealer.com/detail/1251355 https://www.zealer.com/detail/1251354 https://www.zealer.com/detail/1251352 https://www.zealer.com/detail/1251351 https://www.zealer.com/detail/1251350 https://www.zealer.com/detail/1251349 https://www.zealer.com/detail/1251348 https://www.zealer.com/detail/1251347 https://www.zealer.com/detail/1251346 https://www.zealer.com/detail/1251345 https://www.zealer.com/detail/1251343 https://www.zealer.com/detail/1251340 https://www.zealer.com/detail/1251338 https://www.zealer.com/detail/1251337 https://www.zealer.com/detail/1251336 https://www.zealer.com/detail/1251334 https://www.zealer.com/detail/1251333 https://www.zealer.com/detail/1251332