Pytorch grad

Gradient

Clarificaton
导数,derivate
偏微分,partial derivate
梯度,gradient

\[\begin{aligned} &\bigtriangledown{f} = (\frac{\delta{f}}{\delta{x_1}};\frac{\delta{f}}{\delta{x_2}};...;\frac{\delta{f}}{\delta{x_n}})\\ &z = y^2-x^2\\ &\frac{\delta{z}}{\delta{x}}=-2x\\ &\frac{\delta{z}}{\delta{y}}=2y\\ &\bigtriangledown{f} = (-2x,2y)\\ \end{aligned} \]

How to search for minima?

\[\begin{aligned} &\theta_{t+1} = \theta_{t} - \alpha_{t}\bigtriangledown{f(\theta_{t})}\\ &Function:\\ &J(\theta_1,\theta_2) = {\theta_1}^2 + {\theta_2}^2\\ &Objective:\\ &\substack{\min\\{\theta_1,\theta_2}} J(\theta_1,\theta_2)\\ &Update rules:\\ &\theta_1:=\theta_1 - \alpha \frac{d}{d\theta_1}J(\theta_1,\theta_2)\\ &\theta_2:=\theta_2 - \alpha \frac{d}{d\theta_2}J(\theta_1,\theta_2)\\ &Derivatives:\\ &\frac{d}{d\theta_1}J(\theta_1,\theta_2)=\frac{d}{d\theta_1}{\theta_1}^2 + \frac{d}{d\theta_1}{\theta_2}^2=2\theta_1\\ &\frac{d}{d\theta_2}J(\theta_1,\theta_2)=\frac{d}{d\theta_2}{\theta_1}^2 + \frac{d}{d\theta_2}{\theta_2}^2=2\theta_2\\ \end{aligned} \]

Common Functions
Common Functions Function Derivative
Constant $$C$$ $$0$$
Line $$x$$ $$1$$
$$ax$$ $$a$$
Square $$x^2$$ $$2x$$
Square root $$\sqrt{x}$$ $$\frac{1}{2}x^{-\frac{1}{2}}$$
Exponential $$e^x$$ $$e^x$$
$$a^x$$ $$\ln(a)a^x$$
Logarithms $$\ln(x)$$ $$\frac{1}{x}$$
$$\log_a(x)$$ $$\frac{1}{x\ln(a)}$$
Trigonometry $$\sin(x)$$ $$\cos(x)$$
$$\cos(x)$$ $$-\sin(x)$$
$$\tan(x)$$ $${\sec(x)}^2$$
eg.

\[\begin{aligned} &f=[y-(xw+b)]^2\\ &g=xw+b\\ &\frac{\delta f}{\delta w}=2(y-g)\frac{\delta g}{\delta w}=2(y-(xw+b))*(-x)\\ &\frac{\delta f}{\delta b}=2(y-g)\frac{\delta g}{\delta b}=2(y-(xw+b))*(-1)\\ &\bigtriangledown=(2(y-(xw+b))*(-x),2(y-(xw+b))*(-1))\\ \end{aligned} \]

上一篇:$\gcd(q^a-1,q^b-1)=q^{\gcd(a,b)}-1$


下一篇:Raki的统计学习方法笔记0xB(11)章:条件随机场