Linear neuron:

\[y=b+\sum\limits_i{x_i w_i}\]

Binary threshold neuron:

\[z = \sum\limits_i{x_i w_i}\]
\[y=\left\{\begin{aligned} 1,~~~~~~~z\gt \theta \\ 0,otherwise\end{aligned}\right.\]

也可以写作：

\[z = \sum\limits_i{x_i w_i}\]
\[y=\left\{\begin{aligned} 1,~~~~~~~z\gt 0 \\ 0, otherwise\end{aligned}\right.\]

Rectified linear neuron:

\[z=b+\sum\limits_i{x_i w_i}\]
\[y=\left\{\begin{aligned} 1,~~~~~~~z\gt 0 \\ 0, otherwise\end{aligned}\right.\]

Sigmoid neuron:

\[z=b+\sum\limits_i{x_i w_i}\]
\[y=\frac{1}{1+e^{-z}}\]

Stochastic binary neuron:

we treat the output of the logistic as the probability of producing \(s=1\).

\[z = \sum\limits_i{x_i w_i}\]
\[P(s=1)=\frac{1}{1+e^{-z}}\]

Note：

1. \(x\): neuron input.
2. \(w\): input weight matrix.
3. \(b\): bias term.
4. \(y\): neuron output .