Spatial Transformer Network with image classification

2024-01-23 08:47:35

In this tutorial, you will learn how to augment your network using a visual attention mechanism called spatial transformer networks.

Spatial transformer networks are a generalization of differentiable attention to any spatial transformation. Spatial transformer networks (STN for short) allow a neural network to learn how to perform spatial transformations on the input image in order to enhance the geometric invariance of the model. For example, it can crop a region of interest, scale and correct the orientation of an image. It can be a useful mechanism because CNNs are not invariant to rotation and scale and more general affine transformations.

解读: 上面提到的 crop a region of interest, scale and correct the orientation of an image. because CNNs are not invariant to rotation and scale and more general affine transformations. 这两句话，在我看来其实有一部分数据增强可以解决，所以两者在这块儿貌似有点小重合。

Spatial transformer networks boils down to three main components:

The localization network is a regular CNN which regresses the transformation parameters. The transformation is never learned explicitly from this dataset, instead the network learns automatically the spatial transformations that enhances the global accuracy.
The grid generator generates a grid of coordinates in the input image corresponding to each pixel from the output image.
The sampler uses the parameters of the transformation and applies it to the input image.

解读: 此处说明了 STN 的构成主要分为 3 个部分，分别是 Localization network、Grid generator、Sampler。从上图也可以看出来，STN 的参数学习是一种端到端的方式。另外再提一点，如果你看过源码的话，可以发现 STN 能处理的输入分辨率是比较小的，如果分辨率比较大，那么也意味着我们需要扩张 STN 的网络，这个开销就有点大了，可能不太合适，不过可以考虑放在 ResNet 等网络的后半部分，不过还没实验是否能带来准确率提升。

下图是 MNIST 过后，展示的原始图片和转换后的图片。