One by One Convolution
lastmod 2018-10-23created 2016-10-286 edits
About
1x1_conv_layer_-_preview.jpg
As the two images above show, this process (an MLP Conv using n layers) is equivalent to applying a regular convolution followed by a convolution with a 1x1 filter. Remember what the purpose of NIN mentioned earlier was? It was to better represent nonlinear relationships by introducing an MLP. As just shown, the relationship obtained through the MLP can ultimately also be expressed as a combination of a regular CNN and a 1x1 Conv. In other words, the significance of NIN lies in introducing this 1x1 Conv. As a result, using 1x1 Convs appropriately makes it possible to build nonlinear functions better. And that is not the only advantage of the 1x1 Conv. A 1x1 Conv performs pooling across channels. That is, if the number of 1x1 Convs is smaller than the number of input channels, dimension reduction becomes possible.
Why 1x1 convolution ? Is it not redundant?
| | |
| Convolution with kernel of size 3x3 | Convolution with kernel of size 1x1 |
Mean
in the Google Inception article Going deeper with convolutions:
| One big problem with the above modules, at least in this naive form, is that even a modest number of 5x5 convolutions can be prohibitively expensive on top of a convolutional layer with a large number of filters. This leads to the second idea of the proposed architecture: judiciously applying dimension reductions and projections wherever the computational requirements would increase too much otherwise. This is based on the success of embeddings: even low dimensional embeddings might contain a lot of information about a relatively large image patch... 1x1 convolutions are used to compute reductions before the expensive 3x3 and 5x5 convolutions. Besides being used as reductions, they also include the use of rectified linear activation which makes them dual-purpose. |
See also
Favorite site
- One by One (1 x 1) Convolution - counter-intuitively useful 1
- (Deep Learning) What is CS231n "1X1 Convolution"?
References
-
One_by_One_Convolution_-_counter-intuitively_useful.pdf ↩