Convolutional neural network

lastmod 2026-10-11created 2015-10-1551 edits

In machine learning, a convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network where the individual neurons are tiled in such a way that they respond to overlapping regions in the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons which are designed to use minimal amounts of preprocessing. They are widely used models for image and video recognition.

A convolutional neural network is the same as a conventional Neural network. The difference is that it uses Convolution to extract meaningful features.

Introduce

You have a set of images and labels saying whether each image contains a person or not. Your goal is to guess, given an image, whether there is a person in it. The basic approach would be to extract SIFT features and apply a classification model, but the accuracy is poor. This is because the features differ when the person is far away, doing a handstand, or close up. If you want to classify this with Deep learning, you can use a convolutional neural network.

Terams

Filter: the matrix or values used to apply Convolution to the input image.
Feature map: the data resulting from passing an image through a Convolution filter.

Convolution calculate example

okdalto/conv_visualizer - Convolution Visualization Project
TensorFlow and deep learning, without a PhD
[Recommended] 37. Building the Conv layer of a ConvNet (lec 11-1) <- You need to understand this to understand 1x1 convolution.

Theory_-_convolutional_networks.gif

Think of it like a matrix, as shown below.

Ratsgos_blog_-_CNN_Forward_Pass.png

CNN Backpropagation

[Recommended] Backpropagation in CNN - ratsgo's blog ¹

Average Pooling

By the chain rule of differentiation, the gradient at the current point equals the incoming gradient multiplied by the local gradient. The average is computed by summing all values and dividing by the number of elements. If there are m elements, the local gradient at the Average Pooling point is 1/m.

Ratsgos_blog_-_Average_Pooling_Backpropagation.png

Max Pooling

If pooling is done with the maximum value, backpropagation works as below. That is, the local gradient of the element holding the maximum is 1 and the rest are 0, so the result is obtained by multiplying these by the incoming gradient.

Ratsgos_blog_-_Max_Pooling_Backpropagation.png

CNN

Ratsgos_blog_-_CNN_Backpropagation.png

Layer

It usually consists of several layers and basically has three different kinds of layers.

Convolution layer: the layer that extracts Convolution features. It extracts meaningful features.
Polling layer: CNNs are usually applied to images. Because images have too many pixels, the process of sub-sampling to reduce the features is called polling.
Feedforward layer: applied last, and used for classification with the features coming out of the Convolution and Polling layers. It behaves like an ordinary Neural Network.

A typical CNN is structured as follows.

Convolution layer -> Polling layer -> Convolution layer -> Polling layer -> ... -> Feedforward layer

That is, Convolution layers and Polling layers are used alternately to extract features, and classification is finally done through the Feddforward layer.

Intuitively, think of the Pooling layer as reducing the size of the image. If Convolution features are extracted after reducing the image size, both large and small people from the problem in the example above can be detected. For this reason, Pooling layers and Convolution layers are used alternately, and naturally, the more layers there are, the better the classification performance will be.

Cnn_layer.png

Fully Connected Network

Computing the parameters needed to connect every input layer unit to the hidden layer was feasible, but for somewhat larger, realistic images such as 96*96, Fully Connected is unsuitable.

Locally Connected Network

Connects only the pixels in a nearby region of the input to a hidden layer node.

Vision Layers

Convolution
Pooling
Local Response Normalization (LRN)
im2col

Models

Github: Benchmarks for popular CNN models

The following well-known models exist.

Table of Machine learning

Table of Machine learning
Learning Course	Theory	Probability theory → Bayesian probability
	Machine learning	Hypothesis → Loss function → Gradient descent → Overfitting
	Neural network	Artificial neuron → Perceptron → Multilayer perceptron → (Example) → Feedforward neural network → Activation function (Sigmoid function & Euler's number) → Softmax function (Loss function) → Backpropagation (Gradient descent) → (Example)
	Deep learning	Convolutional neural network (CNN) & Region-based Convolutional Network (RCNN, R-CNN)
	ETC	Tutorials
Theory	Basic	Terms
	Probability theory	Bayesian probability, Decision Theory, Probability Density Function (PDF), Radial basis function (RBF), Hyperparameter
	Neuroscience d	Neuron
	Statistics	Covariance, Statistical classification, Confusion matrix, Cross-validation, Root-mean-square deviation (MESD), Mean squared error
	Graph theory
	Matrix	General matrix-matrix multiplication (gemm), Toeplitz matrix, im2col
	Other	Computational learning theory, Empirical risk minimization, Occam learning, PAC learning, Statistical learning, VC theory, Bayesian network, Markov random field, Hidden Markov Model (HMM), Conditional Random Field, Normal Bayes Classifier, Energy Based Model, Occam's razor, Ground truth
Algorithm types	Supervised Learning	Support Vector Machine (SVM), Hidden Markov model, Regression analysis, Neural network, Naive Bayes classifier, K-Nearest Neighbor (K-NN), Decision trees, Ensembles (Bagging, Boosting, Random forest), Relevance vector machine (RVM)
	Unsupervised learning	Clustering, Independent component analysis
	Semi-supervised learning	Generative models, Low-density separation, Graph-based methods, Heuristic approaches
	Other learning	Reinforcement learning, deep learning
By topic	Structured prediction	Graphical models (Bayes net, CRF, HMM)
	Parameter estimation algorithms	Dynamic programming, EM algorithm
	Approximate inference techniques	Monte Carlo method, AdaBoost
	Approaches	Decision tree learning, association rule learning, genetic programming, inductive logic programming, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning
	Modeling	Neural network, SVM K-NN, decision tree, Genetic Algorithm, genetic programming, Gaussian process regression, linear discriminant analysis, perceptron, radial basis function network
	Recommender system (Korean: 추천 시스템)	Collaborative filtering (Korean: 협업 필터링), Content-based filtering (Korean: 컨텐츠 기반 필터링), Hybrid recommender systems
Data mining		Cross-validation (k-fold), Data set (Training set, Validation set, Test set)
Regression analysis		Linear regression, Logistic Regression, Logit function, Multinomial logistic regression (Softmax Regression)
Clustering		k-means clustering, BIRCH, Hierarchical, Expectation-maximization (EM), DBSCAN, OPTICS, Mean-shift
By type		3D Machine Learning
Artificial Neural Networks; ANN	Artificial neuron	Perceptron, Sigmoid neuron
	Combination function & Activation function	Sigmoid function, Rectified linear units (ReLU), 1x1 Convolution
	Loss function or Cost function	Softmax, Sum of squares (Euclidean), Hinge / Margin, Cross entropy, Infogain, Accuracy and Top-k
	Algorithms	Multilayer perceptron (MLP), Feed-forward Neural Network (FNN), Long short-term memory (LSTM), Autoencoder, Self-organizing map (SOM), Network In Network (NIN), Adaptive neuro fuzzy inference system (ANFIS)
	Deep learning	Deep neural network (DNN), Convolutional neural network (CNN) & Regions with Convolutional Neural Network (RCNN, R-CNN, SSD), Recurrent neural network (RNN), Gated Recurrent Unit (GRU), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, Deep hyper network (DHN), Deconvolutional Network, Fully Convolutional Networks for Semantic Segmentation (FCN), Generative Adversarial Networks (GAN), Learning Deconvolution Network for Semantic Segmentation, Multi-Scale Context Aggregation by Dilated Convolutions (Dilated Convolutions), Dynamic Routing Between Capsules (Capsules Network), YOLO, Path Aggregation Network for Instance Segmentation (PANet), Image-to-Image Translation with Conditional Adversarial Networks (pix2pix), CycleGAN, BicycleGAN, SlowFast, Kinetics, AVA, MelGAN, EfficientDet, SinGAN, Panoptic Segmentation, U-Net, MONAI, CenterNet, DeepLab, HRNet, OCRNet, ResNeSt
	Models	GoogLeNet, LeNet, AlexNet, ReNet, RCNN, SPPnet, Fast R-CNN, Faster R-CNN, VGG, SqueezeNet, PointNet, Mask R-CNN, MaskLab, OSMN (Video Segmentation), ColorNet, EfficientNet
Problems	Dimensionality reduction	Curse of dimensionality, Factor analysis, CCA, independent component analysis (ICA), LDA, non-negative matrix factorization (NMF), principal component analysis (PCA), t-SNE
	Underfitting & Overfitting	Early stopping, Model selection, Normalization, Regularization (L1, L2, Dropout), Generalization
	Initialization	Xavier Initialization
	ETC	Bias-variance dilemma, Vanishing gradient problem, Local minimum problem, Batch Normalization, Standardization, AutoML
Optimization		Hyperparameter, Simulated annealing, Early stopping, Feature scaling, Normal Equation, Gradient descent, Stochastic gradient descent (SGD), Backpropagation, Convex optimization, Performance Tuning, im2col, Dense-Sparse-Dense Training for Deep Neural Networks (DSD), Deep Compression, Pruning Neural Networks, Shake-Shake regularization, Accurate Large Minibatch SGD
ETC	Libraries	OpenCV, Deeplearning4j (DL4J), Torch, PyTorch, Theano, Caffe, TensorFlow, MMDetection, ConvNetJS, cuDNN, Netscope, Microsoft Azure Machine Learning Studio, OpenPose, DensePose, Keras, tiny-dnn, Detectron, Stanford CoreNLP, Aifiddle, Kubeflow, OpenNMT, alibi-detect, Flashlight (facebook), MediaPipe, Weights and Biases
	Dataset	MNIST (handwriting data), ImageNet, CIFAR-10, TinyImages, PASCAL VOC, COCO, AVSS, YouTube-VOS
	Annotation tools	labelme, cvat, f-BRS, COCO Annotator
	See also	Artificial intelligence, autonomous robots, bioinformatics, computational intelligence, computer vision, data mining, pattern recognition, big data
Unknown keyward		Classification, Anomaly detection, Association rules, Reinforcement learning, Structured prediction, Feature engineering, Feature learning, Online learning, Semi-supervised learning, Unsupervised learning, Learning to rank, Grammar induction, Local outlier factor, Minimum Description Length (MDL), Bayesian MAP, Structural Risk Minimization (SRM), Long Narrow Valley, Word2vec, Autoregressive integrated moving average (ARIMA)

Documentation

Bayesian probability

Bayesian Reasoning and Machine Learning: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online; http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/020217.pdf; 020217_-_Bayesian_Reasoning_and_Machine_Learning.pdf

Deep learning

ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)

http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

ImageNet_Classification_with_Deep_Convolutional_Neural_Networks.pdf

[Korean translation] ImageNet classification using deep convolutional neural networks

: ImageNet_Classification_with_Deep_Convolutional_Neural_Networks_-_ko.pdf

Going deeper with convolutions (GoogleNet): http://arxiv.org/pdf/1409.4842v1.pdf; Going_deeper_with_convolutions.pdf; Translation: GoogleNet

Feature article: Deep Hypernetwork models (Seoul National University / Byoung-Tak Zhang): Deep_Hypernetwork_models_201508.pdf

[Mocrosoft] Deep Residual Learning for Image Recognition (Winner ILSVRC2015): Deep_Residual_Learning_for_Image_Recognition(Winner_ILSVRC2015)_Microsoft.pdf

[Microsoft] Fast R-CNN, Towards Real-Time Object Detection with Region Proposal Networks (Winner ILSVR2015): Fast_R-CNN,Towards_Real-Time_Object_Detection_with_Region_Proposal_Networks(Winner_ILSVR2015)_Microsoft.pdf

Learning Deconvolution Network for Semantic Segmentation: http://cvlab.postech.ac.kr/research/deconvnet/

Deep EXpectation of apparent age from a single image: https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks.: Inside_outside_net_detecting_objects_in_context_with_skip_pooling_and_recurrent_neural_networks_2015.pdf; A paper on methods for detecting small objects.

Cho Dae-hyup's blog - Understanding deep learning and TensorFlow for those who gave up on math: http://bcho.tistory.com/1208; Machine_learning_ebooks_-_Machine_learning_for_those_who_abandon_math.pdf

Densely Connected Convolutional Networks (DenseNets): Deep learning CVPR2017 Best Paper Award

Deep Learning Interviews book: Hundreds of fully solved job interview questions from a wide range of key topics in AI.

Deep learning interview book

A collection of practical questions and solutions for master's/PhD students learning machine learning and for job seekers

A printed copy can be purchased, but the full PDF is freely available

https://arxiv.org/abs/2201.00650

[Recommended] "2025 Key Terms You Must Know in the AI Era, Seen Through AI Trends and Issues" published by NIA (National Information Society Agency): Key terms you must know in the AI era (168p PDF) | GeekNews; https://www.nia.or.kr/site/nia_kor/ex/bbs/View.do?cbIdx=82618&bcIdx=28928&parentSeq=28928; Nia.or.kr_-ai-_2025_terms.pdf

There Will Be a Scientific Theory of Deep Learning: There will be a scientific theory of deep learning. | GeekNews; There Will Be a Scientific Theory of Deep Learning; A paper arguing that a scientific theory characterizing key properties of deep learning, such as its training process, hidden representations, final weights, and performance, is taking shape; Presents five lines of research (interpretable idealized settings, tractable limits, simple mathematical laws, hyperparameter theory, universal behavior) as the core evidence

Tutorials

Getting Started with Deep Learning Properly (author: Takayuki Okatani / translator: Shim Hyo-seop): Deep_Learning_-Takayuki_Okatani-2015-_sample.pdf

Introductory guide to machine learning - IDG Deep Dive: http://www.itworld.co.kr/techlibrary/97428; IDG_DeepDive_Machine_learning-20160113.pdf

Understanding Deep Learning (unpublished; 2016-08-22 ver): Understanding_deep_learning_0822.pdf

Fundamental of Reinforcement Learning: https://www.gitbook.com/book/dnddnjs/rl/details; Fundamental_of_Reinforcement_Learning.pdf; Modulabs - Fundamentals of reinforcement learning

Deep Learning Papers Reading Roadmap (deep learning papers roadmap): https://github.com/songrotek/Deep-Learning-Papers-Reading-Roadmap/blob/master/README.md; Deep_Learning_Papers_Reading_Roadmap.md.zip

[Recommended] Machine Learning-based Web Exception Detection (reference site for the Financial Security Institute project!): https://cloudfocus.aliyun.com/Machine-Learning-based-Web-Exception-Detection-89782?spm=a2c1b.a2c1b4.a2c1b4.16.ZSQoEd; Machine_Learning-based_Web_Exception_Detection_-Insights_and_Trends-_Alibaba_Cloud_Focus.pdf

Machine learning basics, parts 1–57 (Miscellaneous explorations - Chamstudy ePaiai : Naver blog): http://sams.epaiai.com/220498694383

Microsoft releases ML for Beginners course: A 12-week, 24-lesson curriculum created by the MS Azure Clouds Advocates team; A classic machine learning course using Scikit-learn (deep learning to come in a separate AI course); https://github.com/microsoft/ML-For-Beginners

CS146S: The Modern Software Developer (Korean) | GeekNews: [Original] CS146S Korean Edition | The Modern Software Developer; The official Korean version of Stanford's CS146S course

Compress model

Deep Compression and EIE - Deep Neural Network Model Compression and Efficient Inference Engine: Deep_compression_and_EIE_PPT.pdf

Learning bothWeights and Connections for Efficient Neural Networks: Learning_both_weights_and_connections_for_efficient_neural_networks_2015.pdf