Artificial neural network

lastmod 2026-10-11created 2014-10-3116 edits

An artificial neural network (neural network) is a mathematical model that aims to express some characteristics of brain function through computer simulation.

An artificial neural network refers broadly to any model in which artificial neurons (nodes), which form a network through synaptic connections, change the strength of those synaptic connections through training and thereby gain the ability to solve problems. In a narrow sense it is sometimes used to mean a multilayer perceptron trained with error backpropagation, but this is an incorrect usage; artificial neural networks are not limited to this.

Artificial neural networks include supervised learning, in which the network is optimized for a problem through the input of a teacher signal (correct answer), and unsupervised learning, which does not need a teacher signal. Supervised learning is used when there is a clear answer, and unsupervised learning is used for data clustering. As a result, in order to reduce dimensions, many of them can obtain good answers with relatively little computation for problems that are not linearly separable, on multidimensional data such as images or statistics. For that reason, they are applied in various fields such as pattern recognition and data mining.

Artificial neural networks can be built using specialized computers, but most are implemented by application software on ordinary computers.

Background

Research into the human central nervous system inspired the concept of neural networks. In artificial neural networks, artificial neurons are connected to each other to form a network that mimics biological neural networks.

There is no single formal definition of what an artificial neural network is. However, a set of statistical models is called neural (Neural) if it has the following characteristics.

• It consists of a set of adjustable weights, that is, numerical parameters that can be adjusted by a learning algorithm.
• It can infer non-linear functions of its inputs.

The adjustable weights represent the connection strengths between neurons, and they are activated during training or prediction.

Neural networks resemble biological neural networks in that functions are performed in parallel or collectively by the units, rather than as subtasks assigned to the various units. The word 'neural network' usually refers to models used in statistics, cognitive psychology, and artificial intelligence. Neural network models that emulate the central nervous system are part of theoretical neuroscience and computational neuroscience.

About this

• Sigmoid Function of the Artificial neural network ¹

An artificial neural network uses a Combination function, which combines the input variables, and an Activation function, which combines and transforms the input information. The Combination function turns the pieces of input information into a single piece of information, most commonly using a weighted average, and the Activation function is a function that passes the combined value of the input information to the output layer or hidden layer, converting the combined value of the input information into a value within a certain range.

Category

• Perceptron
  • Adaline (Adaptive Linear Neuron)
  • Madaline (Many Adaline)
• Multilayer Perceptron
  • Has an input layer, an output layer, and one or more hidden layers (Hidden layer).
  • The training method is as follows.
    1. Present the input data to each unit of the input layer.
    2. This signal is transformed at each unit and passed to the intermediate layer.
    3. Finally, the result comes out at the output layer.
    4. Compare the output value with the desired output value and adjust the connection strengths in the direction that reduces the difference.
  • Backpropagation
• Hopfield Network
  • Associative Memory
• Kohonen Network
  • Competitive Learning
    • Each neuron calculates how close its connection-strength vector is to the input vector. The neurons then compete with each other to be granted the privilege of learning, and the neuron with the shortest distance wins.
  • Self-organizing Feature Maps
• ART Network (Adaptive Resonance Theory Network)
• Simulated Annealing
• Boltzmann Machine
• Counterpropagation network
  • A combination of Kohonen's self-organizing map and Grossberg's outstar algorithm.
• Cognitron
  • A pattern recognition multilayer neural network model based on the way the human visual system recognizes things
  • A pattern recognition model developed by Fukushima (K, Fukushima) in 1975; a multilayer neural network model that performs competitive learning.
• Neocognitron
• Fuzzy neural network model
  • SONN (Self-Organizing Neural Network)
    • When the connection strengths between neurons are adjusted, each connection strength is adjusted according to its fuzzy similarity with the input vector.

Table of Machine learning

Documentation

Bayesian probability

Bayesian Reasoning and Machine Learning

http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online

http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/020217.pdf

Deep learning

ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)

http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

[Korean translation] ImageNet classification using deep convolutional neural networks

Going deeper with convolutions (GoogleNet)

http://arxiv.org/pdf/1409.4842v1.pdf

Translation: GoogleNet

Special feature article: Deep Hypernetwork models (Seoul National University/Byoung-Tak Zhang)

[Mocrosoft] Deep Residual Learning for Image Recognition (Winner ILSVRC2015)

[Microsoft] Fast R-CNN, Towards Real-Time Object Detection with Region Proposal Networks (Winner ILSVR2015)

Learning Deconvolution Network for Semantic Segmentation

http://cvlab.postech.ac.kr/research/deconvnet/

Deep EXpectation of apparent age from a single image

https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks.

A paper on Small Object detection methods.

Cho Dae-hyup's blog - Understanding deep learning and TensorFlow for those who gave up on math

http://bcho.tistory.com/1208

Densely Connected Convolutional Networks (DenseNets)

Deep learning CVPR2017 Best Paper Award

Deep Learning Interviews book: Hundreds of fully solved job interview questions from a wide range of key topics in AI.

Deep learning interview book

A collection of practical questions and solutions for master's/PhD students and job seekers learning machine learning

A printed copy can also be purchased, but the full PDF is freely available

• https://arxiv.org/abs/2201.00650

[Recommended] "Key AI Glossary You Must Know in the AI Era, Examined Through 2025 AI Trends and Issues" published by NIA (National Information Society Agency)

Key glossary you must know in the AI era (168p PDF) | GeekNews

https://www.nia.or.kr/site/nia_kor/ex/bbs/View.do?cbIdx=82618&bcIdx=28928&parentSeq=28928

There Will Be a Scientific Theory of Deep Learning

There will be a scientific theory of deep learning. | GeekNews

There Will Be a Scientific Theory of Deep Learning

A paper arguing that a scientific theory characterizing key properties of deep learning, such as the training process, hidden representations, final weights, and performance, is taking shape

Presents five research threads (interpretable idealized settings, tractable limits, simple mathematical laws, hyperparameter theory, universal behaviors) as its core evidence

Tutorials

Getting Started with Deep Learning Properly (author Takayuki Okatani / translator Shim Hyo-seop)

Introductory Guide to Machine Learning - IDG Deep Dive

http://www.itworld.co.kr/techlibrary/97428

Understanding Deep Learning (unpublished; 2016-08-22 ver)

Fundamental of Reinforcement Learning

https://www.gitbook.com/book/dnddnjs/rl/details

MODULABS - Fundamentals of reinforcement learning

Deep Learning Papers Reading Roadmap (deep learning paper roadmap)

https://github.com/songrotek/Deep-Learning-Papers-Reading-Roadmap/blob/master/README.md

[Recommended] Machine Learning-based Web Exception Detection (reference site related to the Financial Security Institute project!)

https://cloudfocus.aliyun.com/Machine-Learning-based-Web-Exception-Detection-89782?spm=a2c1b.a2c1b4.a2c1b4.16.ZSQoEd

Machine learning basics parts 1–57 (Odds-and-ends exploration - Chamstudy ePaiai : Naver Blog)

http://sams.epaiai.com/220498694383

Microsoft releases ML for Beginners course

A 12-week, 24-lesson curriculum made by the MS Azure Clouds Advocates team

A classic machine learning course using Scikit-learn (deep learning will come as a separate AI course)

https://github.com/microsoft/ML-For-Beginners

CS146S: The Modern Software Developer (Korean) | GeekNews

[Original] CS146S Korean Edition | The Modern Software Developer

The official Korean version of Stanford's CS146S course

Compress model

Deep Compression and EIE - Deep Neural Network Model Compression and Efficient Inference Engine

Learning bothWeights and Connections for Efficient Neural Networks

Convolutional neural network

A CNN Presentation using Reveal.js.

Author - me

Gradient-Based Learning Applied to Document Recognition

http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

Visualizing and Understanding Convolutional Neural Networks

http://arxiv.org/abs/1311.2901

Compressing CNN for Mobile Device (Samsung) - the need for CNN model compression etc ...

[http://mlcenter.postech.ac.kr/files/attach/workshop_fall_2015/삼성전자_김용덕_박사.pdf](http://mlcenter.postech.ac.kr/files/attach/workshop_fall_2015/삼성전자_김용덕_박사.pdf)

Using Filter Banks in Convolutional Neural Networks for Texture Classification

https://arxiv.org/abs/1601.02919

Deep belief network

The Applications of Deep Learning on Traffic Identification

https://www.blackhat.com/docs/us-15/materials/us-15-Wang-The-Applications-Of-Deep-Learning-On-Traffic-Identification-wp.pdf

Deconvolution neural network

Learning Deconvolution Network for Semantic Segmentation

http://cvlab.postech.ac.kr/research/deconvnet/

https://arxiv.org/abs/1505.04366

Segmentation

Learning to Segment (Facebook Research)

https://research.fb.com/learning-to-segment/

DeepMask+SharpMask as well as MultiPathNet.

Recurrent Instance Segmentation

https://arxiv.org/abs/1511.08250

Slideshare - Single Shot MultiBox Detector and Recurrent Instance Segmentation

vid2vid

https://github.com/NVIDIA/vid2vid

Pytorch implementation for high-resolution (e.g., 2048x1024) photorealistic video-to-video translation.

It can be used for turning semantic label maps into photo-realistic videos, synthesizing people talking from edge maps, or generating human motions from poses.

Fire Detection

Organized under Fire Detection#Deep learning based.

Background subtraction

Organized under Background subtraction#Deep learning based.

LSTM

Organized under Long short-term memory.

Learning

Siamese Neural Networks for One-Shot Image Recognition

https://jayhey.github.io/deep%20learning/2018/02/06/saimese_network/

Training a network in deep learning requires a very large amount of training data. One-shot learning overcomes this drawback by training the network so that it can classify even with only one image per label.

NVIDIA AI Developer Newsletter

[Recommended] AI Can Transform Anyone Into a Professional Dancer

https://news.developer.nvidia.com/ai-can-transform-anyone-into-a-professional-dancer/

https://arxiv.org/abs/1808.07371

Transforming Standard Video Into Slow Motion with AI

https://news.developer.nvidia.com/transforming-standard-video-into-slow-motion-with-ai/

NVIDIA SPLATNet Research Paper Wins a Major CVPR 2018 Award

https://news.developer.nvidia.com/nvidia-splatnet-research-paper-wins-a-major-cvpr-2018-award/

AI Learns to Play Dota 2 with Human Precision

https://news.developer.nvidia.com/ai-learns-to-play-dota-2-with-human-precision/

[Recommended] This AI Can Automatically Remove the Background from a Photo

https://news.developer.nvidia.com/this-ai-can-automatically-remove-the-background-from-a-photo/

NVDLA Deep Learning Inference Compiler is Now Open Source

https://devblogs.nvidia.com/nvdla/

Nature

Deep learning of aftershock patterns following large earthquakes

https://www.reddit.com/r/MachineLearning/comments/9bo9i9/r_deep_learning_of_aftershock_patterns_following/

https://www.nature.com/articles/s41586-018-0438-y

https://drive.google.com/file/d/1DSqLgFZLuNJXNi2dyyP_ToIGHj94raWX/view

Cancer

Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening

https://medium.com/@jasonphang/deep-neural-networks-improve-radiologists-performance-in-breast-cancer-screening-565eb2bd3c9f

Text Generation

See Text Generation.

VizSeq - A Visual Analysis Toolkit for Text Generation Tasks

https://arxiv.org/abs/1909.05424

Favorite site

Machine learning

• Wikipedia (en) description of machine learning
• Machine Learning - OpenCV machine learning algorithms ²
• UNIVERSITY OF OXFORD: COMPUTER SCIENCE: Machine Learning: 2014-2015 Course materials (Oxford University machine learning course materials and videos)
• History of machine learning 1
• History of machine learning 2
• Parameter tuning and training
• Team AI Korea : We do Artificial Intelligence
• [Recommended] Mad for Simplicity - Enginius/Machine Learning (lots of machine learning content, neatly organized)
• [Recommended] HUB-AI - AI STUDY (lots of AI study materials)
• How a working programmer who gave up on math can start studying machine learning
• A look at basic math for machine learning by mingrammer
• [Recommended] Browse state-of-the-art
• [Recommended] Artificial Intelligence (AI) & Machine Learning (ML) Dictionary - WikiDocs

E-book

• [Recommended] 15 recommended free textbooks on robotics and machine learning/artificial intelligence
• The Element of Statistical Learning: Data Mining, Inference, and Prediction – T. Hastie, R. Tibshirani, J. Friedman (2009)
• Introduction to Machine Learning – A. Shashua (2008)
• Deep Learning – An MIT Press book in preparation - Y. Bengio (2015)
• Learning Deep Architectures for AI – Y. Bengio (2009)
• Gaussian Process for Machine Learning – C. Rasmussen, C. Williams
• Reinforcement Learning – C. Weber, M. Elshaw, N. Mayer (2008)
• Reinforcement Learning: An Introduction – R. Sutton, A. Barto (1998)
• Bayesian Reasoning and Machine Learning – D. Barber (2014)

Data analysis

• A data analyst dad flexing on youth soccer

Artificial neural network

• Wikipedia (en) description of artificial neural networks
• Neural networks - Data Mining ³
• Sigmoid Function of the Artificial neural network ⁴
• [Recommended] Neural network theory : Oh Chang-seok ⁵ (includes explanations of several Activation functions)
• MLWiki (will mainly explain neural networks, including Deep Learning)
• Study/Artificial Intelligence: Artificial Neural Network
• A Neural Network in 11 lines of Python (Part 1) (writes a Neural Network directly in Python code)
  • A Neural Network in 13 lines of Python (Part 2 - Gradient Descent)
  • Hinton's Dropout in 3 Lines of Python
  • Anyone Can Learn To Code an LSTM-RNN in Python (Part 1: RNN)

Convolutional neural network

• Wikipedia (en) description of CNN
• LeNet-5, convolutional neural networks
• TensorFlow - (5) MNIST - CNN
• [Recommended] VGG Convolutional Neural Networks Practical ⁶ (an example that works through CNN step by step)
• Convolutional Neural Networks (formulas for Convolutional, Max-Pooling, Fully-Connected, and Backpropagation)
• SlideShare: Backpropagation in Convolutional Neural Network
• Songorithm ML : Middle Learning = Shallow + Deep Machine Learning
  • 2015.11/9, Songorithm ML : Part 3 – Session 1 : Convolutional Networks (1) / (caffe) Hands-on caffe with Docker
    • Chapter 9 Convolutional Networks
• Gradient-based learning applied to document recognition
• Convolutional Neural Nets II Hands On ⁷
• Forward and Backward Propagation of Convolutional Layer ⁸
• Deep learning made easy to understand and use (summary of CNN concepts and samples using Theano)
• Backpropagation In Convolutional Neural Networks
• Convolutional Neural Networks backpropagation: from intuition to derivation
• [Recommended] CNN, Convolution Neural Network summary
• [Recommended] (CNN) Convolution Neural Network
• [Recommended] Backpropagation in CNN
• [Recommended] The easiest way to understand CNN backpropagation ⁹
• Back Propagation in Convolutional Neural Networks — Intuition and Code

Segmentation

• [Recommended] A LOOK AT IMAGE SEGMENTATION USING CNNS ¹⁰ (CNN)

Deep learning

• Wikipedia (en) description of deep learning
• Deep learning tutorial for beginners
• Korean dependency parsing using deep learning
• Deep Learning, let's take a quick look
• Welcome to Deep Learning
  • Convolutional Neural Networks (LeNet)
• [Recommended] Almost everything about Deep Learning, explained simply
• [Recommended] NVIDIA > Accelerated Computing > Deep Learning Courses
• [Recommended] Deep Learning - Seoul National University : Biointelligence lab ¹¹
• khanrc's blog - Deep Learning Tutorial
• Awesome Deep Learning (collection of resources)
• [Recommended] Must Know Tips/Tricks in Deep Neural Networks (by Xiu-Shen Wei) ¹²
• Taehoon Kim
  • Github - Taehoon Kim
  • (Translation) Handwriting recognition using a Neural Network
  • [Recommended] Slideshare - Deep and wide deep learning for intellectual conversation PyCon APAC 2016
• Youtube - FlowNet (Segmentation) Optical flow estimation - demos from supplementary material
• Youtube - Semantic Optical Flow (CVPR 2016)
• Multi-speaker Tacotron in TensorFlow (Origin: http://carpedm20.github.io/)
  • SlideShare - Deep learning that reads books aloud: what if actress Yoo In-na read Harry Potter to you DEVIEW 2017
• [Recommended] OpenResearch.ai ¹³ - deep learning papers are well organized.
• [Recommended] Papers With Code : the latest in machine learning - organizes papers together with code.
• Deep learning is not so mysterious or different | GeekNews - Generalization
• [Recommended] Deep Learning Bible - 4. Object Detection - Korean - WikiDocs - analyzes and summarizes many papers in Korean.

Deep learning libraries

• Wikipedia (en) Comparison of deep learning software
• Deep Learning Comp Sheet: Deeplearning4j vs. Torch vs. Theano vs. Caffe vs. TensorFlow
• Which Deep Learning Library should I choose?
• [Recommended] Evaluation of Deep Learning Toolkits

Andrew Ng

• [Recommended] Coursera - Machine Learning - Stanford - by Andrew Ng ¹⁴¹⁵
  • Learning machine learning with Corsera Machine Learning : week1
  • Learning machine learning with Corsera Machine Learning : week2
  • Learning machine learning with Corsera Machine Learning : week3
  • Learning machine learning with Coursera Machine Learning : week4
• [Recommended] README SanghyukChun's Blog ¹⁶¹⁷ (Andrew Ng analysis, Machine Learning study, etc...)
  • [Recommended] Machine Learning Study (1) What is Machine Learning?
  • Machine Learning Study (13) Clustering (K-means, Gaussian Mixture Model)
  • [Recommended] Machine Learning Study (18) Neural Network Introduction
  • [Recommended] Machine Learning Study (19) Deep Learning - RBM, DBN, CNN
• Old Lisper - Functional Programming World ¹⁸ (Andrew Ng analysis blog)
  • Machine Learning, Week 1

Machine Learning Exercises In Python

• Part 1 - Simple Linear Regression ¹⁹
• Part 2 - Multivariate Linear Regression ²⁰
• Part 3 - Logistic Regression ²¹
• Part 4 - Multivariate Logistic Regression ²²
• Part 5 - Neural Networks ²³
• Part 6 - Support Vector Machines ²⁴
• Part 7 - K-Means Clustering & PCA ²⁵
• Part 8 - Anomaly Detection & Recommendation ²⁶

Guides

• Deeplearning4j #2 example MNIST
• Classifying MNIST digits using Logistic Regression
• Deep Learning
• Samsung Software Membership: Artificial Neural Network #1 (a simple Neuron implementation in C++ code)²⁷
• Samsung Software Membership: Artificial Neural Network #2 (a simple Feedforward neural network implementation in C++ code) ²⁸
• Number plate recognition with Tensorflow (car license plate recognition)
• [Recommended] GPU: how much faster than CPU? ²⁹ (compares forward time on GPU and CPU for well-known deep learning model architectures)

Tutorials

• WikiDocs - Deep Learning theory and practice (under revision)
• [Recommended] WikiDocs - Machine Learning lecture notes
• [Recommended] Gihub - 1ambda - Data Analysis ³⁰
• UFLDL Tutorial (Unsupervised Feature Learning and Deep Learning)
• opentutorials - Machine learning
• [Recommended] GitBook - leonardoaraujosantos > Artificial Inteligence ³¹
• [Recommended] Other posts in the 'Machine learning_Professor Sung Kim' category
• [Recommended] 170915 Tensorflow seminar: Deep learning from theory to practice - POSTECH ³²
• A page summarizing the prerequisite knowledge needed to understand Reinforcement Learning
• [Recommended] Neural Networks : Zero to Hero - coding neural network development from scratch | GeekNews
  • A neural network development course made by Andrej Karpathy, former head of AI at Tesla
  • Develops everything from the basics of backpropagation up to modern deep neural networks like GPT
  • Language models are good for learning deep learning. Even if you later expand into areas like computer vision, most of what you learn applies immediately
  • [Original] Neural Networks: Zero To Hero
• A collection of AI Engineering resources | GeekNews
  • [Original] aie-book/resources.md at main · chiphuyen/aie-book
  • Only the most helpful picks from the 1200+ links and 1000+ AI Github Repos that Chip Huyen referred to while writing the "AI Engineering" book
• 「Machine Learning Study on Your Own」 | GeekNews
  • [Original] teddylee777/machine-learning: A repository made to help machine learning beginners and those preparing a study group. (This repository is intented for helping whom are interested in machine learning study)

Data pool

• Fueling the Gold Rush: The Greatest Public Datasets for AI ³³ (a collection of datasets for AI!)

Service

• Aifiddle - Design, Train & Share AI Models from your browser (lets you easily assemble deep learning models in the browser)
• PlayGround - Tinker With a Neural Network Right Here in Your Browser. Don’t Worry, You Can’t Break It. We Promise (the graph of data passed between the model's layers is displayed visually)
• DeepL Translate: The world's most accurate translator
  • DeepL - AI translation that sounds human, now supports Korean | GeekNews
  • OpenAI
  • Stability.AI
  • Hugging Face
  • Weights and Biases
• Kagi - Universal Summarizer
  • A "generate a summary of anything on the web" tool provided by the search engine Kagi
  • Universal Summarizer - summarizes anything for you | GeekNews

Sample images good for use in proposals

• [https://www.google.com/imgres?imgurl=https%3A%2F%2Fd3i71xaburhd42.cloudfront.net%2F791a9a8d8fe3110985211b3256b67c0e979fdf8a%2F2-Figure1-1.png&imgrefurl=https%3A%2F%2Fwww.semanticscholar.org%2Fpaper%2FImage-Sequences-Based-Traffic-Incident-Detection-Zou-Shi%2F791a9a8d8fe3110985211b3256b67c0e979fdf8a%2Ffigure%2F0&tbnid=U1aALMVaNKcjhM&vet=10CBAQxiAoAWoXChMImOranIz17gIVAAAAAB0AAAAAECg..i&docid=jy0zapjJfnlb9M&w=564&h=426&itg=1&q=Person%20Tracking%20&client=firefox-b-d&ved=0CBAQxiAoAWoXChMImOranIz17gIVAAAAAB0AAAAAECg#imgrc=zFEtt7J5NRRniM&imgdii=IsNk3wEGRScDPM Google Image Search] - samples of car object detection at intersections

Online demo

• MNIST Demo
  • Github - MNIST Demo
• MNIST Online demo
• Demos - Deep Learning
• Caffe Demos
• CNN Explainer

References