Data mining

lastmod 2026-03-25created 2015-12-2910 edits

In a word, data mining is defined as extracting useful information from large data sets¹.

About

A more detailed definition is as follows.²

데이터마이닝이란 의미있는 패턴과 규칙을 발견하기 위해서 자동화되거나 반자동화된 도구를 이용하여 대량의 데이터를 탐색하고 분석하는 과정이다.

As another definition of data mining, the Gartner Group defines it as follows.³

데이터마이닝은 통계 및 수학적 기술뿐만 아니라 패턴인식 기술들을 이용하여 데이터 저장소에 저장된 대용량의 데이터를 조사함으로써 의미있는 새로운 상관관계, 패턴, 추세 등을 발견하는 과정이다.

Data mining process

1. Problem definition
2. Data acquisition and selection
3. Data exploration
4. Data preprocessing
5. Train and evaluate data mining model
6. Interpret results

Data set

• What is the difference between test set and validation set?

Training set

a set of examples used for learning: to fit the parameters of the classifier In the MLP case, we would use the training set to find the “optimal” weights with the back-prop rule

Validation set

a set of examples used to tune the parameters of a classifier In the MLP case, we would use the validation set to find the “optimal” number of hidden units or determine a stopping point for the back-propagation algorithm

Test set

a set of examples used only to assess the performance of a fully-trained classifier In the MLP case, we would use the test to estimate the error rate after we have chosen the final model (MLP size and actual weights) After assessing the final model on the test set, YOU MUST NOT tune the model any further!

Why separate test and validation sets?

The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model After assessing the final model on the test set, YOU MUST NOT tune the model any further!

See also

• Cross validation
• Korean: 빅데이터 (Bigdata)
• Korean: 빅데이터 플랫폼 (Big Data Platform)
• Korean: 데이터 웨어하우스 (Data Warehouse)
• Korean: 데이터 레이크 (Data Lake)
• Korean: 데이터 레이크하우스 (Data Lakehouse)
• Concept drift (Korean: 컨셉 드리프트)
  • Model Drift (Korean: 모델 드리프트)
  • Drift Detection (Korean: 드리프트 감지; Korean: 드리프트 디텍션)

Favorite site

• Data mining - Wikipedia
• Multiple linear regression - Data mining process
• Overview of data mining
• Machine Learning Chapter 10 - Training-Testing data set -
• Model evaluation, model selection, and algorithm selection in machine learning – Chapter 3. Cross-validation and hyperparameter tuning
• 18. Training-Testing data set (lec 07-2)
• [Recommended] What is the difference between test set and validation set?
• How can you avoid overfitting (Over-fitting)? (Training vs. Validation vs. Test set)

References