Skip to content
Search

Dataset

lastmod 2026-07-27created 2018-07-2457 edits

A data set (or dataset) is a collection of data.

Generally, a data set corresponds to the contents of a single database table or a single statistical data matrix, where every column of the table represents a particular variable and each row corresponds to a given member of the data set in question. The data set lists values for each of the variables, such as the height and weight of an object, for each member of the data set. Each value is called a datum. The data set may comprise data for one or more members, corresponding to the number of rows.

The term data set may also refer to the data in a collection of closely related tables corresponding to a particular experiment or event. An example of this kind is the data sets collected by a space agency performing experiments with instruments aboard space probes.

in Machine Learning

Dataset-training_validation_testing.png

Validation vs Test

  • Why use a validation set in Machine Learning
  • The test set is used to evaluate the model's 'final performance', and differs in that it is not involved in the training process.
  • The validation set, on the other hand, can be seen as taking part in the performance evaluation used to select the final model among several models.
  • Therefore, the validation set is involved in the Training process.

In other words, the validation set is involved in the training process and is the set used to pick the single best model among several trained models. The test set is the set used to finally evaluate the model's performance after all training is complete. If the test set is used to improve the model, it is not a test set but a validation set. If you don't want to evaluate several models and select the best one among them, you don't need to create a validation set. In that case, however, problems will arise. (You can neither predict test accuracy nor prevent overfitting through model tuning.)

Data pools

This section also collects sample datasets for machine learning.

General

  • AI-Hub - AI Hub (Ministry of Science and ICT)
  • Data.gov - US federal public data, includes 300,000 datasets totaling 16TB
  • Datasets - The largest hub of ready-to-use datasets for ML models, with fast, easy-to-use, and efficient data manipulation tools, from HuggingFace.

Image

Image (OCR)

Video

Korean: 라이다 (LiDAR) - Driving sequences (Point Cloud)

  • PandaSet - CC BY 4.0 (commercial use explicitly allowed) - 103 sequences × 80 frames, cuboid + point segmentation labels
  • ZOD (Zenseact) - CC BY-SA 4.0 - 100,000 frames + sequences, 3D boxes, multi-LiDAR·radar - Commercial use is allowed, but note that the SA (ShareAlike) condition propagates.

Korean: 라이다 (LiDAR) - Non-driving (Point Cloud)

  • AHN4/AHN5 (Netherlands) - CC0 - Nationwide aerial LiDAR LAZ, includes ASPRS classification labels (ground/buildings/water, etc.)
    • Can be loaded straight into existing LAS/LAZ pipelines.
    • Download in small tiles from GeoTiles.nl — ideal for semantic labeling examples
  • USGS 3DEP - Public domain (US federal) - US aerial LiDAR LAS/LAZ, includes basic classification
    • Area-based extraction is available on OpenTopography. An alternative filling the same role as AHN
  • Redwood 3D Scans - Public domain - 10,933 scans of everyday objects, PLY meshes/RGB-D
    • No labels, but the small per-object PLY files make it a good starting point for 3D editor demos and cuboid labeling

Non-commercial licenses

  • KITTI·SemanticKITTI·nuScenes·Waymo·Argoverse 2 (CC BY-NC-SA family)
  • MOT Challenge (CC BY-NC-SA 3.0)
  • PersonPath22 (CC BY-NC)
  • Toronto-3D (CC BY-NC 4.0)
  • Paris-Lille-3D (CC BY-NC-ND 3.0)

Regression

Tracking

License Plate Recognition

Pedestrian detection

Face recognition

Car

Parking lot

Korean

  • KorQuAD: KorQuAD 2.0 is a Korean Machine Reading Comprehension dataset of 100,000+ pairs in total, including the 20,000+ question-answer pairs from KorQuAD 1.0. Unlike KorQuAD 1.0, answers must be found in an entire Wikipedia article rather than in 1-2 paragraphs. Since some documents are very long, search time needs to be taken into account. Tables and lists are also included, so understanding document structure through HTML tags is needed as well. This dataset should make machine reading comprehension possible on documents of various forms and lengths.
  • Nemotron-Personas-Korea - Dataset of 1 million synthetic personas based on South Korea's actual population distribution

NLP

Fire

Fall detection (Activity Recognition)

Human Pose

Injured civilians

Road accident

Weld

Point Cloud

Other time series data

Natural language

The General Index - An n-gram index of a million journals released for free
https://archive.org/details/GeneralIndex
Researcher Carl Malamud released an n-gram index extracted with SpaCy from 107,233,728 journals, including paywalled papers
Avoids copyright restrictions because the index contains only sentence snippets from 1 to at most 5 words long, not full text
Released for free on the web archive for use in various research fields
  • e.g.) How many times a particular chemical has been used in papers
Consists of 3 tables
  • 350 billion n-grams and journal ids
  • 19.7 billion keywords and journal ids
  • Journal ids and metadata : paper title, authors, DOI (unique paper identifier)
The catalog is 5TB of compressed files, 38TB when extracted

ETC

See also

Favorite site

References

  1. Handong1587-2015-09-24-Computer_Vision_Datasets.md.zip ↩

author Zer0 sitemap