Skip to content
Search

Mi:dm

lastmod 2025-07-11created 2025-07-111 edits

KT's in-house open-source LLM

About

  • "Mi:dm" is an open-source model available for commercial use that reflects the linguistic/cultural characteristics of Korean society
  • Uses a multi-layered optimization strategy including curation of high-quality Korean data, synthetic data generation, curriculum learning, and a proprietary Korean-specialized tokenizer
  • Three models: mini (2.3B) for on-device use, base (11B) balancing performance and efficiency, and frontier-class pro (41B, to be released)
    • Mi:dm 2.0 Mini (2.3B): a lightweight model optimized for embedded environments and special purposes
    • Mi:dm 2.0 Base (11.5B): a large general-purpose model that deepens an existing 8B model with the Depth-up Scaling technique to boost performance
    • Both Base and Mini support 32K-token input
  • Shows top-level performance on Korean benchmarks such as KMMLU and HAERAE, and is released under the MIT license, free for both research and commercial use

Data composition and strategy

  • Focuses on securing high-quality Korean documents, selected by contextuality, readability, and non-toxicity criteria
  • Uses synthetic data (translation, keyword-based textbook generation, Chain-of-Thought, etc.) to secure domain diversity
  • Resolves imbalance in the training data with curriculum learning and domain balancing
  • A Korean-optimized tokenizer strengthens compression efficiency and reflects language structure
  • Data classification scheme
    • Applies a multi-dimensional classification scheme covering language, domain, data source, expression/style, etc.
    • 6 major domains (humanities, STEM, applied sciences, health/food, lifestyle/culture, other) and 20 subdomains
    • Over 85.7% consists of natural (organic) data, 14% synthetic data
  • Quality control pipeline
    • 8-stage large-scale web document filtering: deduplication, heuristics, perplexity, character corruption/repair, model-based quality filter, toxicity filter, line deduplication, PII de-identification, etc.
    • Separate cleaning and rules applied per source (e.g., news, legal documents, academic papers)
  • Synthetic data generation
    • For low-coverage fields such as STEM and economics, high-reliability open-source data is used as seeds to synthesize Korean textbooks/explanations/problems, etc. to strengthen the data
    • Even unusable (ineligible) web documents are used by extracting and rewriting only their core topics
    • Converts and expands the structural diversity of English web documents into Korean to secure long-form QA and writing data
    • Chain-of-Thought data strengthens step-by-step reasoning training for math, code, etc.

Model architecture and training

  • Transformer decoder-only architecture
  • Base: 8B model → Depth-up Scaling (32→48 layers) → expanded to 11.5B, with two-stage continual training on high-quality data
  • Mini: compresses Base's knowledge through width pruning and multi-stage distillation, enabling efficient inference
  • Supports up to 32,768 input tokens through Long-context training
  • Incorporates recent techniques such as GQA, SiLU, and RoPE

See also

Favorite site

author Zer0 sitemap