Kiwi

lastmod 2024-12-01created 2024-12-013 edits

Intelligent Korean morphological analyzer (Korean Intelligent Word Identifier)

Features

• Kiwi : Intelligent Korean morphological analyzer (Korean Intelligent Word Identifier)
• Korean morphological analyzer library aiming for high speed and general-purpose performance
• Has a built-in lightweight language model, so it analyzes morphemes fairly accurately even when there is ambiguity
• Implemented in C++. The developer is a fan of C++.
  • The library is provided for Python, JavaScript, C#, Java, R, and Go
  • Compiled to wasm, so it can run in the browser
• Models come in three sizes: small (16MB), medium (40MB), and large (90MB)
• The development process is published in detail on a blog
  • https://bab2min.tistory.com/560
• Korean language model training using Kiwi as the tokenizer is also impressive
  • https://github.com/bab2min/kiwi-farm

See also

• Natural language processing (NLP)
• KoNLPy (Korean: 코엔엘파이) - A Python package for Korean language processing.

Favorite site

• bab2min/Kiwi: Kiwi (Intelligent Korean morphological analyzer)