Language model
lastmod 2024-12-12created 2024-12-124 edits
A language model is a probability distribution over sequences of words. Given a sequence of words of length m, a language model assigns a probability to the whole sequence. Because languages can be used to express an infinite variety of valid sentences, language modeling faces the problem of assigning non-zero probabilities to linguistically valid sentences (which may never have been encountered in the training data). Various modeling approaches have been designed to overcome this problem, such as applying the Markov assumption or using neural architectures such as recurrent neural networks or transformers.
Language models are useful for a variety of problems in computational linguistics.
About
A language model (LM) is a model trained to output the statistically[1] most appropriate output based on an input (natural language, usually the user's sentence). Language models (LMs) of a much larger scale are called LLMs (Large Language Models), and models with an enormous number of parameters, such as GPT-4, fall into this category. Conversely, smaller ones are called SLMs (small Language Models); they can handle a narrower scope than LLMs but cost less.[2] LLMs mainly run on mainframes and supercomputers, while SLMs can run on workstations and even some high-performance PCs.
Categories
- Large language model (Korean: 대규모 언어 모델; LLM) - Most deep learning-based projects are organized here.
See also
- Korean: 자연어 처리
- Transformer (Korean: 기계 학습)
- Statistical model
- DSPy - Framework for programming (rather than prompting) language models