Skip to content
Search

Large language model

lastmod 2026-10-01created 2024-04-22127 edits

(For the machine learning method based on generating understandable rules, see Logic learning machine.)

A large language model (LLM), also called a giant language model, is a language model consisting of an artificial neural network with a huge number of parameters (usually billions of weights or more). It is trained on a considerable amount of unlabeled text using self-supervised or semi-self-supervised learning. LLMs emerged around 2018 and perform a wide range of tasks. The focus of natural language processing research has shifted away from the earlier training paradigm of specialized supervised models for specific tasks.

Large language models (LLMs) are an enabler of AI chatbot technology and one of the most talked-about topics. How large language models (LLMs) work is broadly divided into three parts: tokenization, transformer models, and prompts. Tokenization is part of natural language processing and refers to converting ordinary human language into sequences that low-level machine systems (LLMS) can understand; this involves assigning numeric values to sections and encoding them for fast analysis. It is like an AI version of phonetics, and the goal of tokenization is to create context vectors, which act like a training guide or formula for the AI to predict sentence structure. The more language is studied and the better it is understood how sentences are constructed, the more accurate the prediction of the next word in a particular type of sentence becomes. This makes it possible to develop models that reproduce the various communication styles people use online.

A transformer model is a type of neural network that examines sequential data to identify relevant patterns of which words are likely to follow each other, and it consists of layers that each perform different analyses to decide which words are compatible. These models do not learn language; they rely on algorithms to understand words written by people and are trained, for example, to write a standard article about coffee by being given hipster coffee blogs. This transformer model is the basis of large language model (LLM) language generation.

A prompt is the information developers give a large language model (LLM) to analyze and tokenize; prompts are essentially training data that helps the LLM across various use cases. The more accurate the prompts it receives, the better the LLM can predict the next word and construct accurate sentences. Therefore, choosing appropriate prompts is important for properly training deep learning AI.

Categories

AI-friendly GUI frameworks

Korean open-source LLMs

LLM API price-per-token comparison

Numbers every LLM developer should know

A summary of numbers that matter when using LLMs.

  • Adding "be concise" to a prompt can save 40–90% of the cost.
  • GPT-3.5 Turbo is 50 times cheaper than GPT-4.
  • Using OpenAI embeddings for vector search is 20 times cheaper than GPT-3.5 Turbo.
  • Training a LLaMa-class LLM costs a million dollars (about 1.3 billion won).
  • Memory size per GPU - V100: 16GB, A10G: 24GB, A100: 40/80GB H100: 80GB
  • You usually need twice the model size in memory - 7B = 14GB
  • Embedding models usually use 1GB of memory or less
  • Batching LLM requests can make them more than 10 times faster.
  • A 13B model needs about 1MB per token, so batching requests greatly increases memory requirements.

Language Model Stack

Almost every company Sequoia has invested in is bringing LLMs like ChatGPT into its products. Based on a survey of companies in its network, Sequoia outlines two AI stacks

LLM API Stack

  • LLM APIs : OpenAI, Anthropic, Cohere
  • Vector DB: AWS, pgvector, Pinecone, Qdrant, Weaviate, Chroma
  • Frameworks: LangChain, LlamaIndex, Anarchy
  • Text-To-Speech: Resemble.AI, ElevenLabs, Wellsaid
  • Monitoring: DISTYL, Guardrails, Helicone, HoveHive, Autoblocks

Custom Model Training / Tuning Stack

  • Compute: AWS, Azure, Google Cloud, Foundary, Lamba, MosaicML, Modal, Fireworks.ai
  • Model Hub: Hugging Face, Replicate
  • Frameworks: PyTorch, TensorFlow
  • Experimentation: Weights & Biases
  • Monitoring/Observability: Robust Intelligence, Gantry, Arthur, arise, Whylabs
  • Hosting: Replicate, HuggingFace

Key points

  • Most companies are building LLMs into their products
  • Focus on language model APIs, retrieval, and orchestration. Open-source use is growing
  • Companies want to customize language models to fit their own context
  • The LLM API stack and the model training stack look separate now, but they will gradually merge
  • The stack is becoming more and more developer-friendly
  • For language models to be fully adopted, their trustworthiness (quality, data privacy, security) must improve
  • Language model applications will become increasingly multimodal
  • It is still very early days

How to fine-tune

Fine-tuning with LoRA and Adapter using Lit-Parrot, an open-source implementation based on nanoGPT

Text embeddings: Parquet and Polars

Text embeddings are vectors generated by large language models, a way of representing words, sentences, and documents numerically

14 things to keep in mind when writing code with LLMs

  1. Set reasonable expectations (Set reasonable expectations)
  2. Consider the training data cut-off date (Account for training cut-off dates)
  3. The importance of context (Context is king)
  4. Ask for multiple options (Ask them for options)
  5. Give specific instructions (Tell them exactly what to do)
  6. Always test the code it writes (You have to test what it writes!)
  7. Keep the interaction conversational (Remember it’s a conversation)
  8. Use tools that can run code (Use tools that can run the code for you)
  9. Learn through ‘vibe coding’ (Vibe-coding is a great way to learn)
  10. A detailed example with Claude Code (A detailed example using Claude Code)
  11. The need for human intervention (Be ready for the human to take over)
  12. Maximizing development speed (The biggest advantage is speed of development)
  13. Amplifying existing expertise (LLMs amplify existing expertise)
  14. Bonus: answering questions about codebases (Bonus: answering questions about codebases)

How to improve recommender systems and search in the age of LLMs

See Recommender system#How to improve recommender systems and search in the age of LLMs.

What is the best LLM you can use on consumer hardware?

Looking for a model capable of basic conversation on a 5060ti + 16GB VRAM. Ideally fast, running in near real time

Summary of answers

  • Various 8B–14B and 30B parameter models run efficiently on 16GB VRAM; Qwen3, DeepSeek-R1, Mistral, Gemma3 and others are recommended
  • Running LLMs locally has advantages in performance, cost, and privacy, but actual performance and model fit require individual experimentation and tuning
  • Tips for optimizing hardware use, such as model file size, quantization level (Q4–Q6, etc.), and splitting loading across GPU and RAM, are actively shared
  • Many tools exist, such as Ollama, LM Studio, llama.cpp, and OpenWebUI, each with pros and cons in accessibility, flexibility, and ease of model management
  • Community information (e.g. Reddit LocalLLaMA) is useful for the latest news and practical tips, but there is also a lot of exaggeration and misinformation, so be careful
  • Qwen3: available in various parameter sizes such as 8B/14B/30B; the 8B–14B models run comfortably on 16GB VRAM. Strong reasoning performance, and thanks to the MoE (Expert Mixture) architecture some models can run even at large sizes with RAM offloading
  • DeepSeek-R1-0528-Qwen3-8B: regarded as having excellent reasoning performance among recent 8B models. At 8B, fits in 4GB–8GB VRAM with Q4–Q6 quantization
  • Mistral Small 3.1: the 14B or 24B model is recommended; good conversation quality and relatively little censorship. Notably, it supports image input
  • Gemma3: a Google model, strong at intuitive conversation. However, some say it has a strong HR-like tendency and adds many disclaimers. Hallucination is also relatively frequent
  • Devstral: a large model based on Mistral. 30B and above can be slow on 16GB VRAM
  • Dolphin, Abliterated: less-censored versions, useful for non-routine situations

Optimizing hardware and runtime environment

  • Quantization settings: the lower the quantization level, such as Q4, Q5, Q6, the less VRAM is used (Q4 ≒ parameters/2, Q6 ≒ parameters*0.75). Watch out for quality degradation, though
  • Estimating VRAM capacity: e.g. 8B Q4 needs 4GB, 14B Q4 needs 7GB, and 30B Q4 needs about 15GB of VRAM
  • RAM offloading: when VRAM is short, some layers can be offloaded to CPU memory. Expect a slowdown, though
  • KV cache quantization: when enlarging the context window, compressing the cache to around q4 is recommended

Tools and frontends

  • llama.cpp: runs fast and flexibly on many platforms. Supports a REST API and a simple React frontend. Can load a model split across VRAM and RAM
  • Ollama: easy installation and model switching, easy integration with GUI frontends. However, it has limits on support for the latest models and context size
  • LM Studio: convenient model management in a GUI. Predicts whether a model fits in VRAM
  • OpenWebUI: frontend only. Requires a backend such as llama.cpp or vllm. Can manage and test multiple models at once
  • KoboldCPP, SillyTavern: frontends specialized for role-playing/storytelling/games

Community and practical information

  • Reddit LocalLLaMA, HuggingFace, Discord: news on the latest models, usage, benchmarks, and setup know-how are actively shared. However, watch out for misinformation and groupthink
  • Benchmark sites: livebench.ai, aider.chat, and others provide scores and rankings for the latest models

Use cases and real experience

  • Privacy, cost savings: local models are more useful than the cloud for sensitive data/privacy concerns or repeated use
  • Freedom to experiment and tune: more flexible than API models for domain-specific fine-tuning, sampling strategies, prompt engineering, and so on
  • Applications: many practical examples such as RAG (retrieval-augmented generation), local database integration, agent automation, and offline assistants

Frequently asked questions and tips

  • Estimating model size: number of parameters × bits (quantization)/8 = approximate VRAM requirement (GB). Overhead and the context window also need to be considered
  • Characteristics by model: Qwen3 for reasoning/coding, Gemma3 for intuition/conversation, Mistral with less censorship, Dolphin/abliterated as uncensored versions, etc.
  • Performance comparison: running your own benchmarks and custom tests to find the model that suits you is recommended

Conclusion and practical advice

  • There is no "best model"; the best approach is to try a variety of recent 8B–14B models such as Qwen3, Mistral, and Gemma3 depending on your hardware, use case, and preferences
  • Matching specs such as model file size, quantization, and context size is very important, so it is effective to test several models yourself and make use of community tips

See also

Favorite site

Tutorials

Guide

Article

References

  1. What_Weve_Learned_From_A_Year_of_Building_with_LLMs_-_GeekNews.pdf ↩

author Zer0 sitemap