Ollama

lastmod 2026-04-03created 2023-12-1845 edits

Get up and running with large language models.

Run Llama 3, Phi 3, Mistral, Gemma, and other models. Customize and create your own.

Get Llama 2 and other large language models up and running locally

Categories

• Ollama:Python (Ollama Python Library)
• Cactus - Ollama for smartphones

Examples

• AutoGen#Ollama Coding Example - How to connect Ollama in AutoGen

Model Library

• ollama/ollama # Model Library

VRAM Requirements

• Running a 7B model requires at least 8GB of RAM, -> suitable on an NVIDIA GeForce RTX 3070 Ti (8G VRAM).
• Running a 13B model requires 16GB, -> suitable on two NVIDIA GeForce RTX 3070 Ti (8G VRAM) cards.
• Running a 33B model requires 32GB.

Even if two cards are installed in parallel, no separate configuration is needed. It picks them up on its own.

Models that can run on two NVIDIA GeForce RTX 3070 Ti (8G VRAM) cards

• codellama:13b
• deepseek-r1:14b

Coding Banchmarks

The process of finding... the best performer for coding

• CodeLlama-34b-Instruct - A model fine-tuned specifically for coding with excellent code generation, but it may be hard to run even on two 3070 Ti cards.
• DeepSeek Coder-33b-Instruct - A coding-specialized model, but with high GPU memory requirements.
• CodeLlama-13b-Instruct - A mid-sized model, smaller than 34b but still offering excellent coding performance, and it can run on two GPUs.
• WizardCoder-15b-V1.0 - A coding-specialized mid-sized model with excellent performance.
• Phind-CodeLlama-34b-v2 - A model optimized for coding, but with high VRAM requirements.

Considering a setup with two RTX 3070 Ti cards, CodeLlama-13b-Instruct or WizardCoder-15b-V1.0 would be the most suitable choice. Both models offer excellent coding performance while being runnable within the hardware specs. They are especially strong at code understanding, generation, and debugging.

7B lineup

• CodeLlama-7b-Instruct - A coding-specialized model developed by Meta; a smaller version of the larger models, but very effective at code generation and understanding.
• WizardCoder-7b-V1.0 - A model optimized for coding tasks, especially strong at Python code generation.
• DeepSeek Coder-7b-Instruct-v1.5 - A model specialized in code understanding and generation, supporting a variety of programming languages.
• Phind-CodeLlama-7b-v2 - A fine-tuned model specialized in code generation.
• CodeGemma-7b - A coding-specialized model developed by Google, released recently.

Among these, CodeLlama-7b-Instruct and DeepSeek Coder-7b-Instruct-v1.5 show the most balanced performance. DeepSeek Coder in particular performs well on coding benchmarks, so it can be called the best choice among 7B models. On an RTX 3070 Ti, models of this size can run well enough even without quantization, but applying quantization at around Q5_1 for memory efficiency gives faster response times.

CLI

ollama Enter:

Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve       Start ollama
  create      Create a model from a Modelfile
  show        Show information for a model
  run         Run a model
  pull        Pull a model from a registry
  push        Push a model to a registry
  list        List models
  cp          Copy a model
  rm          Remove a model
  help        Help about any command

Flags:
  -h, --help      help for ollama
  -v, --version   Show version information

Use "ollama [command] --help" for more information about a command.

Running with Docker

docker run -d --gpus=all -v /home/zer0/ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Using the GPU

• How to use Ollama: from installation to running local LLM models

It is used automatically. However, when the memory limit is reached, it runs in a mode shared with the CPU. Check with ollama ps.

When running in GPU-only mode:

NAME         ID              SIZE      PROCESSOR    UNTIL
gemma2:2b    8ccf136fdd52    3.6 GB    100% GPU     4 minutes from now

When running in shared CPU/GPU mode:

NAME                          ID              SIZE     PROCESSOR          UNTIL
llama3.3:70b-instruct-q2_K    a6f03da15cbc    28 GB    18%/82% CPU/GPU    4 minutes from now

Environment variables

• Installing and operating Ollama : easily running large language models in a local environment
• Global Configuration Variables for Ollama · Issue #2941 · ollama/ollama
• ollama/envconfig/config.go at main · ollama/ollama

Can be checked with ollama serve --help:

Environment Variables:                                                                                     
      OLLAMA_DEBUG               Show additional debug information (e.g. OLLAMA_DEBUG=1)                   
      OLLAMA_HOST                IP Address for the ollama server (default 127.0.0.1:11434)                
      OLLAMA_KEEP_ALIVE          The duration that models stay loaded in memory (default "5m")             
      OLLAMA_MAX_LOADED_MODELS   Maximum number of loaded models per GPU                                   
      OLLAMA_MAX_QUEUE           Maximum number of queued requests                                         
      OLLAMA_MODELS              The path to the models directory                                          
      OLLAMA_NUM_PARALLEL        Maximum number of parallel requests                                       
      OLLAMA_NOPRUNE             Do not prune model blobs on startup                                       
      OLLAMA_ORIGINS             A comma separated list of allowed origins                                 
      OLLAMA_SCHED_SPREAD        Always schedule model across all GPUs                                     

      OLLAMA_FLASH_ATTENTION     Enabled flash attention                                                   
      OLLAMA_KV_CACHE_TYPE       Quantization type for the K/V cache (default: f16)                        
      OLLAMA_LLM_LIBRARY         Set LLM library to bypass autodetection                                   
      OLLAMA_GPU_OVERHEAD        Reserve a portion of VRAM per GPU (bytes)                                 
      OLLAMA_LOAD_TIMEOUT        How long to allow model loads to stall before giving up (default "5m")    

Combining with PandasAI

• Ollama #1: Running unrestricted LLMs on my own computer without worrying about cost | AI&GameDev
• Ollama #6: Testing PandasAI's powerful data analysis features with GPT and local LLMs | AI&GameDev

Using Korean

• Running Llama3 with Ollama (2/2): Using a Korean Llama3 from HuggingFace
• A Korean-language local LLM using Ollama
• Installing Ollama and a Korean LLM model locally :: Patdeureotsyu

Embedding models

• Embedding models · Ollama Blog
• chromadb

Ollama supports embedding models, making it possible to build retrieval augmented generation (RAG) applications that combine text prompts with existing documents or other data.

What are embedding models?

Embedding models are models that are trained specifically to generate vector embeddings: long arrays of numbers that represent semantic meaning for a given sequence of text. The resulting vector embedding arrays can then be stored in a database, which will compare them as a way to search for data that is similar in meaning.

Quantization options

Quantization options are a technique for reducing the memory usage of large language models (LLMs), and Ollama provides various quantization levels. These options make it possible to run large models even in environments with limited GPU memory.

Ollama's main quantization options:

• Q4_0: The lightest quantization level; greatly reduces memory usage but may lose some accuracy.
• Q4_1: Uses slightly more memory than Q4_0 but offers better accuracy.
• Q5_0: Mid-level quantization offering a balance between memory and accuracy.
• Q5_1: Slightly more accurate than Q5_0, with slightly higher memory usage.
• Q8_0: Reduces model size while maintaining high accuracy. Uses more memory but has the highest accuracy.
• F16: Uses half-precision floating point without quantization. Uses the most memory but has no quality loss.

How to use quantized models in Ollama

Set the quantization when downloading the model (e.g. quantizing CodeLlama-13b to Q4_0)

ollama pull codellama:13b-instruct-q4_0

Or set it directly in the Modelfile

FROM codellama:13b-instruct
QUANTIZE q4_0

To reduce memory usage, you can also consider running a quantized version using the --quantize option in Ollama.

Ollama Turbo

• Ollama Turbo | GeekNews
  • [Original] Ollama Turbo
• Ollama Turbo is a new solution for running large open models quickly on high-performance datacenter hardware ($20/month)
• Provides a way to run large models quickly and efficiently while still using Ollama's app, CLI, API, and JavaScript/Python libraries as-is
• Currently in preview, supporting OpenAI's open models gpt-oss-20b and gpt-oss-120b
• When using Turbo mode, query history is not stored on the server or logged
• Hardware is operated only in datacenters in the US, hourly and daily usage limits apply, and usage-based billing is planned for the future

Ollama alternatives and comparison

• Installing and operating Ollama : easily running large language models in a local environment

ollama

Pace of development - Developing rapidly, with active community support.

Strengths - Ease of use, support for many models, efficient resource management

Weaknesses - May be limited compared to vLLM in some advanced optimization features

vLLM

Pace of development - Developing rapidly on an academic background, with a focus on performance optimization.

Strengths - High-performance inference, memory efficiency, large-scale batch processing

Weaknesses - Complex configuration; may be less user-friendly than ollama

LocalAI

Pace of development - Developing steadily, focusing on support for many models.

Strengths - Support for many model types (text, speech, image), easy Docker-based deployment

Weaknesses - May lag behind vLLM or ollama in some of the latest optimization techniques

FastChat

Pace of development - Developing actively, focusing on support for many open-source models.

Strengths - Support for many open-source models, provides a web UI

Weaknesses - May be less optimized than ollama in terms of resource management

See also

• LLM
• llama
• LangChain - Library that connects LLMs to the outside world
• Open WebUI
• LM Studio
• ollama - Ease of use, support for many models, efficient resource management
• vLLM - High-performance inference, memory efficiency, large-scale batch processing
• LocalAI - Support for many model types (text, speech, image)
• FastChat - Support for many open-source models, provides a web UI
• Dolphin MCP
• Microsoft AutoGen - Framework for next-generation LLM applications
• Shimmy - Lightweight, privacy-focused OpenAI API that can replace Ollama

Favorite site

• Ollama web site -> Ollama web site
• Github - jmorganca/ollama
• Github - ollama/ollama
• Why stop paying for ChatGPT Plus? - YouTube - Nomad Coders
• 09. Ollama - <LangChain Notes> - LangChain Korean Tutorial🇰🇷
• How to use Ollama: from installation to running local LLM models

with MCP

• Anthropic MCP with Ollama, No Claude? Watch This! - YouTube
  • Github - chrishayuk/mcp-cli
  • Lilys AI summary
• MCP server using Ollama for local LLMs - YouTube
  • Lilys AI summary

Articles

• Ollama announces a new engine for multimodal models | GeekNews
  • Ollama's new engine for multimodal models · Ollama Blog
• Do you use open LLMs and coding assistants locally? Share your setup | GeekNews