Skip to content
Search

Llama.vim

lastmod 2025-01-25created 2025-01-251 edits

An LLM-powered code/text autocompletion plugin for VIM

Features

  • Provides local LLM-based text autocompletion inside VIM
  • Automatic suggestions when moving the cursor in Insert mode
  • Toggle manual suggestions with Ctrl+F
  • Accept a suggestion with Tab
  • Accept the first line of a suggestion with Shift+Tab
  • Configurable maximum text generation time
  • Configurable context range around the cursor
  • Builds a context ring from context chunks taken from open files, files being edited, and yanked text
  • Supports very large contexts even on low-end hardware through smart context reuse
  • Displays performance stats

Installation

llama.cpp setup

  • The plugin requires a llama.cpp server instance running at g:llama_config.endpoint
  • Mac OS : brew install llama.cpp
  • Other OS : build from source or use the latest binaries: llama.cpp releases
  • Recommended settings by VRAM size:
    • 16GB VRAM or more: Qwen2.5-Coder-7B-Q8_0-GGUF
    • Less than 16GB VRAM: Qwen2.5-Coder-3B-Q8_0-GGUF
    • Less than 8GB VRAM: Qwen2.5-Coder-1.5B-Q8_0-GGUF
  • See :help llama for details
  • Requires FIM-compatible models: HF collection

Implementation details

  • The plugin aims to be simple and lightweight, providing high-quality, performant local FIM completion even on consumer-grade hardware

Hacker News comments

  • Llama.vim – Local LLM-assisted text completion | Hacker News
  • Recommends looking into the technical details of the server implementation, which include interesting and clever ideas. This plugin is also available for VS Code. Performance improved thanks to client-side caching.
  • This person has contributed a lot to the open-source AI ecosystem; hopes they raise enough funding to keep developing software and release it as truly "no strings attached" open source.
  • Thinks that integrating LLM tools more tightly with LSPs, compilers, and other static analysis tools could provide more context and better output. It would be nice to fine-tune LLMs per language and ship them alongside common editor tooling. Points out that AI tools work better with older, popular languages, which can become an important factor when choosing a language. Would like to install a Gleam-specific model that takes data from the LSP and compiler so it doesn't generate invalid syntax.
  • Wonders how the "ring context" works, and suggests a solution for managing the KV cache from a similar project: keep a tree of tokens and store full LLM state snapshots at fixed depth intervals, so that only a few tokens need to be "replayed" when the buffer changes. Thinks there may be mathematical properties in how the important parts of the state work.
  • The blinking cursor in the demo video causes heart palpitations, but thinks it's very cool. Wonders how Linux works on M* hardware.
  • Wonders whether this plugin is the same as the VSCode version.
  • Wonders what will be possible over the next 10 years with consumer hardware and improved quantization. Currently even a 24GB GPU can't match the performance of hosted services.
  • Prefers coding in the terminal; when stuck, can use askds to do deeper analysis in the terminal.
  • Wonders whether llama can be used on mid-range hardware; thinks it crashes due to insufficient RAM. Feels 2G VRAM and 16G system RAM are not enough; most Apple products work thanks to unified memory, but otherwise thinks you need an expensive Nvidia GPU with lots of VRAM. Wonders if there are cheaper options.
  • Asks for advice from someone who knows this field; looking for a way to buy a reasonably priced graphics card to run local LLMs.

See also

  • llama.vscode (VS Code)

Favorite site

author Zer0 sitemap