Retrieval-Augmented Generation

lastmod 2026-09-04created 2024-04-2618 edits

Retrieval-augmented generation (RAG) is an architectural approach that improves relevance by bringing data into the context of a large language model (LLM).

Categories

• RAG:MLOps
• Chonkie - A fast, lightweight Python text chunking library for RAG
• KAG (Knowledge Augmented Generation)
• Kotaemon - An open-source RAG-based tool that lets you chat with your documents
• Flamehaven FileSearch - A self-hostable open-source RAG document search engine

Building a RAG system with PostgreSQL

• Building a RAG system with PostgreSQL | GeekNews
  • [Original] RAG With PostgreSQL - pgDash

With a Retrieval-Augmented Generation (RAG) system, you can build an AI assistant that answers questions based on an existing internal knowledge base (wikis, manuals, training and reference materials, etc.)

A RAG system can be built with just PostgreSQL, pgvector, ollama, and under 200 lines of Go code

Overview

• Uses a few paragraphs of a story as the "document corpus" and generates a document embedding for each document using Meta's Llama3 (hosted locally with ollama)
• Stores documents and embeddings in a PostgreSQL table, storing and accessing embeddings through the pgvector extension
• For a user query, retrieves the single most relevant document from the table and generates a response with Llama3
• ollama provides an OpenAI-like HTTP API that generates embeddings and chat responses
• The Go code talks to Postgres using jackc/pgx and pgvector-go, and handles HTTP API calls using the ollama client API package

Running models with Ollama

• Ollama is a tool that lets you run open-source models locally, and provides an OpenAI-style REST API
• Run the llama3 model with the ollama pull llama3 command
• ollama's HTTP server is available at 127.0.0.1:11434 by default

Installing pgvector

• pgvector is an extension for PostgreSQL versions 12–16; when using the pgdg APT repository it can be installed with sudo apt install postgresql-16-pgvector
• After installing, set up the extension in the database with create extension vector;
• Create a table for storing documents and embeddings with create table items (id serial primary key, doc text, embedding vector(4096));

Document data

• Uses 4 paragraphs from the Sherlock Holmes story "The Boscombe Valley Mystery" (public domain - Project Gutenberg)

Code

• Demo code released on GitHub under the MIT license is available
• Inserting a document is INSERT INTO items (doc, embedding) VALUES ($1, $2)
• Retrieving the most relevant document is SELECT doc FROM items ORDER BY embedding <-> $1 LIMIT 1 (the <-> operator is provided by pgvector)
• Ollama API calls use the ollama Go package
  • Embedding generation uses api.EmbeddingRequest
  • Chat response generation uses api.ChatRequest (with the retrieved document included in the prompt)

Command-line interface

• Store a document in the database with ragdemo -insert {path-to-doc-file}
• Enter a prompt and generate a response with ragdemo -query {query-text}

Overall process

• When storing a document with the -insert option, it reads the file contents, generates an embedding with Llama3, and stores it in PostgreSQL
• When using the -query option, it generates the prompt embedding and then compares it with the other embeddings in the items table to retrieve the "nearest" document (L2 distance computed with the <-> operator)
• The retrieved document is included in the prompt and passed to Llama3, and the chat response is generated and printed

Additional tips

• Consider using a model specialized for embedding generation (instead of llama3)
• For languages other than English, look for more suitable models
• You can try other distance measures besides L2 distance (pgvector supports other methods)
• Full table scans scale poorly, so use pgvector indexes and the like
• Using more documents in the generation step, or fetching additional documents via keyword matching and so on, also helps
• Output quality can be improved by tuning the generation prompt and trying various LLMs

See also

• LangChain
• Pykoi - A UI library for collecting data & feedback for LLMs
• PEFT
• MUVERA - Makes multi-vector search as fast as single-vector search

Favorite site

• What is retrieval-augmented generation (RAG)? | A comprehensive RAG guide | Elastic
• What is RAG? - Retrieval-augmented generation explained - AWS
• What is retrieval-augmented generation (RAG)? | Databricks

Articles

• Building a RAG system with open-source LLMs - YouTube
• (#langchain meetup talk) R.A.G. Why we can never get results easily - YouTube
• Implementing BGE-M3, the RAG embedding model best at Korean, from scratch | GeekNews
  • A blog post that goes beyond the Python implementation provided on Huggingface and reimplements BGE-M3, the RAG-specific embedding model best at Korean, with TensorFlow/Keras in a form that can be served at scale in enterprise environments.
  • [Original] Implementing the BGE-M3 model
  • sionic-ai/BGE-M3-Model-Converter - A tool for manual conversion of BGE-M3 models with preserved trainable variables and direct control over model outputs.
• (Translation) A guidebook for those swimming in the world of RAG | GeekNews
  • [Original] (Translation) A guidebook for those swimming in the world of RAG. Guidebook to the State-of-the-Art… | by Sigrid Jin | Dec, 2024 | Medium ¹
  • [Original of the original] Guidebook to the State-of-the-Art Embeddings and Information Retrieval
• RAG is not dead | GeekNews
  • [Original] Stop Saying RAG Is Dead – Hamel’s Blog
• No need to parse documents for RAG: just use images | GeekNews
  • [Original] Don't bother parsing: Just use images for RAG | Morphik Blog
  • Traditional OCR and parsing approaches fail to properly preserve meaning when extracting information from complex documents
  • Morphik implemented a way to directly understand even tables, charts, and layout context through a visual document embedding approach based on the ColPali model
  • Compared with existing pipelines, this method is far superior in accuracy and information preservation, reaching up to 95.56% accuracy in benchmark tests
  • Additionally, introducing MUVERA and Turbopuffer brought speed improvements in large-scale document retrieval
  • The goal going forward is practical automation of document work, such as multi-document reasoning, workflow integration, and expert-level interpretation
• Production RAG lessons from processing over 5 million documents | GeekNews
• Want to build a local RAG? | GeekNews
  • Skald was developed with the goal of being a fully self-hostable RAG system that does not send data to third parties
  • RAG components are divided into vector database, embedding model, LLM, reranker, and document parser, and open-source alternatives are suggested for each
  • Skald's default local stack consists of Postgres+pgvector, Sentence Transformers, Docling, and a user-specified LLM
• Ask HN: How are you doing RAG locally? | GeekNews
  • [Original] Ask HN: How are you doing RAG locally? | Hacker News
  • A question on Hacker News asking how users implement Retrieval-Augmented Generation (RAG) in local environments
  • A strong trend emerged that local RAG can run well enough without a vector DB, using text search such as SQLite FTS5, BM25, and grep
  • Many shared experiences that embeddings are slow and noisy for code search, and many argued that keyword-based approaches such as BM25+trigram work better
  • Even when vector search is needed, cases kept coming of solving it with lightweight setups such as Postgres+pgvector, storing vector BLOBs in SQLite, or loading FAISS into memory
  • Search quality can be improved with combinations such as hybrid search merging BM25+vectors, RRF (Reciprocal Rank Fusion), reranking, and multi-query expansion
  • A tendency emerged not to fix on "RAG = vector DB" but to choose simple search → hybrid → agentic depending on document type, scale, and operational burden

References