RAG:MLOps

lastmod 2026-09-04created 2026-09-042 edits

RAG (Retrieval-Augmented Generation) guide for an MLOps platform

RAG use scenarios

Use cases where RAG fits on an MLOps platform:

Platform documentation/ADR search (most realistic)

• 40+ plans under docs/plans/, plus docs/reviews/, READMEs, contracts and more
• Example: "How do I change the YOLO training runtime?", "How does tenant isolation work?"
• Returns document paragraphs together with their sources
• Tool calling is strong for structured data (APIs), but RAG wins for partial matching in unstructured documents

Training Log/Error knowledge base

• Vector indexing of past failed training logs, error messages and fixes
• Example: "I got CUDA out of memory; how did we fix it before?"
• Returns similar log context + fixes
• MLflow metrics via tool calling, log text via RAG

OpenAPI/Swagger spec search

• Questions like "Which endpoints deal with dataset deletion?" when there are many API endpoints
• Retrieve the relevant sections from the spec document

Recommended architecture

As a complement to Tool Calling:

사용자 질문
    │
    ▼
분류기 (의도 판단)
    ├── 구조적 데이터 조회 → Tool Calling (Plan 40 그대로)
    ├── 문서/지식 검색    → RAG (docs, logs, plans)
    └── 둘 다 필요       → Hybrid (RAG context + tool results 모두 주입)

FAQ: RAG architecture - Do I need to rebuild RAG when changing the LLM model?

Need to rebuild RAG when changing the LLM model: No.

RAG consists of two independent models:

┌─────────────────────────────────┐
│  Embedding Model (인덱싱)        │  ← 벡터 생성
│  (예: text-embedding-3-small)    │
│  문서 → 벡터 → 벡터 DB           │
└─────────────────────────────────┘
         │
         ▼  (검색된 컨텍스트)
┌─────────────────────────────────┐
│  LLM (생성)                      │  ← 답변 생성
│  (Qwen, DeepSeek, GPT, 등)      │
│  프롬프트 + 검색 결과 → 답변      │
└─────────────────────────────────┘

• When changing only the LLM: indexing, vector DB and search logic all stay as they are
• When you need to rebuild: only when you change the embedding model
  • A different embedding model maps the same text into a different vector space

Implementation

Recommended stack: PostgreSQL + pgvector

Adds an extension to the existing PostgreSQL:

Comparison of alternatives

Implementation steps

1. Add dependencies: pyproject.toml → "pgvector"',"pgvector-python"''
2. 'Migration:
   • alembic revision -m "add rag_vector_store"
   • CREATE EXTENSION vector
   • CREATE TABLE rag_documents (id, tenant_id, chunk_text, embedding(vector), source_file, metadata(jsonb), created_at)
3. Embedding service:
   • src/geo_mlops/services/rag_embedding.py
   • Calls an OpenAI-compatible /v1/embeddings endpoint
4. 'Indexing Job:
   • docs/, README.md, plan documents → chunking → embedding → DB insert
5. 'Search service:
   • src/geo_mlops/services/rag_service.py
   • query embedding → vector nearest-neighbor search → return top-K chunks
6. 'RAG router:
   • src/geo_mlops/routers/rag.py
   • POST /api/v1/rag/query

Full pipeline

OpenAI API + pgvector handle only 2 stages of the full pipeline.

┌──────────────┐     ┌──────────┐     ┌─────────┐     ┌──────────┐     ┌──────┐
│  원본 문서     │ ──→ │  Chunking │ ──→ │ Embedding│ ──→ │ pgvector │
│  docs/*      │     │  분할      │     │ OpenAI API│     │  저장    │
└──────────────┘     └──────────┘     └─────────┘     └──────────┘
                                                         ↑
┌──────┐     ┌───────────┐     ┌──────────┐     ┌──────────┐     │
│ LLM  │ ←── │ Prompt 조립 │ ←── │  Top-K   │ ←── │ Query   │ ────┘
│ 답변  │     │ context   │     │ 검색      │     │ Embedding│
└──────┘     └───────────┘     └──────────┘     └──────────┘
                                            (pgvector)

OpenAI API + pgvector cover only stage 3 (Embedding) and stage 4 (Storage). The other 6 must be implemented yourself.

Ways to simplify:

• langchain covers most of 1–7
• However, since the project is written consistently in the FastAPI + SQLAlchemy pattern, a custom implementation fits the existing code style better

FAQ: Automation feasibility - Is there anything that requires human intervention and blocks full automation?

Document-based RAG can be fully automated. Fundamentally different from image labeling.

Automation pipeline

docs/*.md, README.md, 코드
         │
         ▼  (자동)
   Chunking — 토큰 기준 분할
         │
         ▼  (자동, API 호출)
   Embedding — 텍스트 → 벡터
         │
         ▼  (자동)
   pgvector 저장
         │
         ▼  (자동)
   HNSW index 생성
         │
         ▼  (자동)
   Query → 검색 → LLM → 답변

Optional human intervention

The following are optional, and everything is fully automatic after setup:

1. Answer quality evaluation: judging "Is this answer right?" (optional)
2. Chunk parameter tuning: adjusting chunk size, overlap (defaults also work)
3. Deciding indexing scope: "Which documents should be included?" (one-time decision)

There is no work requiring ongoing human intervention like image labeling.

See also

• RAG
• MLOps