Skip to content
Search

VectorVFS

lastmod 2025-05-07created 2025-05-071 edits

Using the file system as a vector database

About

  • VectorVFS is a Python package that stores a vector embedding for each file as metadata, letting the Linux file system itself serve as a vector database
  • Achieves zero-overhead indexing through the file system's xattrs (extended attributes), without an external index or DB
  • Enables finding similar files via embedding search; not tied to a specific model, and various embedding models can be plugged in
  • Uses Meta's Perception Encoders (PE) to generate image/video-based vector embeddings, which show higher zero-shot performance than other models
  • Lightweight, portable design; usable right away without a separate daemon or service

Introduction

  • VectorVFS is a lightweight Python library that enables storing and searching embeddings for files using only the built-in features of the Linux file system
  • Stores embedding values in each file's extended attributes (xattrs), without an external database
  • Can extend an existing directory structure, kept as-is, into a system capable of semantic search

Key features

  • Zero-overhead indexing
    • Stores vector embeddings directly in the file's xattr
    • No external indexing service or extra storage needed; exists only as metadata alongside the file
  • Seamless retrieval
    • Can perform vector-based similarity search across the entire file system
    • e.g., search for similar image files with something like find_similar_images('example.jpg')
  • Flexible embedding support
    • Uses Meta's Perception Encoders (PE) model by default
    • Support for various embedding models (e.g., text, audio, multimodal) planned
    • Custom embedding models can also be integrated as plugins
  • Lightweight and portable
    • Built on the Linux VFS (xattr) feature, so no separate daemon or server setup needed
    • Portable; usable in local directories or on external storage
  • Embedding model used: Meta Perception Encoders
    • PE is an image/video-based vision-language model released by Meta
    • Better zero-shot performance than competing models InternVL3, Qwen2.5VL, and SigLIP2
    • More backend embedding models to be added in the future

Summary

  • Can build a vector system capable of semantic search while keeping the existing file structure as-is
  • Almost no embedding storage cost; works without separate infrastructure
  • Well suited to privacy-preserving search on offline/edge devices

See also

Favorite site

author Zer0 sitemap