Skip to content
Search

PgQueuer

lastmod 2024-08-21created 2024-08-211 edits

Python library that uses PostgreSQL as a job queue

  • PgQueuer is a minimal, high-performance job queue library for Python that leverages the robustness of PostgreSQL
  • Designed for simplicity and efficiency, it uses PostgreSQL's LISTEN/NOTIFY to manage job queues with ease

Features

  • Simple integration: integrates easily with existing Python applications that use PostgreSQL
  • Efficient concurrency handling: uses PostgreSQL's FOR UPDATE SKIP LOCKED to enable reliable, concurrent job processing
  • Real-time notifications: leverages LISTEN and NOTIFY to provide real-time updates on job status changes

GN⁺'s summary

  • PgQueuer leverages PostgreSQL's LISTEN/NOTIFY to let Python applications manage job queues efficiently.
  • It offers simple integration and efficient concurrency handling, so it can be easily applied to existing systems.
  • Real-time notifications let you check job status immediately, making job management easy.
  • Other projects offering similar features include RQ and Celery.

Hacker News comments

  • Show HN: PgQueuer – Transform PostgreSQL into a Job Queue | Hacker News
  • Question about how it compares with the Graphile Worker library
  • Thoughts on whether PostgreSQL-based job queue libraries could share a common schema
    • A fan of Elixir's Oban
    • Just as there are Sidekiq-compatible libraries in many languages, a similar approach would benefit PostgreSQL-based job queues
    • A core job queue library could be developed in Rust, with language-specific bindings considered
  • Supports using Postgres for everything and is maximizing that in their project
    • Wants a Postgres job queue that does multi-stage pipelines, fan-out, and accumulation
    • A structured relational database is especially well suited to modeling this
    • The 8k maximum payload size of listen/notify is limiting
    • Wants streaming tables and rich type support
  • Recommends a good PostgresFM episode on implementing queues in Postgres
  • PgQueuer integrates well with Postgres RPC calls, triggers, and cronjobs (pg_cron)
  • The River library is also worth a look
    • Supports scheduled jobs and more
    • Also provides a UI for debugging
  • Using Broadcaster for PUB/SUB jobs
  • The simplest job queue example in MySQL
    • Uses an update job_table set key=value where ... limit 1 statement
    • Simple and atomic
    • PostgreSQL does not allow the update ... limit syntax
  • A MySQL user, but exploring PostgreSQL
    • Provides many features out of the box
    • A very interesting tool
  • There is also the Procrastinate library
    • Uses PostgreSQL's LISTEN/NOTIFY (can be turned off as an option to use polling)
    • Supports sync and async jobs, periodic jobs, retries, job locks, priorities, job cancellation/abort, and Django integration (optional)
    • Is a co-maintainer of Procrastinate

See also

Favorite site

author Zer0 sitemap