Skip to content
Search

Prometheus

lastmod 2026-03-27created 2020-06-0937 edits

The Prometheus monitoring system and time series database.

Categories

Features

Prometheus's main features are:

  • a multi-dimensional data model with time series data identified by metric name and key/value pairs
  • PromQL, a flexible query language to leverage this dimensionality
  • no reliance on distributed storage; single server nodes are autonomous
  • time series collection happens via a pull model over HTTP
  • pushing time series is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support

Components

The Prometheus ecosystem consists of multiple components, many of which are optional:

  • the main Prometheus server which scrapes and stores time series data
  • client libraries for instrumenting application code
  • a push gateway for supporting short-lived jobs
  • special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
  • an alertmanager to handle alerts
  • various support tools

Most Prometheus components are written in Go, making them easy to build and deploy as static binaries.

Architecture

This diagram illustrates the architecture of Prometheus and some of its ecosystem components:

Prometheus-architecture.svg.png

Prometheus vs InfluxDB

InfluxDB's data model

InfluxDB uses the concepts of series (Series) and measurement (Measurement). What does that mean? Let me explain it simply!

  • Measurement (Measurement): This is what we want to measure. For example, things like "CPU usage" or "memory usage".
  • Tag (Tag): Metadata that distinguishes data. Information like "server name" or "region" goes here.
  • Field (Field): The actual measured value. Concrete numbers like "CPU usage 80%" or "memory usage 4GB" go here.
  • Timestamp (Timestamp): Indicates when the data was recorded.

With this structure, you can store and search data really efficiently. For example, you can easily query "CPU usage over the last hour for all servers in Seoul".

Prometheus's data model

Prometheus takes a somewhat different approach. Here it uses the concepts of metric (Metric) and label (Label).

  • Metric (Metric): What you want to measure. A concept similar to InfluxDB's measurement (Measurement).
  • Label (Label): Metadata made of key-value pairs. Similar to InfluxDB's tags, but can be used more flexibly.
  • Sample (Sample): The combination of an actual measured value and a timestamp.

This structure of Prometheus is especially strong at handling multi-dimensional data. For example, you can attach labels like "method", "status_code", and "endpoint" to a metric called "HTTP request count" and analyze the data in great detail.

The conclusion?

The difference in data models has a big impact on each database's performance and how it is used.

  • InfluxDB is optimized for storing and searching time series data really fast.
  • Prometheus, on the other hand, is more flexible at handling multi-dimensional data.

Because of these characteristics:

  • InfluxDB is often used for IoT and sensor data processing,
  • and Prometheus is widely used for managing complex systems, such as microservice monitoring.

Prometheus vs Grafana

Prometheus and Grafana are often used together as tools for monitoring and visualization, but they are designed for different purposes. The main differences are summarized below.

Function/Feature

Prometheus

Grafana

Main role

Time series database and monitoring solution

Data visualization and dashboard management

Functions

  • Metric collection and storage
  • Alert generation
  • Data analysis through a query language (PromQL)
  • Visualization of various data sources
  • Dashboard building
  • Alert trigger management

Data storage

Stores data in its own time series database (TSDB)

Does not store data itself. Instead connects to various sources such as Prometheus, Elasticsearch, and MySQL

Data sources

Mainly collects metrics from applications using a pull model

Supports many sources such as Prometheus, InfluxDB, Elasticsearch, and MySQL

Query language

PromQL (Prometheus Query Language)

Supports query languages suited to each data source

Alerting system

Manages alerts through the built-in Alertmanager

Has its own alerting system, but mainly uses alerts provided by tools such as Prometheus

Scalability

Can run as a single node; scalability is supplemented through Pushgateway and similar tools

Various plugins and data sources can be added

Installation and configuration

Relatively simple, but needs Exporters for metric collection

Easy to use with an intuitive interface

Use cases

  • System and application performance monitoring
  • Metric-based analysis
  • Data visualization and dashboard management
  • Well suited as a team collaboration tool

Integrated use

Mainly used as a data source for Grafana

Often used to visualize data collected by Prometheus in dashboards

Brief summary
Prometheus collects and stores data, and generates alerts or performs analysis based on it.
Grafana pulls data from multiple data sources such as Prometheus, visualizes it, and builds dashboards.
Why use them together
Prometheus excels at collecting and storing data, but is not well suited as a visualization tool. Grafana makes up for this by presenting the data intuitively, so the two tools complement each other.
Real-world example
Prometheus collects metrics such as CPU usage and memory usage.
Grafana visualizes the data collected by Prometheus and displays it on a dashboard.
When a system anomaly occurs, Prometheus Alertmanager triggers an alert.

How to run a demo

Run pushgateway: 

docker run --rm -it -p 9091:9091 prom/pushgateway

First, write a prometheus.yml file: 

global:
  scrape_interval: 1s

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['192.168.100.153:9091']

  - job_name: 'node'
    static_configs:
      - targets: ['192.168.100.153:9100']

  - job_name: 'gpu'
    static_configs:
      - targets: ['192.168.100.161:9454']

Run node-exporter: 

docker run --rm -it --net="host" --pid="host" -v "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host

If you need to monitor GPUs, run something like nvidia-smi-exporter.

Run Prometheus: 

docker run -it --rm -v ./prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 prom/prometheus

Open http://localhost:9090/ in a web browser.

To push sample data to pushgateway: 

echo "new_metric 100" | curl --data-binary @- http://localhost:9091/metrics/job/pushgateway

Troubleshooting

context deadline exceeded 

Error scraping target: Get "http://192.168.1.88:8000/metrics": context deadline exceeded`

This means Prometheus could not finish scraping metrics from the target within the set time (scrape_timeout). It can happen when the target takes a long time to respond, because of network problems or a firewall, or because the scrape timeout (scrape_timeout) in the Prometheus configuration is too short 

scrape_configs:
  - job_name: 'prometheus'

    scrape_interval: 5m
    scrape_timeout: 1m

Note that scrape_interval must not be smaller than scrape_timeout.

See also

Favorite site

with Grafana

author Zer0 sitemap