Log Aggregator

lastmod 2026-03-25created 2020-07-0629 edits

A system that collects logs accumulated on multiple machines so they can be analyzed all at once.

Solutions

• Graylog
  • ElasticSearch - Log storage
  • MongoDB - Metadata storage
  • Uses the GELF library for Log4j log streaming.
• Splunk - Log management and analysis
• Flume
• Fluentd (CNCF; EFK)
• Logstash (ELK)
• Rsyslog
• Syslog-ng
• Papertrail
• CloudWatch
• Sentry
• GlitchTip - Open-source error tracking platform compatible with the Sentry API
• Grafana Loki - Like Prometheus, but for logs.

Log Management

• 5 Data Storage Solutions for Better Log Management

Log management using Data Storage Solutions

• ClickHouse (OLAP)
• PrestoDB
• InfluxDB
• Apache HBase
• Tarantool

Tools

• Best Log Aggregator Tools in 2020
• SolarWinds Papertrail
• Logstash
• Fluentd
• Graylog
• ManageEngine EventLog Analyzer
• Sumo Logic
• LogDNA
• log metric - Real-time log monitoring in your browser
• angle-grinder - A viewer that can parse, aggregate, sum, average, min/max, and sort logs in real time - lets you do sophisticated analysis simply when you don't have graphite/kibana/honeycomb/splunk, etc.
• Logchef - Ultra-lightweight, high-performance log analysis platform based on Clickhouse

Log shippers

There are log shippers that can be combined with other tools to build a log management solution covering the entire service, and each log shipping tool has different features.

• Logstash - A powerful data pipeline tool that can be used to collect and ship log files.
• Fluentd
• Logspout - Logspout is a log router for Docker containers that runs inside Docker.

Log stream processors

Stream processing technology is used to process log streams on the fly as needed.

• Flume
• Use Kafka together with Storm or Spark Streaming.
• Log4j has Flume appenders that are useful for collecting log messages.
• Spring Cloud Stream
• Spring Cloud Stream Modules
• Spring Cloud Data Flow

Storage

• ElasticSearch
• HDFS
• MongoDB
• Cassandra

Dashboard

• Kibana (ElasticSearch)
• Graphite (Grafana)

Projcts

• Scribe
• Apache Flume
• Fluentd
• Logstash

See also

• syslog
• Logging (Log file; Log level, etc ...)
• Metrics Aggregations (Stats Aggregation)
• Log Aggregator
• Log management
• LogFX - Multi-platform open-source log viewer
• tailspin - Log file highlighter
• Dozzle - Real-time container log viewer

Favorite site

• Log Aggregator Comparison - Scribe, Flume, Fluentd, logstash
• A High-Volume Log Collection and Analysis System Built in an Hour
• Data Expert Knowledge Portal DBGuide.net - DB Build Guide - Log Data Collection
• Developing a Log Analysis Platform Using Multi-Cloud
• [Recommended] (MSA) Microservice Logging and Monitoring ¹
• Log Aggregation – Best Tools and Practices in 2020
• Event Log Processing at Airbnb

References