Strobelight
lastmod 2025-03-11created 2025-03-111 edits
Profiling service built on open source
Features
- Strobelight is Meta's profiling orchestrator; it combines several open-source technologies to help engineers improve performance and resource utilization
- Since adoption, it has achieved annual capacity savings equivalent to about 15,000 servers
- Strobelight is not a single profiler but an orchestrator that coordinates multiple profilers
- Runs on all of Meta's production hosts and collects performance metrics such as
- CPU usage
- Memory allocation
- Other performance metrics
- With this, engineers can identify performance bottlenecks and resource waste and optimize their code
- A profiler is a sampling-based statistical analysis tool
- Example: on CPU cycle events, it can analyze function call stacks and how long functions run on the CPU
- Helps improve performance by giving a detailed picture of code execution
- Strobelight includes 42 profilers in total
- jemalloc-based memory profiler
- Function call count profiler
- Language-specific event-based profilers for Python, Java, Erlang, and others
- AI/GPU profilers
- Off-CPU time tracking profiler
- Service request latency tracking profiler
- Profiling tools can be run from a command-line tool or the web UI
- Continuous profiling and trigger-based profiling on specific conditions can be configured
- For engineers, adding a new profiler requires multiple code changes and deployments
- Profilers can be added quickly by writing a bpftrace script
- Engineers can quickly trace and analyze performance problems in specific functions
- Strobelight has a built-in system that prevents resource conflicts between profilers
- Applies rules such as forbidding the use of other PMU counters while tracking CPU cycles
- Minimizes resource conflicts through a concurrency and queue management system
- Strobelight automatically runs profiling on all Meta services
- Automatically adjusts sampling frequency and duration based on each service's workload
- Automatically corrects sampling probability and collection frequency to provide consistent data
- LBR profiler
- Supports sampling with Last Branch Record (LBR), an Intel hardware feature
- Used in FDO (Feedback Directed Optimization) to improve binary performance
- Up to 20% reduction in CPU cycle usage across Meta's top 200 services
- Event profiler
- Works similarly to the Linux perf tool
- Samples and visualizes performance events (CPU cycles, L3 cache misses, etc.)
- Can detect and fix problems in code paths in advance
- Stack Schemas
- Improves visibility by adding tags to function call stacks
- Can remove unneeded functions in filtering and visualization
- Strobemeta
- Adds dynamic metadata to call stacks at runtime
- Provides details related to service endpoints, latency, and so on
- Converts virtual addresses in binaries into function names and source code information
- Based on open-source technologies such as DWARF, ELF, gsym, and blazesym
- Symbolization runs after profiling to avoid performance degradation
Strobelight data visualization tools
- Scuba
- SQL-based query and visualization tool
- Provides time series, distributions, flame graphs, and more for profiling data
- Time-based visualization tool for combined profiling data
- Can visualize service request spans, CPU cycles, and off-CPU data together
- An engineer found a std::vector copy problem through Strobelight data
- Added & after the auto keyword → prevented an unnecessary copy
- As a result, savings of about 15,000 servers per year