GNU parallel
lastmod 2025-04-22created 2022-10-268 edits
GNU parallel is a command-line utility for Linux and other Unix-like operating systems that lets users run shell scripts or commands in parallel.
GNU parallel is free software written in Perl by Ole Tange.
It is available under the terms of GPLv3.
Check the number of CPUs
Check the number of cores
Running multi-line commands
What if a file contains several lines of complete commands and you want to run them in parallel? In other words, what if you want to supply the command lines themselves instead of passing arguments to the parallel command? It is very simple.
Logging stdout from multiple parallel processes to separate log files
You could use GNU Parallel instead and its -k option to keep the output in order, in a single log file:
You can add -j 8 after parallel to keep 8 cores busy, but it will keep all cores busy by default anyway.
See also
Favorite site
- GNU parallel - Wikipedia
- GNU Parallel Tutorial — GNU Parallel 20221022 documentation
- [Recommended] gnu_parallel (Genome Informatics Laboratory at KRIBB)
- Why hadn't I known about GNU Parallel all this time? | GeekNews
- GNU Parallel is useful for running E2E tests in parallel
- The author wanted to collect statistics on the pass/fail ratio of a long-running test suite against a live service over HTTP
- At first the author ran the tests with nested loops in bash, but this was slow because each test took a long time to run
- Then ran the tests in parallel with GNU Parallel, greatly improving performance and functionality with less code
- In GNU Parallel, the command is specified to run in parallel, and placeholders ({1}, {2}, etc.) are replaced with input values
- Input to the command is specified after the
:::operator and is supplied to the command in every possible combination - GNU Parallel provides options to control concurrency and avoid resource contention, such as limiting the number of concurrent jobs or delaying the start of each job
- The tool can also output a CSV file with job completion results: including duration, exit code, and captured stdout/stderr
- The author recommends the book by Ole Tange, the author of GNU Parallel: to explore the tool's features more deeply, such as distributing jobs to other hosts over SSH, monitoring job completion times, retrying jobs likely to fail, resuming jobs when a parallel run is interrupted midway, or limiting job concurrency based on CPU usage