Parallel programming model
lastmod 2025-08-09created 2025-08-092 edits
In computing, a parallel programming model is an abstraction (computer science) of parallel computer architecture, with which it is convenient to express algorithms and their composition in programs. The value of a programming model can be judged on its generality (how well a range of different problems can be expressed for a variety of different architectures) and its performance (how efficiently the compiled programs can execute). The implementation of a parallel programming model can take the form of a library (computing) invoked from a programming language, as an extension to an existing language.
Consensus around a particular programming model is important because it leads to different parallel computers being built with support for the model, thereby facilitating software portability. In this sense, programming models are referred to as bridging between hardware and software.
Categories
Approaches for modern deep learning and hardware
- Parallel programming models: approaches for modern deep learning and hardware | GeekNews
- As Moore's law hits its limits, hardware has evolved from faster single cores toward multiple cores and parallel processing
- Parallel processing is divided into several forms such as data parallelism, model parallelism, and pipeline parallelism, which are used in combination in modern deep learning systems
- Parallelization happens at various levels, such as SIMD (instruction-level data parallelism), thread/core parallelism, and massive GPU parallelism
- Most languages, libraries, and tools for parallel processing are extensions "bolted onto" existing sequential languages, so the trend of integrating parallelism natively into the language (Mojo, etc.) is drawing attention
- Practical performance optimizations such as cache line sharing (unnecessary interaction) optimization, efficient memory partitioning, and auto-vectorization are key challenges