Action Gap
lastmod 2026-06-10created 2026-06-101 edits
Action gap is a concept from Reinforcement Learning that refers to the difference between the value of the optimal action and the value of the second-best action in a given state.
Definition
When the value of each action a in a state s is Q(s, a), the action gap is defined as follows.
-
a*= optimal action (the action with the highest Q-value) - In other words, the Q-value difference between the best and second-best actions
Why it matters
In reinforcement learning, the value function is usually estimated with function approximation (neural network, etc.) and always contains estimation error.
- When the action gap is large: even with some estimation error, the optimal action is unlikely to be chosen wrongly. → The policy becomes robust.
- When the action gap is small: even a small estimation error can easily cause a second-best action to be mistaken for the optimal one. → The policy becomes unstable.
Notable research
The best-known paper is "Increasing the Action Gap: New Operators for Reinforcement Learning" (2016) by Bellemare et al. It proposes the consistent Bellman operator and others, which modify the standard Bellman operator, artificially widening the action gap without changing the optimal policy to improve training stability and performance.
Key idea:
- Among the operators that satisfy the Bellman equation, using an operator that widens the action gap improves tolerance to function approximation error.
- As a result, DQN-family methods performed better on benchmarks such as Atari games.
References
- Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., & Munos, R. (2016). Increasing the Action Gap: New Operators for Reinforcement Learning. AAAI.