RL Can Only Reinforce What It Sees Succeed
RL Can Only Reinforce What It Sees Succeed
A paper investigates how outcome reward reinforcement learning shifts language models' balance between prior knowledge and observations. In puzzle worlds with known optimal policies, the authors find that when correct actions are rare, RL learning success is determined by a predictable count: the expected number of trial groups in which the action succeeds, with approximately 35 groups yielding equal probability. This pattern holds across different problems, trial group sizes (4 to 256), learning rates, optimizers, GRPO variants, and model scales (1.3M to 85M parameters).
The paper quantifies the sample efficiency bottleneck of RL in learning rare actions and provides a universal counting law across model scales and hyperparameters.
Source: arXiv Game AI · arxiv.org