Abstract
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.
Experiments
Action chunking boosts goal-reaching performance. We compare Action-Chunked CRL (CRL + AC) with a fixed action chunk length (H=3) against standard CRL across 3 settings, reporting 95% bootstrapped confidence intervals. CRL + AC consistently outperforms CRL, with average improvements of +31.7% on the OGBench manipulation suite, +69.4% on suboptimal OGBench datasets (noisy and explore), and +93.1% on online JaxGCRL locomotion and navigation tasks.
Online locomotion and navigation results. We report time-at-goal learning curves of CRL and CRL + AC with action chunk lengths H ∈ {3, 5, 10} on 10 JaxGCRL locomotion and navigation tasks, with 95% bootstrapped confidence intervals. Time-at-goal captures both task success and behavioral quality, rewarding policies that reach and stably maintain the goal. Overall, CRL + AC with H ∈ {3, 5} consistently and substantially outperforms CRL, with H=3 being the best or near-best chunk length across the majority of tasks, improving performance by +111.2% on average when all 11 JaxGCRL environments are considered.
Action chunking adds information about the goal; performance peaks at moderate chunk length. Results on cube-single-noisy-v0 (σ = 1). (Left) Validation categorical accuracy (dark bars, left axis) increases with action chunk length H as the critic recovers more information about the goal. The thick line on the x-axis marks the state-only critic's accuracy (H=0, no action conditioning), so each bar's height shows the accuracy gained over the state-only baseline. Accuracy peaks at H=30; at H=50 it drops even as training accuracy (light bars, left axis) continues to rise, indicating that the long-chunk critic overfits. (Right) Aggregated success rate peaks at a short chunk length (H=5) and collapses to near zero by H=30 and H=50. Distilling each long-chunk critic into a short-chunk (Hπ=5) critic, and extracting policy from it (light bars, w/ DQC) recovers success to ~50%. All methods execute a single action and replan at each step (Hexec=1), so the collapse does not stem from open-loop execution, but from the difficulty of extracting an action from a long-chunk critic.
Ablations on Action-Chunked Contrastive RL. (Left) Performance across network depths. We evaluate the effect of network depth scaling on a challenging Humanoid task, for CRL and CRL + AC with action chunk length H=3. CRL + AC matches or exceeds CRL at every depth. (Middle) Effect of replanning on CRL + AC. We compare CRL + AC (H=3) executing the full chunk open-loop (Hexec=H) against replanning every step (Hexec=1), alongside standard CRL. Replanning every step achieves comparable average performance to open-loop execution. (Right) Action-Chunked CRL and noise. We report the success rate gain of CRL + AC over CRL for action chunk lengths H ∈ {3, 5} across cube-single-noisy-v0. We vary the noise level σ injected into expert actions during offline dataset collection, and train both CRL and CRL + AC on the resulting datasets. Gains become consistent for σ ≥ 0.3, reaching up to 100% at the highest noise level.
Conclusions
While we were able to observe consistent trends and significant gains, these remain conditional on the selection of the right chunk length. In practice, we have found a mild horizon of three steps (H=3) to be generally the most beneficial; nevertheless, designing an offline procedure to automatically choose chunk lengths, possibly per state, is an important direction for future work. A natural next direction is whether the benefit extends to other temporal enrichments of the critic's input: action chunking, motivated by its established use in RL, is only one such method, and whether alternatives like conditioning on future states or past history yield similar gains remains an open question. More generally, our work suggests that action chunking synergizes particularly well with CRL, and improves performance through mechanisms beyond those previously studied. Similar phenomena may arise beyond CRL: preliminary results show that action chunking also improves other self-supervised RL methods, motivating its broader study in self-supervised RL.