Give a robot a task-specific reward, and it can learn a policy through reinforcement learning. Give it expert demonstrations, and it can learn by imitation. But what if we give it neither?
Ideally, a robot should learn much like a human: by interacting with the world — reaching for objects, making contact, pushing and picking them up, and discovering the outcomes its actions can produce. Over time, these experiences could give rise to a repertoire of reusable manipulation skills, acquired without expert demonstrations or task-specific rewards. This is the promise of unsupervised skill learning for robotic manipulation: learning diverse behaviors through interaction before any particular downstream task is specified.
Goal-conditioned reinforcement learning offers a natural framework for this vision. By treating different desired outcomes as goals, a single policy can acquire many behaviors from the same stream of experience. In particular, contrastive reinforcement learning (CRL)2,3 has recently shown promising success in learning manipulation skills from scratch, without demonstrations or task-specific rewards. Its performance can also improve substantially with larger model scales, suggesting a promising path toward scaling unsupervised RL6,7. Yet, as manipulation tasks involve increasingly sparse and complex object interactions, this scaling alone may not be enough.
In this post, we first explain why standard CRL struggles with interaction-rich manipulation from a theoretical perspective, and then use this analysis to derive a practical solution. Specifically, we model manipulation as a piecewise-smooth dynamical system: before contact, the object follows its passive dynamics; during interaction, the robot's action activates a different dynamical mode; and afterward, the effect of that interaction propagates into future states. We show that these mode switches induce piecewise nonlinear reachability structures that are difficult for CRL's smooth representation geometry to capture, while errors around the interaction propagate throughout the subsequent trajectory. This analysis motivates Interaction-Weighted Resampling (IWR), a simple method that emphasizes future states near interaction events while preserving the original CRL objective.
With this small change, CRL learns more effective manipulation policies across control and manipulation tasks. Notably, we evaluate IWR on a real-world robot air hockey task, a particularly demanding test of fast inference and reactive control, where the policy must rapidly anticipate the puck's trajectory and continuously close the control loop from recent observations. IWR improves the real-robot success rate from 25% with standard CRL to 60%.
01Basics of Contrastive Reinforcement Learning
In goal-conditioned RL, the quantity that matters is not a one-step reward but a reachability measure: starting from state \(s\) and taking action \(a\), how likely is the agent to visit goal \(g\) somewhere in its discounted future? This is the future-occupancy density,
Contrastive RL doesn't estimate this density directly. Instead it learns two encoders — a state-action encoder \(\phi(s,a)\) and a goal encoder \(\psi(g)\) — whose inner product recovers the log-reachability up to a goal-only baseline:
Given an anchor \((s,a)\), the state that actually occurred later in the same trajectory is a positive goal \(g^+\); states drawn from elsewhere are negatives. An InfoNCE objective pulls each anchor toward its true future and pushes it away from the rest:
Hence, the consequence is geometric. The encoder gives an implicit map on which the energy \(\phi(s,a)^\top\psi(g)\) is high when \(g\) is easy to reach and low when it is far. A policy that climbs this energy toward the goal is, in effect, following the terrain of reachability. Everything downstream — how well the policy generalizes, how efficiently it explores — depends on that terrain being faithful to the true dynamics.
02Smooth for Locomotion, Piecewise for Contact
The terrain is only as well-behaved as the dynamics it has to encode. For a walking or reaching agent, movement is continuous: a small change in action produces a small, predictable change in where you end up. Neighboring states stay neighbors after a step, so the learned map is smooth. Successive states trace an orderly curve through representation space, and interpolating between them is safe.
However, this does not hold for manipulation. Actions do essentially nothing until the gripper contacts the object; then, in a single step, a whole new set of variables like the object's pose and velocity comes under the agent's control. Reachability changes discontinuously at that boundary. The map that was smooth over free-space motion now has a crease exactly at contact, and the two sides of the crease obey different local rules.
To illustrate this, below we roll out trained policies and, alongside each rollout, plot a t-SNE embedding of the learned \(\phi(s,a)\) as it evolves. In a smooth reaching task the embedding drifts gently and continuously. In contact-rich air hockey the same embedding lurches: long stretches of near-passive drift punctuated by sudden jumps whenever the paddle strikes the puck.
Left panel of each clip: the rollout. Right panel: t-SNE of \(\phi(s,a)\) over the same episode. Smooth motion yields a smooth trajectory in representation space; contact yields discontinuities.
03Why Contact Breaks the Map
Hence, we have to model manipulation as a factorized, piecewise-smooth Markov process. The state splits into the agent's own configuration and the object's; between contacts, the object is passive and its factor evolves on its own, decoupled from the action. The action only reaches the object during interaction.
Under a local (Gaussian-interpolation) analysis of the energy landscape, this factorization has a sharp consequence for how the goal representation \(\psi\) advances from one step to the next.
In locomotion, the step map is linear. Because motion is smooth everywhere, the representation of the next state is well approximated by a single linear operator applied to the current one, \(\psi_{t+1}\approx A_0\,\psi_t\). Iterating a rollout just applies \(A_0\) again and again, \(A_0^{\,n-1}\psi_t\), and the trajectory stays on a smooth, predictable arc toward \(\psi(g)\).
At contact, the step map becomes affine. When interaction happens, the next representation can only be locally approximated by adding an offset, \(\psi_{t+1}\approx A_1\psi_t + b_t\) — the bias term \(b_t\) is the signature of the mode switch. A single affine kink is exactly the piecewise nonlinearity that the smooth contrastive geometry cannot represent with one consistent map.
Because the object is passive between contacts, whatever error the encoder makes at the interaction is then transported by the free-space dynamics for the rest of the episode. Formally, the worst-case error in the estimated energy after \(k\) further steps grows with the propagated operator:
Here \(e\) is the local approximation error committed at the contact, and \(A_0^{\,k}\) is the passive dynamics compounding it over the following \(k\) steps. The single most important state in a manipulation episode — the instant of contact — is also the one the smooth model represents worst, and its error is precisely the one that gets amplified. That is the mechanism behind the collapse.
The one moment that decides a manipulation episode is the moment the smooth model fits worst — and its error is the one that propagates.
04Interaction-Weighted Resampling
Once the problem is framed this way, the fix follows from it. The error bound is driven by two things we can influence: how large the contact-time error \(\|e\|\) is, and how many steps \(k\) it is allowed to propagate. Both are controlled by which future states the algorithm sees as goals during training. Standard CRL samples the positive future uniformly along a trajectory. In a manipulation episode dominated by passive, uneventful frames, that means the rare contact transitions — the ones that carry all the action-relevant structure — are almost never sampled. The model is trained hardest on exactly the frames that matter least.
Interaction-Weighted Resampling changes the sampling distribution over futures, and nothing else. Instead of drawing the positive goal uniformly, it up-weights candidate futures that lie near an interaction, using a soft weight that peaks when a proximity signal \(d_{t+k}\) (for example, the gripper–object distance) is close to a contact threshold \(c\):
The floor \(\epsilon\) keeps ordinary transitions in the mix so the smooth free-space geometry is still learned; the Gaussian bump concentrates extra samples in the window just before, during, and after contact. Two knobs do the work. The bandwidth \(\sigma\) sets how tightly coverage clusters around the interaction, and \(\epsilon\) trades off coverage against the propagation horizon \(k\) — together they drive the contact-time error down and keep it from being carried far. Crucially, the weight is soft: it needs only a rough proximity signal, not a hand-labeled contact indicator or a privileged interaction detector.
IWR is a light-weight modification to the contrastive critic's target distribution. Anchors are drawn from the replay buffer, candidate futures are re-weighted by interaction proximity, positives and negatives are resampled accordingly, and the InfoNCE critic and its actor are updated exactly as before. No new networks, no reward shaping, no auxiliary losses.
05Experimental Results
We evaluate on three families of goal-conditioned tasks spanning the smooth-to-contact spectrum: Box2D reaching and pushing, Meta-World12 manipulation, and Air Hockey13 in simulation, in sim-to-real transfer, and on a real robot. The baselines run from model-free standards (PPO11, SAC10, SAC+HER8, SAC+HINT9) to the strongest contrastive methods (CRL4 and CRTR5) that IWR modifies.
IWR helps most exactly where the analysis says it should — the contact-heavy tasks — while leaving the near-smooth tasks essentially untouched. Averaged across the manipulation suite it improves success by +19.8% over the strongest contrastive baseline, with the largest gains on Box2D (hard), Meta-World (pick-place), and the real-robot air-hockey transfer.
| Task | PPO | SAC | SAC +HER | SAC +HINT | CRL | CRTR | IWR (Ours) |
|---|---|---|---|---|---|---|---|
| Air Hockey (Simulation) | 0.617 | 0.145 | 0.398 | 0.422 | 0.695 | 0.727 | 0.742 +2.1% |
| Air Hockey (real-transfer) | 0.160 | 0.215 | 0.129 | 0.125 | 0.477 | 0.465 | 0.500 +4.8% |
| Air Hockey (Real Robot) | 0/20 | 0/20 | 0/20 | 0/20 | 5/20 | 2/20 | 12/20 +140% |
| Box2D (center) | 0.086 | 0.058 | 0.088 | 0.088 | 0.278 | 0.274 | 0.288 +3.6% |
| Box2D (goal) | 0.089 | 0.046 | 0.086 | 0.064 | 0.450 | 0.558 | 0.709 +27.1% |
| Box2D (hard) | 0.060 | 0.042 | 0.064 | 0.076 | 0.317 | 0.365 | 0.565 +54.8% |
| Box2D (hard velocity) | 0.148 | 0.149 | 0.152 | 0.139 | 0.387 | 0.377 | 0.436 +12.7% |
| Box2D (maze) | 0.033 | 0.012 | 0.031 | 0.035 | 0.217 | 0.206 | 0.223 +2.8% |
| Meta-World (peg insert) | 0.000 | 0.000 | 0.000 | 0.000 | 0.430 | 0.367 | 0.438 +1.9% |
| Meta-World (pick place) | 0.000 | 0.000 | 0.004 | 0.000 | 0.266 | 0.305 | 0.570 +86.9% |
| Meta-World (push) | 0.000 | 0.000 | 0.004 | 0.000 | 0.699 | 0.750 | 0.730 |
| Meta-World (sweep into) | 0.000 | 0.004 | 0.020 | 0.004 | 0.805 | 0.910 | 0.926 +1.8% |
| Average IWR improvement over the best contrastive baseline | +19.8% | ||||||
Success rate (higher is better); real-robot column reports goals scored out of 20 trials. Shaded cells are IWR; the model-free baselines never leave the ground on the harder manipulation tasks.
Here are the sampled behaviors learned with IWR. Because IWR keeps the contact-time energy accurate, its rollouts commit to the object instead of hovering near it. In Box2D (hard) the contrastive baseline drifts and stalls; IWR drives the puck to the goal and keeps control far longer. In Meta-World pick-place the difference is the whole task: grasping and carrying the block rather than nudging it.
Box2D (Hard)06Crossing the Sim-to-Real Gap
The hardest test is a physical one. We built a real robot air-hockey setup — a UR arm holding a paddle, an overhead camera tracking the puck, and a goal to score against the far edge — and asked policies trained entirely in simulation to transfer zero-shot, with the camera image aligned to the simulator frame by homography.
On the real table the gap is stark. The model-free baselines — PPO, SAC, SAC+HER, SAC+HINT — score 0 of 20. The best contrastive baselines manage 5 and 2. IWR scores 12 of 20, lifting a real robot from a 25% success rate to 60% on a task where every point is decided at the instant paddle meets puck.
07From Reachability to Contact
The central idea behind contrastive RL is that a representation of reachability can also serve as a representation of skill: if the model understands which outcomes are reachable from each state and action, a goal-conditioned policy can turn that geometry into many different behaviors — without demonstrations or task-specific rewards. Manipulation does not invalidate this idea, but it makes the representation problem more demanding. Skills such as pushing, grasping, and striking are organized around contact events that switch the underlying dynamics.
Interaction-Weighted Resampling addresses this problem. It reallocates contrastive supervision toward the futures around interaction events, where reachability changes most sharply. It requires only a rough indication of when contact is likely, preserves coverage of the rest of the trajectory, and leaves the original CRL objective unchanged. This interaction-aware representation improves performance by 19.8% on average across our simulated manipulation tasks and raises real-robot air-hockey success from 25% to 60%.
So, what does it take to learn manipulation skills from scratch? Our results suggest a concrete answer: learn a representation that respects the boundaries where interaction reshapes what is reachable, and train on those moments as if they matter most, because they are precisely where manipulation skills take form.
08References
- Shen, T., Chuck, C., Feng, F., & Huang, B. Learning Object Manipulation from Scratch via Contrastive Interaction.
- Eysenbach, B., Zhang, T., Salakhutdinov, R., & Levine, S. Contrastive Learning as Goal-Conditioned Reinforcement Learning.
- Eysenbach, B., Myers, V., Salakhutdinov, R., & Levine, S. Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference.
- Liu, G., Tang, M., & Eysenbach, B. A Single Goal Is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals.
- Ziarko, A., Bortkiewicz, M., Zawalski, M., Eysenbach, B., & Miłoś, P. Contrastive Representations for Temporal Reasoning.
- Li, Y., Luo, Z., Zhang, T., Dai, C., Kanervisto, A., Tirinzoni, A., Weng, H., Kitani, K., Guzek, M., Touati, A., Lazaric, A., Pirotta, M., & Shi, G. BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning.
- Wang, K., Javali, I., Bortkiewicz, M., Trzciński, T., & Eysenbach, B. 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities.
- Andrychowicz, M., et al. Hindsight Experience Replay.
- Chuck, C., Feng, F., Qi, C., Shi, C., Agarwal, S., Zhang, A., & Niekum, S. Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning.
- Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. Proximal Policy Optimization Algorithms.
- Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning.
- Chuck, C., et al. Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learning.