TL;DRA Latent Action Space for Transferable World Models
Embodied world models are usually conditioned on raw action commands, but those commands are tied to a particular body, controller, and calibration. SCAR treats action as a distinct factor of visual change. An inverse dynamics model infers stochastic latent actions from pairs of latent observations, while a forward dynamics model learns to predict future latent video dynamics conditioned on those actions.
To make the learned action space transferable, SCAR constrains the latent posterior toward a standard Gaussian prior and uses adversarial embodiment invariance to remove nuisance information. Experiments on Procgen and Robotwin show stronger low-data adaptation, better cross-task transfer, and reduced embodiment leakage compared with raw-action or unconstrained latent-action interfaces.
The latent action is learned from observation transitions, not copied from embodiment-specific command vectors.
KL limits visual shortcuts while GRL suppresses embodiment-discriminative information.
Shared latent actions provide a stronger conditioning interface under low labeled target data.
01Sufficient Conditions for Unified Latent Action Recovery
SCAR's theory starts from a structured data-generation view: an embodiment-invariant latent action \(u_t\) is realized as an embodiment-specific command \(a_t^e = h(u_t, e)\), which drives the transition \(s_{t+1}=F(s_t,a_t^e)\) and is observed through rendered frames \(x_t=R(s_t)\). Under injectivity assumptions on realization, dynamics, and rendering, the theory gives sufficient conditions under which visual transitions can recover a shared latent action interface rather than only robot-specific commands.
If inverse dynamics recovers the realized raw action coordinate up to an invertible reparameterization and the recovered coordinate decomposes into a shared controllable component plus an embodiment-specific nuisance component, then a population-level adversarial invariance objective removes the nuisance component and recovers a unified latent action space up to a per-embodiment invertible bijection \(z=\Phi_e(u)\).
Inverse dynamics identifiability
Because the render map and forward dynamics are injective on the data support, an inverse dynamics model that globally optimizes transition prediction can recover the realized action up to a continuous invertible reparameterization \(\tilde{a}_t = \rho(a_t^e)\).
Embodiment-invariant selection
The recovered coordinate may still contain embodiment bias. With an optimal embodiment classifier, the adversarial objective satisfies \(\min_\omega \mathcal{L}_{\mathrm{CE}}=H(e\mid z)=H(e)-I(e;z)\), so gradient reversal pushes the encoder toward \(z\perp\!\!\!\perp e\).
Geometry of nuisance removal
In the proof, embodiment cluster centers span a nuisance subspace \(V\). Invariance forces all embodiment centers to collapse after encoding, which implies \(\operatorname{row}(M) = V^{\perp}\) when the rank and dimension match.
Unified latent recovery
For each embodiment \(e\), the theorem defines \(\Phi_e(u):=f_2(\rho(h(u,e)))\). In the linear proof setting, \(f_2(\tilde a)=M\tilde a\), so this becomes \(M\rho h(\cdot,e)\). By invariance of domain, \(\Phi_e\) is a bijective coordinate chart on its image.
Cross-embodiment alignment
The theorem frames transfer between embodiments \(e_1\) and \(e_2\) through the alignment map \(T=\Phi_{e_2}\circ\Phi_{e_1}^{-1}\), with the common pushforward law \({\left(\Phi_e\right)}_{\#}p(u)=p(z)\). Recovering \(u\) from \(z\) still requires the embodiment ID \(e\).
Why KL and prediction remain necessary
The formal result explains what invariance selects under idealized conditions. In the model, forward prediction prevents trivial invariant collapse, while KL regularization limits unrestricted visual coding and makes the latent action harder to use as an appearance shortcut.
This is a sufficient-condition analysis under idealized structured-nuisance and non-collapsed prediction assumptions, not a claim that every trained model automatically obtains perfect zero-shot transfer.
02Inverse-Forward Latent Dynamics with Anti-Leakage Constraints
Infer latent actions
The IDM reads latent observation pairs and produces stochastic latent action tokens that explain realized visual change.
Predict future dynamics
The FDM conditions a pretrained video dynamics backbone on those latent actions, keeping the representation transition-relevant.
Remove nuisance factors
KL discourages high-capacity visual encoding, while gradient reversal makes the latent action less predictive of embodiment identity.
Recover controllability
A context-conditioned action-to-latent controller maps raw command sequences into the learned latent action space at inference time.
A raw robot command is a local interface. SCAR instead asks the world model to condition on the part of a transition that explains controllable change.
03SCAR Improves Transfer Under Low Target Data
We evaluate whether a shared latent action interface transfers motion structure across embodiments and supports few-shot world-model adaptation when labeled target data is scarce.
Cross-Embodiment Action Transfer
We extract latent actions from a source embodiment trajectory and apply them to a target embodiment visual context. The goal is to transfer the source motion structure while preserving the target robot appearance. The demos compare raw latent actions, latent actions with KL regularization, and latent actions with both KL and GRL regularization.
Example 01Few-Shot Target Embodiment World Model
We train target-embodiment world models with limited target demonstrations and compare different action-conditioning interfaces. The demos show whether each interface can preserve target embodiment appearance while producing coherent future dynamics under the same visual context.
Example 0104Why the Learned Action Matters
A raw robot command is a local interface. The same command can mean different physical interventions across bodies, controllers, and scenes. SCAR instead asks the world model to condition on the part of a transition that explains controllable change.
This view makes embodiment a nuisance variable. The latent action must remain useful for prediction, but it should not reveal which robot produced it. The KL and GRL terms act as complementary pressure: one limits capacity, the other removes embodiment-discriminative directions.