VLA Robot Learning

The policy foundations beneath modern vision-language-action systems.

A vision-language-action model is still a robot policy: it must turn what the robot sees, what it is told, and what it knows about its body into a physically meaningful next motion. Before asking how web-scale language knowledge or multi-robot data help, it is useful to understand the action-learning methods that make that last step work.

One observation can admit several good motions. The central question is therefore not only how to predict an action, but what object the policy should represent: a single action, a short action plan, or a distribution over complete action chunks.

A Common Policy Interface

At control time t, let I_t denote one or more camera images, q_t the robot state, and g a goal or language instruction. Write the observation as

An action a_t may mean joint positions, an end-effector displacement, gripper commands, or mobile-base velocity. Its dimension is d_a. A conventional one-step policy is

where θ are learned parameters and o_≤t is either the current observation or a short history.

Many methods instead predict an action chunk of horizon H:

The robot normally executes only the first action, or a short prefix, then observes again and queries the policy again. Chunking is therefore not a commitment to open-loop control. It gives the policy a local plan while preserving feedback.

Behavior Cloning: Learn Directly from Demonstrations

Suppose a dataset 𝒟 contains demonstrated observations, goals, and actions. Behavior cloning learns the demonstrated conditional distribution by maximum likelihood:

For a Gaussian policy with fixed variance, minimizing this loss is equivalent to minimizing squared prediction error. Behavior cloning needs demonstrations but no manually designed reward or simulator. It is also the baseline from which the later methods depart.

Its first limitation is distribution shift. A small deployment error produces a state that is absent from the demonstrations. The policy then has to act on an unfamiliar observation, which can cause the next error to grow. Its second limitation is multimodality. If going left and going right are both valid ways to reach an object, a squared-error policy can average them into a motion that reaches neither.

A longer history helps, but does not choose a trajectory

A sequence policy changes the information available to behavior cloning, rather than the objective itself:

Here L is the history length and h_t is a context-dependent feature. Attention can recover information hidden by one image: whether a drawer was already opened, for example, or whether an object is moving. It does not by itself solve action ambiguity. A Transformer can still be trained as a deterministic regressor. The next change therefore concerns the output itself: predict a short plan rather than one action.

FiLM: make the goal alter perception

FiLM [1] originated in visual reasoning, but it is a useful primitive for goal-conditioned control. Given visual features F and a goal representation g, it applies

The functions γ and β produce goal-dependent scale and shift vectors; ⊙ is element-wise multiplication. The FiLM block in Figure 1 applies the scale channel by channel, then adds the shift, producing the modulated feature map F′. A policy head can combine F′ with the robot joint state to predict a concrete action.

Thus a command such as “pick up the red block” can change the visual features passed to the action head before an action is predicted. FiLM does not solve data collection or action generation. It isolates a key VLA design problem: how language should reshape visual evidence for control.

ACT: Predict a Short Coordinated Plan

ACT—Action Chunking with Transformers—was introduced in [2] for precise bimanual manipulation. Its central change is simple: rather than asking the network for one action, ask it for a short coordinated plan.

The paper’s ALOHA setup makes the dimensions concrete. It has two arms. Each arm has six rotational joints and one gripper coordinate, so its joint state has seven degrees of freedom; both arms together give

The observation also contains four 480 × 640 RGB cameras: one global top view, one front view, and one close-up camera on each wrist. An action is the next desired 14-dimensional joint configuration. If the chunk horizon is k, ACT predicts

The two halves of Figure 2 happen at different times. The left conditional-VAE encoder exists only during training; the right Transformer policy is the component retained for control.

Training-time CVAE encoder

During training, the encoder sees the current joint state and the demonstrated future chunk, but not the images. Omitting images makes this auxiliary branch cheaper to train; its job is to summarize variation in human action sequences, not to become the deployed visual policy.

The encoder receives three token groups:

The learned [CLS] token is prepended to the joint token and the k action tokens. After the BERT-like Transformer encoder, its output vector

is sent through two linear heads to obtain the parameters of a Gaussian posterior:

Sampling is implemented with the reparameterization trick,

The latent z represents a style of a demonstrated local motion: for the same state, a human may take slightly different valid paths, especially before the final precise contact.

Deployed visual action-chunk policy

At deployment, there is no future demonstrated action sequence, so the left encoder cannot be used. The policy instead uses the prior mean, z = 0. Each of the four camera images passes through a ResNet-18 visual encoder. A 480 × 640 image becomes a 15 × 20 × 512 feature map; flattening its spatial locations gives

visual tokens per camera. With four cameras, the visual input is 1200 × 512 tokens. ACT adds a 2D sinusoidal positional embedding so that flattening does not discard where a feature came from in an image.

The visual tokens, the projected joint state, and z form the encoder memory:

The Transformer decoder uses k fixed positional query embeddings and cross-attends to this memory. Its output is a sequence of k 512-dimensional action tokens,

which a small MLP maps to the actual robot command:

This is the meaning of action chunking in ACT: predict the next k full bimanual joint targets as one coordinated object, rather than treating every 14-dimensional action independently.

What the loss is asking for

The policy and training-only encoder are optimized together with

The reconstruction term is L1, not L2. A large deviation in one noisy human demonstration contributes linearly rather than quadratically, so it cannot dominate the update as strongly as it would under MSE. The ACT paper reports that this choice produced more precise action-sequence modeling [2]. L1 does not mathematically guarantee precision; its practical value here is robustness to imperfect and multimodal human demonstrations.

The KL term keeps the posterior inferred from a demonstration close to the standard normal prior. It prevents the encoder from hiding arbitrary information in z, which is essential because deployment uses z = 0 rather than an encoded future action chunk.

Why absolute joint targets help

ACT’s 14 action coordinates are absolute joint positions, not joint increments. A predicted target is therefore expressed in one fixed joint-coordinate system. If the policy is queried again after a small execution error, its new target is still an absolute configuration; the representation does not numerically sum a chain of its own predicted deltas.

This does not eliminate physical error accumulation. Contacts, calibration error, latency, and a missed grasp still require visual feedback. The benefit is narrower and useful: the command representation itself does not recursively integrate prediction errors. Querying a fresh chunk at every control step lets the four-camera observation correct the plan in closed loop.

ACT queries a new chunk at every control step. Several predicted chunks can contain an estimate for the same physical action, so the policy temporally ensembles those estimates:

The symbol â_t^[t−i] means the prediction for time t made at time t−i. The coefficient m determines how rapidly newer predictions dominate. Figure 3 illustrates this: the colored boxes aligned in one column are several predictions for one physical control instant. ACT averages that column, not actions from different moments.

ACT reduces the effective planning horizon and can coordinate tightly coupled bimanual motion. It remains a demonstration learner for a specific embodiment: it does not by itself provide experience with new objects, recovery states, or a different robot.

Diffusion Policy: Model the Distribution of Action Chunks

Diffusion Policy [3] changes the output model. Rather than predict one chunk directly, it learns to turn a noisy chunk into a demonstrated chunk, conditional on the observation.

Let A⁰ be a clean demonstrated action chunk. A forward diffusion process corrupts it at noise level k:

The schedule is defined by small noise variances β_k, with

Thus ᾱ_k is the cumulative signal retained after k noising steps. A neural network receives the noisy chunk, observation, goal, and noise level, then predicts the injected noise:

At execution, the policy starts from Gaussian noise and repeatedly applies reverse denoising steps. Given the current noisy chunk A^k, one standard noise-prediction reverse step is

For the usual DDPM variance choice,

The network estimate subtracts the noise thought to be present in A^k; the final term injects the controlled randomness required by a stochastic sampler. Other samplers, such as deterministic DDIM-style updates, modify or remove that final noise term, but retain the same idea: use the conditional network prediction to move one step from A^k toward A⁰.

After the final reverse step, the policy has a coherent action chunk. It executes a prefix, observes again, and repeats the process. Because it generates an entire correlated chunk, it can choose one coherent mode—one approach trajectory or grasp strategy—rather than regress the mean of several incompatible modes.

The price is sampling cost: a reverse trajectory normally requires several network evaluations before the robot can act. Diffusion resolves a modeling failure of deterministic BC; it does not remove latency constraints or the need for demonstrations that include contact and recovery.

Flow Matching: Learn How Noise Moves Toward an Action

Flow matching offers another way to build a continuous generative policy. It is the action-generation idea used by the π0 family [4], discussed in detail in Part V.

Let X₀ be Gaussian noise and X₁ a demonstrated action chunk. Define a straight interpolation between them:

Along this path, the target velocity is X₁ − X₀. A conditional vector field v_θ is trained to predict that velocity:

At inference, start from noise and integrate

from s = 0 to s = 1. With N equal flow steps, define

The simplest Euler solver then performs the concrete update

After N steps, X₁ is the sampled continuous action chunk. Higher-order solvers, such as Heun or Runge-Kutta methods, evaluate the same learned velocity field more carefully, but follow the same path from noise to action. Unlike the stochastic DDPM update above, this basic flow-matching sampler does not inject a fresh random noise term at every step; its randomness comes from the initial draw X₀.

Diffusion and flow matching should not be treated as unrelated buzzwords. Both define a path from a simple noise distribution to the demonstrated action distribution, conditioned on observation and goal. Diffusion commonly learns to reverse a discrete noising process; flow matching learns a continuous velocity field along an interpolation. Their engineering trade-offs concern action quality, number of inference steps, numerical solver, and control latency.

What Changes Across These Foundations?

The progression is not that each method discards the previous one. Every method still depends on an observation interface, an action convention, demonstration quality, and closed-loop replanning. The changing design choice is the probabilistic object the policy represents: a point estimate, a latent-conditioned chunk, a denoising process, or a flow field.

What to Carry into the Next Parts

1. Action representation is architecture. Tokenized, diffusion, and flow-based actions change both learning and runtime behavior.
2. Chunking supplies local temporal structure. It does not excuse the policy from observing and recovering in closed loop.
3. A larger visual-language backbone does not replace a motor policy. It still has to condition a concrete action generator such as one of the methods above.

Part II will broaden the question from how a policy generates actions to how one policy can be trained and adapted across datasets and robot embodiments.

References

1. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI Conference on Artificial Intelligence, 2018.
2. T. Zhao, V. Kumar, S. Levine, and C. Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Robotics: Science and Systems, 2023.
3. C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Robotics: Science and Systems, 2023.
4. Physical Intelligence. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164, 2024.