An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning

All variants we study share a base policy: a Diffusion Transformer (DiT) action head trained with a flow-matching objective, which cross-attends from proprioception and noised-action tokens to image tokens and self-attends over the action chunk. We vary this base policy along three axes — whether visual features are compressed before the action head, which visual encoder produces them, and how token positions are encoded — to isolate which choices drive viewpoint generalization. The combination that wins on all three axes — dense tokens, a cross-view finetuned DINOv2 encoder, and robot-frame 4D rotary positional encodings — is what we call AGP.

Out-of-distribution viewpoints. The comparisons above sample test cameras from the same distribution as training, which measures viewpoint robustness. To measure viewpoint generalization, we additionally evaluate Cabinet with a broader camera distribution that observes the scene from unseen elevated and oblique angles.

Comparison to a 3D policy. Policies that operate directly on point clouds are viewpoint-robust by construction, since their inputs can be expressed in a scene-centric frame. We compare against 3D Diffusion Policy (DP3), which consumes a ground-truth-segmented, fused point cloud of the object plus a 4-point gripper point cloud as proprioception.

Robustness to calibration and depth noise. AGP lifts pixels into the robot frame using camera extrinsics and depth — both imperfect in the real world — so we stress-test a Cabinet policy trained without any noise under increasing test-time perturbations of each.

We build on ArticuBot, an automated pipeline for generating demonstrations of a Franka robot opening articulated objects. From its set of 322 objects we select five microwaves, generate 2,358 demonstrations and over 100,000 state-action pairs, and render them in IsaacLab with extensive domain randomization over microwave colors and textures, the robot, the table, and the background. We render two over-the-shoulder views and one wrist view per trajectory, with camera intrinsics matched to the ZED2 and ZED-mini cameras used in our real-world setup, and augment depth maps to simulate sensor noise and edge artifacts. At deployment, real-world depth is obtained with Fast-FoundationStereo and cropped to the workstation to match the training distribution; camera poses are unknown during data generation but within the randomized training distribution. We mount three cameras and evaluate all three possible camera pairs, for 30 trials per policy (10 per pair), reporting grasp success (visually confirmed every trial) and normalized opening.

We study viewpoint generalization, relaxing the common constraint of fixed cameras at training and deployment. Our controlled study identifies what drives it: retaining dense scene representations instead of compressing them before the action head, and grounding every visual, proprioception, and action token in the geometry of the scene. We instantiate these findings as AGP — deliberately not a new architecture, but the combination of existing ingredients responsible for viewpoint robustness — a policy that is performant for all views in the camera distribution, approaches fixed-camera performance, and transfers zero-shot from simulation to a real robot under random cameras.

Limitations. AGP requires calibrated RGB-D input at both training and deployment — depth (sensed or estimated) and camera extrinsics in the robot base frame are needed to lift each pixel into the shared coordinate system the positional encoding relies on. Removing this requirement, e.g. by jointly estimating extrinsics or operating on monocular RGB with learned depth priors, is an important direction. We also demonstrate sim-to-real as a proof of concept on a single object category (microwaves); scaling these findings to large, heterogeneous real-world datasets such as DROID remains an open question.