PhysPony: Physics-Aware Global Horse Motion Reconstruction from In-the-Wild Videos

Yiming Huang Siyu Li Qiao Feng Lingjie Liu
University of Pennsylvania
SIGGRAPH Asia 2026
arXiv Code Interactive Demo
PhysPony teaser: input video frames above, the reconstructed global horse trajectory in the middle, and the physics-refined simulated rollout below.

Abstract

Reconstructing quadruped motion from in-the-wild videos is vital for understanding animal behavior and advancing legged robot learning, but remains challenging due to the inherent ambiguity of monocular visual cues. Existing methods focus on per-frame pose recovery in camera space while neglecting global trajectory modeling, often resulting in limb ambiguity, inconsistent gait patterns, and physically implausible foot contacts.

We present PhysPony, a framework that unifies global trajectory estimation, local articulation reasoning, and physics-based motion refinement for horses. PhysPony combines global trajectory cues from a foundation 4D reconstruction model, fine-grained articulation features extracted by 2D vision encoders, and physics-based test-time optimization via an hSMAL template-based simulation agent guided by a mocap-pretrained tracking policy. Together, these components enable physically consistent dynamics and stable long-horizon trajectories directly from in-the-wild videos. Experiments on PASCAL, AnimalMotion, and PFERD show that PhysPony substantially outperforms prior approaches in both trajectory coherence and physical plausibility.

Method

Given a video of a horse, PhysPony recovers a metric-scale camera trajectory from 4D vision foundation models, estimates shape and local articulation with a motion fusion transformer over pretrained encoder features, and refines the combined motion in simulation to fix physical artifacts.

PhysPony pipeline: global trajectory estimation from UniDepthV2 and FastVGGT, a motion fusion transformer distilling a pose reconstruction teacher model, and physics-based test-time refinement with a pretrained tracking policy.
The hSMAL simulation agent: articulated skeleton with joint locations on the left, mesh appearance on the right.
The hSMAL simulation agent — 36 joints, 35 actuated, 105 degrees of freedom.
The mocap-pretrained tracking policy on held-out PFERD sequences, rendered in Isaac Gym. Green is the reference motion, blue the simulated agent tracking it.

Interactive Demo

Kinematic reconstruction (ghosted) against the physics-refined rollout (solid), with Ponymation as a baseline, in one world frame and on one timebase. Both kinematic tracks are placed by the same metric-scale global trajectory estimated by our pipeline; Ponymation predicts in camera space only, and learns its mesh instead of posing a template, so it is shown at the scale its own canonical space implies. Ground grid squares are 1 m. Drag to orbit.

initialising renderer…
input video

Scene —

0
30 fps
drag orbitscroll zoomright-drag panspace play← → step

BibTeX

@inproceedings{huang2026physpony,
  title     = {PhysPony: Physics-Aware Global Horse Motion Reconstruction from In-the-Wild Videos},
  author    = {Huang, Yiming and Li, Siyu and Feng, Qiao and Liu, Lingjie},
  booktitle = {SIGGRAPH Asia 2026 Conference Papers},
  year      = {2026}
}
Download .bib