HOT
Learning Generalizable Hand-Object Tracking
without Human Demonstration
This work learns generalizable hand-object tracking controllers purely from synthetic data, without any human demonstrations. HOG synthesizes diverse hand-object trajectories, and HOT tracks them robustly across objects, long-horizon sequences, and different dexterous hands.
Abstract #
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations.
Our approach makes two key contributions: (1) HOG, a highly efficient Hand-Object data generator that rapidly synthesizes diverse, large-scale, and uniformly distributed trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning. Notably, we demonstrate that the fast and intentionally noisy data generation process of HOG is crucial for endowing HOT with robust interaction dynamics.
Our method extends to diverse object shapes and hand morphologies. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from simple, scalable synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
Contribution Summary
-
1.
Synthetic data only
-
2.
HOG trajectory synthesis
-
3.
HOT robust tracking
-
4.
Generalization across settings
Select a contribution
Click a card on the left to view its detailed summary.
Method Overview
Click a highlighted part of the framework below.
Framework
(a) HOG synthesizes manipulation trajectories for meta-skills. (b) HOT is trained through a two-stage teacher-student framework using reinforcement learning with a unified HOI imitation reward, enabling robust tracking of the target HOI trajectories. (c) At inference, the system can accept high-level waypoints from language models, generative models, or human data, which HOG converts into trajectories for HOT to track, enabling diverse applications.HOG synthesizes trajectories and HOT learns to track them for downstream manipulation.
HOG synthesizes manipulation trajectories from grasp poses generated by force-closure optimization and refined by RL. Its grammar-based approach supports eight composable meta-skills, offering multi-source parameter control via randomization, LLM/VLM instructions, or from human demonstrations. The system naturally generalizes across diverse hands and objects for scalable data coverage.
Skill references coming soon
Reference videos for this HOG skill will be added soon.
Diverse Zero-Shot Tracking Applications
Tracking with VLM ➕ HOG Tracking with Generative Model Tracking with Human Demonstration1. HOG Synthetic Trajectories / HOT Tracking Results #
HOG generates diverse, large-scale, and uniformly distributed trajectories. HOT learns robust hand-object interaction dynamics by tracking these synthesized trajectories.
2. Zero-Shot Tracking from Generated and Human HOI References #
HOT zero-shot tracks challenging hand-object sequences from GRAB, and also accepts text-conditioned interaction goals such as Text2HOI prompts.
3. Tracking Long Sequences #
HOT enables dexterous hands to track long-horizon sequences including rearrangement and multi-stage manipulation.
4. VLM Planning + Our Method #
HOT can take high-level waypoints from upstream planners and track the resulting hand-object interaction sequence.
5. Different Dexterous Hand #
The framework extends beyond a single hand morphology and generalizes to different dexterous hand embodiments.
6. Cross-Simulator Results #
We validate sim-to-sim transfer in MuJoCo across two dexterous hand embodiments.
7. Real-World Deployment #
We deploy HOT on a physical robot system comprising a Flexiv Rizon 4 arm and a Revo3 hand.