Maniformer and researchers from Northwestern Polytechnical University and Xiβan Jiaotong University have developed MEgoVista, an offline system that turns footage from the MEgo View headset into metric-scale 4D reconstructions of a wearerβs hands and head.
Paper: https://arxiv.org/abs/2609.16684
The system is intended to make first-person video more useful for embodied AI training. Video can capture activities in homes, workplaces, and other everyday settings, but it does not automatically provide the measurements needed to teach a model how people move. Those measurements must show which hands belong to the wearer, how the fingers are positioned, and where the hands are in real-world space.
The Challenge of Reconstructing Everyday Motion
Capturing human movement accurately outside a lab is difficult. Multi-camera motion-capture studios can provide precise measurements, but require specialized equipment and controlled environments. Meanwhile, large collections of everyday video offer a wider range of activities but generally do not include metric-scale 3D annotations for hand position.
Methods based on a single camera also have trouble establishing absolute distance. They may produce a hand pose that looks right in the image while estimating its location incorrectly in 3D.
Everyday activities introduce additional complications. A bystanderβs hands may appear beside the wearerβs hands or make contact with them. Tools and arms can hide parts of the fingers, and head movement shifts the cameraβs viewpoint. A reliable system needs to handle these conditions while separating the wearerβs motion from movement in the scene.
A Three-Part Motion Reconstruction Pipeline
MEgoVista processes footage in three main stages: hand identification, depth estimation, and sequence refinement.
It first identifies the wearerβs hands. The system detects hands, labels each as left or right, and determines whether it belongs to the person wearing the headset. This is designed to prevent a nearby personβs hand from being assigned to the wearer, even during physical contact.
It then estimates real-world distance. The MEgo View headset uses three fisheye cameras recording at 60 frames per second. The center camera forms one stereo pair with each side camera. MEgoVista rectifies the camera images using pre-calibrated baselines, estimates depth with FoundationStereo, and combines the two depth maps according to their confidence.
It refines the motion over time. MEgoVista evaluates observations across the full video sequence instead of treating every frame independently. It filters for reliable data and optimizes the motion sequence, aiming to reduce jitter and produce smoother, anatomically consistent finger poses.
For head motion, the system uses the multi-camera images to estimate the trajectory, establishes metric scale at the start, and uses inertial measurements to determine the vertical direction. The reconstructed hands and head are represented in a shared metric-scale 4D coordinate system.
Results Against Motion-Capture Reference Data
The research team assessed MEgoVista using reference measurements from a separate optical motion-capture setup with 26 cameras. Missed hand detections were included in the evaluation so that methods could not improve their results by omitting challenging frames.
For finger-pose error (PA-MPJPE-p), MEgoVista achieved 4.29 mm. HaWoR measured 16.30 mm, while Dyn-HaMR measured 49.18 mm. For absolute wrist position in the camera frame (CT-p), MEgoVista recorded an error of 14.30 mm, compared with 46.70 mm for HaWoR and 184.38 mm for Dyn-HaMR.
MEgoVista had no missed or false hand detections in the test set. Its motion jitter was 1.11 mm/frameΒ², lower than that of both comparison systems. In two head-trajectory tests, its average position errors across the full sequences were 2.87 mm and 0.75 mm. The offline-refined ORB-SLAM3 baseline recorded 3.27 mm and 1.00 mm, respectively.
The researchers also examined footage from three everyday tasks:
- Preparing milk tea: MEgoVista and Dyn-HaMR tracked the hands closely during fast movements. HaWoR showed a clear misalignment around the left hand.
- Performing a manicure: HaWoR and Dyn-HaMR sometimes mistook a bystanderβs hand for the wearerβs when the two peopleβs hands touched. MEgoVista identified the wearerβs hands correctly.
- Washing an animal: MEgoVista and Dyn-HaMR detected the hands despite rapid movement and soiling. HaWoR showed displacement in the left hand and wrist.
Across these examples, MEgoVistaβs reconstructions more closely followed the observed hand positions and handled interactions between two people more reliably in the tested scenes.
By converting egocentric video into metric-scale hand and head motion, MEgoVista offers a way to create more directly measurable data for robot learning. Its evaluation compares those reconstructions with independent motion-capture measurements and tests performance in varied real-world activities.
About Maniformer
Maniformer develops data infrastructure for Physical AI and embodied intelligence. Its platform combines collection hardware, data processing, validation, and related services to turn physical interactions in the real world into data for AI development and deployment.
Find out more about Maniformer here: https://maniformer.ai/en