Turning heterogeneous human demonstrations into robot-specific training data.
Human demonstrations capture how people interact with the physical world and offer a potential source of supervision for embodied AI models. Companies and individuals are increasingly collecting these demonstrations through distinct equipment such as smart glasses, chest-mounted and wrist cameras, UMI-style devices, and head-mounted capture systems, yet the resulting data remains fragmented across capture setups and processing pipelines. Turning these heterogeneous recordings into robot training data requires a shared framework that accounts for different sensing capabilities, connects actions to task context, recovers spatial motion, and adapts it to robot embodiments.
Today, we introduce Chorus, Action Intelligence's cross-embodiment data engine for converting human demonstrations into robot training data at scale. Chorus brings demonstrations from heterogeneous capture setups into a shared intermediate representation, enabling the same demonstration to be adapted for different robot embodiments. To the best of our knowledge, it is the first system to unify this range of human demonstration sources within a single framework for cross-embodiment robot data processing.
Motivation
Diverse capture systems, different robot embodiments, one shared representation
Human demonstrations vary with how they are recorded. Smart glasses capture a moving first-person view, while head-mounted capture systems additionally add more cameras; chest-mounted cameras observe from the torso, and wrist cameras follow the hands. These placements affect hand visibility and camera motion, while monocular, stereo, and multi-camera configurations provide different spatial cues for reconstruction. Additionally, recordings from different sources may differ in the supporting information they provide, such as camera calibration.
Robot embodiments also differ in their kinematics and end-effector design, so the same human motion cannot be directly applied to every robot. For example, a motion feasible for one robot may violate another’s joint limits, while a grasp performed by a human hand may require a different strategy with a gripper. The challenge is to preserve the intended operation while finding a suitable realization for each robot.
Chorus connects these two sides through a shared intermediate representation, a common format for task semantics and reconstructed spatial motion. Recordings are processed according to their capture setup to produce this representation. This common representation serves as input to retargeting methods for different robot embodiments, allowing the same processed demonstration to be reused across robots.
Source processing
Bring heterogeneous recordings into a common representation
Chorus supports source data from smart glasses, chest-mounted and wrist cameras, neckbands, head-mounted capture systems, and UMI-style devices. Supported configurations include monocular, stereo, four-camera, and six-camera setups. Camera calibration and inertial measurements, when available, help constrain geometry, scale, and motion. When these additional constraints are unavailable, reconstruction relies on visual evidence.
Chorus brings the reconstruction results together with semantic annotations in the shared intermediate representation. A shared timeline links each action description to the corresponding motion and visual observations, while a common coordinate system aligns the recovered camera trajectories, hand movements, and scene geometry. Reconstruction uncertainty is retained alongside these outputs, allowing subsequent robot adaptation to account for the reliability of the recovered information.
Semantic annotation
Connect each action to its role in the task
Robot learning requires supervision at the granularity of the actions a robot must perform. An instruction such as “fold a paper airplane” leaves the individual operations—and their temporal boundaries—unspecified. Our in-house frame-level atomic action recognition model breaks continuous demonstrations into precisely bounded, individually described operations, such as aligning an edge, folding a corner, and pressing a crease. This provides explicit supervision for each step and makes incidental movements separable from the actions we want robots to learn.
We use a compact 4B model at 10 FPS, with short temporal windows to capture brief actions and enable efficient parallel processing. Temporal token compression reduces redundancy across frames while preserving the detail needed for fine-grained annotation. With this design, semantic annotation costs approximately 1/20 of the Gemini 3.8 Flash baseline in our current comparison. The resulting atomic annotations support instructions at three levels—atomic actions, subtasks, and complete tasks—providing approximately 3× instruction augmentation that connects individual operations to broader task goals.
Spatial reconstruction
Recover spatial motion and scene from the observed sequence
Semantic annotations describe what happens and when, but do not specify how the hands move through space or where an operation takes place. Chorus recovers this information through depth estimation, hand localization, and object segmentation, which provide spatial observations of the hands and the surrounding scene. Obtaining hand trajectories additionally requires relating the hand positions observed over time to a common spatial reference. Chorus establishes this reference using available camera calibration and pose information to align observations from different frames within the reconstructed scene. When this information is incomplete, Chorus estimates the missing calibration parameters or camera poses from the recording.
Fast movements and occlusion pose another long-standing challenge to motion reconstruction, potentially introducing gaps or abrupt changes in the recovered trajectories. Chorus applies temporal and geometric optimization to refine these trajectories across frames. Note that the refined trajectories remain aligned with the semantic annotations on the source timeline, linking each operation to its reconstructed spatial motion for subsequent robot adaptation.
Cross-embodiment adaptation
Reuse reconstructed demonstrations across robot embodiments
The reconstructed trajectories describe how the human hands moved through the scene. However, a robot has different degrees of freedom and end effectors from a human, so it may follow the demonstrated motion differently or be unable to reproduce parts of it.
To adapt this motion to the target robot, Chorus first positions and orients the robot’s base relative to the demonstrated motion, then retargets human hand movements to end-effector poses. Inverse kinematics converts these targets into joint configurations within the robot’s limits. Forward kinematics then checks how closely the resulting end-effector motion follows the target trajectory. Each joint is also assigned a confidence score reflecting the feasibility and reliability of its adapted motion over the segment.
A trajectory that satisfies the robot’s kinematic constraints may still lead to collisions in the scene. For example, a robot arm may collide with nearby clutter because its shape differs from that of the human arm. To retain demonstrations affected by these incidental obstacles, Chorus uses scene editing while preserving manipulated objects, supporting surfaces, and other task-relevant geometry. The edited scene also provides the basis for synthesizing robot-domain observations.
Chorus currently supports retargeting to 16 robot embodiments for optimal inference engineering, with a shared representation that naturally allows support to expand to potentially endless embodiments. Across the currently supported embodiments, we achieve an average usable-data conversion rate of 85%. Under this measure, one hour of demonstration video passing initial screening yields approximately 13.6 aggregate hours of usable robot trajectories:
1 screened source hour × 16 embodiments × 85% ≈ 13.6 trajectory hours
Robot-domain observations
Generate visual observations aligned with robot motion
After retargeting, the trajectories describe a robot’s movement, but the original video still shows human hands performing the task. Chorus closes this visual gap by projecting the robot along its adapted trajectory into the source camera view, incorporating any scene edits made during adaptation. Replacing the human actor also exposes previously occluded regions and changes how the robot overlaps with surrounding objects. Chorus uses background repair to fill the exposed regions and occlusion-aware compositing to preserve the appropriate visibility relationships between the robot and the scene. The resulting observations are paired with the adapted trajectories to form robot training data.
Data scoring
Evaluate data quality and action surprise
Not all converted demonstrations are equally reliable. Synthesized observations may contain visual inconsistencies, retargeted motions may fail to preserve the intended operation, and reconstructed trajectories may exhibit temporal discontinuities.
To assess data reliability, Chorus evaluates the outputs of individual processing stages across semantic completeness, quality and stability of 3D reconstruction, motion consistency, and visual quality. In addition to assessing quality, Chorus also tracks effective duration and action counts to indicate how much usable data remains after conversion.
Beyond quality assessment, Chorus adds a distinct evaluation dimension: action surprise. Given the preceding context, a language model predicts the next action, and Chorus measures the mismatch between this prediction and the action actually observed. This signal captures aspects of behavior that quality scores alone do not describe, such as unexpected choices, strategy changes, and corrective actions. Chorus uses this signal to prioritize such segments for review, helping determine whether they add training value by expanding the range of behaviors represented in the data.
Processing efficiency
Shared processing lowers the cost of expanding across embodiments
In Chorus, converting a demonstration to different robot embodiments does not mean running all the processing from the beginning. Our shared representation brings together semantic annotations and spatial reconstruction, allowing both to be reused across compatible robot embodiments.
Chorus processes video at approximately RMB 200 per input hour. For an hour of screened video yielding 13.6 aggregate hours of usable robot trajectories, the corresponding cost is approximately RMB 15 per trajectory hour:
RMB 200 ÷ 13.6 trajectory hours ≈ RMB 14.7 per trajectory hour
By adding more usable embodiments from each demonstration in future versions, Chorus can further reduce the processing cost per hour of robot training data.
What comes next
Accelerating robot data scaling through diverse sources
In the previous chapter, we introduced Continuo, which stays with humans, providing useful services while learning from their experiences with consent. Through Chorus, we aim to scale robot pretraining by tapping into the breadth and natural scale of human experience—making better use of existing human data and turning demonstrations collected using different devices, across diverse tasks and environments into robot pretraining data.
Chorus currently processes egocentric human demonstrations. Next, we plan to support additional camera perspectives and real robot execution data, connecting human experience with robot practice to build a more complete and diverse training data pipeline.
Chorus turns human experience into robot pretraining data at scale.
From Human. Make Intelligence Act.