STREAM

Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

Seong Jong Yoo, Siyuan Peng, Felix Gu, Stratis Aloimonos, Cornelia Fermüller
Perception and Robotics Group
University of Maryland

ECCV 2026 Accepted!

Dance Design Studio Demo. Try out at https://github.com/SeongJong-Yoo/STREAM

Abstract

Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics. While existing models can synthesize motion from music, they remain largely black boxes. Conversely, attempting to condition generation on both text and music frequently leads to modality collapse, where dense acoustic rhythms overwhelm sparse semantic text prompts, destroying user controllability. To resolve this spatial-temporal conflict, we propose STREAM (Structural-Temporal Rhythmic Energy-based Attention for Motion), a modality-decoupled diffusion transformer. STREAM strictly separates conditioning pathways: global text semantics dictate the kinematic structure via Adaptive Layer Normalization (AdaLN), while a novel Bimodal Energy-Based Attention Module (BEAM) routes these features to the musical beat without overwriting the semantics. Central to this effort is Motorica++, a newly curated dataset enriched with domain-specific dance vocabulary and frame-level semantic annotations. Furthermore, to rigorously quantify zero-shot editability, we introduce the Exchange Evaluation Protocol and Editable Dance Score (EDS). Through extensive experiments, STREAM achieves state-of-the-art alignment between motion and music while perfectly preserving choreographic semantics, positioning AI not merely as a reactive synthesizer, but as a controllable, collaborative partner for artistic direction.

Architecture

**Figure 2**. Overview of STREAM. (a) **Left**: STREAM is first controlled by text (high-level concept and a low-level detailed description). Second, the music condition modulates the motion via the music alignment energy function, transforming general motion into *dance-like* motion aligned to musical beats. (b) **Right**: The Bimodal Energy Attention Module (BEAM) adaptively updates the text conditions (red line) via MAP estimation, and global text information is applied through AdaLN modulation.

Results

**Figure 5.** Visualization of dance motion editing example. The original motion contains two dance techniques: Charleston Opposites (green) and Charleston Messaround (blue). We first edit Charleston Opposites to Charleston Side to Side (red) while preserving the other. Similarly, we can edit one more time to Charleston Knock Knees (yellow).

Dataset

Examples

Genre: charleston
Label: double
Description: The person repeatedly performs a step-side-kick-step sequence while balancing on the supporting leg. Throughout the movement, the arms are held out to the sides with elbows slightly bent for balance. While stepping, the person slowly rotates in one direction.

Genre: charleston
Label: hip sway
Description: The figure sways its hips from side to side while taking small, alternating steps forward. Initially, the arms are held out to the sides, then raised straight up, and finally held sideways with bent elbows.

Genre: charleston
Label: jazz square
Description: The person performs a jazz square by stepping forward with the left foot, crossing the right foot over the left, stepping back on the left, and stepping out to the right. Throughout the four-step sequence, their arms remain bent and swing loosely in rhythm with their legs.

Genre: charleston
Label: skating step
Description: The person alternates kicking each leg forward while holding both arms straight out to the sides. The rhythm of the kicks creates a subtle wave-like motion throughout their body.

Genre: hip_hop
Label: biz markie
Description: A person stands slightly bent over with their arms extended straight out to the sides at shoulder height. They bend their torso to the right and push their arms forward while stepping back, then hop to the left and thrust their arms upward. Next, they lean to the left, thrusting the right arm forward and the left arm back, before straightening with their arms at their sides. They jump sideways to the left and push their arms up, then lean forward and push their arms out, before straightening and repeating the movement to the right. After jumping to the left and repeating the sequence again, they bend their elbows, shift their shoulders from side to side, and straighten. Finally, they jump to the center and thrust their arms out at shoulder height, then rotate clockwise and extend the right arm forward while keeping the left arm at their side.

Genre: hip_hop
Label: body roll up
Description: A person performs a body roll, starting from a crouch and pushing their hips forward and up. The motion flows sequentially up the torso and through the head as the body straightens.

Genre: hip_hop
Label: crisscross
Description: Starting from a crouched position, the individual crosses one leg in front of the other while pulling their arms inward. They then uncross their legs by stepping one foot out to the side and extending their arms back out. This movement is repeated rhythmically, alternating legs with each step.

Genre: hip_hop
Label: gucci
Description: A person hops from side to side on both feet while swinging their elbows forward in rhythm with the jumps. In the middle of the movement, the person performs a squat.

Genre: krump
Label: chest hit
Description: The performer adopts a wide stance with their arms held forward, elbows bent, and forearms angled downward. They lean back and swing their forearms back and forth, then straighten to push both arms and the left leg forward. Stepping back with the left leg, they bend over and punch both arms forward, then swing both arms back while turning slightly clockwise as they continue to step back.

Genre: krump
Label: leg swing slow
Description: The person balances on their right leg with arms raised forward and a bent torso, slowly swinging the left leg from back to front. Then, they stand on the left leg while kicking and swinging the right leg without letting it touch the ground.

Genre: locking
Label: uncle sams points
Description: Starting from a standing position, the person extends their right arm straight forward while holding their left arm downward. They then extend both arms straight out to the sides and bring them back toward the body. Finally, the person repeatedly folds and extends their right arm side-to-side while holding their left arm straight out to the side.

Genre: popping
Label: robot
Description: Starting from a standing position with both forearms held in front of the chest, the person opens their chest by pulling their bent elbows back. They then bow forward while maintaining a scarecrow-like upper-body posture. After swinging their arms side to side and back and forth, they lower themselves into a squat. Finally, they swing their arms forward with their knees turned inward and bring the left leg toward the right.

BibTex

			
@inproceedings{yoo2026stream,
  title     = {Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation},
  author    = {Yoo, Seong Jong and Peng, Siyuan and Gu, Felix and Aloimonos, Stratis and Fermüller, Cornelia},
  journal = {European Conference on Computer Vision, ECCV},
  year      = {2026},
}

		

License

STREAM and Motorica++ are available for non-commercial and research use only and may not be redistributed and should follow requirements under the conditions detailed on the license page. For commercial licensing or if you have any questions, please get in touch with us at yoosj@umd.edu.

The copyright of original Motorica dance dataset belongs to https://github.com/simonalexanderson/MotoricaDanceDataset/