One model, no retraining. Each panel is the same frozen generator; only the test-time cost changes.
A generative tracker
At each 50 Hz tick a one-step diffusion generator plans future states and actions from a sliding reference window. An inverse-dynamics head executes the plan. Train it by distilling a teacher, or with RL from scratch.
Steered by costs
Gradient steps on the input noise pull the plan toward any differentiable cost. Masked body groups give way to the cost while the rest keeps tracking.
Knows where it fails
Feed one reference with different noise samples. Where the plans disagree, the tracker usually fails. No simulator, no extra model.
Real robot
Running on a Unitree G1
DiffWBC tracking reference motions on the real robot, one generator call per 50 Hz tick. The overhead gantry is a slack safety line.
Simulation rollouts
Steering a frozen tracker
Steering bends the plan
Each clip shows the generator’s 0.96 s plan as ghosts. Top: without steering (grey ghosts, red where the plan hits the obstacle; gold is the reference motion). Bottom: the same reference after test-time steering (blue ghosts). The cost moves the plan itself, and the robot executes the moved plan.
Bars, reference motion A
Bars, reference motion B
Bars, reference motion C
Low ceiling, reference motion A
Low ceiling, reference motion B
Low ceiling, reference motion C
Other objectives, side by side
Every clip is a closed-loop physics rollout. Panels run left to right. The leftmost panel is the reference or the RL teacher. The others are DiffWBC, without and with steering.
High-five
A hand-target cost, 12 descent steps. Only the arm group is masked to yield; the gait keeps tracking.
Tray balancing
Teacher, DiffWBC without steering, DiffWBC with a tray-tilt cost (0.3).
Weak knee motor
Teacher, DiffWBC without steering, steered when told about the fault, steered with a fault detector.
Velocity commands
Joystick walk-and-turn command (vx 0.83 m/s, yaw rate −0.67 rad/s; green disc). Teacher, DiffWBC with the leg reference masked and no steering, and DiffWBC with a root-velocity cost. Measured velocities are printed under each panel.
Dynamic: 360° flip
Reference, RL teacher, DiffWBC. Plain tracking; the full clip is completed with the landing.
Dynamic: 360° flip, second clip
Reference, RL teacher, DiffWBC. Plain tracking; the full clip is completed.
Adding a condition
Terrain is one more input
The generator can take a new condition without changing its structure. We encode a 21×21 height scan around the robot into 36 terrain tokens and append them to the generator’s input sequence. The reference stays a flat-ground motion, the inverse-dynamics head is unchanged, and the model is fine-tuned with RL on stairs, slopes, boxes and rough ground.
Stairs up
Stairs down
Slope
Boxes
Boxes, another motion
Rough ground
Left: the flat-ground reference motion. Right: the terrain-conditioned model on the terrain shown. The label gives the terrain family, the difficulty level (L0–L9) and the final distance to the end of the reference path. These clips are picked from cells the model passes; the table below covers all cells.
Success (no fall, ends within 1 m) ↑
Flat
Stairs down
Stairs up
Slope
Boxes
Rough
All
Without terrain input
0.87
0.53
0.16
0.55
0.19
0.32
0.43
Terrain tokens switched off
0.90
0.81
0.25
0.67
0.42
0.37
0.55
With terrain tokens
0.97
0.99
0.91
1.00
0.86
0.76
0.90
88 walking and jogging clips that cross the terrain, each on six terrain families and ten difficulty levels, simulation only. All rows are means of four evaluation passes (the spread of the last column is about 0.02–0.05). “Switched off” keeps the fine-tuned weights and masks the tokens, so the gain comes from the scan. Terrain training costs some tracking accuracy (local error 28 vs. 25–26 mm) and reduces how much extra steering helps on top. We have not run this model on the real robot.
Method
Plan, then execute
A deterministic tracker maps a reference to an action, so the only handle at test time is the reference itself. DiffWBC instead models the future. The plan is regenerated from the robot's real state at every tick, so a cost applied to the plan acts in closed loop by construction.
Plain tracking needs one generator call (3.8 ms). Default steering uses 2 calls × 2 descent steps (20.3 ms per tick). Harder reaching targets use 12 steps (55.1 ms).
Plain tracking
A competitive general tracker from one generator call
All methods run in one common harness: MuJoCo sim-to-sim, official checkpoints, 150 windows per set. Errors are on successful clips. Best in bold, second underlined.
AMASS held-out subjects
BONES unseen actors
Method
Succ. (%) ↑
El (mm) ↓
Eg med. (mm) ↓
Succ. (%) ↑
El (mm) ↓
Eg med. (mm) ↓
Steerable
SONIC (release)
98.7
31.9
99.2
96.7
24.3
50.4
no
SONIC (v1.1)
98.0
30.5
126.2
96.7
23.7
61.5
no
HoloMotion
95.3
43.5
119.1
96.7
35.2
89.4
no
HEFT (pmg)
92.7
29.4
127.0
96.7
21.5
51.6
no
Humanoid-GPT
95.3
44.5
254.3
96.0
30.9
88.9
no
MimicLite Huge (teacher)
94.0
29.9
138.6
97.3
24.4
62.3
no
DiffWBC (distilled)
95.3
31.4
135.5
96.7
24.7
62.2
yes
DiffWBC (RL from scratch)
96.7
26.5
107.6
97.3
20.1
45.5
yes
The distilled model is on par with its RL teacher. The model trained from scratch with RL, with no teacher, has the lowest local error El on both sets.
Steering vs. editing the reference
Steering inside the generator beats editing the reference
The RL teacher tracks a reference edited with the same cost, either offline or at every tick (closed loop). DiffWBC is steered inside the generator. Time is controller time per tick.
Method
Bars loose ↑
Ceiling 1.30 m ↑
Ceiling 1.25 m ↑
Ceiling 1.20 m ↑
Reach hit ↑
Time (ms) ↓
Teacher, original reference
0.48
0.36
0.02
0.00
0.01
2.5
Teacher, offline edit
0.58
0.72
0.06
0.00
0.38
2.5
Teacher, closed-loop edit
0.57
0.87
0.10
0.02
0.60
330.5
DiffWBC, no steering
0.40
0.23
0.02
0.00
0.02
3.8
DiffWBC, steered
0.68
0.95
0.84
0.52
0.75
20.3–55.1
The edited reference asks for crouches the deterministic tracker cannot execute, so editing even at every tick stays at 0.10 under a 1.25 m ceiling. Steering the generator reaches 0.84 and runs 6–16× faster than the closed-loop edit.
New · Self-assessment without a simulator
The tracker knows which motions it cannot track
Give the generator the same reference and proprioception with K = 8 different noise samples. If the model has learned a motion well, the eight plans agree. If it has not, they spread out. We use the 95th percentile, over time, of the per-joint standard deviation of the planned joint positions as the score. It needs only the reference and the model: no simulator rollout and no extra network. Scoring 100 clips takes about 7 seconds.
Evaluation set: 7,884 library clips, including those the base policy finds hardest. Labels: whether the RL-from-scratch tracker (60k iterations) falls in mjlab. 537 clips fail (6.8%).
91%
of the top-2% most uncertain clips really fail (13× the 6.8% base rate)
74%
real failures among the top 329 clips, vs. 50% for plan deviation and 34% for reference atypicality
50%
of the tracker's real failures (250 of 501) fixed by fine-tuning on the 329 clips it is least sure about; reshuffling the seed alone flips 16%
11–56×
cheaper than selecting by simulated rollouts (1 or 5 per clip): the 78.8k-clip library is scored in ~12 min on 8 workers, no simulator
Failure rate in the most uncertain clips
Clips ranked by score, in bins of 2% of the set. Dashed line: base rate 6.8%.
AUC for real failures
All scores are simulator-free. Orange: needs only the diffusion generator. Grey: needs a critic, a world model or the library.
Precision and recall at the top of the ranking
Top
Noise spread
Plan deviation
Critic
Spread + critic
1%
0.91 / 0.13
0.87 / 0.13
1.00 / 0.15
1.00 / 0.15
2%
0.91 / 0.27
0.68 / 0.20
0.85 / 0.25
0.97 / 0.28
3%
0.79 / 0.35
0.58 / 0.25
0.74 / 0.33
0.86 / 0.38
5%
0.68 / 0.50
0.44 / 0.33
0.57 / 0.42
0.67 / 0.49
10%
0.41 / 0.61
0.35 / 0.51
0.37 / 0.54
0.41 / 0.60
20%
0.24 / 0.69
0.22 / 0.64
0.23 / 0.67
0.24 / 0.71
Precision / recall. Imagined world-model rollouts reach 0.93 at the top 2% and 0.71 / 0.52 at 5%, close to the noise spread. A threshold picked on half of the chunks (top ~6%) gives precision 0.61 and recall 0.49 on the other half.
Uncertainty and rarity are complementary
Failure rate when each score is split at its median. Rarity is the kNN distance of a clip's reference statistics to the rest of the library.
common motion
rare motion
low spread
0.4%
2,556 clips
6.1%
1,386 clips
high spread
3.5%
1,386 clips
15.5%
2,556 clips
Combined AUC 0.87. Rarity alone ranks broadly (0.81), but the spread is twice as precise at the very top.
From diagnosis to improvement
Fine-tune the RL-from-scratch tracker for 8k iterations on the 329 clips a score ranks hardest, then re-run 7,788 library clips in simulation and compare clip by clip with the base model (501 failures). To keep what it already masters, half of the environments replay the full library with a consolidation loss toward the frozen base and keep their RL return.
Clips selected by
Failures fixed ↑
… not in the training list ↑
Untouched clips broken ↓
Library failure rate ↓
None (base model, new simulation seed)
16%
18%
1.0%
6.4% → 6.4%
Plan deviation, consolidation loss only
38%
26%
4.6%
6.4% → 8.2%
Plan deviation
42%
32%
1.1%
6.4% → 4.7%
Imagined world-model rollout
43%
37%
1.4%
6.4% → 5.0%
Noise spread (ours)
250 / 501 (50%)
44%
1.9%
6.4% → 5.0%
Without the RL term on mastered clips the model breaks 4.6% of clips it used to track and the library failure rate goes up. With it, regressions stay at 1–2%, close to the 1.0% that a new simulation seed alone produces. Imagined rollouts detect failures best (AUC 0.908) but their list trains no better than plan deviation: a higher AUC does not mean better training data. Single seed; 50% vs 43% is about 33 clips.
Held-out failures and dense tracking
Success on held-out failures: same are unseen clips of trained families, fam are 41 unseen families. hard40 checks that dense tracking is kept.
Clips selected by
same ↑
fam ↑
hard40 succ. / El
No fine-tuning
0.11
0.26
0.95–0.975 / 35 mm
Random clips
0.07
0.24
–
Plan deviation
0.45
0.39
0.95 / 34.4 mm
Imagined world-model rollout
0.40
0.42
0.95 / 34.7 mm
Noise spread (ours)
0.42
0.48
0.975 / 35.7 mm
Noise spread, top 1,500 of the library, 60% consolidation (run in progress, 8k of 24k its)
0.39
0.50
0.975 / 32.3 mm
Single seed. Each fam clip is worth 0.009, so a 0.09 gap is about 10 clips. The 1,500-clip run tracks ordinary motions more precisely than the base model (32.3 vs 35 mm).
Selection cost
Finding the clips a tracker cannot track, extrapolated to the 78,842-clip library from timings on the same machine (RTX 4090, 32 cores) and checkpoint. Simulator numbers are for our mjlab stack, not for other papers' simulators.
Method
Worker-hours
Wall, 8 workers
Needs a simulator
Noise spread (ours), 0.07 s per clip
~1.5
~12 min
no
Simulated rollout, 1 per clip
17–40
2.1–5.3 h
yes
Simulated rollouts, 5 per clip (as in Extreme-RGMT)
84–200
10.5–26 h
yes
Lower bound: 0.77 s per rollout with one long process; upper bound: our actual scans. The base tracker fails on only ~1.2% of the library, so rollouts spend ~83 clips per failure found. Cost only: a list selected by rollouts was not trained with this recipe.
Citation
BibTeX
@misc{diffwbc2027,
title = {DiffWBC: Unlocking Steerable General Humanoid Motion Tracking
via One-Step Diffusion Whole-Body Control},
author = {Anonymous},
year = {2027}
}
