the short version
- On Push-T with expert subgoals every 25 steps, replacing LeWM's learned predictor with the simulator's exact physics changed long-horizon success from 78% to 77%, so the predictor was not the bottleneck.
- Chaining 25-step searches through subgoals took LeWM from 22% to 80% on 75-step tasks. Exact physics planning all 75 steps at once reached only 42%.
- Scoring only the block instead of the whole goal image raised an exact-physics planner from 9 of 50 to 39 of 50 tasks, which makes goal specification a first-order design choice.
- The 600 experiments did not test masking, predictor size, collapse prevention or data scale, because none of them retrained the model.
Across 600 experiments on LeWM, a recent model in the family of JEPA world models, none of the gains came from the training choices engineers usually tune. With expert subgoals every 25 steps, swapping the learned predictor for the simulator's perfect physics moved long-horizon Push-T success from 78% to 77%. Planning changes alone took success from 22% to 85%, and a single change to how the goal is scored added 60 points. The work did not test masking, predictor size, collapse prevention or data scale at all, because it never retrained the model.
Haddy Alchaer ran the experiments and wrote them up on November 13, 2026. LeWM was released this year by a team including Yann LeCun. Its main benchmark is Push-T, a simulation in which a robot pusher moves a gray T-shaped block onto a green target shown in a goal image. The architecture has three parts: an encoder that turns each frame into an embedding, a predictor that maps an embedding plus an action to the next embedding, and a planner that tries hundreds of action sequences inside the predictor and keeps the one whose end state lands closest to the goal embedding.
For an engineer, the useful framing is that this is a study of how to use a trained joint-embedding predictive architecture, not how to train one. If you came looking for masking ratios or regularizers against representation collapse, the source has nothing on them. What it does have is a set of inference-time changes that apply to LeWM as published. The author says they should carry over to larger JEPA world models that plan the same way.
A perfect predictor added 20 points, subgoals added 58
Used exactly as published, LeWM succeeds on 22% of tasks with the goal 75 steps away. It reaches 25-step goals 89.7% of the time, within a point of the exact-physics version at 90.5%. On the same 250 long-horizon tasks, exact physics planning all 75 steps at once reached 42%, while LeWM given a new subgoal every 25 steps reached 80%. The compounding-error explanation would predict that a better model fixes long horizons. These numbers instead point to LeWM's planning horizon of about 25 steps as the constraint, with chained segments of that length as the fix.
Extra search is not a substitute. Running the planner four times and keeping the lowest-cost plan took four times the compute and added less than one point, because the four runs almost always succeeded or failed together. One commenter on Hacker News objected that hierarchical planning, subgoals and action chunking are well understood, so the post should not be framed as discovery. The source calls its methods new, but it also describes hierarchical planning as widely considered one of the main open problems for world models. On this benchmark it is worth about three times what perfect physics is worth.
A memory graph needs a cost on every jump
A deployed robot will not have expert subgoals, so the second method builds them from memory. LeWM's encoder embedded every frame of its training demonstrations, about 460,000 states. Each state was linked to the states that follow it in its own demonstration and to nearby states in other demonstrations. The planner searches this graph for a path to the goal and uses the states along it as subgoals. This relies on embedding distance meaning something physical: given one subgoal reachable within 25 steps and one that is not, LeWM's distance ranked the reachable one closer 87% of the time.
The first version treated a jump between demonstrations as costing the same as a real step, and 318 of 1,015 plans tried to use six or more jumps the block could never follow. Adding a cost on every jump raised success on the 250 long-horizon tasks from 62% to 85%, which the source says matches the planner given expert subgoals. One commenter asked whether probabilistic roadmaps would solve this trivially. Another pointed to an MCTS implementation that solves Push-T without training. The source compares against neither, so it gives no evidence on whether the latent graph beats classical planners on cost or success.
The standard Push-T test overstates LeWM results
Every Push-T demonstration ends with the block in about the same place, and the standard test sets goals a fixed number of steps further along the same demonstration. For 84% of those goals, the memory graph already held a nearly identical state from another demonstration. The sample is also small. On the official 50 episodes, LeWM succeeded 68% of the time with the goal 50 steps away, but only 44% on 250 fresh episodes drawn the same way. Changing only the planner's random seed flipped 38 of 250 episodes between success and failure.
The harder test takes the start from one demonstration and the goal from another, and counts success only if the block stays on target for the last 10 steps. Under these rules, the memory-graph planner scores 75% on standard tasks and 45% on the new ones. Given seed variance of that size, a 50-episode Push-T result is weak evidence on its own, and larger draws are needed before comparing methods.
Coverage gaps drop success from 93% to 32%
A memory graph is only as good as its coverage. Deleting every stored state near the goal cut success from 93% to 32%. The physics of pushing a block does not change when the whole table is shifted, rotated or mirrored, so the planner can transform demonstration segments from elsewhere to fill the gap. That brought success back to 85%, and a graph built from 1% of the demonstrations plus transformed copies did about as well as one built from 10%.
- Harder test overall: 45% with plain memory, 56% with transformed segments.
- Goal with the flow of the demonstrations: 69% with plain memory.
- Goal across the flow: 39% plain, 54% transformed.
- Goal against the flow: 27% plain, 46% transformed.
- Each direction row covers about 83 tasks.
One commenter on Hacker News read the deletion result as damning: planning breaks when memory near the goal is missing, and recovery depends on manually transforming the memories it has. The source does not dispute this. It calls generalizing to tasks that don't have perfect demonstrations the biggest open problem remaining, and it tests the symmetry trick only on Push-T. Another commenter asked why evaluation goals come from demonstrations at all rather than arbitrary poses. The source answers only by building the cross-demonstration test.
Scoring the pusher cut success from 39/50 to 9/50
A goal image has to show the pusher somewhere, so a planner matching the whole image also tries to put the pusher where the expert's happened to be. To isolate this, the exact-physics planner ran twice on the same 50 harder-test tasks with two costs: scoring block and pusher solved 9, and scoring the block only solved 39. Thirty tasks went from failure to success and none went the other way, a split the author says happens by chance about twice in a billion. On the 30 flipped tasks, the planner got the pusher within half a pixel of where the goal image put it and lost the block on the way.
a goal should just be a set of end states, not a copy of an image
LeWM cannot simply be told to ignore the pusher, because its embedding of a goal image holds block and pusher together with no part labeled as the pusher. The workaround compares each prediction against 16 versions of the goal and keeps the closest match. Each version has the block in the same place and the pusher moved to a position taken from stored demonstrations. That doubled LeWM's success on the harder test, from 11% to 23%, without retraining. The source does not report the compute cost of scoring 16 goals per prediction.
What to copy, and what the 600 runs leave untested
- Measure your predictor against ground truth before improving it. Here, with expert subgoals, it matched exact physics at 78% versus 77%.
- Find the model's reliable horizon, about 25 steps for LeWM, and plan in chained segments of that length.
- If you plan through a graph of stored embeddings, put a cost on jumps between demonstrations.
- Specify goals as the parts of the state the task requires, not a full goal image.
- Evaluate on goals taken from a different demonstration than the start, and on samples larger than 50 episodes.
The numbers also show what to avoid. Repeated search added under one point for four times the compute, and the standard test split overstates success. The sources say nothing about the training side. There are no results on masking strategy, predictor size, collapse prevention or data scale, no comparison with roadmap or MCTS baselines, and no evidence on whether these findings hold outside Push-T. One commenter raised that last question, asking whether JEPA techniques apply beyond robotic motion.
Questions this raises
Does a better predictor fix long-horizon planning in JEPA world models?
Not on Push-T. Exact physics planning all 75 steps at once reached 42%, while LeWM with a subgoal every 25 steps reached 80%. With expert subgoals, swapping in perfect physics moved success from 78% to 77%.
How does the memory graph planner find subgoals without expert demonstrations?
It embeds about 460,000 states from LeWM's training demonstrations, links each to its successors and to nearby states in other demonstrations, and searches the graph for a path to the goal. Adding a cost to every jump between demonstrations raised success from 62% to 85% on 250 long-horizon tasks.
Is the standard Push-T benchmark reliable for comparing world models?
The study suggests 50-episode results are weak evidence. LeWM scored 68% on the official 50 episodes but 44% on 250 fresh ones drawn the same way, and changing only the planner seed flipped 38 of 250 episodes.
Spotted an error? Email samir.s.sengupta@gmail.com and I will correct it and date the change at the top of the note. How these notes are researched, drafted and checked is in the editorial policy.
