WorldTUI fine-tunes a 360M model to imitate a terminal and gets 80/106 commands right

The project is described in a post on simedw.com published October 22, 2026. The author says the idea came from seeing Genie 3, where a model predicts the next frame from user inputs instead of a game engine rendering it. Without a large GPU cluster or video data, the author scaled down to a terminal for three reasons: it is a grid of characters rather than pixels, real shells can produce unlimited perfectly labelled data, and a pretrained language model already has a good idea of what ls or cat should print. The question the author set out to answer was whether the model "could maintain a consistent world, or just generate convincing terminal output."

The browser demo is 208 MB and runs best in Chrome or Edge on a desktop with WebGPU and a physical keyboard. The author says it should work on a modern phone, but slowly. Typing pythoon in the demo shows that no real shell is involved.

Appending tokens to the end of a stream fits a terminal badly. A single keypress such as Escape in vim does something on its own, and clear wipes the screen and sends the cursor to the top. The model therefore predicts which rows to rewrite and where the cursor ends up. Each session opens with a snapshot of the screen size, its contents and the cursor position. Each keypress then arrives as the raw bytes a real terminal would send: Enter is \r, Escape is 0x1b and the left arrow is ESC [ D.

In the output above, the model writes l on row 0 at column 17 and moves the cursor to row 0, column 18. The trailing 1 marks the cursor as visible, because full-screen apps sometimes hide it. Nothing is cleared first, so new characters overwrite old ones. Screen size is part of the input rather than fixed. The snapshot also lets a session start partway through a bash prompt, a vim buffer or a game of snake, though the author says the model does better from bash.

Rewriting every row after a scroll would cost around 3,000 tokens for one keypress. A COPY operation instead moves a row from the previous screen, so a scroll costs three tokens per row instead of about 96. The same operation covers clearing the screen, inserting and deleting lines in vim, and navigating man pages. When no key is pressed, the model receives a WAIT token with a duration, which is how snake keeps moving.

The vocabulary is 333 tokens: 256 byte tokens reused from Qwen, 64 row tokens and a handful of special operations. A coordinate such as 17 therefore takes one token. Decoding is grammar-constrained, so every output is a valid screen update. A valid update is not necessarily a correct one.

The author looked at existing terminal recordings, including asciinema, but found none with precise keystroke-to-screen alignment. Instead, the author generated sessions by running real commands in Docker containers and recording every keypress and the screen after each one. The dataset grew as the model improved and ended at 336k episodes, roughly 2B tokens. For comparison, the base model saw around 18T tokens during pretraining.

Training started with supervised fine-tuning: given a screen history and a keypress, predict the delta the real terminal produced. The author chose Qwen2.5-Coder-0.5B because a small model was needed for near-realtime performance and a coding model had already seen plenty of CLI usage. Ordinary typing was learned quickly. In longer sessions, errors compounded, and a mistake in a filename carried through the rest of the session.

The fix borrowed from DAgger, where a model learns from the states it actually visits. A point was picked in a recorded session, usually close to the command that produces output, and the model predicted the next 1 to 4 events. Its screen was compared with the recorded one. If the difference was small enough, a correction delta was built to repair the model's screen on the next keypress, and training continued on the corrected frame and a few subsequent real events.

One rule kept this from rewarding hallucination. A correction could only introduce words that had appeared on an earlier real screen or in the user's typing, not words that existed only in the model's own predictions. About a third of sampled sessions produced useful corrections. The author ran three rounds, each collecting mistakes from the latest model. Together with additional training data, this moved PresetBench (47 scripted cases drawn from demo failures) from 24/47 to 38/47 at int4.

ShellBench scores exact output on 106 commands across 24 handwritten bash sessions. The author says neither ShellBench nor PresetBench appears in the training data. WorldTUI scored 80, the prompted Qwen2.5-Coder-0.5B scored 52 and the prompted Qwen3-0.6B scored 60. The largest gap was in file readbacks: 9/13 for WorldTUI against 2/13 and 3/13. A separate memory test uses 32 worlds, each writing two random values to two files and reading them back in separate sessions, for 64 readbacks. Its names and values never appear in training. The available text of the post does not explain how the 13-readback column relates to that test.

The author says the model is still far from a reliable filesystem. Familiar commands work fairly well, but combining them in new ways can quickly break things. For example, after echo green >> colors, running cat colors often does not show the appended line.

Distillation showed how fragile that memory is. The teacher was an earlier checkpoint scoring 78/106 on ShellBench. Removing layer 7 without retraining moved next-token accuracy only from 99.3% to 98.9%, but exact memory readbacks fell from 64/64 to 0/64, with the true history fed in. A first student that kept even-numbered layers plus layer 23 got 31/64 readbacks, picking the wrong of its two written values about half the time. It had not recovered after 480M training tokens.

A 16-layer student that kept layers 1, 7, 15 and 22 passed all 32 memory worlds with 64/64 readbacks, including at 4-bit and about 138MB. Its ShellBench score fell to 65, against the teacher's 78. The author gave up on distillation.

Decoding one token at a time gave about 38 ms per key while typing. Full redraws such as vim test.txt, man and clear took up to 12 s on an RTX 4090. Most rows in a frame do not depend on each other, so the author gave each row its own batch element. Every element shares the same history, including the KV cache, and generates only its own row. Decoding cost becomes the length of the longest row, roughly 100 steps, instead of up to 3,000. The cursor is a final step that sees all row updates. This needed a matching attention mask in training, and large-frame p95 fell from 6.7 s to 0.87 s.

Rows decoded independently cannot coordinate. Vim opened on row 3 instead of row 0, and several rows added snake food at once. The fix was speculative decoding with the same model as drafter and verifier. The parallel rows produce a draft, the model verifies it sequentially and keeps tokens up to the first disagreement, then generates its own correction and re-proposes the rest of the draft.

For the browser, the author dropped Qwen's 151,936-token embedding table, which holds 136M parameters, down to the 333 tokens the model uses. That took the model from 494M to about 360M parameters (990MB to 716MB), and int4 quantization brought it to 208 MB. A custom JavaScript Qwen decoder runs WGSL compute shaders through WebGPU, unpacking 4-bit weights during inference, with 32-bit float activations and attention cache. All inference runs locally in the tab. In the browser on an M4 Mac, one keystroke takes about 60 ms, and man test takes 0.9 s with parallel drafting and verification against 2.5 s sequentially.

The post does not test the model as an environment for training or evaluating agents, so any answer goes beyond its evidence. Its own numbers show the main risk. On ShellBench, 26 of 106 outputs were not exact, and 4 of 13 file readbacks failed. An agent trained against the model would learn from errors like these, and an agent evaluated against it would be scored on them. Grammar-constrained decoding guarantees a well-formed screen, not a true one. An agent would also have to send raw keypress bytes and read screen deltas, not call tools in the usual way.

Commenters on Hacker News raised related concerns. Discussing learned interfaces generally, one asked whether the model not being completely deterministic would be a problem, adding "You don't really want any sort of randomness in your terminal commands." Another saw value in learning terminal behaviour from bash traces that are expensive to obtain. A third suggested a hybrid: give a small model a tool-calling API for a filesystem and an embedded busybox, so its parameters only learn what busybox cannot handle.

The available text of the post does not say whether sampling is greedy or how often identical inputs give identical screens. It says errors compound in longer sessions but does not report how accuracy changes with session length. Teams weighing a learned sandbox would need those answers before trusting rewards computed against it.