Open-Weight Models

LLM self-rewriting weights lift a 27B model 6.6 points, but the abstract gives no compute cost

An open 27B model edits its own weights with no gradients and gains 6.6 points on unseen problems. Engineers should weigh the missing cost and baseline numbers before building on it.

Published
November 21, 2026
Read
6 min
Published by
Samir Sengupta
Open 27B language model editing its own weights, with a 6.6-point accuracy gain and missing compute cost figures
Note 154 / 155Daily note · Written from 1 source

How this was madeDrafted with Claude, Anthropic’s model, from 1 cited source, then checked against it in a second editing pass. Editorial policy

the short version

  • In arXiv 2611.21885, an open 27B model edits its own weights using only forward passes and becomes 6.6 points more accurate on problems it never saw.
  • The abstract reports no wall-clock time, GPU count, memory footprint or number of candidate edits per session. The cost of a self-rewriting session cannot yet be compared with a fine-tuning run.
  • The abstract reports no head-to-head result against fine-tuning or in-context baselines. The case for building self-updating models rests on the paper body or on replication.
  • In one model family, only the largest model improves itself. Smaller models make no edit or break themselves, so edit validation is a hard requirement.

In arXiv paper 2611.21885, an open 27B language model is given its own weights and graded answers. It diagnoses why it failed and edits the weights responsible, with no gradients and no training run. This is what the paper means by LLM self-rewriting weights: the model acts as its own optimizer. The edit makes it 6.6 points more accurate on problems it never saw, and over eight consecutive sessions its accuracy only rises.

The paper, "Language Models Can Rewrite Their Own Weights", is by Jiajun Fan, Ge Liu and nine co-authors. It was submitted on 16 November 2026 and revised to v2 on 18 November. The authors call it, to their knowledge, the first time a language model has rewritten its own weights by its own reasoning and become better. They frame it as a third route to a better model, alongside retraining and inference-time aids such as prompts, harnesses or extra computation. Unlike those aids, the gain stays in the weights after the task ends.

For engineers, two questions decide whether this matters: what a rewriting session costs, and whether the gain beats a plain fine-tune or a better prompt. The abstract makes a qualitative claim about cost per answer but gives no compute figures. It reports no comparison with either baseline.

How the 27B model rewrites its own weights

The setup gives the 27B model three things: its own weights, its graded answers, and the ability to run itself. It reasons about where it is failing and rewrites the weights behind that failure. It then tests candidate edits on practice problems and submits only edits that work.

The authors say each edit improves on the last, and that over eight consecutive sessions accuracy only rises. Earlier work, they note, used a frontier model to edit a much smaller one. Here the model doing the editing and the model being edited are the same.

One commenter on Hacker News asked where the improvement comes from if no new training data is involved. They suggested it derives from the test problems the model runs itself on. The abstract points to two sources of signal: the graded answers the model receives, and the practice problems it uses to accept or reject edits. Another commenter offered a different reading: the model is unlocking capability that pretraining already put in the weights. The abstract does not test either explanation.

What LLM self-rewriting costs in compute and memory

The authors make one cost claim. Self-rewriting "runs only forward passes, as serving does," and, unlike test-time scaling, adds no cost per answer once the edit is written. With no backward pass, a session should not need gradient or optimizer state. That would put its memory needs closer to inference than to fine-tuning. This is an inference from the method, and the abstract gives no figure to confirm it.


Unlike test-time scaling, self-rewriting writes its gain into the weights, at no cost per answer; unlike training, it runs only forward passes, as serving does.
Fan et al., arXiv 2611.21885 abstract

The abstract does not report the numbers a team would need to budget a session. The missing items are:

  • Wall-clock time per self-rewriting session, or for the eight-session run.
  • Hardware: GPU type, count, or peak memory during a session.
  • How many forward passes go into reasoning, proposing edits and testing them on practice problems before one is submitted.
  • How the weights are presented to the model, and how much of them it inspects.
  • Which 27B model was used and under what licence. The abstract says only that it is open.

"Only forward passes" is not the same as "cheap". A session that tries many candidate edits, each checked against a practice set, could use more total compute than a short fine-tune. The cost per answer after the edit is zero. The cost per edit is not stated.

Whether 6.6 points beats fine-tuning or in-context baselines

The headline result is a 6.6-point accuracy gain on unseen problems, measured against the same model before it edited itself. The abstract does not compare this gain with a gradient fine-tune on the same graded answers. It also does not compare it with a prompt or harness built from the same failure analysis. Those comparisons decide whether self-rewriting is worth building infrastructure for. Without them, 6.6 points shows that the method works, not that it beats the cheaper options.

One Hacker News commenter raised a more specific doubt. They said the only modification the model made was to shorten its reasoning traces so they stopped hitting the token limit. They likened this to a separate weight-space programming write-up, by different authors, which they said found that even Astra could not improve a model without that kind of obvious flaw. The abstract does not describe the content of the edits, so this claim is unverified here. If it is right, a decoding-time change to the length budget would be the natural baseline to compare against.

Other commenters pointed to prior work. One cited Google's "sleep" framework, in which a model updates its weights during an offline phase using self-generated practice examples and pre-allocated Mixture-of-Experts memory blocks. Another replied that this paper describes more advanced weight adjustment. The abstract does not compare itself with either approach.

Another commenter called the method overfitting through self-conversation, with a risk of sycophancy. A reply countered that edits are reasoned toward a designed target, not fitted to a metric by gradient descent. The abstract's design partly addresses the concern: edits are accepted only after passing practice problems, and the 6.6-point gain is reported on problems the model never saw. Whether the practice set and the test set differ in distribution is not stated.

Repairing a damaged copy to 80% of original

The abstract's third result is a repair experiment. The authors give the model a damaged copy of itself that solves almost nothing. With no retraining, and without ever reading the original weights, the model brings the copy back to over 80% of the original accuracy on image tasks it never saw. The abstract does not describe how the copy was damaged. It also does not compare the result with reloading a checkpoint, which is what a production team would do.

Only the largest model in one family self-improves

The authors report that the ability emerges with scale. In one model family, the smaller models either make no edit or break themselves, and only the largest improves and repairs itself. The abstract does not name the family or give the size threshold. For anyone weighing self-modifying weights in a real system, this is the main operational risk: a model below the threshold may make no useful edit or break itself. The practice-problem check is the only safeguard the abstract describes.

One commenter on Hacker News noted that the paper frames self-rewriting as a step toward recursive self-improvement without mentioning alignment. Another suggested that a well-aligned closed model with safeguards on how its weights are updated might be enough. They added that an abliterated open model doing this could be very bad. The abstract does not address who approves an edit, how edits are logged, or how to roll one back. A deployment would need all three, plus a held-out evaluation set the model cannot see.

Do the results justify building self-updating models?

On the abstract alone, not yet. The paper shows three things. An open 27B model can produce weight edits that improve accuracy on unseen problems. Its accuracy rises over eight sessions. It can restore a broken copy to over 80% of original accuracy on unseen image tasks.

The abstract does not show session cost, a comparison with fine-tuning or prompting, or what the edits actually change. The next useful evidence is the full paper's per-session compute figures and a same-data fine-tuning baseline. If the reasoning-length explanation from Hacker News proves correct, the generality of the 6.6-point gain will need re-checking.

Questions this raises

How does an LLM rewrite its own weights without gradients?

The 27B model is given its own weights, its graded answers and the ability to run itself. It reasons about why it failed, rewrites the weights behind that failure, tests candidate edits on practice problems, and submits only edits that work. The process uses only forward passes, as serving does.

How much compute does a self-rewriting session cost?

The abstract does not say. It claims no added cost per answer once an edit is written, but gives no wall-clock time, hardware, peak memory or count of forward passes per edit. A session that tests many candidate edits could use more total compute than a short fine-tune.

Does the 6.6-point gain beat fine-tuning or a better prompt?

Unknown. The gain is measured only against the same model before it edited itself. The abstract reports no comparison with a gradient fine-tune on the same graded answers or with a prompt or harness built from the same failure analysis.

Spotted an error? Email samir.s.sengupta@gmail.com and I will correct it and date the change at the top of the note. How these notes are researched, drafted and checked is in the editorial policy.

More notes

Building something on this?

I ship production LLM, RAG and agentic systems for a living - the infrastructure behind the things these notes are about. Open to roles, contract work and research collaboration.