# Continual-ARC benchmark scores agent memory and weight updates on one scale > Continual-ARC brings ARC-AGI tasks back with fresh instances over a lifelong stream and charges for feedback, so a memory-based agent harness and a test-time parametric learner can be ranked against each other. - Published: 2026-11-11 - By: Samir Sengupta (https://www.samcodeman.com/about) - Canonical: https://www.samcodeman.com/writing/continual-arc-benchmark-agent-memory-weight-updates - How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy ## The short version - Continual-ARC scores each instance by the share of a feedback budget the learner consumes. Demonstrations and retries on incorrect submissions are paid for. - A learner may keep anything between instances, so memory-based agent harnesses and weight-based test-time learners are scored on the same scale. - After the stream, a frozen learner is tested on fresh instances of seen tasks to measure memorization, and on unseen tasks to measure generalization. - The available v1 abstract is cut off before any results, so there is no reported number yet for how memory compares with weight updates. The Continual-ARC benchmark was posted to arXiv on 2026-11-10 as 2611.14505v1. It measures skill acquisition by turning the three ARC-AGI editions into a lifelong stream. Tasks return after dormant gaps with fresh instances, while new tasks keep arriving. Each instance is scored on the share of a feedback budget the learner consumes, and a learner may keep anything between instances. That puts memory-based agent harnesses and test-time parametric learning on the same scale. The available v1 abstract is cut off before it reports any results, so there is no published number yet for how the two approaches compare. The paper describes agent harnesses as a promising mechanism for continual learning, alongside test-time parametric learning. It names three limitations of current benchmarks. They generally focus on a single applied domain, they serve each instance only once, and they assume a single family of learner. The abstract ties the third limitation directly to the comparison problem. > memory-based and weight-based systems are rarely compared on the same scale > > - Continual-ARC abstract, arXiv 2611.14505v1 ## How the Continual-ARC benchmark builds its stream from ARC-AGI Continual-ARC draws its tasks from the three ARC-AGI editions, and the abstract says those tasks need no domain knowledge. That property contrasts with the first limitation the authors name: benchmarks that focus on a single applied domain. For anyone reading the eventual results, it suggests that gains on the stream should reflect what a learner picks up during the stream rather than prior expertise in a field. The abstract does not make that argument itself. The stream addresses the second limitation, serving each instance only once. The same tasks return after dormant gaps, each time with fresh instances, so a learner meets a task again but not the identical problem. New tasks keep arriving throughout, so retention and acquisition are tested in the same run. The abstract does not give the stream length, the number of tasks drawn from each ARC-AGI edition, or how long the dormant gaps are. ## How the feedback budget scores each instance On each instance the learner may do three things: pay for demonstrations, submit answers, or act inside the harness. Retries are permitted on incorrect submissions, and those retries are paid for. - Pay for demonstrations: demonstrations cost budget. The abstract does not say how many are available or at what price. - Submit answers: incorrect submissions may be retried, and retries are paid for. - Act inside the harness: listed as the third option. The abstract does not say whether these actions draw on the same budget. Each instance is scored based on the share of this feedback budget consumed. Read plainly, a learner that has acquired a skill should spend less of the budget when the task returns, because it needs fewer demonstrations and fewer paid retries. The abstract does not give the exact formula, the size of the budget, or how harness actions are priced relative to demonstrations and retries. Those details will decide whether a harness that takes many internal actions scores differently from one that asks for demonstrations early. ## Why memory and weight updates share one score The design choice behind the comparison is permissive: a learner may keep anything between instances. The abstract states that, because of this, memory-based and weight-based systems are scored on the same scale. Context retained by a harness and parameters updated in a model are both allowed. The score observes only how much of the feedback budget each spends. This addresses the third limitation the authors name, benchmarks that assume a single family of learner. The abstract does not say whether compute, wall-clock time or token spend enter the score. If they do not, a weight update and a memory lookup are judged only on how much paid feedback follows them, whatever each costs to run. Teams serving either approach would need to measure that cost separately. ## Two frozen tests separate memorization from generalization After the stream ends, the learner is frozen and tested on two sets. Fresh instances of the tasks it met during the stream measure memorization. Unseen tasks measure generalization. Freezing matters for the comparison. Neither a harness nor a weight-updating learner can keep learning during these tests, so both are judged on what they carried out of the stream. Splitting memorization from generalization also exposes a learner that does well on returning tasks but gains nothing on new ones. The abstract does not report results for either test. ## What the v1 abstract reports about results The abstract says the authors run an agent harness with frozen weights, with and without persist. The available text ends mid-word at that point, so the source does not state in full what the ablation toggles. It contains no scores, no named base models and no memorization or generalization figures. The truncated text also does not show whether a test-time weight-update learner was run. Any claim that memory beats weight updates on Continual-ARC, or the reverse, is not supported by this source. The visible text does establish one thing about the reported runs: the harness keeps its weights frozen. Differences between its two configurations would therefore come from what is kept outside the weights, not from parameter changes. ## What this changes for choosing memory or weight updates For engineers deciding whether an agent should learn through a memory layer or through test-time weight updates, Continual-ARC's immediate value is its protocol, not a leaderboard. It defines learning as spending less paid feedback when a task returns after a gap. It checks memorization and generalization separately with a frozen learner. Teams evaluating an internal agent can borrow that structure: bring tasks back after dormant periods with fresh instances, charge for demonstrations and retries, and freeze the system before a final pass on seen and unseen tasks. The abstract also shows the benchmark's limits. ARC-AGI tasks need no domain knowledge, so results will not directly predict performance in a single applied domain. The scoring formula and budget size are not stated. Until results from arXiv 2611.14505v1 are available, the benchmark is a measurement design for comparing the two approaches, not evidence for either one. ## Questions this raises ### Does Continual-ARC show whether memory or weight updates work better? Not yet. The available v1 abstract ends mid-sentence before reporting any scores, base models or memorization and generalization figures. Any claim that one approach beats the other is not supported by this source. ### How does Continual-ARC score a learner on each instance? Each instance is scored on the share of a feedback budget the learner consumes. The learner can pay for demonstrations, submit answers with paid retries, or act inside the harness. The abstract does not give the exact formula, the budget size or how harness actions are priced. ### Why can memory-based and weight-based learners be compared on Continual-ARC? A learner may keep anything between instances, so both retained context and updated parameters are allowed. The score only observes how much paid feedback each approach spends, putting both on the same scale. ## Key facts - Continual-ARC was posted to arXiv on 2026-11-10 as 2611.14505v1. - The stream draws tasks from all 3 ARC-AGI editions and returns them after dormant gaps with fresh instances. - Each instance is scored on the share of a feedback budget consumed by demonstrations, paid retries and harness actions. - After the stream, a frozen learner faces 2 tests: fresh instances of seen tasks for memorization and unseen tasks for generalization. - The v1 abstract is cut off before any scores, so there is no published memory vs weight-update comparison yet. ## Sources 1. [Continual-ARC: Measuring Skill Acquisition over a Working Lifetime](https://arxiv.org/abs/2611.14505v1) - arXiv