Migrating large system prompts to Ollama: 35KB eats 14% of context

The hardware is a 128 gig AMD Ryzen AI MAX+ 395, with 32 gig allocated to the host OS and everything else allocated to inference. The author is trying to determine whether he can rely on abliterated open weight 27b parameter models, with the stated goal of avoiding cybersecurity refusals and keeping session transcripts off provider hardware. He is explicit that the smaller model size did not produce the problem: self-hosted systems have smaller context windows, and the prompt plus session history exceeds the 65k maximum.

That distinction decides what you change. If the failure were model capability, the answer would be a bigger local model. Because the failure is context accounting, the answer is a different prompt architecture, and most of the work happens before the model is running. Note also what these notes do not cover: there is no prompt-caching measurement, no KV-cache sizing, and no chat-template or tool-call format comparison anywhere in them.

The notes give two reasons for moving off frontier APIs. The first is refusals: the author argues that safety filters block defenders from proving exploitability, and that defenders need models that will do security testing, because a vulnerability finding is useless if you cannot demonstrate it. The second is session privacy - the argument that agent sessions are transcripts of the hardest problems you work on, and that the metadata about those sessions may be worth more than the data in them. The notes cite a Verge report on a dispute over frontier providers and user activity as the trigger for writing them up quickly.

On the frontier APIs, a fat system prompt buys room. The notes argue that the largest asset frontier providers supply is not the model but the context window: large windows give the model space for chain of thought, which lets it infer what a poorly constructed prompt was intended to produce. Anthropic and OpenAI provide only summaries of chain of thought to the user, so the repair work is not directly readable. With a fat window and chain of thought, bad prompts still produce good results, and you do not learn that they are bad.

At 65k tokens that cushion is gone. The notes report that context gets saturated within a few circles, and sometimes before a response comes back at all. The observed behaviour is second-guessing the prompt with unnecessary tool calls and double reads of files the agent has already read.

Two configuration items come before any prompt work. Context length has to be tuned explicitly in Ollama; the notes state the context defaults in Ollama are extremely small. If you are moving onto opencode, agent definition becomes declarative: agents are stored in ~/.config/opencode/agents rather than assembled by a shell script or a direct Claude Code invocation.

The notes add that opencode ships its own system prompt, which has to be overcome through declarative agents, and that opencode's permissions are a separate thing to familiarise yourself with. People already building with the Anthropic SDKs will find the declarative step familiar. Anyone whose agent was a Claude Code file read plus a prompt will be writing something new.

The prescribed rewrite is to split preprompts into single problem/resolution units, one objective each - what the notes call SOP, Single Objective Prompting. Alongside that: log session state to disk to facilitate more frequent session handoffs, and build agents that re-read only the slice they need. Work to reduce the number of tool calls per agentic step. Replace "don't do X" with positive directives, so the instruction becomes "only do Y".

The notes flag tool-call parse failures as a specific hazard: parsing tool call responses shoves so much raw data into context that it destroys sessions. There is no measurement of how much context a failed parse costs, and no comparison of tool-call formats across models, so the mitigation on offer is structural - fewer calls per step - rather than tuned to a format.

The notes list concrete signals of context exhaustion to measure over time and monitor for in logs. These are cheap to instrument and are the closest thing the piece offers to an operational threshold for a local agent.

Commenters on Hacker News largely accepted the diagnosis and disputed the framing. The top-ranked comment compressed the piece to one line - 35kB prompts that worked fine against a hosted 1 million token window crash out at a 65K window locally - and said the commenter had hoped for more substance. The author replied in the thread that he had spent the whole weekend on the problems and the post, and that he would later share details on using automation to accelerate splitting prompts for local inference.

A sharper objection was that the prompt itself is the bug. One commenter argued that, given how most people and companies run their inference engine, you run out of useful context the model can accurately attend to around the 250k mark regardless of advertised context size, frontier and Chinese models included. That commenter's fix is to have the LLM help shape the overall plan, then run each step in a separate session without the context of previous successful steps.

That claim did not go unchallenged in the thread. One reply noted the Claude Code system prompt was over 50KB, though the commenter thought it had been trimmed down heavily recently because newer models need less hand-holding. Another commenter reported a long-running autonomous Astra session reaching about 600k tokens and finishing fine, and Opus 5 running to around 700k with good performance, and asked how anyone keeps context that low on complex tasks. None of this is resolved in the material: the 250k figure is one practitioner's estimate, not a measurement.

The other recurring pushback was hardware. One commenter said the main gotcha for local models is insane hardware requirements, and that even for $10K you get mediocre performance. Another said it feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage. The notes offer no counter-benchmark: they report one machine, one window size, and one qualitative failure mode.

Several things an engineer would want before committing to a port are absent from both the notes and the thread. Prompt caching is never mentioned, so there is no figure for what re-processing a 35KB prefix costs per turn locally versus on the frontier API. There is no KV-cache sizing and no tokens-per-second figure for the 27b model on the Ryzen AI MAX+ 395, and no discussion of chat template or system-role handling differences between a frontier API and whatever template the local build ships with. Tool-call formatting appears only as a failure symptom.

Treat those as the unlogged risks in the port. The one number that is reported - 14% of a 65k window consumed at rest - is the floor, not the total cost of the migration. Test the chat template, the system-role handling and the tool-call format on the specific abliterated build you plan to run before assuming the prompt is the only thing that needs rewriting.

What to watch: the author has said follow-up posts will cover automation for splitting preprompts, which is the part of this workflow that does not scale by hand. Until then the reproducible parts of the note are the arithmetic, 35KB against 65k tokens leaving 86% before the first turn, the five failure signals that say the window is full, and the two configuration changes - explicit context length in Ollama, declarative agents in ~/.config/opencode/agents - that have to happen before any of the prompt work is testable.