Ollama Cuts Long Prompts from the Front

When a prompt is longer than the model’s context window, Ollama doesn’t return an error. It keeps the first few tokens and the end of the prompt, drops everything in between, and the model answers whatever is left. An instruction at the top of a long prompt is the first thing to go. Cap what you send, put the instruction after the text, and set the context length you actually need.

What Ollama does with a prompt that doesn’t fit

In Ollama 0.32 a chat request that is too long goes through two cuts.

First, in server/prompt.go, Ollama drops whole messages from the start of the conversation until the rest fits, always keeping the system messages and the last message. That one is logged only at debug level.

Second, if the rendered prompt is still too long, for example because one message alone is bigger than the window, the runner cuts tokens (llm/llama_server.go). It keeps the first num_keep tokens (4 by default, plus one if the tokenizer adds a start token), throws away what follows, and keeps the end. It doesn’t fill the window back up: it frees about half of it so there is room to generate. With num_ctx at 4,096 the model sees 4 tokens from the start and roughly the last 2,046 tokens of your prompt. Four tokens is the opening of the chat template, a word of the system prompt at most, so the system prompt goes too.

This second cut logs a warning:

level=WARN source=llama_server.go:317 msg="truncating input prompt" limit=2050 prompt=10522 keep=4 new=2050

prompt is what you sent in tokens, new is what the model saw.

This is what happens with context shift on, the default for most models; with it off, you get a 400 instead. It’s the path GGUF models take through llama-server. Models on Ollama’s MLX engine on a Mac take a different one.

The default context is smaller than you think

Unless you set it, Ollama picks the context length from GPU memory: 4,096 tokens below 24 GiB, 32,768 from 24 to 48 GiB, and 262,144 from 48 GiB up, capped at what the model was trained for. Left on automatic, it can also step down to 32,768 or 4,096 when the model doesn’t fit in memory. The FAQ still says 4,096. To see what a loaded model actually got, look at the CONTEXT column:

ollama ps

The fix

1. Cap the input. I clip each message, then keep the start and the end of the transcript and mark the gap. The end gets more room because that’s usually where a conversation resolves.

MAX_MESSAGE_CHARS = 2000
MAX_TRANSCRIPT_CHARS = 24000


def clip(text, cap):
    return text if len(text) <= cap else text[:cap] + " …[truncated]"


def build_transcript(messages, per_msg=MAX_MESSAGE_CHARS, total=MAX_TRANSCRIPT_CHARS):
    lines = [f"{m['role']}: {clip(str(m.get('content') or ''), per_msg)}" for m in messages]
    if sum(len(line) + 1 for line in lines) <= total:
        return "\n".join(lines)

    # Keep the start (30%) and the end (70%), mark the gap.
    head_budget = int(total * 0.3)
    tail_budget = total - head_budget
    head, used, i = [], 0, 0
    while i < len(lines) and used + len(lines[i]) + 1 <= head_budget:
        head.append(lines[i])
        used += len(lines[i]) + 1
        i += 1
    tail, used, j = [], 0, len(lines)
    while j > i and used + len(lines[j - 1]) + 1 <= tail_budget:
        tail.insert(0, lines[j - 1])
        used += len(lines[j - 1]) + 1
        j -= 1
    gap = [f"[… {j - i} message(s) omitted …]"] if j > i else []
    return "\n".join(head + gap + tail)

The per-message clip keeps one pasted file or tool dump from eating the whole budget. Size the budget from your context. A token is roughly four characters of English and fewer for code and logs, so 24,000 characters is somewhere around 6,000 to 7,000 tokens. That fits a 32k window with room to spare, but not the 4,096 default.

2. Put the instruction after the text. If something does get cut, it’s the old part of the transcript, not the task.

INSTRUCTION = """Summarize the conversation above in 2-3 sentences and list main topics.

Return JSON only (no markdown, no explanation):
{"summary": "...", "topics": ["topic1", "topic2"]}"""

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "You summarize conversations. Return only valid JSON."},
        {"role": "user", "content": "Conversation:\n" + build_transcript(messages) + "\n\n" + INSTRUCTION},
    ],
    temperature=0,
    max_tokens=1000,
)

3. Set the context length. On the native /api/chat, send "options": {"num_ctx": 32768} with the request. The OpenAI-compatible /v1/chat/completions has no field for it, so set it on the server instead, with OLLAMA_CONTEXT_LENGTH=32768 (an Environment= line in the systemd unit on Linux, launchctl setenv on a Mac) or the context slider in the Ollama app. You can also bake it into a model:

FROM qwen3:8b
PARAMETER num_ctx 32768
ollama create qwen3-32k -f Modelfile

More context costs memory, so check ollama ps afterward to see that the model still runs fully on the GPU.

How to detect it

  • The log. Search for truncating input prompt: ~/.ollama/logs/server.log on a Mac, journalctl -u ollama on Linux.
  • The token count. prompt_eval_count on the native API, usage.prompt_tokens on the OpenAI-compatible one. If it’s far below the size of what you sent and close to half your context length, the prompt was cut.
  • Fail loudly. The native /api/chat also accepts "truncate": false. With it, an oversized prompt gets a 400 saying the request exceeds the available context size, instead of a quiet cut.

How I ran into it

On August 19 I was looking at why searching my past chats in Catalyst by meaning gave such poor results. It summarizes each chat in the background and indexes the summary, and the summarizer runs on a local model through Ollama’s OpenAI-compatible endpoint. One of the causes was in its log, in lines like this:

Could not extract JSON from summary response: I understand your frustration…

The prompt was the instruction first, then the whole transcript, in one user message. Long agentic chats, with tool output and pasted files, ran past the context window. With only one message, Ollama could only cut tokens: it kept the end of the transcript and lost the instruction and the system prompt. The model replied to the last message, as it should with what it had. It all ran in the background, so the only trace was that log line.

The fix was the first two steps: a 2,000-character clip per message, a 24,000-character cap on the transcript, and the instruction after the transcript in all three extraction prompts, plus JSON mode on the summary call. Thinking models have their own way of eating the token budget; I wrote that up in Ollama ignores think: false.

By: Gavi Narra on: