Image request cap ignores modelCapabilities.limits.vision.max_prompt_images, and its newest-first eviction invalidates the prompt cache on every request #2823

Activity

1. github-actions commented on Oct 8, 2026

   

   github-actionsbot

   on Oct 8, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the detailed report. I labeled this bug.

   What I checked: I searched this repo for max_prompt_images / maxPromptImages. It appears only in the generated RPC types (Node, Python, Go, .NET, Rust, Java), in the hand-written per-SDK ModelCapabilitiesOverride / vision-limits types, and in docs/features/image-input.md. The SDKs only pass the override through in the session.open / setModel payloads. No SDK code enforces or drops the 50-image cap, and the "Removed {n} images..." message is not generated here.

   Findings:

   1. Override ignored. The SDK serializes the field correctly, so the cap is applied inside the Copilot runtime (CLI). Either the runtime ignores the override, or it applies it only to part of the request-building path. That is a runtime defect, or an undocumented limitation if intentional. As you note, accepting the field without effect is the same class of problem as modelCapabilities.limits.maxContextWindowTokens is ignored by the runtime; only the undocumented provider.maxContextWindowTokens works #2756. I couldn't reproduce the runtime behavior from this repo, since the runtime source isn't here.
   2. Newest-first eviction. This also lives in the runtime. Your analysis is plausible: if eviction keeps the newest N images, the retained set changes on each request, which rewrites an early message and breaks the cacheable prefix. A sticky oldest-first policy would be prefix-stable.

   Next steps: The fix belongs in the runtime (github/copilot-cli), so this needs routing to the runtime team. Please link it with github/copilot-cli#4831. On the SDK side, once the runtime's behavior is settled, docs/features/image-input.md should document the image cap and whether max_prompt_images is honoured. Tell us if you have reproduction logs of the outbound request bodies. Those would help the runtime team.

   Generated by Bug Handler for #2823 · copilot · auto · 16.2 AIC · ⌖ 0.472 AIC · ⊞ 7.5K · ◷
2. IslamTayeb commented on Oct 10, 2026

   

   IslamTayeb

   on Oct 10, 2026

   Last edited by IslamTayeb

   Author

   More actions

   Routed to the runtime as github/copilot-cli#5111, which carries this full report, since #4831 there is closed. Please follow up on that issue; this comment stays as the SDK-side record.

   Linked with github/copilot-cli#4831 as requested, with a comment there pointing back here. That issue was auto-closed as resolved in CLI v1.0.88, which raised the cap from 1 to 50; the retention half it asked for ("some way for an old image to drop out instead of holding the slot") is what this issue is about.

   Reproduction logs of the outbound request bodies

   https://github.com/proxy/gist.github.com/IslamTayeb/46cfa44887d318ecec4efe1d3fabb44a

   • probe.mjs — the reproduction. @github/copilot-sdk 1.0.15 + copilot-sdk-linux-x64 1.0.15, Node v25.2.1, claude-opus-5, BYOK Anthropic through a loopback proxy that records every outbound /v1/messages body. createSession passes modelCapabilities: { supports: { vision: true }, limits: { vision: { max_prompt_images: 1000000 } } }. The workspace holds 61 tiny 8×8 PNGs; the model is asked to view tile_01 … tile_60 (10 per response), then reply ok (no new image), then view tile_61.
   • request_bodies.json — all 10 request bodies as the runtime sent them. Image base64 is replaced by sha256 + length, the Copilot system prompt and tool schemas by digests; messages are otherwise verbatim, cache_control markers included. Each entry also carries the assistant.usage event for that request.
   • session_events.jsonl — model.log, assistant.usage, assistant.message, tool.execution_start events.

   request	images in body	tool_results replaced by the notice	earlier message rewritten (index)	cacheReadTokens	cacheWriteTokens
   1	0	0	–	0	13,912
   2	10	0	none	13,912	1,692
   3	20	0	none	15,604	1,703
   4	30	0	none	17,307	1,695
   5	40	0	none	19,002	1,695
   6	50	0	none	20,697	1,695
   7	50	10	2	13,912	10,400
   8	50	10	none	24,312	48
   9	50	10	none	24,360	58
   10	50	11	4	13,715	10,886

   What the bodies show:

   1. max_prompt_images is ignored. With the override at 1,000,000, request 7 (60 images in history) carries exactly 50 and the runtime logged Removed 10 images from the model request to stay within the 50-image request limit…. The same tool_result in message index 2, request 6 → request 7:

   {"type":"tool_result","tool_use_id":"toolu_011jisLSffC4K4nSk18SbGs5","content":[{"type":"text","text":"Viewed image file successfully."},{"type":"image","source":{"type":"base64","media_type":"image/png","data":"<sha256:06e76d363b3c678f base64_chars:100>"}}]}

   {"type":"tool_result","tool_use_id":"toolu_011jisLSffC4K4nSk18SbGs5","content":[{"type":"text","text":"Viewed image file successfully."},{"type":"text","text":"You've reached the maximum number of images you can view (50) so I can't provide the image for you to see."}]}

   2. Each admission past the cap rewrites an already-sent message. The replacement lands in the oldest tool batch (message 2 at request 7, message 4 at request 10), so the cacheable prefix ends there: cacheReadTokens drops from 20,697 to 13,912 (system prompt + tools + first user message) and the rest is re-written. Requests 8 and 9 add no image and nothing is rewritten (48 and 58 write tokens). Request 10 admits one more image, replaces one more old tool_result, and rewrites again.

   3. A correction to my ask 2. The bodies show the runtime's eviction is already oldest-first and sticky: the dropped set goes 10 → 10 → 10 → 11 and never shrinks or reorders. So the problem is not that the retained set is unstable between requests; it is that every image admitted past the cap replaces one tool_result that the provider has already cached. A "sticky oldest-first" policy is what runs today and does not help. The only prefix-stable behaviours are (a) no eviction, i.e. honour max_prompt_images (our workaround: a proxy that puts the evicted images back from earlier requests; the Anthropic API accepted 120 images in one request), or (b) refuse the new image at the tail and keep the old ones, which starves the model of what it just rendered. Which one is the runtime team's call; (a) is the one the SDK already advertises.

   4. The warning under-reports. It fired on every eviction here (4 of 4), but in the production run below it fired twice for 51 evictions, and runtime.node contains mark_image_limit_warning_emitted, so counting warnings does not measure the problem.

   Same mechanism at production scale (same SDK, no override)

   Two runs of the same paper on the same harness; the only difference is whether our proxy restored the evicted images before the request reached the API.

   	requests	tool images	requests after an image past the 50th	full prefix rewrites	cache-write tokens	cache-read tokens	cost
   runtime eviction reaches the API	200	108	51	52 (request 1 + the 51)	16,035,273	32,205,310	$120.63
   evicted images restored by the proxy	174	76	26	1 (request 1)	381,220	34,377,849	$23.70

   In the first run every one of the 51 requests that followed an image arrival past the cap shows cacheReadTokens ≤ 16,144 (system + tools) and a 189K–381K write; no other request does. Those 51 rewrites are 15.68M of the 16.04M write tokens. (This is a different replicate from the table in the issue body, hence the slightly different numbers.)