Repository navigation
fix(agent-runtime): keep image and blob bytes out of tool result fallback text - #1570
Open
xpeng5278-web wants to merge 1 commit into
Open
xpeng5278-web wants to merge 1 commit into
xpeng5278-web wants to merge 1 commit into
Conversation
…back text An image-only or blob-only host result has no text block, so the MCP content and bare content-block fallbacks stringified the raw payload. That copied base64 into the model text channel, duplicating the image block for vision models and dumping bytes as text for every other model. Strip image data and resource blob bytes in that JSON fallback only. Type, mimeType, and uri stay so the attachment is still described.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1569
What
In the host tool bridge (
packages/agent-runtime/src/runtime.ts), the MCPcontentbranch and the bare content-block-array branch now stringify their no-text fallback through a smallwithoutInlineBinaryhelper. It drops imagedataand resourceblobbytes and keeps everything else (type,mimeType,uri, other fields), so the model is still told what was returned.Image block extraction,
details, theimagesbranch,hostContentBlockTextandtoolResultFromUiare unchanged.Docs:
docs/spec/03-runtime/02-agent-runtime.md§3.3 now states that the no-text JSON fallback also keeps image and blob bytes out of the model-visible text.Why
textPartsonly holdstext,resource.textandresource_linkcontent. An image-only or blob-only result therefore fell back toJSON.stringify(rawContent), base64 included. Vision models received the image twice (image block + base64 text). Text-only models received the base64 as plain text. One screenshot could add hundreds of KB to the context. Theimagesbranch already stripsimagesbefore stringifying, and #1360 / #1551 already keep binary out of the text channel; this aligns the two remaining fallbacks.How tested
packages/agent-runtime/src/runtime.test.ts:contentand image-only bare array → no base64 in text, image block kept,imageCount: 1uriandmimeTypestill presentmain(eec988c0f): the 4 new cases fail (expected '…' not to contain 'cG5nLWJ5dGVz'). With this change:runtime.test.ts311/311 pass.packages/agent-runtime:vitest run→ 1333 tests pass;tsc -p . --noEmitclean.node scripts/check-pr-base-main.mjs,node scripts/check-architecture.mjs,pnpm lint:biome,docs/scripts/check-docs.mjspassed.Scope
Only the no-text JSON fallback in those two branches, plus the regression tests and one spec sentence. Results that have a text block, the
imagesshape, and persisted-row restore are unchanged. Other binary-carrying block kinds (e.g. MCPaudio) are not touched here.