Sanitize surrogates and non-UTF-8 bytes in pydantic data converter

[xumaple]

Summary

pydantic_core's Rust to_json() serializer crashes when values contain:

• Strings with Unicode surrogates (U+D800–U+DFFF) — e.g., from subprocess output decoded with errors='surrogateescape'
• Bytes with non-UTF-8 content — pydantic serializes bytes via UTF-8 decode, so binary data like b'\x89PNG...' crashes

This was discovered in a real workload where a sandbox exec activity captured stdout from a command that read binary files (PNG headers, fonts, etc.).

Fix

Adds a _sanitize_for_json() fallback in PydanticJSONPlainPayloadConverter.to_payload():

• On the happy path, the existing Rust serializer runs unchanged
• On failure, strings are sanitized via UTF-16 round-trip (surrogate pairs become proper codepoints, lone surrogates become U+FFFD) and bytes are re-encoded as valid UTF-8, then the same serializer retries — preserving exclude_unset and all existing behavior

Related

• Normalize surrogate characters in ADK invoke_model responses #1427 applied a similar fix in the ADK integration for LLM response text fields
• This PR generalizes the fix to the pydantic data converter layer so all data flowing through it is protected

Test plan

• 9 new unit tests covering surrogates, invalid bytes, exclude_unset preservation, Pydantic models, dataclasses, and nested structures
• All 16 existing pydantic tests still pass
• All lints pass (ruff, pyright, basedpyright, pydocstyle)

🤖 Generated with Claude Code