[itsgaldoron]

The defect

vp8_yv12_realloc_frame_buffer() computes the plane sizes in a signed int:

int aligned_width = (width + 15) & ~15;
int aligned_height = (height + 15) & ~15;
int y_stride = ((aligned_width + 2 * border) + 31) & ~31;
int yplane_size = (aligned_height + 2 * border) * y_stride;
...
int uvplane_size = (uv_height + border) * uv_stride;
const size_t frame_size = yplane_size + 2 * uvplane_size;

At dimensions and a border that fit the int parameters these products wrap, and the wrapped values are used both for the vpx_memalign(32, frame_size) allocation and for the u_buffer/v_buffer plane pointer placement. The only validation is border & 0x1f, which is applied after the allocation. Measured on the unpatched tree (x86-64, gcc 11 -O2), true sizes computed in 64-bit:

width x height	border	true yplane_size	yplane_size (int32)	true frame_size	unpatched outcome
65,536 x 65,536	32	4,303,360,000	8,392,704	6,455,040,000	alloc fails: the frame_size sum wraps a second time to a negative int on this build (undefined behavior — it can as well land in the accepted rows)
46,341 x 46,341	32	2,155,187,712	-2,139,779,584	3,232,781,568	alloc fails (negative -> huge size_t)
100,000 x 100,000	32	10,012,804,096	1,422,869,504	15,019,206,144	accepted: 2,134,304,256-byte buffer for a 15,019,206,144-byte frame
32 x 32	536,870,912 (2^29)	1,152,921,573,326,324,736	1,024	1,729,382,359,989,487,104	accepted: 1,536-byte buffer, y_buffer placed at buffer_alloc + 536,870,912

border = 2^29 is a multiple of 32 and passes the only existing check. In the accepted rows, any encode/decode/fill of the logical frame writes past the end of the allocation, and the plane pointers are computed from the same wrapped products. UBSan reports signed integer overflow at exactly these sites on the unpatched tree (yv12config.c:57 yplane_size, :63 uvplane_size, :64 frame_size, :111/:113 the plane pointer offsets).

The VP9 allocator in the same file, vpx_realloc_frame_buffer(), already does this arithmetic in uint64_t and carries the full guard set (CONFIG_SIZE_LIMIT dimension bound, VPX_MAX_ALLOCABLE_MEMORY / REF_FRAMES bound, frame_size > SIZE_MAX fail-early, external-frame-size cast check). The VP8 path has none of it.

The fix

Port the sibling's proven pattern to the VP8 path, minimal:

• compute yplane_size, uvplane_size, frame_size in uint64_t, and compute the plane pointer offsets from the same 64-bit products (they had the same wrap);
• add the same limit checks as vpx_realloc_frame_buffer(): #if CONFIG_SIZE_LIMIT dimension bound, #if defined(VPX_MAX_ALLOCABLE_MEMORY) frame_size > VPX_MAX_ALLOCABLE_MEMORY / REF_FRAMES, #if UINT64_MAX > SIZE_MAX frame_size > SIZE_MAX, each failing with -1 exactly like the VP9 path;
• validate the inputs before the arithmetic: negative width/height are rejected, and width, height, border are bounded at INT_MAX / 4 so the remaining int stride math (aligned_width + 2 * border + 31 <= 3 * (INT_MAX / 4) + 46 < INT_MAX) cannot overflow; negative border is rejected with the existing -3 code (it would place the plane pointers outside the buffer);
• move the existing border & 0x1f check (comment unchanged) ahead of the allocation instead of after it.

Legal frames are byte-identical to before: same layout formula, same return codes on the allocation-failure paths, vp8_yv12_de_alloc_frame_buffer() untouched. Measured after the patch (same probe): 65,536 x 65,536 reports frame_size = 6,455,040,000, 46,341 x 46,341 reports 3,232,781,568, 100,000 x 100,000 reports 15,019,206,144 (each either allocated at full size or failing cleanly in vpx_memalign - never a wrapped size), border = 2^29 is rejected with -3 before any allocation, and the two legal control rows (16x16, 16383x16383) report the same sizes as the unpatched build. UBSan reports no runtime errors on the patched tree.

Regression test

New test/yv12config_test.cc (registered in test/test.mk under the unconditional whitebox tests), with the expected layout computed independently in 64-bit:

• Vp8NormalFrames - 16x16, 352x288, 16383x16383 with VP8BORDERINPIXELS must still allocate and report the exact frame size and plane offsets. Passes before and after the change (behavior-neutral for legal input).
• Vp8WrappingSizesNotUndersized - the wrapping dimension rows must never be accepted with a wrapped size: either the call fails, or the buffer is the full 64-bit size at the exact plane offsets.
• Vp8OversizedBorderRejected - border = 1 << 29 must be rejected outright.
• Vp8NegativeArgsRejected - negative width/height/border must be rejected.

Unpatched tree (control build, same test file):

[ RUN      ] YV12ConfigTest.Vp8WrappingSizesNotUndersized
test/yv12config_test.cc:53: Failure
Expected equality of these values:
  layout.frame_size
    Which is: 15019206144
  (uint64_t)ybf.frame_size
    Which is: 2134304256
...
[  FAILED  ] YV12ConfigTest.Vp8WrappingSizesNotUndersized
[  FAILED  ] YV12ConfigTest.Vp8OversizedBorderRejected
[  FAILED  ] YV12ConfigTest.Vp8NegativeArgsRejected
[  PASSED  ] YV12ConfigTest.Vp8NormalFrames
 3 FAILED TESTS

Patched tree:

[==========] Running 4 tests from 1 test suite.
[       OK ] YV12ConfigTest.Vp8NormalFrames (0 ms)
[       OK ] YV12ConfigTest.Vp8WrappingSizesNotUndersized (0 ms)
[       OK ] YV12ConfigTest.Vp8OversizedBorderRejected (0 ms)
[       OK ] YV12ConfigTest.Vp8NegativeArgsRejected (0 ms)
[  PASSED  ] 4 tests.

The full test_libvpx suite (13,058 tests) was run on both the unpatched and the patched tree and the per-test results were diffed: the only differences are the three new failing tests turning green; everything else is identical. The environment's ~10,105 failures are missing-external-testdata artifacts (decode/encode corpus, y4m files) that reproduce identically on both trees - they are artifacts of the environment, not of this change.

Honest boundary

vp8_yv12_alloc_frame_buffer() / vp8_yv12_realloc_frame_buffer() are internal (vpx_scale/yv12config.h, not an installed vpx/ header), and every in-tree caller is bounded far below the wrapping range: the VP8 bitstream frame header carries width/height in 14 bits, the encoder config enforces RANGE_CHECK(cfg, g_w, 1, 16383) / g_h, and CONFIG_SIZE_LIMIT bounds the decode interface where enabled - at 16,383 the plane math tops out around 2.7e8. So this is not a demonstrated in-library overflow today; it is API-soundness hardening of an allocator whose sibling function in the same file already carries the full guard set, in the same spirit as pnggroup/libpng#967 (sizing arithmetic that wraps at legal inputs of the API).

[itsgaldoron]

libvpx reviews via Gerrit (codereview.settings); this fix is up for review at https://chromium-review.googlesource.com/c/webm/libvpx/+/8712858 — keeping this PR as the public record.