rdoq: skip uncoded cost setup for trailing zero groups

[xiujunwang-dev]

When the final coefficient group precedes trailing zero groups, rdoQuant currently initializes uncoded distortion for every coefficient. The trailing groups add the same cost to every candidate, so this omits common work while preserving RDOQ decisions.

Validation:

• Quant::rdoQuant function-level timing on arm64, five paired runs (ns/call):
  • HD input, 16x16: 418.5 -> 406.9 (+2.9%), faster in 5/5 runs.
  • HD input, 32x32: 1630.1 -> 1484.9 (+9.8%), faster in 5/5 runs.
  • Portrait input, 16x16: 634.5 -> 618.4 (+2.6%), faster in 3/5 runs.
  • Portrait input, 32x32: 2863.8 -> 2666.5 (+7.4%), faster in 5/5 runs.
  • 4x4 and 8x8 show no stable improvement.
• 36 paired RDOQ output checks passed across bit depths and modes.

[mcw-Lavanya]

Hi @xiujunwang-dev, thanks for the patch.

I verified it on x86 and ARM64 (8/10/12-bit, 26 test cases, asm off/on, AVX2 and AVX512). The output is bit-exact with master, with identical bitrate/PSNR/SSIM, and the testbench passes.

On ARM64 I see a small gain (about +1.8% fps with asm on). On x86, though, encoding is consistently about 6–7% slower in fps with both AVX2 and AVX512, across all presets, bit depths and test cases (low QP and CRF 18 are hit hardest, at about −9 to −10%). With --no-asm, x86 is neutral, so the slowdown comes from the SIMD path. A possible cause: the per-group psyRdoQuant/nonPsyRdoQuant calls replace the vectorized whole-block psyRdoQuantAll/nonPsyRdoQuantAll routines, and psyRdoQuant has no AVX2 version.

With the default preset (medium, rdoq-level 0) there is no impact, as expected (BasketballDrive 1080p, CRF 28: 21.50 vs 21.61 fps). The x86 regression appears only with RDOQ enabled (--preset slow and slower, or --rdoq-level 1/2).

Could you help with the following?

1. Have you seen this slowdown on x86, or did you test only on arm64?
2. Could you share the exact x265 command lines and input clips (the “HD” and “Portrait” inputs) you used for the rdoQuant timings and the 36 RDOQ output checks?
3. Could you check x86 performance on your side (encoder fps and rdoQuant ns/call with --asm avx2 and --asm avx512)?