[julek-wolfssl]

The EC branch of EVP_DigestSignFinal() returned success without writing the signature when the DER encoding did not fit in *siglen, and it set *siglen past the end of the caller's buffer. It now fails in that case and leaves sig and *siglen unchanged.

The sig == NULL size query now uses wc_ecc_sig_size(), matching EVP_PKEY_size() and EVP_PKEY_sign(). The previous size * 2 + 8 calculation had two problems:

• It was too small for brainpoolP512r1 (136 instead of 137).
• It read ecc_sets[-1] for keys with explicit curve parameters.

The reported size is now the DER maximum. For P-521 it is 139 (was 140), and for prime239v1-3 it is 66 (was 68).

Fixes #11904

[Copilot]

🟡 Changes recommended

The test guard enables Brainpool key generation in FIPS 6+ development configurations, where it is prohibited, causing the key-generation assertion to fail before signature sizing is exercised.

1 open issue

• Exclude FIPS 6+ Brainpool configurations from key-generation test · New

What changed in this PR

Fixes #11904 in wolfSSL’s OpenSSL compatibility layer by correcting ECDSA signature buffer handling and size queries.

Changes:

• Rejects undersized signature buffers without changing the buffer or its reported length.
• Uses wc_ecc_sig_size() for size queries, fixing Brainpool sizing and explicit-curve handling.
• Adds regression checks for undersized buffers and the Brainpool P512 maximum signature size.

File	Description
wolfcrypt/​src/​evp.c	Corrects ECDSA buffer validation and signature sizing.
tests/​api/​test_evp_pkey.h	Declares and registers the new size-query test.
tests/​api/​test_evp_pkey.c	Adds buffer-failure checks and Brainpool sizing coverage.

_{Review effort: Balanced}

[github-actions]

MemBrowse Memory Report · Explore what changed →

zephyr-frdm_rw612

• FLASH: log_const_area -8 B, rodata -8 B, text +160 B (+0.1%, 282,608 B / 67,108,864 B, total: 0% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[Marti2203]

Thanks Juliusz! I tested 0a71909 with both reproducers from #11904 (--enable-opensslextra): with a 16-byte buffer EVP_DigestSignFinal() now returns 0 and leaves *siglen and the buffer unchanged, and the size query still returns 72 for P-256. The evp_pkey group passes (18 passed, 41 skipped); test_wolfSSL_EVP_DigestSignFinal_ecc_size is skipped in that configuration since it has no brainpool curves, so I did not exercise the new size query for brainpoolP512r1. Using wc_ecc_sig_size() for the size query, so it matches EVP_PKEY_size(), makes sense to me.