[LinuxJedi]

Description

Builds on #11914. Only the last commit, "Keep the TLS peer certificate as a reference-counted heap object", is new here; the others are #11914's and drop out of this diff once it merges.

With KEEP_PEER_CERT, which OPENSSL_EXTRA turns on, every WOLFSSL embedded a whole WOLFSSL_X509 for the peer's certificate: 3.4 KB on Cortex-M33, 4.3 KB with OPENSSL_ALL. InitSSL() zeroed it. Every connection paid for it, including ones that never have a peer certificate: servers that don't ask for client authentication, PSK connections, and any object before its handshake. wolfSSL_get_peer_certificate() returned wolfSSL_X509_dup() of it, which parses the certificate DER again on every call.

Changes:

• ssl->peerCert is now a WOLFSSL_X509*, NULL until a certificate is kept. The WOLFSSL holds one reference to it.
• KeepPeerCert() replaces the kept certificate with a new object each time instead of reusing it. It is used for the leaf in SetupStoreCtxCallback(), in ProcessPeerCerts() and when restoring a certificate from a ticket. A reference handed out stays valid across renegotiation, wolfSSL_clear() and a ticket restore, and after wolfSSL_free(). A copy that fails part way is still kept and its error returned, as before.
• wolfSSL_get_peer_certificate() returns the kept object with a new reference when WOLFSSL_X509 has a reference count (OPENSSL_EXTRA or OPENSSL_EXTRA_X509_SMALL), as OpenSSL does. Without one (for example --enable-sep) it still returns a copy. Either way the caller frees it with wolfSSL_X509_free(), as before.
• With SESSION_CERTS, a resumed connection's certificate is decoded from the session chain into a new kept object.
• ReinitX509() only existed for the embedded object and is removed.
• Dual-algorithm certificates: DoTls13CertificateVerify() returns PEER_KEY_ERROR when no peer certificate is kept, instead of reading the alternative key from an empty struct.

Behaviour change for applications: repeated calls to wolfSSL_get_peer_certificate() now return the same object, and changes made to it are seen through the WOLFSSL, as with OpenSSL.

Results

STM32U585 (Cortex-M33 at 160 MHz, newlib-nano malloc), TLS 1.3 client and server over memory I/O, ECC P-256 chain with two intermediates, server not asking for a client certificate. Before is #11914, after is this change on top of it.

	OPENSSL_EXTRA	OPENSSL_ALL
sizeof(WOLFSSL)	4560 -> 1144 B	5576 -> 1304 B
Server heap per connection	6842 -> 3426 B	12366 -> 8094 B
Server wolfSSL_new() time	2.43 -> 2.10 ms	2.93 -> 2.50 ms
Client wolfSSL_new() heap	7726 -> 4310 B	14914 -> 10642 B
Client handshake peak, from before wolfSSL_new()	23694 -> 23692 B	34290 -> 34289 B
Client handshake time	116.4 -> 116.4 ms	116.9 -> 117.0 ms
newlib heap high-water mark, whole run	57.0 -> 50.1 KB	82.4 -> 77.6 KB
Flash	-1.3 KB	-1.6 KB

A client that gets a certificate keeps it either way, so its handshake peak and resident heap are unchanged: the certificate is allocated during the handshake instead of being part of wolfSSL_new(). A client handshake makes one more allocation, or two with a verify callback, since the leaf is copied once for the callback and once to keep, as before.

Testing

• test_wolfSSL_get_peer_certificate_ref (new) checks that:
  • the client is handed the kept object, with its reference count going 2, 3, 2 across two calls and a free;
  • a server that did not ask for a client certificate keeps none;
  • a held reference keeps its certificate after wolfSSL_clear() and a new handshake, which keeps a new object;
  • a held reference stays valid after wolfSSL_free().
• test_tls13_ticket_peer_cert_reverify now checks that no certificate is kept, instead of checking the embedded issuer size.
• make check passed on macOS/clang for default, --enable-opensslextra, --enable-opensslall and --enable-all on this branch.
• On Reduce OpenSSL compat layer heap use and time when parsing certificates #11914 before its latest commits:
  • make check passed on macOS/clang for CFLAGS="-DKEEP_PEER_CERT -DSESSION_CERTS" (the copy path) and --enable-experimental --enable-dual-alg-certs --enable-mldsa. --enable-opensslextra=x509small and --enable-sep built.
  • make check passed on Linux x86_64/gcc for --enable-opensslextra and --enable-all. Valgrind on the peer certificate API tests reported no errors and no leaks.
  • --enable-all with clang ASan + UBSan: the peer certificate, ticket, chain and mutual authentication API tests passed with no reports.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

🤖 Generated with Claude Code

[LinuxJedi]

Closing this as it is going to be difficult to review until #11914 is merged

[github-actions]

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m3

• FLASH: .text -44 B (-0.0%, 130,773 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata -24 B, .text -576 B (-0.1%, 792,964 B / 1,048,576 B, total: 76% used)

gcc-arm-cortex-m4-pq

• FLASH: .text -64 B (-0.0%, 309,008 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text -64 B (-0.0%, 340,360 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text -64 B (-0.0%, 131,537 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text -64 B (-0.0%, 246,498 B / 262,144 B, total: 94% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text -64 B (-0.0%, 246,498 B / 262,144 B, total: 94% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• stm32-sim-stm32h753

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}