Skip to content

Openssl3.5.0 merge#805

Open
nfrayer wants to merge 47 commits into
rhboot:mainfrom
nfrayer:openssl3.5.0_merge
Open

Openssl3.5.0 merge#805
nfrayer wants to merge 47 commits into
rhboot:mainfrom
nfrayer:openssl3.5.0_merge

Conversation

@nfrayer

@nfrayer nfrayer commented Jun 24, 2026

Copy link
Copy Markdown

This PR is about updating Cryptlib to use openssl 3.5.0 for:

  • Getting ready for PQC
  • Hopefully be closer to upstream openssl to facilitate more frequent updates in the future

Also a great effort from @vathpela to remove unused code and reduce shim binary size, thanks.

nfrayer and others added 30 commits June 24, 2026 10:35
Signed-off-by: Nicolas Frayer <nfrayer@redhat.com>
Signed-off-by: Peter Jones <pjones@redhat.com>
Parts of openssl throw stringop overflow errors a lot, and for now it's
not time to fix openssl bugs, so turn it off.

Signed-off-by: Peter Jones <pjones@redhat.com>
Currently cryptlib's APIs manually add MD5 digests to the runtime
openssl configuration.  We don't need to verify MD5 certs, so there's no
reason we need this initialization code.

This patch removes it, and in some cases also SHA-1 (for the same
reason.)

Signed-off-by: Peter Jones <pjones@redhat.com>
gettimeofday() is a macro currently implemented with the old "do { ... }
while 0;" C trick.  Our compiler supports statement-expressions, so we
should use them instead.

This patch does that.

Signed-off-by: Peter Jones <pjones@redhat.com>
This adds implementations for strstr, strspn, strcspn, and strerror(),
and tests for strstr, strspn, and strcspn.  The strerror implementation
always returns a constant.

Signed-off-by: Peter Jones <pjones@redhat.com>
When Cryptlib's Pk/CryptX509.c does this:

  #include "InternalCryptLib.h"
  #include <openssl/x509.h>
  #include <openssl/x509v3.h>
  #include <crypto/asn1.h>
  #include <openssl/asn1.h>
  #include <openssl/rsa.h>

Then we're getting InternalCryptLib.h before any system include or
gnu-efi include.

In InternalCryptLib.h, we find:

  #include <stdarg.h>

  #include <efibind.h>
  #include "Base.h"
  #include <Library/BaseLib.h>
  #include <Library/BaseMemoryLib.h>
  #include <Library/MemoryAllocationLib.h>
  #include <Library/DebugLib.h>
  #include <Library/BaseCryptLib.h>

  #include "CrtLibSupport.h"

Which means we're including efibind.h before the definition of
GNU_EFI_USE_MS_ABI that tells it how to declare EFIAPI.  As a result,
EFIAPI is defined with no contents, and thus when we get to:

  BOOLEAN
  EFIAPI
  X509ConstructCertificateStack (
    IN OUT  UINT8  **X509Stack,
    ...
    )
  {
    VA_LIST  Args;
    BOOLEAN  Result;

    VA_START (Args, X509Stack);
    Result = X509ConstructCertificateStackV (X509Stack, Args);
    VA_END (Args);
    return Result;
  }

X509ConstructCertificateStack() is defined as a System V ABI function.
But because stdarg.h was included through CrtLibSupport.h, VA_START was
*not* defined before GNU_EFI_USE_MS_ABI, and in its view VA_START is
__builtin_ms_va_start(), which uses the MS ABI.

This patch makes sure all of Cryptlib is built with GNU_EFI_USE_MS_ABI.

Signed-off-by: Peter Jones <pjones@redhat.com>
Get a default hash for hash-less signing schemes such as ML-DSA, SLH-DSA,
and EdDSA in the case when signed attributes are present as well as for the
no signed attributes case. For the latter case, EdDSA is the only signing
scheme that has a required hash (sha512 for ED25519 and shake256 for
ED448), all other ones have a suggested hash. Only use the suggested hash
if the hash provided by the caller of CMS_add1_signer passed a NULL pointer
for md. Use the required hash in any case, overriding any choice of the
caller.

Fixes: #13523

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from openssl/openssl#28923)
…case

Enable the ability to sign with a hashless signing schemes, such as ML-DSA
in pure mode, in case no attributes are used in CMS. To support this, pass
the BIO with the plain data through to the signing function so that key's
pure mode signing scheme can hash the data itself.

The current implementation relies on a seek'able BIO so that the data
stream can be read multiple times for support of multiple keys.

Some signing schemes, such as ML-DSA, support the message_update function
when signing data, others, such as EdDSA keys do not support it. The former
allows for reading data in smaller chunks and calling
EVP_PKEY_sign_message_update with the data, while the latter requires that
all data are all read into memory and then passed for signing. This latter
method could run into out-of-memory issue when signing very large files.

Fixes: #28279

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
…shless signing)

Enable signature verification for hashless signing schemes, such as ML-DSA
and EdDSA, for the non-attribute case of CMS. Also in this case the BIO
with the plain input data needs to be passed through to the signature
verification function so that the pure-mode signature verification method
can hash the plain data itself.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Implement EVP_SIGNATURE_hash_message_update() to check for support
of EVP_PKEY_sign_message_update() and EVP_PKEY_verify_message_update() and
use this function to replace the has_msg_update column in CMS.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
…spatch

Shim only verifies signatures and never exports or encodes keys.
The OpenSSL 3.x provider dispatch table (encoders.inc) registered
key-to-DER/PEM/text/blob encoders for every algorithm, which caused
encode_key2any.o, encode_key2text.o, encode_key2ms.o, and
encode_key2blob.o to be pulled into the final binary via the
defltprov/baseprov dispatch tables even though shim has no code
path that calls OSSL_ENCODER.

Replace encoders.inc with an empty file and remove the four encoder
implementation objects from the Makefile.  The encoder API
infrastructure (encoder_lib, encoder_meth, encoder_pkey) is left
intact for ABI compatibility; only the concrete implementations are
removed.

Measured on shimx64.efi:
  Before: 3,241,845 bytes
  After:  3,046,280 bytes  (-195,565 bytes, -6.0%)

Section breakdown:
  .text     -33,472 bytes  (code removed)
  .data     -41,024 bytes  (function pointer tables and rodata)
  .rela     -44,136 bytes  (relocations for function pointers)
  .eh_frame -19,160 bytes  (unwind tables, tracks .text)

Signed-off-by: Nicolas Frayer <nfrayer@redhat.com>
Right now we log this if there are four certs in db:
verify.c:150:check_db_cert_in_ram() trying to verify cert 1 (db)
verify.c:150:check_db_cert_in_ram() trying to verify cert 3 (db)
verify.c:150:check_db_cert_in_ram() trying to verify cert 5 (db)
verify.c:150:check_db_cert_in_ram() trying to verify cert 7 (db)

That is obviously wrong.

Signed-off-by: Peter Jones <pjones@redhat.com>
Currently shim supports some questionable hash types for mok passwords.
They're short-lived (typically across one reboot) and brute forcing them
later doesn't get you anything, so it's not terribly important, but
still they're kind of junk.

This patch removes support for them.  It corresponds with mokutil commit
b42e3038f4c44e6594a840986222a8c0ad333ae8 , which removes support for
generating them.

Signed-off-by: Peter Jones <pjones@redhat.com>
Signed-off-by: Peter Jones <pjones@redhat.com>
Signed-off-by: Peter Jones <pjones@redhat.com>
I have no idea why Cryptlib/OpenSSL/crypto/provider/defltprov.c existed,
but nothing is using it.

Signed-off-by: Peter Jones <pjones@redhat.com>
Nothing is using this, and it's kind of confusing.

Signed-off-by: Peter Jones <pjones@redhat.com>
This doesn't change the shim binary at all, so we're definitely not
using any of this.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not using any MACs directly, and this saves 11KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This removes OPENSSL_NO_ERR and also makes shim's init_openssl() use the
correct way to enable the error messages, instead of having to know
which subsystems are needed.

Somehow this saves 1.4KiB, which makes you really wonder what that
setting is for.

Signed-off-by: Peter Jones <pjones@redhat.com>
This mostly stops building PEM, since we never use it.  Saves over 9KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This mostly stops building any Password Based Encryption, including
PKCS12, since we don't use it.  Saves about 22KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not using openssl's compression for anything, so cut it.  Saves
5KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not doing any key exchanges.  Axe it for a cool 109KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This saves almost 5KiB and turns off an undesirable algorithm.

Signed-off-by: Peter Jones <pjones@redhat.com>
No idea why this had the TODO comment in configuration-ec.h, but we
can't be using it and it's easy to turn off.

This removes 22KiB from our shim binary.

Signed-off-by: Peter Jones <pjones@redhat.com>
We don't do any symmetric crypto except in the password routines in Mok,
so we don't need AES at all.

This removes 145KiB from our shim binary.

Signed-off-by: Peter Jones <pjones@redhat.com>
This turns off block ciphers completely.  This saves 11KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
I think this is an actual bug fix.

Anyway, it gets rid of a whole 667 bytes.

Signed-off-by: Peter Jones <pjones@redhat.com>
No idea why we would need CMAC, so turn it off too.

This removes 1.2KiB from our shim binary.

Signed-off-by: Peter Jones <pjones@redhat.com>
vathpela added 17 commits June 24, 2026 10:56
This gets rid of all the KDFs except ecdh_kdf and kdf_lib/ked_meth.

This saves almost 64KiB, which should be enough for anyone.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not doing any protocol negotiation, we don't need key exchange.

This saves ~54KiB from shimx64.efi

Signed-off-by: Peter Jones <pjones@redhat.com>
I don't think we actually need any MAC at all, but nevertheless this
disables the legacy mac functions we *definitely* don't need.

Saves 11KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This gets rid of all* SHA1, SHA2-224, and SHA2-256-192, which saves around 12KiB.

Unfortunately too many protocols use EVP_sha1() to /actually/ get rid of
its implementation in legacy_sha.c, but this removes it from the
providers at least.

Signed-off-by: Peter Jones <pjones@redhat.com>
This disables all of the truncated SHA hashes, which saves around 7KiB

Signed-off-by: Peter Jones <pjones@redhat.com>
This removes all of MD5.  I have no idea if the three direct users of
EVP_md5() are actually safe to switch to EVP_sha256().  We're already
not building x509_r2x.c.  We're already not using PEM at all, but
pem_lib hasn't been eradicated (at least not yet).

This saves around 9KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This disables ECX, Curve25519, Curve448, and EDDSA, for a savings of
172KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This introduces OPENSSL_NO_WEAK_EC, which disables NIST P224 and P256
and similar.

We're only using P384 and larger, and this saves 49KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not using SHA3 or SHAKE yet, we don't need this.  This saves about
28KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
I don't think we need any of this, but I gotta double check about the
RNG.  This cuts out around 25KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
What would we do if an RNG test failed?  When would we call such a
thing?  Anyway, saves 9KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
This takes it out of the function call tables, but doesn't actually not
build it.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not using ML-KEM for Secure Boot, so we don't need it.  Saves
about 63KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
We're not using RSA as a cipher, just as a signature method, so we don't
need to register it as a cipher, or build that API.  This doesn't even
remove most of the implementation, but it already saves almost 5KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
Surprisingly this only saves 855 *bytes*

Signed-off-by: Peter Jones <pjones@redhat.com>
I imagine we *might* have to undo some of this in the glorious future
where we're sealing keys in the TPM, maybe, but until then may as well
not build it.

Saves about 6KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
The only NIST curve we actually need is P384, so for now getting rid of
P521 saves us a hefty 30KiB.

Signed-off-by: Peter Jones <pjones@redhat.com>
@nfrayer nfrayer force-pushed the openssl3.5.0_merge branch from 3d7c6c8 to 827cb3b Compare June 24, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants