CVE-2023-50868 - Understanding and Exploiting the "NSEC3" DNSSEC DoS Vulnerability

DNS is one of the internet's oldest and most critical protocols. To keep it secure, extensions like DNSSEC were developed. However, these security upgrades sometimes introduce new risks. In late 2023, a vulnerability (CVE-2023-50868) was disclosed, exploiting the way DNSSEC handles the Closest Encloser Proof, causing significant CPU exhaustion on DNS servers. This writeup breaks down the issue in simple language, demonstrates how it can be abused, and offers mitigation recommendations for sysadmins.

What’s the Bug?

CVE-2023-50868 targets DNS servers that implement DNSSEC using the NSEC3 mechanism (as specified in RFC 5155) but do not follow newer improvements (RFC 9276). The bug arises while proving that a DNS query is for a non-existent subdomain. When you ask for a random subdomain (that doesn't exist), DNS must "prove" to you that the name doesn't exist. To do this, it runs a function called Closest Encloser Proof—which, with NSEC3, can trigger thousands of SHA-1 hash function calculations if poorly implemented.

When a large number of random queries hit your DNS server, each for a different non-existent subdomain, the server must repeatedly hash inputs using SHA-1. This use of CPU resources can be so severe that it slows down or even crashes the server, leading to a Denial of Service (DoS).

The DNS server takes the subdomain from your query.

2. For non-existent domains, it must prove there is no such entry by calculating hashed closest matches (Closest Enclosers).

Code Snippet: Simulating the Attack

Here’s a Python script using standard libraries that simulates making random DNSSEC queries against a target DNS server. Note: This code is for educational purposes ONLY. Don’t use it on systems you don’t own or have permission to test!

import random
import string
import dns.resolver

def random_subdomain(length=12):
    return ''.join(random.choice(string.ascii_lowercase) for _ in range(length))

target_zone = 'victim-domain.com'
resolver = dns.resolver.Resolver()
resolver.nameservers = ['1.2.3.4']  # Target DNS server IP

for _ in range(100):
    sub = random_subdomain()
    fqdn = f"{sub}.{target_zone}"
    try:
        answer = resolver.resolve(fqdn, 'A')
    except Exception as e:
        pass  # Ignore all errors (NXDOMAIN is expected)

Each request here triggers the DNSSEC Closest Encloser logic, leading the server to perform costly hash calculations.

References

- CVE Details: CVE-2023-50868
- ISC Security Advisory (BIND9)
- RFC 5155 - DNS Security (DNSSEC) Hashed Authenticated Denial of Existence
- RFC 9276 - DNSSEC NSEC3 Guidance for DNS Operators
- NSEC3 Closest Encloser Proof Attack Example

Attack Scenario

- Attacker: Can send many random DNS queries for non-existent domains at a victim’s DNSSEC-enabled server.
- Target: Authoritative DNSSEC server using NSEC3, not patched per RFC 9276, with significant NSEC3 iterations configured.

Technical Breakdown

When NSEC3 is used, the server must separate queries for legitimate non-existent domains from those for random, arbitrary domain names. Each random query triggers hashing of the “closest encloser” with potentially thousands of SHA-1 iterations, as allowed by the original RFC 5155 spec. This is computationally far more expensive than resolving normal queries.

Key Line from RFC 5155

> “...the algorithm must hash each label upper-bound up to NSEC3 iterations to confirm the closest encloser...” (see Section 7.2.2).

The "NSEC3" Issue

Under attack, a poorly protected server will start to consume enormous CPU resources just by handling these proofs, crowding out legitimate requests and potentially making the DNS service unresponsive.

Easy to automate—thousands of random queries per second can be generated with simple scripts.

- Works against any DNSSEC-enabled server with NSEC3 and high iteration counts, unless RFC 9276 mitigations are in place.

Deploy RFC 9276 Guidance:

- Update your DNS software and configuration to follow RFC 9276 best practices.

Rate-limit excessive queries for non-existent domains.

4. Consider NSEC/NSEC3-less Configurations:
- Depending on your security needs, switching from NSEC3 to standard NSEC or minimizing DNSSEC use for less critical domains can help.

Conclusion

CVE-2023-50868 highlights an uncomfortable truth: complexity can lead to new vulnerabilities, especially when legacy specifications are still in use. By understanding how Closest Encloser Proof can backfire into a CPU-eating monster, sysadmins and DNS operators can better defend their infrastructure.

Patch your DNS server. Reduce NSEC3 iteration counts. Stay safe.

*If you want to dig deeper, check out Cloudflare’s deep dive or review the ISC advisory for practical guidance.*


TL;DR: Don’t let your DNS server get hashed to death! CVE-2023-50868 is real—and fixable.

Timeline

Published on: 02/14/2024 16:15:45 UTC
Last modified on: 05/12/2025 15:15:56 UTC