The Moby container framework (most widely recognized as the Docker engine) is the core technology behind Docker, Mirantis Container Runtime, and more. At its heart is a daemon called dockerd, which powers not just simple containers, but also *Swarm Mode*—a baked-in, simple container orchestrator. Swarm Mode networks containers together across multiple hosts using VXLAN overlays, promising isolation, security, and, optionally, encryption.
But if you thought your secrets were safe on an "encrypted overlay network" in Swarm—think twice. CVE-2023-28841 is a critical vulnerability where the "encryption" you opted into in Swarm can fail silently, leaving all your data traveling in plain text. Let’s break down how this happens, why it's dangerous, and what you can do.
How Overlay Networks Work
- Overlay networks let containers talk securely and directly, no matter which node in the cluster they're on.
- This is implemented using VXLAN (Virtual Extensible LAN), which wraps up a container’s traffic into UDP packets, tagged with a unique VXLAN Network ID (VNI).
- When encryption is enabled, a layer of IPsec (specifically, Encapsulating Security Payload or ESP in transport mode) should wrap this up, giving:
How Encryption is Enforced
- When you add a node's endpoint to an encrypted overlay, Moby installs IP tables rules (Linux firewall) using the u32 extension. These rules force outgoing VXLAN packets with the encrypted VNI to be IPsec-encrypted.
Here's roughly what happens on the network stack
# Example: iptables rule to select VXLAN VNI (pseudo)
iptables -A OUTPUT -p udp --dport 4789 -m u32 --u32 ">>22&x3c@8=x123abc" -j [IPSEC-MARK]
Where Things Go Wrong
On affected platforms (most major Linux distributions), the encryption enforcement silently fails when the xt_u32 iptables module is unavailable or not loaded. When this happens:
But the "encrypted" packets are in fact sent as cleartext VXLAN UDP datagrams.
As there's no notification or error, administrators, developers, and users may assume they're protected, but everything is wide open.
Consequences
- Total traffic exposure: Anyone on the network path, including cloud operators, ISP employees, or malicious insiders, can intercept and read the application traffic.
Secret leakage: Database queries, API calls, and internal app data can be sniffed.
- False sense of security: Many trust Swarm’s “encrypted overlay” as their sole line of defense.
Mirantis Container Runtime versions before 20.10.16 (numbered differently!)
- Any downstreams or forks including Swarm/overlay with this bug
The attacker sees
udp.port == 4789
# Traffic is plain text (unencrypted payloads)
A simple tcpdump command
sudo tcpdump -i any udp port 4789 -X
Examining a database query between two containers shows readable SQL statements—no encryption!
*(Illustrative example: Wireshark capture of VXLAN payload showing cleartext SQL query)*
If the rules had worked, the same traffic would show as ESP (IPsec-encrypted) payload, unreadable to attackers.
Moby (Docker engine):
- 23..3 release notes
- 20.10.24 release notes
Mirantis Container Runtime:
Workarounds
- Block VXLAN UDP port 4789 egress at your edge firewall to keep unencrypted overlay traffic from escaping to the internet:
sudo iptables -A OUTPUT -p udp --dport 4789 -j DROP
- Ensure all Swarm nodes have the xt_u32 module loaded:
bash
sudo modprobe xt_u32
<br> - Add to /etc/modules` for permanent loading.
- Add another encryption layer: Don’t rely solely on Swarm overlays for confidentiality—use mTLS, Wireguard, or app-level encryption for critical data.
---
## References
- Official CVE record: CVE-2023-28841
- Original Moby security advisory
- Upgrade Moby/Docker
- Mirantis advisory
---
## Conclusion
CVE-2023-28841 serves as a wake-up call: Always validate your security assumptions. If you use Docker Swarm overlay networks, do not assume traffic is encrypted just because you checked “encrypted”. Upgrade to a fixed version. Block, monitor, and layer your defenses. And, as demonstrated, simplicity in orchestration often hides sharp edges—don’t let your secrets leak silently.
Timeline
Published on: 04/04/2023 22:15:00 UTC
Last modified on: 04/14/2023 15:46:00 UTC