CVE-2023-3171 - How a Deserialization Flaw in EAP-7 Could Let Attackers Crash Your App Server

If you’re running Red Hat’s JBoss EAP 7, there’s a critical deserialization security bug you need to know about: CVE-2023-3171. Many businesses use EAP-7 (Enterprise Application Platform) to run important Java apps, so this vulnerability could hit hard. In this post, let’s walk through what this flaw is, how attackers can use it, and what you should do to stay safe.

What is CVE-2023-3171?

At its core, CVE-2023-3171 is a deserialization flaw found in JBoss EAP-7 and its distributions (also possibly in WildFly, the upstream project). When the server deserializes certain Java classes, it doesn’t properly check how much memory is consumed by the objects created. Specifically, attackers can submit malicious input that instructs the server to create very large HashMap or HashTable objects—or objects with enormous numbers of entries. Because there’s no guardrail here, it’s possible for someone to easily eat up all your JVM’s heap memory with just a few carefully crafted requests, causing a Denial of Service (DoS). The server freezes up, becomes unresponsive, or even crashes.

> "A flaw was found in EAP-7...which permits instantiation of HashMap and HashTable with no checks on resources consumed. This issue could allow an attacker to submit malicious requests...which could eventually exhaust the heap and result in a Denial of Service."

How Does It Work?

Java deserialization is a notorious source of vulnerabilities. When a server accepts serialized Java objects from users, it’s trusting that those objects are safe to reconstruct in memory. But if the user is an attacker, they can craft objects that, when deserialized, are extremely expensive in terms of resources.

Let’s break this specific case down

- The attacker crafts a special serialized object (using Java’s standard serialization, often with ObjectInputStream).

This object is a HashMap (or Hashtable) with a huge specified capacity.

- When the server receives and deserializes the object, it tries to allocate a HashMap with the specified size.

There are no checks on this size during deserialization.

- Result: The JVM’s heap gets flooded, the GC (Garbage Collector) chokes, and the app server is overwhelmed.

Code Example: The Malicious Payload

Here’s a simple snippet showing how an attacker might create such a payload in Java, though in practice, exploit kits automate these steps:

import java.io.*;
import java.util.HashMap;

public class LargeHashMapPayload {
    public static void main(String[] args) throws Exception {
        // Create a HashMap with a huge initial capacity
        int massiveCapacity = Integer.MAX_VALUE; // 2^31-1, or about 2 billion entries
        HashMap<Object, Object> evilMap = new HashMap<>(massiveCapacity);

        // Serialize to a file or send over the network
        ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("evil_map.ser"));
        out.writeObject(evilMap);
        out.close();
    }
}

This file (evil_map.ser) can then be delivered to the server endpoint that expects serialized objects—often via HTTP POST or RMI calls.

Here’s how an attacker could leverage CVE-2023-3171 in practice

1. Find a Deserialization Endpoint: Many Java web apps or EAP-based services accept serialized objects over HTTP (REST endpoints, admin APIs, etc).

Send Malicious Data: The attacker sends the massive HashMap payload.

3. Watch the Heap Fill: On deserialization, the server tries to allocate a gigantic HashMap and quickly runs out of memory.
4. Take Down the Service: If it’s vulnerable, *all* users will experience slowdowns, errors, or downtime.

With automated tools, attackers can craft and send many such payloads—maybe even mixing in legitimate requests to cover their tracks.

Why Is This So Dangerous?

Denial of Service (DoS) impacts are often underestimated. If your main application server crashes, you’re looking at:

Tough incident response

Plus, this bug is easy for attackers to trigger, with little technical know-how.

All versions of Red Hat JBoss EAP 7 (until patched)

- Possibly WildFly server and any downstream Java app that doesn't restrict HashMap/Hashtable instantiation on deserialization

Other Java environments not using EAP-7 or accepting serialized objects might still be at risk if they copy similar deserialization code.

Mitigation and Fixes

Red Hat has patched this flaw. Update to the latest versions of JBoss EAP 7.4 or later (see vendor advisories below).

If you can’t patch right away

- Block/disable all endpoints that accept serialized Java objects

Add input validation or deserialization filters

- Use a Java agent or library that enforces limits on collection allocation during deserialization (like SerialKiller or JEP-290’s built-in filters)

Custom hotfixes must intercept readObject calls for HashMap/Hashtable and restrict constructed sizes.

References and Further Reading

- Official Red Hat Security Advisory: Red Hat: CVE-2023-3171
- NVD Entry: NVD - CVE-2023-3171
- JBoss EAP 7 Security: JBoss EAP 7 Security Documentation
- About Java Deserialization Vulnerabilities: OWASP Serialization Cheat Sheet

Final Thoughts

CVE-2023-3171 is a classic example of how a tiny oversight—failing to check object size on deserialization—can become a major security risk. If your server processes serialized Java objects, make sure to apply patches, tighten input controls, and review how your app handles such data.

Timeline

Published on: 12/27/2023 16:15:13 UTC
Last modified on: 01/04/2024 17:07:40 UTC