In June 2024, security researchers disclosed a critical vulnerability in libaom, the open-source AV1 codec library widely used in browsers (including Chrome and Firefox), video streaming apps, and media processing tools. Tracked as CVE-2024-5171, this flaw involves an integer overflow bug in the internal img_alloc_helper function, potentially causing a heap buffer overflow. This post explains the technical details, offers an example exploit scenario, shows vulnerable code snippets, and links to original references for further reading.

What Happened?

libaom is responsible for encoding and decoding AV1 video streams. For handling images and frames, libaom provides several internal and external APIs to allocate memory for image data.

The bug lives in the code that allocates these image buffers: specifically, the function img_alloc_helper does not properly validate user-supplied width, height, or alignment values before calculating memory offsets. This can be abused to cause an integer overflow while computing buffer sizes. The resulting incorrect buffer size can lead to reading/writing past the allocated buffer's end—a classic heap buffer overflow.

If exploited, this vulnerability could lead to application crashes, data leakage, or even remote code execution—especially when libaom is used in browsers or server-side video encoders handling untrusted content.

aom_img_alloc_with_border() ➡️ calls ➡️ img_alloc_helper

Each function allows the caller to request a buffer using custom width, height, and alignment values. None of these functions have robust checks to ensure that large input values won't break the math behind the buffer swath calculation.

Vulnerable Code Snippet

Here's a simplified extract of the relevant code pattern in *libaom* (source reference):

static int img_alloc_helper(aom_image_t *img,
                            aom_img_fmt_t fmt,
                            unsigned int d_w,
                            unsigned int d_h,
                            int align,
                            unsigned int size_align,
                            unsigned int border,
                            int alloc_data) {
    // ...
    unsigned int stride = ((d_w + align - 1) & ~(align - 1));
    unsigned int plane_size = stride * d_h;
    unsigned int buffer_size = plane_size + 2 * border;

    img->stride[] = stride;
    img->planes[] = malloc(buffer_size);

    // ...snip...
}

Problem: If an attacker provides a sufficiently large value (d_w, d_h, or align), the calculations for stride, plane_size, or buffer_size can wrap around or overflow, causing malloc to allocate a *much smaller buffer* than needed, whereas the code later treats it as large.

Proof-of-Concept Exploit

Let's see a simplified theoretical example in C, simulating a call to aom_img_alloc() with huge dimensions:

#include <stdio.h>
#include <stdlib.h>
#include "aom/aom_image.h" // Simplified for illustration

int main() {
    aom_image_t img;
    // Oversized values aimed to trigger overflow
    unsigned int d_w = xFFFFFFFE; // ~4GB
    unsigned int d_h = 4; 
    int align = 64;
    
    if (aom_img_alloc(&img, AOM_IMG_FMT_I420, d_w, d_h, align) == NULL) {
        printf("Allocation failed as expected\n");
    } else {
        printf("Buffer allocated with stride: %u\n", img.stride[]);
        // Potential out-of-bounds access will occur on use!
    }
}

What happens?
- The calculation stride = ((d_w + align - 1) & ~(align - 1)) can wrap around, resulting in *unexpectedly small* values.
- Downstream code assumes the image buffer is large enough, but actually, it's not, enabling a potential out-of-bounds write or read!

Exploit Potential

An attacker could submit a specially crafted AV1 video (or image) with giant frame sizes or alignment in the header. When processed by an application (for instance, a media player or browser tab), this can corrupt the heap. Given the right conditions, this could be leveraged to:

Or, in more advanced attacks, escalate into remote code execution (RCE)

Browsers are particularly at risk, since AV1 decoding can be triggered by visiting a malicious website.

Recommendations

- Update: Patch your libaom to the latest version. As of June 2024, the fix has landed.

References

- libaom Security Advisory
- Official commit fix on GitHub
- NVD entry CVE-2024-5171
- libaom img_alloc_helper on GitHub
- AV1 Codec GitHub repo

Conclusion

CVE-2024-5171 shows how a seemingly simple math error in a library used almost everywhere can have a major security impact, especially in browsers or server-side video platforms. Integer overflows are easy to miss and can have catastrophic downstream effects. If you use libaom, make sure to patch now—don't let a video buffer overflow your day.

Timeline

Published on: 06/05/2024 20:15:13 UTC
Last modified on: 07/23/2024 18:09:56 UTC