On June 16, 2023, a serious vulnerability CVE-2023-36328 was disclosed in LibTomMath, a popular open-source multi-precision arithmetic library used by many cryptography libraries and applications. This bug occurs in the mp_grow function, and it’s an integer overflow issue. Such simple mistakes can have giant effects, letting attackers crash your app or even run their own code. If you use LibTomMath before commit beba892bcd4e4ded4d667ab1d2a94f4d75109a9, you need to read this!

What Is LibTomMath?

LibTomMath is a library for handling big integers. Since normal numbers in C are pretty small, we need libraries like LibTomMath to deal with the giant numbers needed in cryptography, like for RSA or ECC.

What’s the Integer Overflow About?

In programming, an integer overflow happens when you try to store a number that’s too big for the variable type. For example, with 32-bit unsigned ints ( to 4,294,967,295), if you go above that, it wraps around to .

In LibTomMath version before commit beba892bcd4e4ded4d667ab1d2a94f4d75109a9, the function mp_grow is supposed to increase the size of a number’s digits array. But it doesn’t check if the new size is *so big* that multiplying it by digit size would overflow.

How Overflow Can Lead to Exploits

- Buffer Overflow: The library might allocate less memory than it thinks, and then copy more data than it should—classic buffer overflow, a favorite tool for attackers.
- Arbitrary Code Execution: With careful setup, an attacker can overwrite data like function pointers or return addresses, making the app jump to code the attacker supplied.
- Denial of Service (DoS): Even without code execution, overflows usually crash the app, leading to DoS.

Here’s a simplified/representative version of the problem

int mp_grow(mp_int *a, int old_size, int new_size) {
    if (new_size <= old_size) return MP_OKAY;
    // THIS LINE IS VULNERABLE TO OVERFLOW!
    mp_digit *tmp = realloc(a->dp, new_size * sizeof(mp_digit));
    if (!tmp) return MP_MEM;
    a->dp = tmp;
    // ...
    return MP_OKAY;
}

What’s Wrong?

If new_size is user-controlled or comes from unchecked input, and is very large, new_size * sizeof(mp_digit) can wrap around and become a small number. The memory allocator will happily give you a tiny buffer, but the code will write WAY more than that, right over memory it doesn't own.

The Commit that Fixed It

The LibTomMath maintainers fixed it in commit beba892:

+ if (new_size > (INT_MAX / sizeof(mp_digit))) return MP_MEM;
mp_digit *tmp = realloc(a->dp, new_size * sizeof(mp_digit));

1. Trigger the Overflow

A bad actor can send data that makes your app call mp_grow with a massive new_size. This could happen if you, say, read a giant public key or a manipulated protocol message.

2. Heap Corruption

The code will allocate too little memory (due to integer overflow). When the library later writes data, it will smash over other heap data.

3. Remote Code Execution

With skill, the attacker could place payloads so that when the overflow happens, they end up overwriting sensitive structures, like function pointers. Next time the app calls a function, it jumps to attacker code!

4. Crash (DoS)

If the memory corruption just breaks something important, the app will crash, causing a denial of service.

PoC (Proof-of-Concept) Example

This is only to show how you might crash an app using a similar bug. Imagine an attacker controls new_size:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

typedef unsigned int mp_digit;

void exploit(int new_size) {
    // INTENTIONAL OVERFLOW FOR DEMO:
    mp_digit *arr = (mp_digit *)malloc(new_size * sizeof(mp_digit));
    for (int i = ; i < new_size; i++) {
        arr[i] = i; // THIS WILL OVERWRITE MEMORY IF MALLOC WAS TINY!
    }
    free(arr);
}

int main(int argc, char *argv[]) {
    if (argc < 2) return 1;
    int input_size = atoi(argv[1]);
    exploit(input_size);
    return ;
}

If you pass a very big number (like 4294967296 on a 32-bit machine), the allocation will wrap around and be tiny, but arr[i] = i will still loop a huge number of times, crashing the app or worse.

Who Is Affected?

If you use LibTomMath in your app, especially if users can send big numbers (cryptography, communication protocols, debug features), you’re at risk. Common software using LibTomMath:

Embedded systems

- Security/crypto protocols

How to Fix

Upgrade LibTomMath to at least commit beba892 or newer.

Check your package manager, or manually grab the latest release.

References

- Official CVE Entry
- LibTomMath GitHub: Commit Fix
- Discussion in oss-security
- Common Integer Overflow Vulnerabilities

Conclusion

CVE-2023-36328 is a classic example of how seemingly harmless arithmetic mistakes can open the door for attackers. Don’t wait. Patch your LibTomMath now, and always check for integer overflows when allocating memory!

If you learned something today, please share this post — it could help someone else avoid a dangerous bug.


*This article is an exclusive, plain-English explanation about CVE-2023-36328 and integer overflow risks in security-critical C code. Stay safe out there!*

Timeline

Published on: 09/01/2023 16:15:08 UTC
Last modified on: 11/07/2023 04:16:32 UTC