cybersecurity

What is Integer Overflow? Ways to Exploit, Examples and Impact

Nandini

— 6 min read
What is Integer Overflow? Ways to Exploit, Examples and Impact

In the world of low-level programming and cybersecurity, the way a computer handles numbers is fundamentally different from how we perceive them in mathematics. While humans think of numbers as infinite, computer memory is finite. When a calculation exceeds the maximum capacity of its allocated storage, a critical vulnerability known as an integer overflow occurs. This phenomenon is a cornerstone of binary exploitation and can lead to everything from minor logic errors to complete system compromise.

Understanding Computer Integers: The Foundation

To understand why an integer overflow happens, we must first look at how computers store numerical data. Unlike a calculator that can handle massive strings of digits, a programming language like C, C++, or Java assigns a fixed amount of space to a variable. This space is measured in bits.

Fixed-Width Data Types

Common integer types include:

  • 8-bit (char): Can hold $2^8$ (256) distinct values.
  • 16-bit (short): Can hold $2^{16}$ (65,536) distinct values.
  • 32-bit (int/long): Can hold $2^{32}$ (approx. 4.29 billion) distinct values.
  • 64-bit (long long): Can hold $2^{64}$ distinct values.

Within these types, we have two categories: Unsigned and Signed. An unsigned integer only stores positive values (0 to $2^n - 1$). A signed integer uses one bit (the Most Significant Bit or MSB) to represent whether the number is positive or negative, typically using a system called Two's Complement.

What is Integer Overflow?

An integer overflow occurs when an arithmetic operation attempts to create a numeric value that is outside of the range that can be represented with a given number of bits. When this happens, the value "wraps around" to the other end of the range.

Imagine a car's mechanical odometer that only has five digits. If the car travels 99,999 miles and then moves one more mile, the odometer doesn't show 100,000; it rolls back to 00,000. This is exactly how an integer overflow works in a CPU.

Unsigned Overflow Example

Consider an 8-bit unsigned integer. Its range is 0 to 255.
If we have the value 255 (binary 11111111) and add 1, the binary result is 100000000. However, since we only have 8 bits of space, the leading '1' is dropped, leaving us with 00000000, or 0.

unsigned char val = 255;
val = val + 1;
printf("%u", val); // Output: 0

Signed Overflow Example

Signed integers are more complex. In an 8-bit signed integer, the range is -128 to 127. The value 127 is represented as 01111111. If you add 1 to it, the binary becomes 10000000. In Two's Complement, the MSB being '1' indicates a negative number. Thus, adding 1 to 127 results in -128.

signed char val = 127;
val = val + 1;
printf("%d", val); // Output: -128

The Mechanics of Two's Complement

To truly grasp signed overflows, one must understand Two's Complement. This is the mathematical scheme used by computers to perform subtraction using addition logic. To get the negative version of a number, you flip all the bits and add one.

Because the highest bit determines the sign, any operation that pushes a bit into that position unexpectedly changes the number from a very large positive to a very large negative (or vice versa). This "sign bit flip" is a frequent source of security vulnerabilities, particularly in length checks and memory allocations. For security professionals using tools like Jsmon to map attack surfaces, identifying where user input interacts with these calculations is vital.

How to Exploit Integer Overflow

Attackers don't just cause overflows for fun; they use them as a "primitive" to achieve more dangerous goals, such as Buffer Overflows or Heap Overflows.

1. Bypassing Boundary Checks

Suppose a program checks the size of an incoming packet to ensure it doesn't exceed a buffer.

void process_data(unsigned short packet_count) {
    if (packet_count + 1 > 100) {
        return; // Error: Too many packets
    }
    // Allocate memory and process
}

If an attacker sends a packet_count of 65,535 (the max for a 16-bit unsigned short), the calculation packet_count + 1 overflows to 0. Since 0 is less than 100, the check passes, even though the attacker is providing a massive amount of data that the program isn't prepared to handle.

2. Memory Allocation Vulnerabilities (The malloc Disaster)

This is perhaps the most common way integer overflows lead to Remote Code Execution (RCE). Consider code that allocates memory for an array of objects:

void* create_array(size_t element_size, size_t count) {
    size_t total_size = element_size * count;
    return malloc(total_size);
}

If an attacker controls count, they can provide a very large number. Let's say element_size is 20 bytes and the attacker provides a count that, when multiplied by 20, results in a value just over the 32-bit limit (4,294,967,295). The total_size variable might overflow and wrap around to a very small number, like 50.

malloc(50) will succeed, but the subsequent code will try to write count number of elements (millions) into that 50-byte buffer. This results in a massive heap buffer overflow, allowing the attacker to overwrite function pointers or metadata to take control of the program.

3. Financial Logic Errors

In decentralized finance (DeFi) or legacy banking applications, integer overflows can be used to steal funds. If a balance is stored as an unsigned integer, and an attacker can trigger a subtraction that goes below zero, the balance wraps around to the maximum possible value.

unsigned int balance = 100;
unsigned int withdrawal = 150;
balance = balance - withdrawal; // Underflow occurs
// balance is now 4,294,967,246

Real-World Examples and Impact

The Ariane 5 Flight 501

One of the most famous (and expensive) integer overflows in history occurred in 1996. The Ariane 5 rocket exploded 40 seconds after launch. The cause was a software exception during a data conversion from a 64-bit floating point to a 16-bit signed integer. The value was greater than 32,767 (the max for a 16-bit signed int), causing an overflow that crashed the flight control system.

YouTube's "Gangnam Style" Bug

In 2014, Google had to update YouTube's view counter. Originally, views were stored as a 32-bit signed integer. When Psy's "Gangnam Style" surpassed 2,147,483,647 views, the counter overflowed and turned into a negative number. YouTube subsequently upgraded the counter to a 64-bit integer, which can handle up to 9 quintillion views.

OpenSSH and Linux Kernel Vulnerabilities

Numerous CVEs (Common Vulnerabilities and Exposures) are attributed to integer overflows. For example, CVE-2002-0639 in OpenSSH involved an integer overflow in the nresp variable, which led to a heap overflow and allowed for remote root access. These vulnerabilities prove that even well-vetted, mission-critical code is susceptible to these logic errors.

How to Prevent Integer Overflow

Preventing integer overflows requires a combination of safe coding practices, compiler protections, and robust testing.

1. Use Language-Level Protections

Modern languages like Python automatically handle large integers by dynamically allocating more memory as the number grows. Rust, by default, checks for overflows in debug builds and panics, forcing developers to handle the possibility explicitly using methods like checked_add() or saturating_add().

2. Input Validation and Range Checking

Always validate user-supplied data before using it in arithmetic. If you are expecting a small number, enforce that limit strictly.

if (count > MAX_ELEMENTS) {
    handle_error();
}
size_t total = count * sizeof(element);

3. Use Safe Libraries

In C and C++, use libraries designed to catch overflows. For example, the SafeInt library for C++ or the checked-int headers can perform operations and return a boolean indicating if an overflow occurred.

4. Compiler Flags

Modern compilers offer flags to help detect these issues:

  • -fsanitize=integer in Clang/GCC can detect various integer-related issues during testing.
  • -fwrapv tells the compiler to treat signed overflow as wrapping (defined behavior), which can sometimes prevent certain classes of exploitation but is generally less safe than fixing the logic.

5. Static and Dynamic Analysis

Using static analysis tools (SAST) can help identify potentially vulnerable arithmetic operations in your source code. During the reconnaissance phase of security auditing, utilizing a platform like Jsmon helps security teams identify exposed infrastructure where such vulnerabilities might reside in custom-built web applications or APIs.

Conclusion

Integer overflows are a classic example of how the physical constraints of hardware manifest as software vulnerabilities. While they may seem like simple math errors, their impact on security is profound. By understanding the mechanics of how numbers wrap around in memory and how those wraps can be leveraged to bypass security checks or corrupt memory, developers and security researchers can better defend against them.

As systems become more complex and the volume of data grows, the risk of hitting these numerical ceilings increases. Rigorous testing, safe arithmetic libraries, and a deep understanding of binary representation are the best defenses against this age-old class of bugs.

To proactively monitor your organization's external attack surface and catch exposures before attackers do, try Jsmon.

Read more