cybersecurity

What is Buffer Overflow? Ways to Exploit, Examples and Impact

Nandini

— 6 min read
What is Buffer Overflow? Ways to Exploit, Examples and Impact

In the realm of cybersecurity, few vulnerabilities have stood the test of time as prominently as the buffer overflow. Despite being one of the oldest known software flaws, it remains a critical threat to modern infrastructure, often leading to full system compromise. A buffer overflow occurs when a program attempts to write more data into a fixed-length block of memory, or "buffer," than it was designed to hold. This excess data spills over into adjacent memory locations, potentially corrupting data, crashing the application, or, most dangerously, allowing an attacker to execute arbitrary code. Understanding this vulnerability is foundational for any security professional, especially when using tools like Jsmon to monitor the external attack surface for exposed services that might harbor such flaws.

Understanding Computer Memory Layout

To grasp how a buffer overflow works, one must first understand how a program organizes memory during execution. In most modern architectures, a process's memory is divided into several segments. The two most relevant to buffer overflows are the Stack and the Heap.

The Stack

The stack is a region of memory that stores local variables, function parameters, and return addresses. It operates on a Last-In, First-Out (LIFO) basis. When a function is called, a new "stack frame" is created. This frame contains the data needed for that function to execute. Crucially, it also contains the "Return Address"—a pointer that tells the CPU where to go once the function finishes its execution. Because the stack grows downward (toward lower memory addresses) and buffers are typically filled upward, an unchecked write can easily reach the return address.

The Heap

The heap is used for dynamic memory allocation. Unlike the stack, which is managed automatically by the compiler, the heap is managed manually by the programmer using functions like malloc() and free() in C. While heap overflows are more complex to exploit than stack overflows, they are equally devastating, often involving the corruption of metadata used by the memory allocator.

How Buffer Overflows Work: The Technical Mechanism

At its core, a buffer overflow is a failure of boundary checking. Consider a simple program written in C, a language that does not perform automatic bounds checking on arrays or pointers.

#include <string.h>
#include <stdio.h>

void login(char *input) {
    char password_buffer[16];
    // Vulnerable function: strcpy does not check the size of input
    strcpy(password_buffer, input);
    printf("Input processed.\n");
}

int main(int argc, char *argv[]) {
    if (argc > 1) {
        login(argv[1]);
    }
    return 0;
}

In the example above, password_buffer is allocated 16 bytes on the stack. If a user provides an input string of 10 characters, the program functions normally. However, if the user provides 32 characters, the strcpy function will continue writing past the 16th byte. It will overwrite whatever follows the buffer on the stack. This typically includes the Saved Frame Pointer (SFP) and, most importantly, the Return Address (RET).

By carefully crafting the input, an attacker can overwrite the Return Address with the memory address of their own malicious code (often called "shellcode"). When the login function finishes, the CPU looks at the Return Address to decide where to execute next. Instead of returning to the main function, it jumps straight into the attacker's shellcode.

Step-by-Step: Anatomy of a Buffer Overflow Attack

Exploiting a buffer overflow is a methodical process. While modern operating systems have introduced protections, the fundamental steps remain a core part of exploit development training.

1. Fuzzing the Application

The first step is identifying that a vulnerability exists. This is often done through "fuzzing"—sending increasingly large or malformed strings to an application until it crashes. For example, if a network service crashes when sent 500 "A" characters, it is a strong indicator of a buffer overflow.

2. Finding the Offset

Once a crash is confirmed, the attacker needs to know exactly which bytes in their input are overwriting the Return Address. This is known as finding the "offset." Instead of sending 500 "A"s, the attacker sends a unique cyclic pattern (e.g., Aa0Aa1Aa2...). When the program crashes, the CPU's Instruction Pointer (EIP on x86, RIP on x64) will contain a specific value from that pattern. By looking up that value, the attacker knows exactly where the Return Address sits relative to the start of the buffer.

3. Identifying Bad Characters

Not all characters can be used in an exploit. For instance, the null byte (\x00) often signifies the end of a string in C. If the shellcode contains a null byte, the strcpy function will stop copying, and the exploit will fail. Other "bad characters" might include carriage returns (\x0d) or line feeds (\x0a). Attackers must identify these and encode their shellcode to avoid them.

4. Redirecting Execution Flow

With the offset known, the attacker replaces the Return Address with a new pointer. In a basic stack overflow, they might point it to a JMP ESP instruction found within a loaded library (like libc). This instruction tells the CPU to jump to the location currently pointed to by the Stack Pointer (ESP), which is exactly where the attacker's shellcode is residing.

Types of Buffer Overflow Attacks

While stack-based overflows are the most common entry point for beginners, several variations exist:

  • Stack-Based Overflow: The most common type, targeting local variables and return addresses on the call stack.
  • Heap-Based Overflow: Targets memory in the dynamic heap. These are harder to exploit because the heap doesn't have a direct return address to overwrite; instead, attackers often overwrite function pointers or manipulate memory allocation structures.
  • Integer Overflow: While not a buffer overflow itself, an integer overflow can lead to one. If a program calculates the size of a buffer using an integer that wraps around (e.g., adding 1 to the maximum value of an unsigned integer results in 0), it might allocate a tiny buffer for a huge amount of data.
  • Format String Vulnerability: Occurs when an application passes user input directly into functions like printf(). Attackers can use format specifiers like %x to read the stack or %n to write to memory.

Real-World Examples and Impact

The impact of a buffer overflow can range from a simple application crash to a full-scale network worm.

The Morris Worm (1988)

One of the first major internet-scale events, the Morris Worm, utilized a buffer overflow in the gets() function of the Unix fingerd daemon. It allowed the worm to spread autonomously across the early internet, infecting thousands of systems. This event highlighted the danger of unsafe C functions early on.

Heartbleed (2014)

While technically a "buffer over-read" rather than an overflow, Heartbleed (CVE-2014-0160) in OpenSSL shares the same root cause: a lack of bounds checking. By sending a malformed heartbeat request, an attacker could trick the server into sending back data from its memory, including private encryption keys and user credentials.

Impact Summary

  • Remote Code Execution (RCE): The most severe outcome, allowing an attacker to run any command on the target system.
  • Denial of Service (DoS): Crashing a critical service, making it unavailable to legitimate users.
  • Privilege Escalation: If a vulnerable program runs with administrative rights, a local user can exploit it to gain full control of the machine.

How to Prevent Buffer Overflow Vulnerabilities

Preventing buffer overflows requires a multi-layered approach involving secure coding practices, compiler-level protections, and operating system features.

1. Use Safe Functions

Developers should avoid inherently dangerous functions in C and C++. For every unsafe function, there is a safer alternative that requires a length argument:

  • Use strncpy() instead of strcpy()
  • Use fgets() instead of gets()
  • Use snprintf() instead of sprintf()

2. Compiler Protections

Modern compilers include features to detect and mitigate overflows:

  • Stack Canaries: The compiler places a small, random value (a "canary") on the stack just before the return address. Before the function returns, it checks if the canary is still intact. If it has been changed (due to an overflow), the program terminates immediately.
  • FORTIFY_SOURCE: A compiler feature that replaces unsafe string functions with safer versions that perform light-weight bounds checking at runtime.

3. Operating System Protections

  • ASLR (Address Space Layout Randomization): ASLR randomizes the memory addresses used by the stack, heap, and libraries every time a program runs. This makes it difficult for an attacker to predict where their shellcode or a JMP ESP instruction will be.
  • DEP/NX Bit (Data Execution Prevention / No-Execute): This marks certain areas of memory (like the stack) as non-executable. Even if an attacker successfully redirects the CPU to their shellcode on the stack, the CPU will refuse to execute it, causing a crash instead.

Conclusion

Buffer overflows represent a fundamental flaw in how software handles data and memory. While the industry has moved toward memory-safe languages like Rust, Python, and Go, the massive legacy of C and C++ codebases means that buffer overflows will remain a primary concern for the foreseeable future. By understanding the mechanics of the stack, the process of exploitation, and the necessity of robust mitigations like ASLR and stack canaries, security professionals can better defend their environments.

To proactively monitor your organization's external attack surface and catch exposures before attackers do, try Jsmon. By keeping a constant eye on your infrastructure and the services you expose, you can identify the very entry points that attackers use to launch memory corruption exploits.

Read more