Part 1: x86 Architecture for Exploit Developers — Program Memory, the Stack, and CPU Registers
The foundation behind every buffer overflow — memory layout, the stack, calling conventions, and the registers that matter
TL;DR — A 32-bit Windows process lives in
0x00000000–0x7FFFFFFF. The stack is a per-thread LIFO area (PUSH/POP) holding arguments, locals, and return addresses, bundled per call into a stack frame (prologue/epilogue, EBP-anchored layout). Calling conventions define how args are passed and the stack is cleaned. The CPU has nine 32-bit registers ("E" = Extended); the three that matter most are ESP (current stack top, moves constantly), EBP (stable frame reference per function), and EIP (next instruction — the attacker's prime target).
Before you can debug a memory corruption, write a buffer overflow, or build a ROP chain, you need a clear mental model of three things: how a program's memory is laid out, how the CPU executes code, and how functions return to their caller.
This is the groundwork for OSED (EXP-301) and reverse engineering in general. Throughout we'll lean on Assembly (asm) — you don't need to write it yet, you just need to read it.
Program Memory on Windows
On a 32-bit Windows process, user-mode memory spans from 0x00000000 (lowest) to 0x7FFFFFFF (highest).
0x7FFFFFFF +---------------------------+ highest user-mode address
| Stack (per thread) |
| Heap |
| DLLs / shared libraries |
| Program Image (.exe) |
0x00000000 +---------------------------+ lowest address
There are several regions (program image, heap, DLLs…), but this module focuses on exactly one: the stack.
The Stack
A running thread executes code from the Program Image or from DLLs, and needs a short-term scratch area for:
function arguments
local variables
program control info (especially return addresses)
That's the stack, and each thread gets its own — which is what lets threads run independently.
LIFO: Last-In, First-Out
The CPU treats the stack as LIFO — the last item pushed is the first popped. Two dedicated instructions:
push eax ; put EAX on top of the stack
pop ebx ; take the top value into EBX
Calling Conventions
A calling convention is the agreed protocol between caller and callee — how arguments are handed over and how the result is passed back. x86 has several, differing in:
how params/return value are passed (registers, stack, or both)
what order params are passed
how the stack is prepared/cleaned — and who cleans it
which registers the callee must preserve
The compiler normally chooses it; sometimes a programmer overrides per-function. The point: it's a predictable contract — and predictability is what exploitation relies on.
Function Return Mechanics
When code calls a function, how does the CPU know where to come back to? The return address is stored on the stack alongside the function's parameters and locals. That per-call bundle is a stack frame.
Typical prologue: push ebp → mov ebp, esp → sub esp, N. Typical epilogue: mov esp, ebp → pop ebp → ret. ESP moves during the function; EBP stays fixed as the frame anchor.
Higher Memory
+------------------+
| Function Arg 2 | <-- [EBP+12]
+------------------+
| Function Arg 1 | <-- [EBP+8]
+------------------+
| Return Address | <-- [EBP+4]
+------------------+
| Saved EBP | <-- [EBP]
+------------------+
| Local Variable 1 | <-- [EBP-4]
+------------------+
| Local Variable 2 | <-- [EBP-8]
+------------------+
^
|
ESP
Lower Memory
When the function ends, the CPU takes the return address off the stack and resumes in the caller.
CPU Registers
Registers are the CPU's fastest storage. A 32-bit architecture has nine 32-bit registers. The names come from 16-bit days; the "E" prefix means Extended to 32-bit. Each holds 0–0xFFFFFFFF, with 16/8-bit subregisters:
|<--------------- EAX (32 bits) --------------->|
|<----- AX (16 bits) ---->|
| AH (8) | AL (8) |
General-Purpose Registers
| Register | Name | Primary role |
|---|---|---|
| EAX | Accumulator | Default for math/logic results; holds return values |
| EBX | Base | General scratch register; historically a base address |
| ECX | Counter | Iteration count for loops and shift/rotate |
| EDX | Data | Partners with EAX in mul/div; carries I/O port numbers |
| ESI | Source Index | Source address during string/memory copies |
| EDI | Destination Index | Destination address during string/memory copies |
ESP — Stack Pointer
Always points to the top of the stack; updated by every PUSH/POP (push → ESP −4, pop → ESP +4). ESP = current stack top — the register that moves with stack operations.
EBP — Base Pointer
Points to the base of the current function's stack frame, set in the prologue and held constant for the rest of the function. Stable offsets: mov eax, [ebp+8] (arg), mov ecx, [ebp-4] (local). EBP = stable frame reference vs ESP = live stack top. Optimized builds may omit the frame pointer (-fomit-frame-pointer). 64-bit: EBP→RBP, ESP→RSP.
EIP — Instruction Pointer
Always points to the next instruction. Because it directs program flow, EIP is the attacker's primary target in memory-corruption exploits. Control EIP → control execution.
Seeing It Live in WinDbg
Lab: WinDbg (x86) → File → Open Executable… → C:\Windows\SysWOW64\notepad.exe. First break lands in ntdll during loader init — expected.
WinDbg with 32-bit Notepad at the initial ntdll breakpoint
Dump all registers:
0:000> r
eax=00000000 ebx=00e01000 ecx=589b0000 edx=00000000 esi=03321880 edi=776e307c
eip=7777f0b6 esp=00cef3f4 ebp=00cef420 iopl=0 nv up ei pl zr na pe nc
ntdll!LdrpDoDebuggerBreak+0x2b:
7777f0b6 cc int 3
Notice **esp (00cef3f4)** is lower than **ebp (00cef420)** — ESP at the stack top, EBP anchoring the frame above it.
Frame at EBP:
0:000> dd ebp
00cef420 00cef67c 77779860 3a434e24 00e01000
[ebp] = saved EBP; [ebp+4] = 77779860 → ntdll!LdrpInitializeProcess+0x1b20.
Dump stack top with symbols:
0:000> dds esp
00cef3f4 3a434c78
00cef3f8 776e307c ntdll!`string'
00cef3fc 03321880
...
00cef424 77779860 ntdll!LdrpInitializeProcess+0x1b20
dds esp output
Subregisters — EAX is 0 at this breakpoint; assign a value to see AL:
0:000> r eax
eax=00000000
0:000> r al
al=00
0:000> r eax=41424344
0:000> r al
al=44 ; AL = lowest byte of EAX
r eax / r al subregister demo
WinDbg fluency now makes every later module — overflows, SEH, ROP — far easier.
Why This Matters for Exploitation
A buffer overflow overwrites stack data until it reaches the saved return address. On return, the CPU loads that into EIP — control it, and you control execution. SEH overwrites, ROP, and egghunters are all variations on getting a controlled value into EIP.
Conclusion
Memory:
0x00000000–0x7FFFFFFF; focus = stack.Stack = per-thread, LIFO, via PUSH/POP.
Calling conventions = how args pass, results return, stack cleans.
Stack frame = one call's args + locals + return address.
Nine 32-bit registers ("E" = Extended) with 16/8-bit subregisters.
ESP = current stack top (moves) · EBP = stable frame reference (fixed per function) · EIP = next instruction (the target).
Get comfortable dumping registers and reading the stack in WinDbg now. Next module: stepping through real code on the path to controlling EIP.
I'm documenting my full OSED (EXP-301) journey one topic at a time. Follow along if you're on the same path.
