At DEF CON CTF 2025, a team of three cracked a seemingly locked RSA challenge in under eight minutes — not through mathematical wizardry, but by recognizing a small public exponent combined with an unpadded message. That same pattern shows up in beginner and intermediate CTF crypto rounds constantly. Knowing where to look is half the fight.
Start With Classical Ciphers: Identify Before You Attack
Most CTF crypto tracks open with classical ciphers — Caesar, Vigenère, substitution. The instinct is to start guessing. Don’t. Run frequency analysis first. quipqiup handles substitution ciphers automatically, but for anything that looks like a shifted alphabet, start with a quick Python loop.
# player: ctf_user on machine: kali.local
# Challenge gives: "Gur synt vf: synt{pynffvp_ebg_rnfl}"
python3 -c "
import codecs
ciphertext = 'Gur synt vf: synt{pynffvp_ebg_rnfl}'
print(codecs.decode(ciphertext, 'rot_13'))
"
# Output:
# The flag is: flag{classic_rot_easy}
ROT13 is a Caesar cipher with a fixed shift of 13. Python’s codecs module decodes it in one line. The flag is right there. But here’s what matters for harder variants: if the shift isn’t 13, brute-force all 25 shifts. The correct one produces readable English — your signal that you’ve landed on the right key.
When you see a longer ciphertext that resists single-shift attacks, suspect Vigenère. Run the ciphertext through dcode.fr or use kasiski examination to find the key length, then attack each Caesar shift independently. The moment you find repeating trigrams, you have your foothold.
XOR — The CTF Crypto Workhorse
XOR encryption shows up in nearly every CTF, especially in beginner to intermediate tracks. The reason: it’s trivially reversible if you know anything about the key or plaintext. XOR has a critical weakness — if you XOR the ciphertext with any known portion of the plaintext, you get the key bytes for that position.
Suppose you grab a binary file from a challenge server at 192.0.2.47. The challenge says “the flag starts with flag{.” That’s five known plaintext bytes — enough to start recovering the key.
# Download the encrypted file
curl http://192.0.2.47:8080/secret.bin -o secret.bin
# Python: known-plaintext XOR key recovery
python3 - <<'EOF'
with open('secret.bin', 'rb') as f:
data = f.read()
known_plain = b'flag{'
key_bytes = bytes([data[i] ^ known_plain[i] for i in range(len(known_plain))])
print(f"Recovered key bytes: {key_bytes}")
# Try extending: if key is repeating, test full decryption
key = key_bytes # assume key length = 5 for now
decrypted = bytes([data[i] ^ key[i % len(key)] for i in range(len(data))])
print(decrypted[:80])
EOF
# Output:
# Recovered key bytes: b'\x1fX\x0c\x01R'
# flag{xor_is_not_encryption_lol} -- well done, ctf_user
Those five bytes of known plaintext unlocked the entire message because the key repeated. In a real challenge, the key length might not be five — use the xortool utility to detect key length automatically before guessing. Once you have the repeating key, the rest falls in one pass.
The defender lesson here is brutal: XOR with a short repeating key is not encryption. It's obfuscation, and weak obfuscation at that.
RSA Weak Key Attacks: Small Exponents and Factoring
RSA challenges in CTFs almost never expect you to break strong RSA. They hide a flaw: small public exponent, shared prime between two public keys, or a factorable modulus. Your first move when given an RSA challenge is to throw the modulus at factordb.com and RsaCtfTool.
# Install RsaCtfTool
git clone https://github.com/RsaCtfTool/RsaCtfTool.git
cd RsaCtfTool && pip3 install -r requirements.txt
# Challenge provides: n, e, and ciphertext c
# Values from challenge file on ctf.local
python3 RsaCtfTool.py \
--n 109966677789992229852421204514876078244960820053592980369487699117 \
--e 3 \
--uncipher 5787791992085545066827349723672769388085631958240615668387653 \
--attack smalle
# Output:
# [*] Trying attack: smalle
# [+] Clear text : b'flag{small_e_big_mistake}'
A public exponent of e=3 combined with a small message means the ciphertext is literally m^3 without wrapping around the modulus. Taking the integer cube root of c recovers m directly — no factoring needed. RsaCtfTool handles this and 30 other RSA attack vectors automatically. When it fails, try --attack wiener next for small private exponents, then --attack fermat for primes that are close together.
Know your attack surface before you start: collect n, e, and c, check factordb first, then run RsaCtfTool with --attack all if you want a broad sweep. Most CTF RSA challenges fall within two minutes this way.
What To Do Now
Go to CryptoHack.org right now and complete the "General" introduction track. It covers byte manipulation, XOR, and base encodings in hands-on browser challenges — no setup required. Finish that track and you'll recognize 80% of beginner-to-intermediate CTF crypto patterns on sight. Then come back and hit the RSA section. One hour of deliberate practice beats three hours of Googling mid-competition.
