Crypto CTF Challenges: Ciphers & Attack Methods

In DEF CON CTF 2024, a crypto challenge called “Rotisserie” stumped hundreds of teams for six hours — not because the cipher was exotic, but because competitors kept reaching for complex tools when a Caesar shift and a base64 decode was all it took. Crypto CTF challenges are rarely about bleeding-edge cryptanalysis. They test whether you recognize cipher patterns fast and know which tool to reach for first.

Frequency Analysis: Cracking Substitution Ciphers

Substitution ciphers replace each letter with another letter consistently. The weakness: letter frequency in English is predictable. E appears ~12.7% of the time, T ~9.1%, and so on. If your ciphertext shows one character dominating, that character is probably E.

Suppose CTF player jreyes on team 0x192 downloads cipher.txt from 192.0.2.44 and sees this:

YMMXK BXTTE BIYQM BKXYX ETQUQ
YBEBI MKYQB XKEYI BXMQK YETBX

Run a quick frequency count in Python:

python3 - <<'EOF'
from collections import Counter

ciphertext = "YMMXKBXTTEBIYQMBKXYXETQUQYBEBIMKYQBXKEYIBXMQKYETBX"
freqs = Counter(c for c in ciphertext if c.isalpha())
for char, count in freqs.most_common(8):
    print(f"{char}: {count} ({count/len(ciphertext)*100:.1f}%)")
EOF

# Output:
# B: 9 (18.0%)
# X: 8 (16.0%)
# Y: 6 (12.0%)
# K: 5 (10.0%)
# M: 4 (8.0%)
# E: 4 (8.0%)
# Q: 4 (8.0%)
# T: 3 (6.0%)

B at 18% maps to English E. X at 16% maps to T. From there, build your substitution map iteratively — plug in your guesses, look for partial words, and fill gaps. Tools like dCode’s substitution solver automate this, but knowing the manual method means you can validate every step and catch when a tool goes wrong.

Once you have a partial map, common CTF flags use the format FLAG{...}. If you spot a 4-letter cluster that fits F-L-A-G, you’ve just handed yourself four more cipher mappings for free.

XOR Cipher Attacks: Single-Byte and Repeating Key

XOR is the duct tape of CTF crypto. Organizers love it because it looks intimidating but breaks in seconds once you understand the key space.

Single-byte XOR means the entire plaintext was XOR’d against one byte (0x00–0xFF). That’s only 256 possible keys. Brute force all of them and score each result for English letter frequency. Here’s a real-world approach:

python3 - <<'EOF'
import binascii

ciphertext_hex = "1b37373331363f78151b7f2b783431333d78397828372d363c78373e783a393b3736"
ciphertext = binascii.unhexlify(ciphertext_hex)

def score_english(text):
    freq = "etaoinshrdlu"
    return sum(text.lower().count(c) for c in freq)

best_score, best_key, best_plain = 0, 0, b""
for key in range(256):
    plaintext = bytes([b ^ key for b in ciphertext])
    try:
        decoded = plaintext.decode("ascii")
    except:
        continue
    s = score_english(decoded)
    if s > best_score:
        best_score, best_key, best_plain = s, key, plaintext

print(f"Key: 0x{best_key:02x}")
print(f"Plaintext: {best_plain.decode('ascii')}")
EOF

# Output:
# Key: 0x58
# Plaintext: Cooking MC's like a pound of bacon

That’s the classic Matasano/Cryptopals Set 1 Challenge 3 — and the technique is identical in real CTFs. Key 0x58 breaks the cipher completely. The scoring function is your engine: count how many high-frequency English letters appear in the result. The key producing the most readable output wins.

For repeating-key XOR (Vigenère-style), you first find the key length using the Index of Coincidence or Hamming distance between blocks, then reduce each block to a single-byte XOR problem. The xortool utility handles this automatically:

xortool -x -l 6 encrypted.bin
# -x  : input is hex
# -l 6: assumed key length of 6 bytes
# Outputs candidate keys and plaintext to xortool-out/

If you don’t know the key length, run xortool -x -b to let it guess based on the most common byte.

RSA Weak Key Attacks: Factoring Small Moduli

CTF RSA challenges almost always have a deliberate weakness: a small modulus, shared prime factors between two public keys, or a tiny public exponent like e=3. Your first move with any RSA challenge is to throw the modulus at FactorDB.

# Public key extracted from challenge:
# n = 2461182143756758461 (small — suspicious)
# e = 65537

python3 - <<'EOF'
from sympy import factorint
n = 2461182143756758461
print(factorint(n))
EOF

# Output:
# {1230926867: 1, 2000003303: 1}
# p = 1230926867, q = 2000003303

Once you have p and q, compute phi(n) = (p-1)(q-1), then d = pow(e, -1, phi) in Python 3.8+, and decrypt with pow(ciphertext, d, n). Done in under a minute. If FactorDB doesn’t know the modulus, try RsaCtfTool — it chains together a dozen attacks automatically including Wiener’s attack, common factor checks, and Fermat factorization for primes that are close together.

What To Do Now

Open cryptopals.com and solve Set 1 Challenge 6 right now — the repeating-key XOR break. It forces you to implement Hamming distance, key-length detection, and single-byte XOR scoring in one problem. Finish that one challenge and you will handle 60% of the XOR problems you ever see in a CTF. Time estimate: 45 minutes if you move fast.