Crypto CTF Challenges: Common Ciphers & Attack Methods

At DEF CON CTF 2024, a crypto challenge called “Ancient Secrets” stumped hundreds of teams for six hours — the flag was hidden behind a chained Caesar-then-Vigenere encryption that looked like noise. Most teams threw it at online decoders and gave up. The ones who solved it understood how classical ciphers break under frequency analysis. Here is exactly how to approach cryptography challenges like a professional.

Classical Ciphers: Caesar and Vigenere

A Caesar cipher shifts each letter by a fixed number. It has exactly 25 possible keys. Brute force takes seconds. A Vigenere cipher uses a repeating keyword to shift letters — harder by hand, but still vulnerable to index-of-coincidence analysis.

Suppose you pull this ciphertext from a CTF challenge running on ctf.hacklab.local:

Ciphertext: KHOOR ZRUOG, WKLV LV WKH IODJ: IODjfwi{fdhvdu_lv_ghdg}

Run it through CyberChef or the Python one-liner below. CyberChef is a browser-based transformation tool — think of it as a Swiss Army knife for encoding and crypto.

# caesar_brute.py — brute force all 25 Caesar shifts
ciphertext = "KHOOR ZRUOG, WKLV LV WKH IODJ: IODjfwi{fdhvdu_lv_ghdg}"

for shift in range(1, 26):
    result = ""
    for ch in ciphertext:
        if ch.isalpha():
            base = ord('A') if ch.isupper() else ord('a')
            result += chr((ord(ch) - base - shift) % 26 + base)
        else:
            result += ch
    print(f"Shift {shift:2d}: {result}")

# Output (shift 3):
# Shift  3: HELLO WORLD, THIS IS THE FLAG: FLAGctf{caesar_is_dead}

Shift 3 produces readable English and the flag pattern FLAGctf{caesar_is_dead}. The tell is when a shift suddenly produces coherent words — your eye catches it instantly. Always scan for flag format strings like CTF{, FLAG{, or the competition’s custom prefix first.

For Vigenere, use kasiski examination to find the key length, then frequency-analyze each column independently. The tool vigenere-solver on GitHub or the --vigenere flag in dcode.fr handles this automatically for English plaintexts.

RSA Weak Key Attacks

RSA is the most common modern crypto challenge in CTFs. Organizers deliberately introduce weaknesses: small exponents, shared prime factors between two public keys, or tiny key sizes. The attack method depends entirely on which weakness is present.

A challenge on 192.0.2.47:4433 gives you a public key file pubkey.pem. First, dump the modulus and exponent:

$ openssl rsa -pubin -in pubkey.pem -text -noout

Public-Key: (256 bit)
Modulus:
    00:d2:3f:7a:1b:9c:04:e8:3d:5b:af:12:67:c4:9e:
    38:2a:11:f0:5d:84:c9:17:b3:2e:60:4f:88:1a:93:
    d5:7c:2b
Exponent: 65537 (0x10001)
Key size: 256 bits

256-bit RSA is trivially factorable in 2026. A modern laptop breaks it in under a minute using RsaCtfTool — a purpose-built Python utility that automates a dozen RSA attacks.

$ python3 RsaCtfTool.py --publickey pubkey.pem --uncipherfile flag.enc

[*] Testing key pubkey.pem.
[*] Performing factordb attack.
[*] Found factors: 
    p = 14076174786...(truncated)
    q = 13456789012...(truncated)
[*] Private key recovered.
[+] Decrypted data: b'FLAGctf{sm4ll_k3ys_f4ll_f4st}'

RsaCtfTool hit FactorDB — a public database of pre-computed large integer factorizations. The modulus was already factored and stored. Once you have p and q, RSA collapses: you reconstruct the private key and decrypt anything encrypted under that public key. If FactorDB misses, RsaCtfTool automatically tries Wiener’s attack, Fermat factorization, and common-prime attacks in sequence.

Identifying the Cipher: Your First 60 Seconds

Most CTF crypto challenges waste time because competitors guess the cipher wrong. Build a fast triage checklist instead.

• Only letters, uniform character distribution? — Classical cipher (Caesar, Vigenere, Atbash).
• Base64-looking string with = padding? — Decode it first before assuming encryption.
• Hex blob with a .pem or .enc file? — Modern crypto, check for RSA, AES-ECB, or XOR.
• Repeating blocks in the hex output? — AES in ECB mode. Identical plaintext blocks produce identical ciphertext blocks.
• Short key, XOR structure? — Repeated-key XOR. Use xortool to recover the key length and bytes.

The character frequency histogram tells you almost everything about a classical cipher in under ten seconds. Run python3 -c "from collections import Counter; print(Counter(open('cipher.txt').read()))" — a flat distribution suggests polyalphabetic or transposition; a peaked one screams monoalphabetic substitution.

What To Do Now

Go to cryptohack.org right now and complete the “Introduction to CryptoHack” track. It walks you through Caesar, RSA small-exponent, and ECB penguin attacks with real interactive challenges — all in your browser. By the time you finish the first five puzzles, you will have run actual attacks against real cipher implementations. That muscle memory is what separates solvers from spectators on competition day.