BYUCTF RSA Dreams Cybersecurity Writeup 2026

Let’s solve this RSA cryptography challenge together!

Challenge Analysis:

We are given the output file and code:

c = 5074616349947930347771128443869249667723941019037011379843932659330729580197593522845748676276068149443504037403162962894625104017380398816152166168830833
n = 6452268004013779272669102227661703532150635430524568657091997086066784917218113937677647594597481724133073200003104968955173212323278046973034541033497147
e = 65537
hint = 161697499284577475400347684012866511237569864647822807778480533925514941939388

from Crypto.Util.number import getPrime
from Crypto.Util.number import bytes_to_long

flag = b"REDACTED"

p = getPrime(256)
q = getPrime(256)
n = p*q
hint = p + q
e = 0x10001

c = pow(bytes_to_long(flag), e, n)

print(f"c = {c}")
print(f"n = {n}")
print(f"e = {e}")
print(f"hint = {hint}")

The variable “hint” is composed of p and q being added. Since this is RSA, we also know that we need p and q to be multiplied in order to get the value of n. Luckily, we have the value of n; so we can plug these numbers into an equation to find p and q.

But before we get to the decryption, let’s understand how RSA works.

RSA Encryption:

RSA is an asymmetric encryption where two keys are used (a private and public key). It is secure because RSA uses the mathematical difficulty of factoring very big problems.

Key Terms:

• Prime numbers: Numbers that can be divided by itself and one.
• Modulo (mod): It finds the remainder after dividing it one number by another.
• Euler’s Totient Function(ϕ): If n is the product of two primes (p and q), then ϕ(n)=(p−1)(q−1). For instance, ϕ(91) = (7−1)(13−1) = (6)(12) = 72.
• Coprime: Two numbers are coprime if their GCD is 1. For instance, 8 and 15 are coprime. Factors of 8: 1,2,4,8. Factors of 15: 1,3,5,15. The greatest common factor they both share is 1.
• Modular Inverse: Given numbers a and m, the modular inverse of a is a number b such that: (a×b) mod m=1. For instance: The modular inverse of 3 mod 11 is 4, since (3×4) mod 11 = 12 mod 11 = 1.

RSA is encrypted by:

p = 61 
q = 53

// Now we will multiply the primes
n = p x q
n = 61×53 = 3233

// Now we will calculate Euler's Totient Function
ϕ(n) = (p−1)(q−1)
ϕ(3233)=(61−1)(53−1)=60×52=3120

// Now we have to find a public exponent 
// "e", in such that  1 < e < φ(n), is coprime with ϕ(n), are 
// commonly chosen as a small prime (like 3, 5, 17, 257, 65537):
e = 17

// Now we will calculate the private exponent d
d × e = 1(modϕ(n))
d × 17 = 1(mod3120)  /*3120 found from step 3  */

d = 2753

// Lastly, we will encrypt this by doing: 
C = plaintext^e  (mod n)

Now we understand how RSA works, so let’s solve this challenge!

Decryption:

In order to decrypt this RSA, we need to find the values of p and q.

We know:

 n = p * q 
 and
 hint = p + q

So we will do the equation: x² + hintx + n. The positive and negative values of x resemble the values of p and q.

Now, we can plug the values into the code shown below.

# RSA.py
from Crypto.Util.number import inverse, long_to_bytes

n = 6452268004013779272669102227661703532150635430524568657091997086066784917218113937677647594597481724133073200003104968955173212323278046973034541033497147
e = 65537
c = 5074616349947930347771128443869249667723941019037011379843932659330729580197593522845748676276068149443504037403162962894625104017380398816152166168830833

p = 71669843677648724047578634482796162927151697026318521175318409958080978797761
q = 90027655606928751352769049530070348310418167621504286603162123967433963141627

phi = (p - 1) * (q - 1)
d = inverse(e, phi)

# To decrypt: c^d mod n = m
m = pow(c, d, n)
print(long_to_bytes(m))

We got the flag 🥳: byuctf{great_job_recovering_the_flag}

If you want to learn more about cybersecurity, check out one of my articles about Google Dorking! https://medium.com/bugbountywriteup/12-must-know-google-dorking-commands-in-2026-9c8538c313c9

BYUCTF RSA Dreams Cybersecurity Writeup 2026 was originally published in InfoSec Write-ups on Medium, where people are continuing the conversation by highlighting and responding to this story.