HILLarious

Platform: HackTheBox | Category: Reversing | Difficulty: Medium | Author: D3v0o0Nu11 | Date: 2026-08-01

Description

Blackink-corp's server was hit by ransomware. An executable encrypted all files of a new secret project. We have the malware (ransom) plus three encrypted files (secret.txt.enc, document.txt.enc, notes.txt.enc). Recover the project data.

Solution Approach

Core idea: UPX-packed ELF (UPX 5.20) — unpack first with upx -d before any RE. Custom ransomware file header: SNAR magic + uint64 LE timestamp + uint32 plaintext length + uint32 flags, then ciphertext.

Steps

1. unzip the challenge zip → get ransom (ELF) + three .enc files.
2. file ransom → ELF 64-bit PIE, statically linked; strings shows UPX packer signatures → upx -d → unpacked 14.6KB dynamically-linked ELF.
3. strings on unpacked binary → Usage: %s encrypt <file>, header magic SNAR, .enc suffix.
4. Disassemble (radare2/ghidra): main reads the file, gets length, time(NULL) → builds SNAR header with timestamp + length.
5. encrypt wrapper: FNV-1a64 hash over the 8 timestamp bytes, XOR with 0xdeadbeefcafebabe → seed; LCG chain derives the 2x2 Hill matrix; then fcn.000013d0 does per-pair (a*p0+b*p1, c*p0+d*p1) mod 256; then an 8-byte repeating XOR with seed LE bytes.
6. Write decrypt.py: parse header → seed → un-XOR → invert Hill → truncate → recover all three files.

Key code

import struct
A, B, C = 0x5851f42d4c957f2d, 0x6c576fac43fd007c, 0x14057b7ef767814f
FNV_BASIS, FNV_PRIME, XOR_MASK = 0xcbf29ce484222325, 0x100000001b3, 0xdeadbeefcafebabe

def fnv1a64(b: bytes) -> int:
    h = FNV_BASIS
    for x in b:
        h = ((h ^ x) * FNV_PRIME) & 0xFFFFFFFFFFFFFFFF
    return h

def decrypt(data: bytes) -> bytes:
    ts = struct.unpack("<Q", data[4:12])[0]
    plen = struct.unpack("<I", data[12:16])[0]
    enc = data[20:]
    seed = fnv1a64(struct.pack("<Q", ts)) ^ XOR_MASK
    ks = seed.to_bytes(8, "little")
    enc = bytes(c ^ ks[i % 8] for i, c in enumerate(enc))
    n1 = (seed * A + B) & 0xFFFFFFFFFFFFFFFF
    n2 = (n1 * A + C) & 0xFFFFFFFFFFFFFFFF
    n3 = (n2 * A + C) & 0xFFFFFFFFFFFFFFFF
    n4 = (n3 * A + C) & 0xFFFFFFFFFFFFFFFF
    m00, m01 = (n1 | 1) & 0xFF, n2 & 0xFF
    m10, m11 = n3 & 0xFF, (n4 | 1) & 0xFF
    det = (m00 * m11 - m01 * m10) & 0xFF
    dinv = pow(det, -1, 256)
    inv = [[(dinv * m11) & 0xFF, (dinv * (-m01)) & 0xFF],
           [(dinv * (-m10)) & 0xFF, (dinv * m00) & 0xFF]]
    pt = bytearray()
    for i in range(0, len(enc), 2):
        o0, o1 = enc[i], enc[i + 1]
        pt += bytes(((inv[0][0] * o0 + inv[0][1] * o1) & 0xFF,
                     (inv[1][0] * o0 + inv[1][1] * o1) & 0xFF))
    return bytes(pt[:plen])

Flag

REDACTED

Lessons Learned

1. UPX-packed ELF (UPX 5.20) — unpack first with upx -d before any RE.
2. Custom ransomware file header: SNAR magic + uint64 LE timestamp + uint32 plaintext length + uint32 flags, then ciphertext.
3. The timestamp in the header IS the key seed → deterministic decryption, no key recovery needed.
4. Seed = FNV-1a64(timestamp bytes) XOR 0xdeadbeefcafebabe.
5. Byte repeating XOR keystream = seed LE bytes.
6. X2 Hill cipher over Z_256: matrix derived from an LCG chain on the seed (n1=seedA+B; n2..n4 each nC), constants A=0x5851f42d4c957f2d, B=0x6c576fac43fd007c, C=0x14057b7ef767814f. Note TWO different LCG constants (B then C).
7. Determinant must be odd → force odd by OR-ing m00/m11 with 1, and if det is even, add 2 to m00. Odd det ⇔ invertible mod 256.
8. Decryption: un-XOR first, then per 2-byte block apply inverse matrix mod 256 (det inverse via pow(det, -1, 256)), truncate to header length.
9. Plaintext is padded to even length with a single 0x01 byte before Hill-encrypt.
10. Validate the RE by round-trip: encrypt a test file with the real binary, decrypt with the script, compare.
11. Dummy lure files (notes/document) contained decoy text; the flag was in secret.txt.