Virtually Mad

Platform: HackTheBox | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: NA | Author: D3v0o0Nu11 | Date: 2026-02-28 | Status: Solved Techniques: custom_vm_reverse_engineering, opcode_format_analysis, bitfield_extraction, register_state_constraint_solving, dispatch_loop_analysis

Summary

Given a stripped ELF 64-bit PIE executable (virtually.mad) that implements a custom virtual machine. The program takes a hex string as input, interprets it as VM opcodes, executes them, and checks the final register state.

Recon

Port scan

nmap -p- -sV -sC <TARGET> --min-rate 1000 -Pn

Enumeration highlights

• Event: HackTheBox | ID: 20260228_hackthebox_virtuallymad
• Tags: elf, x86_64, custom_vm, stripped_binary, opcode_encoding, bitfield, register_machine
• Indicators: custom VM with dispatch loop, opcodes as hex string input, register state comparison after execution, bitfield-encoded instructions, Executing %d opcodes
• Source: 20260228_hackthebox_virtuallymad.md

Foothold

Vulnerability / Misconfiguration

1. Custom_vm_reverse_engineering
2. Opcode_format_analysis
3. Bitfield_extraction
4. Register_state_constraint_solving
5. Dispatch_loop_analysis

<command>

Exploitation

• See original writeup content for detailed exploitation.

Privilege Escalation

Enumeration

sudo -l
find / -perm -4000 2>/dev/null
getcap -r / 2>/dev/null
cat /etc/crontab
ps aux

Exploitation

1. N/A for challenge-type writeup; see exploitation above.
2. Flag obtained via challenge solve.

<command>

Flags

Key Takeaways / Lessons

• custom_vm_reverse_engineering
• opcode_format_analysis
• bitfield_extraction
• register_state_constraint_solving
• dispatch_loop_analysis
• Tags: elf, x86_64, custom_vm, stripped_binary, opcode_encoding, bitfield, register_machine

Original Writeup

<details><summary>Click to expand original content</summary>

Description

Your friend loves to make pretty odd programs. This time you are given a special "machine" and you have to crack the correct code.

Given a stripped ELF 64-bit PIE executable (virtually.mad) that implements a custom virtual machine. The program takes a hex string as input, interprets it as VM opcodes, executes them, and checks the final register state.

Analysis

Initial Reconnaissance

$ file virtually.mad
virtually.mad: ELF 64-bit LSB pie executable, x86-64, stripped

$ strings virtually.mad
Give me code to execute:
Invalid code.
Executing %d opcodes.
Skipping opcode #%d, value too high (0x%x).
This is the right answer! Validate the challenge with HTB{%s}

Key strings immediately indicate:

• Input of "code" for execution
• Opcode validation (checking for values that are too high)
• Answer verification with flag output

VM Architecture

Decompilation in Ghidra revealed a custom virtual machine with the following architecture:

Registers:

• 4 general-purpose registers: a (0), b (1), c (2), d (3) — 32-bit each
• Flags register flags — set by the CMP instruction

Instruction Set (5 instructions):

Opcode Format

Each opcode is a 32-bit value, entered as an 8-character hex string. Bit layout:

31-24  23-20  19-16  15-12  11-0
 inst   type   dst    mode   operand

Input Validation

• Input string is parsed as a sequence of 8-character hex blocks
• Each block is converted to a 32-bit number
• Lower 12 bits of each opcode must be ≤ 0x100
• Exactly 5 opcodes (40 hex characters input)

Required Final State

After executing all 5 opcodes, the VM checks the registers:

a     = 0x200
b     = 0xFFFFFFFF  (-1 in signed 32-bit)
c     = 0xFFFFFFFF  (-1 in signed 32-bit)
d     = 0x0
flags = 0x10000000

If all values match — the flag is output (the input code itself).

Solution

Constructing Opcodes

Initial state: all registers = 0, flags = 0.

Need to bring registers to the required state in 5 instructions:

Opcode 1: 02100100 — ADD a, 0x100

inst=0x02 (ADD), type=1, dst=0 (a), mode=0 (imm), operand=0x100
a = 0 + 0x100 = 0x100

Opcode 2: 02100100 — ADD a, 0x100

inst=0x02 (ADD), type=1, dst=0 (a), mode=0 (imm), operand=0x100
a = 0x100 + 0x100 = 0x200 ✓

Opcode 3: 03110001 — SUB b, 1

inst=0x03 (SUB), type=1, dst=1 (b), mode=0 (imm), operand=0x001
b = 0 - 1 = 0xFFFFFFFF ✓

Opcode 4: 01121100 — MOV c, reg_b

inst=0x01 (MOV), type=1, dst=2 (c), mode=1 (reg), operand=0x100 (src_reg = bits 11-8 = 1 = b)
c = b = 0xFFFFFFFF ✓

Opcode 5: 04130000 — CMP d, 0

inst=0x04 (CMP), type=1, dst=3 (d), mode=0 (imm), operand=0x000
d == 0 → equal → flags = 0x10000000 ✓

Why ADD Twice Instead of MOV 0x200?

Constraint: lower 12 bits (operand) ≤ 0x100. The value 0x200 doesn't fit in a 12-bit immediate, so two additions of 0x100 are needed.

Verification

$ echo "REDACTED" | ./virtually.mad
Give me code to execute: Executing 5 opcodes.
=====
a: 0x200
b: 0xffffffff
c: 0xffffffff
d: 0x0
flags: 0x10000000
=====
This is the right answer! Validate the challenge with HTB{REDACTED}

</details>

---

Auto-tracked: saved to WriteUps; run /xesor-revise to fold lessons into XESXor_Methodology.md.

signed by XESXOR