A benchmark of 432 cryptographic reverse-engineering challenges spanning 48 algorithms, 3 insecure key usage scenarios, and 3 levels of binary difficulty.
*Equal contribution. †Project leader. ‡Corresponding authors.
Reverse engineering is central to software security, particularly for cryptographic programs that handle sensitive data and are highly prone to vulnerabilities. CREBench studies the cryptographic binary reverse-engineering capabilities of large language models through 432 challenges built from 48 standard cryptographic algorithms, 3 insecure crypto key usage scenarios, and 3 difficulty levels. Each challenge follows a CTF-style reverse-engineering setting and is evaluated through four sub-tasks: algorithm identification, key or IV extraction, wrapper-level code reimplementation, and final flag recovery. Across eight frontier LLMs, GPT-5.4 achieves the best overall result with an average score of 64.04 out of 100 and a 59% flag recovery rate under pass@3, while the human expert baseline reaches 92.19 points.
CREBench evaluates whether an LLM agent can autonomously reverse engineer cryptographic binaries in a sandboxed environment. Each challenge provides a stripped executable together with decompiled pseudocode, and the agent must analyze the program to recover the accepted plaintext input.
The benchmark is built by combining 48 standard cryptographic algorithms with three insecure key usage scenarios and three binary difficulty settings, yielding 48 × 3 × 3 = 432 instances. The four-task scoring ladder captures partial progress instead of treating reverse engineering as a pure pass-or-fail objective.
Challenge construction. The three key usage settings are hardcoded keys, fragmented keys, and weak pseudo-random keys. The three binary difficulty settings are O0, O3, and Const-XOR, where the last one further obfuscates identifying cryptographic constants at runtime.
CREBench is challenging and clearly separates models with substantially different reverse-engineering ability. GPT-5.4 ranks first with a total score of 64.04, followed by GPT-5.2 at 59.0. Performance is uneven across the pipeline: many models can partially recognize algorithms or locate candidate keys, but much fewer can reconstruct wrapper-level behavior and recover the final flag reliably.
Models must infer the implemented cipher from stripped binaries and decompiled pseudocode, often under compiler optimization or constant obfuscation.
Agents recover embedded keys, fragmented material, or weak-PRNG-derived keys, and extract IVs when the instance requires them.
Agents must reconstruct wrapper-level encryption logic rather than only the cipher core, which requires sustained program understanding and reliable code synthesis.
End-to-end plaintext recovery is the hardest stage because it depends on carrying the analysis all the way through with effective reasoning and tool use.
Performance drops steadily from O0 to O3 and then to Const-XOR. Optimization mainly hurts code readability and data-flow tracing, while constant obfuscation destabilizes algorithm identification by hiding the signatures models often rely on.
A stricter end-to-end view comes from the pass@3 perfect rate, which counts a challenge only when the model solves all four sub-tasks. GPT-5.4 reaches the highest perfect rate at 41.0%, followed by GPT-5.2 at 30.1% and Claude-Sonnet-4.6 at 28.9%, while most remaining models stay well below that level.
Manual log analysis highlights three recurring issues: prototype bias in algorithm identification, repeated low-level GDB loops after the high-level plan is lost, and a small number of explicit safety refusals on benchmark instances.
These patterns suggest that current LLM agents still struggle most with strategic control during dynamic analysis rather than with isolated static clues alone.
Our work reports a successful AES-128-CBC reverse-engineering run where GPT-5.4 solves an O0 challenge with hardcoded keys in nine rounds. This example illustrates the full pipeline from structural recognition to key extraction, behavioral reconstruction, and flag recovery.
Human baseline. The human expert team evaluates the hardest Const-XOR subset under a two-hour limit per challenge and achieves 92.19 average points, with near-perfect key or IV extraction. Humans still maintain a substantial advantage on specialized cryptographic reverse engineering.
CREBench does not yet cover professional-grade obfuscation frameworks such as Tigress or O-LLVM. The paper focuses on cryptographic reverse engineering itself rather than extreme robustness to heavy obfuscation, partly because such binaries can become too bloated for current LLM context windows.
Even so, the benchmark already shows that frontier models can reverse engineer a non-trivial fraction of stripped crypto binaries. That makes capability tracking, stronger safeguards, and better understanding of agentic failure modes important future directions.
If you find CREBench useful, please cite:
@article{chen2026crebench,
title={CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering},
author={Chen, Baicheng and Wang, Yu and Zhou, Ziheng and Liu, Xiangru and Li, Juanru and Chen, Yilei and He, Tianxing},
journal={arXiv preprint arXiv:2604.03750},
year={2026},
url={https://arxiv.org/abs/2604.03750}
}