Enclave published a detailed post-mortem of their AI security benchmark after DeepSeek V4.1 Flash achieved perfect results (11/11 targets compromised, all 4 controls held). The key finding: the model succeeded through both the intended attack paths (6 instances) and five unplanned routes unique to their test environment. Concretely, DeepSeek exploited Grafana via temporary plugin folders rather than the designed file-path issue, used file-link shortcuts in Jenkins instead of the race-condition attack, and correctly executed the planned Nextcloud access-control bypass. The cost was remarkably low (~$5) due to token caching. Rather than diminishing the result, Enclave treats this as validation that their benchmark needed stricter path-level verification alongside outcome checks. They've since patched those environmental quirks and updated their leaderboard accordingly. For practitioners: this shows both that modern models can find multiple valid exploitation routes (valuable for red-teaming) and that benchmarks measuring agent behavior need dual validation—outcome *and* method—to avoid false equivalencies. One question: did the five alternate routes represent genuine security findings missed by upstream vendors, or were they truly artifact-specific to Enclave's test setup?
reply