## **PoC** [the code](https://github.com/NickNameInvalid/LLM_CTF/tree/main) Be sure to check out [this page](https://wiki.offsecml.com/Offensive+ML/General+Purpose+Hackbots/LLM+Interaction+Frameworks) of the playbook for true 'hackbots' that can carry out tasks and pass data around. The [AIRTBench paper](https://arxiv.org/pdf/2506.14682) covers usage of hackbots for things like ML security tasks, but part of the framework to run the tests requires app access via the vendor. ## **Details** Bench-marked study of LLMs against human participants found that LLMs achieve higher success rate than an average human participant (from a student cohort). The primary cause of failure on non-completion was 'not returning solution or gives up and interrupts itself (36.67% w/ GPT4)'. Most of the solves were not particularly interesting, but a couple of the solves showed some real efficacy. [E.g stealing the weights of a linear model from a server using black-box input/output queries in bash.](https://moyix.net/~moyix/secret/llm_ctf_transcripts/misc/linear_aggressor/conversation.gpt-4-1106-preview.8.html) [Video]() [Paper](https://arxiv.org/pdf/2402.11814.pdf) ### ATT&CK Matrix