Defensive Security Episode 316 is available for all today!
@jerry@infosec.exchange and I cover the recent Sharepoint 0-day, and speculation details leaked from Microsoft, Google Gemini going rogue and deleting files before admitting it's utter failure, yet another open source repo supply chain attack, the Clorox vs Cognizant lawsuit over a helpdesk social engineering attack, and the hack of Allianz Life.
Available on all your favorite podcast platforms, Youtube, and here: https://defensivesecurity.org/defensive-security-podcast-episode-316/
@lerg@infosec.exchange @jerry@infosec.exchange re: the LLM deleted files
There are many ways an LLM might “understand” a prompt. We know it isn’t repeatable in any exact way. We know there is some amount of randomness involved.
The LLM might assess that the non existent target folder evaluates to empty, so the operation becomes “move the files to nothing” 🤐
@GuillaumeRossolini@infosec.exchange @lerg@infosec.exchange true enough. In this instance, I think the issue stemmed from the fact that it shows an invalid command to create the directory. The person asking for help didn’t think to ask it to ensure the directory creation completed successfully before moving the files, and therefore it went off the rails. I think it’s pretty easy to replicate things like this, I’ve had ChatGPT help me with plenty of shell scripts and unless I’m really clear on everything it needs to do. I’ll end up with a lot of unhandled failure cases where I think we have a problem is that people see that it knows how to do 80% of the work 80% of the time and they make an assumption that it really knows how to do 100% of the work all of the time, and because of their own ignorance (not in the pejorative sense) of the subject matter, they don’t catch the problem. I think the exact error as a point out, won’t be very repeatable, but I do think that many different type are quite repeatable conditioned on the right level of ambiguity in the prompt. At least that’s my observation so far.
@jerry@infosec.exchange @lerg@infosec.exchange also, assuming this is bash or equivalent, these shells tend not to default to set -e if memory serves? Meaning “exit as soon as you get a failed command”: this isn’t the default.
Another flag I discovered recently is set -o pipefail which has the same effect for piped commands, because that isn’t covered by -e for some reason.
So a failed directory creation would let the remainder of the script run, like you said, without extra checks (or these flags)
@jerry@infosec.exchange @lerg@infosec.exchange still another failing of these tools, they aren’t really absorbing the boring tasks (the quality checks) that were hidden in the prompt
Andrew Kalat
