IT consultant Mark Pesce was building an LLM-based similarity finder for a legal client. He discovered a prompt that reliably caused multiple LLMs to go nuts and output complete gibberish: “it desc…
yeah find me one single instance of someone doing this “genuine question” shit that doesn’t result in the most bad faith interpretation possible of the answers they get
If I’m missing something obvious I’d love it if you told me.
most security vulnerabilities look like they cause the targeted program to spew gibberish, until they’re crafted into a more targeted attack
it’s likely that gibberish is the LLM’s training data, where companies are increasingly being encouraged to store sensitive data
there’s also a trivial resource exhaustion attack where you have one or more LLMs spew garbage until they’ve either exhausted their paid-for allocation of tokens or cost their hosting organization a relative fuckload of cash
either you knew all of the above already and just came here to be a shithead, or you’re the type of shithead who doesn’t know fuck about computer security but still likes to argue about it
yeah find me one single instance of someone doing this “genuine question” shit that doesn’t result in the most bad faith interpretation possible of the answers they get
the amount of times I’ve had to clean shit up after someone like this “didn’t think $x would matter”…