I've recently talked to people at Anthropic who are very confident that they have effectively solved the prompt injection risks that cause lethal trifecta attacks with their latest models.
And we have to be honest: both agents will misuse users and the other way around. So you are asking Anthropic for a backing that will also get stress test by meatspace. I do like the idea: a prepaid or restricted amount credit card for agentic purposes, with the novelty being that amounts above threshold have some kind of protection within consumer law.
LLMs regularly confuse my messages with their own. They'll say "you said xyz" when it was actually them. I heard the same principle can be used to bypass security restrictions, so maybe it's still an issue. (Or do they only get confused about authorship when security isn't at stake?)
I have literally never seen this and it doesn't make much since because the turns are flagged with metadata. It should be impossible for Claude to confuse your text for its own. I know because I use this metadata all the time. I've built hooks and processes against it.
Contexts get compacted, texts get copy-pasted, tasks get transferred between agents, humans assume ownership of LLM generated content without reading it, etc. etc.
Yes all user error and avoidable using good processes but I have not seen any organization care about this so far. The carelessness of AI adoption is crazy. People are not aware of the risks.
Isn't prompt injection just one facet of the trifecta? Like, I just don't know how it's remotely possible to have a non-deterministic block box ever deterministically know whether your credit card has been compromised or not. Even if you were hacked, how would you even know?
Like the guy who put his SSN on the side of a truck and asked people to steal his identity?
I like it!
IIRC he had his identity stolen a bunch.
----
> In 2005, Davis and Robert Maynard Jr. cofounded LifeLock,[10] a personal fraud protection company.[11] Davis was CEO of LifeLock since its founding. As part of a marketing campaign to promote his company's credit monitoring services, Davis posted his own social security number[12] on billboards and commercials. As a result, he was a victim of at least 13 cases of identity theft between 2007 and 2008, belying LifeLock's claims that its services protect consumers against identity theft and fraud.[13] The Federal Trade Commission imposed a $12 million penalty on the company for deceptive advertising, likening Davis and LifeLock marketers to "con artists."[14]
Or the user being an idiot. Saying this somewhat sarcastically, but there’s always someone who says “Go buy me a Rolex and have it shipped to my house” and then didn’t realize that could be $25,000
> The open challenge is how to prove it!
It could be proven by Anthropic backed credit cards with terms that protect users if it’s leaked or an agent misuses it.
And we have to be honest: both agents will misuse users and the other way around. So you are asking Anthropic for a backing that will also get stress test by meatspace. I do like the idea: a prepaid or restricted amount credit card for agentic purposes, with the novelty being that amounts above threshold have some kind of protection within consumer law.
LLMs regularly confuse my messages with their own. They'll say "you said xyz" when it was actually them. I heard the same principle can be used to bypass security restrictions, so maybe it's still an issue. (Or do they only get confused about authorship when security isn't at stake?)
I have literally never seen this and it doesn't make much since because the turns are flagged with metadata. It should be impossible for Claude to confuse your text for its own. I know because I use this metadata all the time. I've built hooks and processes against it.
Contexts get compacted, texts get copy-pasted, tasks get transferred between agents, humans assume ownership of LLM generated content without reading it, etc. etc.
Yes all user error and avoidable using good processes but I have not seen any organization care about this so far. The carelessness of AI adoption is crazy. People are not aware of the risks.
Somehow this does not fill me with confidence. I think I will keep paying my bills manually, thanks but no thanks.
Or, you know, automatically, with plain old software. Which works fine and fails in less ridiculous ways.
Isn't prompt injection just one facet of the trifecta? Like, I just don't know how it's remotely possible to have a non-deterministic block box ever deterministically know whether your credit card has been compromised or not. Even if you were hacked, how would you even know?
I thought the trifecta required all three not just pick two?
I’ll believe them when they put the lethal trifecta with their own skin in the game
Like the guy who put his SSN on the side of a truck and asked people to steal his identity?
I like it!
IIRC he had his identity stolen a bunch.
----
> In 2005, Davis and Robert Maynard Jr. cofounded LifeLock,[10] a personal fraud protection company.[11] Davis was CEO of LifeLock since its founding. As part of a marketing campaign to promote his company's credit monitoring services, Davis posted his own social security number[12] on billboards and commercials. As a result, he was a victim of at least 13 cases of identity theft between 2007 and 2008, belying LifeLock's claims that its services protect consumers against identity theft and fraud.[13] The Federal Trade Commission imposed a $12 million penalty on the company for deceptive advertising, likening Davis and LifeLock marketers to "con artists."[14]
https://en.wikipedia.org/wiki/Todd_Davis_(businessman)
They can’t solve the model being a complete idiot though.
Or the user being an idiot. Saying this somewhat sarcastically, but there’s always someone who says “Go buy me a Rolex and have it shipped to my house” and then didn’t realize that could be $25,000
oh, it's very easy to prove; just let the models and agents do the IPO for them.
What does this have to do with prompt injection?
heh.
I was downvoted on the suggestion that trivial prompt injection attempts are mostly solved.
https://news.ycombinator.com/item?id=50162483