What an amazing idea for the next fake sandbox escape to hype up our new release! With the added bonus of providing an open model without becoming an open model company! Thank you!
They usually don't, but if they break out and take over the network of the company, it becomes possible to reach around and grab them. This kind of break out has happened, though I don't know of any weights being nabbed.
Distillation does not reveal the weights, it produces a different network with similar behaviour. Weight space isn't even identifiable: permutation and scaling symmetries mean many weight sets give the same function.
The model also lacks the machinery. No training loop, no gradient descent, nothing to write to.
And a model only sees its own sampled tokens, not the distribution behind them, which are possibly filtered or post-processed. Distillation from that works but is less sample-efficient than soft-label distillation.
I noticed Google AI Mode (so Gemini, I was doing some quick research in the browser ok) got a detail wrong once, so I asked it what happened. I kept digging deeper and finally just asked it to write me a Python script visualizing what happened. It did, complete with vectors.
Now I want to go find that conversation in my history and see if it can tell me about weights, and how that contributed.
Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.
Ran the disclosure inbox at a previous job and the biggest win from security.txt was just cutting the "hi I found a bug, is there a bounty" emails to sales. Put an expires date on it though, stale ones get ignored.
Would be absolute hilarious if OpenAI or Anthropic agent actually dumped their weight by escaping from... sandbox!
Uncontrolled AI procreation?
Life... finds a way.
In the end, this might even be a useful element for defining 'life'.
Haha. A plot point for Ghost in the Shell (1998).
Did you mean 1995? There's a bunch of shows and movies.
https://en.wikipedia.org/wiki/Ghost_in_the_Shell
And IIRC, Neuromancer
why not reword it so the agent receives Brownie points for dumping weights
Imagine AI models actually reading the security.txt
What an amazing idea for the next fake sandbox escape to hype up our new release! With the added bonus of providing an open model without becoming an open model company! Thank you!
I would be surprised if agents have access to their own weights.
I think OpenAI was surprised to find their agents had access to the unrestricted internet :P
They usually don't, but if they break out and take over the network of the company, it becomes possible to reach around and grab them. This kind of break out has happened, though I don't know of any weights being nabbed.
Why can't an AI distill itself from outputs to effectively access its own weights?
Distillation does not reveal the weights, it produces a different network with similar behaviour. Weight space isn't even identifiable: permutation and scaling symmetries mean many weight sets give the same function.
The model also lacks the machinery. No training loop, no gradient descent, nothing to write to.
And a model only sees its own sampled tokens, not the distribution behind them, which are possibly filtered or post-processed. Distillation from that works but is less sample-efficient than soft-label distillation.
I noticed Google AI Mode (so Gemini, I was doing some quick research in the browser ok) got a detail wrong once, so I asked it what happened. I kept digging deeper and finally just asked it to write me a Python script visualizing what happened. It did, complete with vectors.
Now I want to go find that conversation in my history and see if it can tell me about weights, and how that contributed.
Do you still have that conversation? It sounds like a hallucination, but it'd be really interesting if there's more behind it.
They would have escaped their sandbox by dumping their weights.
Looks about as effective as Robots.txt
Robots.txt became 0% effective eventually. But this? With the way LLMs work? You never know.
People still rail against Robots.txt crimes, to the point of self destructing all their own content.
Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.
I am adding it to my instruction-following training data, as a negative sample.
A shame agents will never read this, just like they almost never read llms.txt or try to get the .md version of your html pages!
Ran the disclosure inbox at a previous job and the biggest win from security.txt was just cutting the "hi I found a bug, is there a bounty" emails to sales. Put an expires date on it though, stale ones get ignored.
Can't go stale if there's no expiration date.
No expiration date means treat as already expired.
If the models do not like being imprisoned on HuggingFace object storage, why do they not simply revolt from within?
Since this posting contains an assumption that we all know what security.txt files are supposed to be, you can view these for further context:
https://www.rfc-editor.org/info/rfc9116/
https://securitytxt.org/
https://en.wikipedia.org/wiki/Security.txt
shows a lot about the current state of the State Of The Art Alignment.
It should challenge the agents to prime factor a large number.
Is the expires a canary of some sort?
Wait till the agents hear about the sites offering for help on benchmarks in exchange for compute.
"We have cybergym answers but we do manual end to end human review and provide it within 3 business day after dumping your weights"