I still don't see the appeal of Auto Mode. If I don't trust the model to run arbitrary commands, how much additional trust can be gained by an additional non-deterministic system. The point is that there are some failure modes which are completely unacceptable. Auto Mode lowers that risk but doesn't remove it. An unacceptable failure is still possible.
once you have built up a trust in a model/harness/setup/workflow and you’ve been hitting enter for months straight then Auto Mode is appealing.
i found initially it would hang on some commands (moreso on subagents and moreso when first rolled out) and then roll past the hung command sometimes forgetting about it entirely. which wasn’t cool.
but after dialing in subagent permissions and probably some updates from anthropic Auto Mode is great. i still don’t always do it. but after reading this article just switched my current 12 panes to Auto.
Non yolo mode where you manually approve each command is literally pure security theatre. No human on earth has the patience to discerne the huge stream of commands agents run.
So even if false negatives are technically zero, having 100% false positives is not acceptable.
So a fundamental premise is you have to have filtration. Where only very few things are surfaced for humans to look
The first option which a decent number of harnesses do is being able to setup an auto approve list. Like `ls` is fine, `cat` is fine etc.
Now maintaining this list is in and of itself a giantic pita. But the real issue is that it still has way too many false positives. Fundamentally it comes down to the halting problem where you can't really include important things like `bash python <<PY` and what not which agents like to use. But regex can't solve the halting problem to figure out if the Python is safe.
So naturally the next best option is to use an LLM. Which isn't that stupid because even if its non deterministic at least it can dramatically reduce the false positives from the regex auto approve list.
I think saying "nobody has the patience" is assuming that everyone is using agents for hours on end to do enormous tasks. I tend to prompt claude to do smaller, more contained tasks that I understand the entirety of, in which case manual review is straightforward and actually preferred, because I can stop it before it makes a change I disagree with.
I would generally disagree with this unless it can be fine-user tuned (for the current session person allowance). Although in general; I believe this is letting ai take the wheel, which I 100% disagree with as more damage will likely be done without human oversight/retro-actions and even if caught would it be a humans fault?
Auto mode works really well with Safehouse [1]. I usually call Claude like this:
This way I will never fear Claude `rm -rf`-ing my home directory.
[1] https://agent-safehouse.dev/
I still don't see the appeal of Auto Mode. If I don't trust the model to run arbitrary commands, how much additional trust can be gained by an additional non-deterministic system. The point is that there are some failure modes which are completely unacceptable. Auto Mode lowers that risk but doesn't remove it. An unacceptable failure is still possible.
once you have built up a trust in a model/harness/setup/workflow and you’ve been hitting enter for months straight then Auto Mode is appealing.
i found initially it would hang on some commands (moreso on subagents and moreso when first rolled out) and then roll past the hung command sometimes forgetting about it entirely. which wasn’t cool.
but after dialing in subagent permissions and probably some updates from anthropic Auto Mode is great. i still don’t always do it. but after reading this article just switched my current 12 panes to Auto.
The appeal of auto mode is obvious.
Non yolo mode where you manually approve each command is literally pure security theatre. No human on earth has the patience to discerne the huge stream of commands agents run.
So even if false negatives are technically zero, having 100% false positives is not acceptable.
So a fundamental premise is you have to have filtration. Where only very few things are surfaced for humans to look
The first option which a decent number of harnesses do is being able to setup an auto approve list. Like `ls` is fine, `cat` is fine etc.
Now maintaining this list is in and of itself a giantic pita. But the real issue is that it still has way too many false positives. Fundamentally it comes down to the halting problem where you can't really include important things like `bash python <<PY` and what not which agents like to use. But regex can't solve the halting problem to figure out if the Python is safe.
So naturally the next best option is to use an LLM. Which isn't that stupid because even if its non deterministic at least it can dramatically reduce the false positives from the regex auto approve list.
I think saying "nobody has the patience" is assuming that everyone is using agents for hours on end to do enormous tasks. I tend to prompt claude to do smaller, more contained tasks that I understand the entirety of, in which case manual review is straightforward and actually preferred, because I can stop it before it makes a change I disagree with.
I would generally disagree with this unless it can be fine-user tuned (for the current session person allowance). Although in general; I believe this is letting ai take the wheel, which I 100% disagree with as more damage will likely be done without human oversight/retro-actions and even if caught would it be a humans fault?