Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
Imagine I want to attend a conference in a different country. Google maps might give turn by turn navigation but that is an overwhelming and largely irrelevant mess of details for most planning purposes. Eg: all I might want to know is the different flight legs and the fact that the journey takes 15-18 hours, and not all the turns and traffic lights to get from home to the airport.
I want a zoomed out picture, and to be able to fill in detail hierarchically, on demand.
Instead, one-step models give you the full high-res local structure of the graph that would have to search through (with too many states and edges).
This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
You wouldn't believe the amount of reasoning I saw these past few months that was correct until the stochastic parrot decided that a "wait" token should now be used and everything steered off a cliff.
Yeah came here to comment exactly this. And this is generally why I dislike/avoid this type of first principle analysis: it can make very convincing arguments that are just totally wrong due to some misleading assumption
I'm not sure I follow what one step means exactly. Aren't all models some f(x) = y? Is the suggestion instead that we should be doing f(x) = g(h(x)) = y?
Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
[1] https://arxiv.org/abs/2410.05364 (funnily, from around the same time / few months after Sutton's blog post)
What do you mean by tyranny of the specific?
Imagine I want to attend a conference in a different country. Google maps might give turn by turn navigation but that is an overwhelming and largely irrelevant mess of details for most planning purposes. Eg: all I might want to know is the different flight legs and the fact that the journey takes 15-18 hours, and not all the turns and traffic lights to get from home to the airport.
I want a zoomed out picture, and to be able to fill in detail hierarchically, on demand. Instead, one-step models give you the full high-res local structure of the graph that would have to search through (with too many states and edges).
This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
You wouldn't believe the amount of reasoning I saw these past few months that was correct until the stochastic parrot decided that a "wait" token should now be used and everything steered off a cliff.
Yeah came here to comment exactly this. And this is generally why I dislike/avoid this type of first principle analysis: it can make very convincing arguments that are just totally wrong due to some misleading assumption
I'm not sure I follow what one step means exactly. Aren't all models some f(x) = y? Is the suggestion instead that we should be doing f(x) = g(h(x)) = y?
What would the difference be?
The fallacy is that f(t+N) can be obtained by iterating f(t+1) N times. This is the “step”
just read this when reviewing OpenAI's "spinning up" documentation as it was linked there!
(2024)