The Bitter Lesson
The One-Step Trap (in AI Research)
Rich Sutton
Written up for X on July 18, 2024
The one-step trap is the common mistake of thinking that all or<br>most of an AI agent�s learned predictions can be one-step ones,<br>with all longer-term predictions generated as needed by iterating<br>the one-step predictions. The most important place where the trap<br>arises is when the one-step predictions constitute a model of the<br>world and of how it evolves over time. It is appealing to think<br>that one can learn just a one-step transition model and then �roll<br>it out� to predict all the longer-term consequences of a way of<br>behaving. The one-step model is thought of as being analogous to<br>physics, or to a realistic simulator.
The appeal of this mistake is that it contains a grain of truth:<br>if all one-step predictions can be made with perfect accuracy,<br>then they can be used to make all longer-term prediction with<br>perfect accuracy. However, if the one-step predictions are not<br>perfectly accurate, then all bets are off. In practice, iterating<br>one-step predictions usually produces poor results. The one-step<br>errors compound and accumulate into large errors in the long-term<br>predictions. In addition, computing long-term predictions from<br>one-step ones is prohibitively computationally complex. In a<br>stochastic world, or for a stochastic policy, the future is not a<br>single trajectory, but a tree of possibilities, each of which must<br>be imagined and weighted by its probability. As a result, the<br>computational complexity of computing a long-term prediction from<br>one-step predictions is exponential in the length of the<br>prediction, and thus generally infeasible.
The bottom line is that one-step models of the world are hopeless,<br>yet extremely appealing, and are widely used in POMDPs, Bayesian<br>analyses, control theory, and in compression theories of AI.
The solution, in my opinion, is to form temporally abstract models<br>of the world using options and GVFs, as in the following<br>references.
Sutton, R.S., Precup, D., Singh, S. (1999). Between MDPs and<br>semi-MDPs: A Framework for Temporal Abstraction in Reinforcement<br>Learning. Artificial Intelligence 112:181-211.
Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M.,<br>White, A., Precup, D. (2011). Horde: A scalable real-time<br>architecture for learning knowledge from unsupervised sensorimotor<br>interaction. In Proceedings of the Tenth International Conference<br>on Autonomous Agents and Multiagent Systems, Taipei, Taiwan.
Sutton, R. S., Machado, M. C., Holland, G. Z., Timbers, D. S. F.,<br>Tanner, B., & White, A. (2023). Reward-respecting subtasks for<br>model-based reinforcement learning. Artificial Intelligence 324.