Paper page - HRM-Text: Efficient Pretraining Beyond Scaling

Log In<br>Sign Up

HRM-Text explores a different approach to language model pretraining: hierarchical recurrent computation, task-completion training, and latent-space reasoning.\nAt just 1B parameters, HRM-Text achieves competitive performance with dramatically lower training cost and data requirements.\n1B parameters40B unique tokens~1 day of pretraining~$1000 training cost\n","updatedAt":"2026-05-21T03:19:24.382Z","author":{"_id":"61b6cbbdbfb266841ec0f24a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61b6cbbdbfb266841ec0f24a/PHUVNOOMEw_R2CF3u-sMS.png","fullname":"One","name":"imone","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":58,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7599409818649292},"editors":["imone"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/61b6cbbdbfb266841ec0f24a/PHUVNOOMEw_R2CF3u-sMS.png"],"reactions":[{"reaction":"🔥","users":["diwank","maesaer","m-ric"],"count":3},{"reaction":"❤️","users":["diwank"],"count":1},{"reaction":"🚀","users":["diwank"],"count":1}],"isReport":false}},{"id":"6a15b40523c30dbbbdf7acaf","author":{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","fullname":"Urro","name":"urroxyz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false},"createdAt":"2026-05-26T14:53:57.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"One of the most exciting papers of the year.","html":"One of the most exciting papers of the year.\n","updatedAt":"2026-05-26T14:53:57.318Z","author":{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","fullname":"Urro","name":"urroxyz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9666399955749512},"editors":["urroxyz"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png"],"reactions":[],"isReport":false}},{"id":"6a26777f1546853bde1270cf","author":{"_id":"686241ae58095f461aa02a71","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kMC2nzkk3OWdKDk2QFocY.png","fullname":"Matthias Kronenberger","name":"chronos75","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false},"createdAt":"2026-06-08T08:04:15.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Looking forward to specific training using this, for example on classic literature\n","html":"Looking forward to specific training using this, for example on classic literature\n","updatedAt":"2026-06-08T08:04:15.996Z","author":{"_id":"686241ae58095f461aa02a71","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kMC2nzkk3OWdKDk2QFocY.png","fullname":"Matthias Kronenberger","name":"chronos75","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8403600454330444},"editors":["chronos75"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kMC2nzkk3OWdKDk2QFocY.png"],"reactions":[],"isReport":false}},{"id":"6a2c00c7f7f66fcaa0cb66d6","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":371,"isUserFollowing":false},"createdAt":"2026-06-12T12:51:19.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models](https://huggingface.co/papers/2605.11011) (2026)\n* [Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models](https://huggingface.co/papers/2605.07721) (2026)\n* [Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility](https://huggingface.co/papers/2605.14037) (2026)\n* [LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling](https://huggingface.co/papers/2606.04438) (2026)\n* [MiniGPT: Rebuilding GPT from First...