[2402.17152] Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Skip to main content

Search arXiv

Press Enter to search · Advanced search

-->

Computer Science > Machine Learning

arXiv:2402.17152 (cs)

[Submitted on 27 Feb 2024 (v1), last revised 6 May 2024 (this version, v3)]

Title:Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Authors:Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, Yu Shi<br>View a PDF of the paper titled Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations, by Jiaqi Zhai and 11 other authors

View PDF<br>HTML (experimental)

Abstract:Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute.

Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data.

HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

Comments:<br>26 pages, 13 figures. ICML'24. Code available at this https URL

Subjects:

Machine Learning (cs.LG); Information Retrieval (cs.IR)

Cite as:<br>arXiv:2402.17152 [cs.LG]

(or<br>arXiv:2402.17152v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2402.17152

Focus to learn more

arXiv-issued DOI via DataCite

Submission history<br>From: Jiaqi Zhai [view email]<br>[v1]<br>Tue, 27 Feb 2024 02:37:37 UTC (1,966 KB)

[v2]<br>Thu, 18 Apr 2024 03:38:55 UTC (1,768 KB)

[v3]<br>Mon, 6 May 2024 02:05:45 UTC (1,765 KB)

Full-text links:<br>Access Paper:

View a PDF of the paper titled Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations, by Jiaqi Zhai and 11 other authors<br>View PDF<br>HTML (experimental)<br>TeX Source

view license

Current browse context:

cs.LG

next >

new<br>recent<br>| 2024-02

Change to browse by:

cs<br>cs.IR

References & Citations

NASA ADS<br>Google Scholar

Semantic Scholar

export BibTeX citation<br>Loading...

BibTeX formatted citation

&times;

loading...

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender<br>(What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn...