[2209.01188] Petals: Collaborative Inference and Fine-tuning of Large Models

Skip to main content

arXiv is now an independent nonprofit!<br>Learn more<br>&times;

Search arXiv

Press Enter to search &middot; Advanced search

-->

Computer Science > Machine Learning

arXiv:2209.01188 (cs)

[Submitted on 2 Sep 2022 (v1), last revised 2 Mar 2023 (this version, v2)]

Title:Petals: Collaborative Inference and Fine-tuning of Large Models

Authors:Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel<br>View a PDF of the paper titled Petals: Collaborative Inference and Fine-tuning of Large Models, by Alexander Borzunov and 7 other authors

View PDF

Abstract:Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.

Comments:<br>10 pages, 4 figures. The version 2 updates the benchmarks and the description of the chat application. Source code and docs: this https URL

Subjects:

Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:<br>arXiv:2209.01188 [cs.LG]

(or<br>arXiv:2209.01188v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2209.01188

Focus to learn more

arXiv-issued DOI via DataCite

Submission history<br>From: Alexander Borzunov [view email]<br>[v1]<br>Fri, 2 Sep 2022 17:38:03 UTC (1,727 KB)

[v2]<br>Thu, 2 Mar 2023 19:33:31 UTC (9,066 KB)

Full-text links:<br>Access Paper:

View a PDF of the paper titled Petals: Collaborative Inference and Fine-tuning of Large Models, by Alexander Borzunov and 7 other authors<br>View PDF<br>TeX Source

view license

Current browse context:

cs.LG

next >

new<br>recent<br>| 2022-09

Change to browse by:

cs<br>cs.DC

References & Citations

NASA ADS<br>Google Scholar

Semantic Scholar

export BibTeX citation<br>Loading...

BibTeX formatted citation

&times;

loading...

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender<br>(What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Which authors of this paper are endorsers? |<br>Disable MathJax (What is MathJax?)

Major funding support from