[2607.15115] Don't Predict, Prioritize: Rethinking GPU Reliability Assessment

Skip to main content

Search arXiv

Press Enter to search · Advanced search

-->

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2607.15115 (cs)

[Submitted on 16 Jul 2026]

Title:Don't Predict, Prioritize: Rethinking GPU Reliability Assessment

Authors:Difeng Ma, Changhua Pei, Yuanwei Lu, Quan Zhou, Zexin Wang, Yibo Zhu, Daxin Jiang, Dan Pei, Jingjing Li, Gaogang Xie<br>View a PDF of the paper titled Don't Predict, Prioritize: Rethinking GPU Reliability Assessment, by Difeng Ma and 9 other authors

View PDF<br>HTML (experimental)

Abstract:The reliability of Graphics Processing Units (GPUs) is a criticalbottleneck for modern large-scale AI infrastructure, where a sin-gle node failure can disrupt synchronous training jobs and causesignificant financial losses. While predictive maintenance is widelyused in other hardware domains, we demonstrate that accuratelypredicting the exact timing of GPU failures is inherently this http URL an in-depth analysis of telemetry data from a productioncluster, we find that major GPU failures, including Double Bit Er-rors (DBEs) and GPU Lost events, exhibit strong stochasticity andlow signal-to-noise ratios in time-series telemetry, which makesconventional time-based prediction ineffective.

This insight motivates a paradigm shift: instead of attempting topredict the absolute timing of a failure, we propose a more robustapproach focused on ranking nodes by their relative failure risk. Wepropose HeaRank (Health Rank), a Learning-to-Rank (LTR) frame-work that leverages stable historical failure patterns to computea global risk ranking of GPU nodes. Evaluated on a production-scale cluster with thousands of GPUs, HeaRank achieves an AUCof 0.83, significantly outperforming both heuristic baselines andstate-of-the-art ranking algorithms. In online deployment, HeaRanksuccessfully captures 64% of future failures within the top 5% ofranked nodes, compared to only 21% by the incumbent productionsystem. These results suggest that relative risk ranking can serveas a robust alternative in environments where absolute failure pre-diction is inherently limited. Our work highlights the importanceof risk-aware scheduling and proactive resource management inmodern GPU clusters.

Comments:<br>Accepted at ACM SIGKDD 2026; 13 pages, 13 figures

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as:<br>arXiv:2607.15115 [cs.DC]

(or<br>arXiv:2607.15115v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2607.15115

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

Journal reference:<br>Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026)

Related DOI:

https://doi.org/10.1145/3770855.3818373

Focus to learn more

DOI(s) linking to related resources

Submission history<br>From: Zexin Wang [view email]<br>[v1]<br>Thu, 16 Jul 2026 15:24:47 UTC (3,743 KB)

Full-text links:<br>Access Paper:

View a PDF of the paper titled Don't Predict, Prioritize: Rethinking GPU Reliability Assessment, by Difeng Ma and 9 other authors<br>View PDF<br>HTML (experimental)<br>TeX Source

view license

Current browse context:

cs.DC

next >

new<br>recent<br>| 2026-07

Change to browse by:

cs

References & Citations

NASA ADS<br>Google Scholar

Semantic Scholar

export BibTeX citation<br>Loading...

BibTeX formatted citation

&times;

loading...

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to...