Use the built-in GELU, don't roll your own! :: Giles' blog
el.dataset.currentDropdown = '')<br>}">
Giles' blog
Writing the post that I wished I'd found when I started learning whatever it was...
About
Contact
Archives
Categories
Blogroll
August 2026 (2)
July 2026 (8)
June 2026 (7)
May 2026 (2)
April 2026 (11)
March 2026 (3)
February 2026 (4)
January 2026 (4)
December 2025 (1)
November 2025 (3)
October 2025 (9)
September 2025 (3)
August 2025 (5)
July 2025 (1)
June 2025 (2)
May 2025 (3)
April 2025 (2)
March 2025 (7)
February 2025 (10)
January 2025 (6)
December 2024 (7)
September 2024 (1)
August 2024 (2)
July 2024 (2)
May 2024 (2)
April 2024 (2)
February 2024 (2)
April 2023 (1)
March 2023 (2)
September 2022 (1)
February 2022 (1)
November 2021 (1)
March 2021 (1)
February 2021 (2)
August 2019 (1)
November 2018 (1)
May 2017 (1)
December 2016 (1)
April 2016 (1)
August 2015 (1)
December 2014 (1)
August 2014 (1)
March 2014 (1)
December 2013 (1)
October 2013 (3)
September 2013 (4)
August 2013 (2)
July 2013 (1)
June 2013 (1)
February 2013 (1)
October 2012 (1)
June 2012 (1)
May 2012 (1)
April 2012 (1)
February 2012 (1)
October 2011 (1)
June 2011 (1)
May 2011 (1)
April 2011 (1)
March 2011 (1)
February 2011 (1)
January 2011 (1)
December 2010 (3)
November 2010 (1)
October 2010 (1)
September 2010 (1)
August 2010 (1)
July 2010 (1)
May 2010 (3)
April 2010 (1)
March 2010 (2)
February 2010 (3)
January 2010 (4)
December 2009 (2)
November 2009 (5)
October 2009 (2)
September 2009 (2)
August 2009 (3)
July 2009 (1)
May 2009 (1)
April 2009 (1)
March 2009 (5)
February 2009 (5)
January 2009 (5)
December 2008 (3)
November 2008 (7)
October 2008 (4)
September 2008 (2)
August 2008 (1)
July 2008 (1)
June 2008 (1)
May 2008 (1)
April 2008 (1)
January 2008 (4)
December 2007 (3)
March 2007 (3)
February 2007 (1)
January 2007 (2)
December 2006 (4)
November 2006 (18)
AI (97)
TIL deep dives (77)
Python (76)
LLM from scratch (48)
Resolver One (34)
PyTorch (22)
TIL (21)
Blogkeeping (19)
PythonAnywhere (17)
Linux (16)
Startups (15)
Gadgets (13)
Hugging Face (13)
NSLU2 offsite backup project (13)
Funny (11)
Musings (11)
Finance (10)
Fine-tuning LLMs (10)
C (9)
JAX (9)
Personal (8)
Robotics (8)
Website design (8)
3D (5)
Quick links (5)
Rants (5)
Cryptography (4)
JavaScript (4)
Music (4)
Oddities (4)
Talks (4)
Dirigible (3)
Eee (3)
GPT-2 mysteries (3)
Memes (3)
Politics (3)
Django (2)
GPU Computing (2)
LaTeX (2)
MathML (2)
Microprojects (2)
OLPC XO (2)
Retro Language Models (2)
Space (2)
VoIP (2)
Copyright (1)
Golang (1)
poppy the training box (1)
Raspberry Pi (1)
Software development tools (1)
Agile Abstractions
antirez
Astral Codex Ten
:: (Bloggable a) => a -> IO ()
David Friedman's Substack
Econ & Energy
Entrepreneurial Geekiness
For some value of "Magic"
Hackaday
kaleidic.ai newsletter
Knowing.NET
Language Log
Millennium Hand
ntoll.org
Obey the Testing Goat!
One Useful Thing
PK
PythonAnywhere News
Simon Willison's Weblog
Societive
Software Deviser
Some opinions, held with varying degrees of certainty
tartley.com
the singularity is nearer
Theia Vogel's website
Use the built-in GELU, don't roll your own!
Posted on 20 August 2026
in
AI,
PyTorch,
Python
Unsurprisingly, PyTorch's own built-in GELU function<br>is faster than the hand-rolled one I've been using to date. But I was surprised at<br>how much faster using it made things when training my models. I discovered this<br>accidentally just now while working on something unrelated, but am logging the details<br>here for anyone else that might find it useful.
The headline numbers: the same code, training the same model on the same data, ran<br>at about:
21,000 tokens per second using the hand-rolled GELU from Sebastian Raschka's book<br>"Build a Large Language Model (from Scratch)".
25,000 tokens per second using PyTorch's built-in GELU with no arguments.
25,000 tokens per second using the built-in GELU with approximate="tanh", which<br>uses the same maths as Raschka's version under the hood.
That's a 20% increase in throughput for both of the built-in versions -- definitely nothing to be sneezed at.
And what is particularly<br>interesting is that there aren't that many GELUs going on -- it's a GPT-2 small-style<br>model, with 12 layers. So that's 12 GELUs handling tensors<br>shaped (batch_size, seq_len, 4 * d_emb), which is (6, 1024, 3072) for my training setup.<br>Given that the rest of the model is doing all of the normal full attention stuff for GPT-2, it's<br>really surprising that the GELUs alone must have been taking up so much of the time. The throughput<br>numbers mean that we must have been spending about 17% of our time on the extra overhead from the hand-rolled<br>version, so that sets a lower bound for how much time the GELUs were taking up.
More info below the fold.
Back when I was doing the "interventions" part of my LLM from scratch...