Linux BPF Superpowers
Brendan's site:
Start Here
Homepage
Blog
Full Site Map<br>--><br>Sys Perf book
BPF Perf book
Linux Perf
eBPF Tools
perf Examples
Perf Methods
USE Method
TSA Method
Off-CPU Analysis
Active Bench.
WSS Estimation
Flame Graphs
Flame Scope
Heat Maps
Frequency Trails
Colony Graphs
DTrace Tools
DTraceToolkit
DtkshDemos
Guessing Game
Specials
Books
Other Sites
Book sale until Dec 1, 2021: 55% off for 2 or more
-->
Systems Performance 2nd Ed.
BPF Performance Tools book
I'm speaking at AWS re:Invent 2019<br>-->
Recent posts:
07 Feb 2026 "
Why I joined OpenAI
05 Dec 2025 "
Leaving Intel
28 Nov 2025 "
On "AI Brendans" or "Virtual Brendans"
22 Nov 2025 "
Intel is listening, don't waste your shot
17 Nov 2025 "
Third Stage Engineering
04 Aug 2025 "
When to Hire a Computer Performance Engineering Team (2025) part 1 of 2
22 May 2025 "
3 Years of Extremely Remote Work
01 May 2025 "
Doom GPU Flame Graphs
29 Oct 2024 "
AI Flame Graphs
22 Jul 2024 "
No More Blue Fridays
24 Mar 2024 "
Linux Crisis Tools
17 Mar 2024 "
The Return of the Frame Pointers
10 Mar 2024 "
eBPF Documentary
28 Apr 2023 "
eBPF Observability Tools Are Not Security Tools
01 Mar 2023 "
USENIX SREcon APAC 2022: Computing Performance: What's on the Horizon
17 Feb 2023 "
USENIX SREcon APAC 2023: CFP
02 May 2022 "
Brendan@Intel.com
15 Apr 2022 "
Netflix End of Series 1
09 Apr 2022 "
TensorFlow Library Performance
19 Mar 2022 "
Why Don't You Use ...
Blog index
About
RSS
I am program co-chair for LISA 2018<br>-->
Brendan Gregg's Blog
home
Linux BPF Superpowers
05 Mar 2016
Last month I spoke at Facebook's Performance @Scale event about Linux BPF Superpowers. These are coming to Linux in the 4.x series, and I've been using them in new open source performance tools.
Video is on Facebook (30 mins):
Linux 4.x Performance: Using BPF Superpowers (Brendan Gregg)<br>Posted by At Scale on Friday, February 26, 2016
Slides are on slideshare (PDF):
We've stopped calling it eBPF (extended Berkeley Packet Filter), and are now just calling it BPF, although we need a better backronym: Bytecode Probe Framework? Naming things is hard.
BPF is the in-kernel bytecode machine that can be used for tracing, virtual networks, and more. Alexei Starovoitov is the lead developer (he's now at Facebook), and there are developers from several companies contributing, including myself at Netflix, Daniel Borkmann at Cisco, and Brenden Blanco at PLUMgrid.
As an example of BPF, I opened with off-CPU analysis, and why BPF was making new things possible. I summarized some other examples as well, including gethostlatency, which instruments DNS lookups system wide without needing to restart anything:
# ./gethostlatency<br>TIME PID COMM LATms HOST<br>06:10:24 28011 wget 90.00 www.iovisor.org<br>06:10:28 28127 wget 0.00 www.iovisor.org<br>06:10:41 28404 wget 9.00 www.netflix.com<br>06:10:48 28544 curl 35.00 www.netflix.com.au<br>06:11:10 29054 curl 31.00 www.plumgrid.com<br>06:11:16 29195 curl 3.00 www.facebook.com<br>06:11:25 29404 curl 72.00 foo
gethostlatency, and the other tools I demonstrated, are in bcc tools, which is a Python front end for BPF. For this talk I created a diagram of all the bcc tracing tools so far:
So many of my favourites (from other tracing languages) now have equivalents in bcc, which is pretty exciting. Tools like execsnoop, opensnoop, ext4slower, tcpretrans, tcpconnect, and runqlat.
These bcc tools are still in development and require at least Linux 4.1, which many people aren't running yet. You can think of them as a preview of things to come. But they are coming sooner rather than later: Ubuntu 16.04 (for example) will have a 4 series kernel, and isn't far away.
Please watch my talk video above, and check out the other talk videos which were pretty interesting as well (although that link plays the low-res versions in Chrome; high-res versions, like I linked to above, do exist).
Thanks to Facebook for having me – it was a great event.
Links from the talk
iovisor bcc:
https://github.com/iovisor/bcc
http://www.brendangregg.com/blog/2015-09-22/bcc-linux-4.3-tracing.html
http://blogs.microsoft.co.il/sasha/2016/02/14/two-new-ebpf-tools-memleak-and-argdist/
BPF Off-CPU, Wakeup, Off-Wake & Chain Graphs:
http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html
http://www.brendangregg.com/blog/2016-02-01/linux-wakeup-offwake-profiling.html
http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html
Linux Performance:
http://www.brendangregg.com/linuxperf.html
Linux perf_events:
https://perf.wiki.kernel.org/index.php/Main_Page
http://www.brendangregg.com/perf.html
Flame Graphs:
http://techblog.netflix.com/2015/07/java-in-flames.html
http://www.brendangregg.com/flamegraphs.html
Netflix Tech Blog on Vector:
http://techblog.netflix.com/2015/04/introducing-vector-netflixs-on-host.html
Click here for Disqus comments (ad supported).
You are welcome to comment here,...