Exploring BPF LSM support on aarch64 with ftrace

Home<br>Platform

Industries

Resources

Company

Login

Book a demo

This blog post is an extract of an internal research we made about BPF LSM support on aarch64 in Linux. It's very difficult to start looking inside the kernel source if you are not familiar with the codebase, so we decided to publish this post to show our approach as it can be useful to anyone who wants to explore kernel internals.<br>Introduction<br>We were already using BPF LSM on x86_64, while on aarch64 we are relying on Kprobes, so we wanted to known what is missing inside the kernel to have this features on aarch64.<br>We dug many times into the kernel source code, but in general we search for something that is already there to understand how it works. In this case instead we were looking for something that doesn't exists, looking for something that returns an error because is not implemented.<br>Remembering Steven Rostedt talks on how to start learning the Linux kernel, we started with ftrace (and tools built on the tracing infrastructure) to understand what happens when we load an unsupported BPF program into the kernel.<br>The problem<br>This is the output when we try to load a BPF LSM program into a aarch64 5.15 Linux kernel using our software pulsar:7<br>root@pine64-1:/home/exein# ./pulsar-enterprise-exec pulsard<br>[2023-02-16T14:52:45Z INFO pulsar::pulsard::daemon] Starting module process-monitor<br>[2023-02-16T14:52:45Z INFO pulsar::pulsard::daemon] Starting module file-system-monitor<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module network-monitor<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module logger<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module rules-engine<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module desktop-notifier<br>[2023-02-16T14:52:46Z ERROR pulsar::pulsard::module_manager] Module error in file-system-monitor: failed program attach lsm path_mknod

Caused by:<br>0: `bpf_raw_tracepoint_open` failed<br>1: No error information (os error 524)<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module anomaly-detection<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module malware-detection<br>[2023-02-16T14:52:46Z ERROR pulsar::pulsard::module_manager] Module error in malware-detection: /var/lib/pulsar/malware_detection/models/parameters.json not found<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module platform-connector<br>[2023-02-16T14:52:46Z INFO platform_connector::client] Connected to https://platform-dev-instance.exein.io:8001/<br>[2023-02-16T14:52:46Z INFO pulsar::pulsard::daemon] Starting module threat-response<br>[2023-02-16T14:52:46Z ERROR pulsar::pulsard::module_manager] Module error in network-monitor: failed program attach lsm socket_bind

Caused by:<br>0: `bpf_raw_tracepoint_open` failed<br>1: No error information (os error 524)We got error 524, or ENOTSUPP, while pulsar tries to load the BPF program related to path_mknod LSM hook. Let's try to dig into this problem.<br>Note: At the time of this research we were not able to find a precompiled aarch64 with BPF and BTF enabled, so we had to compile a custom kernel. We also enabled the tracing option and the function_graph plugin to use the tools below.<br>All the experiments were done on a Pine A64 with custom Armbian images.<br>This images have custom kernels with a standard Ubuntu 22.04 LTS Jammy userspace.<br>The tools<br>To investigate on this issue we used the following tools:<br>bpftrace: BPF based tool to dynamically attach probes with a custom C-like language.<br>trace-cmd: wrapper around tracefs filesystem that interacts with ftrace infrastructure.<br>To use this tools you need some options enabled into the Linux kernel , please check the official documentation for the full requirements.<br>Note: It's also possible to use other tools to do the same job, for example funcgraph and kprobe from perf-tools<br>Linux 5.15<br>Now we start using these tools to see what happens in the kernel 5.15 when we try to load our BPF program.<br>From this point to the end of this post we will use the probe binary instead of pulsar, because it is simpler. To recap how it work this is the command line help:<br>exein@pine64-1:~$ ./probe<br>Test runner for eBPF programs

Usage: probe [OPTIONS]

Commands:<br>file-system-monitor Watch file creations<br>process-monitor Watch process events (fork/exec/exit)<br>network-monitor Watch network events<br>help Print this message or the help of the given subcommand(s)

Options:<br>-v, --verbose<br>-h, --help Print help<br>-V, --version Print versionIn the examples, we will try to load the file-system-monitor probe.<br>Running the following commands we can see the function graph calls of __sys_bpf function, the BPF syscall entrypoint:<br>trace-cmd record -p function_graph -g __sys_bpf ./probe file-system-monitor<br>trace-cmd reportThe output is a giant function graph , too large to paste here. Because we have an error we are interested in the last functions before the program stopped. These...