Run LLM Without OS

NightRun UEFI application boots a local LLM on Raspberry Pi 5 and x86 PCs without an OS - CNX Software

Running LLMs locally on an SBC or mini PC is a good way to keep your data private, but they typically require a conventional operating system, which uses a good portion of system memory. NightRun is an experimental open-source project that addresses this issue by booting a machine directly into a local LLM from a USB drive or microSD card, without loading a conventional OS. By removing the operating system, NightRun makes more RAM and memory bandwidth available for AI inference. The runtime is written in Rust, and during boot it loads a quantized model (1.3 GB to 2.4 GB) directly into RAM while verifying its CRC-32 checksums. After the model is loaded, storage is “sealed,” meaning any later attempt to read from the disk will trigger a hard fault.

NightRun key features:

Hardware support – Supports 64-bit x86_64 UEFI PCs (Secure Boot disabled) and Raspberry Pi 5 via USB or microSD card

Supported models – Supports Llama 3.2 1B (4GB RAM), Llama 3.2 3B and Granite 4.1 3B (6GB RAM), and Qwen3 4B Instruct 2507 (8GB RAM) models converted to the .nrm format

Optimized inference – Uses AVX2/FMA/F16C kernels on x86 and NEON kernels on Raspberry Pi

Zero-Copy design – Runs quantized Q8_0, Q4_K, and Q6_K models directly from RAM without dequantization

Built-in UI – Includes a framebuffer chat interface with live stats, prompt editing, and conversation history

NightRun generating a response with the Granite 4.1 3B model in its built-in framebuffer chat interface. Completed response from the Granite 4.1 3B model in NightRun’s framebuffer chat interface, with live inference statistics displayed in the status bar.

It might be easy to think that NightRun is a stripped-down Linux distribution designed to save memory, but that is not the case. It is a UEFI application that runs without a conventional operating system, so there is no Linux kernel, scheduler, or network stack underneath. Instead, it relies on UEFI Boot Services to handle keyboard input, drive the display framebuffer, access storage, and even control the Raspberry Pi 5 cooling fan. In terms of performance, the developers say decoding speed is mainly limited by memory bandwidth. In an 8-core x86 QEMU virtual machine, Llama 3.2 1B (Q8_0) reaches around 20 tokens per second during decoding. On a real 8GB Raspberry Pi 5, Granite 4.1 3B (Q4_K_M) generates about 3.0 tokens per second using the early pre-sdot NEON kernels. We have previously written about Binh Pham’s LLMStick that uses C/C++ inference engines like llama.cpp deployed on top of lightweight Linux distributions (such as Alpine or DietPi) to free up memory. Compared to that, NightRun is a complete, production-ready UEFI runtime appliance with parity-tested tokenization specifically designed for consumer hardware. The majority of the code for the NightRun project was written using Claude Code with the Fable 5 model and released under the MIT license on GitHub. Installation is handled by a Linux shell script that downloads a verified model and flashes the bootable image. The developers have implemented various safety checks in the installer to explicitly type FLASH /dev/sdX to prevent accidental overwriting of system drives. More details on the technical implementation can be found at nightrun.io.

Nightrun demo

Debashis Das Debashis Das is a technical content writer and embedded engineer with over five years of experience in the industry. With expertise in Embedded C, PCB Design, and SEO optimization, he effectively blends difficult technical topics with clear communication

Run LLM Without OS

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

Google will expand age checks on Android worldwide till the end of the year