Language integrated LLMs as an OCaml function | Anil Madhavapeddy
>_ Anil Madhavapeddy
>_esc<br>PapersNotesProjectsIdeasTalksLinks<br>Type to search across all content
↑↓ nav↵ openesc close
Anil MadhavapeddyFable cut out on me at 1am on Saturday while I was sweeping over the OCaml runtime looking for concurrency bugs. There have been excellent takes on the sovereignity implications of this, and I figured I'd roll my sleeves up and get serious about using the open weights models. DeepSeek's models have been getting more capable since their first release, and v4 Flash is small enough to run on my Mac (admittedly, very high-end Macs with 128GB/512GB of RAM respectively for my laptop and desktop).
The question is whether the agentic CLIs I've been using can be easily replaced. The best way to learn how a system works is to build it unikernel style, and so I aimed to expose the LLM as a normal OCaml library.<br>This avoids routing via bloated CLIs, and<br>lets the linking application drive the agentic loop according to its specific needs.
What makes this practical is Antirez'<br>Dwarfstar, a self-contained<br>native inference engine that supports Apple Metal and portable(ish) C.<br>I bound this directly to OCaml 5 and Eio as<br>ocaml-deepseek , and now a<br>plain function call on my laptop gets me an LLM in my application.
The Humpty OCaml deepseek agent in full (local) poetic flow<br>For example, I can now embed Deepseek inference directly into the OCaml webserver that drives this very<br>site in order to look for suspicious bot activity, and because it's open<br>weights and running locally, there's no dependency on external services!
(* A traffic-triage agent in-process in OCaml. The agent is handed two<br>OCaml function tools and works out for itself how to combine them. *)<br>let agent =<br>Agent.create engine ~system:"You are a web-traffic analyst."<br>~tools:[<br>Toolbox.read ~dir:logs; (* read-only sandboxed handle to the logs dir *)<br>query_db ~conn; (* a SELECT-only tool over the local database *)<br>in<br>Agent.send agent ~on_event<br>"Cross-reference today's 404 spikes in the access log against the \<br>client IPs in the requests table. Anything coordinated indicating a bad bot?"
The log reader and the database query are just two OCaml functions the model is<br>allowed to call, each scoped and sandboxed (using Eio) to exactly what it needs. The model decides when and<br>how to combine them.
1 Trying out Humpty the OCaml agent
I've submitted the package to opam, so opam install deepseek or opam pin add deepseek https://tangled.org/anil.recoil.org/ocaml-deepseek.git should work.<br>The package also ships a binary called humpty[1] with two variants: humpty-metal for Apple Silicon and a portable humpty-cpu that should run anywhere (slowly).
There are four subcommands that we'll use to explain how to build an agent up in OCaml:<br>first list the available models and download one, then chat with it statelessly, and then wrap that into an agent.
1.1 Choose the right Deepseek model
Before we can get started you'll first need the open model weights downloaded.<br>humpty list<br>prints a the catalogue of available weights:
$ humpty-metal list<br>Models (download dir: /Users/avsm/.local/share/ds4)
TARGET ALIASES DESCRIPTION<br>[ ] q2-imatrix q2 2-bit Flash routed experts (~81 GB); for 96-128 GB RAM.<br>[*] q2-q4-imatrix q2q4 Mixed Flash quant (~98 GB); higher quality for 128 GB.<br>[ ] q4-imatrix q4 4-bit Flash routed experts (~153 GB); for 256 GB+ RAM.<br>[ ] pro-q2-imatrix pro-q2 PRO q2 single file (~430 GB); for 512 GB RAM.
[*] = present, [ ] = not downloaded
Pick one based on how much RAM you have; I use q2q4 on my laptop (with 128GB RAM),<br>and the extremely beefy pro-q2 on my Mac Studio (with 512GB RAM).<br>There are also split files for running the model distributed across several machines, which I'll<br>skip here for now.
1.2 Grab the Deepseek model weights
Once you've chosen, humpty download q4<br>(or pro-q2, or whichever) shells out to the Hugging Face CLI to fetch the GGUF.<br>You'll either need the Huggingface CLI installed or have uvx in your path.
Once this gets doing go have a cup of tea while the gigabytes of LLM weights<br>download, and then we'll start to build an agent from the camel up!
2 Building an agent from the ground up
I first want to pin down what an "agent" actually means, as the term seems to have<br>accreted much mystique this year. The whole OCaml Deepseek stack is a small library you<br>can read through quickly, so let me build an agent up from scratch.<br>The code below links to the Tangled source.
2.1 An LLM is a stateless request-reply function
A basic LLM takes in a text prompt, performs inference on some weights, and generates a text reply back.<br>To illustrate this in our OCaml code, we need to load the model weights and spin up an<br>engine with a cache directory for the compiled Metal kernels (if using the<br>Apple GPU version):
let engine = Deepseek.V4.create ~cache ~model ~domain_mgr ~sw () in<br>V4.generate engine "Explain monads in one sentence." ~on_token:print_string;<br>- :...