Run your own ChatGPT: self-hosted LLMs with Docker Compose | TotalDebug

≡ menu

// technology. inside out

$ tree ~

├─~/homeh

├─~/categoriesc

├─~/tagst

├─~/archivea

├─~/projectsp

└─~/abouti

© 2026 Steven Marks<br>built with >_

>_ Docker · Homelab

Run your own ChatGPT: self-hosted LLMs with Docker Compose

Steven Marks

06 Jul 2026 · 8 min read

I use Claude Code every day but not everything belongs in the public domain: personal data, private notes, half-formed ideas. For those I’d rather keep the conversation local.

So I also run a local assistant. A small Docker Compose stack gives me a private LLM with a chat UI that looks a lot like the commercial ones and for that kind of prompt it means no per-token bill and nothing leaving my local network.

The whole thing is two services: Ollama to run the models and Open WebUI to talk to them.

What you get

Ollama pulls models, keeps them on disk, and serves them over a simple HTTP API on port 11434.

Open WebUI is the front end. It’s a polished, self-hosted chat interface with conversations, multiple models, user accounts and file uploads. It talks to Ollama over the API, so you get a ChatGPT-style experience locally.

Two containers, one network, two volumes for persistence. That’s the entire design.

Before you start

You need Docker which is likely already installed if you run anything in a homelab.

For anything beyond small models you also want an NVIDIA GPU and the NVIDIA Container Toolkit installed on the host. The toolkit is what lets a container see the GPU. Without it, the GPU block in the Compose file will fail to start, and you’ll want the CPU-only variant instead (I cover that later). You can check the toolkit is working with:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that prints your GPU, you’re ready. If it errors, fix the toolkit first, because the LLM stack relies on exactly that plumbing.

The Compose file

Here’s the stack in full. I’ll break down the parts that matter underneath.

10<br>11<br>12<br>13<br>14<br>15<br>16<br>17<br>18<br>19<br>20<br>21<br>22<br>23<br>24<br>25<br>26<br>27<br>28<br>29<br>30<br>31<br>32<br>services:<br>ollama:<br>image: ollama/ollama<br>command: serve<br>expose:<br>- 11434/tcp<br>healthcheck:<br>test: ollama --version || exit 1<br>volumes:<br>- ollama:/root/.ollama<br>deploy:<br>resources:<br>reservations:<br>devices:<br>- driver: nvidia<br>device_ids: ["all"]<br>capabilities: [gpu]

webui:<br>image: ghcr.io/open-webui/open-webui:main<br>ports:<br>- 8080:8080/tcp<br>environment:<br>- OLLAMA_BASE_URL=http://ollama:11434<br>volumes:<br>- open-webui:/app/backend/data<br>depends_on:<br>- ollama

volumes:<br>ollama:<br>open-webui:

Save it as docker-compose.yml and run docker compose up -d to bring the full stack online.

Before you do it’s worth understanding how it works in a little more detail below.

How the two services find each other

Notice OLLAMA_BASE_URL=http://ollama:11434. That ollama is not a hostname I’ve configured, it’s the service name for ollama. Compose gives every service a DNS entry on the default network for the project. So Open WebUI reaches the Ollama by name, over the internal network, with no host ports or IP addresses involved.

That’s also why Ollama uses expose rather than ports. expose makes 11434 reachable to other containers on the same network but does not publish it to the host. The only thing I actually publish is Open WebUI’s 8080, because that’s the one interface a human needs to reach. Keeping the raw model API off the host is a small but real bit of hygiene and I’ll come back to why it matters.

GPU passthrough, the part everyone gets stuck on

This block is what hands the GPU to the container:

deploy:<br>resources:<br>reservations:<br>devices:<br>- driver: nvidia<br>device_ids: ["all"]<br>capabilities: [gpu]

A few things worth knowing:

capabilities: [gpu] is required. If you omit it, Docker will not grant GPU access even with the rest of the block present.

device_ids: ["all"] gives the container every GPU. If you have more than one and want to pin Ollama to a specific card, swap it for something like device_ids: ["0"].

This is the Compose equivalent of docker run --gpus all. It only works if the NVIDIA Container Toolkit from the previous section is installed. This is where “it won’t start” almost always traces back to.

Get this right and Ollama will load models into VRAM and run them on the GPU, which is the difference between a usable assistant and one you wait on.

Persistence and health

The two named volumes persist the models to stop re-downloading several gigabytes of model every time a container restarts:

ollama:/root/.ollama keeps your pulled models.

open-webui:/app/backend/data keeps your accounts, chats and settings.

The healthcheck on Ollama gives Docker a real readiness signal instead of “the process is running, probably”. It’s simple, but it means depends_on and your own tooling can reason about whether the engine is actually up.

Bringing it up

docker compose up -d<br>docker compose ps

Open WebUI is now on http://localhost:8080. The first account you create becomes the admin, so...