Building an Infinite Memory Local AI Stack on Fedora | lead > prompt #

If you’ve been following my recent experiments, you know I’m a huge advocate for running open-weight models locally. But one of the biggest friction points with local LLMs is the context window. Even with a massive 256k context limit, you can't, and shouldn't, dump your entire history into every prompt. It bloats VRAM and destroys inference speeds.

The solution is a Personalized Long-Term Memory RAG (Retrieval-Augmented Generation) . Today, I'm going to walk you through exactly how I built an autonomous, privacy-focused memory layer that seamlessly injects context into my chats using Mem0 , Qdrant , and Open WebUI , completely bypassing the need for clunky UI document uploads.

The Hardware & OS

My current daily driver for this setup is a modular mini PC from Framework running Fedora 44 .

Here are the specs of my machine:

Ryzen™ AI Max+ 395 - 128GB (I've allocated 118GB of this to VRAM for hosting models)

Storage: WD_BLACK™ SN7100 NVMe™ - M.2 2280 - 4TB

For local inference, I'm relying entirely on an integrated AMD GPU . Offloading a heavy model like qwen3-coder-next pushes the APU to its limits and I regularly see GPU utilization pinned at 100% with VRAM usage hovering right around 40-50GB . Despite being integrated graphics, this configuration handles the RAG pipeline flawlessly without breaking a sweat.

We will be using Podman instead of Docker, keeping things native to the Fedora ecosystem.

Step 1: Spin Up the Vector Database (Qdrant)

Qdrant acts as the limitless hard drive for our AI's memories. It stores the mathematical vectors and text payloads on disk so they persist across reboots.

Run this Podman command to spin up Qdrant using host networking:

Bash

podman run -d \<br>--name qdrant \<br>--network=host \<br>-v ~/.local/share/qdrant_data:/qdrant/storage:z \<br>--restart always \<br>qdrant/qdrant:latest

Step 2: Prepare the Ollama Models

We need two models running in Ollama:

The Embedder: Translates text into 768-dimensional vectors.

The Chat Model: The brains of the operation.

Bash

ollama pull nomic-embed-text:latest<br>ollama pull qwen3-coder-next:q4_K_M

(Tip: Make sure the Ollama daemon is running in the background via systemctl status ollama).

Step 3: The Magic Middleware (Mem0 Proxy)

Instead of hacking memory directly into Open WebUI, we are deploying a lightweight FastAPI proxy. This sits between your UI and Ollama. When you send a message, the proxy intercepts it, searches Qdrant for relevant past facts, injects them into the system prompt, and forwards it to Qwen. This is the RAG step in action!

Crucially, it also features a background task that silently extracts new facts from your conversation and saves them to Qdrant without adding any latency to your chat response.

Create a file at ~/memory_proxy.py (or wherever you wish) and paste this in:

Python

import os<br>import uvicorn<br>import asyncio<br>from fastapi import FastAPI, Request, BackgroundTasks<br>from fastapi.responses import StreamingResponse<br>import httpx<br>from mem0 import Memory

os.environ["MEM0_TELEMETRY"] = "False"

app = FastAPI(title="Mem0 Local Proxy with Auto-Extract")

# Configure Mem0<br>mem0_config = {<br>"vector_store": {<br>"provider": "qdrant",<br>"config": {<br>"collection_name": "lead_prompt_memories",<br>"host": "localhost",<br>"port": 6333,<br>"embedding_model_dims": 768,<br>},<br>"llm": {<br>"provider": "ollama",<br>"config": {<br>"model": "qwen3-coder-next:q4_K_M",<br>"temperature": 0.1,<br>"max_tokens": 4096,<br>"ollama_base_url": "http://127.0.0.1:11434",<br>},<br>"embedder": {<br>"provider": "ollama",<br>"config": {<br>"model": "nomic-embed-text:latest",<br>"ollama_base_url": "http://127.0.0.1:11434",

memory = Memory.from_config(mem0_config)<br>OLLAMA_BASE_URL = "http://127.0.0.1:11434"

# Background extraction logic<br>def extract_and_save_memory(messages: list, user_id: str):<br>try:<br>memory.add(messages, user_id=user_id)<br>print(f"\n[MEM0 BACKGROUND]: Memory extraction complete for user '{user_id}'.")<br>except Exception as e:<br>print(f"\n[MEM0 BACKGROUND ERROR]: Failed to extract memory: {e}")

@app.get("/v1/models")<br>async def list_models():<br>async with httpx.AsyncClient(timeout=30.0) as client:<br>resp = await client.get(f"{OLLAMA_BASE_URL}/v1/models")<br>return resp.json()

@app.post("/v1/chat/completions")<br>async def chat_completions(request: Request, background_tasks: BackgroundTasks):<br>body = await request.json()<br>messages = body.get("messages", [])<br>user_id = body.get("user", "john")

user_query = ""<br>for msg in reversed(messages):<br>if msg.get("role") == "user":<br>user_query = msg.get("content", "")<br>break

# 1. READ: Retrieve existing memories<br>memories_str = ""<br>if user_query:<br>try:<br>results = memory.search(query=user_query, filters={"user_id": user_id})<br>mem_list = [r["memory"] for r in results.get("results", [])]<br>if mem_list:<br>memories_str = "\n".join(f"- {m}" for m in mem_list)<br>except Exception as e:<br>print(f"[Proxy Memory Error]: {e}")

# 2. INJECT: Add retrieved context to system prompt<br>if memories_str:<br>context_prompt =...