I Built an AI Companion That Actually Plays With You

The goal

Simple: Let's build a super-charged next-level gaming companion that actually feels good.

There are already multiple frameworks that let LLMs control NPC dialogue. They are fantastic for role-playing and staying in character, but they have two recurring problems: weak world agency and latency. Good at talking but far less reliable at performing actions and terrible at complex instruction sets. You may have also noticed how popular AI NPC demos often cut between the player speaking and the AI replying, trying to mask latency. Can we do better?

I wanted a companion that:

Useful and instant. It should fight, fetch, loot, inspect, carry and give items etc etc, follow complex multi-step instructions without feeling buggy or experimental. This matters especially in VR, where navigating menus is cumbersome and immersion-breaking. it needs to be FAST fast, not just fast

Alive and present. It should have a fun, endearing personality, not canned robotic pre-written responses. Remember shared experiences and change over time. The microphone stays active while a session is running: you do not summon Varkos through a dialogue menu, you talk to him. When immersion kicks in, it should feel like you are not playing alone.

Local and private wherever practical. The elephant in the room is that cloud LLM calls can get quite pricy (especially with multi-thousand-token LLM calls) and the added latency can be an experience killer. And why turn a private single-player game into a metered and surveilled experience? Let's try to give as much control to the user as possible (bonus it's a fun technical challenge).

Basically: a single-player game where you are not playing alone.

Complex commands

Varkos can handle commands that extend beyond one immediate action. Plans can wait for events, preserve targets between steps, monitor progress and repair or stop when world state changes. Nothing is pre-scripted.

Let's see some examples

Varkos receives a conditional instruction involving the next arrow. He registers the future trigger instead of acting immediately, waits for the correlated projectile impact and then continues the plan.

Long-form multi-step command

“I want you to wait here and I’m gonna go over there. Once you see the signal, the signal is going to be an arrow I fire up in the sky, I want you to pick up this potion and come and bring it to me. Okay?”

Your browser does not support embedded video.

A deferred command follows a real projectile event in Skyrim.

Item search

Varkos can search the grounded world state for a requested item, identify where it is and respond using what is actually present in the game.

“Do you see the ceremonial sword anywhere?”

Varkos picks up a different sword and brings it to us. We tell him that’s not the one, then he offers to be on the lookout.

Your browser does not support embedded video.

Finding an item through game state rather than inventing an answer.

Hide-and-seek

Hide-and-seek is not a single API call. It becomes a persistent goal with movement, waiting, monitoring and completion conditions.

“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.”

Your browser does not support embedded video.

A game represented as a persistent plan rather than a line of dialogue.

Loot this chest and give me the potion

This combines a grounded container, a filtered loot step and an inventory transfer. Each physical result advances the next part of the plan.

Your browser does not support embedded video.

Loot, select and transfer while preserving the requested item.

Pick up all the items

“Pick up all the items and give them to me” becomes a bounded collection plan over real references. Varkos gathers them, returns and transfers them without pretending that one magical action means “all.”

Your browser does not support embedded video.

A collection plan operating on grounded world objects.

Combat

Varkos receives grounded events from the game, can warn the player through a fast reflex path and uses native body control to act. Instruction plans can strategize (e.g. attack this, then retreat, etc.), and his emotional state can affect how and if he chooses to fight.

Your browser does not support embedded video.

Combat footage 1: dungeon combat.

Your browser does not support embedded video.

Combat footage 2: perception, warning and physical action on the latency-sensitive path.

Personality evolution

Varkos is fully customizable. He does not have to be a demon dog, and the runtime does not have to control only a single character. What systems are applied and what they do, is up to open configuration.

One part of my current build still fully depends on big model/cloud LLM calls: slow personality evolution. This work happens away from the real-time action path. As the player and Varkos travel together, important interactions become evidence for gradual...