GitHub - sachinkesiraju/jax-realtime: Full‑duplex realtime voice assistant in your browser with jax‑js · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
sachinkesiraju
jax-realtime
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>33 Commits<br>33 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>docs
docs
public/vad
public/vad
src
src
tests
tests
.gitignore
.gitignore
.nvmrc
.nvmrc
LICENSE
LICENSE
README.md
README.md
index.html
index.html
netlify.toml
netlify.toml
package-lock.json
package-lock.json
package.json
package.json
tsconfig.json
tsconfig.json
vite.config.ts
vite.config.ts
View all files
Repository files navigation
jax-realtime
A real-time, full-duplex voice assistant that runs entirely in your browser<br>on WebGPU, built with jax-js.
Every stage (speech → ASR → LLM → TTS → speech, plus optional vision) runs<br>locally in the tab; nothing is sent to a server.
It's inspired by the Thinking Machines<br>interaction model and<br>GPT-Live, rebuilt as a<br>small-model cascade that fits in a browser tab. The goal is a conversation<br>that feels live: you can interrupt it mid-sentence, pause mid-thought<br>without losing your turn, and it keeps searching in the background while you<br>talk.
Stage<br>Model<br>Runs on
Ear (ASR)<br>Whisper base.en (int8, dequantized to fp16)<br>WebGPU via jax-js
Turn-taking (VAD)<br>Silero VAD v5, ported to TypeScript<br>CPU (~2 ms / 32 ms frame)
Brain (LLM)<br>SmolLM2-360M-Instruct (fp16)<br>WebGPU via jax-js
Voice (TTS)<br>Kyutai Pocket TTS + Mimi codec (fp16)<br>WebGPU via jax-js
Eye (vision)<br>D-FINE small (COCO-80)<br>WebGPU via @jax-js/onnx
Everything shares the single WebGPU device. The streaming ASR lane is paused<br>while the assistant speaks so it doesn't contend with TTS for the GPU; barge-in<br>is therefore energy-based (see below), and captions resume the moment the<br>assistant stops.
Interaction
Full-duplex micro-turns : a ~150 ms tick loop drives a deterministic,<br>priority-ordered policy: adaptive barge-in (talk over the assistant,<br>including its tool narrations, and the audio cuts in ~300 ms; the threshold<br>auto-calibrates to the echo floor of each reply), adaptive endpointing ,<br>and time-awareness timers. A watchdog force-recovers the session if a reply<br>ever stalls, so it can't wedge.
Continuation-merge : if the endpoint fires on a mid-thought pause and you<br>resume speaking before the reply's first audio, the unheard reply is aborted<br>and both halves are answered as one turn ("append, don't restart").
Learned turn signal : a pure-TypeScript port of Silero VAD v5 scores<br>P(speech) every 32 ms on the CPU and drives speech onset, silence tracking,<br>and the phantom-turn guard: keyboard noise and ambient swells never even<br>latch an utterance, near-silence never reaches Whisper (so no hallucinated<br>"thank you"s), and quiet speech still passes. A repetition-degeneracy gate<br>drops decoder loops on top.
Eye (vision) : enabled by default for webcam context, with a pre-load<br>toggle to skip its 42 MB model, camera access, and GPU residency. D-FINE<br>runs low-priority object detection (it yields the GPU to audio), smooths the<br>person count, and answers<br>"what do you see?" / "how many people?" / "tell me about the person"<br>directly from the measurements. Proactive interjections (stepped away,<br>phone spotted, slouching) are best-effort rule heuristics. The webcam shows<br>as a corner PiP with detection boxes.
Typed conversation memory : bounded facts the user explicitly states<br>(name, trip, pet, favorite, plans, relationships) are retained and injected<br>only when relevant; exact recall bypasses small-model guessing.
Two-tier tools : factual asks are delegated so the small on-device model<br>isn't left guessing: weather ("what's the weather in Tokyo" → open-meteo,<br>in °F/mph), facts ("who is Ada Lovelace" → Wikipedia), plus instant offline<br>calculator and clock/date . Web lookups speak a holding line and fetch<br>in the background, then answer on the next silence and render a card; the<br>card clears when the conversation moves on, and the spoken answer can be<br>interrupted like any reply.
Performance (all on jax-js / WebGPU)
The turn-latency floor is set by the single GPU, so the work went into cutting<br>GPU cost per token/frame rather than overlapping stages (which a single device<br>can't do; see docs/BENCHMARKS.md for the full<br>map-reduce campaign log, including the negative results):
Fused decode : the LLM decode step is fused from dozens of per-layer jit<br>dispatches into one, and Pocket TTS from ~11 into two, cutting the<br>command-buffer submit overhead that dominated per-step...