I Built a Digital Human You Can Talk To

Skip to content

AI & Machine Learning

I built a Digital Human you can talk to

Published by

krishnagokula

on

August 3, 2026

Most digital-human demos start with a prepared script or audio file. I wanted a live conversation.

A user had to speak through the browser. The system had to understand the question, retrieve relevant information, generate a response, synthesize speech, animate a face, and stream the result back as video.

The notes from our 21 June 2024 demo recorded about five seconds to stream video for a short, 20-token response. We ran that demo on an AWS G5.2xlarge instance using Groq, Chroma, AWS Polly, GeneFace++, FastAPI, and WebRTC.

A separate end-to-end test on the AWS A10G measured an 18-second minimum and a 25-second average for a ten-word response. The experiment log does not contain enough detail to reconcile the difference. I treat the five-second figure as a demo observation and the end-to-end table as the repeatable benchmark.

The five-second result came from a pipeline of separate systems. Each system had its own latency, output format, failure modes, and hardware requirements.

Defining the conversation

The proof of concept supported a turn-based, two-way loop:

The user spoke to the avatar through a browser.

The browser sent the audio to the backend.

The backend transcribed the speech and generated an answer.

The avatar spoke the answer and streamed video back to the browser.

Full-duplex barge-in remained outside the PoC. The notes do not show a user interrupting the avatar mid-response and forcing the system to cancel queued speech and video frames. That requires a different session state machine and cancellation path.

The architecture

Browser microphone → FastAPI session → Deepgram VAD + STT → Groq + Chroma → AWS Polly → GeneFace++ → WebRTC video → Browser

The system processed one turn through eight stages:

Capture: React captured the user’s microphone input in the browser.

Session handling: FastAPI accepted the audio stream and maintained the conversation session.

Turn detection: Deepgram’s voice activity detection identified speech boundaries.

Transcription: Deepgram converted the recorded turn into text.

Retrieval and generation: Chroma retrieved relevant document chunks, and Groq generated the response.

Chunking: The backend split the response into units that the speech and animation stages could process without waiting for the full answer.

Speech and animation: AWS Polly produced audio. GeneFace++ used that audio and the trained avatar data to generate video frames.

Delivery: aiortc sent the frames to the browser over WebRTC.

Chroma held indexed reference material for retrieval. It did not train the language model. That distinction matters because teams often label any data connected to an LLM as training data.

The latency budget

Tvisible ≈ Tendpoint + Tretrieval + Tfirst token + Tspeech chunk + Tavatar chunk + Ttransport

The individual benchmark values below cannot be added into one exact end-to-end total. We tested some components with different input sizes, GPUs, cloud regions, and levels of overlap. The equation shows where the delay came from and where we had to measure it.

The avatar renderer consumed much of the budget. GeneFace++ generated ten seconds of video in about five seconds on an RTX 4070. A one-second LLM response still felt slow if the system waited for a complete audio track and complete video before sending the first frame.

Text chunking gave the pipeline a way to overlap work. The speech and animation stages could process the first chunk while the LLM produced later chunks. Chunk size created a trade-off: small chunks reduced the wait but could damage prosody and create visible transitions; large chunks produced smoother output at the cost of a longer pause.

Technical specification

LayerPoC choiceRoleComputeAWS G5.2xlarge with NVIDIA A10GHosted the API and GPU avatar workloadBackendPython 3.8 and FastAPIManaged sessions and pipeline orchestrationVoice activity and STTDeepgramDetected turns and converted speech to textLLMGroq, with Mixtral 8x7B in the RAG testsGenerated the responseRetrievalLocal Chroma DBRetrieved document context without a managed search serviceText-to-speechAWS PollyGenerated the response audioAvatar rendererGeneFace++Generated lip-synced video frames for the demoVideo transportWebRTC through Python aiortcStreamed low-latency video to the browserFrontendTypeScript and React with chat UI componentsCaptured audio and displayed the avatar conversation

The proposed architecture kept the avatar engine replaceable. We also evaluated SyncTalk and several other lip-sync systems. GeneFace++ powered the measured demo stack.

The source notes did not record the microphone codec, sample rate, chunk duration, VAD thresholds, WebRTC topology, or concurrent-session capacity. Those values belong in a production specification before another team tries to reproduce the...