Building Real-Time Voice Agents from Scratch - Learning Roadmap | Nemorize

← Back to Roadmaps

Learning Topics

This roadmap covers the following topics:

✅ Part I: Foundations ✅ Shape of a Voice Agent⚪ mic → ASR → LLM → TTS Loop

⚪ Trade Matrix

✅ Audio Fundamentals⚪ SR_IN vs SR_OUT

⚪ float32 ↔ int16 Conversions

✅ VAD: Detecting Speech⚪ Threshold Tuning

⚪ Pre-roll Buffer

✅ Part II: The Pipeline ⚪ ASR with faster-whisper⚪ Model Size Trade-offs

⚪ ASR as a Blocking Call

⚪ LLM Streaming & State⚪ Speakable System Prompt

⚪ The Commit Pattern

⚪ TTS & Latency Trick⚪ pop_sentences Deep Dive

⚪ Kokoro vs Piper Backends

✅ Part III: The Hard Parts ⚪ Barge-in: Interruption⚪ Yield-Point Latency

⚪ Cancel Wire Protocol

⚪ The Feedback Loop⚪ Browser AEC

⚪ Playback State Machine⚪ Three Distinct Moments

✅ Part IV: Engineering It Well ⚪ Frontend Audio Scheduling⚪ AudioWorklet for Mic Capture

⚪ Gapless playHead Scheduling

⚪ Concurrency & Orchestration⚪ run_in_executor Pattern

⚪ asyncio vs Threads — Same Shape

✅ Part V: Make It Yours ⚪ Capstone Extensions⚪ Measurable Latency Fork

⚪ Extension Projects

⚪ The Production Bridge⚪ Trade-offs You Now Own

⚪ Why Hosted APIs Choose as They Do

Community Feedback

Share your thoughts and rate this roadmap

Your rating:

Cancel<br>Post Comment

Sign in to share your feedback and rate this roadmap

Sign In

Loading comments...

No comments yet. Be the first to share your feedback!

Node Title

&times;

Add Child Node

&times;

Adding child to:

Title

Description (optional)

Node Type

Branch<br>Leaf

Side

Left<br>Right

Cancel<br>Add Node

Generate Content

&times;

Generating content for:

Configure what to generate for this node.

Include lesson content

Generate educational text with explanations, examples, and diagrams

Number of questions:

Words per section:

Min:

Max:

Words per section (max capped at 3000)

Number of sections:

Min:

Max:

Number of sections in the lesson (2-10)

Cancel

Generate Content

Generating lesson and questions...

This may take a minute or two.

Lesson

&times;

Loading lesson...

Ready to practice?

This lesson has 0 questions

📑<br>ToC

📑<br>Contents

&times;

&times;

Sign in to Nemorize

Choose your preferred sign-in method

Continue with Google

Continue with Microsoft