Building Voice Agents in Regulated Environments

Ullas Sankhla

SubscribeSign in

Building Voice Agents in Regulated Environments<br>In a series of posts I am going to document my journey of building voice agents for lendeasy.ai

Ullas Sankhla<br>Aug 17, 2026

Share

Subscribe

Introduction

One of the most important parts of our technology stack at lendeasy.ai is voice agents. A collections voice agent has to be very carefully designed to make sure that it can ask for payment, negotiate terms with the customer on a smaller amount or on dates — and, most importantly, direct them to an appropriate path in case of dispute or hardship.<br>A collections voice agent should also be able to handle several kinds of informational questions, FAQs and account data specific questions as well.<br>In this post I will be covering the thought process of building an agent and the designs we have iterated over the last few weeks of refining our collections voice agent. I will not be covering system-wide reusable pieces like the voice gateway or the workflow engine — those are for another time. This post goes over useful concepts that I have extracted and can be used to build a voice (or even a more general) agent that is driven by data, context and sound data driven decision making principles.<br>When designing a voice agent we have to handle a lot of uncertainty in what a human (in this case a borrower) might say. A borrower might decide to pay off the entire loan, they may decide to pay only a part of it, they may also change their decision mid call and decide to defer a payment.<br>A borrower may also ask several informational questions out of order of the conversation line.<br>The voice agent needs to be able to handle all of these without losing track of where it was, what data it has collected and what next steps it needs to do.<br>Evals and Requirements

We started with a set of requirements and then consolidated those requirements with evals. These evals helped us ground the most basic request-response expectations from our application that maintain compliance.<br>A multi-step workflow definition

A typical agentic workflow will have several steps, where each step is a sub-agent, and it will have one or more tool calls for that sub-agent. In the case of a collections workflow, we will have a sub-agent that will verify the identity of the caller, another sub-agent that will do payment negotiation (how much can you pay?), another sub-agent for when the caller discloses hardship, another for authorizing a credit card payment and so on. The workflow edges dictate which node runs next once the current node finishes with a specific code. Some edges pass control back to the user to get an utterance, others will pass control to another subagent which can ask a followup question.<br>This works if we can figure out which step the borrower needs to be in using a classifier for utterance.<br>The intent classifier

The intent classifier maps the intent of the borrower’s utterance to one of the sub-agents (steps) of the workflow graph above. The intent classifier tells which specific agent to invoke. This is needed to make sure that partial utterance (like a “Yes”) to an availability question is not taken as a “Yes” for payment authorization. The intent classifier maps the intent explicitly to a step or signals continuation of a step (such as a response, affirmation or denial to a question asked in a previous turn). This also works really well for questions that arrive out of nowhere and are not defined in any workflow sub-agent. These are handled as interrupts.

The interrupt mechanism

The interrupt mechanism exists as a bunch of free-hanging nodes within our workflow graph. These are useful to redirect intent to a node that is normally not in the call flow. This typically maps to a “FAQ” question, which will interrupt the ongoing flow but resume the conversation back to where it was. Another example is, a request to not send any message on the phone, this is an ask that does not fit or change the current conversation flow - but is a preference that can be set out of order and then the regular conversation flow can be resumed.<br>This acts much like the interrupt mechanism for a processor in the classic computer architecture sense. We stack whatever we are working on - keep it on the side - we service the interrupt and then we resume processing.<br>A pivot mechanism

So far what we have described works very well when there is a linear flow for a call. However, most of these calls do not go in a linear fashion. A caller, once verified, would like to know what their options are before committing to a particular payment - if at all. Worse, they may pivot completely from one path to another as we have talked above.<br>These pivots will mean we have to backtrack all the information we have collected so far and start on another thread.<br>In order to solve this problem, we developed a path classifier. It is similar to the planner in a typical agentic application. The path...