Building the AI Retrieval Infrastructure Behind 20 Billion+ Vectors at HubSpot

EngineeringBuilding the AI Retrieval Infrastructure Behind 20 Billion+ Vectors at HubSpot

Oleg Tereshin & Xin Liu

on<br>Jun 25, 2026

Semantic search plays a critical role in HubSpot operations and powers a wide variety of use cases, from agents and RAG (Retrieval Augmented Generation) to contact deduplication. Starting from the POC a few years back, it has grown to a scale of tens of billions of vectors, utilized by 38+ teams. With the growth of agent usage, semantic search capabilities have become even more important to ensure agents can quickly retrieve the information they need while maintaining retrieval quality. This article describes the journey of vector DB infrastructure, from a small POC to the tens of billions of entries powering hundreds of use cases across HubSpot.

What is VaaS?

VaaS (Vector as a Service) is HubSpot's centralized vector storage and search platform built on top of a vector database. It provides an API layer and features such as:

Access control

Embeddings generation based on the collection setup

Data versioning

Feedback collection

It sits in front of the vectors database, in our case Qdrant, and is directly exposed to the consumers. It handles access control, basic validation, embeddings generation and then proxies requests to Qdrant. For read operations, requests are handled synchronously, whereas for write operations the flow is asynchronous with minimal consumer lag.

VaaS platform high-level architecture

Why Qdrant?

HubSpot uses the Qdrant vector database for all semantic search operations. Qdrant is an open-source vector database that can be deployed on-premises and provides an extensive API covering most modern semantic search features, such as named vectors, hybrid search, multi-stage querying, and weighted reranking. It uses HNSW (Hierarchical Navigable Small World) as an ANN (Approximate Nearest Neighbor) algorithm, which provides very low latency at the trade-off of higher memory consumption.

Qdrant also supports an extensive set of cost-efficiency optimizations, such as quantization, on-disk storage options for payloads and vectors, and configurable precision. This allows us to tune the setup to find a balance between providing the required level of retrieval quality and associated costs.

Why we need to run Qdrant in-house

HubSpot has a lot of expertise in running self-managed open-source database systems in-house, and over the years our infrastructure team has built numerous solutions, rules, and best practices that we want to take advantage of. Having such an extensive infrastructure layer lets us seamlessly integrate with and benefit from all of HubSpot’s internal tooling, including tracing, cost tracking, rate limiting, and scaling. Running Qdrant in-house enables us to easily fine-tune the infrastructure to our needs and control costs by utilizing our corporate AWS rates. Finally, running in-house gives us full control over customer data and infrastructure, and ensures all of HubSpot’s extensive security best practices are applied to this data store.

Current scale

The VaaS (Vector-as-a-Service) platform is currently used by 38+ teams and has more than 200 indexes that run on a fleet of 140+ clusters across 5 regions and 2 environments. We isolate clusters by product teams to minimize the blast radius from failures affecting the entire platform. Collectively, the underlying vector database stores 20+ billion vectors across different regions, with the largest index holding 9.5 billion vectors and an average index size of 95 million vectors.

The platform handles 5,000+ RPS for writes, with spikes up to 100,000 RPS, and serves 1,000+ RPS for reads across all regions. The traffic can also be spiky, specifically in cases when large backfills run in parallel with the live traffic; therefore, the underlying infrastructure should be scalable and reliable to provide the best customer experience while supporting engineering use cases.

Total reads across all regions

Total writes across all regions

The Early Proof of Concept

Back in 2023, semantic search had just started to take off, and the internal POC was set up to run Qdrant alongside the required pipelines and an API layer on top of it. There was a limited number of consumers and a dozen Qdrant clusters that were relatively easy to operate. As a first solution, Helm was utilized to define per-cluster/environment setup and parameters such as number of replicas, disk space, RAM, CPU, etc., were adjusted manually. The cluster definition and related cluster artifacts (Kafka topics, workers, etc.) were defined in code and created manually. For a small fleet of clusters, it was a great solution that allowed us to iterate fast, utilize overprovisioning to handle traffic spikes, and keep the setup simple. All the clusters were defined as StatefulSets and shared the same namespace. At that time, Qdrant was already handling 2+...