How Discord Creates Insights from Trillions of Data Points

Discover3 of 9<br>Safety 4 of 9

Quests 5 of 9

Support 6 of 9

Blog 7 of 9

Developers 8 of 9

Careers9 of 9

Log InLog in

Download<br>Nitro<br>Discover

Safety

Quests

Support

Blog

Developers

Careers<br>Log In

App Store

Lorem ipsum dolor sit<br>Mi neque maecenas

Engineering & Developers

How Discord Creates Insights from Trillions of Data Points

Daniel Meas

November 18, 2021

Updated November 19, 2021<br>At Discord, the Data Platform team empowers the organization to analyze, understand, and leverage data to help Discord create space for everyone to find belonging. Discord uses data for a number of reasons: to identify bad actors and harmful communities; to develop insights that inform critical product and strategy decisions; and to train and assess the effectiveness of machine learning models (check out our Privacy Policy for more information on what information we collect and how we use it!). Without regular and rigorous analysis of how our product is used, our ability to make informed decisions about company strategy at scale would be severely lacking.

Raw data comes to us as production datastore exports and product telemetry data (over 15 trillion records so far and billions generated daily). When Discord was a smaller company and data use cases were simpler, it was somewhat tenable, if not ideal, to manually compute useful datasets as needed. Today, we process petabytes of data with 30,000 vCPUs in the cloud. To be useful, the raw data must be cleaned, privatized according to our data governance policies, and then transformed into a complex schema of thousands of precomputed tables in our 30+ petabyte data warehouse (we use Google BigQuery).<br>As of this writing, the part of the Data Platform team responsible for ingesting raw data and making it accessible consists of eight people and we were even fewer in number during the history described below. Given the team’s size relative to the rest of Discord, it was important to build a system that was self-serve and as automated as possible. This is the story of how we turned petabytes of raw data into a structured data warehouse and the system we built to maintain it, internally referred to as Derived.

Requirements and Approach<br>What we needed was a system for maintaining a complex Directed Acyclic Graph (DAG) of precomputed data—in our case, this meant a DAG of derived tables in our BigQuery data warehouse:

A derived table essentially represents a data transformation that may have predecessor tables in the DAG as input dependencies: in other words, a derived table definition may be thought of as a SQL SELECT statement that references raw data or other derived tables.<br>Assuming the DAG flows top to bottom, one might imagine that at the top of the DAG would be the raw data sources and lookup tables; in the middle, a core set of reusable “golden” core data tables (e.g. normalized daily sign-ups across platforms); and towards the bottom of the DAG, tables that are intended to be consumed directly in analysis, BI tools or machine learning models.<br>The DAG may contain thousands of tables, so it needs to scale.

Though the system would be broken down into a series of deliverable milestones, we wanted the eventual system to meet the following requirements:

Table updates should run as soon as new data is available (but no sooner!)<br>Maintain an audit trail of mutations to each table.<br>Include primitives for powering data lineage and a data catalog.<br>Intuitive, self-serve table modifications  for stakeholder teams like engineering, data science, and machine learning.<br>Allow for integration with  data access controls and support for  scalable privacy policy enforcement.<br>Ability to automatically export derived data to production datastores for use in Discord’s user-facing product.<br>Simple and easy to operate in the context of Discord’s infrastructure environment.

While existing solutions such as dbt, Airflow, and Looker solve for some of the above, we ultimately decided that we wanted a more custom solution that would integrate nicely with our existing systems  and give us the flexibility to extend to use cases beyond analytics.

We were already using Airflow to schedule batch jobs and to process simpler datasets, but we found the following limitations:

Writing jobs was complicated and required people to have a deep understanding of Python, SQL, and Airflow. This violated our requirement that DAG modifications should be self-serve.<br>Scheduling queries that depended on each other on different schedules was difficult to reason about (e.g., knowing when to update a table that is scheduled weekly and reads from a table that is updated monthly).<br>Knowing exactly where to insert your table build into the dependency graph and understanding its impact on other tables was not straightforward (e.g., knowing when tables you own should be backfilled because predecessor datasets owned by another team were incomplete, introduced...