Databricks vs AWS Native at 20TB a Day - The Data Bytes
The Data Bytes
SubscribeSign in
Databricks vs AWS Native at 20TB a Day<br>Ingestion, transformation, load, governance, BI, Genie AI, GTM timeline, maintenance, integrations and cold hard dollars. All of it. At scale.
Data bytes<br>Jun 20, 2026
Share
There is a particular kind of meeting that happens at every company around the time their data volumes start looking embarrassing. Someone opens a spreadsheet. Someone else says “have we considered Databricks?” A third person says “but AWS is already our cloud provider.” And then everyone stares at each other for forty five minutes before agreeing to “align offline.”<br>This article exists so you can walk into that meeting with actual numbers instead of vibes.<br>The Data Bytes is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Subscribe
We are benchmarking at 20TB of data processed per day. That is not a toy workload. It is not a proof of concept. It is the kind of number that makes your cloud bill a line item in the board deck, your data platform a hiring dependency, and your infrastructure choices something you will be living with for the next three to five years.<br>The two stacks we are comparing:<br>Databricks on AWS (Premium or Enterprise tier) versus AWS Native managed services meaning EMR Serverless, MSK, Kinesis, Glue, MWAA, Redshift, Lake Formation, and QuickSight assembled into something resembling a data platform.<br>INGESTION
At 20TB per day you are moving roughly 14GB per minute on average, with real peaks probably hitting 30 to 40GB per minute during business hours. This is where the two architectures immediately diverge in philosophy.<br>DATABRICKS APPROACH<br>Databricks uses Auto Loader for incremental ingestion from S3, Azure Data Lake, or GCS. It detects new files automatically, handles schema evolution without you writing a migration script at 11pm, and scales the underlying Spark cluster to match arrival rate. For streaming, Delta Live Tables handles Kafka or Kinesis as a source and writes directly to Delta Lake tables.<br>One service. One configuration file. One place to look when something breaks.<br>AWS NATIVE APPROACH<br>AWS gives you choices. Several of them. Whether you wanted them or not.<br>Kinesis Data Firehose for managed streaming delivery. MSK (Managed Kafka) for high-throughput streaming with more control. Kinesis Data Streams if you need custom consumer logic. AWS Glue for batch file ingestion from S3. Or you wire Kafka Connect to MSK to write to S3 and then use Glue crawlers to catalogue it.<br>All of these are legitimate tools. Using all of them simultaneously because your use case grew organically is a very normal way to end up with a very complicated ingestion layer.
This is the largest single cost difference in the entire comparison and the one most people miss when evaluating the two platforms. Kafka on MSK with 7-day retention and the EBS storage that comes with it is expensive. Auto Loader on Databricks is just compute.<br>Critical assumption: If you only need batch ingestion and not streaming, AWS Glue alone is $704/month versus Databricks Auto Loader at roughly the same compute cost. The gap collapses for pure batch. It opens wide for streaming.<br>TRANSFORMATION
Where 20TB of raw data becomes something a business analyst can actually use without crying.<br>DATABRICKS<br>Databricks runs Spark. Specifically it runs Databricks Runtime which is a heavily optimised fork of open source Spark, plus Photon, a vectorised query engine written in C++ that replaces the JVM-based Spark SQL engine for eligible operations.<br>Photon is not marketing. Multiple independent benchmarks show 2 to 5x speedup on aggregations, joins, and window functions compared to vanilla Spark. At 20TB per day that speed difference means either smaller clusters or faster completion times. Both translate to money.<br>Delta Live Tables adds pipeline orchestration, data quality expectations (think assertions on your data that fail loudly instead of silently corrupting downstream tables), and CDC (change data capture) support out of the box.<br>You write Python or SQL. Databricks handles dependency resolution, retries, and backfill.<br>AWS NATIVE<br>EMR Serverless is AWS’s managed Spark option. You submit a job, it spins up the compute, runs, scales down. No cluster management. Supports Spark 3.x, Hive, Presto.<br>What it does not have is Photon. You are running standard open source Spark which is fine but not faster than fine.<br>AWS Glue is available for smaller transformations. It uses a DPU (Data Processing Unit) model and is convenient for light ETL but becomes expensive and slow for heavy transformation at 20TB/day scale. Most data teams at this volume end up using Glue for cataloguing and metadata, not for the actual transformation work.<br>PERFORMANCE AT 20TB<br>Databricks with Photon: transforms 20TB in approximately 8 to 10 hours on a 25 node r5.8xlarge cluster<br>EMR Serverless without...