The tiniest logging stack: Fluent Bit, Parquet and DuckDB – David Guerrero
The tiniest logging stack: Fluent Bit, Parquet and DuckDB
4 Jun 2026
A tiny logging stack for a tiny amount of logs, or more?
The often-recommended Loki can be quite heavy to set up and the highly available version has a complex architecture. I also tried Parseable which initially seemed simpler, though in the end it generated excessive traffic to and from S3 for my use case, with relatively slow query performance.
So what are our options for small environments, like a k3s cluster running on a few small nodes?
The tiniest logging stack.
Fluent Bit writes Parquet files into S3, DuckDB queries them. That’s it.
This is the tiniest logging stack that can aggregate logs, keep them in long-term storage, maintain good availability, and still scale to some extent.
For my 3-node setup the total memory footprint of Fluent Bit pods is under 100 MiB and CPU usage negligible. The object storage layer can be any S3-compatible alternative, like Scaleway in my case.
What’s in the diagram is technically all you need: Fluent Bit and an object storage bucket. However there are additional considerations to ensure a high query performance and plug a browsing interface in front of it.
Schema
Using Apache Parquet means you’ll need a schema. It’s better to keep it lean here as well, only defining the fields common to all logs.
For Kubernetes logs for example it could look like this:
time timestamp<br>log varchar<br>id varchar<br>namespace varchar<br>pod varchar<br>container varchar<br>node varchar
log is the raw log as a JSON string, which can also be queried (non-optimally) from DuckDB via its JSON processing functions.
id in this case is a UUID added by Fluent Bit (UUID_Key id), used later on for query-time deduplication.
Each log category might use a different schema, the important part here is to avoid mixing Parquet files with different schemas in a single query.
When updating the schema for a log category, you can also use union_by_name to perform a union over the fields in both schemas.
Too many (small) files
The core problem with each Fluent Bit daemon writing Parquet files to S3 is the sheer number of files this produces by default. Thousands of files can accumulate quickly and make any querying impossible in a reasonable amount of time.
Fortunately this can be greatly alleviated with some architectural patterns and appropriate configuration.
Partitioning
DuckDB supports Hive partitioning, which is the main step towards organizing our Parquet files and improving query performance.
The S3 output for Fluent Bit conveniently supports setting a matching key format:
s3_key_format /raw/containers/year=%Y/month=%m/day=%d/hour=%H/$UUID.parquet
Since the logs are always queried through a specific time window, time-based partitioning is the obvious choice here, with hourly partitioning being a good starting point. Using more dimensions (namespace, deployment…) would also make sense for larger volume use cases.
In this example containers is a log category. It’s not part of the partitioning as I consider each log category to possibly have a different schema, and thus the files shall not be mixed.
Only the necessary files will be read from S3 at query time, as DuckDB can perform partition pruning.
Buffering
By default the file count can still be fairly high within a given hourly partition. Buffering the logs for some time before writing them to S3 is the best way to address this issue.
For example we can configure Fluent Bit to only write files every 5 minutes, or when they reach a target size (whichever happens first):
Total_File_Size 5M<br>Upload_Timeout 5m
Note that Fluent Bit will do its best to flush the logs during clean shutdowns and not lose any logs even while buffering.
Buffering has of course implications on how soon you’ll be able to query new logs. An upload timeout of 5 minutes means logs can be delayed up to 5 minutes.
Aggregation
The small-file problem is amplified by the number of nodes you run: more nodes means more files, as each Fluent Bit daemon writes its own set.
A cleaner architectural solution can be achieved through Fluent Bit’s forward input and output. We can set up an aggregator deployment with a fixed number of pods, and have each daemon forward the logs to the aggregator service.
Even a single pod can be used for aggregation, as each daemon can also buffer logs for a while if the aggregator goes down for a short amount of time. Adequate monitoring can help ensure no logs are lost.
Ultimately the number of aggregator pods will also need to be balanced with the Parquet file size (e.g. 5-minute blocks), as these are the key parameters driving the number of files.
Compaction
Once we’ve finished writing log files to a given hour’s partition, we can compact them into a single Parquet file. This step is optional but helps in reducing the file count even more.
A...