GitHub - el10savio/obsIngest: A real-time, high-cardinality observability ingestion pipeline with Otel, Kafka, Go and ClickHouse · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
el10savio
obsIngest
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>2 Commits<br>2 Commits
cmd/ingester
cmd/ingester
deploy
deploy
docs
docs
migrations
migrations
src
src
.dockerignore
.dockerignore
.gitignore
.gitignore
Dockerfile
Dockerfile
Makefile
Makefile
README.md
README.md
docker-compose.yml
docker-compose.yml
go.mod
go.mod
go.sum
go.sum
View all files
Repository files navigation
obsIngest
A real-time, high-cardinality observability ingestion pipeline. OpenTelemetry logs, metrics, and traces are generated, buffered durably in a Kafka cluster, consumed by Go ingesters, and stored in ClickHouse for exploration.
Architecture
Running
Needs Docker + Docker Compose (provision ~4 GB ; ≥ 8 GB if you also start Superset)
make provision # brings up infra, creates topics, runs migrations, starts the ingesters<br>make superset # brings up Superset to analyse the ingested events<br>make load RATE=2000 # varied logs/metrics/traces across several fake services
make load fans telemetrygen across services (checkout cart payment search auth) with<br>mixed severities and a failing payment service, so the dashboards show real variance.<br>Tune with make load RATE= DURATION=60s SERVICES="a b c".
Then open:
Grafana (system health): http://localhost:3000 (admin/admin)
Kafka Console : http://localhost:8090
Superset (data exploration): make superset-up, then http://localhost:8088 (admin/admin)
Grafana
Superset
1000-ft view
Telemetrygen : OTLP load generator; generates synthetic logs/metrics/traces at a certain rate to have real data ready to send through.
OTel Collector : Consumes telemetry and sends it to Kafka, with each signal having its own topic, thereby decoupling producer and storage.
Kafka : Deals with spikes and ensures fault-tolerance: 3 brokers, RF=3, so that one broker can fail and no data gets lost; consumers can replay messages from committed offsets.
Go ingesters : Independent consumer groups with scalability options. Consume OTLP messages, aggregate them, batch insert into ClickHouse; commit offsets after a successful insertion (at-least-once delivery).
On a failed insert the batch is dead-lettered to a .dlq topic and the ingester keeps running (no retry, no crash, no health flip). For simplicity the demo dead-letters immediately; an industrial setup would first retry through Kafka, republishing to one or more delayed retry topics (non-blocking, so the ingester never stalls or holds state) and only park a record in the DLQ once those retries are exhausted.
ClickHouse : Both storage and processing engine, optimized for high cardinality telemetry (sorting, compression, skip-indexes), with a 30 days TTL and roll-up/cardinality materialized views.
Superset : Data-plane exploration tool for ClickHouse; provisions a connection and pre-made dashboard automatically.
Prometheus + Grafana : System plane self-observability; each component exposes /metrics, Prometheus scrapes them and Grafana visualizes pipeline lag/health.
Scaling to 100M events/sec
Ingest parallelism : Raise partition count and add ingester replicas. The consumer group splits partitions across replicas, so throughput grows linearly until you hit the partition ceiling; then add partitions.
Buffer : Scale Kafka to a larger multi-broker (and multi-region) cluster; RF=3 + acks tuning keeps durability while brokers absorb the fan-in.
Sampling : Add head and tail sampling to discard certain events based on rules to reduce traffic volume and reduce network costs.
Store : Shard ClickHouse and replicate with ReplicatedReplacingMergeTree ON CLUSTER. The schema is written so ReplacingMergeTree → ReplicatedReplacingMergeTree ON CLUSTER is a near one-line swap, distributing both writes and queries.
Insert path : async_insert (with...