GitHub - el10savio/duckWatch: Real-time observability for DuckLake Ingestion levaraging CDC on the attached Postgres catalog using Debezium, a Go collector, Kafka and Clickhouse. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
el10savio
duckWatch
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>2 Commits<br>2 Commits
.github/workflows
.github/workflows
clickhouse
clickhouse
cmd
cmd
config/debezium
config/debezium
deploy/superset
deploy/superset
docs
docs
scripts
scripts
src
src
.dockerignore
.dockerignore
.gitignore
.gitignore
Dockerfile
Dockerfile
Makefile
Makefile
README.md
README.md
docker-compose.yml
docker-compose.yml
go.mod
go.mod
go.sum
go.sum
setup.sh
setup.sh
View all files
Repository files navigation
duckWatch
Real-time observability for a DuckLake levaraging CDC on the attached Postgres catalog database using Debezium, a Go collector, Kafka and Clickhouse.
Introduction
A file-based catalog (Iceberg, Delta) records only committed state: the snapshots and file<br>statistics that survived. DuckLake's catalog is a running database, so it additionally<br>exposes an entire class of signal that a file catalog structurally cannot produce at any<br>polling frequency, like in-flight transactions, live lock contention, and the commit<br>conflict/retry rate of optimistic-concurrency writers racing on the shared ducklake_snapshot. We leaverage CDC and a scheduled Go poller to ingest these events and populate ClickHouse via Kafka to get analytics on them.
Architecture
Path<br>Mechanism<br>Yields<br>Landed in
CDC<br>Debezium tails the Postgres WAL<br>committed catalog mutations (snapshots, file stats)<br>cdc_* tables
Stats<br>a Go collector polls pg_stat_activity / pg_locks / pg_stat_database<br>live in-flight / contention / conflict state<br>stat_* tables
Quick start
Requires Docker (Compose v2) and Go 1.25
more contention/conflict<br>make teardown # drop the connector and volumes">make setup # write path + observability plane, wired end to end<br>make superset-up # dashboard at http://localhost:8088 (admin/admin)<br>make load RPS=10 WRITERS=4 # re-run load; more writers -> more contention/conflict<br>make teardown # drop the connector and volumes
Superset
About
Real-time observability for DuckLake Ingestion levaraging CDC on the attached Postgres catalog using Debezium, a Go collector, Kafka and Clickhouse.
Resources
Readme
Uh oh!
There was an error while loading. Please reload this page.
Activity
Stars
stars
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Go<br>35.8%
Python<br>32.3%
Shell<br>20.0%
Makefile<br>7.7%
Dockerfile<br>4.2%
You can’t perform that action at this time.