What Four Hours of Focused Human-AI Engineering Actually Ships · Canonical<br>Receipts before claims. Fifteen models. Thirteen releases. One evening.

The hook

On May 30th I published v0.1 of a Databricks integration pack for Swamp, the new declarative automation framework from System Initiative. By the end of the same evening it was at v0.13, with fifteen models covering jobs, DLT pipelines, SQL warehouses, the full Unity Catalog tree, workspace secrets, permissions, DBSQL queries, and Git Repos. Every release scored 100/A on the platform's quality rubric. Every model was end-to-end smoke-tested against a real Databricks workspace. One release shipped a real bug; two releases later I shipped the fix. The whole thing took about four hours of focused work, paired with Claude.

This post is about what's in the pack, why it matters if you have Databricks alongside anything else in your stack, and what I learned about engineering velocity along the way.

What I shipped

Fifteen Swamp models, published to swamp.club/extensions/@mfbaig35r/databricks, with source at github.com/mfbaig35r/swamp-databricks. They cover the Databricks data engineering surface:

Compute and orchestration : jobs (Jobs API 2.2), DLT pipelines, SQL warehouses with statement execution

Workspace storage : notebooks and workspace files (two distinct Databricks object types, two distinct models, distinct upload semantics)

Unity Catalog : catalogs, schemas, tables, volumes

Access control : workspace permissions (jobs, pipelines, warehouses, notebooks, repos, queries, dashboards, alerts...), UC permissions (full grant model for catalogs, schemas, tables, volumes, functions, external locations, storage credentials)

Workspace secrets : scopes and keys, distinct from Swamp's own vault, with secret values never persisted in Swamp's data layer

DBSQL : saved queries that pair with the job model's sql_task.query.query_id field

Git Repos : workspace-attached repos with pull, branch switching, sparse checkout, and full lifecycle

Each model exposes the surface a real workspace user would touch. The job model covers Jobs API 2.2 with full task-type schema validation including for_each_task (recursive via Zod z.lazy), dbt_task, and spark_python_task. The uc_permissions model covers all 15 UC securable types with the changes-style add/remove PATCH semantics. The sql_warehouse model includes warehouse lifecycle and the SQL Statement Execution API for running queries from a workflow. Lifecycle-managing models expose idempotent create_or_update semantics with workspace-first reconciliation, which correctly handles delete-then-recreate and out-of-band workspace deletes.

Every release scored 100/A on Swamp's quality rubric. Every model was smoke-tested end-to-end against a real Databricks Free workspace before the pack moved to the next release. "Smoke" means the actual API path: notebook uploaded to the workspace, job created via the API, run triggered, run waited to terminal state, result validated. Not type-check. That discipline caught a genuine semantic bug in v0.8's create_or_update that I shipped a fix for in v0.11. More on that below.

One reference workflow ships alongside the models: a Met Museum API ingestion pipeline at examples/api-ingest/met-museum/, end-to-end validated, runnable on Databricks Free without credentials. It implements a universal pattern for pulling external HTTP APIs into Bronze + Silver Delta tables (rate-limited Spark fan-out via mapInPandas, raw-JSON Bronze for schema-drift immunity, typed Silver via SQL). Forking it for Stripe, GitHub, Salesforce, or any other API is a 45-90 minute exercise: change the config block at the top of the notebook, change the Silver SQL. Everything between stays.

Distribution is the part that isn't done. The pack has zero pulls as of writing. It's day one for distribution; the engineering artifact is finished. More on that at the end.

Why this exists

Databricks already has two ways to manage itself: the Terraform provider for infra lifecycle, and Databricks Asset Bundles (DAB) for in-Databricks pipeline definition. Both are good at what they do. Neither composes with anything outside Databricks, and neither helps you write the Python that actually does the ingest, transform, or training work inside a notebook.

The non-obvious answer the pack is built on: don't ship a generic ingest runtime. Let the agent write the notebook code on demand for the specific case, then orchestrate the agent-written notebook through Swamp's declarative graph.

The Met Museum example in the pack is the proof. I didn't hand-write the 200-line bronze ingest notebook. I described the task to Claude (pull the Met's open catalog into a Delta table, respect their rate limit, capture errors), Claude wrote the notebook, I reviewed it, smoke-tested it on Databricks Free, fixed one bug (the serverless RDD restriction), committed it. The pack's notebook.upload step then uploads that committed notebook to the workspace, and...