Agentic Airbyte | Visual Execution Tutorial

Skip to main content

Flow Board / intent becomes evidence<br>step 1 / plan

Agentic Airbyte execution flow<br>Goal enters the AI harness, the harness calls Crabbox, Crabbox injects a credential profile into a worker, Airbyte moves data from source to target, and evidence returns to the harness.

Goal<br>policy

AI harness<br>writes spec<br>chooses next action

Crabbox<br>lease + run<br>collect artifacts

Profile<br>scoped env

Worker<br>repo + env<br>Airbyte runs

Source<br>API / DB

Evidence<br>logs / JUnit<br>metrics / config

Target<br>warehouse

AI Harness Reads goal and writes a bounded spec.

calls Crabbox ->

Crabbox Leases a worker and injects the named profile.

runs command ->

Worker + Airbyte Reads source and writes target. Agent never sees rows.

returns evidence ->

Evidence Logs, metrics, JUnit, redacted config.

control = intent + command<br>credentials = profile -> env<br>data = source -> target<br>evidence = artifacts -> decision

1Plan<br>2Lease<br>3Inject<br>4Move<br>5Prove<br>6Repair

owner: agent<br>The agent compresses intent into a job spec.

It reads the goal, repo state, schemas, previous evidence, and policy. It emits a bounded run. It does not move data.

Input goal + policy + repo state

Output job JSON + Crabbox command

||

run replay<br>One job, traced from request to repair.

Click a row. The main flow jumps to the same boundary.

00:00<br>Request Sync CRM accounts into the warehouse before 08:00.

00:02<br>Spec Agent writes refs, profile, validation, retry, artifacts.

00:05<br>Lease Crabbox finds a warm worker and returns a run id.

00:31<br>Sync Airbyte reads source.crm.accounts and writes target rows.

02:14<br>Validate Worker emits counts, schema drift check, JUnit, config.

02:18<br>Decision Agent sees one failing freshness check and emits a repair job.

phase: request<br>Human or schedule gives intent.

The input is a business goal, not a connector config dump. The agent still has to choose the run shape.

Owner human / schedule

Input goal + deadline + policy

Output bounded intent for the harness

Evidence none yet

mental model<br>Everything is easier when each box owns one question.

Read the system as 4 contracts. Each box gets a narrow input, owns one decision, and emits a narrow output.

agent / driver<br>What should run next?

Turns intent and previous evidence into the next bounded job spec.

Reads goal, repo, evidence<br>Writes job spec, next action

crabbox / runner<br>Where can it run safely?

Leases the sandbox, hydrates code, injects scoped env, captures outputs.

Reads pool, command, profile<br>Writes run id, artifacts

airbyte / mover<br>How do rows move?

Runs the connector in the worker. Source and target never enter model context.

Reads source, target, env<br>Writes target rows, sync status

evidence / judge<br>What happened?

Converts logs and checks into finish, retry, repair, or alert.

Reads logs, JUnit, metrics<br>Writes decision, repair input

owner: agent<br>The agent is the driver, not the pipe.

It can inspect repo state and evidence, then write the next bounded job. It should never carry production rows or raw secrets.

Can do choose refs, profile, validation, retry, next action

Must not do move rows, store secrets, sanitize artifacts

Intent -> Spec<br>Goal becomes refs, profile, retry policy, validation, artifacts.

Spec -> Run<br>Spec becomes a sandboxed command with a durable run id.

Profile -> Env<br>Profile name becomes scoped variables inside the worker only.

Source -> Target<br>Connector moves rows directly. The prompt never becomes the data plane.

Worker -> Evidence<br>Execution becomes logs, JUnit, metrics, counts, redacted config.

Evidence -> Action<br>Signals become finish, retry, repair, or alert.

bad design<br>Agent becomes the integration runtime.<br>Secrets, rows, logs, and retries collapse into one prompt loop.

bounded design<br>Agent chooses experiments. Systems execute them.<br>Each boundary has one owner, one input shape, and one evidence shape.

red flags<br>The model context becomes the data plane.

The agent sees too much, owns too much, and cannot prove what happened. Debugging depends on an unstructured transcript.

Failure mode secret leakage, copied rows, non-repeatable retries

Fix move data movement back into a worker with scoped credentials and artifacts

runnable shape<br>The runnable shape has 3 contracts.

A useful agent output is not prose. It is a spec contract, an execution handoff, and an evidence contract.

ai-agent-dispatch.sh<br>Copy

# Goal: sync CRM accounts into the warehouse safely.

crabbox pool ensure example-org/data-movement/main/provider/linux/etl \<br>--min-ready 3 \<br>--create -- \<br>--cache-volume airbyte-etl

mkdir -p .crabbox/generated<br>cat > .crabbox/generated/accounts-sync.json --json<br>crabbox artifacts download --out evidence/

job spec anatomy

Refs source + target<br>Profile scoped env<br>Proof artifacts + tests

Spec Contract plan

Agent writes: refs, profile name, allowlists, validation, retry, artifact globs. Agent never writes: raw secrets or copied rows.

Execution Handoff lease

Crabbox...