---
title: "Recipe: Data sync and ETL to a warehouse"
description: "TL;DR: On a schedule, this playbook pulls records from a source API, transforms them in a Code step, and loads them into your warehouse (Snowflake or BigQuery) with incremental updates. Trigger is a schedule; output is fresh, transformed data in the warehouse."
canonical_url: "https://docs.ballet.dev/articles/recipe-data-sync-and-etl-to-a-warehouse-XBdNlNuOsn"
md_url: "https://docs.ballet.dev/articles/recipe-data-sync-and-etl-to-a-warehouse-XBdNlNuOsn.md"
---
# Recipe: Data sync and ETL to a warehouse

**TL;DR:** On a schedule, this playbook pulls records from a source API, transforms them in a Code step, and loads them into your warehouse (Snowflake or BigQuery) with incremental updates. Trigger is a schedule; output is fresh, transformed data in the warehouse.

## Who this is for

Data teams that move data between systems and a warehouse on a recurring basis.

## What you'll need

- **Connectors:** Source API (HTTP), Snowflake or BigQuery.
- **Secrets:** Source API key, warehouse credentials.
- **Trigger:** Schedule (cron) for the sync cadence.

## How to build it with Ask Ballet

1. Open the Console on Home.
2. Paste the prompt below; set source, destination table, and cadence.
3. Review the plan and confirm.
4. Connect the source and warehouse, then set the schedule.

```text
Build a playbook that runs every hour. Call the Stripe API for charges created since
the last successful run (use a stored cursor/timestamp). Transform each record into our
warehouse schema in a Code step (flatten nested fields, convert currency to cents,
normalize timestamps to UTC). Upsert the rows into the Snowflake table analytics.charges
keyed by charge id. Store the new high-water timestamp for the next run and log the row
count.
```

## What the playbook does

1. **Schedule trigger** — runs on the configured interval.
2. **Code step (read cursor)** — loads the last high-water mark.
3. **HTTP step (extract)** — pulls new/changed records from the source.
4. **Code step (transform)** — maps records to the warehouse schema.
5. **HTTP/Code step (load)** — upserts into the warehouse and saves the new cursor.

## Variations and tips

- Use incremental sync with a high-water mark to avoid reprocessing all data.
- Make the load idempotent (upsert by key) so retries are safe.
- Surface the loaded tables in [Scheduled reports](/articles/recipe-scheduled-reports-and-readouts-rqFtYUEyJN).

## Related articles

- [Scheduled reports and readouts](/articles/recipe-scheduled-reports-and-readouts-rqFtYUEyJN)
- [How do I manage Schedules?](/articles/how-do-i-manage-schedules-NTU9cNrk8w)
- [How do I store credentials with Secrets?](/articles/how-do-i-store-credentials-with-secrets-1D8OUcCo0n)
- [How do I read runs and run history?](/articles/how-do-i-read-runs-and-run-history-WukfYP5du3)