From cron job to pipeline in 20 minutes

You already wrote the query. It works. It lives in a .sql file, or in your client's history, and something runs it — a cron entry on a box someone set up, or you, on Monday mornings.

That works right up until it doesn't: the query changes and nobody knows which version ran, it fails at 3am and nobody finds out until a stakeholder asks, or the person who set up the cron leaves.

Here is the same query as a scheduled, versioned, monitored pipeline. It takes about twenty minutes, and you don't have to learn a DAG framework to do it.

1. Connect the database (2 minutes)

Credentials → Add Integration, pick your database, fill in the connection.

Two things worth knowing before you start. Connectors that look duplicated usually differ by how you authenticate: "PostgreSQL — Host / Port" wants host, port, database, user and password, while the connection-string variant wants a single postgresql:// URI. Pick whichever matches the credential you actually hold.

And create a dedicated user for this rather than reusing your own. Grant it only the tables it needs. You'll want to rotate it one day without wondering what else breaks.

Hit Test before saving. A connection that can't be reached fails later, at execution time, where the cause is much harder to see.

2. Paste the query in (5 minutes)

Data Tools → Data, then Tools → SQL Modeling. Name the step something you'll recognise in a list of thirty, and paste the query in.

Pick your connection in the Source dropdown, then hit Run. The pill next to Run says QA, and that's deliberate: running defaults to your QA values, not production. An accidental click on a half-finished query doesn't reach the live database.

If your query has anything environment-specific in it — a schema name, a date window, an account id — replace it with a {{ placeholder }} now rather than later:

SELECT customer_id, SUM(amount) AS revenue
FROM {{ schema }}.orders
WHERE created_at >= NOW() - INTERVAL '{{ window_days }} days'
GROUP BY 1;

Save the step. That's version 1.

3. Give the placeholders values (3 minutes)

Open the Variables tab. Every {{ placeholder }} you used shows up as a row with a column per environment, and the save flow won't let you leave QA blank.

Fill in QA and PROD. They can be the same value; the point is that the choice is written down rather than implied. Substitution happens on the server at run time, using the environment you picked, so that selector next to Run isn't decoration.

4. Schedule it (5 minutes)

Deployments → New Deployment. Point it at your step, choose how it runs:

• CRON — on a schedule, in your organization's timezone
• On demand — when someone clicks
• API — when something else calls it

Pick the environment it runs in. A deployment pins a version. That's the part the cron job could never do: editing your step tomorrow creates version 2, and the deployment keeps running version 1 until you move it. No more wondering which SQL actually ran.

5. Find out when it breaks (5 minutes)

This is the part the cron job never had.

Open the Dashboard. Steps w/ Errors is the first card, and its report is what you land on: failed steps in the last 30 days with the real error attached. Not "exit code 1" — the message the database itself returned, which is usually the fastest way to the cause.

Monitoring → Runs is the same view full-page. Workflow Executions answers the other question: did the whole thing run, and what set it off.

What you have now

The same query, plus the parts that were missing:

Before	After
SQL in a file somewhere	A named, versioned step
Env values hard-coded or edited by hand	QA and PROD values, substituted at run time
A cron line on a box	A deployment pinned to a version
Silence when it fails	The database's own error, on the Dashboard

You didn't write a DAG, learn a scheduler's DSL, or stand up infrastructure. The query is still just SQL. You can read it, and so can whoever inherits it.

Next

Once you have two steps, you'll want the second to run only if the first returned rows. That's a workflow, and it's worth its own post.

---

🧩 Keywords

sql pipeline, cron replacement, scheduling, data pipeline, versioning, postgres