The pipeline step that decides whether to publish

The nightly load finishes at 03:40. By the time you look, the mart has been rebuilt and the dashboard is showing whatever came out.

Most mornings that is fine. The awkward mornings are the ones where the numbers are odd rather than wrong: revenue down a third, three thousand rows short, a plan that has stopped appearing. Sometimes that is a bad load. Sometimes it is the first of the month, or a customer who churned, or a genuine quiet week.

You already know what you do about it. You look at the numbers each morning and decide. This post is about moving that decision into the pipeline — and, just as importantly, about when you should not.

First: if the rule is a threshold, it is not this

Write an assertion.

If you can express the check as "no orders may have a null customer", or "the load must write at least 100 rows", then an assertion is strictly better than anything on this page. It is deterministic, it costs nothing to run, it cannot change its mind between Tuesday and Wednesday, and it either warns, pauses or stops the run depending on what you chose.

Reach for a model only when the judgement genuinely resists a threshold — when it depends on several numbers at once, on what normal looks like for this month, on the kind of thing you would explain to a colleague in a sentence rather than encode in a WHERE clause.

That is a narrower set of cases than it first appears. It is worth being honest with yourself about which one you are in, because the deterministic check is cheaper and more reliable, and reaching past it for a model is the most common way this feature gets misused.

The shape

Three steps.

1. A SQL step that summarises the load. Row counts, nulls on the keys that matter, the month-on-month delta. It returns rows, like any step.
2. An agent step that reads that summary and answers whether the numbers are safe to publish.
3. The publish step, which runs only if the answer was yes.

The agent is a step like the other two. It has inputs, it produces output, and the steps after it can read what it produced.

Building the agent

Data Tools → Agents → New Agent → Pipeline Agents. You need a model credential first — your own key, with Claude, Gemini, OpenAI, Perplexity or xAI Grok.

The parts that matter:

Role is the standing instruction, applied every run:

You are a careful data engineer reviewing a nightly load before it is published to downstream reports.

Prompt Template is the question, with a placeholder where the data goes:

Last night's load produced this summary:

{{ load_summary }}

Decide whether these numbers are safe to publish.

Response Format is the part people skip, and the part that makes this a pipeline step rather than a chat:

{ "proceed": true, "reason": "one short sentence" }

Those keys are what the rest of the pipeline is allowed to ask about the answer. Without them you get prose, which is readable but not branchable.

Wiring it up

In the pipeline, connect the summary step to the agent step, then double-click the agent step and declare its input: which upstream step, which field, and the name the prompt uses — load_summary.

That declaration is the whole data contract. The agent is sent what its prompt asks for by name, and nothing else. Your result set does not travel to the model because it happens to be in the same pipeline; a field nobody declared is not sent. If the prompt asks for something nothing supplies, the step fails and names it, rather than handing the model the literal text {{ load_summary }} and getting a confident answer about nothing.

Then double-click the edge leaving the agent and branch on proceed.

Test it before you trust it

There is a Test button in the agent editor. It sends the prompt to the model there and then, and shows you what went and what came back.

Use it. A prompt that reads perfectly to you can still come back in the wrong shape, and the 03:40 run is an expensive place to discover that. Test calls are billed like any other call, which is the correct incentive: it is a real call, so it is a real cost, and it is a cost you chose at a moment you will remember.

Make being wrong safe

This is the part worth designing deliberately.

The agent will sometimes be wrong. Build the branch so that the wrong answer is the cautious one: not publishing when it should have published costs you a morning; publishing a broken mart costs you whatever reads from it.

That falls out of the structure if you let it. The downstream step runs only when the condition passes, and an agent step that goes wrong — a dead model, a missing input, a deactivated credential — fails the run rather than skipping. So every failure mode lands on "did not publish", which is the one you can recover from over coffee.

What you should not build is the inverse: a pipeline that publishes unless the agent objects. That reads almost the same on the canvas and fails in the opposite direction.

What it cannot do

An agent step has no tools. It cannot query a database, call an API, or act on anything. It is given a prompt and it returns an answer. There is nobody present at 03:40 to approve an action, so we do not give it any to take.

That constraint is the reason this is a reasonable thing to put in a pipeline at all. The agent's blast radius is one JSON object that a condition reads.

---

Full details in the Agents documentation, and what is and is not sent to a model is spelled out on the Security page.