Skip to content

How To

Files, CSV, Parquet and object storage

Most data work starts with a file somebody sends you. A CSV export, a Parquet drop in a bucket, a quarterly spreadsheet that arrives by email whether you asked for it or not.

Data Conductor has two ways to deal with that, and they are not interchangeable. Picking the wrong one is the difference between a job that takes thirty seconds and one that quietly does nothing.

From cron job to pipeline in 20 minutes

You already wrote the query. It works. It lives in a .sql file, or in your client's history, and something runs it — a cron entry on a box someone set up, or you, on Monday mornings.

That works right up until it doesn't: the query changes and nobody knows which version ran, it fails at 3am and nobody finds out until a stakeholder asks, or the person who set up the cron leaves.

Here is the same query as a scheduled, versioned, monitored pipeline. It takes about twenty minutes, and you don't have to learn a DAG framework to do it.