What shipped for launch
First of these. The format will stay the same: what changed, why it matters, and what's still missing. A changelog that only lists wins is marketing with a version number on it.
Field Notes¶
Practical writing for the engineer who ended up owning the data stuff.
How to get something specific done in Data Conductor, how we think about problems like testing and environments, and what changed in the platform this month. Short, concrete, and written against real pipelines rather than architecture diagrams.
Recent posts¶
QA and PROD without a second database
Ask a small engineering team how they separate test from production data work and you usually get one of three answers.
"We have a staging database." Sometimes true, and good, though it's often months out of date — so passing there proves less than people think.
"We comment out the production bit before running." Honest, and terrifying.
"We have report.sql and report_prod.sql." The most common, and the one that quietly rots: the two files drift, and nobody
can say which is correct.
None of these are laziness. They're what you get when environments are a convention rather than a mechanism. Here's the mechanism we chose instead.
From cron job to pipeline in 20 minutes
You already wrote the query. It works. It lives in a .sql file, or in your
client's history, and something runs it — a cron entry on a box someone set up,
or you, on Monday mornings.
That works right up until it doesn't: the query changes and nobody knows which version ran, it fails at 3am and nobody finds out until a stakeholder asks, or the person who set up the cron leaves.
Here is the same query as a scheduled, versioned, monitored pipeline. It takes about twenty minutes, and you don't have to learn a DAG framework to do it.
An Approach to Pipeline Testing
Data teams can learn a lot from software engineering. The same pyramid that guides unit, integration, and end-to-end testing for code also provides a blueprint for building confidence in your data pipelines