Analytics datasets you declare, not pipelines you orchestrate.

Where the imperative code comes from

Almost nobody writes a pipeline because they wanted a program. They write one because the data arrived in a shape SQL couldn't address — a nested JSON response, a document with an array of sub-documents, a directory of files, a graph, an event stream — and shaping it took code. That first shaping step is where the procedure enters, and once a procedure is in the path, everything downstream inherits it.

Our premise is that the shaping step doesn't need code, because non-relational data is not infinitely various. It falls into a modest set of structure types, and each one has a short declarative rule for projecting it into rows and columns: the path to the collection in a response and the fields to lift out of it, the sampled document schema for a collection with the steward's explicit types overriding what was inferred, the crawl of a location that registers each CSV, Parquet, or SQLite file it finds. That is a declaration per source type, not a program per dataset.

Once everything is two-dimensional, the rest is SQL. Which is the whole point: SQL is a language a machine can read back, so lineage, governance, rewriting, and audit become properties of the definition. Determinism is still on you — SQL gives you enough rope, and this system will refuse the parts of that rope it can detect. After SQL, the only remaining problem is timing.

The other reason: getting data from A to B

The rest of the pipelines exist because something in one system was needed in another. That sounds like plumbing, and it isn't. Writing it forces a developer to settle questions that were never theirs to settle: which copy is now authoritative, how stale is acceptable, what happens when the pull fails, where a write is supposed to land. Those are policy decisions, and they get made in a script, one at a time, by whoever drew the ticket — then they harden, because the answer is now a program somebody depends on.

Here data movement is a property of the system rather than something you do. Registering a source resolves how it joins the query surface: reached live through a connector, read in place as a file, or landed into a store because it has no live representation at all. Nobody chooses that; it follows from what the source can do and what the engine can do, and the same source can resolve differently on a different engine without anyone rewriting anything.

The questions that used to land on a developer are settled once, as rules. A pull that fails with no fresh cached data is an error, not a silent serve of yesterday's numbers — stale is served only where a freshness policy explicitly says it may be. A write goes to the upstream system of record and invalidates the cached copy, because a cache is never allowed to act like the source of truth. A store the engine cannot read back is rejected at registration rather than discovered later.

Which is why re-pointing is uneventful. Move the source from Oracle to Postgres, move the target to a different store, switch the engine underneath — the strategy re-resolves and the movement follows. There is no pipeline to rewrite, because there was never a pipeline; there was a registration, and you changed it.

An imperative pipeline is already a hole

Most derived data is produced by a program: a script, a notebook, a job in an orchestrator. It runs, it writes a table, and the only durable record of what happened is the table plus a log line. Everything you later need to know has to be reconstructed by reading code and hoping it's the version that ran.

• How do you audit it? By reading the transform. The audit surface is a program, so auditing it is code review, repeated every time anyone changes it.
• What did it actually do? Whatever the code said on the day it ran. Which columns fed which output is a question you answer by tracing the source by hand.
• When did it do it? You get the job's start time, which is incidental — when the machine woke up — rather than semantic: the version of the data it consumed. Those two differ by however late the upstream was, and the log records the one that doesn't matter.
• What does the result mean? The name someone typed on the output table, and a wiki page written at a different time by a different person.

None of that is fixable with better discipline, because the artifact is a procedure. A procedure can only be understood by simulating it. Declared datasets invert that: the definition is the artifact, and every one of those four questions is answered by reading the model instead of the code. A pipeline tells you how a number was computed. A definition tells you what the number is.

The four questions, answered from the definition

• What it did — column-level lineage is resolved statically from the definition: every output column back to the leaf source columns it derives from, with the intermediate CTEs and joins preserved as a graph. Each edge carries a named transform, matched against SQL operators, engine built-ins, and registered commands, so a derivation reads as concat → geocode rather than as a blob of SQL you have to parse yourself.
• When it did it — every refresh stamps two versions. A definition version, which is a content hash of the MV's logic, so you can tell whether two runs used the same rules. And an input version, captured at the strongest fidelity the source offers: an Iceberg snapshot id, else a relational watermark, else a freshness token, else the refresh clock. The fidelity kind is recorded alongside the value, so a lineage query knows whether it is looking at a precise data version or a wall-clock guess.
• How you audit it — those stamps land in the refresh ledger and the trace, which are queryable tables. Point-in-time lineage is a query you write, joined against the business data it describes, rather than a report a separate tool renders.
• What it means — a dataset is declared as an entity or a fact, with a key, a grain, and history, and that declaration lowers to the underlying view machinery. Meaning lives in the model that produced the table rather than in the name on it.

The declarative form also lets the system refuse definitions that could not be audited. A distributed MV whose SQL calls now() or random(), or takes a LIMIT with no total ordering, is rejected outright: two copies computing from the same sources would never agree, and a number nobody can reproduce is not a metric.

Layered by design

MVs stack on views and on other MVs, so your whole analytics layer becomes one declarative graph — expressed in the same model that describes and governs your data, not in a second tool with its own scheduler and its own idea of what your tables mean.

Recorded transforms

Each MV records the SQL or join that built it, plus the input signals it read — an Iceberg snapshot id, a relational watermark. Because the transform is recorded rather than merely executed, queries against the underlying tables are transparently rewritten onto a fresh MV when one covers them, partial join matches included. Nobody has to know the MV exists to benefit from it.

Refresh by freshness

An MV rebuilds only when an upstream input actually changed, and a change fans out through the graph to its dependents. Refresh is demand-driven rather than recompute-everything, which is where the compute bill for a derived layer usually goes.

Refresh by calendar

Or on a TTL or interval, when time rather than data is the trigger — a daily close, a monthly snapshot. Freshness and calendars are the only two knobs; there is no hand-maintained schedule to drift out of sync with the model.

A pipeline is a hole in your governance

A pipeline is a policy bypass. It reads governed data and writes a new table somewhere else, and that new table is outside the policy model — same numbers, no row filter, no masking, no record of who may see it. Every rule you wrote upstream stops at the write. So the derived layer becomes the reliable way around the controls, built by the team that was asked to make data more available, and nobody involved intended a bypass.

Declared datasets close it, because the derived table never leaves the model. Row-level security, masking, and relationship rules hold on an MV exactly as they hold on a source table. There is no second copy to govern separately, and no materialization that quietly outranks the policy that produced it.

Promotion carries data, too

Where a release also has to move data, a materialized view can carry the transform and land its output into a real source table, not just the internal store. True data migrations may still call for imperative scripts outside Provisa — the model promotes cleanly; the heavy data movement is yours to run.

"But ours is mostly declarative"

Often true. Plenty of orchestrators spend most of their runtime shuttling data between declarative SQL transforms, and the SQL in them is fine. The trouble is that the surrounding thing can still do anything, and what a system can do is what you have to audit it for. A mostly-declarative pipeline has to be reviewed as a program, because proving it stayed inside the declarative part means reading it — which is the cost you were trying to avoid.

And it never stays inside. The escape hatch is right there, the deadline is real, and one custom step is always the fastest way through — so the model gets a hook, then a macro that writes SQL, then a step that calls out to a script nobody revisits. Nobody decides to make the layer untrustworthy. It happens one reasonable exception at a time, and afterward you cannot tell by looking which datasets went through the clean path.

A closed vocabulary is the difference. When the only thing you can declare is a definition the system can read back, "is this reproducible" stops being a question about the team's discipline. It becomes a property the system can check — which is why a definition that reaches for now() or random() where reproducibility is required gets refused rather than reviewed.

Custom logic is fine. Undeclared custom logic isn't.

Real work needs code the model can't express: text pulled out of a PDF, an embedding from a model endpoint, a third-party API with its own rate limits and its own bad days. Banning that would be a fantasy, and it isn't the rule here.

The price of admission is a two-dimensional transform: models in, a model out, with data quality checked on both sides. Most of the ones you need already ship as declarations — that's what the source shaping rules are. What's left is the rare case, and it enters the same way everything else does.

Which means the result is always a DAG of models. Lineage isn't something a command interrupts; the call is a node with real edges, so the trace closes across it. That's the difference from a graveyard of stored procedures nobody can inventory. This is a live graph, and every exception in it is a labeled node you can go and look at.

What you end up with

None of this is a productivity feature. Writing the transform was never the expensive part; living with it was. A procedure has to be re-read every time someone asks what a number means, re-audited every time it changes, and worked around every time it turns out to hold data that policy says a person shouldn't see. That cost compounds per dataset, and it is why derived layers end as a pile of tables nobody fully trusts.

Declared datasets pay it once. The lineage is in the definition, the data version is stamped on the refresh, the policy is the same policy that governs the source, and the decision about whether to move data at all belongs to the system. What you have at the end is not a set of output tables and the scripts that made them. It is a model — and a model is a thing you can ask questions of.