Skip to content

Change Signal Unification

Status: implemented (REQ-932, 2026-07). Tracked in docs/arch/requirements.yaml.

Problem

The inbound question "how do we learn a source changed" is encoded in three places with overlapping value sets:

Signal Values Runtime role
change_signal (Table/Source) ttl probe ttl_probe native debezium kafka none (captured, inert)
MV freshness_mode ttl probe ttl_probe refresh gate
live.strategy poll native debezium kafka subscription provider

change_signal is the union: ∩ freshness_mode = {ttl, probe, ttl_probe} (identical), ∩ live.strategy = {native, debezium, kafka} (identical). poll is ttl/probe with a watermark. The same axis is expressed three times, and change_signal — the field designed to be authoritative — is read nowhere.

Target model

Three orthogonal axes:

Axis Field Decides
Inbound detection change_signal (resolve table.change_signal or source.change_signal) how we learn the source changed
Append vs replace + subscribable Table.watermark_column set → incremental append + subscribable; unset → full replace
Is it landed materialize / reachability / cache_ttl whether a local copy exists

freshness_mode and live.strategy derive from change_signal; they are no longer configured independently.

Derivation (provisa/core/change_signal.py)

resolve(table, source) -> str          # table.change_signal or source.change_signal
is_poll(sig) = sig in {ttl, probe, ttl_probe}
is_push(sig) = sig in {native, debezium, kafka}

to_freshness_mode(sig) -> str | None   # poll → same value; push → None (event-driven, no gate)
to_provider(sig, source_type) -> str   # native → source_type; debezium → "debezium"; kafka → "kafka"; poll → source_type

Push signals skip the freshness gate — their landed copy is updated by applying incoming events, not by polling.

Rewired sites

  1. MV/landing build derives freshness_mode from change_signal via to_freshness_mode(resolve(...)) at the MVDefinition(...) construction sites (api/app.py, api/admin/schema.py), threading probe_query. Previously these omitted freshness_mode and always defaulted ttl.
  2. Provider dispatch keys off change_signal. api/data/subscribe.py:_resolve_provider_type resolves via to_provider(resolve_effective(...)) rather than reading live.strategy directly.
  3. Poll-job reconcile keys off change_signal + Table.watermark_column. live/reconcile.py builds a poll spec when is_poll(sig); a watermark selects append vs replace landing.
  4. Validation keys off change_signal in core/config_loader.py: debezium/kafka require a source cdc block; push signals require a CDC-capable source or materialization store.

Delete / deprecate

  • live.strategy and live.watermark_column (core/models.py) are superseded by change_signal and Table.watermark_column. They are retained on the model as a legacy read-through: provider dispatch (api/data/subscribe.py) and poll reconcile (live/reconcile.py) go through change_signal.resolve_effective(...), which falls back to signal_from_strategy(live.strategy) only when no change_signal is set. change_signal is authoritative when present.
  • MV freshness_mode becomes an internal derived value only.

Landing (consumption layer, shipped)

change_signal.select_landing_shape(sig, watermark_column) maps the signal to a landing shape, applied by the DB-agnostic Core ops in federation/materialize_exec.py (driven by federation/store_writer.py) — not by engine CTAS in mv/refresh.py:

  • Replace: DELETE+INSERT — poll signal with no watermark.
  • Append: INSERT ... WHERE watermark > cursor — poll signal with a watermark_column set.
  • CDC-apply: consume the debezium/kafka provider and upsert/tombstone by primary key into the landed table via the dialect-agnostic Connection.upsert. This is what makes landed deletes work. (Subscription delivery grain remains insert/update only per REQ-928.)

Migration

V1, no migrations. Derivation happens at the read sites rather than by dropping the fields: change_signal is authoritative, and live.strategy/live.watermark_column remain on the model as a legacy fallback read through resolve_effective / signal_from_strategy. Existing JSONB rows carrying live.strategy keep working — the resolver maps strategy → change_signal when the latter is unset.

Sequencing (each phase leaves the tree green)

  1. Derivation module + rewire 1 (freshness_mode) + replace-path test.
  2. Rewire 2/3/4 (provider + reconcile + validation) + publish-downstream test.
  3. New append + CDC-apply landing + append/delete test.
  4. Delete live.strategy/live.watermark_column.

Tests

Coverage is in unit tests (tests/unit/test_change_signal.py, test_materialize_landing.py, test_subscribe_publish.py); the integration tests named in early planning (test_landing_replace / append_delete / publish_downstream) were not created.

  • Derivation: change_signal → freshness_mode / provider / landing shape for six values plus inherit.
  • Publish downstream: a Debezium/Kafka event through the resolved provider reaches the SSE stream and a Kafka sink.
  • Landing replace: poll signal, no watermark, full rebuild.
  • Landing append + CDC-apply: watermark signal → incremental insert; op=d tombstone removes the row.