Requirements¶

Generated from docs/arch/requirements.yaml. Do not hand-edit.

Each requirement is anchored — link to one directly with #REQ-NNN (e.g. REQ-001). Inline REQ-NNN mentions deep-link to their section, and the docs search box indexes every requirement by id, category, and description.

1. Access Governance & Security¶

REQ-001 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Any authenticated identity can query using any supported language (GraphQL, SQL, pgwire, Arrow Flight). Data returned is governed solely by table/column visibility, RLS, and masking — there is no capability gate on querying itself. Relationship join enforcement applies unless the role holds the ignore_relationships capability. There is no query registry and no pre-approval concept.

Use case: Removing the query-execution capability gate means governance is expressed entirely through data-layer controls (visibility, RLS, masking), not through access lists.

Code: provisa/security/, provisa/compiler/

Tests: tests/integration/test_governance_integration.py, tests/unit/test_governance.py, tests/unit/test_registry_removed.py, provisa-ui/e2e/governance-core.spec.ts

REQ-002 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

Rights and Stage 2 governance enforcement is platform-level — applied to every query at compile time. No client path can bypass it without bypassing the server (REQ-266).

Use case: Platform-level rights enforcement prevents any client path from bypassing governance controls.

Code: provisa/security/, provisa/compiler/

Tests: tests/unit/test_governance.py, tests/integration/test_governance_integration.py

REQ-003 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

All queries and mutations are governed by user rights alone — table/view rights plus relationship rights. No registry membership or query approval is required for any operation.

Use case: Uniform rights-based governance lets users run any query their rights permit without approval overhead.

Code: provisa/core/

Tests: tests/unit/test_governance.py, tests/integration/test_governance_integration.py

REQ-004 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

Test endpoint accepts arbitrary queries against registered schema with full guards; MUST NOT be exposed in production.

Use case: Isolated test endpoint lets developers validate queries safely without risking production exposure.

Code: provisa/core/

Tests: tests/unit/test_governance.py, tests/unit/test_test_endpoint_guard.py

REQ-005 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Result-size ceilings are defined per role/table in config (max_rows); Stage 2 injects LIMIT when a query would exceed the role's ceiling for any referenced table. Clients may always narrow further (fewer columns, additional filters).

Use case: Per-role/table row ceilings bound result size without an approved-query registry.

Code: provisa/core/

Tests: tests/unit/test_governance.py, tests/integration/test_governance_integration.py

REQ-006 · Query Governance¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Large-result redirect and Arrow output are available to any query the user's rights permit, subject to configured thresholds (REQ-029, REQ-137).

Use case: Large-result and Arrow output available to any rights-permitted query, governed by configured thresholds.

Code: provisa/core/

Tests: tests/e2e/test_large_result.py, tests/unit/test_governance.py, tests/unit/test_redirect_s3.py

REQ-038 · Security¶

Status: ✅ complete · Priority: MUST · Type: structural

Two independent enforcement layers: schema visibility and SQL enforcement (Stage 2). Both derive from the user's table/view and relationship rights.

Use case: Independent visibility and SQL-enforcement layers ensure governance cannot be bypassed by compromising one layer.

Code: provisa/security/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/unit/test_masking.py, tests/unit/test_rls.py, tests/unit/test_visibility.py

REQ-039 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Schema visibility layer: unauthorized tables/columns do not appear in SDL or query builder; compiler rejects at parse time.

Use case: Schema visibility enforcement ensures unauthorized tables and columns are invisible to clients at design time.

Code: provisa/security/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/unit/test_masking.py, tests/unit/test_rls.py, tests/unit/test_visibility.py, provisa-ui/e2e/governance-core.spec.ts

REQ-040 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

SQL enforcement layer: executor injects RLS WHERE clauses and strips unauthorized columns before execution, every request.

Use case: SQL-level RLS and column stripping ensure governance is enforced at execution time, not just at schema time.

Code: provisa/security/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/integration/test_rls_execution.py, tests/unit/test_inherited_roles.py, tests/unit/test_layer_contracts.py, tests/unit/test_masking.py, tests/unit/test_rls.py, tests/unit/test_sampling_rls.py, tests/unit/test_visibility.py

REQ-041 · Security¶

Status: ✅ complete · Priority: MUST · Type: structural

RLS rules defined at table registration as PG-style SQL filter expressions mapped to user roles.

Use case: Declarative RLS rules let stewards define row-level access without writing database triggers or views.

Code: provisa/security/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/integration/test_rls_execution.py, tests/unit/test_layer_contracts.py, tests/unit/test_masking.py, tests/unit/test_rls.py, tests/unit/test_visibility.py

REQ-042 · Security¶

Status: ✅ complete · Priority: MUST · Type: structural

Source registration, table registration, relationship definition, security configuration, query development, query authorization, and ignore-relationships rights are distinct and independently configured.

Use case: Fine-grained, independently assignable rights prevent privilege creep across registration and execution roles.

Code: provisa/security/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/unit/test_governance.py, tests/unit/test_masking.py, tests/unit/test_rls.py, tests/unit/test_visibility.py

REQ-203 · ABAC Approval Hook¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Pluggable operation approval hook for enterprises with complex ABAC that can't be expressed as static RLS rules. Evaluated at query time. Request: user_id, roles, session_vars, tables, columns, operation. Response: approved/denied + optional additional filter. Position: after RLS injection, before execution.

Use case: ABAC approval hook lets enterprises enforce attribute-based policies too complex for static RLS rules.

Code: provisa/security/

Tests: tests/integration/test_abac_hook_endpoint.py, tests/integration/test_governance_integration.py, tests/unit/test_abac_hook.py, tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py

REQ-204 · ABAC Approval Hook¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Approval hook scoping — per-table (approval_hook: true), per-source (approval_hook: true on source config), or global (auth.approval_hook). Compiler checks at query time: if no table in the query has approval_hook enabled (directly, via source, or global), skip the call entirely. Zero overhead for unscoped tables.

Use case: Scoped hook evaluation skips the approval call entirely for unscoped tables, keeping overhead at zero.

Code: provisa/security/

Tests: tests/integration/test_governance_integration.py, tests/unit/test_abac_hook.py, tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py

REQ-246 · ABAC Approval Hook¶

Status: ✅ complete · Priority: MAY · Type: structural

Approval hook protocols — three transport options, configured via auth.approval_hook.type (Part of Phase AE): - webhook (default): HTTP POST to URL. ~5-50ms. Simplest, works with any external ABAC service. - grpc: gRPC with persistent connection + multiplexing. ~1-5ms. Binary protocol for high-volume same-datacenter deployments. Proto service definition shipped with Provisa. - unix_socket: Unix domain socket for same-machine sidecars (e.g., OPA). <0.5ms. Path configured via auth.approval_hook.socket_path.

Use case: Three approval hook transports (HTTP/gRPC/Unix socket) let enterprises match latency requirements to ABAC infra.

Code: provisa/security/

Tests: tests/integration/test_governance_integration.py, tests/unit/test_abac_hook.py, tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py

REQ-247 · ABAC Approval Hook¶

Status: ✅ complete · Priority: MAY · Type: structural

Approval hook config (Part of Phase AE): auth.approval_hook.type: webhook | grpc | unix_socket; url for webhook/grpc; socket_path for unix_socket; timeout_ms: 500; fallback: deny | allow on timeout; scope: all or omit for per-table/per-source scoping. Per-table: tables[].approval_hook: true. Per-source: sources[].approval_hook: true. Global: auth.approval_hook.scope: all.

Use case: Granular approval hook scoping (global/source/table) avoids adding governance overhead to unscoped tables.

Code: provisa/security/

Tests: tests/integration/test_governance_integration.py, tests/unit/test_abac_hook.py, tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py

REQ-262 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: MUST · Type: structural

Compiler refactored into two explicit stages. Stage 1: GraphQL → plain PG-style SQL (semantic compilation only — resolves aliases, domains, relationships, emits physical column names and table names with AS aliases). Stage 2: plain SQL → governed SQL (AST rewrite applying RLS, masking, visibility, and the row cap). Stage 2 is a standalone transformer that accepts SQL + role and returns governed SQL, with no knowledge of GraphQL or the semantic model.

Use case: Two-stage compiler separates semantic translation from governance so Stage 2 can be reused by any SQL client.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_governance_integration.py, tests/unit/test_graphql_alias.py, tests/unit/test_stage2.py, tests/unit/test_two_stage_compiler.py

REQ-263 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Stage 2 governance transformer applies four governance concerns: (1) RLS — inject WHERE predicate per table reference per role (SQLGlot AST rewrite); (2) column masking — wrap masked column expressions with masking function (AST rewrite); (3) column visibility — remove or NULL-out columns invisible to the role (AST rewrite); (4) row cap — inject/cap LIMIT to the most restrictive per-role/table max_rows ceiling for any referenced table. A role without the FULL_RESULTS capability receives the configured default cap. The row cap is the single mechanism for bounding result size; statistical sampling is a user query feature, not a governance concern (REQ-478).

Use case: Stage 2 governance transformer enforces RLS, masking, visibility, and the row cap uniformly on every SQL path and transport.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_governance_integration.py, tests/unit/test_mask_inject.py, tests/unit/test_stage2.py, tests/unit/test_stage2_complex.py, tests/unit/test_two_stage_compiler.py

REQ-264 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Stage 2 must handle all SQL structural patterns: subqueries, CTEs, JOINs, SELECT * (expand via schema introspection then apply visibility), UNION, and nested expressions. RLS and masking must be injected at every table reference in the full AST, not only the outermost SELECT.

Use case: Stage 2 handling of CTEs and subqueries ensures governance is applied at every table reference, not just outermost.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/e2e/test_sql_endpoint.py, tests/integration/test_governance_integration.py, tests/unit/test_mask_inject.py, tests/unit/test_sql_endpoint_governance.py, tests/unit/test_stage2.py, tests/unit/test_two_stage_compiler.py

REQ-265 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Stage 2 operates on physical column names (left side of AS alias pairs). Stage 1 guarantees all aliases are explicit in emitted SQL, so Stage 2 never needs to resolve semantic model names.

Use case: Stage 2 operating on physical column names makes it independent of the semantic model and alias resolution.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/unit/test_stage2.py, tests/unit/test_two_stage_compiler.py, tests/integration/test_governance_integration.py

REQ-266 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: MUST · Type: constraint

With Stage 2 as a standalone SQL transformer, any SQL may enter the pipeline from any client — GraphQL, raw SQL via /data/sql, JDBC, DB-API, SQLAlchemy — and governance is uniformly enforced. No client can bypass governance without bypassing the server.

Use case: Stage 2 as standalone SQL transformer ensures no client can bypass governance without bypassing the server.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_governance_integration.py, tests/unit/test_sql_endpoint_governance.py, tests/unit/test_stage2.py, tests/unit/test_two_stage_compiler.py, provisa-ui/e2e/governance-core.spec.ts

REQ-267 · Two-Stage Compiler (Governed SQL)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A /data/sql REST endpoint accepts raw PG-compatible SQL + role (via auth), passes it through Stage 2 governance, routes and executes identically to the GraphQL path. Subject to the user's rights: any table or view the user's rights permit may be queried freely; Stage 2 governance is applied uniformly.

Use case: /data/sql endpoint lets raw SQL tools submit queries that still pass through full governance enforcement.

Code: provisa/compiler/stage2.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_governance_integration.py, tests/unit/test_sql_endpoint_governance.py, tests/unit/test_stage2.py, tests/unit/test_two_stage_compiler.py, provisa-ui/e2e/governance-core.spec.ts

REQ-369 · Rate Limiting¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Per-role rate limits configurable in provisa.yaml: max requests per second, max concurrent SSE subscriptions, max concurrent Arrow Flight streams. Limits enforced at the API layer before compilation or execution. Requests exceeding limits rejected with HTTP 429 and a Retry-After header.

Use case: Per-role rate limits prevent any single role from exhausting server resources or upstream connection pools.

Code: provisa/api/

Tests: tests/unit/test_rate_limiting.py, tests/integration/test_rate_limiting_integration.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-370 · Rate Limiting¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The NL query service (POST /query/nl) has an independent rate limit configurable via nl.rate_limit (requests per minute per role). LLM API cost is the primary driver — unbounded NL requests are a cost risk regardless of server capacity. Requests exceeding the limit are rejected before any LLM call is made.

Use case: NL rate limit caps LLM API cost exposure without restricting other query paths.

Code: provisa/nl/

Tests: tests/unit/test_rate_limiting.py, tests/integration/test_rate_limiting_integration.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-371 · Rate Limiting¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Rate limit state stored in Redis (existing cache.redis_url) using a sliding window counter. No rate limit state is held in-process — stateless Provisa containers (REQ-057) share limit state across all instances.

Use case: Redis-backed rate limit state ensures limits are enforced correctly across all horizontal Provisa instances.

Code: provisa/api/

Tests: tests/integration/test_cache_store.py, tests/unit/test_cache_store.py, tests/unit/test_rate_limiting.py

REQ-402 · Security¶

Status: ✅ complete · Priority: SHOULD · Type: structural

RLS rules can be scoped to a domain instead of a single table by setting domain_id (nullable FK to domains) instead of table_id (now also nullable); a single domain-level rule applies to all tables in that domain unless overridden by a table-specific rule.

Use case: Domain-level RLS rules reduce configuration burden when the same row filters apply across many tables in a semantic domain.

Code: provisa/core/schema.sql, provisa/core/repositories/rls.py, provisa/compiler/

Tests: tests/e2e/test_security.py, tests/integration/test_governance_integration.py, tests/unit/test_rls.py

REQ-531 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Masked columns are rejected from WHERE and HAVING clauses at query parse time, before execution, preventing inference of the unmasked value through binary search filtering. Enforced via V005 validation in the SQL validator (provisa/compiler/sql_validator.py). Applies across all query interfaces (GraphQL, SQL, Cypher).

Use case: Predicate guard closes the filtering side-channel so callers cannot infer unmasked values by submitting successive WHERE comparisons against a masked column.

Code: provisa/compiler/sql_validator.py

Tests: tests/e2e/test_security.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_stage2.py, tests/integration/test_security_integration_extra.py

REQ-554 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

All roles receive sampled results capped at DEFAULT_SAMPLE_SIZE (default: 100 rows) unless the role holds the full_results capability. The cap is controlled by the PROVISA_SAMPLE_SIZE environment variable and enforced via the Stage 2 row cap mechanism.

Use case: Default row cap prevents accidental large result delivery to unprivileged roles without requiring per-query LIMIT clauses.

Code: provisa/compiler/sampling.py, provisa/compiler/stage2.py

Tests: tests/e2e/test_security.py, tests/unit/test_stage2.py, tests/integration/test_security_integration_extra.py

REQ-555 · ABAC Approval Hook¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The gRPC approval hook maintains a single persistent channel per Provisa instance (one grpc.aio channel reused across all calls to that endpoint), reducing connection overhead for high-throughput co-located policy services.

Use case: Persistent gRPC channel eliminates per-call connection overhead for high-volume approval checks against a co-located policy service.

Code: provisa/auth/approval_hook.py

Tests: tests/integration/test_audit_auth_integration.py, tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-556 · ABAC Approval Hook¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The approval hook client implements a circuit breaker that opens after 5 consecutive failures (configurable via circuit_breaker_threshold) and enters half-open state after 30 seconds (configurable via circuit_breaker_cooldown_s), preventing cascading failures from a slow or unavailable hook endpoint.

Use case: Circuit breaker prevents a slow or unavailable approval hook from blocking all query execution when the fallback policy is 'allow'.

Code: provisa/auth/approval_hook.py

Tests: tests/unit/test_abac_hook_extended.py, tests/unit/test_approval_hook.py, tests/integration/test_security_integration_extra.py

REQ-557 · Security¶

Status: ✅ complete · Priority: MUST · Type: constraint

Data source credentials use ${env:VAR_NAME} syntax in the Provisa config YAML; values are resolved at runtime from environment variables via the secrets provider in provisa/core/secrets.py. Plaintext passwords are never stored in the config database.

Use case: Environment-variable secret references keep credentials out of config files and the config database, satisfying standard secrets hygiene requirements.

Code: provisa/core/secrets.py

Tests: tests/e2e/test_security.py, tests/unit/test_secrets.py

REQ-591 · Security¶

Status: ✅ complete · Priority: MUST · Type: constraint

SET LOCAL scopes app.tenant_id to the current database transaction. When the transaction commits or rolls back, the variable resets, preventing tenant context leakage across transactions.

Use case: Transaction-scoped tenant context prevents cross-tenant data leakage if middleware is bypassed or connection is reused across requests.

Code: provisa/core/db.py

Tests: tests/e2e/test_security.py, tests/unit/test_core_registration.py, tests/integration/test_security_integration_extra.py

REQ-594 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

TenantMiddleware defines a skip-path set {/billing/signup, /billing/webhook, /health, /docs, /openapi.json}. Requests to these paths bypass tenant resolution entirely and do not require a JWT with a tenant_id claim.

Use case: Skip-path exemptions allow Lemon Squeezy billing webhooks, health checks, and OpenAPI docs to function without tenant JWTs while all other paths enforce tenant isolation.

Code: provisa/api/middleware/tenant_middleware.py

Tests: tests/e2e/test_security.py, tests/unit/test_tenancy_requirements.py, tests/integration/test_security_integration_extra.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-596 · Audit Logging¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Every query is recorded in query_audit_log with tenant_id, user_id, role_id, SHA-256 query hash, table_ids, source, status_code, duration_ms, and logged_at. The table is append-only: PostgreSQL rules block DELETE and UPDATE at the database level. Query text is never stored verbatim — only its SHA-256 hash. Two indexes support tenant-scoped and per-user time-range queries.

Use case: Append-only audit log with hash-only query storage satisfies data minimization requirements while providing immutable evidence of query activity for compliance and investigation.

Code: provisa/audit/query_log.py

Tests: tests/unit/test_audit_requirements.py, tests/integration/test_audit_auth_integration.py

REQ-603 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

V002 relationship governance: every JOIN ON condition in SQL and Cypher queries must match an approved, registered relationship (source_col = target_col). SQL and Cypher queries that traverse a join not backed by a registered relationship are rejected at compile time. GraphQL queries that traverse relationships defined in the SDL are always pre-approved and exempt from the V002 check.

Use case: V002 enforces approved navigation paths so joins between sensitive tables require explicit steward registration and approval, providing an audit surface for cross-table data access.

Code: provisa/compiler/sql_validator.py

Tests: tests/unit/test_cypher_graph_fns.py, tests/unit/test_stage2.py, tests/integration/test_security_integration_extra.py

REQ-613 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Status: ✅ complete · Priority: MUST · Type: structural

The assignments_source config field controls how role assignments are resolved: claims reads role assignments directly from JWT token claims; provisa reads role assignments from Provisa's own internal assignment store. Defaults to claims.

Use case: Two assignment resolution modes let organizations choose between IdP-managed roles (claims) and Provisa-managed roles (provisa store) without changing auth provider.

Code: provisa/core/models.py, provisa/auth/

Tests: tests/unit/test_auth_providers.py, tests/integration/test_auth_integration.py

3. Source Registration & Data Modeling¶

REQ-012 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Source registration is privileged; validates connection, calls Trino dynamic catalog API, no restart required, available within seconds.

Use case: Privileged, live source registration lets stewards connect new data sources in seconds without downtime.

Code: provisa/core/

Tests: tests/integration/test_introspect.py, tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py, tests/unit/test_source_registration_validation.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-013 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

Source registration does not expose data — no table queryable until explicitly registered by a steward.

Use case: Data is invisible until explicitly registered, ensuring no accidental exposure of raw database tables.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-014 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

Unregistered tables do not exist — cannot be referenced in queries, do not appear in schema browser, cannot be mutation targets.

Use case: Unregistered tables are completely absent from schema and runtime, eliminating enumeration attacks.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-015 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

There is no per-table governance mode. Every table and view is queryable directly under the user's rights (table/view rights + relationship rights) with Stage 2 governance applied uniformly. No registry-required mode exists.

Use case: Uniform rights-based access removes per-table governance modes; every table follows the same model.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-016 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Table publication triggers schema generation pass; table immediately available in query builder.

Use case: Immediate schema availability after table publication removes friction for query developers.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-017 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

NoSQL/non-relational sources are exposed read-only through their native Trino connector (e.g. the MongoDB connector), driven by the type-specific mapping DSL (REQ-251); no mutations. (Revised 2026-06-18: the original "automatic Parquet materialization" wording is superseded by the connector approach, which is the implementation.)

Use case: Native Trino connectors make NoSQL sources queryable without manual ETL pipelines.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-018 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Trino FK metadata used to infer candidate intra-source relationships for steward confirmation/rejection.

Use case: FK-inferred relationship suggestions reduce manual steward work when registering related tables.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-019 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cross-source relationships defined manually by steward with cardinality (many-to-one, one-to-many). (Revised 2026-06-18: one-to-one removed — the relationship-field model is a strict binary, single object vs list, so a 1:1 collapses to many-to-one; model a true 1:1 as a many-to-one in each direction.)

Use case: Manual cross-source relationship definitions let stewards federate data across different databases with full control.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-020 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Relationships owned by defining steward, versioned, flagged for re-review on schema changes affecting join fields.

Use case: Versioned, owner-tracked relationships flag stale joins when schema changes, preventing silent data errors.

Code: provisa/core/

Tests: tests/integration/test_introspect.py, tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py, tests/unit/test_relationship_metadata.py

REQ-021 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: structural

GraphQL schema reflects registration model (business intent), not raw database structure.

Use case: Business-intent schema means the GraphQL API reflects concepts consumers understand, not raw table structures.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-119 · JSONB & API Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Stewards can promote specific nested fields from JSONB columns into native PostgreSQL generated columns (GENERATED ALWAYS AS ... STORED). Promoted columns are filterable, indexable, relationship-eligible, and auto-maintained by PostgreSQL. Supports dot-path extraction for nested fields.

Use case: JSONB field promotion to generated columns makes nested document fields indexable and relationship-eligible.

Code: provisa/api_source/

Tests: tests/unit/test_api_promotions.py, tests/integration/test_schema_compiler_integration.py

REQ-133 · Views (Governed Computed Datasets)¶

Status: ✅ complete · Priority: MUST · Type: structural

Views are SQL-defined computed datasets registered in the Provisa config with full column-level governance (visibility, masking, descriptions, aliases).

Use case: SQL-defined governed views let stewards expose computed datasets with full column-level governance applied.

Code: provisa/compiler/, provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/e2e/test_mv_optimization.py

REQ-134 · Views (Governed Computed Datasets)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Views go through the same governance pipeline as tables — RLS, masking, sampling, role-based schema visibility, approval workflow.

Use case: Views through the same governance pipeline as tables ensures computed datasets receive identical security treatment.

Code: provisa/compiler/, provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/e2e/test_mv_optimization.py, tests/integration/test_schema_compiler_integration.py

REQ-135 · Views (Governed Computed Datasets)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Views with materialize: true are backed by a periodically refreshed MV (CTAS). Views without materialization run as live subqueries via Trino.

Use case: Materialized views enable pre-computed result caching, while live views support real-time freshness needs.

Code: provisa/compiler/, provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/e2e/test_mv_optimization.py, tests/integration/test_registration_governance_integration.py

REQ-136 · Views (Governed Computed Datasets)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Views are the governed mechanism for adding computed semantics (aggregations, transformations) to the platform. This preserves the GraphQL constraint that no new semantics can be added outside the platform.

Use case: Views as the governed mechanism for computed semantics ensures all aggregations and transforms are auditable.

Code: provisa/compiler/, provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/e2e/test_mv_optimization.py, tests/integration/test_schema_compiler_integration.py

REQ-151 · Column Path Extraction¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Columns with path extract values from JSON source columns using PG >> syntax. SQLGlot transpiles to json_extract_scalar for Trino.

Use case: Column path extraction from JSON columns makes nested document fields queryable as first-class columns.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_sql_gen.py, tests/integration/test_schema_compiler_integration.py

REQ-152 · Column Path Extraction¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Path columns on PostgreSQL sources route direct. Non-PG sources force Trino routing.

Use case: Path column routing rules ensure JSON extraction uses the correct engine for each source type.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_sql_gen.py, tests/integration/test_registration_governance_integration.py

REQ-153 · Column Path Extraction¶

Status: ✅ complete · Priority: MUST · Type: constraint

Path columns are read-only computed fields — mutations unaffected.

Use case: Path columns are read-only, preventing mutations from targeting computed derived fields.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_sql_gen.py

REQ-154 · Naming & Schema¶

Status: ✅ complete · Priority: MAY · Type: structural

Optional domain_prefix prepends domain_id__ (double underscore) to all GraphQL names.

Use case: Domain prefix with double underscore lets stewards namespace all GraphQL names to avoid type conflicts.

Code: provisa/compiler/

Tests: tests/integration/test_schema_compiler_integration.py, tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-155 · Naming & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Table and column alias fields override GraphQL names.

Use case: Column and table aliases let stewards expose business-friendly names without altering underlying schema.

Code: provisa/compiler/

Tests: tests/integration/test_schema_compiler_integration.py, tests/unit/test_graphql_alias.py, tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-156 · Naming & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Table and column description fields included in GraphQL SDL.

Use case: Description fields in SDL give query developers in-schema documentation without leaving the GraphQL tooling.

Code: provisa/compiler/

Tests: tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-157 · Naming & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Order-by enum values preserve original column case (not uppercased).

Use case: Preserved original column case in order-by enums prevents case mismatch errors in consumer queries.

Code: provisa/compiler/

Tests: tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-158 · Auto-Materialized Relationships¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cross-source relationships with materialize: true auto-generate MV definitions at startup.

Use case: Automatic MV generation for materialized cross-source relationships eliminates manual ETL setup for joins.

Code: provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/unit/test_mv_refresh.py, tests/integration/test_registration_governance_integration.py

REQ-159 · Auto-Materialized Relationships¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Only cross-source relationships generate MVs. Same-source relationships are already fast via direct routing.

Use case: MVs only for cross-source relationships avoids unnecessary materialization overhead for same- source fast joins.

Code: provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/unit/test_mv_refresh.py

REQ-160 · Auto-Materialized Relationships¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Auto-MVs start STALE and are populated by the background refresh loop.

Use case: Auto-MVs starting STALE ensures the refresh loop populates them before they serve live traffic.

Code: provisa/mv/

Tests: tests/unit/test_mv_registry.py, tests/unit/test_mv_refresh.py, tests/integration/test_registration_governance_integration.py

REQ-194 · Naming Convention¶

Status: ✅ complete · Priority: MUST · Type: structural

The naming authority is the single source of truth for all client-facing names. Physical backend column names are never exposed to clients. Alias (column.alias in config): the canonical name. SQL uses it verbatim (no convention applied). GQL applies its configured convention to it. CQL derives from GQL. No alias: each query language derives its name from the physical column name via its configured convention. SQL convention: configurable, default snake_case. Applied via apply_sql_name(). GQL convention: configurable (apollo_graphql default = camelCase, hasura_graphql = snake_case). Applied via apply_gql_name(). CQL: derived from the GQL name.

Use case: Consistent multi-language naming with a single authority ensures client-facing names are predictable and independent of physical schema.

Code: provisa/compiler/

Tests: tests/unit/test_graphql_alias.py, tests/unit/test_naming.py

REQ-195 · Naming Convention¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Aligns with Hasura v2 naming_convention setting: hasura-default maps to snake_case, graphql-default maps to camelCase. DDN namingConvention: graphql maps to camelCase.

Use case: Hasura v2 parity — naming convention alignment lets teams familiar with Hasura conventions feel at home.

Code: provisa/compiler/

Tests: tests/unit/test_naming.py

REQ-250 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: structural

All Trino catalog configuration must flow through Provisa's config YAML — users never manage Trino .properties files directly. Provisa generates catalog properties, table definition files, and auto-registers tables at runtime following the Kafka source config pattern as exemplar.

Use case: Config-driven Trino catalog generation hides Trino properties files from operators entirely.

Code: provisa/core/

Tests: tests/e2e/test_kafka_flow.py, tests/integration/test_elasticsearch_introspect.py, tests/integration/test_introspect.py, tests/integration/test_nosql_catalogs.py, tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py, tests/unit/test_prometheus_source.py, tests/unit/test_redis_source.py, tests/unit/test_trino_catalog_files.py

REQ-251 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: structural

Each NoSQL/non-relational source type requires a type-specific mapping DSL in the Provisa config to define how its native data structures map to relational tables. Examples: Redis key patterns → rows, hash fields → columns; Elasticsearch index patterns with nested field paths; Prometheus metric names with label-to-column mapping.

Use case: Type-specific mapping DSL lets stewards define how Redis keys, Elasticsearch fields, and Prometheus metrics become SQL columns.

Code: provisa/core/

Tests: tests/integration/test_introspect.py, tests/integration/test_nosql_catalogs.py, tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py, tests/unit/test_table_search.py

REQ-363 · Semantic Layer / Semantic Model¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The SQLAlchemy dialect introspects table and column metadata via POST /data/graphql (GraphQL introspection query) instead of any admin or unfiltered endpoint. This ensures the dialect's get_table_names() and get_columns() responses are role-governed: the GraphQL server applies the Semantic Layer filter before returning the introspection result, so callers see only tables and columns their role is permitted to access.

Use case: SQLAlchemy dialect introspects via governed GraphQL endpoint; role determines table and column visibility.

Code: provisa/compiler/, provisa/security/

Tests: tests/unit/test_visibility.py, tests/unit/test_enforcement_metadata.py, tests/integration/test_registration_governance_integration.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-366 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Views require an approval workflow, OR the originator must already hold the rights to the underlying tables and to any joins used within the view. Any join within a view likewise requires approval or originator rights. Convenience views (adding no new semantics) are discouraged — instead grant the relationship rights and query in any form. Creating a view implies new semantics: derived/calculated values, or the view name itself as a new business concept. Approval gates therefore apply to both views (for the semantics they introduce, consistent with REQ-134) and relationships (for navigational intent).

Use case: View approval (or originator-rights) plus join approval ensures new semantics and cross-domain imports are governed; convenience views are discouraged in favor of relationship rights.

Code: provisa/core/

Tests: tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py

REQ-367 · Domain Model¶

Status: ✅ complete · Priority: MUST · Type: structural

There is only one view type — always intradomain (the output belongs to a domain). Two valid reasons to create a view: (1) the source is cross-domain (importing external data into the domain as a named business concept), (2) the source is local (deriving new or calculated data from existing domain assets). Two hard constraints follow: cross-domain data may only enter a domain via a view, and new or derived data may only exist as a view. No other reason to create a view is valid; there is no separate cross-domain view type — just views, always intradomain, used for exactly these two purposes.

Use case: Unified view model with two hard constraints ensures cross-domain imports are read-only adapters while computed data is always created within a single domain.

Code: provisa/compiler/sql_gen.py, provisa/mv/

Tests: tests/unit/test_domain_views.py

REQ-392 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Schema endpoint returns node_labels with a pk: string | null field per label, designating the primary key column name. Allows graph UI to construct reliable WHERE NOT n.<pk> IN [<value>] exclusion clauses (instead of heuristic-based id(n) fallback), enabling "Exclude from query", deep-linking to nodes, "show related to this entity", and inspector PK highlighting.

Use case: Explicit PK designation in schema enables semantic graph operations and node-centric navigation without node ID heuristics.

Code: provisa/api/rest/cypher_router.py, provisa/cypher/label_map.py

Tests: tests/unit/test_core_registration.py, tests/unit/test_graph_schema.py, tests/unit/test_graph_schema_pk.py, tests/unit/test_relationship_alias.py, tests/unit/test_schema_pk_requirements.py

REQ-393 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The semantic layer must support user-designated primary key columns per table column via an is_primary_key: bool field. Designation is informational only — Provisa does not enforce uniqueness; the underlying datastore is authoritative.

Use case: User-designated PKs enable semantic graph operations without requiring Provisa to validate database constraints.

Code: provisa/core/, provisa/compiler/

Tests: tests/unit/test_core_registration.py, tests/unit/test_domain_table_uniqueness.py

REQ-394 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Multiple PK checkboxes on a table infer a composite key; the first designated PK column is used as the canonical id_column for Cypher node identity resolution, taking priority over all heuristics.

Use case: First PK column as canonical id_column ensures consistent node identity across Cypher graph operations.

Code: provisa/cypher/, provisa/compiler/

Tests: tests/unit/test_core_registration.py

REQ-399 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: structural

When a Relationship is saved, the source_column on the source table is marked as is_foreign_key=true in table_columns. Two new boolean columns is_foreign_key and is_alternate_key are added to table_columns via ALTER TABLE ... ADD COLUMN IF NOT EXISTS.

Use case: Foreign key and alternate key tracking on columns enables semantic relationship metadata without external constraints.

Code: provisa/core/schema.sql, provisa/core/repositories/

Tests: tests/unit/test_core_registration.py

REQ-400 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When a Relationship is saved, the target_column on the target table is marked as is_primary_key=true if no other column in that table already has is_primary_key=true; otherwise it is marked as is_alternate_key=true.

Use case: Automatic PK/AK designation on relationship target columns reduces steward manual key configuration work.

Code: provisa/core/repositories/relationship.py, provisa/core/models.py

Tests: tests/unit/test_core_registration.py

REQ-413 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-generate GQL relationships from FK constraints in database schema introspection — relationships discoverable from FK metadata in addition to manual steward configuration and AI-assisted hints.

Use case: Auto-generated FK relationships reduce manual relationship registration overhead while preserving steward control.

Code: provisa/compiler/introspect.py

Tests: tests/unit/test_discovery_requirements.py, tests/integration/test_registration_governance_integration.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-414 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Demo/install example schema must include at least one FK relationship to exercise auto-generated relationship discovery.

Use case: Demo schema coverage of FK auto-discovery ensures feature is validated during onboarding.

Code: demo/files/create_demo_files.py

Tests: tests/e2e/test_installer.py, tests/unit/test_core_registration.py, tests/integration/test_registration_governance_integration.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-415 · Registration & Governance¶

Status: ✅ complete · Priority: MAY · Type: behavioral

The hasura_v2_relationship_style option controls whether FK-derived relationships use Hasura V2's naming conventions — singular for many-to-one, plural for one-to-many using inflection.

Use case: V2-compatible relationship naming enables teams to adopt Hasura conventions without schema migration.

Code: provisa/compiler/introspect.py

Tests: tests/unit/test_discovery_requirements.py, tests/unit/test_fk_relationship_style.py

REQ-417 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Hasura v2 migration tool (provisa/hasura_v2/mapper.py) maps Hasura Remote Schemas to Provisa graphql_remote source registrations instead of skipping them with "NOT SUPPORTED" warning. Provisa already has GraphQL Remote source capability (graphql_remote_adapter.py and graphql_remote_router.py); the mapper was a gap. Migration preserves Remote Schema name, URL, headers, and authentication configuration in the registered source.

Use case: Hasura Remote Schema migration removes GTM blocker — customers see supported sources instead of "NOT SUPPORTED" during migration path evaluation.

Code: provisa/hasura_v2/mapper.py, provisa/source_adapters/graphql_remote_adapter.py, provisa/api/admin/graphql_remote_router.py

Tests: tests/integration/test_graphql_remote_source.py, tests/unit/test_core_registration.py, tests/unit/test_hasura_remote_schema.py, tests/unit/test_hasura_v2_comprehensive.py, tests/unit/test_import_shared.py

REQ-418 · Domain Model¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Report authoring workflow: analysts pull cross-domain data into their own domain via views (data-import adapters), then define calculations and relationships only within that domain. New or derived data exists only as views; no direct cross-domain calculations or relationships.

Use case: Unified view model ensures domain logic stays local and views serve exactly two purposes: importing external data and creating derived data within a single domain.

Code: provisa/compiler/, provisa/mv/

Tests: tests/unit/test_domain_views.py

REQ-432 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

A registered table must be unique by (domain_id, table_name) — stronger guarantee than the DB constraint UNIQUE(source_id, schema_name, table_name). Table registration (admin registerTable/updateTable mutations) rejects any collision of domain+table against a different physical table. Startup/schema-rebuild validates the entire registry and fails if any duplicate exists.

Use case: Domain+table uniqueness prevents ambiguous table references across all query layers (GraphQL, Cypher, view SQL).

Code: provisa/core/

Tests: tests/unit/test_core_registration.py, tests/unit/test_schema_visibility_filters.py

REQ-433 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

A datasource may be associated with multiple domains. Any domain owner may register any unclaimed table from that source. Once a table is claimed by one domain, no other domain may register it — first-come ownership model. Unique constraint enforced on (source_id, normalized_table_name). The UI greys out claimed tables regardless of which domain claimed them.

Use case: Multi-domain with first-claim ownership prevents the same physical table from being registered multiple times while allowing flexible domain-to-source associations.

Code: provisa/core/, provisa-ui/src/pages/TablesPage

Tests: tests/unit/test_dataset_uniqueness.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-434 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Creation-request mechanism: any governed create operation (view, relationship, etc.) attempted by a user lacking the authority to perform it produces a persisted request rather than an error. The request enters a queue (REQ-063) visible to every user holding the rights to execute that create. An authorized user may execute or reject the request; rejection carries a specific, actionable reason. No create is performed until an authorized user executes the request.

Use case: Creation-request queue lets users without create authority propose views/relationships for an authorized holder to execute.

Code: provisa/core/, provisa-ui/src/

Tests: tests/unit/test_core_registration.py, tests/unit/test_creation_requests.py

REQ-542 · Naming & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Regex naming rules in config are applied to table names in order when generating GraphQL field names, before uniqueness resolution.

Use case: Ordered naming rules let stewards strip common prefixes (e.g., prod_pg_) from auto-generated GraphQL names without manual aliasing.

Code: provisa/compiler/naming.py

Tests: tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-543 · Auto-Materialized Relationships¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Mutations to source tables of a materialized cross-source relationship mark the corresponding MV as stale for re-refresh. The refresh_interval defaults to 300 seconds (5 minutes) when not explicitly set.

Use case: Mutation-triggered staleness ensures the MV is refreshed promptly after writes, keeping cached join results consistent.

Code: provisa/mv/

Tests: tests/unit/test_mv_refresh.py, tests/integration/test_registration_governance_integration.py

REQ-604 · Naming & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Relationships support an optional graphql_alias field that names the SDL field exposed on the parent type. When graphql_alias is absent, the SDL field name is derived from the target table's field_name and the relationship cardinality via rel_field_name(target.field_name, cardinality) (singular for many-to-one, plural for one-to-many).

Use case: graphql_alias lets stewards expose relationship traversal fields under business-friendly names independent of the target table's raw field_name.

Code: provisa/compiler/schema_gen.py

Tests: tests/unit/test_graphql_alias.py, tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-605 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When root_table_ids is set on a SchemaInput, tables whose IDs are absent from that set are excluded from root query fields in the generated SDL but remain present as GraphQL named types reachable via relationship fields. Domain access filtering applies the same mechanism: tables in inaccessible domains become type-defs only. Full removal of a type (including its definition) requires deleting the table registration entirely.

Use case: Type-def-only tables let stewards expose navigable object graphs where some types are reachable only via relationship traversal, not by independent client query, without requiring separate registration.

Code: provisa/compiler/schema_gen.py

Tests: tests/unit/test_visibility.py

REQ-609 · Domain Model¶

Status: ✅ complete · Priority: MUST · Type: constraint

Every domain must have a designated steward before it can serve governed data. A domain may exist in a pending state until a steward is assigned; a domain without a steward cannot expose tables, views, or relationships to consumers.

Use case: Steward requirement ensures accountability is always assigned before governed data is exposed.

Code: provisa/core/domain_policy.py

Tests: tests/unit/test_domain_policy.py

REQ-610 · Domain Model¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A field access grant belongs to the requesting domain, not to the specific view that prompted it. Any subsequent view in the requesting domain may use the granted fields without additional cross-domain approval. New fields not covered by the existing grant require a new request.

Use case: Domain-scoped field grants prevent repeated approval ceremonies for the same cross-domain data access pattern.

Code: provisa/core/

Tests: tests/unit/test_core_registration.py

REQ-611 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Discovery is structured across five tiers of increasing governance: (1) Registered source schema — raw inventory, admin-level visibility; (2) Unclaimed tables — introspected from registered sources with no domain owner; (3) Domain assets — claimed tables and steward-defined views, fully governed; (4) Relationships — approved traversal paths between Tier 3 assets; (5) Field grants — domain-to-domain field access permissions. Each tier is a prerequisite for the next.

Use case: Structured discovery tiers give stewards and analysts a clear mental model of what data is governed and at what level.

Code: provisa/core/, provisa/discovery/

Tests: tests/unit/test_discovery_requirements.py

REQ-612 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Relationship candidates are ranked by a four-level confidence hierarchy: (Highest) Approved catalog relationship validated by both stewards; (High) Intra-source FK constraint — explicit modeling intent; (Medium) Intra-source semantic inference — column name/type similarity; (Low) Cross-source semantic inference — naming conventions diverge, high false-positive risk. Candidates corroborated by multiple evidence types accumulate confidence.

Use case: Confidence hierarchy lets stewards prioritize which relationship suggestions to review first, reducing false-positive noise.

Code: provisa/discovery/analyzer.py, provisa/discovery/collector.py

Tests: tests/unit/test_discovery_requirements.py

REQ-635 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The schema name presented to users must be the name the data source itself uses to group datasets. For relational databases this is the native schema (or database for MySQL). For flat/API sources with no native grouping concept, a fixed constant naming the source type is used.

Use case: Schema names derived from native source grouping prevent user confusion and keep the semantic layer consistent with the underlying data source.

Code: provisa/api/admin/introspect.py

Tests: tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-636 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

When a Trino connector is configured for a source type (i.e., the type is in SOURCE_TO_CONNECTOR), Trino is the preferred path for schema and table introspection. Native driver introspection is only used for source types with no Trino connector or those that override via native_schemas/native_tables returning a non-None value.

Use case: Trino-first introspection ensures schema discovery uses the same path as query execution, keeping introspection results consistent with what is queryable.

Code: provisa/api/admin/introspect.py, provisa/core/models.py

Tests: tests/integration/test_introspect.py, tests/unit/test_core_registration.py, tests/unit/test_native_introspect.py, tests/unit/test_compiler_introspect.py

REQ-637 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

PG cache schemas, Trino catalog names, and other implementation-layer namespaces must never appear as schema or table names presented to the user. Only native source schema/table names and registered aliases are exposed.

Use case: Hiding implementation namespaces prevents users from seeing or depending on internal infrastructure details that may change.

Code: provisa/api/admin/introspect.py

Tests: tests/integration/test_introspect.py, tests/unit/test_catalog_cache.py, tests/unit/test_core_registration.py

REQ-638 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The UI calls one availableSchemas endpoint and one availableTables endpoint. Backend routing selects the correct introspection strategy per source type internally, with no source-type-specific endpoints exposed to the UI.

Use case: Single introspection endpoints for schemas and tables simplify the UI and centralize routing logic in one place.

Code: provisa/api/admin/schema.py, provisa/api/admin/introspect.py

Tests: tests/integration/test_introspect.py, tests/unit/test_available_tables.py, tests/unit/test_core_registration.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-639 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

A source type not explicitly handled by the introspection router must return [] (empty list) for both schemas and tables — never None. Returning None triggers a Trino fallback path that is not appropriate for non-Trino sources.

Use case: Returning empty list instead of None for unknown source types prevents broken Trino fallback execution on source types that have no Trino connector.

Code: provisa/api/admin/introspect.py

Tests: tests/integration/test_introspect.py, tests/unit/test_core_registration.py

REQ-640 · Naming & Schema¶

Status: ✅ complete · Priority: MUST · Type: constraint

Dataset name comparisons must always normalize both sides through the centralized normalization function (toSnakeCase in the UI via naming.ts; provisa.compiler.naming in the backend) before comparing. Inline ad-hoc normalization such as name.toLowerCase() or hardcoded replace(/-/g, '_') is forbidden. This applies to isRegistered checks, alias lookups, duplicate detection, and cross-source name matching.

Use case: Centralized normalization for all name comparisons prevents silent mismatches caused by different casing conventions across source types.

Code: provisa-ui/src/naming.ts, provisa-ui/src/pages/TablesPage.tsx, provisa/compiler/naming.py

Tests: tests/unit/test_naming.py, tests/unit/test_schema_service.py

REQ-641 · Registration & Governance¶

Status: ✅ complete · Priority: MUST · Type: structural

Every registered dataset has two identity forms: (1) physical identity — the native name as returned by the source, used by the query engine; (2) semantic identity — domainId + tableName (alias, normalized), used by the API, GraphQL schema, and end users. Resolution between forms uses registered_tables: physical-to-semantic via (sourceId, normalizedPhysicalName); semantic-to-physical via (domainId, tableName).

Use case: Dual identity model with centralized resolution prevents ad-hoc string comparisons that break across naming conventions and source types.

Code: provisa/core/repositories/table.py

Tests: tests/unit/test_graphql_alias.py, tests/unit/test_naming.py

4. Source Connectors¶

REQ-147 · Kafka Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Kafka topics queryable via Trino Kafka connector. Routed through Trino (TRINO_ONLY source).

Use case: Kafka topic querying via Trino lets analysts query streaming data with the same SQL and governance as relational tables.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_schema.py, tests/integration/test_kafka_source.py

REQ-148 · Kafka Sources¶

Status: ✅ complete · Priority: MUST · Type: constraint

Default time window (default_window) auto-injected as WHERE clause on _timestamp. Prevents unbounded reads.

Use case: Auto-injected time-window filter prevents unbounded Kafka topic scans that would exhaust cluster resources.

Code: provisa/kafka/

Tests: tests/integration/test_kafka_source.py, tests/unit/test_kafka_schema.py, tests/unit/test_kafka_window.py

REQ-149 · Kafka Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Discriminator filter for multi-type topics — multiple table configs on the same physical topic, each filtered by a discriminator field/value.

Use case: Discriminator filters let a single physical Kafka topic serve multiple logical table definitions safely.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_schema.py, tests/integration/test_kafka_source.py

REQ-150 · Kafka Sources¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Manual schema definition for topics without Schema Registry.

Use case: Manual schema definition for Schema Registry-less topics lets stewards govern message structure explicitly.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_schema.py, tests/integration/test_kafka_source.py

REQ-229 · Direct-Route Dialect Expansion¶

Status: ✅ complete · Priority: MUST · Type: structural

For every source type, three things must be true for direct-route capability: (1) Trino connector is packaged in the Trino deployment, (2) SQLGlot has the dialect for transpilation, (3) SOURCE_TO_DIALECT and SOURCE_TO_CONNECTOR entries exist in Provisa config. New direct-route sources to add: clickhouse, mariadb, singlestore, redshift, databricks, hive, druid, exasol. File and lake sources (iceberg, hive_s3, delta) are TRINO_ONLY — no direct-route, no SQLGlot dialect entry required; only Trino connector packaging and Provisa config entries needed. Each source must have all applicable items verified.

Use case: Verified direct-route support for new RDBMS dialects lets data engineers connect more source types without Trino. File and lake sources require only connector and config entries since they always route through Trino.

Code: provisa/transpiler/

Tests: tests/unit/test_dialect_expansion.py

REQ-295 · Query-API Sources (Neo4j & SPARQL)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Neo4j registered as a non-Trino source using the existing API source pipeline. Source config accepts Neo4j HTTP API endpoint URL, target database name, and auth (none/basic/bearer token). Tables defined as steward-authored Cypher SELECT queries. The Neo4j HTTP API v2 response envelope (data.fields / data.values) is unwrapped by the built-in neo4j_tabular response normalizer into flat row dicts before the existing flattener runs. Column names match the aliases declared in the Cypher RETURN clause. Does not use Trino.

Use case: Neo4j registered as an API source lets stewards expose Cypher query results as governed GraphQL tables.

Code: provisa/neo4j/, provisa/sparql/, provisa/api_source/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_neo4j_exec.py, tests/integration/test_sparql_exec.py, tests/unit/test_neo4j_normalizers.py

REQ-296 · Query-API Sources (Neo4j & SPARQL)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Neo4j table registration includes a query preview step. On submission, Provisa executes the steward's Cypher query against the live endpoint with an implicit LIMIT 5 appended and displays the resulting rows. If the response contains node or edge objects rather than flat scalar values, registration is blocked with an error directing the steward to rewrite the query using explicit scalar projections (RETURN n.name AS name, n.age AS age rather than RETURN n). Steward owns the response shape via query authoring; Provisa does not auto-flatten graph objects.

Use case: Query preview at Neo4j table registration catches non-scalar projections early before they cause runtime errors.

Code: provisa/neo4j/, provisa/sparql/, provisa/api_source/

Tests: tests/integration/test_direct_exec.py, tests/integration/test_neo4j_exec.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_neo4j_normalizers.py, tests/unit/test_neo4j_preview.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-297 · Query-API Sources (Neo4j & SPARQL)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

SPARQL connector supports any SPARQL 1.1 compliant triplestore endpoint (Apache Jena Fuseki, Ontotext GraphDB, Stardog, Blazegraph, Virtuoso). Source config accepts endpoint URL, auth (none/basic/bearer), and optional default graph URI. Tables defined as SPARQL SELECT queries. The built-in sparql_bindings response normalizer unwraps the standard SPARQL 1.1 JSON results envelope (results.bindings[].variable.value) into flat row dicts before the existing flattener runs. Column names match SPARQL SELECT variable names. Does not use Trino.

Use case: SPARQL connector lets stewards expose any SPARQL 1.1 triplestore as governed GraphQL query results.

Code: provisa/neo4j/, provisa/sparql/, provisa/api_source/

Tests: tests/unit/test_neo4j_normalizers.py, tests/integration/test_neo4j_exec.py

REQ-298 · Query-API Sources (Neo4j & SPARQL)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

API source caller extended to support HTTP POST with a request body. Required for Neo4j (POST to /db/{database}/query/v2 with Content-Type: application/json body {"statement": "<cypher>"}) and SPARQL (POST with Content-Type: application/x-www-form-urlencoded body query=<encoded-sparql> per SPARQL 1.1 protocol). ApiEndpoint.method already accepts arbitrary HTTP verbs; this closes the implementation gap where POST bodies were not transmitted. Existing GET endpoints unaffected.

Use case: HTTP POST support for API source caller enables Neo4j and SPARQL endpoints that require POST requests.

Code: provisa/neo4j/, provisa/sparql/, provisa/api_source/

Tests: tests/unit/test_neo4j_normalizers.py, tests/integration/test_neo4j_exec.py

REQ-299 · Query-API Sources (Neo4j & SPARQL)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

A response_normalizer: str | None field added to ApiEndpoint. When set, the named normalizer is applied to the raw API response before response_root navigation and the existing flattener. Built-in normalizers: neo4j_tabular (zips data.fields[] with each data.values[][] entry into a flat dict) and sparql_bindings (extracts results.bindings[] and maps each {variable: {type, value}} binding to {variable: value}). Normalizer names are validated at source registration time; unknown names are rejected. Extensible for future query-API envelope formats without changes to the core flattener.

Use case: Named response normalizers (neo4j_tabular, sparql_bindings) cleanly unwrap API envelopes before flattening.

Code: provisa/neo4j/, provisa/sparql/, provisa/api_source/

Tests: tests/integration/test_neo4j_exec.py, tests/unit/test_api_flattener.py, tests/unit/test_api_normalizer_requirements.py, tests/unit/test_misc_requirements.py, tests/unit/test_neo4j_normalizers.py

REQ-307 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Provisa supports a "remote GraphQL schema" source type. The steward registers an external GraphQL endpoint URL plus optional auth (none/bearer/basic). Provisa introspects the remote schema via a standard __schema introspection query and stores the result.

Use case: Remote GraphQL schema source type lets stewards expose external GraphQL APIs as governed Provisa tables.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-308 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Each Query field on the remote schema is exposed as a virtual read-only table in Provisa. Column names and types are inferred from the field's return type (object fields → columns). Each Mutation field is exposed as a tracked function with return_schema derived from the mutation's return type. This eliminates manual table and function registration for remote GraphQL APIs.

Use case: Auto-registering remote Query fields as tables and Mutation fields as functions eliminates manual registration.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-309 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

At query execution time, Provisa translates the incoming GraphQL request into a remote GraphQL query and forwards it to the remote endpoint. The response rows are materialized as Parquet in a Trino Iceberg table on S3 (results.api_cache, s3a://provisa-results/api_cache/). Repeated calls within TTL are served from the Iceberg table via Trino — zero remote hops. The cache table is automatically dropped after TTL expires.

Use case: S3/Iceberg materialization of remote schema results eliminates repeated network hops and enables federated SQL (WHERE/ORDER BY/LIMIT) over cached data.

Code: provisa/graphql_remote/, provisa/api_source/trino_cache.py

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_api_cache.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-310 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: MUST · Type: constraint

RLS, column masking, domain access control, and column visibility are applied to remote schema results in Stage 2 of the governance pipeline, identically to local table results. The remote endpoint has no knowledge of Provisa's governance rules.

Use case: Stage 2 governance applied to remote schema results ensures external APIs never bypass RLS or masking.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-311 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Remote schema introspection is re-run on demand via an "Refresh Schema" admin mutation. Provisa does not poll for remote schema changes automatically — the steward triggers refresh when the remote API evolves. On refresh, existing table and function registrations derived from the remote schema are updated; custom RLS/masking rules applied on top are preserved.

Use case: On-demand schema refresh lets stewards update remote schema registrations when the upstream API evolves.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-312 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Remote schema type name conflicts are resolved by prefixing all generated type names with a steward-supplied namespace (e.g. Shopify__Order rather than Order). Namespace is set at source registration time and cannot be changed without re-registering.

Use case: Namespace prefixing prevents remote schema type name collisions with local registered table types.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-313 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Relationships between remote schema virtual tables and local registered tables are supported via the standard relationship model. Provisa resolves the local side from cache/DB and the remote side via the cached remote call. Cross-source join behaviour follows the existing federation routing rules.

Use case: Relationships between remote and local tables follow standard federation routing so joins work transparently.

Code: provisa/graphql_remote/

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_introspect.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-314 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: structural

An OpenAPI source is registered by providing an OpenAPI 3.x or Swagger 2.0 spec. The spec path may be a local file path (absolute or relative to the config directory) or a remote URL. Provisa fetches remote specs at registration time and caches them; local specs are read directly.

Use case: OpenAPI spec registration auto-discovers all GET endpoints as governed virtual tables without manual config.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-315 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: MAY · Type: behavioral

If auto-discovery of a spec is not possible (behind auth, no spec endpoint, hand-written API), the steward may manually author an OpenAPI 3.x spec in the admin UI or upload a YAML/JSON file. The manually created spec is stored locally and treated identically to a fetched one.

Use case: Manual OpenAPI spec authoring lets stewards govern APIs that have no public spec endpoint.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-316 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

On registration, Provisa parses the spec and auto-registers all GET operations as virtual query tables. Path parameters and query parameters become GraphQL arguments. The responses.200 (or responses.2xx) schema determines the virtual table's column set.

Use case: GET operation auto-registration maps path and query params to GraphQL arguments automatically.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-317 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

All non-GET operations (POST, PUT, PATCH, DELETE) are auto-registered as tracked functions (mutations). Request body schema properties become GraphQL mutation input arguments. The responses.200/2xx schema becomes the mutation's return_schema.

Use case: Non-GET operations auto-registered as mutations expose REST write endpoints as governed GraphQL mutations.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-318 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GET operation results are materialized as Parquet in a Trino Iceberg table on S3 (results.api_cache, s3a://provisa-results/api_cache/). The cache key is a SHA-256 hash of source_id + operation path + native args. Repeated calls within TTL hit Trino directly. The cache table is dropped after TTL expires. Mutations are never cached — they are side-effecting.

Use case: S3/Iceberg materialization of GET operation results prevents repeated upstream REST calls and enables Trino SQL filtering over cached rows.

Code: provisa/api_source/trino_cache.py, provisa/api_source/router_integration.py

Tests: tests/unit/test_api_cache.py, tests/integration/test_openapi_source.py

REQ-319 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: MUST · Type: constraint

Full governance is applied to all OpenAPI source results: RLS, column masking, domain access control, and column visibility are enforced in Stage 2 identically to local tables.

Use case: Stage 2 governance applied to OpenAPI results ensures REST API data never bypasses RLS or masking.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-320 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: MUST · Type: constraint

Auth configuration for the upstream REST API (bearer token, API key header, basic auth, OAuth2 client credentials) is stored per-source in the secrets store and injected into outgoing requests. The caller's Provisa token is not forwarded.

Use case: Per-source upstream auth config keeps API credentials in the secrets store and out of client requests.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py

REQ-321 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Spec refresh is triggered on demand via an admin mutation. On refresh, existing virtual table and tracked function registrations derived from the spec are updated; governance rules applied on top are preserved.

Use case: On-demand spec refresh preserves governance rules while updating virtual table registrations from updated specs.

Code: provisa/openapi/

Tests: tests/unit/test_openapi_loader.py, tests/integration/test_openapi_source.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-322 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Provisa supports a "grpc_remote" source type. The steward registers an external gRPC server address plus a .proto file path or URL. Provisa parses the proto and auto-registers virtual tables and tracked functions. The provisa/grpc_remote/ module handles all external gRPC connectivity and is entirely distinct from provisa/grpc/, which serves Provisa's own gRPC server.

Use case: gRPC remote source type lets stewards expose external gRPC services as governed GraphQL tables and mutations.

Code: provisa/grpc_remote/

Tests: tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-323 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Query vs mutation classification defaults to a name-prefix rule: methods whose names start with Get, List, Find, Fetch, Search, or Stream are classified as queries (virtual tables); all others are classified as mutations (tracked functions). Per-method override precedence is retained in the admin UI. Structural heuristic (server-streaming / repeated-message) is kept as fallback for non-prefixed methods.

Use case: Default name-prefix classification removes manual tagging while preserving per-method overrides and structural fallback.

Code: provisa/grpc_remote/

Tests: tests/e2e/test_grpc_query.py, tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_grpc_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py, tests/unit/test_table_search.py

REQ-324 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Proto scalar types are mapped to SQL/GraphQL types as follows: string/bytes → text; int32/uint32/sint32/fixed32/sfixed32 → integer; int64/uint64/sint64/fixed64/sfixed64 → bigint; float → real; double → numeric; bool → boolean; repeated <T> → jsonb; nested message → jsonb; enum → text.

Use case: Proto scalar type mapping to SQL/GraphQL types gives gRPC message fields correct query and filter semantics.

Code: provisa/grpc_remote/

Tests: tests/integration/test_graphql_remote_source.py, tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-325 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Each query-classified gRPC method is exposed as a virtual read-only table. Output message fields become columns using the type mapping in REQ-324. Input message fields become GraphQL query arguments. Server-streaming methods (Stream*) collect all streamed response messages into a list before returning rows.

Use case: Query-classified gRPC methods exposed as virtual tables let clients query remote services with GraphQL syntax.

Code: provisa/grpc_remote/

Tests: tests/e2e/test_grpc_query.py, tests/integration/test_graphql_remote_source.py, tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-326 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Each mutation-classified gRPC method is exposed as a tracked function (mutation). Input message fields become GraphQL mutation input arguments. The output message schema becomes the mutation's return_schema.

Use case: Mutation-classified gRPC methods exposed as tracked functions let clients call remote services via GraphQL mutations.

Code: provisa/grpc_remote/

Tests: tests/integration/test_graphql_remote_source.py, tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-327 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Query method results are materialized as Parquet in a Trino Iceberg table on S3 (results.api_cache, s3a://provisa-results/api_cache/). The cache key is a SHA-256 hash of source_id + method + native args. Mutations are never cached — they are side-effecting. One grpc.aio.Channel is reused per registered source across requests, stored in AppState.grpc_remote_channels. The cache table is dropped after TTL expires.

Use case: S3/Iceberg materialization of gRPC query results enables Trino SQL filtering over cached rows and eliminates repeated remote calls. Channel reuse reduces connection overhead.

Code: provisa/grpc_remote/, provisa/api_source/trino_cache.py

Tests: tests/e2e/test_grpc_query.py, tests/integration/test_cache_store.py, tests/integration/test_grpc_execution.py, tests/unit/test_api_cache.py, tests/unit/test_cache_store.py, tests/unit/test_grpc_remote_cache.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-328 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Full governance is applied to gRPC remote source results: RLS, column masking, domain access control, and column visibility are enforced in Stage 2 identically to local table results. The remote gRPC server has no knowledge of Provisa's governance rules.

Use case: Stage 2 governance applied to gRPC results ensures remote service data never bypasses RLS or masking.

Code: provisa/grpc_remote/

Tests: tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-329 · gRPC Remote Schema Connector (REQ-322–329)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Proto schema refresh is triggered on demand via an admin mutation. On refresh, Provisa re-parses the proto, updates virtual table and tracked function registrations, and preserves any RLS/masking rules applied on top. Proto import paths (for well-known types such as google/protobuf/timestamp.proto) are stored at registration time and re-used on refresh.

Use case: On-demand proto refresh preserves governance rules while updating registrations when the remote proto changes.

Code: provisa/grpc_remote/

Tests: tests/integration/test_grpc_execution.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-331 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A new ingest source type allows external services (OTEL Collector, Fluentd, custom webhooks, etc.) to POST JSON events to Provisa via POST /events/ingest/{source_id}/{table}. Provisa owns the write path and persists events to a steward-configured backing relational store.

Use case: Ingest source type lets external push producers (OTEL Collector, Fluentd) write events to governed tables.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py, tests/integration/test_ingest_push_integration.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-332 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Each ingest source specifies a SQLAlchemy-compatible connection (dialect, host, port, database, username/password). Provisa creates one AsyncEngine per source at startup and reuses it across all requests. The dialect field on the sources table carries the SQLAlchemy driver string (e.g. postgresql+asyncpg, mysql+aiomysql).

Use case: Per-source SQLAlchemy AsyncEngine lets ingest tables target any supported relational store asynchronously.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py

REQ-333 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Stewards define the schema for each ingest table as a list of columns with column_name, data_type (SQL type, allowlisted), and path (dot-notation extraction path into the POST payload). Provisa auto-generates and executes CREATE TABLE IF NOT EXISTS DDL at startup using these definitions. System columns _received_at and _updated_at (TIMESTAMPTZ) are always injected.

Use case: Auto-generated DDL from column definitions creates ingest tables at startup with system audit columns included.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py, tests/integration/test_ingest_push_integration.py

REQ-334 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The path field on table_columns uses dot-notation to walk nested JSON payloads. Array index segments are supported (e.g. resourceLogs.0.resource.attributes). If path is absent, the column name is used as the top-level key. Missing paths yield NULL.

Use case: Dot-notation path extraction lets stewards map deeply nested JSON payloads to flat relational columns.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py

REQ-335 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The ingest endpoint accepts a single JSON object or a JSON array of objects per request. Each event in the array is extracted and written as a separate row. Returns HTTP 202 with a count of inserted rows. Source or table not found returns 404; engine unavailable returns 503.

Use case: Batch ingest endpoint accepts arrays of events and returns inserted count so producers confirm delivery.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py, tests/integration/test_ingest_push_integration.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-336 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Ingest tables are subscribable via the standard SSE endpoint (GET /data/subscribe/{table}). The subscription provider polls _updated_at watermark at a configurable interval (default: 5 s). Full governance is applied: RLS row filtering and column masking are enforced identically to local table subscriptions.

Use case: Ingest tables subscribable via SSE lets consumers receive pushed events as a live stream with full governance.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py, tests/integration/test_ingest_push_integration.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-337 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

table_columns.data_type TEXT stores the steward-declared SQL type for ingest columns. This field is also used by future schema introspection tooling for sources where Trino is not available (ingest, API sources).

Use case: data_type column on table_columns stores the steward-declared SQL type needed for non-Trino source introspection.

Code: provisa/ingest/

Tests: tests/unit/test_ingest_ddl.py, tests/unit/test_ingest_router.py

REQ-338 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

websocket is a supported source type. Connects to a WebSocket server, optionally sends a JSON subscribe payload on connect, receives JSON messages as a stream of ChangeEvents.

Use case: WebSocket sources feed real-time events (market data, IoT, CDC streams) into the governed data fabric.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py, tests/integration/test_websocket_rss_integration.py

REQ-339 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

WebSocket provider auto-reconnects on disconnect or error with configurable reconnect_interval (default: 5 s). The reconnect loop runs until close() is called.

Use case: Auto-reconnect ensures continuous event ingestion without operator intervention on transient disconnects.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py, tests/integration/test_websocket_rss_integration.py

REQ-340 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

WebSocket message fields: op (insert/update/delete, default insert), _ts/timestamp (ISO 8601 or Unix epoch float, default now), all other fields become the row. event_path dot-notation extracts a sub-object from the message before mapping.

Use case: Standardized message field mapping lets stewards define schema once regardless of upstream message shape.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py

REQ-341 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

WebSocket source connection URL is constructed from host, port, path, and use_ssl (federation_hints). subscribe_payload (JSON string in federation_hints) is sent on connect. event_path (federation_hints) selects a sub-field of each message.

Use case: URL construction from config allows WebSocket sources to be registered via standard Provisa YAML config.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py

REQ-342 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

rss is a supported source type. Polls an RSS 2.0 or Atom feed URL at a configurable interval (default: 300 s = 5 min). Only items with a publication date newer than the last-seen watermark are emitted as ChangeEvents with operation="insert".

Use case: RSS/Atom feeds become governed polling sources for news, alerts, and changelog data.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py, tests/integration/test_websocket_rss_integration.py

REQ-343 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

RSS parser handles both RSS 2.0 (<item>) and Atom (<entry>) formats. Fields extracted: title, link, description/summary, published (pubDate/updated), id (guid/id). Dates accept RFC 2822 and ISO 8601 formats; unparseable dates fall back to a sentinel minimum date (datetime.min, UTC) to explicitly mark them unparseable rather than silently substituting the current time.

Use case: RSS 2.0 and Atom format parity ensures any feed can be registered without format-specific handling.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py, tests/integration/test_websocket_rss_integration.py

REQ-344 · Phase AT — WebSocket & RSS Sources (REQ-338–344)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

RSS source URL is resolved from federation_hints.feed_url if present, otherwise built from host, port, path, and use_ssl. poll_interval (federation_hints) overrides the default.

Use case: Feed URL resolution from config enables RSS sources to be registered the same way as other API sources.

Code: provisa/subscriptions/

Tests: tests/unit/test_websocket_provider.py, tests/unit/test_rss_provider.py

REQ-372 · File & Lake Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Iceberg and Delta Lake sources support time-travel queries via an optional as_of argument on root query fields (snapshot ID, version number, or ISO 8601 timestamp). The compiler translates as_of to Trino FOR TIMESTAMP AS OF / FOR VERSION AS OF syntax. Time-travel is read-only and only available on source types whose Trino connector supports it. Queries against non-time-travel-capable sources that supply as_of are rejected at compile time.

Use case: Time-travel queries let consumers audit historical snapshots of Iceberg and Delta tables without separate table copies.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_time_travel.py, tests/integration/test_registration_governance_integration.py

REQ-419 · Vector Search¶

Status: ✅ complete · Priority: MUST · Type: structural

A model registry must be declared in Provisa config listing available embedding models with id, provider, dimensions, API key env var or base URL, and enabled flag. Models not in the registry cannot be used for embedding generation or query vectorization.

Use case: Explicit model registry enables air-gapped deployments and audit trails for embedding model usage.

Code: provisa/vector/registry.py

Tests: tests/unit/test_vector.py

REQ-420 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Model registry must support multiple provider types: OpenAI-compatible APIs, Ollama (local), and local HuggingFace model paths. This enables air-gapped and regulated deployments to use on-premise models throughout.

Use case: Multi-provider support removes vendor lock-in and enables regulated deployments without external API calls.

Code: provisa/vector/providers.py

Tests: tests/unit/test_vector.py

REQ-421 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: structural

A column may be declared as an embedding vector by setting embedding: true with attributes: model id (must reference model registry), dimensions, and optional source_column mapping to the physical column name in the source.

Use case: Schema-level embedding declarations enable Provisa to manage vector columns as first-class citizens.

Code: provisa/compiler/introspect.py

Tests: tests/unit/test_vector.py

REQ-422 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Source capability auto-detection at registration time must identify native vector support: pgvector extension for PostgreSQL, Atlas Vector Search for MongoDB, Cortex for Snowflake. Sources without detected capability are flagged as requiring fallback.

Use case: Early capability detection allows fallback materialization strategy to be planned at source registration.

Code: provisa/source_adapters/introspect.py

Tests: tests/unit/test_vector.py

REQ-423 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A cosine_similarity(column, query_vector) UDF must be available in Provisa SQL. It translates to the native vector operator for sources with native capability (pgvector <=>, Snowflake VECTOR_COSINE_SIMILARITY(), etc.) and to the pgvector fallback cache for sources without.

Use case: Unified UDF hides native vs. fallback complexity from analysts.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_vector.py

REQ-424 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

For sources without native vector capability, Provisa must transparently materialize the embedding column to an internal pgvector-enabled PostgreSQL cache table, build an HNSW index, rewrite the query against the cache joining PKs back to the source, and return unified results. The caller must not be aware of the fallback path.

Use case: Transparent fallback enables vector search on any source without source-side changes.

Code: provisa/vector/fallback_cache.py

Tests: tests/unit/test_table_search.py, tests/unit/test_vector.py

REQ-425 · Vector Search¶

Status: ✅ complete · Priority: MUST · Type: constraint

The pgvector fallback cache must support TTL-based invalidation, mutation-triggered invalidation, manual refresh via admin API, and row count drift detection. Stale cache must not silently serve results beyond configured TTL.

Use case: Multiple invalidation strategies with drift detection prevent silent data staleness.

Code: provisa/vector/cache_invalidation.py

Tests: tests/unit/test_config_validation_edge_cases.py, tests/unit/test_vector.py

REQ-426 · Vector Search¶

Status: ✅ complete · Priority: MUST · Type: constraint

Embedding columns are subject to the full Provisa governance stack: RLS, column masking, domain boundary rules, and sensitivity tiers. A restricted sensitivity tier on an embedding column must prevent cosine similarity search for unauthorized roles.

Use case: Embedding vectors inherit standard governance, preventing privacy bypass through vector search.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_vector.py

REQ-427 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A virtual embedding column may be declared on a table with a generated_from subquery that must return exactly one text value per row. The subquery may reference multiple columns, apply transformations, and include joins to related tables. Provisa validates the subquery against a sample row at declaration time.

Use case: Generated embedding columns enable text-to-vector transformations without storing raw text.

Code: provisa/vector/generation.py

Tests: tests/unit/test_table_search.py, tests/unit/test_vector.py

REQ-428 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Generated embedding columns must support scheduled incremental refresh — only rows where the source data has changed since last embedding are re-embedded. Full rebuild is triggered on model change or schema change affecting the generating subquery.

Use case: Incremental refresh avoids unnecessary embedding API calls and reduces operational cost.

Code: provisa/vector/scheduled_refresh.py

Tests: tests/unit/test_vector.py

REQ-429 · Vector Search¶

Status: ✅ complete · Priority: MUST · Type: constraint

Once an embedding column is generated with a declared model, that model is locked for that column. Provisa must reject queries that attempt to use a query vector from a different model or incompatible dimensions, with a clear error identifying the conflict.

Use case: Model locking prevents silent semantic errors from mismatched embedding spaces.

Code: provisa/compiler/sql_gen.py

Tests: tests/unit/test_vector.py

REQ-430 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Query-time vectorization must be supported: when a similarity search is expressed with a text string rather than a raw vector, Provisa calls the declared embedding model to generate the query vector before executing the search. Both text input and raw vector input must be supported interfaces.

Use case: Query-time vectorization enables natural text-based search without pre-vectorization ceremony.

Code: provisa/vector/query_vectorization.py

Tests: tests/unit/test_schema_visibility_filters.py, tests/unit/test_vector.py

REQ-431 · Vector Search¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Vector search features are organized as three independent phases: (1) native vector search with model registry, embedding column declaration, and cosine_similarity UDF translation; (2) non-native fallback via pgvector cache materialization; (3) declarative embedding generation from source text columns. Each phase is independently deployable and delivers standalone value.

Use case: Phased decomposition enables incremental feature adoption and reduces delivery risk.

Code: provisa/vector/

Tests: tests/unit/test_schema_visibility_filters.py, tests/unit/test_vector.py

REQ-673 · Cost Statistics¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Connector cardinality capability — cardinality(source, table) -> Estimate{value, exact, method}. A source may expose a CHEAP volume metric so a caller can size a table without a full scan. Resolution order: (1) a cheap native statistic when the source has one (GraphQL totalCount, OpenAPI X-Total-Count, PG pg_class.reltuples, Trino SHOW STATS, Parquet/Iceberg row-group metadata); (2) an exact COUNT() only when the connector reports counting is cheap; (3) UNKNOWN when sizing is expensive and no native estimate exists — fail-open, never a hidden full scan. The Estimate carries exact (true for a native count()/totalCount, false for reltuples-style estimates) and method, so a consumer knows what the number is and what it may be used for. The producer is SPARSE and OPT-IN: a native count is authored only for expensive-count sources — the API sources (openapi/graphql_remote/grpc), where a count today means materializing the full pull, and huge un-indexed row-stores. Cheap-count sources contribute nothing and carry no maintenance. Generalizes and supersedes the GQL-specific count_query helper (provisa/sources/counts.py) under the connector abstraction (REQ-840-843). DONE (2026-07): the estimate type + resolution order are implemented — provisa/federation/ cardinality.py::Estimate(value, exact, method: CardinalityMethod) and resolve_cardinality(native_stat, exact_count_cheap, exact_count) returning, in order, a source's native statistic (exact totalCount or approximate reltuples), else an exact COUNT(*) only when the connector says counting is cheap, else UNKNOWN (fail-open, never a hidden full scan). REMAINING (kept in-progress): the per-source native-stat PRODUCERS wired onto the connectors (graphql totalCount, openapi X-Total-Count, PG reltuples, Trino SHOW STATS, Parquet/Iceberg row-group metadata) — sparse/opt-in for expensive-count sources.

Use case: One estimate serves two consumer classes. The AUTOMATED PLANNER reads it to choose a federation strategy (materialize vs virtual, REQ-826), gate hot-table promotion (REQ-236) without a blind COUNT(*), and set redirect/probe thresholds. HUMAN PLANNERS read the same estimate — in the catalog, in plan/EXPLAIN output, and on the cold graph-counts path — to size a source before committing to an expensive federated query, with exact/estimate/unknown surfaced honestly. An "unknown, sizing requires a full pull" answer is itself actionable, unlike today's bare 0.

Code: provisa/federation/cardinality.py

Tests: tests/unit/test_cardinality.py

REQ-540 · GovData Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Sources of type govdata expose U.S. government open data partitioned by subject grouping. Configuring a govdata source with a subject exposes all schemas for that subject automatically.

Use case: Subject-grouped GovData sources let stewards selectively grant access to government data by domain area.

Code: provisa/core/models.py

Tests: tests/integration/test_govdata_source.py, tests/unit/test_core_registration.py

REQ-541 · GovData Sources¶

Status: ✅ complete · Priority: MUST · Type: constraint

The ref and geo schemas are always included as linker schemas in every GovData source — they are not configurable and not tied to any subject grouping.

Use case: Always-included linker schemas ensure GovData joins on reference and geography data are always available without explicit configuration.

Code: provisa/core/models.py

Tests: tests/integration/test_govdata_source.py, tests/unit/test_core_registration.py

REQ-550 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Sources are classified into three execution categories: Direct-capable (has both native driver and federated connector; executes single-source queries via native driver), Federation-only (no direct driver; always routed through Trino), and Materialize (no federated connector; data fetched and cached as Parquet in S3 or PostgreSQL before being queryable by the federation engine).

Use case: Tripartite source categorization gives stewards a clear mental model of how each source type executes queries and when the federation engine is involved.

Code: provisa/core/models.py, provisa/transpiler/router.py

Tests: tests/integration/test_direct_exec.py, tests/unit/test_routing.py

REQ-553 · Registration & Governance¶

Status: ✅ complete · Priority: SHOULD · Type: structural

File-based sources (csv, parquet, sqlite) use the path config field instead of host/port connection parameters. The path may be a local filesystem path or an fsspec-supported remote object-storage URL (s3://, ftp://, sftp://); the crawler detects fsspec URIs via fsspec.core.url_to_fs and walks them the same way as local directories.

Use case: File path config for file-based sources lets stewards register local files and remote object-storage files as governed tables using the same config structure as network-connected sources.

Code: provisa/core/models.py, provisa/file_source/source.py, provisa/file_source/crawler.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_core_registration.py

REQ-597 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GraphQL remote source registration accepts a field_overrides map ({fieldName: "query" | "mutation"}) that is applied after introspection and takes priority over structural classification. Only query-type fields can be reclassified as mutations via overrides; mutation-type fields have no override path in GraphQL.

Use case: field_overrides lets stewards correct mis-classified remote fields (e.g. a mutation named like a query) at registration time without modifying the upstream schema.

Code: provisa/graphql_remote/mapper.py

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_graphql_remote_overrides_requirements.py, tests/unit/test_misc_requirements.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-598 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Remote schema source registrations (GraphQL, gRPC, OpenAPI) accept a relationships list that stores FK/PK join paths as manually declared relationships (no remote_managed flag). Relationships auto-detected during introspection are stored with remote_managed: True. On schema refresh, auto-detected (remote_managed: True) relationships are re-run and may change; manually declared relationships (no flag) are preserved unchanged.

Use case: Distinguishing manual from auto-detected relationships ensures steward-declared join paths survive schema refreshes without being overwritten by introspection results.

Code: provisa/graphql_remote/mapper.py, provisa/api/admin/graphql_remote_router.py, provisa/api/admin/grpc_remote_router.py, provisa/api/admin/openapi_router.py

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-599 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

For all remote schema source types (GraphQL, gRPC, OpenAPI), required input parameters that are not already response fields are registered as _nf_-prefixed native filter columns. The native_filter_type is query_param for GQL and OpenAPI query parameters, path_param for OpenAPI path parameters, and grpc_input for gRPC. When a parameter name matches a response field name, parameter metadata is merged into the existing column rather than creating a duplicate.

Use case: Native filter columns let consumers filter remote-source queries using standard where arguments that are passed through to the upstream API rather than applied as post-fetch SQL predicates.

Code: provisa/graphql_remote/mapper.py, provisa/api/admin/grpc_remote_router.py, provisa/openapi/register.py, provisa/api/app.py

Tests: tests/e2e/test_grpc_query.py, tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_source.py, tests/integration/test_grpc_execution.py, tests/integration/test_openapi_source.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-600 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Remote schema sources enforce depth limits on nested object expansion: graphql_remote.max_object_depth (default: 5) caps GQL OBJECT nesting; graphql_remote.max_list_depth (default: 2) caps LIST-typed nested OBJECT inclusion; graphql_remote.max_list_items (default: 100) injects first: N to cap array size. For gRPC and OpenAPI, nested message/object sub-field resolution is one level deep only. Fields beyond these limits are excluded from the fetch.

Use case: Depth limits prevent unbounded data expansion and runaway remote API calls when remote schemas contain deeply recursive or list-heavy structures.

Code: provisa/graphql_remote/mapper.py, provisa/grpc_remote/mapper.py, provisa/openapi/register.py

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_grpc_remote_loader.py, tests/unit/test_openapi_loader.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-601 · OpenAPI Auto-Registration Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

OpenAPI virtual table names are derived from the operation's operationId. If no operationId is defined, Provisa slugifies {method}_{path}. An alias is derived by stripping the leading verb segment and singularizing the noun (e.g. findPetsByStatus → pet_by_status). The alias is used as the consumer-facing name in GraphQL and other query interfaces.

Use case: operationId-based naming and verb-stripped aliases give OpenAPI tables predictable, consumer-friendly names without manual configuration.

Code: provisa/openapi/register.py

Tests: tests/integration/test_openapi_pet_by_status.py, tests/unit/test_graphql_alias.py, tests/unit/test_openapi_loader.py

REQ-602 · GraphQL Remote Schema Connector (REQ-307–313)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Schema generation synthesizes ColumnMetadata for remote schema tables (GraphQL remote, gRPC remote, OpenAPI) at schema build time since these sources have no Trino catalog to introspect. Type mapping (GQL scalar → Provisa type, proto scalar → SQL type, JSON Schema → Provisa type) is applied during synthesis to produce typed column metadata equivalent to what catalog introspection produces for local relational tables.

Use case: Synthesized ColumnMetadata lets remote schema tables participate in schema generation and query compilation identically to locally connected relational tables.

Code: provisa/compiler/schema_gen.py, provisa/api/app.py

Tests: tests/integration/test_graphql_execution.py, tests/integration/test_graphql_remote_source.py, tests/integration/test_schema_gen.py, tests/unit/test_graphql_remote_mapper.py, tests/unit/test_remote_adapter_contract.py, tests/unit/test_schema_service.py

REQ-652 · Vector Search¶

Status: ✅ complete · Priority: MUST · Type: constraint

Column masking does not apply to embedding columns — a vector cannot be partially masked and remain semantically meaningful. Access to an embedding column is enforced via visibility (hide entirely), RLS (filter rows), sensitivity tier (block search), and domain boundary rules; the column is either visible and searchable for a given role, or absent from the schema entirely.

Use case: Preventing partial-masking of vectors ensures access control is all-or-nothing for embedding columns, which cannot be meaningfully redacted at the field level.

Code: provisa/vector/governance.py

Tests: tests/unit/test_vector.py

5. Query Languages, Compilation & Operations¶

REQ-007 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Compiler is purpose-built with no dependency on PostGraphile, DuckDB, or any third-party GraphQL server framework.

Use case: Purpose-built compiler eliminates third-party framework constraints and reduces attack surface.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-008 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Schema generation pass runs at registration time; queries Trino INFORMATION_SCHEMA, applies per-role column visibility, incorporates relationships, produces GraphQL SDL.

Use case: Schema generation at registration time means the GraphQL schema always reflects current data and role visibility.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-009 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Query compilation produces PG-style SQL from validated GraphQL AST — single SQL statement, no resolver chain, no N+1. Exception: action query fields (REQ-361) and remote-schema relationships (REQ-313) resolve nested relationships via batched secondary fetches, which is outside this single-statement guarantee.

Use case: Single-statement SQL compilation eliminates N+1 query patterns and improves query execution performance.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_graphql_remote_source.py, tests/integration/test_schema_gen.py, tests/unit/test_layer_contracts.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-010 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: structural

Trino type mapping: VARCHAR→String, INTEGER→Int, BOOLEAN→Boolean, TIMESTAMP→DateTime, JSONB→JSON. Nullability preserved.

Use case: Consistent Trino type mapping ensures GraphQL consumers receive predictable, well-typed field values.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-011 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: constraint

References to unregistered tables, excluded columns, undefined relationships, or type mismatches rejected at compile time with precise errors.

Use case: Compile-time rejection of invalid references catches developer errors early before they reach the data layer.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-032 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: behavioral

DB mutations are single-source by definition, bypass Trino, no routing decision, no registry approval — but always require write authority on the target table (REQ-033). Webhook mutations are governed separately and require specific steward approval for use (REQ-209).

Use case: DB mutations skip the registry approval path but still require table write authority; webhook mutations require explicit approval.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-033 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: constraint

User must have write rights to target table for any mutation.

Use case: Write-rights enforcement ensures users can only mutate tables they are explicitly authorized to change.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-034 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Mutation input types reflect only columns user's role is permitted to write; excluded column references rejected at parse time.

Use case: Role-filtered mutation input prevents users from writing to columns their role cannot access.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-035 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: behavioral

RLS WHERE clauses injected into UPDATE and DELETE before execution.

Use case: RLS injection into UPDATE and DELETE ensures row-level security is enforced on all write operations.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-036 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: constraint

Mutations can only target registered tables — compiler does not generate mutation types for unregistered tables.

Use case: Mutations only target registered tables, preventing writes to unmanaged or unknown database objects.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-037 · Mutation Execution¶

Status: ✅ complete · Priority: MUST · Type: constraint

Cross-source transactions not supported. NoSQL sources do not support mutations.

Use case: Restricting mutations to single-source and relational targets prevents partial cross-source transaction failures.

Code: provisa/compiler/sql_gen.py, provisa/executor/

Tests: tests/unit/test_mutation_sql.py, tests/integration/test_mutations.py

REQ-066 · SQLGlot Transpilation¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Compiler emits PG-style SQL as canonical output; SQLGlot translates to Trino SQL or target RDBMS dialect.

Use case: PG-style canonical SQL output gives a single, well-understood intermediate representation for all dialects.

Code: provisa/transpiler/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_layer_contracts.py, tests/unit/test_transpile.py, tests/unit/test_transpiler.py

REQ-067 · SQLGlot Transpilation¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Target dialect determined by source type captured at table registration time.

Use case: Dialect detection at table registration time means transpilation is automatic and requires no per-query config.

Code: provisa/transpiler/

Tests: tests/unit/test_transpile.py, tests/unit/test_transpiler.py, tests/integration/test_compiler_integration.py

REQ-068 · SQLGlot Transpilation¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Supported dialects: PostgreSQL, MySQL, SQL Server, Trino, DuckDB, Snowflake, BigQuery.

Use case: Multi-dialect SQLGlot support lets Provisa federate across PostgreSQL, MySQL, SQL Server, Trino, and more.

Code: provisa/transpiler/

Tests: tests/unit/test_transpile.py, tests/unit/test_transpiler.py, tests/integration/test_compiler_integration.py

REQ-196 · Aggregates¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-generated aggregate queries following Hasura v2 pattern. Every table gets a <table>_aggregate root field. Numeric columns get sum/avg/stddev/variance. All comparable columns get min/max. All columns get count. No configuration required for default behavior.

Use case: Hasura v2 parity — auto-generated aggregate fields expose sum/avg/min/max without any steward configuration.

Code: provisa/compiler/sql_gen.py, provisa/mv/

Tests: tests/unit/test_sql_gen_aggregate.py, tests/integration/test_compiler_integration.py

REQ-197 · Aggregates¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Per-role aggregate gating via allow_aggregations (matching v2) or per-table aggregates config section for explicit override of auto-detected functions and role visibility.

Use case: Hasura v2 parity — per-role aggregate gating prevents unauthorised users from deriving statistical insights.

Code: provisa/compiler/sql_gen.py, provisa/mv/

Tests: tests/unit/test_sql_gen_aggregate.py, tests/integration/test_compiler_integration.py

REQ-198 · Aggregates¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Aggregate MV routing — when a query requests aggregates over a pattern already materialized in an MV, the compiler rewrites the query to use the MV. Requires aggregate catalog + query rewriter.

Use case: Aggregate MV routing transparently accelerates aggregate queries without requiring query rewrites.

Code: provisa/compiler/sql_gen.py, provisa/mv/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_aggregate_mv_catalog.py, tests/unit/test_aggregate_mv_routing.py, tests/unit/test_sql_gen_aggregate.py

REQ-199 · Aggregates¶

Status: ✅ complete · Priority: MAY · Type: behavioral

View auto-materialization for aggregate optimization — expensive views auto-materialized and registered in aggregate catalog instead of requiring bespoke materialized_views entries. Default TTL configurable globally via materialized_views.default_ttl (seconds, default: 3600). Individual views can override with refresh_interval. Stale MVs refreshed by background loop; queries against stale MVs fall back to live execution. (Explicit, defined exception to REQ-064 fail-fast — stale-MV→live is designed behavior, not silent error handling.)

Use case: View auto-materialization avoids repeated expensive subquery execution for high-traffic computed datasets.

Code: provisa/compiler/sql_gen.py, provisa/mv/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_aggregate_mv_catalog.py, tests/unit/test_aggregate_mv_routing.py, tests/unit/test_sql_gen_aggregate.py

REQ-200 · OrderBy Alignment¶

Status: ✅ complete · Priority: MUST · Type: structural

GraphQL order_by schema must follow Hasura v2 convention: column-keyed input type {column_name: direction} instead of current {field: ENUM, direction: ENUM} struct.

Use case: Hasura v2 parity — column-keyed order_by schema lets Hasura-trained developers sort results intuitively.

Code: provisa/compiler/

Tests: tests/unit/test_orderby_alignment.py, tests/integration/test_compiler_integration.py

REQ-201 · OrderBy Alignment¶

Status: ✅ complete · Priority: MUST · Type: structural

OrderBy direction enum must include 6 values: asc, asc_nulls_first, asc_nulls_last, desc, desc_nulls_first, desc_nulls_last. SQL compiler maps to ORDER BY col ASC NULLS FIRST, etc.

Use case: Hasura v2 parity — six-value direction enum gives precise NULL-ordering control matching standard SQL.

Code: provisa/compiler/

Tests: tests/unit/test_orderby_alignment.py, tests/integration/test_compiler_integration.py

REQ-202 · OrderBy Alignment¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Relationship ordering — order by related object fields (e.g., order_by: {author: {name: asc}}). Matches Hasura v2 default behavior.

Use case: Hasura v2 parity — relationship ordering lets clients sort parent rows by child attributes in one request.

Code: provisa/compiler/

Tests: tests/unit/test_orderby_alignment.py, tests/integration/test_compiler_integration.py

REQ-205 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Database functions (stored procedures, UDFs) registered in Provisa config and exposed as GraphQL mutations or queries. VOLATILE functions exposed as mutations; STABLE/IMMUTABLE as queries. Follows Hasura v2's pg_track_function pattern.

Use case: Tracked DB functions expose stored procedures as GraphQL mutations so clients use one API for all writes.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-206 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Function config section in provisa.yaml (|) provides structured registration of stored procedures with governance metadata.

Use case: Function config section provides a structured way to register stored procedures with governance metadata.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-207 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

Function return type MUST reference a registered table. The result set maps back to GraphQL using that table's type definition, column governance (visibility, masking), and RLS rules. This ensures function results go through the same security pipeline as direct queries.

Use case: Requiring functions to return a registered table ensures their results go through the same security pipeline.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/e2e/test_query_pipeline.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-208 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Functions execute via direct DB connection (same as mutations). Never routed through Trino.

Use case: Direct DB execution for functions avoids Trino round-trips on mutation-class operations.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/integration/test_direct_exec.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-209 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Webhook-backed mutations — external HTTP endpoint called as a GraphQL mutation (Part of Phase AC). Config includes name, url, method, domain_id, governance (requires_approval), arguments with name and type, returns (registered table or inline type), visible_to, and timeout_ms. Webhook mutations are NOT DB mutations (REQ-031/REQ-032) and require specific steward approval for use; governance: registry-required gates them.

Use case: Webhook mutations let GraphQL clients trigger external services without learning a second API.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-210 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MAY · Type: structural

Webhook mutations support inline return type definitions (not backed by a registered table) for cases where the webhook returns a custom shape. Inline types define fields with names and GraphQL types.

Use case: Inline return types for webhooks let custom response shapes be exposed without creating a registered table.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-211 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

Function and webhook argument types map to GraphQL input types. Arguments are validated at parse time. SQL injection prevented via parameterized calls for DB functions and JSON serialization for webhooks.

Use case: Argument validation at parse time prevents SQL injection and type errors before any DB or HTTP call is made.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-252 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Schema inference must be supported where the Trino connector provides auto-discovery (MongoDB, Cassandra, Elasticsearch). Sources with inference support should offer a discover: true flag that introspects and generates a starting column list. Explicit column definitions always take precedence over inferred schemas. Sources without inference capability (Redis) require explicit column definitions.

Use case: Schema inference for supported connectors (MongoDB, Cassandra) reduces manual column definition work.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_discover_inference.py, tests/unit/test_mongodb_discovery.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-253 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Naming convention changes (global, source, or table level) reflected immediately in GraphQL schema. Admin mutations trigger _rebuild_schemas() which regenerates in-memory graphql-core schema objects. GraphiQL and Voyager fetch fresh introspection on each page load. No client-side schema caching.

Use case: Immediate schema rebuild on naming convention changes ensures GraphQL clients always see consistent names.

Code: provisa/compiler/sql_gen.py

Tests: tests/e2e/test_naming_reload.py, tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py, provisa-ui/e2e/registration-admin.spec.ts

REQ-259 · Compiler & Schema¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Apollo Federation v2 subgraph support — when enabled, Provisa generates a Federation v2 compliant schema with @key directives on entity types (derived from primary keys), _service and _entities root fields, and batch entity resolution. Provisa can be composed into an Apollo Gateway/Router supergraph. Entity resolution respects RLS, masking, and role-based visibility. Disabled by default.

Use case: Apollo Federation v2 subgraph support lets Provisa participate in a federated supergraph alongside other APIs.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-300 · GraphQL Variable Defaults¶

Status: ✅ complete · Priority: MUST · Type: behavioral

GraphQL operations may declare variable default values (e.g. query Q($limit: Int = 10)). The compiler MUST apply those defaults for any variable not present in the request variables dict, per GraphQL spec §6.4.1. Missing a declared default must not produce a 500 — the default value is used as if the caller had supplied it.

Use case: GraphQL variable defaults applied by compiler prevent 500 errors when callers omit optional parameters.

Code: provisa/compiler/

Tests: tests/integration/test_compiler_integration.py, tests/integration/test_graphql_execution.py, tests/unit/test_params.py

REQ-301 · GraphQL Variable Defaults¶

Status: ✅ complete · Priority: MUST · Type: constraint

LIMIT and OFFSET values MUST be emitted as positional parameters ($N) in compiled SQL, never interpolated as literals. This ensures consistent parameterization regardless of whether the value originated from a GraphQL variable or an inline literal argument.

Use case: Parameterized LIMIT/OFFSET prevents SQL injection and ensures consistent query plans regardless of value source.

Code: provisa/compiler/

Tests: tests/integration/test_compiler_integration.py, tests/integration/test_graphql_execution.py, tests/unit/test_params.py

REQ-304 · Tracked DB Functions — Custom Return Schema (REQ-304–306)¶

Status: ✅ complete · Priority: MAY · Type: structural

Tracked DB functions MAY declare a return_schema (JSON Schema) in place of a returns (registered table reference). When return_schema is present and returns is empty, the schema compiler converts the JSON Schema definition to a GraphQLObjectType at schema-build time. type: array yields GraphQLList(GraphQLNonNull(obj)); type: object yields a nullable single object. This enables functions that return custom shapes not backed by any registered table.

Use case: Custom return_schema for tracked functions lets stored procedures returning non-table shapes be exposed safely.

Code: provisa/compiler/, provisa/api/admin/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_function_gen.py, tests/unit/test_schema_service.py

REQ-305 · Tracked DB Functions — Custom Return Schema (REQ-304–306)¶

Status: ✅ complete · Priority: MAY · Type: ui

The admin UI for tracked DB functions provides a "Return Type" mode toggle: "Registered Table" (existing schema.table select) or "Custom Schema". In custom schema mode, the steward pastes a sample JSON response (single object or array); Provisa infers a JSON Schema from it client-side (field names + primitive types). The steward may edit the inferred schema before saving. The inferred+edited JSON Schema is stored as return_schema JSONB on tracked_functions.

Use case: Admin UI return type toggle lets stewards define custom function return schemas by pasting a sample response.

Code: provisa/compiler/, provisa/api/admin/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_function_gen.py, tests/unit/test_schema_service.py

REQ-306 · Tracked DB Functions — Custom Return Schema (REQ-304–306)¶

Status: ✅ complete · Priority: MUST · Type: structural

JSON Schema → GraphQL type mapping for return_schema: "string" → GraphQLString, "integer" → GraphQLInt, "number" → GraphQLFloat, "boolean" → GraphQLBoolean. Unknown or missing type defaults to GraphQLString. Properties with nested objects are not recursively expanded in V1 — only top-level scalar fields are mapped.

Use case: JSON Schema to GraphQL type mapping for return_schema ensures functions with custom shapes have typed fields.

Code: provisa/compiler/, provisa/api/admin/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_function_gen.py, tests/unit/test_schema_service.py

REQ-345 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa exposes a POST /query/cypher endpoint that accepts a Cypher SELECT query and optional named parameters ($param). The query is compiled to SQL and executed via Trino. All existing governance (RLS, column masking, domain visibility, row ceiling) applies via Stage 2 identically to GraphQL-compiled queries.

Use case: Graph-native users write Cypher against Provisa without learning GraphQL or SQL.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py, provisa-ui/e2e/cypher-api.spec.ts

REQ-346 · Cypher Query Frontend (Phase AU)¶

Status: ✗ rejected · Priority: MUST · Type: constraint

SUPERSEDED BY REQ-818. Originally: the Cypher compiler is strictly read-only, rejecting any write clause (CREATE/MERGE/SET/DELETE/DETACH/REMOVE) or APOC reference at parse time. Cypher now supports governed CREATE/DELETE/SET writes (REQ-818); only MERGE/DETACH/REMOVE and APOC remain rejected. Use REQ-818 and REQ-671 instead.

Use case: Read-only enforcement prevents graph users from accidentally mutating data via Cypher.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-347 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher clauses map to SQL as follows: MATCH → JOIN; OPTIONAL MATCH → LEFT JOIN; WHERE → WHERE; RETURN → SELECT; ORDER BY → ORDER BY; SKIP/LIMIT → OFFSET/LIMIT; WITH (pipeline) → CTE or subquery. Node label :Label and relationship type :TYPE are resolved via the steward-declared label mapping (REQ-351) to physical table and join references.

Use case: Cypher clause mapping enables graph queries to execute as governed SQL via Trino.

Code: provisa/cypher/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-348 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Path queries — shortestPath(...), allShortestPaths(...), and variable-length relationship patterns [*1..n] — translate to Trino recursive CTEs (WITH RECURSIVE) against the adjacency relation defined in the label mapping. Maximum hop depth is enforced at compile time; unbounded [*] is rejected.

Use case: Path queries and variable-length traversals translate to recursive CTEs for graph algorithms over relational data.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-349 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When a RETURN clause references a whole node variable, relationship variable, or path variable (not a scalar property), a Stage 3 SQLGlot rewrite pass wraps the projected columns for that variable into a single JSON object column using CAST(ROW(...) AS JSON). This rewrite runs after Stage 2 governance and before execution. It does not modify scalar property projections.

Use case: Node/edge/path return types give graph clients structured output matching their mental model.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-350 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Three graph output types are defined in the GraphQL schema and returned when Stage 3 wrapping is applied: Node { id: ID!, label: String!, properties: JSON }, Edge { id: ID!, type: String!, startNode: Node!, endNode: Node!, properties: JSON }, Path { nodes: [Node!]!, edges: [Edge!]! }. Scalar property projections return plain column types as normal.

Use case: Typed graph output (Node, Edge, Path) lets graph clients parse results without custom deserialization.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-351 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: structural

The Cypher label map is derived entirely from the existing Provisa CompilationContext — no separate config. TableMeta.type_name (PascalCase GraphQL type name) becomes the Cypher node label; registered JoinMeta entries become relationship types. Consumers write Cypher using the same type names already visible in the Provisa GraphQL schema. No additional registration step is required.

Use case: Cypher label map derived from existing compilation context — no extra registration step for Cypher users.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-352 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher named parameters ($param) are translated to Trino positional parameters at compile time. Parameter types are inferred from the label mapping schema. Missing parameters with no default are rejected at compile time.

Use case: Named parameters in Cypher translate to parameterized SQL; prevents injection and enables query reuse.

Code: provisa/cypher/

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-353 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

WITHDRAWN (2026-06-19). Cross-source Cypher queries are allowed — Trino joins across catalogs natively, so a query whose labels resolve to tables on different sources translates and executes normally. No cross-source restriction is enforced. (Supersedes REQ-481.)

Use case: Cross-source graph traversal is a core capability, not an error.

Code: provisa/cypher/

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-354 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa exposes a POST /query/nl endpoint accepting a natural language question. The service submits an async job and returns a job_id immediately. The consumer polls GET /query/nl/{job_id} or receives the result via SSE when complete.

Use case: Natural language entry point for non-technical users who cannot write SQL, GraphQL, or Cypher.

Code: provisa/nl/

Tests: tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py, tests/integration/test_nl_endpoint.py, provisa-ui/e2e/cypher-api.spec.ts

REQ-355 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

For each submitted NL query the service runs three independent parallel generation loops — one targeting Cypher, one GraphQL, and one SQL. Each loop uses an LLM to generate a candidate query, validates it through the existing Provisa compiler pipeline (parse + compile, no execution), feeds any compiler error back to the LLM as a correction signal, and retries until the query is valid or a max iteration limit is reached. The three loops are independent — failure in one does not block the others.

Use case: Parallel three-target generation maximizes answer quality; one valid result is enough to respond.

Code: provisa/nl/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_nl_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py, tests/steps/steps_natural_language_query_service_phase_av.py

REQ-356 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: MUST · Type: constraint

The LLM prompt for each generation loop is scoped to the consumer's role. The schema context supplied to the LLM is the GraphQL SDL visible to that role — tables, fields, and relationships the consumer is not permitted to see are absent from the prompt. Generated queries referencing invisible schema elements are rejected by the compiler, triggering a refinement loop iteration. Governance is enforced at two layers: prompt scoping and compiler validation.

Use case: Role-scoped LLM prompt prevents the AI from generating queries referencing data the user cannot see.

Code: provisa/nl/

Tests: tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py, tests/integration/test_nl_endpoint.py

REQ-357 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Once all three generation loops produce valid queries, all three are executed in parallel via the standard Provisa pipeline (Stage 2 governance applies to each). The response includes all three query texts and their results: { cypher: { query, result }, graphql: { query, result }, sql: { query, result } }. A branch that exhausts its iteration limit returns { query: null, result: null, error: "<last compiler error>" } for that branch without blocking the response.

Use case: All three query forms returned lets power users choose the most legible or efficient representation.

Code: provisa/nl/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_nl_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py, tests/steps/steps_natural_language_query_service_phase_av.py

REQ-358 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The NL query service is differentiated from commodity text-to-SQL tools by: (1) three-target generation (SQL, GraphQL, Cypher) — Cypher output for graph-shaped questions is not available in any comparable tool; (2) role-scoped schema context — the LLM cannot generate queries referencing data the consumer is not permitted to see; (3) compiler-driven refinement loop — deterministic validation replaces heuristic result quality checks.

Use case: Differentiator: only platform offering role-scoped, compiler-validated, multi-target NL query generation.

Code: provisa/nl/

Tests: tests/integration/test_nl_endpoint.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py, tests/steps/steps_natural_language_query_service_phase_av.py

REQ-359 · Natural Language Query Service (Phase AV)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Any valid query produced by the NL service is executed directly under the consumer's rights, and may be saved by the consumer for reuse. Generated queries are governed by the same rights + Stage 2 enforcement as hand-written queries — the NL service never bypasses governance.

Use case: NL-generated queries run under the consumer's rights with full Stage 2 governance — never bypassed.

Code: provisa/nl/

Tests: tests/integration/test_direct_exec.py, tests/integration/test_nl_endpoint.py, tests/unit/test_nl_loop.py, tests/unit/test_nl_runner.py

REQ-360 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Action query fields (tracked functions and webhooks with exposed_as: query) must support standard filter/sort/pagination GraphQL arguments: where, order_by, limit, offset — applied as Python post-processing after the function executes and results are materialized.

Use case: Filter/sort/pagination on action query fields gives function-backed results the same query ergonomics as tables.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-361 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Action query fields returning a known table type must resolve nested relationship fields by batching lookups against the related source table and merging results back onto each row. Relationship resolution applies the same governance rules (column visibility, RLS, column masking) as direct table queries.

Use case: Governed relationship resolution on action results ensures nested fields respect RLS, masking, and visibility like direct queries.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-362 · Tracked Functions & Custom Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

One-to-many relationships on action result rows must return an array field; many-to-one relationships must return an object field or null. Relationship cardinality is sourced from the JoinMeta declarations that define the table relationships.

Use case: Cardinality-correct relationship fields (array vs object) on action results match the JoinMeta declarations.

Code: provisa/compiler/, provisa/executor/, provisa/webhooks/

Tests: tests/e2e/test_mutations.py, tests/integration/test_compiler_integration.py, tests/unit/test_actions.py, tests/unit/test_function_gen.py

REQ-403 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: behavioral

RLS injection via inject_rls() checks for table-specific rules first, then falls back to domain-level rules; table-level rules take precedence.

Use case: Table-level RLS precedence lets stewards override domain rules selectively for exceptions without duplicating configuration.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_rls_compiler_fallback.py, tests/unit/test_schema_service.py

REQ-409 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher translator detects ISO 8601 datetime string literals in WHERE clauses and wraps them as TIMESTAMP '...' before SQLGlot parsing, fixing Trino type mismatch errors when filtering on timestamp primary key columns.

Use case: Automatic timestamp coercion eliminates manual casting for timestamp PK comparisons in Cypher queries.

Code: provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_schema_service.py

REQ-411 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Naming convention option hasura-default provides snake_case mutation names (insert_orders, update_orders, delete_orders) and snake_case field names, matching Hasura V2's default naming style.

Use case: Alternative naming convention supports teams migrating from Hasura by providing compatible mutation and field name generation.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-412 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Naming convention option graphql-default provides camelCase field names, PascalCase types, and camelCase mutations (insertOrders, updateOrders), matching Hasura V2's graphql-default and GraphQL best practices. This is now the Provisa default.

Use case: GraphQL best-practice naming convention improves developer experience and aligns with ecosystem defaults.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-416 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: structural

Naming convention now uses three preset enums (snake, hasura_graphql, apollo_graphql default) with domain_prefix: bool as orthogonal option. snake (PascalCase types, snake_case fields/mutations), hasura_graphql (PascalCase types, camelCase fields, snake_case mutations matching Hasura V2), apollo_graphql (PascalCase types, camelCase fields/mutations per Apollo/GraphQL idiom). Old free-form strings deprecated.

Use case: Enum-based preset naming eliminates ambiguous string values and standardizes on clear, documented conventions compatible with Hasura and Apollo ecosystems.

Code: provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_naming_convention_validation.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-478 · Compiler & Schema¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Statistical row sampling is a user query feature, not governance. Root query fields accept an optional sample: Float argument (percentage in (0, 100]); the compiler emits TABLESAMPLE BERNOULLI (<pct>) on the base table, which SQLGlot transpiles per dialect (direct PG and Trino both supported). Out-of-range values are rejected at compile time. Not combinable with as_of (time-travel) or op-relationship lateral joins. Raw-SQL clients use native TABLESAMPLE directly. Distinct from the REQ-263 row cap, which bounds result size.

Use case: Statistical sampling lets analysts request a random subset for exploration, separate from the governance row cap.

Code: provisa/compiler/sql_gen.py, provisa/compiler/schema_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_schema_service.py, tests/unit/test_sql_gen.py

REQ-525 · Compiler & Schema¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Auto-generated .proto from the data schema is generated per role. Each role receives a proto definition reflecting only the tables and columns visible to that role.

Use case: Per-role proto generation ensures gRPC clients only see schema elements they are authorized to access.

Code: provisa/grpc/proto_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_grpc_requirements.py, tests/unit/test_schema_service.py

REQ-526 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

gRPC server reflection is enabled on the Provisa Protobuf gRPC server (port 50051), allowing clients to discover available RPCs and message schemas at runtime without a pre-shared proto file.

Use case: Server reflection enables tooling (grpcurl, grpcui, Postman) to discover gRPC services without distributing proto files.

Code: provisa/grpc/reflection.py, provisa/grpc/server.py

Tests: tests/e2e/test_grpc_query.py, tests/integration/test_compiler_integration.py, tests/unit/test_grpc_reflection.py, tests/unit/test_grpc_server.py, tests/unit/test_schema_service.py

REQ-534 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GraphQL queries with multiple root fields are compiled into separate SQL queries and executed independently. Results are merged into a single response: fields below the redirect threshold are returned inline in data; fields above the threshold are redirected with per-field entries in redirects. Binary formats (Parquet, Arrow) are only supported for single-root queries.

Use case: Multi-root query execution lets GraphQL clients fetch multiple independent datasets in a single request while applying per-field redirect thresholds.

Code: provisa/api/data/endpoint.py

Tests: tests/integration/test_compiler_integration.py, tests/integration/test_direct_exec.py, tests/unit/test_compiler_requirements.py, tests/unit/test_multi_root_queries.py, tests/unit/test_schema_service.py

REQ-537 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GET /data/schema-version returns a string combining a per-boot UUID nonce with a monotonically incrementing rebuild counter in the format <boot-id>-<counter>. Clients use this value to detect schema changes and invalidate local schema caches.

Use case: Schema version endpoint lets clients detect server-side schema changes and re-fetch SDL or introspection without polling.

Code: provisa/api/data/sdl.py

Tests: tests/integration/test_schema_gen.py, tests/unit/test_compiler_requirements.py, tests/unit/test_schema_service.py, provisa-ui/e2e/cypher-api.spec.ts

REQ-571 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: structural

The Cypher parser is a custom recursive-descent implementation with no dependency on any external Cypher library. It produces a CypherAST from read-only Cypher queries.

Use case: Custom parser with no external library dependency gives full control over the parse grammar and error messages for Provisa-specific Cypher extensions.

Code: provisa/cypher/parser.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_parser.py, tests/unit/test_sql_to_cypher.py

REQ-572 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa handles CALL db.labels(), CALL db.relationshipTypes(), and CALL db.propertyKeys() as introspection procedures that return data from the in-memory semantic layer (CypherLabelMap) without generating or executing any SQL.

Use case: Returning schema introspection data from the semantic layer (not the database) lets Neo4j-compatible clients discover the graph schema via standard procedure calls.

Code: provisa/api/rest/cypher_router.py

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-573 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Correlated CALL subqueries of the form CALL { WITH x MATCH (x)-[:R]->(n) RETURN n.prop AS alias } are translated to CROSS JOIN LATERAL expressions. The outer-scope variable must appear in WITH; multiple imported variables are supported. Non-correlated top-level CALL blocks (without WITH) are handled by cypher_calls_to_sql_list.

Use case: CROSS JOIN LATERAL translation of correlated CALL subqueries lets graph clients use subquery-pattern idioms that are common in openCypher workloads.

Code: provisa/cypher/correlated_call.py, provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-574 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Relationships in Provisa exist solely as join metadata in the semantic layer (JoinMeta). They carry no stored attributes. Relationship property access (e.g. WHERE r.since > 2020 or RETURN r.weight) has no meaning and is not supported.

Use case: Restricting relationships to join metadata avoids the complexity of relationship property storage while covering the vast majority of graph traversal use cases.

Code: provisa/cypher/label_map.py, provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-575 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Bidirectional traversal syntax (a)-[]-(b) is rewritten at compile time to a UNION ALL of all matching directed forward and backward relationships from the semantic layer. Every relationship in the semantic layer is directional; bidirectional syntax is sugar that expands to both directions.

Use case: Bidirectional sugar covers common graph traversal patterns without requiring stewards to register both directions of every relationship.

Code: provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-576 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When shortestPath endpoints have different node types and no self-referential relationship exists in the schema, the translator emits a flat JOIN chain (structurally shortest schema path) rather than a recursive CTE with ORDER BY hops. Hops are not tracked in this code path, so the result is the structurally shortest schema path, not the data-shortest path across multiple rows.

Use case: Flat JOIN for heterogeneous shortestPath avoids unnecessary recursive CTE overhead when the schema topology determines the path uniquely.

Code: provisa/cypher/path_translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_path_translator.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-577 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When multiple schema paths of equal hop count connect the same start and end node types, all matching paths are emitted as UNION ALL branches. Row-level deduplication across branches is not performed.

Use case: Emitting all schema paths as UNION ALL branches ensures no valid traversal is silently omitted when multiple relationship types connect the same pair of node types.

Code: provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-578 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MAY · Type: constraint

WITH clause CTEs are named _w0, _w1, ... using a positional index assigned within a single translation call. Composing multiple independently translated queries (e.g. in a batch) can produce colliding CTE names if the results are concatenated without renaming.

Use case: Positional CTE naming is deterministic within one translation call; the collision risk on batch composition is a known limitation that callers must handle.

Code: provisa/cypher/translator.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-642 · Graph Analytics Pipeline¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A POST /data/graph-analytics endpoint accepts a Cypher query and algorithm name, executes the query via the existing cypher_router pipeline, builds an in-memory NetworkX DiGraph from the resulting nodes and edges, runs the named algorithm, merges a _analytics dict into each node/edge, and returns the augmented nodes and edges as JSON with an elapsed_ms field.

Use case: Server-side graph analytics endpoint lets the browser request algorithm output on any Cypher-defined subgraph without building the graph client-side.

Code: provisa/api/rest/graph_analytics_router.py

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_compiler_integration.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_graph_analytics_requirements.py, provisa-ui/e2e/cypher-api.spec.ts

REQ-643 · Graph Analytics Pipeline¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The graph analytics response merges a _analytics dict into every node and edge in the result. The keys present in _analytics vary by algorithm: centrality algorithms produce score; community detection produces cluster; k-core produces core_number; degree centrality also produces in_degree and out_degree.

Use case: Uniform _analytics key convention lets the UI apply visual encodings without algorithm-specific branching.

Code: provisa/api/rest/graph_analytics_router.py

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_compiler_integration.py, tests/unit/test_graph_analytics_requirements.py

REQ-650 · Graph Analytics Pipeline¶

Status: ✅ complete · Priority: MUST · Type: constraint

The graph analytics endpoint enforces a configurable maximum graph size. When the input graph exceeds the configured limit (default: 10,000 nodes or 50,000 edges), the endpoint returns HTTP 413 before running any algorithm.

Use case: Graph size cap prevents unbounded memory allocation from oversized graphs passed to in-process algorithm libraries.

Code: provisa/api/rest/graph_analytics_router.py

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_compiler_integration.py, tests/unit/test_graph_analytics_requirements.py

REQ-651 · Graph Analytics Pipeline¶

Status: ✅ complete · Priority: MUST · Type: constraint

The Girvan-Newman community detection algorithm is restricted to graphs with fewer than 500 nodes. Requests for Girvan-Newman on larger graphs are rejected unless the caller supplies force=true in the params, making the computational risk explicit.

Use case: Girvan-Newman restriction prevents O(n³) runtime from making large-graph analytics requests impractical.

Code: provisa/api/rest/graph_analytics_router.py

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_compiler_integration.py, tests/unit/test_graph_analytics_requirements.py

REQ-653 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Table-level enable_aggregates and enable_group_by flags in provisa.yaml (both default false). enable_aggregates replaces the role-level no_aggregations capability as the primary gate for {table}_aggregate root fields. enable_group_by gates {table}_group_by root fields. Stewards opt in per table, controlling SDL noise at the table level rather than globally per role.

Use case: Steward can expose aggregate and group-by fields only on tables where they are meaningful, keeping the SDL small and navigable for API consumers.

Code: provisa/compiler/schema_gen.py, provisa/core/models.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_core_registration.py, tests/unit/test_schema_service.py, provisa-ui/e2e/tables-enable-aggregates.spec.ts

REQ-654 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

{table}_group_by root query field. Requires by: [SelectColumn!]!; accepts same args as the base table query (where, order_by, limit, offset, distinct_on). Returns [{TypeName}GroupByRow!]! where {TypeName}GroupByRow has groupKey: JSON! (object mapping group-by column names to their values) and aggregates: {TypeName}AggregateFields!. Reuses existing {TypeName}AggregateFields type unchanged. Maps to SELECT <by_cols>, <agg_funcs> FROM <table> WHERE <where> GROUP BY <by_cols> ORDER BY <order_by> LIMIT <limit> OFFSET <offset>.

Use case: Exposes grouped analytics (e.g. count per user, sum per category) as a first-class query without requiring a database view.

Code: provisa/compiler/schema_gen.py, provisa/compiler/aggregate_gen.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_group_by.py, tests/unit/test_schema_service.py

REQ-655 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

{table}_group_by supports two additional filter clauses beyond the base where: (1) aggregates(where: {TypeName}BoolExp) — a where arg on the aggregates field of {TypeName}GroupByRow that applies conditional aggregation within each group (SQL FILTER (WHERE ...)); (2) having: {TypeName}HavingExp arg on the root field that filters groups post-aggregation (SQL HAVING). {TypeName}HavingExp is a new input type with the same numeric/comparable fields as {TypeName}AggregateFields but with comparison operators. Three filters compose orthogonally: _group_by(where:) → SQL WHERE, aggregates(where:) → SQL FILTER, having: → SQL HAVING.

Use case: Enables queries like "users with more than 3 active inquiries" without client-side filtering or custom views.

Code: provisa/compiler/aggregate_gen.py, provisa/compiler/sql_gen.py

Tests: tests/integration/test_compiler_integration.py, tests/unit/test_group_by.py, tests/unit/test_schema_service.py

REQ-661 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

Labels in Cypher write clauses (CREATE, SET, DELETE) must resolve to exactly one NodeMeta via the semantic label map. Ambiguous or unknown labels are hard errors at translation time; no fuzzy matching or implicit selection is permitted.

Use case: Single-label resolution ensures write operations are unambiguous and cannot accidentally target the wrong table.

Code: provisa/cypher/write_translator.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-662 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

Labels in write operations must already exist in the governed semantic label map. New labels cannot be created via Cypher write clauses — only pre-registered tables are writable.

Use case: Restricting writes to pre-registered tables ensures schema integrity and prevents untracked table creation.

Code: provisa/cypher/write_translator.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-663 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

All Cypher write operations (CREATE, SET, DELETE) are gated on writable_by ACL defined on the target table's NodeMeta. If a role lacks write permission, the query is rejected at compile time. The check applies uniformly to CREATE, SET, and DELETE.

Use case: Consistent write access control across all mutation types prevents unauthorized writes to sensitive tables.

Code: provisa/cypher/write_translator.py, provisa/api/rest/cypher_router.py, provisa/api/data/endpoint.py, provisa/compiler/stage2.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_mutations.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-664 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

A source connector must explicitly support DML (INSERT, UPDATE, DELETE) to be writable. Read-only connectors (e.g., Trino-federated, iceberg catalog without Delta connector) reject write operations at translation time with a clear error message.

Use case: Restricting writes to DML-capable sources prevents silent failures on read-only systems.

Code: provisa/cypher/write_translator.py, provisa/core/models.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-665 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

Relationships in Cypher cannot be written — they are derived from foreign-key joins in the semantic layer, not stored edges. CREATE, SET, and DELETE cannot target relationship entities. Attempting to write a relationship is a hard error at translation time.

Use case: Relationship immutability prevents accidental FK manipulation and keeps relationship metadata consistent with the declared join paths.

Code: provisa/cypher/write_translator.py, provisa/cypher/label_map.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_mutations.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-666 · Cypher Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

CREATE (n:Label {props}) is translated to INSERT INTO catalog.schema.table (columns) VALUES (values). Property names in the Cypher object are mapped to table columns via domain-prefix stripping and alias resolution. Type coercion aligns Cypher scalar values with target column types.

Use case: Property-to-column mapping with type coercion enables straightforward graph-idiomatic CREATE statements without explicit SQL casting.

Code: provisa/cypher/write_translator.py, provisa/cypher/select_builder.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py, tests/integration/test_cypher_endpoint.py

REQ-667 · Cypher Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

MATCH (n:Label) WHERE ... DELETE n is translated to DELETE FROM catalog.schema.table WHERE .... The WHERE clause is reused from the existing MATCH translator; pattern conditions must compile to standard SQL WHERE predicates before deletion.

Use case: WHERE clause reuse avoids duplicating filter translation logic between read and write paths.

Code: provisa/cypher/write_translator.py, provisa/cypher/select_builder.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py, tests/integration/test_cypher_endpoint.py

REQ-668 · Cypher Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

MATCH (n:Label) WHERE ... SET n.prop = val is translated to UPDATE catalog.schema.table SET column = value WHERE .... Property-to-column mapping applies domain-prefix stripping; assignment expressions must be valid SQL expressions. Multiple SET clauses compose as comma-separated column updates.

Use case: Property-to-column mapping with multi-assignment composition lets graph clients issue idiomatic UPDATE patterns.

Code: provisa/cypher/write_translator.py, provisa/cypher/select_builder.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_label_map.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py, tests/integration/test_cypher_endpoint.py

REQ-669 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: structural

Write operations use a separate parser entry point or translation flag to avoid breaking the read-only query path. Existing read-only parser must remain unchanged. The WriteTranslator class encapsulates CREATE, SET, and DELETE translation logic separately from QueryTranslator.

Use case: Separate write translation path prevents mutation syntax from breaking existing read-only query parsing and execution.

Code: provisa/cypher/write_translator.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_path_translator.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

REQ-670 · Cypher Mutations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cypher write endpoints return the number of rows affected (rows inserted for CREATE, rows updated for SET, rows deleted for DELETE). The response includes an affected_rows field in the JSON response body.

Use case: Affected-row counts let clients validate mutation success and detect silent failures.

Code: provisa/api/rest/cypher_router.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py, tests/integration/test_cypher_endpoint.py, provisa-ui/e2e/cypher-write-affected-rows.spec.ts

REQ-671 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: constraint

DETACH DELETE (cascading delete across derived relationships) is not supported. Attempting DETACH DELETE is a hard error at parse or translation time. Clients must explicitly DELETE each table in the correct dependency order.

Use case: Explicit cascade prevention avoids accidental data deletion across unintended table relationships.

Code: provisa/cypher/write_translator.py, provisa/cypher/parser.py

Tests: tests/e2e/test_mutations.py, tests/unit/test_cypher_translator.py, tests/unit/test_sql_to_cypher.py

6. Execution, Routing, Caching & Performance¶

REQ-027 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Single-source queries route to direct RDBMS connection; SQLGlot transpiles to target dialect. Target: sub-100ms to low hundreds of ms.

Use case: Single-source direct routing delivers sub-100ms query latency for the majority of requests.

Code: provisa/executor/, provisa/transpiler/

Tests: tests/e2e/test_routing.py, tests/integration/test_direct_exec.py, tests/integration/test_execution_routing.py, tests/unit/test_routing.py, provisa-ui/e2e/execution-routing.spec.ts

REQ-028 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cross-source queries route to Trino; SQLGlot transpiles to Trino SQL. Target: 300-500ms.

Use case: Cross-source Trino routing enables federated queries across databases within a 300–500ms target.

Code: provisa/executor/, provisa/transpiler/

Tests: tests/unit/test_routing.py, tests/e2e/test_routing.py, tests/integration/test_execution_routing.py, provisa-ui/e2e/execution-routing.spec.ts

REQ-029 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Large results above threshold redirect to blob storage with presigned URL and TTL.

Use case: Large-result redirect to blob storage prevents memory exhaustion on the Provisa server for big datasets.

Code: provisa/executor/, provisa/transpiler/

Tests: tests/e2e/test_large_result.py, tests/e2e/test_routing.py, tests/integration/test_direct_exec.py, tests/integration/test_execution_routing.py, tests/unit/test_redirect_s3.py, tests/unit/test_routing.py, provisa-ui/e2e/execution-routing.spec.ts

REQ-030 · Execution & Routing¶

Status: ✅ complete · Priority: MAY · Type: structural

Steward routing override configurable per table/view for edge cases where default routing is inappropriate.

Use case: Per-table steward routing overrides handle edge cases where the default routing heuristic is suboptimal.

Code: provisa/executor/, provisa/transpiler/

Tests: tests/unit/test_routing.py, tests/e2e/test_routing.py, tests/integration/test_execution_routing.py

REQ-031 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: constraint

DB mutations ALWAYS route to direct RDBMS connection — Trino never involved in mutation execution. Webhook mutations are NOT DB mutations: they reference an external HTTP endpoint that presumably changes underlying data. Provisa cannot verify whether anything changed, so this is the steward's responsibility to define (see REQ-209).

Use case: DB mutations always bypass Trino, ensuring writes are atomic and target the correct source directly.

Code: provisa/executor/, provisa/transpiler/

Tests: tests/e2e/test_routing.py, tests/integration/test_direct_exec.py, tests/integration/test_execution_routing.py, tests/unit/test_routing.py, provisa-ui/e2e/execution-routing.spec.ts

REQ-052 · Data & Storage¶

Status: ✅ complete · Priority: MUST · Type: structural

Each registered RDBMS source maintains warm connection pool; min pool size configurable per source.

Use case: Warm connection pools eliminate per-request TCP handshake latency for frequently queried RDBMS sources.

Code: provisa/executor/, provisa/core/

Tests: tests/unit/test_connection_pool.py, tests/integration/test_pool.py

REQ-053 · Data & Storage¶

Status: ✅ complete · Priority: MAY · Type: structural

PgBouncer is the documented per-PostgreSQL-source opt-in via use_pgbouncer (default off); it is NOT forced on, because a default-on would require a running PgBouncer for every PG source. Default is direct asyncpg pooling (REQ-052).

Use case: Optional PgBouncer per-source avoids mandatory sidecar infrastructure while offering connection pooling where needed.

Code: provisa/executor/, provisa/core/

Tests: tests/integration/test_direct_exec.py, tests/integration/test_pool.py, tests/unit/test_connection_pool.py

REQ-054 · Data & Storage¶

Status: ✅ complete · Priority: MUST · Type: structural

Trino read path maintains single persistent connection to coordinator.

Use case: Single persistent Trino coordinator connection avoids repeated authentication and session setup overhead.

Code: provisa/executor/, provisa/core/

Tests: tests/unit/test_connection_pool.py, tests/integration/test_pool.py

REQ-230 · Hot Tables (Redis-Cached Lookups)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Hot tables stored as a single JSON blob in Redis and injected as a VALUES CTE; column governance (RLS/masking/visibility) is NOT applied at storage time but by Stage-2 apply_governance over the governed SQL that wraps the CTE (pipeline order: governance -> cache/CTE -> route). The per-row PK-hash/sorted-set storage in the spec was judged unnecessary because hot tables are tiny inline tables.

Use case: Centralised JSON blob storage with governance applied at query time simplifies cache maintenance and ensures all governance rules are enforced on hot data.

Code: provisa/cache/

Tests: tests/integration/test_cache_store.py, tests/integration/test_hot_tables_real.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_cache_store.py, tests/unit/test_hot_tables.py, tests/unit/test_mask_inject.py

REQ-231 · Hot Tables (Redis-Cached Lookups)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Hot table refresh follows MV pattern: TTL-based via refresh_interval (default: materialized_views.default_ttl), background refresh loop, stale fallback to live query. Mutations to the source table trigger immediate invalidation + async reload.

Use case: TTL-based hot table refresh keeps cached lookup data current without manual cache invalidation.

Code: provisa/cache/

Tests: tests/integration/test_hot_tables_real.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_hot_tables.py, tests/unit/test_hot_tables_integration.py

REQ-232 · Hot Tables (Redis-Cached Lookups)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Hot table JOIN optimization -- when a query joins a hot table (e.g., orders JOIN countries ON country_code), the compiler injects the hot table data as constants into the SQL via a VALUES-based CTE. The DB engine sees literal rows, not a table reference. This eliminates the second table scan and works across sources (constants travel with the query to Trino/Snowflake/ClickHouse). SQLGlot transpiles the VALUES clause per dialect.

Use case: Hot table JOIN optimisation eliminates a second table scan by injecting lookup data as SQL constants.

Code: provisa/cache/

Tests: tests/integration/test_hot_tables_real.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_hot_tables.py, tests/unit/test_hot_tables_integration.py

REQ-233 · Hot Tables (Redis-Cached Lookups)¶

Status: ✅ complete · Priority: MUST · Type: constraint

Hot caching is bounded by both max_rows (own config default, falls back to auto_threshold) and max_bytes (default 10MB, measured after JSON serialization) — a wide table within max_rows but over the byte ceiling is not cached.

Use case: Dual byte and row limits prevent hot caching of unexpectedly wide tables that would exceed Redis memory budgets.

Code: provisa/cache/

Tests: tests/integration/test_hot_tables_real.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_hot_tables.py

REQ-234 · Materialized View Lifecycle¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-materialized view storage reclamation -- when a view is removed from config, disabled, or its source table is unregistered, the backing MV table is dropped (Trino DROP TABLE). Background cleanup task runs on config reload and periodically (default: daily). Orphaned MV tables (present in target schema but not in MV registry) are flagged and optionally auto-dropped after a grace period.

Use case: MV storage reclamation prevents orphaned Iceberg tables accumulating when views are removed from config.

Code: provisa/mv/

Tests: tests/integration/test_execution_routing.py, tests/unit/test_config_reload.py, tests/unit/test_mv_lifecycle.py

REQ-235 · Materialized View Lifecycle¶

Status: ✅ complete · Priority: MUST · Type: constraint

Auto-materialized aggregate views must have a size guard -- materialized_views.max_rows (default: 1,000,000). Views whose source query would produce more rows than the limit skip materialization and fall back to live execution. Size estimated via SELECT COUNT(*) probe before CTAS. Configurable per view to override the global default.

Use case: MV size guard prevents materialising oversized views that would exhaust storage or exceed refresh budgets.

Code: provisa/mv/

Tests: tests/unit/test_mv_lifecycle.py, tests/integration/test_execution_routing.py

REQ-236 · Hot Table Auto-Detection¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A table is auto-hot if it is the target of a many-to-one relationship OR its row count is <= auto_threshold (SELECT COUNT(*) at schema build via detect_hot_tables_by_count).

Use case: Auto-detection of small lookups and relationship targets caches them in Redis without explicit config.

Code: provisa/cache/

Tests: tests/unit/test_hot_tables.py, tests/integration/test_hot_tables_real.py

REQ-237 · Hot Table Auto-Detection¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Auto-hot tables can be opted out via hot: false on the table config. Explicit hot: true overrides the auto-detection criteria (forces caching regardless of row count or relationship status). Auto-detection runs on every schema rebuild; tables that grow beyond the threshold are automatically evicted from Redis on next rebuild.

Use case: Explicit hot:false opt-out lets stewards prevent specific tables from being auto-cached in Redis.

Code: provisa/cache/

Tests: tests/unit/test_hot_tables.py, tests/integration/test_hot_tables_real.py

REQ-238 · Warm Tables (Replica Profile — Low-Latency Placement)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Warm tables -- frequently queried RDBMS tables materialized into a low-latency, engine-reachable location. In materialization_store terms (REQ-844) this is the warm profile of the REPLICA role — an engine-relative low-latency PLACEMENT of a reachable source (distinct from reactive, which lands an unreachable source). The mechanism below is the current Trino realization of that placement rather than its definition — it materializes into the Iceberg results catalog so Trino's built-in file system cache (fs.cache.enabled=true) caches the Parquet files on local SSD; other engines realize the low-latency tier differently. Provides ~10-50ms reads vs 100ms+ network round-trip to remote source. Same TTL/refresh pattern as MVs.

Use case: Warm tables on local SSD reduce RDBMS round-trip latency from 100ms+ to ~10-50ms for frequent queries.

Code: provisa/mv/

Tests: tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_catalog_cache.py, tests/unit/test_warm_tables.py

REQ-239 · Warm Tables (Replica Profile — Low-Latency Placement)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Warm table auto-promotion -- track query frequency per table (increment counter on each compiled query). Tables exceeding warm_tables.query_threshold (default: 100 queries per refresh interval) are auto-materialized into Iceberg. Tables falling below the threshold are demoted (backing table dropped) on next refresh cycle.

Use case: Auto-promotion of frequently queried tables to warm tier improves performance without manual configuration.

Code: provisa/mv/

Tests: tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_warm_tables.py, tests/integration/test_execution_routing.py

REQ-240 · Warm Tables (Replica Profile — Low-Latency Placement)¶

Status: ✅ complete · Priority: MAY · Type: structural

Warm table config: warm: true to force, warm: false to opt out. Global warm_tables.query_threshold, warm_tables.max_rows (default: 10,000,000), warm_tables.refresh_interval (default: same as materialized_views.default_ttl). Trino file cache config (fs.cache.enabled, fs.cache.directories, fs.cache.max-sizes) managed in Trino catalog properties.

Use case: Warm table config guards (max_rows, threshold) prevent promotion of tables too large for local SSD cache.

Code: provisa/mv/

Tests: tests/integration/test_direct_exec.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_catalog_cache.py, tests/unit/test_warm_tables.py

REQ-241 · Warm Tables (Replica Profile — Low-Latency Placement)¶

Status: ✅ complete · Priority: MUST · Type: constraint

A table lives in at most one cache tier — hot wins over warm. WarmTableManager.check_promotions skips any table the hot tier manages (HotTableManager.managed_tables()).

Use case: Strict tier exclusivity prevents hot tables from being accidentally promoted to warm, ensuring optimal latency.

Code: provisa/mv/

Tests: tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_warm_tables.py, tests/integration/test_execution_routing.py

REQ-275 · Federation Performance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

On source registration, Provisa runs ANALYZE against the registered source's tables (where the connector supports it) to prime the federation engine's cost-based optimizer with baseline row counts and column statistics.

Use case: Running ANALYZE on registration primes the federation cost-based optimizer so join plans are efficient immediately.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-276 · Federation Performance¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Admin API exposes a "Refresh Statistics" mutation per source that re-runs ANALYZE on demand. Useful for volatile sources where baseline stats have aged.

Use case: On-demand Refresh Statistics mutation lets stewards refresh cost stats for volatile sources without a restart.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-277 · Federation Performance¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Per-query and per-table/view session property overrides: steward or developer can attach named session hints (join_distribution_type, join_max_broadcast_table_size, join_reordering_strategy). Provisa injects matching SET SESSION statements before execution.

Use case: Per-query session property hints let stewards tune federation join strategies for known expensive queries.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-278 · Federation Performance¶

Status: ✅ complete · Priority: MAY · Type: structural

Source-level default session properties: each registered source config accepts a federation_hints: block. Properties in that block are injected for any query that touches that source. Per-query hints override source-level defaults.

Use case: Source-level federation hints apply performance tuning to all queries touching a source without per-query config.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-279 · Federation Performance¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Provisa-branded comment hint syntax /*+ hint */ in query text. Supported hints: BROADCAST(<table>), NO_REORDER, BROADCAST_SIZE(<size>). Provisa parser strips the comment before forwarding SQL to the federation engine and translates to the equivalent session properties. The federation engine never sees the comment.

Use case: Provisa-branded hint syntax decouples query authors from Trino session property names and future engine changes.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-280 · Federation Performance¶

Status: ✅ complete · Priority: MUST · Type: behavioral

ANALYZE runs on the API cache table after each materialization CTAS; ANALYZE failure is logged, not raised (connector tolerance, matching analyze_source_tables / REQ-275).

Use case: ANALYZE after API cache CTAS keeps cost estimates fresh without failing the entire materialization on ANALYZE errors.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-281 · Federation Performance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Source-level federation_hints use the Provisa-branded @provisa vocabulary (join=broadcast|partitioned, reorder=none|auto, broadcast_size=<size>), translated to Trino session props by provisa/compiler/directives.py:translate_federation_hints at query time. Raw Trino session-prop keys still pass through (deprecated) for backward compatibility.

Use case: Source-level federation hints vocabulary is translated to Trino session props, keeping query authors decoupled from engine-specific property names.

Code: provisa/executor/, provisa/compiler/

Tests: tests/integration/test_direct_exec.py, tests/integration/test_execution_routing.py, tests/integration/test_federation_integration.py, tests/unit/test_compiler_requirements.py, tests/unit/test_federation.py, tests/unit/test_federation_hints.py, tests/unit/test_federation_v2_extended.py

REQ-397 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

When a PK is available, the exclusion WHERE clause must use n.<pk_col> IN [<pk_value>] instead of id(n) IN [<nodeId>].

Use case: PK-based exclusion is semantic and stable across node re-materialization.

Code: provisa-ui/src/components/graph/graph-model.ts, provisa/cypher/

Tests: tests/unit/test_graph_exclusion.py, tests/integration/test_execution_routing.py

REQ-536 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

All data responses include cache status headers: X-Provisa-Cache: HIT|MISS on every response, and X-Provisa-Cache-Age: <seconds> on cache HITs, indicating how many seconds old the cached result is.

Use case: Cache headers let clients and proxies distinguish fresh vs cached results and detect stale responses.

Code: provisa/cache/middleware.py

Tests: tests/unit/test_cache_headers_requirements.py, tests/unit/test_misc_requirements.py, tests/unit/test_cache_headers_requirements.py, tests/integration/test_execution_routing.py, provisa-ui/e2e/security-rate-limiting.spec.ts

REQ-544 · Cache¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cache TTL resolves in order: table-level TTL > source-level TTL > global default TTL. Setting cache_enabled: false on a source disables caching for all tables in that source regardless of table-level TTL. Cache keys always include role_id and RLS context values for security partitioning.

Use case: Hierarchical TTL lets operators tune cache freshness per source or per table while security-partitioned cache keys prevent cross-role result leakage.

Code: provisa/cache/policy.py, provisa/cache/key.py

Tests: tests/integration/test_execution_routing.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_cache_policy.py

REQ-552 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cross-source JOINs routed through the federation engine apply automatic type coercion when joining columns across sources with differing native types.

Use case: Automatic type coercion during cross-source joins prevents type mismatch errors when federating tables from heterogeneous source systems.

Code: provisa/transpiler/, provisa/executor/

Tests: tests/unit/test_routing.py, tests/e2e/test_routing.py, tests/integration/test_execution_routing.py

REQ-595 · Caching¶

Status: ✅ complete · Priority: MUST · Type: constraint

In multi-tenant mode, RedisCacheStore prefixes every cache key with the tenant_id (provisa:cache:<tenant_id>:<sha256_key>). Table invalidation keys follow provisa:table:<tenant_id>:<table_id>. The APQ cache uses provisa:apq:<tenant_id>:<sha256_hex>. Default APQ TTL is 86400 seconds, configurable via PROVISA_APQ_TTL. One tenant's cache flush never touches another tenant's cached results. check_cache and store_result in cache/middleware.py pass tenant_id (org_id) through to RedisCacheStore.get/set, so prefix isolation is active for query result caching.

Use case: Tenant-prefixed cache keys guarantee that cache invalidation, eviction, and key collisions cannot cross tenant boundaries on shared Redis infrastructure.

Code: provisa/cache/store.py, provisa/cache/middleware.py, provisa/apq/cache.py

Code: provisa/api/flight/

Tests: tests/integration/test_arrow_flight_integration.py, tests/unit/test_flight_server_helpers.py, tests/unit/test_zaychik_flight_unit.py

REQ-146 · Arrow Flight¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Falls back to materializing via Trino REST if Zaychik unavailable.

Use case: Fallback to Trino REST ensures Flight delivery works even when the Zaychik proxy is unavailable.

Code: provisa/api/flight/

Tests: tests/integration/test_arrow_flight_integration.py, tests/unit/test_flight_server_helpers.py, tests/unit/test_zaychik_flight_unit.py

8. Client Access & Protocols¶

REQ-043 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GraphQL endpoint is primary entry point for queries and mutations.

Use case: GraphQL as the primary entry point gives consumers a strongly typed, self-documenting API surface.

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/unit/test_rest_generator.py, tests/unit/test_jsonapi.py, tests/integration/test_client_access_integration.py

REQ-044 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Presigned URL redirect for large result consumers with TTL-bounded access.

Use case: Presigned URL redirect gives large-result consumers time-bounded, secure access without server-side buffering.

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/e2e/test_large_result.py, tests/integration/test_client_access_integration.py, tests/unit/test_jsonapi.py, tests/unit/test_redirect_s3.py, tests/unit/test_rest_generator.py

REQ-045 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

gRPC Arrow Flight endpoint for high-throughput consumers; Trino produces Arrow natively for zero-copy delivery.

Use case: Arrow Flight endpoint enables high-throughput consumers to stream data with zero-copy native Arrow delivery.

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/unit/test_rest_generator.py, tests/unit/test_jsonapi.py, tests/integration/test_arrow_flight_integration.py

REQ-126 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

JDBC driver that exposes registered tables and views as virtual tables. Connection authenticates against Provisa, maps user to role.

Use case: JDBC driver exposes registered tables/views as virtual tables, enabling any JDBC-compatible tool to query governed data.

Code: provisa-client/

Tests: tests/unit/test_drivers.py, tests/unit/test_flight_modes.py, tests/integration/test_arrow_flight_integration.py, tests/integration/test_client_access_integration.py

REQ-127 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

getTables() returns registered tables and views visible to the authenticated role, by their registered names.

Use case: getTables() returning registered tables/views lets BI tools discover governed datasets through standard JDBC metadata.

Code: provisa-client/

Tests: tests/unit/test_drivers.py

REQ-128 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

getColumns(tableName) introspects the registered table/view output schema — column names and types from compiled metadata, filtered by role visibility.

Use case: getColumns() introspection lets JDBC tools render correct column types without manual schema configuration.

Code: provisa-client/

Tests: tests/integration/test_pgwire_column_visibility_integration.py, tests/unit/test_drivers.py, tests/unit/test_view_column_types.py

REQ-129 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

executeQuery(sql) runs arbitrary SQL against registered tables/views, passes it through Stage 2 governance, executes via Provisa's HTTP API, and deserializes the result into a JDBC ResultSet. (Amended 2026-06-19: transport is Arrow IPC / JSON, not Parquet — Arrow IPC streams a ResultSet far better than a columnar file format, which would buffer the whole result.)

Use case: executeQuery lets JDBC tools run governed SQL against registered tables/views using standard SQL syntax.

Code: provisa-client/

Tests: tests/unit/test_drivers.py

REQ-130 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: MUST · Type: constraint

Full security pipeline (RLS, masking, sampling) applied at query time — not baked into views.

Use case: Full security pipeline at JDBC query time ensures BI tool users receive only data their role permits.

Code: provisa-client/

Tests: tests/e2e/test_query_pipeline.py, tests/unit/test_drivers.py

REQ-131 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Connection string format: jdbc:provisa://host:port. Authentication via standard JDBC username/password properties.

Use case: Standard JDBC connection string format means no custom driver configuration is needed in most BI tools.

Code: provisa-client/

Tests: tests/unit/test_drivers.py

REQ-132 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The driver ships as a single self-contained shaded JAR. (Amended 2026-06-19: the Flight transport (REQ-293) requires gRPC/Netty, so "JDK + Arrow only" is not achievable; instead all bundled third-party packages — Gson, gRPC, Netty, protobuf, guava — are shade-relocated under io.provisa.shaded.* so they never collide with a host application's own copies. The JDBC SPI and Apache Arrow remain at their original packages.)

Use case: Single self-contained JAR eliminates dependency conflicts when deploying the driver in enterprise environments.

Code: provisa-client/

Tests: tests/unit/test_drivers.py

REQ-161 · Query Development Tools¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

POST /data/compile returns compiled SQL with RLS/masking applied, route decision, and params without executing.

Use case: compile endpoint lets developers inspect governed SQL and routing before submitting a query for approval.

Code: provisa/api/rest/, provisa/api/data/

Tests: tests/integration/test_compile_endpoint.py, tests/unit/test_client_access.py, tests/unit/test_compile_route.py

REQ-163 · Query Development Tools¶

Status: ✅ complete · Priority: SHOULD · Type: ui

GraphiQL Provisa plugin with View SQL (compiled governed SQL preview).

Use case: GraphiQL plugin with View SQL integrates governed-SQL inspection into the IDE.

Code: provisa/api/rest/, provisa/api/data/

Tests: tests/integration/test_compile_endpoint.py, tests/unit/test_client_access.py

REQ-256 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-generated plain REST endpoints for every registered table via GET /data/rest/{table} with query string mapping to GraphQL args (?limit=10&where.id.eq=1). Compiles and executes via existing pipeline (RLS, masking, routing) with same security as GraphQL.

Use case: Auto-generated plain REST endpoints let REST-only tools query governed data without learning GraphQL.

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_client_access_integration.py, tests/unit/test_jsonapi.py, tests/unit/test_rest_generator.py

REQ-257 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-generated JSON:API compliant endpoints for every registered table via GET /data/jsonapi/{table} following spec (jsonapi.org): resource objects with type/id/attributes/relationships, sparse fieldsets (?fields[orders]=amount), inclusion of related resources (?include=customer), filtering (?filter[region]=US), sorting (?sort=-created_at), pagination (?page[number]=2&page[size]=25), compound documents. Compiles and executes via existing pipeline.

Use case: JSON:API compliant endpoints let clients use standard JSON:API tooling (sparse fields, includes, pagination).

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/integration/test_jsonapi_integration.py, tests/unit/test_jsonapi.py, tests/unit/test_jsonapi_include.py, tests/unit/test_rest_generator.py

REQ-258 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SSE subscriptions via GET /data/subscribe/{table} with pluggable notification providers per source type. PostgreSQL uses LISTEN/NOTIFY via asyncpg, MongoDB uses Change Streams via motor collection.watch(), Kafka uses consumer groups. Each provider implements a common async watch() interface returning change events. RLS filtering and schema validation apply regardless of provider.

Use case: Pluggable SSE subscription providers let each source type (PG, Mongo, Kafka) use its native change mechanism.

Code: provisa/api/rest/, provisa/api/jsonapi/, provisa/api/flight/

Tests: tests/integration/test_client_access_integration.py, tests/unit/test_jsonapi.py, tests/unit/test_rest_generator.py, tests/unit/test_subscription_providers.py

REQ-268 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Python DB-API 2.0 (PEP 249) interface in provisa-client. provisa_client.connect(url, user, password) returns a PEP 249 connection. cursor.execute(query) accepts GraphQL (detected by leading { or query/mutation keyword) or SQL. Authenticates via username/password; server assigns role.

Use case: DB-API 2.0 interface lets Python code query Provisa with standard cursor.execute() without a custom client.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_client_role.py, tests/unit/test_proto_gen.py, tests/unit/test_protocol_clients.py

REQ-269 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

DB-API 2.0 connection exposes all registered tables and views the user's rights permit for arbitrary SQL. There is no connection mode parameter — governance is uniform via rights + Stage 2.

Use case: Uniform rights-governed SQL access in DB-API removes mode selection; every query is governed identically.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py

REQ-270 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SQLAlchemy dialect for Provisa. create_engine("provisa+http://user:password@host:8001"). Dialect maps SQLAlchemy Core expressions to Provisa SQL. engine.connect() returns a DB-API 2.0 connection. pandas.read_sql(query, engine) works out of the box. inspector.get_table_names() returns registered table and view names the role may access.

Use case: SQLAlchemy dialect lets pandas and ORM users read governed data with standard SQL tooling and no custom code.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py

REQ-271 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

ADBC (Arrow Database Connectivity) interface in provisa-client. provisa_client.adbc_connect(url, user, password) returns an ADBC connection using Provisa's Arrow Flight endpoint as transport. Results stream as Arrow RecordBatches natively. Compatible with adbc_driver_manager and pandas.read_sql via ADBC. Authenticates via username/password.

Use case: ADBC interface streams Arrow RecordBatches natively so analytics tools get zero-copy columnar data.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py

REQ-272 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: MUST · Type: constraint

JDBC driver runs arbitrary SQL against registered tables and views with full Stage 2 governance — RLS, masking, sampling, and ceilings applied on every query. There is no ungoverned access path.

Use case: Uniform Stage 2 governance in JDBC ensures arbitrary SQL from BI tools always has RLS and masking applied.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py

REQ-273 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: MUST · Type: constraint

All clients (DB-API, SQLAlchemy, ADBC, JDBC) authenticate via username/password. Server assigns role from configured auth provider. No client-supplied role parameter accepted or required.

Use case: Server-assigned roles prevent clients from escalating privileges by supplying their own role parameter.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_client_role.py, tests/unit/test_proto_gen.py, tests/unit/test_protocol_clients.py

REQ-274 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Query language selection is per-call for DB-API and GraphQL clients: pass a GraphQL string to execute via Stage 1+2; pass a SQL string to execute via Stage 2 only. ADBC and SQLAlchemy always use SQL (Stage 2 only). JDBC uses SQL always.

Use case: Per-call query language selection lets the same client send GraphQL or SQL based on what it has available.

Code: provisa-client/, provisa/api/

Tests: tests/integration/test_adbc.py, tests/integration/test_sqlalchemy_dialect.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py

REQ-288 · Automatic Persisted Queries (APQ)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa implements the Apollo APQ wire protocol (GraphQL over HTTP with extensions.persistedQuery.sha256Hash). Client sends hash only; if server has it, executes without the query text. If server does not have it, returns PersistedQueryNotFound; client resends with full query text and hash; server stores and executes. Standard Apollo client behaviour works without modification.

Use case: APQ wire protocol support lets Apollo clients reduce bandwidth by sending only a hash after the first request.

Code: provisa/apq/

Tests: tests/integration/test_abac_hook_endpoint.py, tests/integration/test_apq_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_apq.py, tests/unit/test_registry_removed.py

REQ-289 · Automatic Persisted Queries (APQ)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

APQ cache stored in Redis (existing cache.redis_url). TTL configurable via apq.ttl (default: 24h). Cache miss on cold start is expected and handled by the standard APQ retry flow — no preloading required.

Use case: Redis APQ cache with configurable TTL reuses query text across process restarts without preloading.

Code: provisa/apq/

Tests: tests/integration/test_apq_integration.py, tests/integration/test_cache_store.py, tests/unit/test_apq.py, tests/unit/test_cache_store.py, tests/unit/test_config_reload.py

REQ-290 · Automatic Persisted Queries (APQ)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

APQ applies to any query the authenticated caller's rights permit. APQ is fully automatic — any successfully executed query is registered in the APQ cache and reusable by hash on subsequent calls. No steward involvement.

Use case: Automatic APQ registration lets clients benefit from hash-based execution with no config.

Code: provisa/apq/

Tests: tests/unit/test_apq.py, tests/integration/test_apq_integration.py

REQ-291 · Automatic Persisted Queries (APQ)¶

Status: ✅ complete · Priority: MUST · Type: constraint

APQ registration is gated only by the standard rights + Stage 2 governance check. A query referencing tables or columns the caller's rights do not permit is rejected at the governance check — before it can be registered in the APQ cache. Only queries that pass governance can be cached.

Use case: Rights-gated APQ registration prevents hash-only execution of queries the caller is not permitted to run.

Code: provisa/apq/

Tests: tests/unit/test_apq.py, tests/integration/test_apq_integration.py

REQ-293 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

JDBC driver transport via Arrow Flight — connect to Provisa's existing Flight server (grpc://host:8815) for streaming query results. Arrow record batches stream from the first row with backpressure, zero serialization overhead, and no full-result buffering. Flight is used automatically when the server is reachable; falls back to HTTP silently if not. No user configuration required. The Flight ticket carries the SQL or GraphQL query + role + variables as JSON; Stage 2 governance applies uniformly.

Use case: JDBC transport via Arrow Flight streams results with backpressure so BI tools handle large result sets safely.

Code: provisa-client/

Tests: tests/e2e/test_grpc_query.py, tests/unit/test_drivers.py, tests/integration/test_client_access_integration.py

REQ-398 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The /data/graph-schema REST endpoint must expose pk_columns (list of column names per node label) so the UI can determine exclusion eligibility.

Use case: Schema endpoint pk_columns expose lets UI conditionally enable/disable "Exclude from query" per node.

Code: provisa/api/rest/cypher_router.py

Tests: tests/unit/test_graph_schema.py, tests/integration/test_client_access_integration.py

REQ-405 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

graphql_api and graphql_remote source sub-types are collapsed into a single graphql type in the SourcesPage UI; grpc_api and grpc_remote are collapsed into a single grpc type.

Use case: Collapsing API/remote variants eliminates redundant form paths and simplifies steward source registration.

Code: provisa-ui/src/pages/SourcesPage.tsx

Tests: tests/integration/test_client_access_integration.py, tests/integration/test_graphql_remote_source.py, tests/unit/test_admin_requirements.py

REQ-406 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

OpenAPI source registration supports an inline spec editor (monospace textarea) as an alternative to providing a file path or URL; the mode is selected via a radio toggle "Spec path / URL" vs "Write spec inline". Inline content is sent as spec_content; file/URL path is sent as spec_path.

Use case: Inline spec editing lets stewards paste a spec directly into the UI without first writing it to a file.

Code: provisa-ui/src/pages/SourcesPage.tsx, provisa/api/admin/openapi_router.py

Tests: tests/unit/test_admin_requirements.py, tests/integration/test_client_access_integration.py

REQ-407 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

OpenAPI source backend accepts optional spec_content: str on OpenAPIRegisterRequest and OpenAPIPreviewRequest; when provided it is parsed (YAML then JSON fallback) and used in place of loading from disk; path is stored as ":inline:" sentinel.

Use case: Backend inline-spec support decouples OpenAPI registration from file-system access on the server.

Code: provisa/api/admin/openapi_router.py

Tests: tests/unit/test_admin_requirements.py, tests/integration/test_client_access_integration.py

REQ-408 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

OpenAPI operations can carry x-provisa-kind: query or x-provisa-kind: mutation extension to override the GET-heuristic that classifies POST operations as mutations by default. Allows POST-as-read endpoints to be exposed as GraphQL queries.

Use case: x-provisa-kind override lets stewards correctly classify non-standard POST read operations without changing the upstream API.

Code: provisa/openapi/mapper.py

Tests: tests/unit/test_admin_requirements.py, tests/integration/test_client_access_integration.py

REQ-527 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

pgwire server is disabled by default and starts only when PROVISA_PGWIRE_PORT environment variable is set to a non-zero integer. Binds to 0.0.0.0 on the configured port.

Use case: Disabled-by-default pgwire prevents accidental exposure of the PostgreSQL wire protocol endpoint when it is not needed.

Code: provisa/api/app.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-529 · pgwire Server¶

Status: ✅ complete · Priority: MUST · Type: behavioral

pgwire authentication uses PG auth type 3 (cleartext password) for both trust and simple modes. Trust mode: username becomes role_id, password ignored. Simple mode: password verified via bcrypt through SimpleAuthProvider. Any provider value other than none or simple returns a FATAL login error.

Use case: Restricting pgwire auth to trust or simple providers prevents misconfiguration with OIDC or Firebase providers that require HTTP redirect flows incompatible with the wire protocol.

Code: provisa/pgwire/server.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-530 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

pgwire TLS is enabled by setting PROVISA_PGWIRE_CERT and PROVISA_PGWIRE_KEY to PEM certificate and key paths. When both are set the server wraps connections in TLS. When absent the server replies N to SSL negotiation requests.

Use case: Optional TLS for pgwire allows encrypted connections in production without requiring TLS in local development.

Code: provisa/pgwire/server.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-532 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Queries against information_schema and pg_catalog are intercepted and answered from an in-memory DuckDB database built per request from the role's compilation context — no Trino round-trip occurs. Intercepted tables include pg_namespace, pg_class, pg_attribute, pg_type, pg_constraint, pg_roles, pg_settings, pg_stat_activity, schemata, tables, columns, views, table_constraints, key_column_usage, referential_constraints, and others. Stubs return zero rows.

Use case: Local catalog intercept allows BI tools (DBeaver, Tableau, JDBC schema browsers) to populate their schema tree from pg_catalog without generating Trino queries.

Code: provisa/pgwire/catalog.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_catalog.py, tests/unit/pgwire/test_phase3.py, tests/unit/pgwire/test_phase4.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-538 · JDBC/ODBC Integration¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The auto-generated .proto file maps Provisa/Trino column types to protobuf scalar types: integer → int32, bigint → int64, varchar → string, decimal → double, boolean → bool, timestamp → google.protobuf.Timestamp. Each registered table produces one proto message. Relationships between tables produce nested message fields.

Use case: Documented proto type mapping lets gRPC clients generate correct typed stubs without manual type translation.

Code: provisa/grpc/proto_gen.py

Tests: tests/unit/test_grpc_requirements.py

REQ-579 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The pgwire server reports server version 14.0.provisa to connecting clients. Tools that gate features on the PostgreSQL version number behave as though connected to PostgreSQL 14.

Use case: Reporting a stable PostgreSQL version ensures JDBC drivers, DBeaver, and other tools that gate features on version behave predictably.

Code: provisa/pgwire/server.py, provisa/pgwire/catalog.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-580 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

pgwire simple-query protocol supports multi-statement queries. Semicolon-separated statements are split and executed sequentially within a single simple-query message.

Use case: Multi-statement simple queries allow JDBC tools and psql scripts that send batched DDL or DML to work without modification.

Code: provisa/pgwire/server.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-581 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Parameterized queries using $1, $2, ... positional parameters are supported in both simple-query and extended-query (Bind/Execute) protocol modes. Parameters are substituted as SQL literals before execution — the upstream engine never sees a prepared statement.

Use case: Parameterized query support enables standard JDBC/psycopg2 prepared statement patterns. Literal substitution avoids prepared-statement protocol complexity with Trino.

Code: provisa/pgwire/server.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_phase2.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-582 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

pgwire DDL dispatches to one of two paths based on ddl_catalog. Trino path (non-registered catalog such as iceberg, hive, otel, results): supports only CREATE TABLE and CREATE VIEW; ALTER, DROP, and CREATE INDEX are rejected. Direct path (registered source ID): supports full DDL including CREATE, ALTER, DROP, indexes, and sequences.

Use case: Two DDL paths let stewards write tables to Iceberg for lake storage or directly to registered RDBMS sources for full DDL control.

Code: provisa/pgwire/ddl_handler.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-583 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

After DDL execution on either the Trino or direct path, the new table is registered into the role's compilation context so it is immediately queryable without a server restart.

Use case: Immediate post-DDL registration lets analytic workflows create tables and query them in the same session.

Code: provisa/pgwire/ddl_handler.py

Tests: tests/integration/test_direct_exec.py, tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-584 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

DDL write target is resolved from the domain's ddl_catalog and ddl_schema config fields. If ddl_catalog is unset, the system defaults to the Iceberg catalog. If ddl_schema is unset, it defaults to the domain ID. The domain is resolved through the role's domain_access list.

Use case: Domain-driven DDL target resolution ensures DDL writes land in the correct catalog and schema without per-query configuration.

Code: provisa/pgwire/ddl_handler.py, provisa/api/app.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-585 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

pgwire supports COPY ... TO STDOUT (export) and COPY ... FROM STDIN (import). COPY TO STDOUT supports both table reference and arbitrary subquery forms. Supported formats for COPY output: text (tab-delimited, default) and csv. Binary COPY format is not supported.

Use case: COPY protocol support enables bulk data export and import via psql, pgdump-compatible tools, and JDBC copy managers without requiring the HTTP API.

Code: provisa/pgwire/copy_handler.py

Tests: tests/e2e/test_output_formats.py, tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-586 · pgwire Server¶

Status: ✅ complete · Priority: MUST · Type: constraint

COPY ... FROM STDIN is restricted to sources with types postgresql, mysql, sqlite, or mariadb. Attempting COPY FROM against a Trino-only source raises SQLSTATE 42501. If no column list is provided, columns are inferred from the registered schema.

Use case: Restricting COPY FROM to writable source types prevents accidental writes to read-only Iceberg or Trino-federated tables.

Code: provisa/pgwire/copy_handler.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-587 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Transaction control commands (SET, BEGIN, START TRANSACTION, COMMIT, ROLLBACK, SAVEPOINT, RELEASE, DISCARD, RESET, DEALLOCATE) are intercepted and return an empty success response. The server is stateless with respect to transactions — in_transaction() always returns False; there is no transaction isolation or rollback support.

Use case: Accepting but ignoring transaction commands allows JDBC drivers and ORMs that issue implicit BEGIN/COMMIT to function without errors, while making the stateless nature explicit.

Code: provisa/pgwire/server.py, provisa/pgwire/catalog.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_phase2.py, tests/unit/pgwire/test_server.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-588 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The catalog intercept layer handles scalar expression queries: current_user / session_user → authenticated role_id; current_database() → "provisa"; current_schema() → "public"; version() → "PostgreSQL 14.0 on Provisa"; pg_backend_pid() → 0; current_setting(...) → value from a fixed settings table; SHOW <setting> → value from the same settings table.

Use case: Scalar expression intercepts prevent JDBC drivers and ORMs from failing on startup queries that probe server identity and settings.

Code: provisa/pgwire/catalog.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_phase2.py, tests/unit/pgwire/test_phase4.py, tests/unit/pgwire/test_server.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-589 · pgwire Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The extended-query protocol (Bind/Execute) supports binary-encoded parameters. Supported OIDs for binary decode: 16 (bool), 17 (bytea), 20 (int8), 21 (int2), 23 (int4), 25 (text), 700 (float4), 701 (float8), 1043 (varchar), 1082 (date), 1114 (timestamp), 1184 (timestamptz), 1700 (numeric), 2950 (uuid). Unsupported OIDs raise "Unsupported binary parameter type: <oid>". Result columns are also sent in binary when the client requests it.

Use case: Binary parameter encoding support enables psycopg2 and asyncpg to send typed parameters without text serialization, preserving full type fidelity for numeric and temporal values.

Code: vendor/buenavista/buenavista/postgres.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-590 · pgwire Server¶

Status: ✅ complete · Priority: MUST · Type: constraint

pgwire enforces hard-coded timeouts: DDL operations time out after 60 seconds; query operations time out after 120 seconds. These are enforced in the handler threads via future.result(timeout=N).

Use case: Hard-coded timeouts prevent long-running DDL or query operations from blocking pgwire handler threads indefinitely.

Code: provisa/pgwire/ddl_handler.py, provisa/pgwire/server.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-606 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

ProvisaClient accepts a bearer token (token parameter) as an alternative to username/password for authentication. When a token is present, it is sent as Authorization: Bearer <token> on every request. When omitted, the client relies on password-based login or no auth (unsecured deployment).

Use case: Bearer token support lets ProvisaClient integrate with identity systems that issue tokens directly, without requiring a Provisa-local username/password pair.

Code: provisa-client/provisa_client/client.py

Tests: provisa-client/tests/test_client.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py, tests/integration/test_client_access_integration.py

REQ-607 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

ProvisaClient error contract: query() raises httpx.HTTPStatusError on HTTP-level errors (4xx/5xx). query_df() raises RuntimeError when the GraphQL response body contains an errors field, regardless of HTTP status.

Use case: Distinct error types for transport errors vs. GraphQL errors let callers handle network failures and schema/data errors with separate recovery paths.

Code: provisa-client/provisa_client/client.py

Tests: provisa-client/tests/test_client.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py, tests/integration/test_client_access_integration.py

REQ-608 · SQL & Multi-Protocol Client Access¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The ADBC interface (adbc_connect) connects to the Arrow Flight server on port 8815. The Flight port is hardcoded in adbc_connect and is not configurable via any parameter — callers must run the Flight server on port 8815 or reconfigure the server.

Use case: Hardcoded Flight port in adbc_connect simplifies the ADBC connection signature; the server config is the single place to change the port.

Code: provisa-client/provisa_client/adbc.py

Tests: provisa-client/tests/test_adbc.py, tests/unit/test_client_access.py, tests/unit/test_proto_gen.py, tests/integration/test_client_access_integration.py

REQ-614 · pgwire¶

Status: ✅ complete · Priority: MUST · Type: constraint

The pgwire listener accepts only SQL statements. GraphQL and Cypher query strings are not parsed or executed over the pgwire protocol.

Use case: SQL-only restriction on pgwire keeps the protocol surface minimal and avoids ambiguity between query languages on a single connection.

Code: provisa/pgwire/_pipeline.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_cypher_graph_fns.py, tests/unit/test_pgwire_requirements.py

REQ-615 · pgwire¶

Status: ✅ complete · Priority: MUST · Type: constraint

The pgwire listener does not support DML mutations (INSERT, UPDATE, DELETE). SQL statements of these types are not routed to a write path.

Use case: Read-only restriction on pgwire keeps the connection appropriate for analytics tools and prevents accidental data modification via SQL clients.

Code: provisa/pgwire/_pipeline.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-616 · pgwire¶

Status: ✅ complete · Priority: MUST · Type: constraint

COPY and DDL statements submitted over pgwire require the ddl role capability. Roles without ddl receive a permission error (PG error code 42501). Roles with ddl have COPY and DDL executed via the DDL handler and COPY handler respectively.

Use case: Capability-gated DDL and COPY lets power users perform schema operations via psql or DBeaver while protecting other roles from accidental destructive statements.

Code: provisa/pgwire/server.py, provisa/pgwire/ddl_handler.py, provisa/pgwire/copy_handler.py

Tests: tests/integration/test_pgwire_integration.py, tests/unit/pgwire/test_wire_protocol.py, tests/unit/test_pgwire_requirements.py

REQ-617 · gRPC¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Role selection on every Provisa gRPC RPC is via the x-provisa-role metadata key. Missing or unrecognised role metadata causes the call to be rejected with UNAUTHENTICATED. Streaming query RPCs emit one response message per result row; mutation RPCs are unary.

Use case: Metadata-based role selection lets a single gRPC channel serve multiple roles by setting the key per-call without reconnecting.

Code: provisa/grpc/server.py

Tests: tests/e2e/test_grpc_query.py, tests/unit/test_grpc_requirements.py, tests/integration/test_client_access_integration.py

9. Live Data & Events¶

REQ-172 · Dataset Change Events¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Mutations emit a dataset change event to a Kafka topic — no row-level detail, just {table, source, timestamp}.

Use case: Mutation change events let downstream consumers react to data changes without polling the source.

Code: provisa/events/

Tests: tests/unit/test_kafka_change_events.py, tests/integration/test_live_data_integration.py

REQ-173 · Dataset Change Events¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Change events fire on the same mutation hook that invalidates cache and marks MVs stale.

Use case: Change events and cache invalidation fire together, keeping all derived state consistent after mutations.

Code: provisa/events/

Tests: tests/unit/test_kafka_change_events.py, tests/integration/test_live_data_integration.py

REQ-174 · Dataset Change Events¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Producers running complex ETL outside Provisa can signal changes via a trivial mutation (touch operation).

Use case: Touch mutations let external ETL pipelines signal Provisa that source data changed without a direct DB write.

Code: provisa/events/

Tests: tests/unit/test_kafka_change_events.py, tests/integration/test_live_data_integration.py

REQ-175 · Dataset Change Events¶

Status: ✅ complete · Priority: MAY · Type: structural

Change event topic configurable via PROVISA_CHANGE_EVENT_TOPIC (default: provisa.change-events).

Use case: Configurable change event topic lets operators isolate event streams by environment or team.

Code: provisa/events/

Tests: tests/unit/test_kafka_change_events.py, tests/integration/test_live_data_integration.py

REQ-176 · Kafka Sinks (Table/View Publishing)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Registered tables and views can optionally have a Kafka sink — results published to a topic on trigger. Sinks may use change_event, schedule, manual, or poll depending on source capability.

Use case: Table/view Kafka sinks push governed results to topics so consumers never bypass security.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_sink.py, tests/integration/test_kafka_sink.py

REQ-177 · Kafka Sinks (Table/View Publishing)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Sink triggers: change_event (re-run when source table changes), schedule (cron/interval), manual (on-demand), poll (watermark-based continuous polling — required when CDC is unavailable).

Use case: Flexible sink triggers (CDC, schedule, manual, poll) match delivery to what each source physically supports.

Code: provisa/kafka/

Tests: tests/integration/test_kafka_sink.py, tests/unit/test_event_triggers.py, tests/unit/test_kafka_sink.py

REQ-178 · Kafka Sinks (Table/View Publishing)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Sinks are opt-in per registered table or view, configured by the steward.

Use case: Opt-in sink configuration prevents accidental data publishing without steward intent.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_sink.py, tests/integration/test_kafka_sink.py

REQ-180 · Kafka Sinks (Table/View Publishing)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Sinks can be added to or removed from a table/view at any time, independently of other config.

Use case: Independent sink attachment lets operators add or remove delivery channels without touching other config.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_sink.py, tests/integration/test_kafka_sink.py

REQ-181 · Kafka Sinks (Table/View Publishing)¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Sink output format is JSON (one message per row, keyed by optional column).

Use case: JSON-per-row Kafka messages let any downstream consumer deserialize results without custom decoders.

Code: provisa/kafka/

Tests: tests/unit/test_kafka_sink.py, tests/integration/test_kafka_sink.py

REQ-260 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Polling-based subscription provider for sources without native CDC. A watermark_column (monotonic timestamp or sequence, e.g. updated_at) must be declared on the table config. Without a watermark_column, poll subscriptions are unavailable for that source. Part of Phase AM (Live Query Engine). Note: Poll-based subscriptions cannot detect hard deletes — a deleted row leaves no updated watermark value and will not appear in any poll. Applications requiring delete visibility must use soft deletes (e.g. a deleted_at flag) that update the watermark column; the delete is then visible as an update event containing the soft-delete marker.

Use case: Poll-based subscriptions enable live delivery for sources that do not support CDC or LISTEN/NOTIFY.

Code: provisa/subscriptions/

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_sse_subscriptions.py, tests/unit/test_live_engine_full.py, tests/unit/test_subscribe.py

REQ-261 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Debezium CDC subscription provider for non-PG RDBMS sources (MySQL, MariaDB, SQL Server, Oracle). Debezium captures changes from the source's transaction log and publishes to Kafka topics. Provisa's Kafka notification provider consumes these CDC events and streams them as SSE subscriptions. Requires Kafka + Debezium connector infrastructure.

Use case: Debezium CDC subscriptions enable real-time events for MySQL, MariaDB, SQL Server, and Oracle sources.

Code: provisa/subscriptions/

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_debezium_cdc.py, tests/integration/test_sse_subscriptions.py, tests/unit/test_debezium_provider.py, tests/unit/test_subscribe.py

REQ-282 · Live Query Engine (Unified Subscription & Sink Delivery)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A single Live Query Engine powers all poll-based live delivery. It is the common implementation for: (a) SSE subscriptions on tables/views without CDC, (b) Kafka sinks on tables/views without CDC. Delivery mode (CDC vs poll) is orthogonal to the output mechanism (SSE vs Kafka sink).

Use case: Single Live Query Engine powers all poll-based delivery so CDC vs poll and SSE vs sink are independent choices.

Code: provisa/live/, provisa/subscriptions/

Tests: tests/integration/test_live_sse_integration.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-283 · Live Query Engine (Unified Subscription & Sink Delivery)¶

Status: ✅ complete · Priority: MUST · Type: constraint

watermark_column is a required config field for any poll-based live delivery. The column must be monotonically increasing (e.g. updated_at, created_at) and is declared on the table/view config (REQ-260). Without watermark_column, poll delivery is unavailable and config validation fails at startup.

Use case: Required watermark_column config makes poll delivery explicit and prevents unbounded or duplicate event delivery.

Code: provisa/live/, provisa/subscriptions/

Tests: tests/integration/test_live_sse_integration.py, tests/unit/test_config_validation_edge_cases.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-285 · Live Query Engine (Unified Subscription & Sink Delivery)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Tables declare delivery mode in the subscription/sink config: delivery: cdc or delivery: poll. cdc is available for PostgreSQL (LISTEN/NOTIFY), Debezium-connected sources (REQ-261), and MongoDB (Change Streams per REQ-258). All other sources — Trino-federated, JDBC with restricted access, Kafka topics, API sources — must use delivery: poll. Config validation rejects delivery: cdc for sources that do not support it.

Use case: Delivery mode validation at config load time prevents misconfiguring CDC on sources that do not support it.

Code: provisa/live/, provisa/subscriptions/

Tests: tests/integration/test_live_sse_integration.py, tests/unit/test_available_tables.py, tests/unit/test_config_validation_edge_cases.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-286 · Live Query Engine (Unified Subscription & Sink Delivery)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SSE subscription and Kafka sink are equivalent output mechanisms for a table or view. A single live definition may list both sse_subscription and kafka_sink in its outputs. The poll engine executes once per interval; each output tracks its own watermark independently in a live_query_state table (keyed by source name + output_type). A slow Kafka consumer does not block SSE delivery and vice versa.

Use case: Independent watermark tracking per output type ensures a slow Kafka consumer never delays SSE subscribers.

Code: provisa/live/, provisa/subscriptions/

Tests: tests/integration/test_live_sse_integration.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-287 · Live Query Engine (Unified Subscription & Sink Delivery)¶

Status: ✅ complete · Priority: MUST · Type: structural

Live query state persisted in a live_query_state PG table: (source, output_type, last_watermark, last_polled_at, status). On restart, polling resumes from the last committed watermark — no events are replayed or skipped due to process restart. status tracks active, paused, error.

Use case: Persisted live query state lets poll delivery resume from exactly the last committed watermark after a restart.

Code: provisa/live/, provisa/subscriptions/

Tests: tests/integration/test_live_sse_integration.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-565 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

At startup, Provisa idempotently installs AFTER INSERT OR UPDATE OR DELETE triggers on all registered PostgreSQL subscription tables. Each trigger calls pg_notify('provisa_{table}', json_build_object('op', lower(TG_OP), 'row', ...)) so that raw DML from any writer is picked up by SSE subscriptions.

Use case: Auto-installed triggers ensure external writes to PostgreSQL tables are captured by SSE subscriptions without requiring the writer to call pg_notify manually.

Code: provisa/subscriptions/pg_triggers.py

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_sse_subscriptions.py, tests/unit/test_event_triggers.py, tests/unit/test_subscribe.py

REQ-566 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When PostgreSQL trigger installation fails (e.g. insufficient privilege), Provisa logs a warning and falls back to watermark-based polling for that table, provided watermark_column is configured on the table. Tables where installation succeeds are tracked in-memory to select the LISTEN/NOTIFY provider path.

Use case: Graceful fallback to polling prevents a trigger permission failure from making a table entirely unsubscribable.

Code: provisa/subscriptions/pg_triggers.py

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_live_data_integration.py, tests/unit/test_subscribe.py

REQ-567 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When a subscription field selects columns from joined tables via registered relationships, the subscription engine collects all physical table names referenced by the join walk and calls watch_many(all_watch_tables) on the PG notification provider. A change to any joined physical table re-fires the subscription query.

Use case: Watching all involved physical tables ensures that relationship-traversing subscriptions surface changes to nested data without requiring a separate subscription per table.

Code: provisa/api/data/subscription_sse.py

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_sse_subscriptions.py, tests/unit/test_audit_requirements.py, tests/unit/test_live_engine_full.py

REQ-568 · Subscriptions¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Each SSE change event is emitted as a JSON object with the shape {"event": "<op>", "table": "<table>", "row": {...}} where op is one of insert, update, or delete.

Use case: Standardized event shape lets clients deserialize change events without custom parsing per table.

Code: provisa/api/data/subscription_sse.py, provisa/subscriptions/pg_provider.py

Tests: tests/e2e/test_subscriptions.py, tests/integration/test_sse_subscriptions.py, tests/unit/test_subscribe.py

10. UI & Admin Surfaces¶

REQ-058 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Branded, custom React-based UI — rendered surface determined entirely by user's assembled role set.

Use case: Role-driven rendered UI ensures each user sees only the capabilities relevant to their assigned permissions.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-059 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Role composition system: admin assembles capabilities from independently assignable building blocks.

Use case: Composable role building blocks let admins precisely grant capabilities without overprivileging any user.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-060 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Capabilities: Source Registration, Table Registration, Relationship Registration, Security Configuration, Query Development, Admin. Holders of a create capability (e.g. Relationship/View Registration) also execute creation requests queued by users lacking that capability (REQ-063).

Use case: Distinct assignable capabilities let organizations enforce separation of duties across data governance roles.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-061 · UI & Frontend¶

Status: ✅ complete · Priority: MUST · Type: ui

Every destructive or consequential action requires explicit confirmation with consequence summary.

Use case: Explicit confirmation for destructive actions prevents accidental data loss or misconfiguration in the UI.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-062 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Test endpoint execution shows RLS filters applied, columns excluded, schema scope enforced in result metadata.

Use case: Test endpoint metadata display lets developers see exactly which security rules applied to their query result.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-063 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Creation-request queue: when a user lacks authority for a create operation (view, relationship), they submit a request that enters a queue visible to anyone holding the rights to execute it. Designed for steward efficiency; rejection reasons must be specific and actionable.

Use case: Actionable rejection reasons in the creation-request queue help requesters quickly fix and resubmit.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_ui_role_requirements.py

REQ-164 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

GET/PUT /admin/config for config YAML download/upload with backup and reload.

Use case: Config YAML download/upload with backup lets admins version-control platform configuration outside the UI.

Code: provisa/api/admin/

Tests: tests/e2e/test_admin_flow.py, tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py, tests/unit/test_config_reload.py, tests/unit/test_config_secrets.py

REQ-165 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

GET/PUT /admin/settings for runtime platform settings (redirect, sampling, cache).

Use case: Runtime settings API lets operators adjust redirect, sampling, and cache behaviour without a restart.

Code: provisa/api/admin/

Tests: tests/e2e/test_admin_flow.py, tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-166 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Editable relationships page with materialize toggle, delete, and add form.

Use case: Editable relationships page with materialize toggle lets stewards manage cross-source joins interactively.

Code: provisa/api/admin/

Tests: tests/e2e/test_admin_flow.py, tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-167 · Admin & Configuration¶

Status: ✅ complete · Priority: MAY · Type: ui

AI-suggested relationships via LLM discovery integration on relationships page.

Use case: AI-suggested relationships reduce steward effort when registering foreign-key links across complex schemas.

Code: provisa/api/admin/

Tests: tests/e2e/test_admin_flow.py, tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-242 · Commands UI¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Admin UI "Commands" page listing all registered functions and webhooks. Grouped by type (DB Function / Webhook). Shows source, domain, exposed_as (mutation/query), governance level, return table, argument count.

Use case: Commands admin page gives stewards visibility into all registered functions and webhooks in one place.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_commands_requirements.py

REQ-243 · Commands UI¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Add command form with type selector: DB Function (source, schema, function name, exposed_as, returns registered table, arguments, visible_to/writable_by) or Webhook (name, URL, method, timeout_ms, returns registered table or inline type, arguments, visible_to).

Use case: Add command form lets stewards register DB functions and webhooks without editing YAML files directly.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_commands_requirements.py

REQ-244 · Commands UI¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Inline type builder for webhook return types — dynamic rows of field name + GraphQL type. Used when webhook returns a custom shape not backed by a registered table.

Use case: Inline type builder lets stewards define custom webhook return shapes visually without writing JSON Schema.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_commands_requirements.py

REQ-245 · Commands UI¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Test command button — execute function/webhook with sample arguments, display result + governance pipeline applied (which columns masked, RLS filters, role). Same pattern as query test endpoint.

Use case: Test command button lets developers verify function execution and governance pipeline before approving.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/e2e/test_query_pipeline.py, tests/unit/test_commands_requirements.py

REQ-248 · UI & Design Patterns¶

Status: ✅ complete · Priority: SHOULD · Type: structural

GraphQL Voyager integration uses iframe with React 18 CDN standalone bundle (not a native React component fork). The iframe approach avoids MUI v5/React 19 incompatibility and provides reliable isolation. No component fork is planned.

Use case: GraphQL Voyager iframe integration provides schema visualisation without React version compatibility issues.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_commands_requirements.py

REQ-249 · UI & Design Patterns¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Column-level masking configuration stored as inline fields on ColumnConfig in models.py (mask_type, mask_pattern, etc.) — not as a separate MaskingRule Pydantic model or separate database table. Masking rules are loaded from table_columns DB rows at startup in app.py. This co-locates masking config with the column definition it applies to.

Use case: Inline masking config on ColumnConfig keeps column governance co-located, avoiding split config lookups.

Code: provisa-ui/src/

Tests: provisa-ui/e2e/no-domain-mode.spec.ts, provisa-ui/e2e/relationships-header.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py

REQ-395 · UI & Frontend¶

Status: ✅ complete · Priority: MUST · Type: ui

The PK designation must be configurable in the TablesPage UI via checkbox per column, visible in both add-form and edit-form views, and in the read-only view.

Use case: UI-based PK designation lets stewards configure keys without editing YAML files.

Code: provisa-ui/src/pages/TablesPage, provisa-ui/src/components/ColumnForm

Tests: provisa-ui/e2e/tables-register.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py

REQ-396 · UI & Frontend¶

Status: ✅ complete · Priority: MUST · Type: ui

The graph node context menu "Exclude from query" option must be disabled (greyed out) when no primary key is configured for that node label.

Use case: Disabling exclusion without a PK prevents incomplete or ambiguous row-level filtering.

Code: provisa-ui/src/components/GraphContextMenu

Tests: provisa-ui/e2e/graph-query-panel-height.spec.ts, provisa-ui/e2e/graph-show-children.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py

REQ-401 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Foreign key (FK) and alternate key (AK) badges surface as read-only indicators in the column editor UI, showing which columns are linked by relationships.

Use case: FK/AK badges provide visual feedback on relationship column usage without allowing direct editing.

Code: provisa-ui/src/components/ColumnForm, provisa-ui/src/pages/TablesPage

Tests: provisa-ui/e2e/tables-register.spec.ts, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py

REQ-404 · UI & Frontend¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Security page RLS form includes "Apply To" toggle: "Specific Table" or "Entire Domain". Selection determines whether table_id or domain_id is populated; the other is NULL.

Use case: UI toggle lets stewards choose rule scope without understanding the underlying schema design.

Code: provisa-ui/src/pages/SecurityPage

Tests: provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py

REQ-410 · UI & Frontend¶

Status: ✅ complete · Priority: MUST · Type: ui

GraphFrame Cypher WHERE clause generation must use single-quoted string literals (not double-quoted SQL identifiers) for non-numeric PK values to produce valid Cypher syntax.

Use case: Correct single-quote quoting prevents SQL identifier errors when filtering by string PK values in the graph view.

Code: provisa-ui/src/pages/GraphFrame.tsx

Tests: provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_column_governance_requirements.py, tests/unit/test_cypher_graph_fns.py

REQ-533 · Admin & Configuration¶

Status: ✅ complete · Priority: MUST · Type: structural

The admin GraphQL API is a Strawberry-based endpoint mounted at POST /admin/graphql, served by FastAPI via strawberry.fastapi.GraphQLRouter. It is wholly separate from the data GraphQL endpoint at /data/graphql with distinct schemas, independent routers, and independent access control.

Use case: Distinct admin API endpoint prevents governance operations from being confused with data query operations and allows independent auth enforcement.

Code: provisa/api/app.py, provisa/api/admin/schema.py

Tests: tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-620 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The admin GraphQL API is mounted at /admin/graphql on the backend server (default port 8001). It is distinct from the data GraphQL API at /data/graphql and is used for platform administration operations.

Use case: Separate admin GraphQL endpoint allows tooling and scripts to target administrative mutations and queries without mixing them with data queries.

Code: provisa/api/app.py line 3389

Tests: tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-622 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

The admin GraphQL API ships with Strawberry's built-in GraphiQL IDE accessible via GET /admin/graphql in a browser, allowing interactive exploration of the full admin schema.

Use case: Integrated GraphiQL removes the need for external GraphQL tooling when exploring or debugging admin operations.

Code: provisa/api/app.py, provisa/api/admin/schema.py

Tests: tests/integration/test_admin_api.py, tests/unit/test_admin_mv.py, tests/unit/test_admin_requirements.py

REQ-644 · Graph Grouping — View Transform¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Node grouping in the graph UI is a view transform: the underlying node and edge data is never modified; the rendering layer derives a groupKey from any node attribute and applies visual encodings (color, hull ellipse) without altering model state. GroupingState is held in GraphFrame and passed to GraphCanvas and HullOverlay.

Use case: View-transform grouping keeps the data model clean and allows switching between grouping attributes without re-fetching data.

Code: provisa-ui/src/components/graph/GraphFrame.tsx, provisa-ui/src/components/graph/graph-clusters.ts

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py

REQ-645 · Graph Grouping — Attribute Discovery¶

Status: ✅ complete · Priority: SHOULD · Type: ui

After any query result, the UI scans all node properties to build a list of groupable attributes: attributes with more than one distinct value across nodes are included. domain (derived from the node label prefix) is always the first entry. Schema cluster virtuals (schema_L1, schema_L2, schema_L3) are included when their underlying scl1/scl2/scl3 properties have multiple distinct values. The grouping dropdown is populated dynamically from this list — no attribute names are hardcoded.

Use case: Dynamic attribute discovery lets users group by any categorical property in the current result set without prior configuration.

Code: provisa-ui/src/components/graph/GraphFrame.tsx

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py, tests/unit/test_llm_discovery.py

REQ-646 · Graph Grouping — Color Encoding¶

Status: ✅ complete · Priority: SHOULD · Type: ui

When a grouping attribute is active, each node's background-color in the Cytoscape style function is overridden to the color assigned to that node's group value. Color assignment uses clusterColor(groupValue) which derives a stable color from the group key string.

Use case: Color-by-group gives immediate visual differentiation of node clusters without requiring a separate legend interaction.

Code: provisa-ui/src/components/graph/GraphCanvas.tsx, provisa-ui/src/components/graph/graph-clusters.ts

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py

REQ-647 · Graph Grouping — Hull SVG Overlay¶

Status: ✅ complete · Priority: SHOULD · Type: ui

When a grouping attribute is active, an SVG ellipse is drawn over the Cytoscape canvas around each group's node positions. Ellipses are computed by fitting bounding-box centroids and radii from node positions in graph coordinates, then converted to screen coordinates using Cytoscape zoom and pan. The SVG layer re-renders on every Cytoscape viewport event so hulls track pan and zoom.

Use case: SVG hull ellipses provide spatial grouping cues that remain accurate during pan and zoom without requiring re-layout.

Code: provisa-ui/src/components/graph/GraphCanvas.tsx

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py

REQ-648 · Graph Grouping — Collapse to Supernode¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Each group cluster supports a collapse toggle. When collapsed: member nodes are removed from the Cytoscape graph; a synthetic supernode with id __collapsed_<level>_<cidToId(cid)> is added showing the group label and member count; edges crossing the group boundary are rewritten to connect to/from the supernode. Double-clicking the supernode expands the group and restores original nodes and edges. Collapsed state is held in a Set<string> per frame and reset when the grouping attribute changes.

Use case: Collapse-to-supernode reduces visual complexity for large clusters while preserving cross-cluster edge visibility.

Code: provisa-ui/src/components/graph/graph-clusters.ts, provisa-ui/src/components/graph/GraphCanvas.tsx

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py

REQ-649 · Graph Node Size Encoding¶

Status: ✅ complete · Priority: MAY · Type: ui

Node size can be driven by any numeric node property via sizeByProperty (a per-label map of property name). The property value is linearly scaled between the observed min and max across all nodes of that label to produce a node diameter between base and base * multiplier. The sizeByProperty setting is persisted in localStorage under key provisa.graph.sizeByProperty.

Use case: Size-by-property encoding lets analysts visually compare centrality scores or any other numeric metric across nodes without reading raw values.

Code: provisa-ui/src/components/graph/GraphCanvas.tsx, provisa-ui/src/pages/GraphPage.tsx

Tests: provisa-ui/src/__tests__/GraphFrame.test.tsx, provisa-ui/src/__tests__/ui_requirements.test.ts, tests/unit/test_graph_grouping_requirements.py

11. Platform, Infrastructure & Delivery¶

REQ-055 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Docker Compose for development/small-team: single command, Provisa + Trino coordinator + configurable workers, all connectors pre-loaded.

Use case: Single-command Docker Compose setup lets developers run the full Provisa stack without manual configuration.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-056 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Helm chart for production Kubernetes: horizontal Trino worker scaling, resource groups, HPA autoscaling.

Use case: Helm chart with HPA autoscaling lets production deployments scale Trino workers automatically under load.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/e2e/test_helm_minikube.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-057 · Infrastructure¶

Status: ✅ complete · Priority: MUST · Type: constraint

Provisa container is stateless; deployment topology behind Trino endpoint is configuration concern.

Use case: Stateless Provisa container enables horizontal scaling and rolling deployments without session affinity concerns.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-064 · Error Handling & Reliability¶

Status: ✅ complete · Priority: MUST · Type: constraint

Never add fallback values or silent error handling — all errors must be explicit and fail-fast.

Use case: Fail-fast error handling ensures production failures surface immediately instead of silently returning bad data.

Code: —

Tests: tests/unit/test_error_handling.py, tests/unit/test_query_timeout.py

REQ-065 · Error Handling & Reliability¶

Status: ✅ complete · Priority: MUST · Type: constraint

No migrations in version 1 development.

Use case: No-migration policy eliminates schema migration risk and keeps V1 development iteration fast.

Code: —

Tests: tests/unit/test_error_handling.py

REQ-069 · Architecture & Design Patterns¶

Status: ✅ complete · Priority: MUST · Type: constraint

Architecture docs in docs/arch/ ARE the planning documents — update when requirements change, don't implement without planning.

Use case: Architecture docs as planning documents ensure implementation always reflects a reviewed design.

Code: —

Tests: tests/unit/test_process_requirements.py

REQ-070 · Architecture & Design Patterns¶

Status: ✅ complete · Priority: MUST · Type: constraint

Maximum brevity in communications — code and facts only, no pleasantries or explanations unless asked.

Use case: Maximum brevity discipline keeps communications focused on actionable information rather than ceremony.

Code: —

Tests: tests/unit/test_process_requirements.py

REQ-071 · Architecture & Design Patterns¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

New requirements tracked via requirements-tracker agent appending to docs/arch/requirements.md.

Use case: Automated requirements tracking ensures every design decision is recorded without interrupting development flow.

Code: —

Tests: tests/unit/test_infra_requirements.py

REQ-072 · Commercial Positioning¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Core product is open source: Docker Compose, Helm chart, UI, compiler, SQLGlot layer, Trino backend.

Use case: Open-source core makes Provisa accessible to teams that cannot budget for commercial data access platforms.

Code: —

Tests: tests/unit/test_infra_requirements.py

REQ-073 · Commercial Positioning¶

Status: ✅ complete · Priority: MAY · Type: infrastructure

SaaS tier: two isolation lanes. (a) Pooled shared-Trino cluster for free/small orgs ("trino-level isolation"). (b) BYO (bring-your-own) federation engine for enterprise — customer points Provisa at their own Databricks/Snowflake cluster. Both lanes route through the same _govern_and_route/_execute_plan pipeline (REQ-1244). Org-scoped sessions are pinned to one engine per deployment (subdomain → org → engine → credentials).

Use case: Two-lane model lets Provisa serve both self-service free/small orgs on shared Trino and enterprise customers on their own federation engines without code branches or dual pipelines.

Code: —

Tests: tests/unit/test_control_plane.py, tests/unit/test_process_requirements.py, tests/integration/test_multi_org_routing.py

REQ-074 · Commercial Positioning¶

Status: ✅ complete · Priority: MAY · Type: infrastructure

Enterprise tier: SLA guarantees, dedicated support, advanced audit logging, compliance reporting.

Use case: Enterprise tier SLA and audit logging meets regulated-industry compliance and procurement requirements.

Code: —

Tests: tests/unit/test_process_requirements.py, tests/unit/test_sla_monitor.py

REQ-169 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Trino 480 with Iceberg results catalog (JDBC on PG, native S3 filesystem).

Use case: Trino 480 with Iceberg results catalog provides modern open-table-format storage for all redirected results.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-170 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

start-ui.sh --reset-volumes for Docker crash recovery.

Use case: reset-volumes flag gives developers a one-command recovery path after Docker volume corruption.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-171 · Infrastructure¶

Status: ✅ complete · Priority: MUST · Type: behavioral

MinIO results bucket auto-created at startup.

Use case: MinIO bucket auto-creation at startup prevents first-run failures due to missing redirect storage.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-223 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Single-executable installer that bundles the Provisa platform into one download. Underlying technology (Python server, PostgreSQL admin DB, Trino query engine, React UI) is hidden from the user. Source datasets are NOT bundled -- they connect over the wire.

Use case: Single-executable installer hides platform complexity so analysts can run Provisa without DevOps knowledge.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py, tests/unit/test_live_engine_full.py

REQ-224 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Installer expands into a hidden directory (~/.provisa/) containing all services. User interacts via provisa start, provisa stop, provisa status, provisa open (opens browser).

Use case: CLI commands (start/stop/status/open) give non-technical users a simple interface to manage the platform.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-225 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Default deployment uses embedded PostgreSQL (postgres:16) for admin DB and bundled Trino for query federation. Vertical scaling by default -- single machine, increase resources as needed.

Use case: Embedded PostgreSQL and Trino let a single machine run the full platform out of the box.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-226 · Installer & Packaging¶

Status: ✅ complete · Priority: MAY · Type: infrastructure

Users can later connect their own Trino cluster, Spark, external auth provider, or external PostgreSQL for the admin DB via config. Pointing to an external Trino instance is the primary scale-out mechanism.

Use case: External Trino and auth provider support lets organisations scale Provisa onto existing enterprise infrastructure.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-227 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

AF2 is delivered in OS phases. AF2a (immediate): macOS arm64/x86 signed + notarized .dmg. AF2b (follow-on): Linux .AppImage. AF2c (follow-on): Windows signed .exe. Each phase bundles the container runtime for its platform — no Docker, OrbStack, or Colima prerequisite visible to the user.

Use case: OS-native signed bundles (DMG/AppImage/EXE) let users install Provisa like any desktop app without Docker.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-228 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Phase 1 (AF1, complete): Shell script installer with Docker Compose, provisa CLI wrapper, state in ~/.provisa/. Requires Docker/OrbStack/Colima. Phase 2 (AF2): Airgapped native app bundle — Lima + containerd embedded, all service images bundled as .tar archives, zero outbound network at install or runtime. AF3 (native OS packages via Homebrew/.deb/.rpm) is superseded by AF2.

Use case: AF2 airgapped bundle removes the Docker prerequisite so enterprise users with no internet access can install.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-254 · Testing & Quality¶

Status: ✅ complete · Priority: MUST · Type: constraint

Integration tests must use Docker — spin up required containers (PG, Trino, Redis, etc.), install dependencies, run tests, and tear down containers. Tests must not assume a pre-existing Docker Compose stack.

Use case: Integration tests that spin up Docker containers ensure tests validate against real service behaviour.

Code: —

Tests: tests/unit/test_infra_requirements.py

REQ-255 · Testing & Quality¶

Status: ✅ complete · Priority: MUST · Type: constraint

Unit tests must mock all external components (databases, Trino, Redis, HTTP endpoints, file system where applicable). No unit test should require a running external service.

Use case: Unit tests that mock all external components run fast and independently of any running infrastructure.

Code: —

Tests: tests/unit/test_infra_requirements.py

REQ-294 · Installer & Packaging¶

Status: ✅ complete · Priority: MUST · Type: constraint

The distribution must be fully airgap-capable. No outbound network calls at install time or first launch. All container images are bundled inside the app and loaded via ctr images import. Image references in docker-compose use digests, not tags. Suitable for enterprise environments with no internet access.

Use case: Fully airgapped bundle lets enterprises in no-internet environments install and run Provisa from local media.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-302 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

OpenTelemetry (OTel) instrumentation across all Provisa components — traces, metrics, and logs exported via OTLP. Covers: FastAPI app, GraphQL compiler, SQL executor, Trino driver, admin GraphQL API, Arrow Flight server, cache layer, MV subsystem, discovery pipeline, and all source adapters. Structured log emission with trace_id/span_id correlation.

Use case: OTel instrumentation across all components gives operators unified traces, metrics, and logs in one backend.

Code: provisa/otel_compat.py, provisa/api/app.py

Tests: tests/unit/test_executor_trino.py, tests/unit/test_flight_modes.py, tests/unit/test_otel_requirements.py, tests/unit/test_otel_trace_pipeline.py

REQ-303 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

OTel integration via opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-httpx, and OTLP exporter. Auto-instrumentation applies to FastAPI and httpx. Manual span/metric instrumentation at: compile_query, execute_trino, execute_direct, handle_api_query, cache check/store, MV rewrite, RLS inject, masking inject. Export endpoint configured via environment variable (default: http://localhost:4317).

Use case: Auto-instrumentation of FastAPI and httpx combined with manual spans gives full request visibility end-to-end.

Code: provisa/otel_compat.py, provisa/api/app.py

Tests: tests/integration/test_cache_store.py, tests/integration/test_direct_exec.py, tests/unit/test_cache_store.py, tests/unit/test_mask_inject.py, tests/unit/test_otel_requirements.py

REQ-330 · Infrastructure & Observability¶

Status: ✅ complete · Priority: MAY · Type: infrastructure

Development observability stack available in docker-compose under observability profile (opt-in). OTel Collector receives OTLP on 4317/4318 and exports metrics to Prometheus and traces to Tempo. Grafana on port 3100 with Prometheus and Tempo datasources pre-provisioned and a Provisa dashboard included. Provisa app configured via OTEL_EXPORTER_OTLP_ENDPOINT env var (default: http://localhost:4317).

Use case: Opt-in observability stack with OTel, Prometheus, Tempo, and Grafana lets developers debug performance locally.

Code: helm/, docker-compose.yml

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-528 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The path to the Provisa config file is controlled by the PROVISA_CONFIG environment variable (default: config/provisa.yaml).

Use case: Configurable config path lets operators run multiple Provisa instances with different configs on the same host.

Code: provisa/api/app.py, provisa/api/admin/_config_io.py

Tests: tests/integration/test_infra.py, tests/unit/test_admin_requirements.py

REQ-539 · Infrastructure¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The GET /health (or HEAD /health) and GET /setup/status endpoints are always unauthenticated — they bypass the Authorization: Bearer requirement even when an auth provider is configured. All other endpoints require authentication when auth.provider is set.

Use case: Unauthenticated health and setup-status endpoints let load balancers, orchestrators, and first-run UIs check server availability without credentials.

Code: provisa/api/app.py

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-545 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Provisa runs two independent OTLP export paths: the operator's internal collector and an optional Provisa support endpoint. Each path has its own telemetry filter configuration and applies independently.

Use case: Independent export paths let operators share diagnostic telemetry with Provisa support while maintaining separate redaction policies for their internal collector.

Code: provisa/api/otel_setup.py

Tests: tests/unit/test_otel_requirements.py

REQ-546 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: MUST · Type: constraint

Telemetry filters run inside a wrapping _FilteringExporter before spans leave the process. Original span objects are never mutated — filtering operates on copies.

Use case: Non-mutating span filtering ensures telemetry redaction cannot corrupt spans in the internal pipeline.

Code: provisa/api/otel_setup.py

Tests: tests/unit/test_otel_requirements.py

REQ-547 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: MUST · Type: constraint

SQL literal redaction defaults to true on the support telemetry path (support_telemetry_filter.redact_sql_literals), ensuring query data is not shared with Provisa support by default.

Use case: Default-on SQL redaction on the support path protects sensitive query data without requiring operator configuration.

Code: provisa/api/otel_setup.py

Tests: tests/unit/test_otel_requirements.py, tests/integration/test_registration_governance_integration.py

REQ-548 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: MUST · Type: constraint

The Provisa support OTLP endpoint (support_endpoint config or PROVISA_SUPPORT_OTLP_ENDPOINT env var) is disabled by default. When unset, no telemetry data leaves the operator's infrastructure via this path.

Use case: Opt-in support endpoint ensures no data reaches Provisa infrastructure without explicit operator configuration.

Code: provisa/api/otel_setup.py

Tests: tests/unit/test_otel_requirements.py

REQ-549 · OpenTelemetry Instrumentation¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa auto-detects OTLP transport from the endpoint URL scheme: http:// or https:// schemes use OTLP/HTTP with /v1/traces, /v1/metrics, and /v1/logs path suffixes appended automatically; any other scheme uses OTLP/gRPC with insecure mode.

Use case: URL-scheme-based transport detection lets operators configure the OTLP endpoint without a separate transport config field.

Code: provisa/api/otel_setup.py

Tests: tests/unit/test_otel_requirements.py, tests/integration/test_registration_governance_integration.py

REQ-558 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

The development backend API (uvicorn main:app) listens on port 8001 when launched via start-ui.sh or manually. The AppImage and installer paths use port 8000 as the default.

Use case: Consistent port convention between dev and production paths prevents connection errors when switching environments.

Code: start-ui.sh, provisa/api/app.py

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-559 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

The start-ui.sh script starts the Vite UI dev server on port 3000. The production installer serves the UI via provisa/ui_server.py on port 3000.

Use case: Fixed port 3000 for the UI dev server ensures developers can rely on a consistent URL during local development.

Code: start-ui.sh, provisa/ui_server.py

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-560 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The Provisa API default port is 8000 in AppImage/installer/Kubernetes deployments (configurable via --api-port wizard prompt or api_port in config). The Arrow Flight server listens on port 8815 by default.

Use case: Documented default ports let operators configure firewalls and load balancers without reading source code.

Code: provisa/core/models.py, provisa/api/app.py

Tests: tests/e2e/test_health_checks.py, tests/unit/test_core_registration.py

REQ-561 · Installer & Packaging¶

Status: ✅ complete · Priority: MUST · Type: constraint

In a multi-node AppImage deployment, secondary nodes run only the Provisa API and a federation engine worker. Singleton stateful services (PostgreSQL, PgBouncer, MinIO, Redis) run exclusively on the primary node.

Use case: Clear primary/secondary service split prevents operators from accidentally running duplicate stateful services that would cause data inconsistency.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py, tests/unit/test_live_engine_full.py

REQ-562 · Installer & Packaging¶

Status: ✅ complete · Priority: MUST · Type: behavioral

In a multi-node deployment, secondary Provisa API instances are stateless and read all configuration (sources, tables, relationships, roles, RLS rules) from the primary node's PostgreSQL database at startup. No manual config sync between nodes is required.

Use case: Stateless secondary nodes reading from a single PostgreSQL source ensures all API nodes serve identical, authoritative configuration without synchronization overhead.

Code: provisa/api/app.py, provisa/core/catalog.py

Tests: tests/e2e/test_installer.py, tests/unit/test_core_registration.py

REQ-563 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The AppImage first-launch in --non-interactive mode installs a systemd unit (/etc/systemd/system/provisa.service) so Provisa starts automatically on boot. The AppImage generates credentials during first-launch and writes them to ~/.provisa/config.yaml.

Use case: Systemd integration and auto-generated credentials enable unattended cloud-init and Terraform provisioning without manual post-install steps.

Code: packaging/

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-564 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

The Terraform AWS deployment path provisions a full multi-node Provisa cluster including VPC with two public subnets across two AZs, EC2 instances, an ALB on port 8000 for the HTTP API, and an NLB on port 8815 for Arrow Flight/gRPC. All nodes are attached to both load balancers.

Use case: Declarative Terraform provisioning lets operators stand up a production-grade multi-node cluster reproducibly without manual AWS console steps.

Code: terraform/aws/

Tests: tests/e2e/test_installer.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-592 · Multi-Tenancy¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Each tenant maps to an org. The orgs table stores org namespaces. The root org is seeded automatically for single-tenant deployments. In multi-tenant mode, one org is created per customer via the admin API. user_org_memberships tracks which users belong to which org. The current interim model adds org_id FK columns to domains and roles for row-level scoping; these columns are superseded by REQ-695 schema-per-org, after which org membership is implicit in schema placement and the org_id columns on domains/roles can be dropped.

Use case: Org registry and user membership model — the provisioning anchor consumed by REQ-695 (schema creation) and REQ-699 (PG role creation). The orgs table and user_org_memberships remain valid under the schema-per-org model; the org_id FK columns on domains/roles do not.

Code: provisa/core/schema.sql, provisa/api/admin/orgs_router.py

Tests: tests/unit/test_core_registration.py

REQ-593 · Multi-Tenancy¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The KMS client reads AWS_KMS_REGION from the environment and defaults to us-east-1. Standard AWS credential chain (environment variables, instance profile, ECS task role) applies.

Use case: Region configuration via environment variable lets operators deploy the KMS client to any AWS region without code changes.

Code: provisa/api/billing/kms.py

Tests: tests/unit/test_tenancy_requirements.py

REQ-618 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Backend API server runs on port 8001 with uvicorn hot-reload enabled for changes to provisa/ and config/ directories. Log output is written to .logs/server.log. The server starts as part of start-ui.sh.

Use case: Hot-reload on source and config changes lets developers iterate without restarting the server manually.

Code: start-ui.sh line 243

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_config_reload.py, tests/unit/test_infra_requirements.py

REQ-619 · Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

start-ui.sh manages the full dev lifecycle: Ctrl+C stops the backend, UI dev server, and all Docker Compose services and reverts any Trino jvm.config patches. --keep-docker flag leaves Docker services running on Ctrl+C. Ctrl+R (SIGUSR1) restarts only the backend without affecting Docker services or the UI.

Use case: Lifecycle controls (stop-all, keep-docker, backend-only restart) give developers precise control over which components restart during development.

Code: start-ui.sh lines 248–258, start-ui.sh lines 301–306, start-ui.sh lines 340–347

Tests: tests/e2e/test_health_checks.py, tests/integration/test_infra.py, tests/unit/test_infra_requirements.py

REQ-630 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Provisa ships as three distinct artifact packages per platform (Core, Obs, Demo) to stay within the GitHub Actions 2 GB artifact limit. CI builds all three in parallel, each uploaded as a separate artifact. The package split maps directly to the three docker-compose layers.

Use case: Three-package split keeps each CI artifact under the 2 GB GitHub Actions limit while preserving logical separation between core, observability, and demo service sets.

Code: packaging/macos/build-dmg.sh, packaging/macos/build-dmg-obs.sh, packaging/macos/build-dmg-demo.sh, packaging/windows/build-sfx.ps1, packaging/windows/build-installer-obs.ps1, packaging/windows/build-installer-demo.ps1, packaging/linux/build-appimage.sh

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-631 · Installer & Packaging¶

Status: ✅ complete · Priority: MUST · Type: constraint

Package dependency chain: Core must be installed before Obs; Obs must be installed before Demo. The Demo installer checks for the presence of ~/.provisa/extensions/observability/ before proceeding. Dependency is enforced at install time by the installer scripts, not enforced by CI job ordering.

Use case: Explicit install-time dependency checks prevent Demo from being installed without Obs, which would leave demo services unable to emit observability data.

Code: packaging/macos/build-dmg-demo.sh, packaging/windows/build-installer-demo.ps1

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-632 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Linux AppImage bundles both core and observability (obs) images directly — there is no separate Obs download for Linux. The Demo package is not included in Linux distribution. The Linux first-launch.sh always starts core + obs together; no extension mechanism is used.

Use case: Bundling obs directly on Linux serves the server/technical user base where observability is production-useful, and eliminates the extension download step on a platform where a single monolithic AppImage is simpler to distribute.

Code: packaging/linux/build-appimage.sh, packaging/linux/first-launch.sh

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-633 · Installer & Packaging¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

macOS and Windows use an extension model: the container/compute tier is the only tier that requires an external container runtime. On macOS, it uses the user's Docker (Docker Desktop or colima); on Windows, it provisions WSL2 + containerd — never VirtualBox. App images (provisa, provisa-ui, zaychik) are built at DMG-build time, shipped in core-images.tar, docker load-ed at first launch, and resolved via docker-compose.airgap.yml. Obs and Demo are extension packages that load images into the existing runtime and write a compose file into ~/.provisa/extensions/<name>/. The launcher enumerates extensions/*/docker-compose.*.yml at startup and appends each found file to the compose file list, composing the service set dynamically. PROVISA_REDIRECT_ENABLED, MinIO, and OTel env vars are set only when the obs extension is present. The native base tier (REQ-979) has no runtime VM at all.

Use case: Extension model lets users install observability and demo services incrementally without reinstalling Core, and lets the launcher automatically discover installed extensions without hard-coded feature flags.

Code: packaging/macos/build-dmg-obs.sh, packaging/macos/build-dmg-demo.sh, packaging/macos/ProvisaLauncher/Sources/ProvisaLauncher/Services/ScriptRunner.swift, packaging/windows/build-container.ps1, packaging/windows/install-container.ps1, packaging/windows/provisa-container.ps1, packaging/windows/wsl/provision-containerd.sh, packaging/windows/wsl/start-containerd.sh, packaging/windows/build-installer-obs.ps1

Tests: tests/e2e/test_installer.py, tests/unit/test_infra_requirements.py

REQ-634 · Installer & Packaging¶

Status: ✅ complete · Priority: MUST · Type: constraint

The dev environment runs the Python backend (uvicorn, port 8000) and UI (vite dev server, port 3000) on the host, never in containers. docker-compose.app.yml and docker-compose.airgap.yml are packaged-product only and must never be included in dev compose stacks. start-ui-install.sh --demo is the only dev mode flag; --demo always implies obs. Ports 8000 and 3000 must never appear in any dev compose file.

Use case: Host-based uvicorn + vite in dev avoids port conflicts with containerised services and enables hot-reload without rebuilding images.

Code: start-ui-install.sh, docker-compose.dev-install.yml, docker-compose.observability.yml, docker-compose.demo.yml

Tests: tests/e2e/test_health_checks.py, tests/unit/test_infra_requirements.py

12. Migration & Compatibility (Hasura)¶

REQ-182 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Hasura v2 metadata converter -- CLI tool that reads a Hasura v2 metadata export directory and emits valid Provisa YAML config. Converts tracked tables, relationships, permissions, roles, and auth.

Use case: Hasura v2 parity — automates migration from Hasura v2 so teams avoid manual YAML rewriting.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-183 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Hasura DDN (v3) HML converter -- CLI tool that reads a DDN supergraph project and emits valid Provisa YAML config. Converts ObjectTypes, Models, Relationships, TypePermissions, ModelPermissions, and DataConnectorLinks.

Use case: Hasura v2 parity — automates migration from Hasura DDN so teams avoid rewriting supergraph configs.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-184 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Shared boolean expression-to-SQL converter for Hasura filter expressions. Supports _eq, _neq, _gt, _gte, _lt, _lte, _in, _nin, _like, _ilike, _regex, _is_null, _and, _or, _not. Session variable mapping: X-Hasura-<Name> -> current_setting('provisa.<name>').

Use case: Hasura v2 parity — shared filter converter ensures RLS rules from Hasura boolean expressions work correctly.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-185 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter maps select_permissions[].columns per role -> Provisa column visible_to. columns: "*" means all columns visible to that role.

Use case: Hasura v2 parity — column visibility from v2 select permissions is preserved automatically on import.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-186 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter maps insert/update_permissions[].columns per role -> Provisa column writable_by.

Use case: Hasura v2 parity — column write permissions from v2 are preserved automatically on import.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-187 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter maps select_permissions[].filter -> Provisa rls_rules[] via boolean expression-to-SQL conversion. filter: {} (empty) means no RLS filter.

Use case: Hasura v2 parity — RLS filters from v2 select permissions are preserved automatically on import.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-188 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter maps object_relationships -> cardinality=many-to-one and array_relationships -> cardinality=one-to-many. Physical column names used directly (no GraphQL resolution needed).

Use case: Hasura v2 parity — relationship cardinality from v2 metadata is preserved automatically on import.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-189 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

DDN converter resolves GraphQL field names to physical column names through ObjectType.dataConnectorTypeMapping[].fieldMapping for all field references in relationships, permissions, and column definitions.

Use case: Hasura v2 parity — DDN field mappings are resolved to physical columns so no manual remapping is needed.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-190 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 auth conversion via optional --auth-env-file flag. JWT with jwk_url -> Provisa provider: oauth. JWT claims_map -> Provisa role_mapping[]. Admin secret -> Provisa superuser. Webhook auth emits warning (no Provisa equivalent).

Use case: Hasura v2 parity — auth config is migrated automatically so JWT and role mappings carry over.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-191 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

DDN AggregateExpression metadata preserved in sidecar provisa-aggregates.yaml and converted to Provisa aggregate config.

Use case: Hasura v2 parity — aggregate expressions from DDN are preserved so analytics queries keep working.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-192 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Converters emit warnings for unmappable features (event_triggers, remote_schemas, cron_triggers, BooleanExpressionType) without failing conversion. v2 Actions and DDN Commands convert to Provisa functions config where backed by stored procedures; webhook-backed actions emit warning with handler URL. Part of Phases AG/AH (Hasura Converters).

Use case: Hasura v2 parity — unmappable features emit warnings instead of failing so partial migrations complete.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_event_triggers.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-193 · Hasura Migration Converters¶

Status: ✅ complete · Priority: MUST · Type: constraint

Both converters produce output that passes ProvisaConfig.model_validate() -- Pydantic-valid config or nothing. Part of Phases AG/AH (Hasura Converters).

Use case: Hasura v2 parity — converter output is always Pydantic-valid so imported configs never cause silent runtime errors.

Code: provisa/hasura_v2/, provisa/ddn/

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py

REQ-212 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Upsert mutations -- INSERT ... ON CONFLICT ... DO UPDATE. New upsert_<table> mutation field. Conflict columns inferred from primary key metadata. SQLGlot transpiles to dialect-specific syntax (MySQL ON DUPLICATE KEY UPDATE, etc.).

Use case: Hasura v2 parity — upsert mutations remove the need for client-side read-then-write logic.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/unit/test_upsert_distinct_on.py, tests/unit/test_column_presets.py, tests/integration/test_mutation_parity_integration.py

REQ-213 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

DISTINCT ON query argument -- deduplicate results by specified columns. Added as distinct_on arg on root query fields. SQLGlot handles dialect differences (window function fallback for non-PG).

Use case: Hasura v2 parity — DISTINCT ON deduplication lets clients request unique result sets in a single query.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/unit/test_upsert_distinct_on.py, tests/unit/test_column_presets.py, tests/integration/test_mutation_parity_integration.py

REQ-214 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Column presets -- auto-set column values on insert/update from session variables or built-in functions. Config per table: column_presets: [{column: created_by, source: header, name: x_user_id}, {column: updated_at, source: now}]. Preset columns removed from user input, injected before SQL generation.

Use case: Hasura v2 parity — column presets auto-stamp audit fields like created_by without trusting client input.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/unit/test_upsert_distinct_on.py, tests/unit/test_column_presets.py, tests/integration/test_mutation_parity_integration.py

REQ-215 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Inherited roles -- role hierarchy where child role inherits capabilities and domain_access from parent. Config: parent_role_id on role definition. Flattened at startup (merge capabilities/domain_access up the chain). Lookups remain O(1).

Use case: Hasura v2 parity — inherited roles eliminate redundant permission duplication across similar role definitions.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/integration/test_mutation_parity_integration.py, tests/unit/test_column_presets.py, tests/unit/test_inherited_roles_extended.py, tests/unit/test_upsert_distinct_on.py

REQ-216 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Scheduled triggers -- time-based execution of registered webhooks or internal functions. APScheduler in-process, cron expression syntax. Config per trigger in provisa.yaml. Reuses existing async background task pattern.

Use case: Hasura v2 parity — scheduled triggers run webhooks or functions on a cron schedule without external cron infra.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/integration/test_mutation_parity_integration.py, tests/unit/test_column_presets.py, tests/unit/test_scheduled_triggers.py, tests/unit/test_scheduler.py, tests/unit/test_upsert_distinct_on.py

REQ-217 · Hasura v2 Parity: Low-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Batch mutations already supported by GraphQL spec -- multiple mutations in one request execute sequentially. Document existing behavior.

Use case: Hasura v2 parity — documenting existing batch mutation support clarifies the capability for developers.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/integration/test_mutation_parity_integration.py, tests/unit/test_batch_mutations.py, tests/unit/test_column_presets.py, tests/unit/test_upsert_distinct_on.py

REQ-218 · Hasura v2 Parity: Medium-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cursor-based pagination -- first, after, last, before args on root query fields. Returns edges[{cursor, node}] + pageInfo{hasNextPage, hasPreviousPage, startCursor, endCursor}. Cursor encoded as base64 of sort key values. Coexists with existing offset/limit.

Use case: Hasura v2 parity — cursor-based pagination lets clients page large result sets stably without offset drift.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/integration/test_mutation_parity_integration.py, tests/unit/test_column_presets.py, tests/unit/test_cursor_pagination.py, tests/unit/test_upsert_distinct_on.py

REQ-219 · Hasura v2 Parity: Medium-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Subscriptions via Server-Sent Events (SSE) -- GET /data/subscribe/<table> endpoint using FastAPI StreamingResponse. PostgreSQL LISTEN/NOTIFY via asyncpg .add_listener() for change detection. No WebSocket complexity. Streams INSERT/UPDATE/DELETE events.

Use case: Hasura v2 parity — SSE subscriptions push live DB changes to clients without WebSocket complexity.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/unit/test_upsert_distinct_on.py, tests/unit/test_column_presets.py, tests/integration/test_mutation_parity_integration.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-220 · Hasura v2 Parity: Medium-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Database event triggers -- table changes (insert/update/delete) fire webhooks. PostgreSQL trigger + pg_notify() -> asyncpg listener -> HTTP POST to configured URL. Config per table with operation filter and retry policy.

Use case: Hasura v2 parity — DB event triggers automate webhook calls on row changes without application-layer polling.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/integration/test_event_triggers_integration.py, tests/integration/test_mutation_parity_integration.py, tests/unit/test_column_presets.py, tests/unit/test_event_triggers.py, tests/unit/test_upsert_distinct_on.py

REQ-221 · Hasura v2 Parity: Medium-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Enum table auto-detection -- introspect pg_enum at schema build time, generate GraphQL enum types for columns using PostgreSQL user-defined enums. Map enum columns to GraphQL enum type instead of String.

Use case: Hasura v2 parity — enum type auto-detection exposes PostgreSQL enums as GraphQL enums with no manual config.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/unit/test_column_presets.py, tests/unit/test_enum_detection.py, tests/unit/test_upsert_distinct_on.py, tests/integration/test_registration_governance_integration.py

REQ-222 · Hasura v2 Parity: Medium-Complexity Features¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

REST endpoint auto-generation -- for each root query field, generate GET /data/rest/<table> FastAPI endpoint. Map query args to URL query params (?limit=10&where.id.eq=1). Reuses GraphQL compilation pipeline internally.

Use case: Hasura v2 parity — auto-generated REST endpoints let REST-only clients consume governed data without GraphQL.

Code: provisa/compiler/, provisa/scheduler/

Tests: tests/e2e/test_query_pipeline.py, tests/integration/test_rest_endpoints.py, tests/unit/test_column_presets.py, tests/unit/test_upsert_distinct_on.py, provisa-ui/e2e/infrastructure.spec.ts

REQ-621 · Hasura Migration Converters¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Both Hasura v2 and DDN converters emit placeholder connection credentials in the output config (host: localhost, password: ${env:DB_PASSWORD}). Operators must update connection details before starting Provisa.

Use case: Placeholder credentials ensure the generated config is syntactically valid while signaling that real credentials must be supplied before production use.

Code: provisa/hasura_v2/mapper.py, provisa/ddn/mapper.py

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-623 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter maps Hasura source kind to Provisa SourceType: pg/postgres -> postgresql, mssql -> sqlserver, bigquery -> bigquery, citus -> postgresql, mysql -> mysql. Connection URL (database_url) is parsed into host/port/database/username/ password components. Pool settings (pool_settings.min_connections, pool_settings.max_connections) are preserved as pool_min/pool_max.

Use case: Hasura v2 parity — source kind and connection URL normalization ensures source configs carry over without manual rewriting.

Code: provisa/hasura_v2/mapper.py

Tests: tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-624 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter upgrades a role to write capability when that role has any delete_permissions entry on any table. No per-table delete mapping is produced — the capability upgrade is the only artefact.

Use case: Hasura v2 parity — delete permission grants are reflected as write capability in Provisa role definitions.

Code: provisa/hasura_v2/mapper.py

Tests: tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-625 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When a Hasura v2 source database_url is an environment variable reference ({"from_env": "VAR"}) or cannot be parsed, the converter substitutes placeholder values: host: localhost, port: 5432, database: default, username: postgres, password: ${env:DB_PASSWORD}. Operators must fix these via --source-overrides before the config is used.

Use case: Explicit placeholder values on parse failure make unparseable connection URLs visible rather than producing silent misconfiguration.

Code: provisa/hasura_v2/mapper.py

Tests: tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-626 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Roles are collected exclusively from permission entries (select, insert, update, delete permissions on tables, action permissions, and inherited role definitions). A Hasura role that exists without any permission entry on any table or action does not appear in the converter output.

Use case: Permission-driven role collection ensures only roles that carry actual access rules are emitted, keeping the output minimal.

Code: provisa/hasura_v2/mapper.py

Tests: tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/unit/test_inherited_roles_extended.py, tests/steps/steps_hasura_migration_converters.py

REQ-627 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

v2 converter derives the Provisa table alias from custom_root_fields.select (first priority), then custom_root_fields.select_by_pk (second priority), then custom_name (third priority). All other custom root fields (select_aggregate, insert, update, delete) are not mapped to any Provisa equivalent.

Use case: Hasura v2 parity — table alias is derived from the select root field, the most common customization point, without requiring operators to manually specify aliases.

Code: provisa/hasura_v2/mapper.py

Tests: tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/unit/test_hasura_v2.py, tests/steps/steps_hasura_migration_converters.py

REQ-628 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When converting DDN HML, if a Model references an ObjectType name that is not found in any scanned .hml file, that table is skipped and a warning is emitted to the WarningCollector; the conversion continues rather than aborting.

Use case: Partial DDN projects (where some HML files are missing) produce a best-effort output with actionable warnings rather than a hard failure.

Code: provisa/ddn/mapper.py

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py, tests/steps/steps_hasura_migration_converters.py

REQ-629 · Hasura Migration Converters¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Subgraph assignment for DDN HML documents is derived from the directory structure, not from a subgraph field in the document. The first directory component under the project root is taken as the subgraph name. Files under a globals/ directory are assigned the globals subgraph and excluded from domain discovery.

Use case: Correct subgraph-to-domain mapping without requiring stewards to add metadata to individual HML files.

Code: provisa/ddn/parser.py

Tests: tests/unit/test_ddn.py, tests/unit/test_ddn_converter.py, tests/unit/test_hasura_converter.py

15. Proposed Features¶

REQ-684 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

An EncryptionService abstraction with pluggable implementations — LocalKeychain (AES-256-GCM, key from OS keychain), AwsKms, AzureKeyVault, and NullEncryption (dev/test passthrough). All encrypt/decrypt operations route through this interface. Configured via provisa.yaml encryption.provider and encryption.key_id.

Use case: Single interface decouples encryption policy from storage and transport layers, enabling the same application code to run encrypted on desktop, SMB SaaS, and enterprise BYOK deployments.

Code: provisa/encryption/service.py, provisa/encryption/providers.py, provisa/encryption/factory.py

Tests: tests/unit/test_encryption.py

REQ-685 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Envelope encryption for all encrypted payloads. A unique Data Encryption Key (DEK) is generated per payload, used to encrypt data locally with AES-256-GCM in streaming chunks. The DEK is then encrypted by the org master key via KMS. Storage format is encrypted_DEK | IV | ciphertext. DEKs are cached in-process with a short TTL to minimize KMS round-trips. Chunk size is configurable (default 64KB) to bound memory usage on large payloads.

Use case: Prevents large payload memory issues and excessive KMS costs. One KMS call per payload, not per row. Streaming chunks mean memory overhead is constant regardless of payload size.

Code: provisa/encryption/envelope.py

Tests: tests/unit/test_encryption.py

REQ-686 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Application-layer column encryption for sensitive PG data. API keys and RLS rules are encrypted via EncryptionService before write and decrypted after read; encrypted columns stored as BYTEA. Structural metadata (table names, column names) remains plaintext to support query planning. DONE (2026-07) — the two sensitive DB-resident columns are both encrypted at rest. (1) api_sources.auth (keys/tokens) is BYTEA, encrypted on write (openapi register) via the process-configured EncryptionService accessor and decrypted before use (api_source loader). (2) rls_rules.filter_expr — the RLS predicate is injected as SQL by inject_rls at every governance read, so it is sensitive metadata. Encryption is confined to the repository boundary (provisa/core/repositories/rls.py) where upsert encrypts and get_for_table_role/list_for_role/ list_all decrypt. The two raw reads that bypassed the repo (the app startup RLS loader and the admin GraphQL rls_rules field) were funnelled through list_all so decryption is applied uniformly; upsert is the single write choke point (config_loader + admin both use it). The meta-RLS projection exposes the row over the introspection surfaces as ciphertext, consistent with the threat model. NullEncryption passthrough stores plaintext-bytes in the same BYTEA column, so the encrypted path is exercised identically whether or not a provider is configured. Both proven end-to-end against the live Postgres store (ciphertext at rest, decrypted round-trip through the repo, wrong master key rejected). COMPLETE — these are the only sensitive DB-resident columns: the sources and kafka_sources tables persist no credentials at all (the schema stores host/username/auth_type only; passwords/keys are never written, resolved at runtime via the secrets provider), and there is no tenant_config table. Nothing else in the config DB requires column encryption.

Use case: Compromised PG instance exposes only opaque blobs for sensitive fields. Defense in depth beyond schema-per-org isolation.

Code: provisa/encryption/runtime.py, provisa/api_source/loader.py, provisa/openapi/register.py, provisa/core/repositories/rls.py, provisa/core/schema_org.py, provisa/core/schema.sql, provisa/api/app.py, provisa/api/admin/schema.py

Tests: tests/integration/test_api_auth_encryption.py, tests/integration/test_rls_filter_encryption.py, tests/unit/test_encryption_runtime.py, tests/unit/test_rls.py

REQ-687 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

DECIDED (bulk-only, client-held key): encrypt ONLY the S3/Iceberg BULK-RESULT payloads emitted by the large-result redirect path (provisa/executor/redirect.py), NOT the Trino non-SQL query cache. Mechanism = client-side envelope encryption: Provisa generates a per-result DEK, encrypts the serialized body (AES-256-GCM) via EncryptionService BEFORE upload, and stores CIPHERTEXT in S3. The presigned GET URL therefore delivers ciphertext; the client must possess the DEK to decrypt. The DEK travels to the client over the authenticated redirect response channel (not via S3), so a leaked presigned URL or a directly-accessed bucket object is useless without it. This is stronger than object-store SSE (which trusts the store to hold plaintext + the key). Presigned-URL compatibility is preserved precisely because Provisa hands back ciphertext — no SSE-C headers on the GET are required. WHY NOT the Trino non-SQL cache: that cache exists so Trino can QUERY cached rows as SQL (Phase-2 pushdown applies compiled WHERE/ORDER BY/LIMIT against the cache table; provisa/api_source/ trino_cache.py). Encrypting those payload columns makes them opaque to Trino — it can no longer filter/sort/join them, defeating the cache. The query cache is instead protected by short TTL + Trino/PG access control, and is explicitly OUT of scope for payload encryption. KEY-DELIVERY (IMPLEMENTED — role-bound grant + authenticated unwrap): the redirect response carries a GRANT = the raw per-result DEK sealed together with the creating role under the master key (a second envelope blob). To decrypt, the client POSTs the grant to the authenticated POST /data/redirect/unwrap; the server opens it under the MasterKeyProvider, verifies the caller is the creating role (or holds ADMIN/SUPERADMIN), and returns the DEK. The client then AES-256-GCM decrypts the iv/ciphertext parsed from the self-describing S3 blob (provisa.encryption. split_envelope). Possession of the presigned URL + grant alone is useless — decrypt-time authz is enforced AND the grant is integrity-protected so it cannot be re-scoped to another role. Rejected alternatives: inline raw DEK (no decrypt-time authz); SSE-C pre-shared key (breaks plain-HTTP clients). SECOND-CHOICE fallback, not selected: sensitive-data column-level encryption only. IMPLEMENTATION: opt-in via PROVISA_REDIRECT_ENCRYPT (default off, fail-closed — requires a real envelope provider); provisa/executor/redirect.py::_encrypt_payload seals the grant; endpoint at provisa/api/data/endpoint.py::redirect_unwrap. Proven against live MinIO end-to-end (tests/integration/test_redirect_encryption_minio.py) + unit round-trip (tests/unit/test_redirect_encryption.py).

Use case: Bulk result objects in S3/Iceberg are unreadable from the object store or a leaked presigned URL without the per-result DEK, which is released only over the authenticated redirect channel.

Code: —

Tests: tests/unit/test_redirect_encryption.py, tests/integration/test_redirect_encryption_minio.py

REQ-688 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Redis hot table payloads (CTE fragment row data) encrypted via EncryptionService at _store_rows() before write and decrypted at get_rows() after read. Redis ACL prefix isolation and payload encryption operate as two independent controls. Encrypted payload stored as bytes blob under the existing key scheme.

Use case: Redis ACL isolation prevents accidental cross-tenant key access. Payload encryption ensures that even a Redis ACL misconfiguration does not expose plaintext row data.

Code: provisa/cache/hot_tables.py

Tests: tests/unit/test_hot_tables_encryption.py

REQ-689 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Audit log query text column encrypted via EncryptionService before write and decrypted on authorised admin reads. Query text may reveal schema shape and data intent. Stored as BYTEA.

Use case: Audit log compliance records must not expose query semantics to a party who has PG access but not the org encryption key.

Code: provisa/audit/query_log.py, provisa/core/schema_org.py

Tests: tests/unit/test_audit_query_encryption.py, tests/integration/test_audit_encryption.py

REQ-690 · Encryption¶

Status: ✅ complete · Priority: MAY · Type: structural

Client-side decryption in the JDBC driver. Provisa backend passes encrypted blobs through without decrypting. Driver decrypts result columns flagged as encrypted via column metadata using EncryptionService (Java) configured via connection URL params (kms_provider, kms_key_arn). DEK cached in driver process with TTL. Backend never sees plaintext query results.

Use case: Strongest threat posture — a fully compromised Provisa backend cannot read query result data. Modelled on MongoDB CSFLE pattern.

Code: jdbc-driver/src/main/java/io/provisa/jdbc/EnvelopeDecryptor.java, jdbc-driver/src/main/java/io/provisa/jdbc/KmsProvider.java, jdbc-driver/src/main/java/io/provisa/jdbc/LocalKmsProvider.java, jdbc-driver/src/main/java/io/provisa/jdbc/AwsKmsProvider.java, jdbc-driver/src/main/java/io/provisa/jdbc/AzureKmsProvider.java, jdbc-driver/src/main/java/io/provisa/jdbc/GcpKmsProvider.java, jdbc-driver/src/main/java/io/provisa/jdbc/KmsProviders.java, jdbc-driver/src/main/java/io/provisa/jdbc/ArrowStreamResultSet.java, jdbc-driver/src/main/java/io/provisa/jdbc/ProvisaConnection.java, jdbc-driver/src/main/java/io/provisa/jdbc/ProvisaDriver.java, jdbc-driver/pom.xml

Tests: jdbc-driver/src/test/java/io/provisa/jdbc/AwsKmsProviderTest.java, jdbc-driver/src/test/java/io/provisa/jdbc/AzureKmsProviderTest.java, jdbc-driver/src/test/java/io/provisa/jdbc/GcpKmsProviderTest.java, jdbc-driver/src/test/java/io/provisa/jdbc/KmsProvidersTest.java, jdbc-driver/src/test/java/io/provisa/jdbc/EnvelopeDecryptorTest.java

REQ-691 · Encryption¶

Status: ✅ complete · Priority: MAY · Type: structural

Client-side decryption in the Python client (DB-API, SQLAlchemy dialect, ADBC). EncryptionService Python implementation shared across dbapi.py, sqlalchemy_dialect.py, and adbc.py. KMS provider and key configured via connection params. DEK cached in process with TTL.

Use case: Python users (pandas, notebooks, data pipelines) get same client-side decrypt guarantees as JDBC users without a separate decrypt step.

Code: provisa-client/provisa_client/encryption.py, provisa-client/provisa_client/dbapi.py, provisa-client/provisa_client/adbc.py, provisa-client/provisa_client/sqlalchemy_dialect.py

Tests: tests/unit/test_client_encryption.py

REQ-692 · Encryption¶

Status: ✅ complete · Priority: MAY · Type: structural

Client-side decryption for GraphQL clients via a thin wrapper. Provisa GraphQL schema marks encrypted fields with an @encrypted directive. The Provisa GQL client wrapper intercepts responses and decrypts flagged fields using EncryptionService before returning to caller. Scoped to non-browser clients only — browser-side KMS access weakens the threat model.

Use case: GraphQL API consumers (non-browser) participate in client-side decrypt without backend plaintext exposure.

Code: provisa-client/provisa_client/graphql_decrypt.py, provisa/api/data/encrypted_directive.py, provisa/api/data/sdl.py

Tests: tests/unit/test_client_encryption.py, tests/unit/test_high_security_mode.py

REQ-693 · Encryption¶

Status: ✅ complete · Priority: MAY · Type: constraint

High-security mode (security.mode=high in provisa.yaml). When enabled — pgwire server is not started; REST and GraphQL data API endpoints return 403; JDBC and Python client connections without kms_key_arn are rejected at auth. Only JDBC and Python clients with client-side decrypt configured may connect. Provisa backend handles only encrypted blobs and query planning metadata.

Use case: Regulated or airgapped environments where the Provisa backend must not handle plaintext data under any circumstances.

Code: provisa/security/high_security.py, provisa/api/app.py, provisa/api/app_startup.py, provisa/api/app_loaders.py, provisa/core/models.py

Tests: tests/unit/test_high_security_mode.py

REQ-694 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Client-owned KMS key model. Org master key is a CMK in the client's own cloud account (AWS KMS, Azure Key Vault, or GCP KMS). Client grants Provisa a scoped role permission (kms:Decrypt and kms:GenerateDataKey) via cross-account IAM or equivalent. Provisa never stores raw key material. Client revokes grant to immediately render all their data unreadable. Supported for both SMB managed tier and enterprise BYOB tier.

Use case: Client holds the kill switch — revoking KMS access immediately cuts Provisa's ability to read their data without requiring data deletion or migration.

Code: provisa-client/provisa_client/encryption.py

Tests: tests/unit/test_client_encryption.py

11. Platform, Infrastructure & Delivery¶

REQ-695 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

All internal PostgreSQL tables (semantic metadata, non-SQL cache, audit log, materialized view definitions) are scoped to a per-org schema named org_. The asyncpg connection pool sets search_path=org_ on every connection via the connection init hook. A default org_id of "default" is used when ORG_ID env var is not set, producing schema org_default. Existing single-org deployments are unaffected — they transparently use org_default.

Use case: Schema-per-org provides structural isolation between tenants on a shared PostgreSQL instance. Accidental cross-tenant data access requires bypassing both the search_path and the role-level grants.

Code: provisa/core/db.py, provisa/api/app.py

Tests: tests/unit/test_org_isolation.py, tests/unit/test_table_search.py

REQ-696 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

Platform tables (tenants, tenant_config) reside in a dedicated platform schema, never touched by org search_path. The platform schema has a separate asyncpg pool or uses fully-qualified queries (platform.tenants). Org schemas cannot reference or join to the platform schema.

Use case: Tenant registry and billing data must not co-mingle with per-org application data. A compromised org connection cannot enumerate other tenants.

Code: provisa/core/db.py, provisa/api/app.py

Tests: tests/unit/test_org_isolation.py, tests/unit/test_table_search.py

REQ-697 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

init_schema() in provisa/core/db.py accepts an org_id parameter. It creates the org_ schema if not exists, then runs core schema SQL, audit schema SQL, and MV schema SQL within that schema. Called at startup from _init_pg_pool_and_schema() in provisa/api/app.py using ORG_ID env var (default "default").

Use case: Schema creation is idempotent and org-scoped from first boot. No manual schema setup required for new tenant provisioning.

Code: provisa/core/db.py, provisa/api/app.py

Tests: tests/unit/test_org_isolation.py

REQ-698 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

The Trino non-SQL cache schema is org-scoped. cache_location() in provisa/api_source/trino_cache.py accepts org_id and sets cache_schema=org__api_cache. All callers (endpoint.py, router_integration.py, cypher_router.py) pass org_id from request context. The provisa_admin Trino catalog requires no change as Trino uses fully-qualified catalog.schema.table references.

Use case: Non-SQL cached rows for different orgs land in separate PG schemas within the provisa_admin catalog. A Trino query for org A cannot reference org B cache tables without an explicit schema qualifier.

Code: provisa/api_source/trino_cache.py, provisa/api/data/endpoint.py

Tests: tests/unit/test_catalog_cache.py, tests/unit/test_org_isolation.py

REQ-699 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

Each org has a dedicated PostgreSQL role named role_ with USAGE and CREATE on org_ schema only. The asyncpg pool connects as this role. The role has no access to platform schema or any other org schema.

Use case: Database-level role scoping provides a second isolation layer below search_path. A malformed query using an unqualified table name cannot accidentally read another org's data even if search_path is misconfigured.

Code: provisa/core/db.py

Tests: tests/unit/test_org_isolation.py

REQ-700 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

Extends REQ-595 for on-prem multi-org deployments. Redis cache keys, hot table keys, APQ keys, and query result keys are all prefixed with org_id (org::...) rather than tenant_id. Redis ACL rules scope each org's credentials to their key prefix. One org's cache flush or eviction never touches another org's keys. Requires wiring org_id through check_cache/store_result in cache/middleware.py (the same gap blocking REQ-595 completion).

Use case: Redis key prefix isolation complements PG schema isolation (REQ-695). Both layers must be independently correct — ACL misconfiguration in one should not cascade to the other.

Code: provisa/cache/middleware.py, provisa/cache/store.py, provisa/core/org_provisioning.py

Tests: tests/integration/test_cache_store.py, tests/unit/test_cache_store.py, tests/unit/test_org_isolation.py

REQ-701 · Multi-Tenancy¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Org provisioning creates all required infrastructure atomically — org PG schema, PG role, Redis ACL user, and Trino cache schema — in a single provisioning transaction or compensating workflow. Partial provisioning leaves no orphaned resources. Deprovisioning drops all org resources in reverse order.

Use case: Atomic provisioning prevents half-initialised orgs from entering a queryable but misconfigured state. Clean deprovisioning enables tenant offboarding without manual cleanup.

Code: provisa/core/org_provisioning.py

Tests: tests/unit/test_org_isolation.py

REQ-702 · Multi-Tenancy¶

Status: ✅ complete · Priority: MUST · Type: structural

Demo init code and static demo assets are org-scoped. _seed_built_in_sources() passes state.org_id to _seed_meta_domain(); _seed_ops_pg() inherits the correct schema via the asyncpg connection pool search_path set by REQ-695. Static demo file assets (Parquet, SQLite, CSV) contain no schema references. No migration tool is required — org scoping is handled entirely through the connection pool and seed function wiring.

Use case: Demo environments must participate in org isolation (REQ-695 through REQ-701) without special-casing. Seeded data lands in org_ automatically via the pool init hook, ensuring demo deployments and production multi-tenant deployments use the same code path.

Code: provisa/api/app.py

Tests: tests/unit/test_org_isolation.py

10. Execution & Query Planning¶

REQ-703 · High Availability¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Provisa implements a two-tier recovery model for all deployment modes. Tier 1 retries read operations for up to 30 seconds on transient errors using exponential backoff with full jitter (configurable via PROVISA_RETRY_BUDGET_SECS). Tier 2 uses an internal watcher to detect and restart failed software components within 2–3 minutes. Write operations are never retried internally. Machine-level and cluster-level failures are the customer's responsibility.

Use case: Read-heavy query workloads may encounter transient coordinator unavailability, brief network partitions, or temporary query queue saturation. Automatic retry provides resilience without requiring client-side retry logic for all read operations. Software component failures are self-healing within the deployment boundary; cluster-wide HA is configured by the customer based on their failure domain (single-node, VM/metal, K8s).

Code: provisa/executor/trino.py, provisa/executor/direct.py, provisa/scheduler/jobs.py, terraform/aws/main.tf, terraform/gcp/main.tf, terraform/azure/main.tf

Tests: tests/unit/test_high_availability.py

8. Client Access & Protocols¶

REQ-711 · ADBC Client¶

Status: ✅ complete · Priority: SHOULD · Type: structural

adbc_connect() must accept an optional port parameter (default 8815) so callers can connect to Arrow Flight servers running on non-default ports. The hardcoded port 8815 must be replaced with this configurable value throughout provisa_client/adbc.py.

Use case: Operators who cannot run the Flight server on port 8815 (due to port conflicts or network policy) can connect via ADBC without falling back to DB-API or SQLAlchemy.

Code: provisa_client/adbc.py

Tests: provisa-client/tests/test_adbc.py

6. Execution, Routing, Caching & Performance¶

REQ-712 · Cache¶

Status: ✅ complete · Priority: MUST · Type: constraint

check_cache and store_result in provisa/cache/middleware.py pass tenant_id (org_id) through to RedisCacheStore.get/set, so the per-tenant cache key prefix implemented in the store is applied at the call site and multi-tenant cache key isolation is active for query result caching.

Use case: Tenant-prefixed cache keys at the middleware layer ensure one tenant's cached query results can never be returned to another tenant.

Code: provisa/cache/middleware.py

Tests: tests/integration/test_cache_store.py, tests/unit/test_apq_cache.py, tests/unit/test_cache_key.py, tests/unit/test_cache_store.py, tests/unit/test_mv_tenant_isolation.py

10. Graph Export¶

REQ-713 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Neo4j export endpoint accepts graph nodes and edges via POST /data/neo4j-export, including target URL, credentials, database, and node/edge lists.

Use case: Provisa users can export subgraphs queried via Cypher to external Neo4j servers for analysis, backup, or integration with other tools.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-714 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Nodes are MERGE'd into Neo4j using _provisa_id as the deduplication key, with labels derived from tableLabel or compound "Domain:Table" label fields.

Use case: Preserves node identity across export runs and handles domain-scoped node types.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-715 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Node properties are written to Neo4j using SET n += {...} with Cypher literal encoding to safely handle nested objects, arrays, nulls, and special types.

Use case: Ensures property values (booleans, numbers, strings, nested structures) are correctly transmitted to Neo4j without injection or serialization errors.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-716 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Edges are MERGE'd by matching source and target nodes on _provisa_id, with relationship type preserved from the source graph.

Use case: Creates or updates edges in Neo4j with correct source and target identity, supporting graph structure reconstruction.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-717 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Neo4j export endpoint uses HTTP Basic authentication via Authorization header (base64-encoded username:password) to authenticate with the target server.

Use case: Securely authenticates to external Neo4j instances without embedding credentials in the URL.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-718 · Neo4j Export¶

Status: ✅ complete · Priority: MUST · Type: constraint

Connection errors to Neo4j return HTTP 502; timeouts return 504; authentication failures return 401.

Use case: Clients can distinguish transient (502, 504) from permanent (401) failures and retry or fix credentials appropriately.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-719 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Neo4j export endpoint returns {imported: N, errors: [...]} structure, allowing partial success where some statements fail while others succeed.

Use case: Clients can detect which nodes/edges failed to import and retry or log them without treating the entire export as a failure.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

REQ-720 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

E2E export flow batches nodes and edges separately to avoid transaction timeouts, with default batch size of 200 nodes per request.

Use case: Allows exporting large graphs by chunking them into manageable transaction sizes, avoiding Neo4j timeout errors on large statements.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_export.py

4. Source Connectors¶

REQ-721 · Splunk Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Splunk is registered as a source type and exposed via Apache Calcite connector (kenstott/calcite), enabling Splunk search results to be queried as tables through Trino.

Use case: Allows users to register Splunk instances as data sources and query their search results via the Provisa GraphQL API.

Code: provisa/core/models.py, provisa/core/catalog.py

Tests: provisa-ui/e2e/splunk-connector.spec.ts, tests/unit/test_splunk_connector.py

REQ-722 · Splunk Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Splunk connector accepts a URL (from base_url or constructed via https://host:port, default port 8089) to connect to the Splunk management API.

Use case: Provides a flexible connection method: users can specify a complete base_url or rely on host:port defaults to simplify configuration.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/splunk-connector.spec.ts, tests/unit/test_splunk_connector.py

REQ-723 · Splunk Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Splunk authentication: when use_token=true (default), the password field is passed as token; when use_token=false, username and password fields are passed as separate credentials.

Use case: Splunk supports both token-based (preferred for API automation) and username/password authentication, allowing users to choose their auth method.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/splunk-connector.spec.ts, tests/unit/test_splunk_connector.py

REQ-724 · Splunk Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Splunk connector supports optional app and datamodel-filter properties from source.database and source.mapping, plus disable-ssl-validation for self-signed certificates.

Use case: Allows filtering queries to a specific Splunk app and disabling SSL validation for development/test environments with self-signed certs.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/splunk-connector.spec.ts, tests/unit/test_splunk_connector.py

REQ-725 · Splunk Connector¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Splunk connector always sets case-insensitive-name-matching=true, allowing case-insensitive table and column lookups.

Use case: Ensures consistent behavior when users reference Splunk table names regardless of case, matching Splunk's own case-insensitive semantics.

Code: provisa/core/catalog.py

Tests: tests/unit/test_splunk_connector.py

REQ-726 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SharePoint is registered as a source type in the Provisa data connector registry and can be configured as a queryable data source via the Apache Calcite sharepoint connector.

Use case: Users can register SharePoint lists as tables in their Provisa domain, enabling SQL queries and GraphQL access to SharePoint data.

Code: provisa/core/models.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

REQ-727 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SharePoint connector supports two authentication methods: CLIENT_CREDENTIALS (default) and certificate-based authentication via PFX certificate.

Use case: Different organizational security policies require different auth methods; CLIENT_CREDENTIALS suits service accounts while certificates support PKI-based auth.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

REQ-728 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SharePoint connector builds connection properties from source base_url/host (site-url), username (client-id), password (client-secret), database field (tenant-id), and optional mapping fields (certificate_path, certificate_password, auth_type).

Use case: Consolidates SharePoint auth and connection configuration into the standard Provisa Source model, allowing centralized secret management.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

REQ-729 · SharePoint Connector¶

Status: ✅ complete · Priority: MUST · Type: constraint

Secret values in SharePoint connector mapping (auth_type, certificate_path, certificate_password) are resolved via the secrets engine before being passed to the Calcite connector.

Use case: Prevents plaintext credentials from being hardcoded in configuration, enabling vault/environment variable secret injection.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

REQ-730 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

SharePoint connector always enables case-insensitive-name-matching when building connection properties to the Calcite connector.

Use case: SharePoint list names are case-insensitive; enabling this flag ensures consistent behavior across queries regardless of case variation in table/column references.

Code: provisa/core/catalog.py

Tests: tests/unit/test_sharepoint_connector.py

REQ-731 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SharePoint lists are enumerated as schemas and exposed as queryable tables via Trino, allowing users to discover available lists and register them as Provisa tables.

Use case: Users can browse SharePoint lists through the Provisa UI without manual configuration, improving UX for data source discovery.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

REQ-732 · SharePoint Connector¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

SharePoint tables can be registered in Provisa with known column definitions when the Calcite connector does not expose information_schema.columns.

Use case: Allows users to manually define columns obtained from the Microsoft Graph API, bypassing the connector limitation and enabling table registration and queries.

Code: provisa/core/catalog.py

Tests: provisa-ui/e2e/sharepoint-connector.spec.ts, tests/unit/test_sharepoint_connector.py

3. Source Registration & Data Modeling¶

REQ-733 · Aggregate Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Table edit form exposes enable_aggregates and enable_group_by as checkboxes in the admin UI. Stewards toggle these per-table settings to enable or disable aggregate and group-by root fields for the schema.

Use case: Admin UI checkboxes give stewards an intuitive way to manage aggregate/group-by settings without manually editing YAML configuration files.

Code: provisa/api/admin/types.py, provisa/api/admin/db_queries.py, provisa-ui/src

Tests: provisa-ui/e2e/tables-enable-aggregates.spec.ts

REQ-734 · Aggregate Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Table configuration persists enable_aggregates and enable_group_by boolean columns in the registered_tables database table. Both columns default to false. Values are readable via the table edit API endpoint and reflect the current configuration state.

Use case: Persistent storage of aggregate/group-by settings ensures steward configuration choices survive across deployments and are accessible via admin APIs.

Code: provisa/api/admin/db_queries.py, provisa/core/models.py

Tests: provisa-ui/e2e/tables-enable-aggregates.spec.ts, tests/unit/test_aggregate_config.py

1. Access Governance & Security¶

REQ-740 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Masking SELECT expressions only; WHERE, JOIN ON, and other predicates use physical unmasked columns unchanged. Masking injection operates on the SELECT projection list exclusively, preserving query filtering and join semantics.

Use case: Separating masked SELECT projections from unmasked predicates ensures query filters remain executable while users see masked values.

Code: provisa/compiler/mask_inject.py, provisa/compiler/stage2.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py, tests/integration/test_security_integration_extra.py

REQ-741 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Column masking output uses ANSI SQL dialects (REGEXP_REPLACE, DATE_TRUNC, SQL literals) independent of source type. Dialect-specific transpilation via SQLGlot occurs downstream, after masking injection.

Use case: Dialect-agnostic masking generation ensures consistent governance behavior across heterogeneous federated sources (PostgreSQL, MySQL, Snowflake, BigQuery).

Code: provisa/security/masking.py, provisa/compiler/mask_inject.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py

REQ-742 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Masking respects column type constraints — NULL masks allowed only on nullable columns; regex and truncate masks validated for applicable types (regex on text, truncate on temporal). Validation enforced at config load time.

Use case: Type-aware masking validation at config time prevents runtime errors and ensures masks produce valid SQL.

Code: provisa/security/masking.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py

REQ-743 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Masking constant expressions must emit syntactically valid SQL for their type — numeric literals without quotes, string literals with single quotes (escaped apostrophes doubled), NULL keyword (not quoted), boolean TRUE/FALSE keywords (not Python string literals). Special constants MAX/MIN resolve to integer subtype bounds (tinyint=127, smallint=32767, integer=2147483647, bigint=9223372036854775807).

Use case: Correct literal syntax ensures masked values are valid SQL across all dialects without requiring downstream escaping.

Code: provisa/security/masking.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py

REQ-744 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Masking preserves query structure — ORDER BY, LIMIT, GROUP BY, and other clauses remain unchanged; only SELECT projection is rewritten. Masking returns a new CompiledQuery object (immutable transformation), never mutating the input.

Use case: Immutable masking injection and preservation of query structure ensure correct sorting, pagination, and aggregation behavior.

Code: provisa/compiler/mask_inject.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py, tests/integration/test_security_integration_extra.py

REQ-745 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Role-based masking — different roles can have different masks for the same column; admin roles with no masking rules see raw values; other roles see role-specific masks (regex, constant, or truncate). Selection determined by (table_id, role) key in masking rules dictionary.

Use case: Role-specific masking lets different users see different representations of the same data based on their role permissions.

Code: provisa/compiler/mask_inject.py, provisa/compiler/stage2.py

Tests: tests/unit/test_masking_edge_cases.py, tests/e2e/test_masking.py, tests/integration/test_security_integration_extra.py

REQ-746 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Capability enforcement via check_capability and has_capability functions — capabilities are independently assigned per role (query_development, source_registration, table_registration, relationship_definition, security_configuration, query_authorization, ignore_relationships). Admin capability grants all permissions. InsufficientRightsError raised on missing capability.

Use case: Fine-grained, independently verifiable capabilities prevent privilege escalation.

Code: provisa/security/rights.py

Tests: tests/unit/test_rights.py

REQ-747 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

SQL validator bypass for remote same-source relationship pairs — when bypass_uncovered_relationships=True and both tables share the same remote source (graphql_remote or grpc_remote from same source_id), uncovered joins pass V002 validation. Cross-source joins (local↔remote or different remote sources) always require a covered_pair registration. Registered pairs always require correct column matches regardless of bypass flag.

Use case: Remote relationship bypass allows federated GraphQL/gRPC sources to define their own relationship model without Provisa relationship registration.

Code: provisa/compiler/sql_validator.py

Tests: tests/unit/test_sql_validator_bypass.py

REQ-748 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Inverse relationship collision fix — joins producing inverse relationships (e.g., pets←assignments, assignments→pets) must emit distinct rel_types based on target label, not just column pairs. Keyed by (source_label, target_label, join_columns) to ensure reverse joins resolve to correct relationship types.

Use case: Correct inverse relationship resolution prevents relationship type confusion in bidirectional federation scenarios.

Code: provisa/cypher/sql_to_cypher.py

Tests: tests/unit/test_sql_validator_bypass.py, tests/integration/test_security_integration_extra.py

REQ-749 · Security¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Domain policy tri-state mode (legacy/single-domain/namespaced) — legacy mode (use_domains absent) stores declared domain_id verbatim; single-domain mode (use_domains=False) coerces all tables to default_domain; namespaced mode (use_domains=True) stores domain_id as declared. Reload validation sweep rejects pre-existing foreign domains when switching to single-domain mode.

Use case: Domain policy modes enable multi-tenancy architectures without breaking legacy single-domain deployments.

Code: provisa/core/domain_policy.py, provisa/core/config_loader.py

Tests: tests/integration/test_domain_policy_integration.py, tests/unit/test_domain_policy.py

5. Query Languages, Compilation & Operations¶

REQ-750 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher graph variables (nodes, edges, paths) returned in the RETURN clause are serialized as JSON objects with canonical shape. Nodes include id, label, tableLabel, and properties. Edges include identity, start, end, type, properties, startNode, and endNode. Paths include nodes, edges, and length/hops.

Use case: Consistent JSON serialization of graph objects enables client libraries to deserialize typed node/edge/path representations.

Code: provisa/cypher/assembler.py, provisa/cypher/graph_rewriter.py

Tests: tests/unit/test_cypher_assembler.py, tests/unit/test_cypher_graph_rewriter.py, tests/integration/test_cypher_integration_extra.py, provisa-ui/e2e/cypher-api.spec.ts

REQ-751 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Variable-length relationship patterns [*1..n] are compiled to recursive CTEs in Trino SQL. The CTE enforces max-hop bounds and returns edge sequences as JSON arrays. Multiple edge hops are preserved in the array for flat-join paths; recursive paths project NULL (intermediate edges not tracked in CTE).

Use case: Variable-length traversals enable clients to explore multi-hop relationships without writing nested joins.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, provisa-ui/e2e/cypher-variable-length-path.spec.ts, tests/integration/test_cypher_integration_extra.py

REQ-752 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Intermediate node property access in multi-hop MATCH patterns (e.g., (p:Person)-[:WORKS_AT]->(c:Company)-[:HAS_DEPT]->(d:Department)) resolves all three table aliases and preserves property references in WHERE and RETURN clauses. No aliasing conflicts occur.

Use case: Complex traversals require filtering and returning properties from intermediate nodes without manual projection.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-753 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Path object RETURN (e.g., RETURN p) emits a JSON_OBJECT with nodes (array of node objects), edges (array of edge objects), start/end (node identities), and length (hop count). For recursive CTE paths, hops field counts intermediate hops.

Use case: Returning entire paths as objects enables clients to navigate graph traversals without unpacking individual nodes/edges.

Code: provisa/cypher/translator.py, provisa/cypher/graph_rewriter.py

Tests: tests/unit/test_cypher_translator_exprs.py, provisa-ui/e2e/cypher-variable-length-path.spec.ts, tests/integration/test_cypher_integration_extra.py

REQ-754 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Correlated CALL subqueries (e.g., CALL { WITH x MATCH (x)-[:REL]->(y) RETURN y }) translate to CROSS JOIN LATERAL subqueries in Trino, preserving the outer variable binding.

Use case: Correlated subqueries enable filtering inner results based on outer row context without materializing cartesian joins.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-755 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Node label alternation (e.g., (n:Label1|Label2)) translates to UNION ALL branches in SQL, one per label candidate. Each branch includes only the relevant node type and relationships that originate from it.

Use case: Label alternation enables flexible node matching without explicit OR conditions.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-756 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Subquery predicates — EXISTS { ... }, COUNT { ... }, COLLECT { ... } — translate to correlated SQL subqueries. EXISTS becomes EXISTS (SELECT ...), COUNT becomes (SELECT count(...)), COLLECT becomes ARRAY(...).

Use case: Subquery predicates enable complex filtering based on related-node aggregations without intermediate MATCH clauses.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-757 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Map projections — n { .prop1, .prop2 }, n { .* }, n { key: expr } — translate to Trino MAP(...) function calls with ARRAY keys and ARRAY values. Star projections expand schema properties; custom keys are added as named entries.

Use case: Map projections enable client-friendly reshaping of returned objects without manual column aliasing.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-758 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Bidirectional edge traversal (e.g., (a)-[]-(b) without direction marker) expands to UNION ALL when multiple relationship directions exist in the schema. Each branch covers one direction; single-direction schemas skip the UNION.

Use case: Bidirectional traversal enables undirected graph exploration without explicit forward/backward branches.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-759 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Backward relationship traversal (e.g., (c:Company)<-[:WORKS_AT]-(p:Person)) inverts the join condition from the forward relationship definition. Join columns are swapped; ON condition evaluates to tgt.id = src.join_column.

Use case: Backward traversal enables reverse-direction path queries without schema bidirectionality.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-760 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

String, list, and numeric functions (LEFT, RIGHT, TRIM, SIZE, COUNT DISTINCT, REDUCE) translate directly to Trino equivalents or adapters. REDUCE syntax is normalized to Trino lambda form. Legacy {param} syntax is normalized to $param before parsing.

Use case: Function support enables rich query expressions without learning Trino-specific syntax.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-761 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Implicit GROUP BY is inferred in the RETURN clause when non-aggregated columns are mixed with aggregate functions. All non-aggregated columns in RETURN are included in GROUP BY without explicit declaration.

Use case: Implicit GROUP BY matches Neo4j Cypher semantics and reduces query verbosity.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_exprs.py, tests/integration/test_cypher_integration_extra.py

REQ-762 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

CASE expressions (both searched CASE WHEN ... THEN ... ELSE ... END and simple CASE x WHEN ... THEN ... ELSE ... END) translate directly to Trino CASE syntax. Multiple WHEN branches and optional ELSE are preserved.

Use case: CASE expressions enable conditional logic in projections and filters without external control flow.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-763 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

UNWIND clauses (e.g., UNWIND [1,2,3] AS x) translate to Trino UNNEST with CROSS JOIN. Parameters in UNWIND (e.g., $list) are passed through and referenced as positional parameters in the SQL output.

Use case: UNWIND enables iteration over lists without explicit array expansion in the application.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-764 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

IN list predicates (e.g., n.age IN [25, 30, 35]) translate directly to SQL IN clauses with literal or parameter values.

Use case: IN predicates enable efficient filtering on multi-value sets without repeated OR conditions.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-765 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Pattern comprehensions (e.g., [(p)-[:WORKS_AT]->(c:Company) | c.name]) translate to ARRAY(...SELECT...) subqueries with the pattern matched inside and the pipe expression projected.

Use case: Pattern comprehensions enable inline aggregation of related rows without explicit GROUP BY.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-766 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

length(p) function on a path variable returns the hop count (number of edges). For recursive CTE paths, it returns the hops field value; for flat-join paths, it returns 1 or the explicit edge count.

Use case: length(p) enables path-based filtering and ordering without explicit edge counting.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-767 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Recursive CTE shortestPath and allShortestPaths functions enforce max-hop bounds at compile time. shortestPath injects LIMIT 1 and ORDER BY hops; allShortestPaths orders by hops without LIMIT.

Use case: Shortest-path functions enable efficient shortest-path queries without client-side traversal algorithms.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-768 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Node graph variable deserialization from SQL result rows. JSON columns marked as NODE in graph_vars are parsed into Node objects with id, label, tableLabel, and properties extracted.

Use case: Node deserialization enables type-safe access to node objects in the assembler stage before HTTP response serialization.

Code: provisa/cypher/assembler.py

Tests: tests/unit/test_cypher_assembler.py, tests/integration/test_cypher_integration_extra.py

REQ-769 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Edge graph variable deserialization from SQL result rows. JSON columns marked as EDGE are parsed into Edge objects with identity, type, start/end node references, and properties. Legacy (id/startNode/endNode) and Neo4j (identity/start/end) formats are both supported.

Use case: Edge deserialization enables type-safe access to relationship objects with start/end node tracking.

Code: provisa/cypher/assembler.py

Tests: tests/unit/test_cypher_assembler.py, tests/integration/test_cypher_integration_extra.py

REQ-770 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Path graph variable deserialization from SQL result rows. Rows with _path_id and _depth columns are collapsed into single Path objects with nodes and edges arrays. Multiple paths with different _path_ids produce separate Path objects.

Use case: Path deserialization enables client-friendly navigation of multi-hop graph results without manual array unpacking.

Code: provisa/cypher/assembler.py

Tests: tests/unit/test_cypher_assembler.py, tests/integration/test_cypher_integration_extra.py

REQ-771 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Variable-length relationship edge columns (e.g., [c*..5] variable) are deserialized as lists of Edge objects from JSON_ARRAY columns. NULL values from CTE null-projection are preserved as None. Empty lists result from zero-edge paths.

Use case: Variable-length edge deserialization enables structured access to multi-hop relationship sequences.

Code: provisa/cypher/assembler.py

Tests: tests/unit/test_cypher_assembler.py, tests/integration/test_cypher_integration_extra.py

REQ-772 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Graph rewriter applies JSON object wrapping to all graph variables in the SELECT clause. Scalar columns remain unwrapped. The wrapping ensures nodes/edges/paths are serializable to JSON.

Use case: Graph rewriter preserves data integrity by ensuring all graph variables are JSON-compatible before HTTP serialization.

Code: provisa/cypher/graph_rewriter.py

Tests: tests/unit/test_cypher_graph_rewriter.py, tests/integration/test_cypher_integration_extra.py

REQ-773 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Domain-scoped node projections (e.g., MATCH (n:PetStore) RETURN n) include all properties from nodes in the domain in the JSON_OBJECT output. Multiple node types within a domain produce UNION ALL branches, each populating the full property set for that type.

Use case: Domain-scoped queries enable exploration of all entities within a business domain without explicit type specification.

Code: provisa/cypher/graph_rewriter.py, provisa/cypher/translator.py

Tests: tests/unit/test_cypher_graph_rewriter.py, tests/integration/test_cypher_integration_extra.py

REQ-774 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Duplicate JSON key prevention. When a table's id_column differs from 'id' but the table has a column named 'id', the JSON_OBJECT projection emits only one 'id' key (the primary identity key, not the column).

Use case: Duplicate key prevention ensures well-formed JSON output from federated tables with unconventional id column naming.

Code: provisa/cypher/graph_rewriter.py

Tests: tests/unit/test_cypher_graph_rewriter.py

REQ-775 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Anonymous all-rels path pattern MATCH p=()-->() (no node/edge labels or variables) generates a valid SQL subquery referencing all relationships without crashing. No empty table-prefix column references are emitted.

Use case: Anonymous paths enable exploratory graph queries without schema knowledge.

Code: provisa/cypher/translator.py

Tests: tests/unit/test_cypher_translator_paths.py, tests/integration/test_cypher_integration_extra.py

REQ-776 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

OPTIONAL MATCH chains (e.g., OPTIONAL MATCH (a)-[r1]->(b) OPTIONAL MATCH (b)-[r2]->(c)) translate to left-joined multi-hop paths. Each OPTIONAL MATCH becomes a separate LEFT JOIN in sequence, preserving null propagation for unmatched branches.

Use case: OPTIONAL MATCH chains enable exploratory traversals where intermediate results may be absent without failing the entire query.

Code: provisa/cypher/translator.py

Tests: provisa-ui/e2e/cypher-breed-filter.spec.ts, tests/unit/test_cypher_frontend.py, tests/integration/test_cypher_integration_extra.py

REQ-777 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

UNION ALL queries (e.g., MATCH (n) RETURN ... UNION ALL MATCH (m) RETURN ...) preserve distinct column aliases in both branches. Result column order matches the first UNION branch. Queries with property filters on both nodes and edges execute without FederationError.

Use case: UNION ALL queries enable multi-entity result merging and cross-entity property filtering in a single query.

Code: provisa/cypher/translator.py

Tests: provisa-ui/e2e/cypher-union-all-email.spec.ts, tests/unit/test_cypher_frontend.py, tests/integration/test_cypher_integration_extra.py

REQ-778 · Cypher Query Frontend (Phase AU)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher /query/cypher endpoint returns a typed response with columns (array of column names), rows (array of result objects), error (null or error detail), and type (always "cypher"). No row truncation or implicit result limiting occurs beyond query-specified LIMIT.

Use case: Consistent response shape enables client libraries to deserialize typed Cypher query results.

Code: provisa/api/rest/cypher_router.py, provisa/api/data/endpoint.py

Tests: provisa-ui/e2e/cypher-breed-filter.spec.ts, provisa-ui/e2e/cypher-variable-length-path.spec.ts, provisa-ui/e2e/cypher-union-all-email.spec.ts, tests/unit/test_cypher_frontend.py, tests/integration/test_cypher_integration_extra.py

10. UI & Admin Surfaces¶

REQ-779 · Graph Explorer¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Graph Explorer Favorites panel persists and displays user-saved Cypher queries with custom labels, stored in browser localStorage.

Use case: Users need quick access to frequently-used Cypher queries without re-typing them.

Code: provisa-ui/src/components/graph-explorer

Tests: provisa-ui/e2e/graph-favorites.spec.ts

REQ-780 · Graph Explorer¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Graph Explorer Favorites: hovering over a favorite item reveals action buttons (run, rename, delete) that were previously hidden.

Use case: Hover-reveal of action buttons keeps the favorites panel uncluttered while making operations discoverable.

Code: provisa-ui/src/components/graph-explorer

Tests: provisa-ui/e2e/graph-favorites.spec.ts

REQ-781 · Graph Explorer¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Graph Explorer Favorites: clicking a favorite's label loads the Cypher query into the editor for execution or modification.

Use case: Users can quickly load and run a saved query or modify it before execution.

Code: provisa-ui/src/components/graph-explorer

Tests: provisa-ui/e2e/graph-favorites.spec.ts, tests/unit/test_graph_explorer_favorites.py

REQ-782 · Graph Explorer¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Graph Explorer Favorites: inline rename input allows editing a favorite's label with Enter to commit and Escape to cancel; label persists in localStorage.

Use case: Users can update favorite labels to keep them organized and meaningful as their work evolves.

Code: provisa-ui/src/components/graph-explorer

Tests: provisa-ui/e2e/graph-favorites.spec.ts, tests/unit/test_graph_explorer_favorites.py

REQ-783 · Graph Explorer¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Graph Explorer Favorites: delete button removes a favorite from the panel and localStorage immediately.

Use case: Users can clean up outdated or unwanted saved queries.

Code: provisa-ui/src/components/graph-explorer

Tests: provisa-ui/e2e/graph-favorites.spec.ts, tests/unit/test_graph_explorer_favorites.py

5. Query Languages, Compilation & Operations¶

REQ-784 · Cypher Graph Analytics¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Auto-impute endpoint generates relationship edges for visible graph nodes based on their labels and the schema relationship map, executing one Cypher query per relationship pair.

Use case: Graph Explorer users need to automatically discover edges between visible nodes without manually querying each relationship type.

Code: provisa/api/rest/cypher_router.py, provisa/cypher/assembler.py

Tests: provisa-ui/e2e/cypher-impute-edges.spec.ts, tests/unit/test_cypher_analytics.py

REQ-785 · Cypher Graph Analytics¶

Status: ✅ complete · Priority: MUST · Type: constraint

Imputed edge startNode/endNode ids must be stable integers (registered via node_ids table), not raw database primary keys, to enable canvas lookup and correct relationship rendering.

Use case: Graph Explorer relies on stable node ids (as registered in node_ids.id) to correlate imputed edges with visible nodes; raw PKs cause canvas lookup failures.

Code: provisa/api/rest/cypher_router.py, provisa/cypher/assembler.py

Tests: provisa-ui/e2e/cypher-impute-edges.spec.ts, tests/unit/test_cypher_analytics.py

REQ-786 · Cypher Graph Analytics¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Impute-relationships endpoint accepts visible node set with stable integer ids and resolves them to database primary keys via node_ids table lookup before executing relationship queries.

Use case: Graph Explorer passes only the stable integer ids (already loaded); the endpoint must map them to raw PK values for correct relationship filtering.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/cypher-impute-edges.spec.ts, tests/unit/test_cypher_analytics.py

4. Source Connectors¶

REQ-788 · File & Lake Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

File connector sources accept a directory glob pattern to enumerate CSV files. Discovered files are introspected to extract schema (column names and types). Multiple CSV files matching the glob are consolidated into a single logical table when registered.

Use case: File glob patterns enable users to query across multiple CSV files without manual discovery or registration of individual files.

Code: provisa/source_adapters/file_connector.py

Tests: provisa-ui/e2e/file-connector.spec.ts, tests/unit/test_file_lake_sources.py

REQ-789 · File & Lake Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

CSV column headers are automatically mapped to GraphQL field names using LINQ4J convention: camelCase headers are converted to snake_case column names. This mapping is transparent at the schema definition level.

Use case: Automatic header normalization allows CSV files with mixed naming conventions to produce consistent, queryable schemas without manual column mapping.

Code: provisa/source_adapters/file_connector.py

Tests: provisa-ui/e2e/file-connector.spec.ts, tests/unit/test_file_lake_sources.py

REQ-790 · File & Lake Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

File connector table enumeration (via directory glob discovery) is accessible through the Provisa UI table registration form. When a file source is selected in the form, a schema dropdown is populated with discovered schemas, and a table dropdown lists all tables available in the selected schema.

Use case: UI-driven table enumeration allows users to discover and register file-based tables without knowledge of directory structure or manual schema definitions.

Code: provisa-ui/src/pages/tables, provisa/api/rest/tables_router.py

Tests: provisa-ui/e2e/file-connector.spec.ts, tests/unit/test_file_lake_sources.py

REQ-791 · File & Lake Sources¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Registered file-based tables are queryable via the data GraphQL endpoint. Tables appear in the GraphQL schema with domain-prefixed field names. Queries execute read-only against the underlying CSV files with all columns visible (subject to column masking and RLS rules if defined).

Use case: File-based data becomes immediately consumable via standard GraphQL queries, enabling analytics and reporting without additional transformation.

Code: provisa/api/data/endpoint.py, provisa/compiler/sql_gen.py

Tests: provisa-ui/e2e/file-connector.spec.ts, tests/unit/test_file_lake_sources.py

10. Graph Export¶

REQ-792 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

GET /data/graph-schema endpoint returns the current graph schema including all node labels (as array of {label: "Domain:Table"} objects) and relationship types (as array of {type, source, target} objects indicating source and target node labels).

Use case: E2E export clients can discover which node labels and relationship types exist in the current graph before querying and exporting them.

Code: provisa/api/data/endpoint.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

REQ-793 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

POST /data/cypher endpoint accepts parameterless Cypher queries and returns row data. Queries requiring parameters are rejected. Endpoint returns {rows: [...], error?: string} structure.

Use case: E2E export clients can execute discovery and validation queries against the graph without parameter substitution complexity.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

REQ-794 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Query result values are introspected for node and edge structures. Nodes are identified by presence of {id, tableLabel, properties} fields. Edges are identified by {identity, startNode, endNode, type} fields. Both are recursively extracted from arbitrary result objects.

Use case: E2E export clients can query arbitrary graph patterns and automatically extract nodes and edges regardless of result structure nesting.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

REQ-795 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

POST /data/neo4j-export endpoint accepts edge-only requests (empty nodes array) and exports relationships without requiring node re-export. Edges reference nodes via start/end node IDs (matching deduplication key).

Use case: E2E export workflows can batch export large graphs by sending nodes first, then edges in a separate request without resending node data.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

REQ-796 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

X-Role header (e.g., "DEV") grants access to /data/cypher and /data/graph-schema endpoints. Requests without valid role are rejected.

Use case: Data export workflow endpoints require explicit role-based access control distinct from standard query execution permissions.

Code: provisa/api/rest/cypher_router.py, provisa/api/data/endpoint.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

REQ-797 · Neo4j Export¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

E2E neo4j export validates exported graph integrity: node and relationship counts in target Neo4j instance match expected counts. MERGE deduplication on _provisa_id ensures node count ≤ exported count; relationship count must equal exported edge count.

Use case: E2E export clients can verify that export was complete and accurate by querying node and relationship counts in the target Neo4j instance.

Code: provisa/api/rest/cypher_router.py

Tests: provisa-ui/e2e/neo4j-docker-export.spec.ts, tests/unit/test_neo4j_cypher_endpoint.py

5. Query Languages, Compilation & Operations¶

REQ-798 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher mutations (CREATE/DELETE/UPDATE) must be transpiled through the full semantic SQL write pipeline, applying RLS injection, dialect transpilation, and all post-mutation hooks (response cache invalidation, MV stale marking, Kafka change events, Kafka sink triggers, hot-table reload).

Use case: Cypher mutations must enforce the same security policies, maintain consistency across multiple backends, and trigger the same change-propagation workflows as SQL and GraphQL mutations.

Code: provisa/api/rest/cypher_router.py

Tests: tests/unit/test_cypher_mutations.py, tests/integration/test_mutations.py, tests/steps/steps_cypher_mutations.py

6. Query Interfaces & Explorers¶

REQ-799 · UI Explorer Pages¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Browser-based gRPC Explorer UI page at /grpc allows interactive query execution against gRPC service methods. Proto schema displayed inline, request bodies edited as JSON, responses rendered as JSON. Method selection filters by operation type (query vs mutation). Auto-selects first query method on navigation.

Use case: Developers can explore and test gRPC queries without native gRPC client tools, using the same governance pipeline as the gRPC servicer.

Code: provisa/api/data/endpoint_grpc_proxy.py, provisa-ui/src/pages/GrpcPage.tsx

Tests: provisa-ui/e2e/grpc-explorer.spec.ts

REQ-800 · UI Explorer Pages¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Browser-based JSON:API Explorer UI page at /jsonapi allows interactive query composition with sparse fieldsets, filtering (eq, neq, gt, gte, lt, lte, like), sorting, pagination, and relationship inclusion. Table selection grouped by domain, field/include selectors with checkbox dropdowns, result summary view with resource cards and pagination controls.

Use case: Developers can explore and test JSON:API endpoints without manual URL construction, inspecting governance-filtered data across all domains and relationships.

Code: provisa/api/jsonapi/spec.py, provisa-ui/src/pages/JsonApiPage.tsx

Tests: provisa-ui/e2e/jsonapi-explorer.spec.ts

REQ-801 · UI Explorer Pages¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Browser-based OpenAPI Explorer UI page at /openapi embeds Swagger UI iframe serving dynamically generated OpenAPI 3.1 spec from /data/rest/docs. Supports auto-run navigation to pre-selected REST endpoints with method/path detection and try-it-out activation via Swagger UI DOM manipulation.

Use case: Developers can browse REST API endpoints with interactive try-it-out experience powered by Swagger UI, with support for navigation from other explorers (e.g., NL query results linking to generated REST endpoints).

Code: provisa/api/rest/openapi_spec.py, provisa-ui/src/pages/OpenApiPage.tsx

Tests: provisa-ui/e2e/openapi-explorer.spec.ts

4. Query Languages, Compilation & Operations¶

REQ-802 · Protocol Support¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Bolt protocol (Neo4j binary protocol) TCP server at port 5251 accepts Cypher queries and mutations, transpiles through Provisa query pipeline (compile, govern, route, execute). PackStream codec encodes/decodes scalars, lists, dicts, nodes, relationships, and messages. Framing layer chunks messages for TCP transport.

Use case: Neo4j-compatible clients (DBeaver, Cypher shells, drivers) can execute Cypher queries against Provisa schema with full governance applied, expanding query language surface beyond GraphQL/SQL/gRPC/REST.

Code: provisa/bolt/server.py, provisa/bolt/packstream.py, provisa/bolt/session.py, provisa/bolt/messages.py, provisa/bolt/framing.py, provisa/bolt/websocket.py

Tests: tests/unit/test_bolt_packstream.py, tests/integration/test_bolt_server.py, tests/e2e/test_bolt_cypher.py

REQ-803 · Protocol Support¶

Status: ✅ complete · Priority: MUST · Type: structural

HTTP→gRPC proxy endpoint POST /data/grpc/{TypeName} translates JSON request body into GraphQL query using the gRPC schema, compiles and executes through the full query pipeline, returns JSON rows. Supports read_mask with dot-notation for field filtering and native filter args (path_param, query_param) from request body["filter"].

Use case: Browser-based gRPC Explorer and programmatic gRPC clients can query via HTTP+JSON without requiring native gRPC/Protobuf tooling, while applying the same governance and backend routing as native gRPC.

Code: provisa/api/data/endpoint_grpc_proxy.py

Tests: tests/unit/test_grpc_proxy_translation.py, tests/integration/test_grpc_proxy.py

REQ-804 · Protocol Support¶

Status: ✅ complete · Priority: MUST · Type: structural

Dynamic OpenAPI 3.1 spec generator for /data/rest endpoints. Generates per-table paths, typed query parameters (limit, offset, fields, filter, orderBy), response schemas with attributes and relationships. Supports domain filtering, describes limit/offset pagination and sparse fieldsets.

Use case: Swagger UI and REST API consumers can discover and understand auto-generated REST endpoints with parameter types, filter operators, pagination, and output schemas derived from the GraphQL schema.

Code: provisa/api/rest/openapi_spec.py

Tests: tests/unit/test_openapi_spec.py, tests/integration/test_openapi_spec.py

6. Query Interfaces & Explorers¶

REQ-805 · UI Explorer Pages¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Dynamic JSON:API OpenAPI 3.1 spec generator accessible at /data/jsonapi/openapi.json. Generates per-table paths with sparse fieldsets (fields[table]), filtering (filter[col], filter[col][op] for all filter operators), sorting (sort ± prefix), pagination (page[size], page[number]), and include (relationship names). Returns application/vnd.api+json in spec.

Use case: Tools and developers can discover JSON:API endpoints and their constraints (filter operators, field lists, relationship names) without manual inspection.

Code: provisa/api/jsonapi/spec.py

Tests: tests/unit/test_jsonapi_spec.py, tests/integration/test_jsonapi_spec.py

3. Data Storage & Indexing¶

REQ-806 · Bolt Protocol¶

Status: ✅ complete · Priority: MUST · Type: structural

Bolt relationships receive durable integer IDs via a rel_ids Postgres table (BIGSERIAL id, composite_id "Type:startPk-endPk" UNIQUE, rel_type, properties JSONB), mirroring the node_ids design. The register_rel_ids function upserts edges and replaces composite identities with integer IDs.

Use case: Bolt clients require stable relationship identity across browser sessions and multiple query paths (Bolt native, REST cypher, REST impute); numeric IDs enable efficient indexing and relationship traversal.

Code: provisa/core/schema.sql, provisa/bolt/session.py, provisa/bolt/packstream.py

Tests: tests/unit/test_bolt_rel_ids.py, tests/integration/test_bolt_rel_ids.py

1. Access Governance & Security¶

REQ-807 · Protocol Support¶

Status: ✅ complete · Priority: MUST · Type: structural

Bolt auth model establishes principal = user, database = role. A user holds a set of roles; SHOW DATABASES lists one database per (view × role) pair. Database names follow the pattern provisa_<role> (business domains only) and provisa_ops_<role> (includes system/meta/ops domains). The Bolt Browser's database dropdown / :use command selects the active role and view. Home database defaults to the user's first role.

Use case: Bolt Browser users can switch contexts by role and view (ops vs business) without reauthenticating. The database abstraction maps cleanly to role-based domain visibility.

Code: provisa/bolt/session.py, provisa/bolt/packstream.py

Tests: tests/unit/test_bolt_show_databases.py

REQ-808 · Query Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

The Bolt endpoint must never subvert the connecting user's domain rights. The active role's domain_access is the hard ceiling for every query, catalog call (db.labels), relationship counts, and relationship imputation. The ops/business view only narrows within that ceiling. Unknown or unauthorized db values return DatabaseNotFound error (no silent fallback). System, meta, and ops domains are identified by domain_id in {"", "meta", "ops"}.

Use case: Bolt's flexible query language (cypher) could bypass view/role filtering if not enforced at the session/catalog level. This requirement ensures domain visibility is the immovable constraint, not an advisory filter.

Code: provisa/bolt/session.py, provisa/cypher/assembler.py, provisa/api/rest/cypher_router.py

Tests: tests/unit/test_bolt_domain_ceiling.py, tests/integration/test_bolt_domain_ceiling.py

5. Query Languages, Compilation & Operations¶

REQ-809 · SQLGlot Transpilation¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Correlated scalar subqueries in PG-style canonical SQL are lifted into CTEs before the query is forwarded to Trino. rewrite_correlated_subqueries_for_trino in provisa/transpiler/transpile.py rewrites any correlated scalar subquery into a set-based CTE join so the federation engine receives a form it can execute across sources.

Use case: Correlated-subquery-to-CTE rewriting keeps cross-source SQL correct on Trino, which does not execute arbitrary correlated scalar subqueries the way a single RDBMS would.

Code: provisa/transpiler/transpile.py

Tests: tests/unit/test_transpile.py

6. Execution, Routing, Caching & Performance¶

REQ-810 · Materialized Views¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Materialized-view transparent rewriting supports partial join-pattern matches. rewrite_if_mv_match in provisa/mv/rewriter.py matches a query's joins against an MV per-join; when the MV covers only a subset of the query's joins, _partial_rewrite_to_mv removes the covered joins and preserves the remaining joins so the rewritten query reads the MV for the matched portion and joins the rest live.

Use case: Partial MV matching lets a materialized view accelerate queries that join additional tables beyond the MV's definition, instead of only queries whose join pattern matches the MV exactly.

Code: provisa/mv/rewriter.py

Tests: tests/integration/test_execution_routing.py, tests/e2e/test_mv_optimization.py, tests/unit/test_mv_partial_rewrite.py

REQ-811 · Federation Performance¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The # @provisa key=value GraphQL comment hint vocabulary includes a route=federated|direct directive parsed by extract_graphql_hints in provisa/compiler/hints.py. route=federated forces the query through the federation engine; route=direct forces single-source direct execution. This is distinct from the /*+ BROADCAST(...) */ SQL block-hint family (REQ-279) and source-level federation_hints (REQ-281).

Use case: A per-query route hint lets query authors override automatic routing for a specific query without changing source configuration.

Code: provisa/compiler/hints.py

Tests: tests/unit/test_routing.py

8. Client Access & Protocols¶

REQ-812 · API & Integration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The X-Provisa-Sink request header on a subscription request redirects the subscription's change-event output to a Kafka sink instead of the SSE response stream. subscription_sse.py parses the header and launches a Kafka sink consumer for the subscription, returning 202 Accepted rather than an open SSE stream.

Use case: An X-Provisa-Sink header lets a client wire a live subscription directly into a Kafka topic for downstream stream processing without holding an SSE connection.

Code: provisa/api/data/subscription_sse.py

Tests: tests/unit/test_subscription_sink_header.py, tests/integration/test_kafka_sink.py

9. Live Data & Events¶

REQ-813 · Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: structural

The live block (LiveDeliveryConfig on Table) is a unified change-feed / CDC abstraction. The binary delivery: poll|cdc field is replaced by a multi-strategy model: strategy: poll|native|debezium|kafka. Per-table params: poll → watermark_column/poll_interval; native → none; kafka → LiveKafkaParams (topic/format/key_column/field_mapping). Reconciled with REQ-824: Debezium/Kafka delta-transport (bootstrap_servers/topic_prefix/schema_registry_url/consumer_group_id) is inherited from Source.cdc, entered once per source — NOT repeated in per-table debezium params. query_id and outputs are optional so live covers both raw table change-feeds and live persisted-query output fan-out. Debezium topic names are derived from the source transport: {topic_prefix}.{schema}.{table}.

Use case: A single live config abstraction unifies all capture mechanisms (polling, PostgreSQL LISTEN/NOTIFY, MongoDB change streams, Debezium, arbitrary Kafka deltas), eliminating the misleading binary of "poll vs CDC" where poll IS a change-feed mechanism. Strategy-specific params prevent schema pollution. Optional query_id and outputs enable both low-level table subscriptions and high-level live persisted-query delivery.

Code: provisa/core/models.py

Tests: tests/unit/test_live_delivery_config.py

REQ-814 · Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Provider selection (_resolve_provider_type in provisa/api/data/subscribe.py + get_provider in provisa/subscriptions/registry.py) dispatches on live.strategy, NOT on source_type. Validation capability-gates strategy by source_type: warehouses → poll only; MongoDB → native/poll; PostgreSQL → native/debezium/poll/kafka; non-PG RDBMS (MySQL, MariaDB, SQL Server, Oracle) → debezium/poll/kafka (debezium/kafka require source-level cdc transport per REQ-824); Kafka topics → kafka only. No source is registered with a synthetic "debezium" type; strategy=debezium routes to the Debezium provider.

Use case: Dispatching on strategy instead of source_type correctly models that strategy is a choice of delta-capture mechanism (independent of where the data is queried from), not a data source type. Capability-gating per source_type prevents misconfiguration (e.g., native change streams on Snowflake).

Code: provisa/api/data/subscribe.py, provisa/subscriptions/registry.py, provisa/core/config_loader.py, provisa/core/models.py

Tests: tests/unit/test_source_cdc_config.py, tests/unit/test_live_delivery_config.py, tests/steps/steps_subscriptions.py

8. Deployment & Infrastructure¶

REQ-815 · Docker Compose Organization¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Telemetry sink services (otel-collector, prometheus, tempo, grafana, otlp2parquet) are organized into an independently start/stoppable compose project (provisa-observability) separate from the core runtime stack. The app degrades gracefully (fire-and-forget) when no OTLP collector is listening. MinIO base definitions move to docker-compose.core.yml; Trino and trino-worker remain core with observability-specific env/exporter overlays applied only when telemetry is enabled.

Use case: Telemetry infrastructure is optional overhead for test and CI runs. Isolating telemetry sinks reduces startup time and resource consumption while ensuring app correctness does not depend on collector availability.

Code: docker-compose.core.yml, docker-compose.observability.yml, provisa/telemetry/exporter.py

Tests: tests/unit/test_infra_requirements.py

REQ-816 · Docker Compose Organization¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Demo/sample upstream sources (graphql-demo, petstore-mock) are organized into an independently start/stoppable compose project separate from the core runtime, categorized alongside other test-only services (neo4j, elasticsearch, kafka, schema-registry, fuseki, mongodb). Integration tests requiring these fixtures bring them up via harness markers rather than assuming they are already running in the dev stack.

Use case: Demo sources serve as test fixtures for federation and remote-source integration tests. Organizing them with other test services ensures test harness controls their lifecycle and reduces assumptions about the dev environment state.

Code: docker-compose.demo.yml, docker-compose.test.yml, tests/conftest.py

Tests: tests/integration/test_openapi_pet_by_status.py, tests/integration/test_graphql_remote_integration.py

REQ-817 · Trino Memory Management¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Trino memory overflow handling uses Fault-Tolerant Execution (FTE) with task.retry-policy=TASK and task.low-memory-killer.policy=total-reservation-on-blocked-nodes. Docker deployments use a shared named-volume filesystem exchange manager (trino/etc/exchange-manager.properties, /data/trino/exchange). Helm deployments require an S3/MinIO-backed exchange manager with a MinIO bucket-init Job or external S3 endpoint+credentials, enforced by a fail-loud guard.

Use case: FTE replaces legacy spill-to-disk (deprecated in Trino 481) and mitigates noisy-neighbor OOM in multi-tenant deployments by isolating task memory failures and enabling graceful task retry.

Code: docker-compose.core.yml, trino/etc/config.properties, trino/etc/worker/config.properties, trino/etc/exchange-manager.properties, helm/provisa/templates/trino-exchange-secret.yaml, helm/provisa/templates/trino-exchange-bucket-job.yaml

Tests: tests/integration/test_trino_fte_exchange.py

5. Query Languages, Compilation & Operations¶

REQ-818 · Cypher Mutations¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Cypher supports WRITES via the /data/cypher endpoint. CREATE, DELETE, and SET statements execute as direct table writes through the same write pipeline as GraphQL and SQL mutations, with RLS injection, dialect transpilation, and post-mutation hooks. Non-direct write patterns (MERGE, DETACH, REMOVE) are unsupported and rejected. All writes are governed by table/view write rights like any other write. This supersedes REQ-346 (read-only constraint), which is now inaccurate.

Use case: Write support enables graph users to mutate data via Cypher while maintaining governance consistency with SQL and GraphQL mutations.

Code: provisa/api/rest/cypher_router.py, provisa/cypher/write_translator.py, provisa/cypher/parser.py

Tests: tests/integration/test_cypher_endpoint.py, tests/unit/test_cypher_parser.py, tests/unit/test_cypher_translator.py

6. Real-Time & Live Data¶

REQ-819 · Live Delivery Configuration¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Per-table live delivery configuration is now configurable via the admin GraphQL API and admin UI (TablesPage). Configuration is persisted on registered_tables.live (JSONB column). Configuration includes: query_id (identifies the live query), watermark_column (tracks delivery progress), poll_interval (frequency in seconds), delivery mode ("poll" or "cdc"), and outputs array (SSE or Kafka sinks). Previously configuration was YAML-only.

Use case: UI-based configuration makes live delivery settings accessible to non-YAML-fluent operators and enables runtime changes without server restart.

Code: provisa/api/admin/types.py, provisa/api/admin/schema.py, provisa/core/repositories/table.py, provisa/core/schema.sql, provisa-ui/src/pages/TablesPage.tsx

Tests: tests/unit/test_live_delivery_config.py

REQ-820 · Live Query Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Live poll execution routes through Trino (federated query), not the PostgreSQL connection pool. Any federated SQL source with a watermark column is pollable. The engine's pg_pool is used only for watermark bookkeeping (live_query_state table). This enables polling over remote sources (not just PostgreSQL) via Trino's federation layer.

Use case: Routing through Trino enables live polling of any federated SQL source (e.g., BigQuery, Snowflake, S3 via Iceberg) without embedding source-specific clients into the live engine.

Code: provisa/live/engine.py, provisa/api/app.py

Tests: tests/unit/test_live_engine.py

REQ-821 · Live Delivery Mechanisms¶

Status: ✅ complete · Priority: MUST · Type: constraint

PostgreSQL data is never polled for live delivery. PostgreSQL live delivery uses delivery=cdc (LISTEN/NOTIFY with database triggers). Poll delivery (delivery=poll) is reserved for non-push federated sources reached through Trino. This prevents duplicate load on PostgreSQL replication and uses push mechanisms where available.

Use case: PostgreSQL's native LISTEN/NOTIFY is more efficient than polling. Restricting poll to federated sources avoids duplicate query load on the primary database.

Code: provisa/live/engine.py, provisa/core/config_loader.py

Tests: tests/unit/test_live_delivery_config.py

REQ-822 · Live Delivery Mechanisms¶

Status: ✅ complete · Priority: MUST · Type: constraint

delivery=cdc is restricted to source types with a real push provider in the subscription registry: postgresql (LISTEN/NOTIFY triggers), debezium and kafka (Kafka consumers), mongodb (change streams). All other sources default to delivery=poll. CDC connection config is inherited from the source definition, never per-table. This ensures CDC is only used when a push mechanism is available.

Use case: Restricting CDC to sources with native push mechanisms prevents attempts to use CDC on sources that don't support it. Connection config inheritance avoids credential duplication per table.

Code: provisa/core/config_loader.py, provisa/subscriptions/registry.py

Tests: tests/unit/test_live_delivery_config.py

REQ-823 · Live Delivery Configuration¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The LiveEngine reconciles poll jobs from the database at startup and after every admin mutation via _rebuild_schemas(). This ensures that admin-set live config takes effect without server restart. Configuration changes trigger immediate engine reconciliation.

Use case: Startup reconciliation catches live jobs even if the engine crashes mid-execution. Admin mutations trigger immediate reconciliation so operators see changes reflected in real-time without restart overhead.

Code: provisa/live/engine.py, provisa/api/app.py

Tests: tests/unit/test_live_engine.py, tests/unit/test_live_delivery_config.py

REQ-824 · Live Delivery Mechanisms¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Source-level CDC transport configuration. Debezium/Kafka delta-transport (bootstrap_servers, topic_prefix, schema_registry_url, consumer_group_id) is entered ONCE per source via a cdc block on the Source model (persisted to sources.cdc JSONB), never repeated per-table. Per-table live config only selects delivery=cdc; the runtime inherits transport from the source. A source-level cdc block is only valid on CDC-capable RDBMS sources (postgresql uses native LISTEN/NOTIFY and needs no transport; mysql/mariadb/sqlserver/oracle require it to enable delivery=cdc via Debezium). Provider dispatch routes non-PG RDBMS sources with a cdc block to the Debezium provider. Polling remains per-table (watermark_column, poll_interval).

Use case: Eliminates repeated data entry of Debezium/Kafka connection details across every table of a source; transport is a per-source connector property, not a per-table one.

Code: provisa/core/models.py, provisa/core/config_loader.py, provisa/core/repositories/source.py, provisa/core/schema.sql, provisa/api/app.py, provisa/api/data/subscribe.py, provisa/api/admin/types.py, provisa/api/admin/schema.py, provisa-ui/src/liveCapability.ts, provisa-ui/src/pages/SourcesPage.tsx, provisa-ui/src/types/admin.ts, provisa-ui/src/hooks/admin.graphql

Tests: tests/unit/test_source_cdc_config.py, provisa-ui/src/__tests__/liveCapability.test.ts

6. Execution, Routing, Caching & Performance¶

REQ-825 · Execution & Routing¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The Provisa FederationEngine is a pluggable, abstract execution substrate behind a single contract. It is one of four primitives composed as a data-flow pipeline: DATASOURCES → FEDERATION ENGINE → SEMANTIC LAYER → EXECUTION PIPELINE. DATASOURCES are external systems with a capability profile. The FEDERATION ENGINE (this requirement) federates those datasources into one raw unified queryable surface; the method per source is the federation strategy (REQ-826). The SEMANTIC LAYER is the governed virtual model projected over that surface — registered tables/columns, relationships, domains, RLS, masking — engine-independent; it is where governance is DEFINED and the thing queries target. The EXECUTION PIPELINE turns a query into results as a sequence of total IR→IR passes (nanopass discipline), bookended by transport: (1) TRANSPORT ingress — receive the query over the client protocol (pgwire, Bolt, Arrow Flight, GraphQL, SQL); (2) TRANSPILE TO IR — frontend parse of the source language into a dialect-neutral logical IR (plural frontends converge here); (3) TRANSPILE TO GOVERNED IR — apply the semantic layer: inject RLS predicates, masking expressions, and relationship joins (engine-independent; the IP pass); (4) PLAN — the one IMPURE stage: not a pure transpile but a planner that (a) chooses the ROUTE (direct native-driver execution for a single reachable source vs handing the query to the federation engine), (b) resolves RESIDENCY PREREQUISITES — for each source whose federation strategy is MATERIALIZED and stale, emit a side-effecting prep step that loads/refreshes it into the DB cache before execute (REQ-826), and (c) CODEGENs the engine-dialect SQL via SQLGlot for the chosen route. The output is not a string but an ordered plan: a prep phase (0..n cache materializations, may be empty) feeding a single terminal step (DIRECT driver or federation-engine execute); this is where today's router (DIRECT/TRINO/API) and the cache tiers (hot/warm/MV) converge. (5) EXECUTE — the federation engine runs the terminal step and does its own opaque physical planning (NOT Provisa's plan); (6) TRANSPORT egress — stream results back over the same client protocol. Stages 1–3 are pure and engine-independent; stage 4 reads mutable freshness state and emits effects, and stages 4–5 vary per engine — so governance-parity conformance (REQ-827) must PIN residency (pre-warm the prep phase) and then test only the codegen + execute. No governance or compilation logic is duplicated per engine — governance is defined in the semantic layer and ENFORCED at stage 3; the engine only executes. There is no "federation vs warehouse" dichotomy: EVERY engine federates; only the METHOD of federation changes with capability (see REQ-826). An engine advertises a capability profile with two axes: (a) REACH — for each datasource, which federation strategies it can support; and (b) SCALE — single-node/memory-bound vs distributed. The profile RANKS transports per source (native > ADBC > ODBC), not a has_connectors boolean. Concrete engines: DuckDB is a full federation engine whose reach nearly matches Trino's — native ATTACH for postgres/mysql/sqlite; ODBC (community nanodbc odbc_attach, or the DuckDB-team odbc/odbc_scanner functions) for any ODBC source; ADBC (community adbc_scanner, Arrow-native, preferred over ODBC); NoSQL via community extensions (mongo ATTACH TYPE MONGO, cassandra scan/query/ATTACH, elasticsearch_query); and zero-copy file scans. Its ONLY differentiator vs Trino is SCALE (single-node, memory-bound, no distributed shuffle), making it ideal on a developer desktop with no Trino dependency. Trino has broad reach and distributed scale. Snowflake/Databricks/Dremio are reused as engines to leverage a customer's existing infrastructure. A pure RDBMS (e.g. SQL Server) has near-zero reach and federates only by the materialized strategy. The engine is selected at deployment via top-level config (engine: duckdb | trino | snowflake | databricks | dremio | ...), not per-source. All query dispatch that today reads the hardcoded Trino singleton (provisa/api/app.py, provisa/api/data/endpoint.py, provisa/executor/trino.py) must route through the FederationEngine contract. The IR->physical mapping is intentionally thin — the divergence is confined to dialect transpilation and the per-source federation strategy (REQ-826). DONE (2026-07): the PLAN stage OUTPUT is modelled — provisa/federation/plan.py::build_execution_plan composes the pure part of stage 4 into an ordered Plan: a prep phase (0..n PrepStep residency loads/refreshes, one per source whose federate() strategy is MATERIALIZED and stale) feeding a single terminal Route (DIRECT for a single VIRTUAL source, ENGINE otherwise). This is the four-primitive flow's planner surface built on the connector/federate contract (REQ-840/826). REMAINING (kept in-progress): the other pipeline stages as engine-agnostic passes (transport ingress/egress, transpile to IR, transpile to governed IR, codegen, execute) and routing the hardcoded-Trino dispatch through the FederationEngine contract.

Use case: Decouples Provisa's governed semantic layer from any single execution vendor. The same semantic layer, execution pipeline, governance, IR, and multi-protocol query surface run on an embedded desktop engine (DuckDB, no Trino), the Trino federation engine, or a customer's existing warehouse (Snowflake/Databricks/Dremio) — making the engine a swappable deployment choice and concentrating Provisa's IP in the semantic layer, pipeline, governance, and federation-strategy logic rather than the engine. Intended model: develop on DuckDB (laptop, zero infra, full governance and full source reach), then scale out to Trino/Snowflake/Databricks in production, unchanged.

Code: provisa/federation/plan.py, provisa/api/app.py, provisa/api/data/endpoint.py, provisa/executor/trino.py, provisa/transpiler/router.py, provisa/core/config_loader.py

Tests: tests/unit/test_federation_plan.py

REQ-826 · Caching¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Federation strategy and freshness management — the datasources → federation engine arrow of REQ-825. The binding between a datasource table and the engine is a FEDERATION STRATEGY: the method by which that source joins the engine's unified surface. federate(datasource, table) resolves each table to one of three strategies and returns which was chosen, because freshness differs per strategy: (1) VIRTUAL — the engine reaches the source live (a connector, or DuckDB ATTACH to postgres/mysql/sqlite, or a Trino connector); no copy, always fresh, cache_ttl irrelevant. (2) SCAN — the source is a file/object the engine reads in place, exposed as a CREATE VIEW with no data moved; freshness follows the underlying file, cache_ttl irrelevant. For DuckDB this spans read_parquet/read_csv/read_json, httpfs over S3/HTTP/GCS/Azure, Iceberg/Delta via extension, ATTACH of external DuckDB/SQLite, and other scanner extensions; the engine declares what it can scan. (3) MATERIALIZED — no virtual or scan representation (live APIs, NoSQL, or an RDBMS deliberately cached for latency); data is INSERTed/COPYed in. MATERIALIZED is the ONLY strategy where cache_ttl means a reload interval and the load-scheduling/invalidation logic runs; VIRTUAL and SCAN are effectively free. The federation strategy is STATEFUL — it owns residency, reload state, and freshness; the execution pipeline (transpile → transport) is stateless per query and must not carry this state. The chosen strategy depends on BOTH datasource capability and engine capability, so the same source may federate by different strategies on different engines. This unifies today's three separate mechanisms — OpenAPI PG response cache (provisa/openapi/pg_cache.py), NoSQL mapping catalogs, and warm-tables/MV materialization — into one federate() operation. NoSQL scanners (mongo, cassandra, elasticsearch) infer schema on read; because governance in the semantic layer binds to columns, federate() MUST pin the schema on the CREATE VIEW from the semantic layer's mapping (REQ-251), never trust re-inference. The router must not attempt a live/VIRTUAL route when federate() reports only a MATERIALIZED strategy for that source; it uses the strategy federate() returns. Only the TENANT MATERIALIZATION STORE (durable — API/GraphQL/gRPC results, warm tables, MVs; REQ-830) is a true MATERIALIZED strategy: the data becomes a real relation the engine reads, so it PARTICIPATES in federation (joinable/scannable like any source). The TENANT HOT CACHE is NOT a federation strategy and is orthogonal to virtual|scan|materialized — it is a codegen-time VALUES-CTE injection inlined into the SQL at the PLAN stage, never an engine relation and never federated; tier exclusivity (REQ-241) chooses hot over materialization store. DONE (2026-07): federate(source, engine) is implemented (provisa/federation/strategy.py), resolving each source to VIRTUAL | SCAN | MATERIALIZED and returning which was chosen. The choice depends on BOTH source capability and engine capability: an ATTACH connector on a live DB → VIRTUAL, an ATTACH connector on a file/object type (csv/parquet/iceberg/...) → SCAN, a LAND connector (warehouse-native) → MATERIALIZED, and an API/NoSQL source with no connector → MATERIALIZED (loaded into the store). The same source therefore federates differently per engine (csv SCANs on DuckDB but is UnreachableSource on a Trino build without a csv connector). prefer_materialized forces MATERIALIZED for a live source deliberately cached for latency. requires_residency(strategy) is the REQ-825 stage-4b signal (only MATERIALIZED needs a prep step). REMAINING (kept in-progress): the STATEFUL side — wiring residency / reload-scheduling / invalidation and unifying the three existing mechanisms (openapi pg_cache, NoSQL mapping catalogs, warm-tables/MV) behind this one federate() operation, and having the router consume the returned strategy — lands with the materialization store (REQ-844-848/830) and the PLAN stage.

Use case: Concentrates the real complexity of federation — strategy selection, residency, reload scheduling, and invalidation — in one stateful federate() operation, so any datasource (RDBMS, files, NoSQL, APIs) joins the engine's surface by the best available strategy with predictable freshness, independent of which engine is deployed.

Code: provisa/federation/strategy.py, provisa/federation/engine.py, provisa/executor/trino.py, provisa/transpiler/router.py, provisa/openapi/pg_cache.py, provisa/mv/registry.py, provisa/cache/warm_tables.py, provisa/cache/hot_tables.py, provisa/core/config_loader.py

Tests: tests/unit/test_federation_strategy.py

REQ-827 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: constraint

Governance-parity conformance across federation engines. Any engine admitted to the FederationEngine contract (REQ-825) MUST produce byte-identical governed results — RLS filtering, column masking, relationship-join enforcement, and row/aggregate output — as the reference engine for the same query and same identity. Because governance is defined in the semantic layer and enforced when the query compiles to IR, only the transpile → transport stages differ per engine; SQLGlot transpiles syntax, but semantic edges still differ per engine (NULL ordering, type coercion, decimal/numeric precision, regex and collation semantics, timezone and timestamp handling, string comparison case sensitivity), and these must not change which rows are visible, which cells are masked, or which joins are enforced. An engine is certified only by passing a shared governance-parity conformance suite run against a fixed golden dataset and identity matrix; results are diffed against the reference engine and any divergence fails certification. This is a hard gate: an uncertified engine MUST NOT be selectable in production config (REQ-825). The suite is authoritative for DuckDB, Trino, Snowflake, Databricks, Dremio, and any future engine. DONE (2026-07): the comparator + certification GATE are implemented — provisa/federation/conformance.py:: compare_governed_results diffs a candidate engine's governed rows against the reference as an order-INDEPENDENT multiset (NULL ordering is an allowed semantic edge) but visibility/masking-EXACT: a row visible in exactly one side (an RLS leak or over-filter) or a differently-masked cell is a Divergence, and any divergence fails certification. ConformanceRegistry tracks certified engines (the reference is always certified) and require_certified raises UncertifiedEngineError — the hard gate that an uncertified engine must not be selectable. REMAINING (kept in-progress): the conformance SUITE itself — the fixed golden dataset + identity matrix and the runner that executes each query on every candidate engine and feeds the diff — which needs the live multi-engine execution paths.

Code: provisa/federation/conformance.py, provisa/security/, provisa/compiler/, provisa/executor/trino.py, provisa/transpiler/router.py

Tests: tests/unit/test_conformance.py, tests/integration/test_governance_integration.py, tests/integration/test_governance_parity_e2e.py

11. Platform, Infrastructure & Delivery¶

REQ-828 · Deployment & Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Pluggable admin/metadata store, decoupled from Postgres, selected via a SQLAlchemy-compliant URI. The control-plane store (sources, domains, registered tables, columns, naming rules, governance config) plus the OpenAPI/GraphQL cache substrate must run on an embedded engine (DuckDB or SQLite) with zero external infra on a developer desktop, and on Postgres in production — same schema, same behavior. This is required for the desktop deployment model in REQ-825 (PG today requires Docker, which breaks the zero-infra laptop story). Three couplings must be removed: (1) DRIVER — repositories take asyncpg.Connection directly (provisa/core/repositories/source.py) and the pool factory is asyncpg-only (provisa/core/db.py); these move behind a store abstraction driven by a SQLAlchemy URI. (2) DIALECT — schema.sql uses Postgres-specific DDL (SERIAL, JSONB); metadata DDL must be dialect-neutral. (3) META-GOVERNANCE RLS — the hard gate: _init_meta_rls (provisa/api/app.py, REQ-041/REQ-402) enforces control-plane governance via Postgres row-level security, which DuckDB/SQLite lack. Meta-RLS enforcement MUST move from the database into the app layer so control-plane governance is store-independent (the same parity principle as REQ-827, applied to the admin DB). Until meta-RLS is app-enforced, embedded stores cannot be certified for production. On desktop the OpenAPI/GraphQL cache lands in the embedded store rather than a Docker Postgres; where Postgres exists (prod) the existing pg_cache path is unchanged and DuckDB may ATTACH it (REQ-826). This store is the TENANT CONTROL PLANE REGISTRY (REQ-830) — per-tenant, distinct from the PLATFORM CONTROL PLANE REGISTRY (tenant directory, federation-engine registry, platform governance). The meta-RLS move generalizes to a single app-layer tenant-isolation boundary enforced across all tenant-scoped stateful components, since swappable media (Redis, DuckDB, S3) have no native RLS.

Use case: Makes the full Provisa deployment — control plane and cache, not just the query engine — infra-free on a developer laptop, completing the develop-on-DuckDB / scale-out-in-prod model in REQ-825 without requiring Docker or an external Postgres.

Code: provisa/core/meta_rls.py, provisa/core/db.py, provisa/core/database.py, provisa/core/duckdb_async.py, provisa/core/repositories/source.py, provisa/core/schema.sql, provisa/core/schema_org.py, provisa/api/app.py, provisa/openapi/pg_cache.py

Tests: tests/unit/test_admin_store.py

REQ-829 · Deployment & Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Pluggable Redis medium with a lightweight, code-identical embedded realization for desktop. Sibling to REQ-828: where REQ-828 makes the SQL-store medium of the TENANT CONTROL PLANE REGISTRY pluggable (Postgres -> DuckDB/SQLite), this makes the Redis medium pluggable (Redis -> embedded fakeredis), so both halves of the REQ-830 desktop collapse (stateful components 1-4 into one process, zero infra) are covered. Redis is REQ-830's named production medium for the TENANT HOT CACHE (component #4, hot tables REQ-230, provisa/cache/hot_tables.py) and additionally backs the fast result/APQ caches (provisa/cache/store.py, provisa/apq/cache.py) and the Redis-runtime governance state that is orthogonal to the five data-flow components — the sliding-window rate limiter and concurrency gauge (provisa/api/rate_limit.py) and the table-based cache-invalidation index (provisa/cache/store.py). When no REDIS_URL is configured, every one of these MUST transparently use an embedded fakeredis (FakeAsyncRedis) client that is API-identical to redis.asyncio.Redis. All four current connection factories (rate_limit.py:120, cache/store.py:138, apq/cache.py:97, cache/hot_tables.py:174) route through a single make_redis(url, decode_responses) factory that returns redis.asyncio.from_url when a URL is set and FakeAsyncRedis otherwise, sharing one process-wide fakeredis.FakeServer so the four clients (which differ in decode_responses) see the same in-memory store. Zero changes to existing call sites. TENANT ISOLATION for the Redis medium is enforced in the app layer via key namespacing (existing tenant_id key prefixes: provisa:cache::..., provisa:apq::...), NOT any store-native RLS — the same app-layer isolation boundary REQ-828/REQ-830 mandate for all swappable media (Redis, DuckDB, S3); the single shared FakeServer is acceptable because desktop is single-tenant. The stub is not a KVS: it MUST support the full command surface actually invoked — get/set/delete, expire/TTL, sorted sets (zadd/zcard/zremrangebyscore/zrange for the sliding-window rate limiter), sets (sadd/smembers/scan_iter for the invalidation index), incr/decr (concurrency gauge), and pipelines. An aiocache/cashews-style KVS abstraction is explicitly rejected because it cannot express the sorted-set and set operations without rewriting rate limiting and invalidation, and the pure get/set caches it could serve already degrade to no-ops without Redis, so it buys almost nothing. Rationale — desktop already runs without Redis today, but silently: build_rate_limiter(None) returns NoopRateLimiter (rate_limit.py:118) and NoopCacheStore no-ops invalidate_by_table/invalidate_by_pattern (store.py:93-96), so rate limiting and cache invalidation are simply disabled and those code paths are never exercised on the laptop. Yet both surfaces are live in production: rate limiting runs on every HTTP request (rate_limit_middleware.py:28) plus Arrow Flight, subscriptions, and NL endpoints (flight/server.py:248, data/subscribe.py:404, nl_router.py:58), and table invalidation fires after every write (cypher_router.py:560, data/endpoint.py:3307, admin/schema.py:2458). The purpose is therefore NOT to add missing Redis features but to make desktop exercise the SAME hot-cache, result-cache, and rate-limit code paths as production instead of silently falling back to no-ops — so that behavior is testable on a laptop with zero infra. Production with a real REDIS_URL is unchanged; in production the same medium scales out to Redis (REQ-830). Completes the infra-free desktop model (REQ-825/828/830) by removing Redis as the last mandatory Docker dependency alongside the SQL store.

Use case: Lets a developer run the full Provisa backend on a laptop with no Redis server and no Docker, collapsing the Redis medium of the TENANT HOT CACHE and Redis-runtime state into an embedded fakeredis while exercising the identical hot-cache, result-cache, APQ, rate-limit, and invalidation code paths that production runs against a real Redis — rather than the current silent no-op fallback — so behavior is the same, and testable, in both environments, with tenant isolation enforced in the app layer.

Code: provisa/core/redis_factory.py, provisa/api/rate_limit.py, provisa/cache/store.py, provisa/apq/cache.py, provisa/cache/hot_tables.py

Tests: tests/unit/test_redis_factory.py

REQ-830 · Deployment & Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Stateful-component topology. The four data-flow primitives (datasources -> federation engine -> semantic layer -> execution pipeline; REQ-825) describe a STATELESS per-query flow. Orthogonal to them, a deployment holds FIVE PLUGGABLE STATEFUL COMPONENTS, each independently swappable and — except the platform registry — TENANT-ISOLATED: (1) PLATFORM CONTROL PLANE REGISTRY — platform-scoped, shared: tenant directory, federation-engine registry, platform-level governance (REQ-041/REQ-402), and per-tenant routing (which registry/caches/engine each tenant uses). (2) TENANT CONTROL PLANE REGISTRY — per-tenant: persists that tenant's datasource definitions and semantic layer (registered tables/columns, relationships, domains, RLS, masking, naming); the store REQ-828 makes pluggable. (3) TENANT MATERIALIZATION STORE — per-tenant durable store realizing the MATERIALIZED federation strategy (REQ-826): API/GraphQL/gRPC result caches, warm tables, MVs; disk/object-backed. It is a real relation the federation engine reads, so materialized/MV data PARTICIPATES in federation (joinable/scannable like any source). (4) TENANT HOT CACHE — per-tenant fast tier realizing hot tables (REQ-230): small, low-latency, ephemeral. STRICTLY a CTE-injection mechanism — its data is inlined as a VALUES CTE into the generated SQL at codegen and NEVER becomes an engine relation, so it does NOT participate in federation (cannot be joined or scanned by the engine as a table) and is orthogonal to the virtual|scan|materialized strategies. Tier exclusivity (REQ-241) chooses it over the materialization store. (5) FEDERATION ENGINE — the execution substrate (REQ-825); tenant-dedicated or shared. Each is independently pluggable: on a developer desktop components 1-4 collapse into one embedded engine (DuckDB/SQLite) and #5 is DuckDB (zero infra); in production they scale out independently (platform registry in Postgres, tenant registries sharded, materialization store in Iceberg/S3, hot cache in Redis, engine in Trino/Snowflake). Tenant isolation for components 2-5 MUST be enforced in the app layer (REQ-828), not delegated to any single store's native RLS, so isolation holds across every medium.

Use case: Names the stateful components multi-tenancy must isolate and deployment must place, separating them from the stateless execution pipeline, and makes each independently pluggable so the same platform collapses to zero-infra on a laptop and scales each tier independently in production with tenant isolation enforced uniformly in the app layer.

Code: provisa/api/app.py, provisa/core/db.py, provisa/core/repositories/source.py, provisa/openapi/pg_cache.py, provisa/cache/hot_tables.py, provisa/cache/warm_tables.py, provisa/mv/registry.py

Tests: tests/unit/test_arch_invariants.py

REQ-831 · Deployment & Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Demo datasources run as host processes with no Docker. The demo GraphQL service (Strawberry + Starlette ASGI, demo/graphql_server/server.py, port 4000) and the demo OpenAPI petstore service (Starlette ASGI, demo/petstore_server/server.py, port 18080) are launched as background uvicorn subprocesses by demo/run-demo-servers.sh {start|stop}, replacing the former docker-compose services (swaggerapi/petstore3 'petstore-mock' and the built 'graphql-demo' container in docker-compose.demo.yml). start-ui.sh --demo starts and stops them and exports PETSTORE_BASE_URL=http://localhost:18080/api/v3 and GRAPHQL_DEMO_URL=http://localhost:4000/graphql; config/provisa.yaml reads those env vars with defaults that preserve the old Docker service names for containerized runs. This removes the demo-source Docker dependency for local/desktop development, advancing the zero-infra deployment model (REQ-829, REQ-830).

Use case: Lets a developer run the full demo (governed GraphQL + OpenAPI sources) on a laptop without Docker, consistent with the embedded-engine / embedded-store zero-infra desktop model.

Code: demo/run-demo-servers.sh, demo/graphql_server/server.py, demo/petstore_server/server.py, start-ui.sh, start-ui-install.sh, config/provisa.yaml

Tests: tests/unit/test_demo_servers.py

REQ-832 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: structural

Control-plane database is accessed through a SQLAlchemy-Core abstraction (provisa/core/database.py: Database/Connection/Row) backed by an AsyncEngine, replacing raw asyncpg. Connection pool replaces asyncpg.create_pool with equivalent behavior.

Use case: Enables multi-database support (PostgreSQL Tier-1, SQLite/MySQL Tier-2) by centralizing SQL dialect translation and connection management in one module.

Code: provisa/core/database.py

Tests: tests/unit/test_admin_database.py

REQ-833 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: structural

Control-plane data model is split into two SQLAlchemy MetaData registries—platform control plane (orgs, user_profiles, user_org_memberships, local_users, org_invites, tenants, tenant_config) and per-org tenant control plane (sources, registered_tables, table_columns, relationships, roles, rls_rules, domains, materialized_views, kafka_, api_, creation_requests, tracked_functions/webhooks, live_query_state, node_ids, rel_ids, query_audit_log, etc.). Exposed at runtime as state.admin_db and state.org_db.

Use case: Separates multi-tenant platform metadata from per-tenant schema, enabling per-tenant state isolation and independent schema evolution.

Code: provisa/core/schema_admin.py, provisa/core/schema_org.py

Tests: tests/unit/test_schema_metadata_parity.py

REQ-834 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: constraint

Portable-type policy for cross-backend support maps PG-specific types to portable equivalents (JSONB/TEXT[] → JSON, SERIAL/BIGSERIAL → Integer/BigInteger identity, TIMESTAMPTZ → DateTime(timezone), UUID → Uuid, BYTEA → LargeBinary). Dialect-aware upsert() and insert_returning() helpers replace PG-only ON CONFLICT/RETURNING.

Use case: Ensures Tier-1 PostgreSQL, Tier-2 SQLite ≥3.35, and Tier-2 MySQL 8 can all execute the same control-plane schema without migration.

Code: provisa/core/database.py

Tests: tests/unit/test_admin_database.py

REQ-835 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: constraint

A Capabilities flag object gates database features by dialect—LISTEN/NOTIFY, advisory locks, arrays, append-only RULEs, RETURNING are PG-only and must not be used on SQLite or MySQL backends.

Use case: Prevents runtime errors when control plane is deployed on non-PostgreSQL backends by enforcing compile-time feature gating.

Code: provisa/core/database.py

Tests: tests/unit/test_admin_database.py

REQ-836 · Docker Composition & Deployment¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Trino FTE exchange spooling directory (provisa_exchange Docker named volume, mounted at /data/provisa/exchange) must be owned by uid 1000 before Trino starts. A trino-exchange-init busybox one-shot service in docker-compose.core.yml chowns the volume and runs before both trino coordinator and trino-worker via condition: service_completed_successfully.

Use case: Trino runs as uid 1000 but freshly-created named volumes are owned root:root, causing write failures and GENERIC_INTERNAL_ERROR on every FTE query (retry-policy=TASK). Pre-initialization via service dependency ensures filesystem readiness.

Code: docker-compose.core.yml

Tests: tests/unit/test_trino_exchange_init.py, tests/integration/test_trino_fte_exchange.py

REQ-837 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: structural

Platform control plane and tenant control plane use two independent SQLAlchemy engines and connection pools (state.admin_db for platform registry; state.org_db for per-tenant data). Platform database is configured by PLATFORM_DATABASE_URL env var (full SQLAlchemy URI supporting any async backend: postgresql+asyncpg, sqlite+aiosqlite, mysql+aiomysql); required at startup with no fallback.

Use case: Enables deployment models where platform registry and per-tenant data live in separate databases on different backends (platform on SQLite/MySQL, tenant on PostgreSQL, or vice versa), allowing independent lifecycle and scaling.

Code: provisa/core/database.py, provisa/core/schema_admin.py, provisa/api/app.py, provisa/auth/wiring.py, provisa/auth/middleware.py, provisa/auth/providers/basic.py

Tests: tests/unit/test_platform_tenant_engines.py

REQ-838 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: constraint

Tenant control-plane isolation is a CONTRACT, not a mechanism: within a tenant-scoped control-plane connection (state.org_db bound to one org_id), no other org's rows are reachable — reads and writes see only the connecting org's data, and cross-org access requires explicitly binding a different org. The ENFORCEMENT MECHANISM is backend-relative, selected by the control-plane store's dialect and never assumed to be PostgreSQL-specific: PostgreSQL realizes it with a per-connection SET search_path to org_ plus a per-org role (physical schema isolation); single-tenant portable backends (SQLite / MySQL bootstrapped from schema_org) realize it as the sole default schema of a per-tenant database; a backend offering row-security may realize it via a tenant predicate. The invariant is identical across backends — only the realization differs, and unavailable mechanisms degrade explicitly, never silently (REQ-889).

Use case: Per-tenant state is isolated without row-level filtering logic on every query, on whatever control-plane store the deployment chooses. The mechanism is chosen from the backend's capabilities (PG schema+role, single-tenant default schema, or row-security predicate) so the same isolation guarantee holds when the control-plane store is swapped (REQ-850).

Code: provisa/core/database.py, provisa/core/db.py, provisa/api/app.py

Tests: tests/unit/test_org_isolation.py, tests/unit/test_tenant_isolation_contract.py

REQ-839 · Data Model & Persistence¶

Status: ✅ complete · Priority: MUST · Type: constraint

Platform registry tables (orgs, user_profiles, user_org_memberships, local_users, org_invites, tenants, tenant_config) are created from portable SQLAlchemy metadata (schema_admin.init_registry_schema via metadata.create_all), not raw SQL schema files. No cross-database foreign key constraints between platform and tenant planes (FKs from domains.org_id, roles.org_id, etc. dropped to plain columns). id/token generation for local_users and org_invites moved app-side (uuid4) instead of PG-specific gen_random_uuid() server defaults.

Use case: Ensures platform registry can run on any SQLAlchemy-supported async backend (PostgreSQL, SQLite ≥3.35, MySQL 8) without migration or schema.sql maintenance. Plain columns (no FK enforcement) allow platform and tenant planes to exist in separate physical databases with independent schemas.

Code: provisa/core/schema_admin.py, provisa/api/auth_router.py, provisa/api/setup_router.py, provisa/api/admin/orgs_router.py, provisa/api/admin/local_users_router.py, provisa/api/admin/invites_router.py

Tests: tests/unit/test_admin_database.py, tests/integration/test_sqlalchemy_dialect.py

4. Source Connectors¶

REQ-840 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

The federation engine is a pluggable abstraction with per-engine drivers. Three driver classes exist: broad federators (Trino, reaches many external source types), partial federators (DuckDB, attaches a subset of external databases via extensions/scanners), and self-only warehouse-native engines (Snowflake, Oracle, Databricks — reach only their own store). Reachability is connector-presence-defined and binary: reachable(table) is true iff a connector exists for (selected_engine, table.source.type). The connector's mechanism — attach (reference in place) or land (materialize into the engine's reachable store) — is an internal property of the connector, not a reachability tier; warehouse-native engines have only land mechanisms (into self), broad/partial federators use attach where a connector supports it and land otherwise. A federation engine INSTANCE owns its connector collection — a map keyed by source_type (connectors: dict[source_type -> Connector]). Reachability is a lookup into that collection: reachable(table) == (table.source.type in engine.connectors). The three driver classes are defined by collection contents — broad federator (Trino: many source_types), partial federator (DuckDB: postgres/csv/json/parquet where an extension/scanner exists), self-only warehouse-native (Snowflake: own store only). Swapping the engine swaps the connector collection; planner/cache/freshness-gate logic is unchanged. Each Connector projects CatalogEntry rows into the engine catalog. DONE (2026-07): provisa/federation/engine.py::FederationEngine owns a connector collection keyed by source_type; reachable(source_type) is the binary connector-presence lookup; swapping the engine swaps the collection. driver_class() classifies by collection contents — build_trino_engine (postgres/mysql/sqlserver → BROAD), build_duckdb_engine (postgres + csv scanner → PARTIAL), build_snowflake_engine (land-into-self only → SELF_ONLY).

Use case: Support deployments whose engine ranges from a broad federator to a self-only data warehouse without changing planner or cache logic.

Code: provisa/federation/engine.py, provisa/federation/connector.py

Tests: tests/unit/test_federation_engine.py

REQ-841 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: MUST · Type: behavioral

A reachable source (one with a connector for the selected engine) is exposed by that connector's mechanism — attach (reference the source in place, no data movement) or land (physically materialize the source into the engine's reachable store). The mechanism is fixed by the connector, not chosen per query; a source with no connector is rejected as unreachable.

Code: provisa/federation/engine.py, provisa/federation/connector.py

Tests: tests/unit/test_federation_engine.py

REQ-842 · Connector Abstraction¶

Status: ✅ complete · Priority: MUST · Type: structural

A Connector, indexed by (federation_engine, source_type), encapsulates all engine-specific catalog operations — capability(), catalog_add(asset), catalog_remove(asset), land(source), typemap(source->engine). For Trino, catalog_add updates the Trino catalog; for DuckDB it issues ATTACH/scanner setup plus view DDL; for warehouse-native engines it is a no-op (the asset is already a native table). Adding a source type, a federation engine, or a cache backend is a connector registration; planner, cache-role, and freshness-gate logic are unchanged. The connector's derived engine-catalog entry has an explicit persisted row shape: (name: text, engine/platform: text, source_type: text, mechanism: attach|land, details: jsonb). The connector exposes catalog_add, catalog_update, and catalog_remove operations over individual entries, plus a catalog_refresh operation that re-projects the entire engine catalog from the asset registry (REQ-843 reconcile). The details jsonb field is engine+source_type specific — e.g., Trino postgresql = connection-url/user /password .properties; DuckDB postgresql = ATTACH dsn; DuckDB csv/json = read_csv_auto/read_json_auto view DDL over file paths+columns; warehouse -native (Snowflake) = no-op land-into-self. Per the V1 no-migrations constraint, the catalog-entry table is derived/rebuildable engine state, not a migrated Postgres table. DONE (2026-07): provisa/federation/connector.py defines the Connector ABC keyed by (engine, source_type) with capability() and details(source), projecting the persisted CatalogEntry shape exactly — (name, engine, source_type, mechanism: attach|land, details: dict/jsonb). Concrete connectors realize the three engine families: Trino postgres/mysql/sqlserver (attach, connection-url/user/password .properties details), DuckDB postgres (attach dsn) + csv (read_csv_auto view DDL), and a warehouse-native land-into-self no-op. catalog_add/remove/refresh are the EngineCatalog.add/remove/refresh operations over these projected entries. REMAINING (kept in-progress): typemap(source→engine) per connector, and the live APPLICATION of a catalog entry to a running engine (writing Trino .properties / issuing DuckDB ATTACH+view DDL) — wired with the executor in the federate() work (REQ-825-827).

Use case: A single extensibility seam so new engines/sources/backends require only a connector, not core changes.

Code: provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/unit/test_federation_engine.py

REQ-843 · Connector Abstraction¶

Status: ✅ complete · Priority: MUST · Type: constraint

The asset registry is the source of truth for asset existence; the engine catalog is derived, rebuildable state. Connectors project registry state into the engine catalog on asset create/drop and on a full startup reconcile. A missing or stale catalog entry triggers re-projection from the registry — never a fallback or error-and-continue. DONE (2026-07): EngineCatalog is derived state (name → CatalogEntry). FederationEngine.reconcile rebuilds the whole projection from the source registry (only reachable sources project); ensure_entry re-projects when the entry is missing or stale (!= the fresh projection) — never keeps stale, never falls back. on_asset_create/on_asset_drop keep the catalog in step with registry mutations.

Code: provisa/federation/engine.py, provisa/federation/connector.py

Tests: tests/unit/test_federation_engine.py

REQ-844 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: structural

materialization_store — materialized (landed physically by ANY write means: engine-native CTAS/load, SQLAlchemy upsert, or app-side land; REQ-848) into a durable STORE. Configured through the same ${env:...} secrets chokepoint as ControlPlaneConfig. Its members are characterized by three properties that together exclude every other cache: FEDERATABLE (the engine reads it as a real relation it can join/scan — REQ-826; excludes hot tables, which are VALUES-CTE plan injections, and the APQ/result memos, which are not relations); DURABLE (survives restart, disk/object-backed — excludes the ephemeral tier); and REFRESHABLE (actively kept in sync by the one centralized freshness gate — REQ-855/856/857/858; excludes a passive scanned source, which is federatable and durable but unmanaged). It holds TWO ROLES on ONE categorical axis — whether the entry has a semantic-SQL definition: (1) REPLICA — undefined verbatim copies of source rows, no standalone identity; governance/derivation applied at query time on top, never baked in. Two population profiles: reactive (source is UNREACHABLE — landed so it is queryable at all; on-miss, single-flighted pull-through; REQ-845/847; formerly api_cache, renamed as the role is source-agnostic) and warm (source is REACHABLE — an optional latency copy; usage-promoted/demoted by query frequency; REQ-238/239). (2) VIEW — mv: governed, named datasets each defined by semantic SQL (view_sql; implicit relationship-generated or explicit view materialize:true). The definition IS the identity and the basis for lineage, provenance (REQ-862), and definition-versioning. Two orthogonal, ENGINE-RELATIVE attributes apply to any entry and are NOT roles: PLACEMENT — where the entry lands, chosen relative to engine capability (object store, a low-latency fast-read tier, engine-native store, or Iceberg); warm's "fast tier" is just placement=engine's-low-latency-store (Trino realizes it as the Iceberg results catalog + local-SSD file cache, other engines differently), never a Trino-specific definition. FRESHNESS POLICY — TTL / probe / transitive (REQ-855). Population trigger and eviction are NOT roles: both roles are lazy and use the one freshness method; the view background refresh loop and warm's frequency promotion are optional warm-ahead only. SUBSTRATE is selected independently per role (roles may occupy distinct schemas of one backend OR distinct backends), engine-capability-gated (REQ-846): replica → a relational/engine-native store suffices (undefined copies have no forensic value); view → SQLAlchemy DB OR engine-native Iceberg, defaulting to Iceberg whenever a reachable Iceberg catalog exists, because the derived dataset's snapshot history is the forensic/bitemporal value (time-travel, provenance, restatement; REQ-372/862). Iceberg view refresh MUST use overwrite semantics — one clean snapshot per refresh (not delete+insert) — so time-travel maps exactly one snapshot to one refresh state. When no Iceberg catalog is reachable the substrate degrades to the relational/native store as an explicit, surfaced capability signal (never a silent fallback; REQ-847).

Use case: One engine-agnostic materialization store replacing per-protocol hardcoded caches, defined by three properties (federatable, durable, refreshable) that cleanly separate it from the ephemeral hot/APQ/result tiers and from passive scan sources. The replica-vs-view axis drives substrate: cheap relational storage for undefined source copies (reactive/warm), snapshot-capable Iceberg for the governed derived datasets whose history underpins audit, provenance, and bitemporal restatement. Supports security-conscious tenants pointing the store at their own data plane.

Code: provisa/federation/materialization.py, provisa/federation/engine.py

Tests: tests/unit/test_federation_materialization.py, tests/integration/test_materialization_store_lifecycle_e2e.py

REQ-845 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The set of sources served by reactive replicas is engine-relative — reactive = { source : reach(source) == land } for the configured federation engine — generalizing the previously hardcoded REST/GraphQL/gRPC pull-through set (renamed from api_cache; the role is source-agnostic). Pull-through is TTL/freshness-gated (per the REQ-855 freshness policy) and single-flighted to prevent upstream stampede on concurrent misses. DONE (2026-07): the engine-relative reactive set is derived — provisa/federation/materialization.py:: reactive_sources returns { source.id : federate(source, engine) is MATERIALIZED }, so the pull-through set is a function of engine reach (a VIRTUAL/SCAN source is never a reactive replica; an unreachable source is excluded). REMAINING (kept in-progress): the TTL/freshness-gated, single-flighted pull-through execution that lands rows into the store on miss — lands with the write face (REQ-848) and the freshness gate (REQ-855).

Code: provisa/federation/materialization.py, provisa/federation/strategy.py

Tests: tests/unit/test_federation_materialization.py, tests/integration/test_materialization_store_lifecycle_e2e.py

REQ-846 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: constraint

The engine's usable materialized-store set is DERIVED from its connectors — a materialized_store flag on a Connector marks a reachable backend that can also be a store; FederationEngine.materialize_stores is a computed property over the connectors so flagged. PG-only today. Native→IR type translation for MATERIALIZED source landing occurs at the landing boundary in resolve_landing_args (provisa/federation/residency.py), not in the shared config/repository write path. Each column's stored engine-normalized native type is translated to a canonical IR name via provisa.core.ir_types.to_ir(native, platform), where platform is the federation engine's dialect. This containment is necessary because stored data_type across the system is engine-physical and consumed as such by GraphQL/SQL generation, masking, and filters. Unmapped native types raise (IR vocabulary gap), never silent varchar defaults. DONE (2026-07): provisa/federation/connector.py (materialized_store flag on PG connectors), provisa/federation/engine.py (materialize_stores property), provisa/federation/residency.py (resolve_landing_args with to_ir translation).

Code: provisa/federation/materialization.py, provisa/federation/engine.py, provisa/federation/residency.py

Tests: tests/unit/test_federation_materialization.py, tests/unit/test_residency.py

REQ-847 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: constraint

Read-through and write-back discipline. On a pull-through read, a pull failure with no fresh cached data is a hard error; serving stale data is permitted only under an explicit per-source freshness policy (REQ-855), never as a silent fallback. Mutations on a pulled-through relation target the upstream source of truth and invalidate the cache entry; the cache is never written as if it were the system of record. DONE (2026-07): the discipline is implemented as pure policy — provisa/federation/read_through.py:: resolve_read returns SERVE_FRESH (re-pull ok, or cache still fresh), SERVE_STALE (only when stale cache exists AND an explicit per-source stale policy allows it), or HARD_ERROR (pull failed with no fresh data — never a silent stale fallback); plan_mutation states a write targets "upstream" and invalidates the cache entry, never the cache as system of record. REMAINING (kept in-progress): wiring these decisions into the live pull-through read path and the mutation executor.

Code: provisa/federation/read_through.py

Tests: tests/unit/test_read_through.py, tests/integration/test_materialization_store_lifecycle_e2e.py

REQ-848 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The federation engine NEVER writes a materialization store; it merely attaches/links and READS the landed replica. ALL data-source landing goes through ONE write-face abstraction (store_writer.land / ensure_table / reconcile_table), vanilla SQLAlchemy Core today, with pluggable platform branches behind it (pyiceberg for an Iceberg store). The "collapse into the engine" applies only to an engine writing its OWN native store, never a separate/attached one. Read-only engines (Databricks/Snowflake external-linking Iceberg) are first-class readers. DONE (2026-07): provisa/federation/store_writer.py, provisa/federation/materialize_exec.py, provisa/api_source/engine_cache.py (land_api_cache), provisa/federation/duckdb_runtime.py.

Code: provisa/federation/materialization.py, provisa/federation/engine.py

Tests: tests/unit/test_federation_materialization.py, tests/integration/test_materialization_store_lifecycle_e2e.py

3. Source Registration & Data Modeling¶

REQ-849 · Views (Governed Computed Datasets)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

After SQL compilation, the query rewriter inspects FROM/JOIN clauses and transparently rewrites the query to read from a materialized view target table when a matching MV exists — without the query author referencing the MV by name. Rewrites are gated by MV enabled status, freshness, and TTL. Match selection is coverage-optimal and independent of candidate ordering: a full match (all query JOINs covered → entire query redirected to the MV) is always preferred over any partial match, and among partial matches the widest coverage (most MV-covered JOINs replaced, remainder preserved) wins. A fully-matching MV is never preempted by an earlier partially-matching one.

Use case: Transparent MV substitution improves query performance without requiring application changes; TTL gating ensures stale data is not served without explicit policy.

Code: provisa/mv/rewriter.py, provisa/mv/registry.py

Tests: tests/unit/test_mv_partial_rewrite.py

11. Platform, Infrastructure & Delivery¶

REQ-850 · Control-Plane Store Portability¶

Status: ✅ complete · Priority: MUST · Type: structural

Control-plane store is SQLAlchemy-URI-pluggable across dialects, bootstrapped from portable SQLAlchemy Core metadata (provisa/core/schema_org.py) via metadata.create_all, replacing PG-only raw schema.sql (which uses SERIAL, JSONB, DO $$/pg_advisory_lock, CREATE SCHEMA, search_path). PostgreSQL retains the raw schema.sql path unchanged.

Use case: Allows control plane to deploy on non-PostgreSQL backends (SQLite, MySQL, DuckDB, Oracle) without migration, supporting infra-free deployments while maintaining backward compatibility with Postgres.

Code: provisa/core/db.py, provisa/core/schema_org.py, provisa/audit/query_log.py

Tests: tests/unit/test_schema_metadata_parity.py

REQ-851 · Multi-Tenant Org Isolation¶

Status: ✅ complete · Priority: MUST · Type: structural

Org isolation is polymorphic on a schema-capable vs not-schema-capable axis. Schema-capable backends (PostgreSQL, MySQL, Oracle) scope an org as a namespace on a single shared connection via SQL commands (SET search_path TO org_, USE org_, ALTER SESSION SET CURRENT_SCHEMA). Not-schema-capable backends (SQLite, DuckDB) carry org in the database file; org selection happens at engine construction via per-org file paths.

Use case: Provides unified multi-tenant abstraction across heterogeneous databases while respecting native schema isolation capabilities, enabling schema-based multi-tenancy on Postgres/MySQL and file-per-org isolation on SQLite/DuckDB.

Code: provisa/core/database.py

Tests: tests/unit/test_org_isolation.py, tests/unit/test_tenant_isolation_contract.py

REQ-852 · Multi-Tenant Org Isolation¶

Status: ✅ complete · Priority: MUST · Type: structural

OrgRouter (provisa/core/database.py) is the multi-tenant mechanism for not-schema-capable backends, mapping org_id → Database and lazily caching one engine per org_ database file. OrgRouter guards against construction on schema-capable backends and enforces that single-tenant SQLite uses the default namespace with no router. Cross-org queries within a single not-schema-capable connection are impossible (file boundary).

Use case: Enables transparent multi-tenant routing for file-based backends (SQLite, DuckDB) while preventing misconfiguration on schema-capable systems.

Code: provisa/core/database.py

Tests: tests/unit/test_org_router.py

REQ-853 · Control-Plane Store Portability¶

Status: ✅ complete · Priority: MUST · Type: constraint

NOT NULL JSON columns in schema_org.py carry server_default values ('[]' for list, '{}' for dict) so raw-text INSERTs that omit them succeed on portable backends (previously only Python-side Core defaults existed, causing raw inserts to fail NOT NULL constraints on SQLite).

Use case: Ensures compatibility with raw SQL inserts across all backends, preventing NOT NULL violations when JSON columns are omitted in direct SQL statements.

Code: provisa/core/schema_org.py

Tests: tests/unit/test_schema_metadata_parity.py, tests/unit/test_admin_database.py

REQ-854 · Release Artifact Packaging¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Core runtime image set (docker-compose.core.yml + OVA/Lima bundles) must include busybox as a core dependency; no third-party billing mock service is bundled in any compose (dev/core/airgap); python:3.12-slim is build-time only and excluded from packaged/airgapped runtime images.

Use case: Busybox is required at runtime (trino-exchange-init service uses it to chown /data/provisa/exchange before Trino starts). Lemon Squeezy is called over the public REST API; tests stub it via HTTP fixtures (LEMONSQUEEZY_BASE_URL), not bundled mocks. Python:3.12-slim exists only as a build base for OVA/Lima source builds. This separation ensures airgapped deployments include all runtime dependencies and excludes unnecessary dev/mock components.

Code: docker-compose.core.yml, docker-compose.dev.yml, .github/workflows/build-dmg.yml

Tests: tests/unit/test_release_packaging.py

4. Source Connectors¶

REQ-855 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

materialization_store freshness policy — the ONE centralized freshness gate referenced by REQ-845 (reactive replica pull-through) and applicable to REQ-844's view role; both roles evaluate it lazily on access. Per cache entry (per-table, keyed like the existing replica/view targets, NOT per-query-slice) the steward selects one of three modes: (1) TTL only — bounded staleness, current behavior (reactive-replica _mem_fresh expiry / view refresh interval); (2) probe — re-probe the upstream on access and rebuild only on change, ignoring TTL; (3) TTL+probe — probe only after a TTL floor elapses, capping probe frequency. The probe returns an OPAQUE token; Provisa never interprets it, only compares stored vs. fresh by equality. Unchanged token → keep the materialized rows, skip the re-pull/rebuild (zero lag at probe cost); changed token → invalidate the entry and re-pull, then store the new token. MUTUAL EXCLUSIVITY with delta (REQ-874): Probe (opaque-token change detection) and delta (monotonic-cursor incremental reload) are MUTUALLY EXCLUSIVE PER ENTRY, selected by token type. Opaque-token entries ⇒ REQ-855 probe + full re-pull, no delta. Monotonic entries ⇒ delta-as-probe (delta query IS the freshness evaluation), no separate probe runs. Probe transport is source-type-agnostic behind a single dispatch (freshness_token(source, table) -> str | None); None means the source cannot produce a token and the entry degrades to TTL (a capability signal, not a silent fallback — REQ-847). Transports: graphql_remote posts a steward GQL query (generalizing the REQ-673 count_query/graph_counts prototype); openapi uses ETag/Last-Modified or a steward-configured endpoint; DB-SQL sources run a steward SQL query via the engine; file-based sources use file mtime.

Use case: Lets stewards trade probe cost against TTL lag per cached source/table — zero-lag freshness for volatile pull-through sources and materialized views where lag is unacceptable, cheap TTL for static ones — inside the one materialization_store gate (REQ-845) rather than a per-protocol bolt-on. view targets (REQ-844 derivation role) may carry both a TTL/refresh interval and a freshness gate; the gate suppresses needless CTAS rebuilds when the upstream token is unchanged. DONE (2026-07): the centralized gate is implemented — provisa/federation/freshness_gate.py:: evaluate_freshness(mode, now, last_refresh_at, ttl, stored_token, probe) returns a FreshnessDecision (fresh + new_token) for the three modes: TTL (bounded staleness), PROBE (probe every access, ignore TTL), TTL_PROBE (probe only past the TTL floor). The probe yields an opaque token compared by equality — unchanged keeps the rows (zero lag), changed marks not-fresh with the new token to persist — and a probe returning None degrades the entry to TTL (a capability signal, never a silent stale fallback, REQ-847). Pure and reused by both the reactive-replica (REQ-845) and view (REQ-844) roles. REMAINING (kept in-progress): the per-source probe TRANSPORTS behind freshness_token(source, table) -> str | None (graphql query, openapi ETag/Last-Modified, DB-SQL steward query, file mtime) and wiring the gate into the store's lazy-on-access path + delta mutual-exclusivity (REQ-874).

Code: provisa/federation/freshness_gate.py

Tests: tests/unit/test_freshness_gate.py, tests/integration/test_materialization_store_lifecycle_e2e.py

REQ-856 · Freshness Module¶

Status: ✅ complete · Priority: MUST · Type: structural

Freshness gate (REQ-855: three modes TTL/probe/TTL+probe, source-agnostic freshness_token dispatch, transports incl. DB-SQL steward query and file mtime) is generalized into a standalone, context-agnostic module answering one question: given a subject and its observable state, is it fresh? Returns fresh/stale/failed + reason. Pure decision function with no triggering or refresh semantics (caller responsibilities). Enables MV, Source, cache, pgvector all consume one implementation.

Use case: Unifies freshness logic currently scattered across MV (TTL-based, provisa/mv/models.py:94-102), Source introspection, and ad-hoc cache checks (provisa/openapi/pg_cache.py:49). Single implementation enables consistent freshness gating across all data paths.

Code: provisa/freshness/__init__.py, provisa/freshness/decision.py, provisa/freshness/predicate.py

Tests: tests/unit/test_freshness.py

REQ-857 · Freshness Module¶

Status: ✅ complete · Priority: MUST · Type: structural

FreshnessSubject protocol exposes: (1) last-refresh timestamp; (2) last-refresh outcome (success/failure); (3) upstream-subject handles for transitive freshness; (4) content-query capability — this IS REQ-855's freshness_token(source,table) -> str | None dispatch (opaque token; None = source cannot produce one, degrades to TTL per REQ-847).

Use case: Enables FreshnessPredicate to evaluate any subject uniformly—MV, Source, cache all conform to the same interface. Transitive strategy can check upstream freshness by following handles.

Code: provisa/freshness/subject.py

Tests: tests/unit/test_freshness.py

REQ-858 · Freshness Module¶

Status: ✅ complete · Priority: MUST · Type: structural

FreshnessPredicate with pluggable strategies that unify REQ-855's three modes plus a boolean content predicate: TTL (time-based, exists today); PROBE (REQ-855 opaque- token change detection via freshness_token — the same mechanism as REQ-855's DB-SQL steward-query transport, do not treat as a separate "SQL strategy"); TRANSITIVE (fresh iff own predicate holds AND all upstream subjects are transitively fresh — the one net-new capability beyond REQ-855, which is per-entry only); composable (TTL+PROBE floors probe frequency). Caller supplies subject and strategy; predicate returns decision without side effects.

Use case: Decouples freshness evaluation from refresh logic. Strategies are swappable and composable, enabling rich policies (e.g., TTL for cost-sensitive tables, PROBE for zero-lag accuracy, TRANSITIVE for derived caches/medallion chains).

Code: provisa/freshness/predicate.py

Tests: tests/unit/test_freshness.py

REQ-859 · Freshness Module¶

Status: ✅ complete · Priority: MUST · Type: behavioral

MV and the ad-hoc API/pg cache freshness checks (provisa/openapi/pg_cache.py) become FreshnessSubjects (via the StateSubject adapter) and delegate their TTL decision to the one unified FreshnessPredicate. MVDefinition.is_fresh_at keeps its FRESH-status lifecycle gate and delegates the TTL comparison; pg_cache._is_fresh delegates likewise. Behaviour-preserving unification of the scattered TTL logic. Source-level conformance is the gating work in REQ-860 (its interesting PROBE mode depends on the REQ-855 probe transport).

Use case: Consistency: the same freshness semantics and strategies apply to every data path, and there is one place to change freshness policy instead of per-consumer copies.

Code: provisa/freshness/adapters.py, provisa/mv/models.py, provisa/openapi/pg_cache.py

Tests: tests/unit/test_freshness_adapters.py

REQ-860 · Freshness Gating¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Freshness gating is now available at the Source level (provisa/core/models.py), not just on MV. A source can declare a freshness predicate; queries reading from that source are gated by freshness before execution.

Use case: Prevents serving stale data from pull-through sources. Stewards can enforce freshness without materializing into an MV, reducing latency and compute overhead.

Code: provisa/core/models.py, provisa/freshness/source_gate.py, provisa/federation/plan.py, provisa/federation/native_backend.py

Tests: tests/unit/test_source_freshness_gate.py

REQ-861 · Freshness Gating¶

Status: ✅ complete · Priority: MAY · Type: behavioral

File-based sources (CSV, Parquet, etc.) may carry an optional producer command (argv) that runs on-stale (gate-triggered) before the file is read. Allows refresh without modifying the file path or materializing into an MV. DONE (2026-07) - a file source (csv/parquet/sqlite/files) carries an optional producer_command argv (provisa/core/models.py); provisa/freshness/producer.py runs it with shell=False (injection-safe) and fails loud on a non-zero exit. The read path (provisa/events/source_loader.py SourceRowLoader.load) is invoked only after the REQ-860 source gate reports stale, so it runs the producer BEFORE the file read - freshening the file in place without touching path or defining an MV.

Use case: Supports workflows where stale file detection should trigger a script to refresh the file in place. Example: fetch latest data from an upstream API and write to the same CSV path when staleness is detected. Pairs with REQ-855: REQ-855 file mtime DETECTS staleness, this producer command is the ACT-on-stale response.

Code: provisa/core/models.py, provisa/freshness/producer.py, provisa/events/source_loader.py

Tests: tests/unit/test_file_producer_command.py

5. Data Lineage & Provenance¶

REQ-862 · Column-Level Trace Instrumentation¶

Status: ✅ complete · Priority: MUST · Type: structural

MV creation/refresh is instrumented with detailed column-level trace records: for each refresh, the emitted OTel spans capture the resolved per-output-column derivation (upstream source column(s) and transform, from the view_sql), the input snapshot ids/watermark epochs consumed, the active definition-version, and the refresh trace_id. These records land in the governed telemetry substrate (OTel→Parquet, otel catalog). The system does not provide a lineage-resolution engine; lineage is DERIVED BY THE USER querying these trace records (optionally materialized as an MV), which — combined with Iceberg time-travel — yields point-in-time, column-level lineage as a query result. DONE (2026-07): the mv.refresh.column_lineage span now carries all four store-independent stamps — per-output-column derivation (provisa/lineage/columns.py, sqlglot), the definition-version content hash (provisa/lineage/versions.py), the resolved input-version + fidelity kind, and the trace_id. The final piece — actual per-source signal gathering — is wired: at refresh (provisa/mv/input_signals.py::gather_input_signals) each source table is asked, through the same Trino connection, for the strongest signal it offers — an Iceberg snapshot id from "

$snapshots", else an RDB watermark MAX(watermark_column) with the column discovered from the provisa_admin registry catalog (REQ-260) — so input_version reflects the real data version rather than always degrading to the refresh epoch. Best-effort telemetry: sources offering neither contribute nothing and any query error is skipped, never failing the refresh; resolve_input_version grades iceberg_snapshot > watermark > freshness_token > refresh_epoch. Enables audit, impact analysis, and compliance workflows without a bespoke lineage service: because the trace substrate is itself queryable and time-travelable, users answer "where did this column value come from, as of time T" by querying the column-level MV-refresh traces rather than by invoking a dedicated lineage API.

Code: provisa/lineage/columns.py, provisa/lineage/versions.py, provisa/mv/refresh.py, provisa/mv/input_signals.py, provisa/mv/registry.py, provisa/transpiler/, provisa/mv/aggregate_catalog.py

Tests: tests/unit/test_lineage_columns.py, tests/unit/test_mv_input_signals.py

6. Execution, Routing, Caching & Performance¶

REQ-863 · Query Planning Pipeline¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Query planning phases are ordered so that source-set-mutating transforms complete before routing. The pipeline is: query language -> semantic SQL (IR) -> governance (which may ADD sources via RLS subquery predicates) -> a post-governance optimization stage (which may REMOVE sources, e.g. hot-table VALUES-CTE inlining, API-cache rewrites, union-branch pruning) -> route decision + physical lowering -> execute. Routing (extract_sources / decide_route) MUST consume the output of the post-governance optimization stage, never the pre-optimization governed SQL. Consequence: when hot-CTE inlining collapses a federated (multi-source) query to a single remaining live source, routing observes one source and selects DIRECT execution instead of Trino federation; the VALUES CTEs are then lowered in the chosen route's dialect (source dialect for direct pushdown, Trino for federated).

Use case: Direct single-source execution is materially faster than Trino federation. Today the route is decided on the governed SQL BEFORE hot-CTE inlining runs (inlining is nested inside the Route.TRINO branch), so a query whose second source is fully inlined still executes federated. Correct phase ordering makes the DIRECT collapse fall out naturally instead of requiring special-case logic, and unblocks the pushdown of inlined lookups on the direct path. The optimization stage sits after governance because hot inlining depends on the governed logical table reference (REQ-233: governance binds to the table name, then inlining swaps in the CTE, and the governed wrapper filters/masks the CTE rows).

Code: provisa/pgwire/_pipeline.py, provisa/transpiler/router.py, provisa/compiler/sql_gen.py, provisa/cache/hot_tables.py, provisa/compiler/stage2.py, provisa/compiler/sql_rewrite.py

Tests: tests/unit/test_pipeline_order.py

REQ-864 · Query Result Cache¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The query result cache key is computed from a NORMALIZED IR — a canonical, AST-normalized form of the GOVERNANCE-NORMALIZED IR (the post-Stage-2 governed SQL — REQ-263 — after RLS, masking, and column visibility are applied with per-identity values resolved, REQ-041/REQ-402) — rather than the serialized compiled-SQL string it hashes today (provisa/cache/key.py hashes compiled.sql as text). Normalization canonicalizes cosmetic variation that does not change results (whitespace, alias naming/ordering, literal formatting, commutable predicate/JOIN ordering, projection ordering) so two semantically-identical queries resolve to the SAME cache entry instead of missing. Because the keyed form is the governed IR, the resolved persona — role, RLS predicate values, masking, and column visibility — is ALREADY embedded in it (no separate concatenation), and the per-tenant prefix (REQ-595) is applied on top; the RLS-unresolved guard (key.py ValueError) is retained. CRITICAL: normalization canonicalizes cosmetic/structural form ONLY and MUST preserve every identity-encoding element — the RESOLVED RLS predicate values, role, masking, and column visibility — so it can never collapse two personas onto one key (the hypercritical isolation constraint REQ-866). Refines, does not replace, the existing result cache (REQ-544).

Use case: Raises cache hit-rate and eliminates duplicate entries for queries that differ only in text, making the result cache robust to cosmetic query variation from different clients, formatters, or GraphQL shapes that compile to equivalent IR.

Code: provisa/cache/key.py, provisa/api/data/endpoint.py

Tests: tests/unit/test_cache_key.py

REQ-865 · Query Routing¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The route decision includes CACHED as a first-class route alongside DIRECT (REQ-027) and FEDERATED/Trino (REQ-028): the Route enum gains Route.CACHE and decide_route (provisa/transpiler/router.py) evaluates the result cache — keyed per REQ-864/REQ-544 — as the first candidate route. A cache hit resolves to Route.CACHE and serves the stored result with no direct or federated execution; a miss falls through to DIRECT/FEDERATED as today. This elevates the current outer-wrapper cache check (which runs before decide_route in endpoint.py and is invisible to routing) into the routing model, so every query is served by exactly one uniformly-decided route: {cached, direct, federated}. CACHED belongs in the route decision precisely BECAUSE the routing point is where the query is GOVERNANCE-NORMALIZED: routing consumes the post-Stage-2 governed IR (REQ-263, persona-resolved via REQ-041/REQ-402), so the cache key derived there ALREADY encodes the requester's persona and a cache serve is inherently isolated (REQ-866) — a pre-governance wrapper cannot guarantee that structurally. The no-cache bypass (@noCache / no_cache / @cached(ttl:0); REQ-544) removes CACHED from the candidate set for that query. Routing consumes the post-optimization IR (REQ-863), so cache-key computation and route selection operate on the same normalized form.

Use case: Unifies the serving model — one decision point owns how every query is served — makes cache participation observable in the route/plan (traceable like any route), and lets routing policy reason about the cache alongside direct and federated execution instead of it being a hidden pre-step.

Code: provisa/transpiler/router.py, provisa/api/data/endpoint.py, provisa/cache/store.py

Tests: tests/unit/test_routing.py

1. Access Governance & Security¶

REQ-866 · Cache Isolation¶

Status: ✅ complete · Priority: MUST · Type: constraint

HYPERCRITICAL isolation invariant for the query result cache. The cache key MUST fully partition by the RESOLVED governance identity of the requester, so a cache hit can NEVER serve rows computed for a different persona. This matters because personas can switch on the same device, connection, or pooled session; a mis-scoped key would leak one user's data to another. The key MUST bind to: the per-tenant prefix (REQ-595), role_id, and the fully-RESOLVED RLS predicates plus masking and column visibility — i.e. RLS current_setting/session values substituted to their actual per-identity values (REQ-041/REQ-402), NOT the unresolved rule template (identical across users, which would collapse distinct personas onto one key). This isolation is STRUCTURAL when the cache is a first-class route (REQ-865): the route decision operates on the governance-normalized IR, so the key intrinsically encodes the persona. IR/SQL normalization (REQ-864) canonicalizes cosmetic and structural form ONLY and MUST preserve every identity-encoding element. FAIL-CLOSED: if any identity dimension is unresolved or unknown, the cache MUST NOT be read or written for that query (the key.py RLS-unresolved guard), and CACHED (REQ-865) is removed from the candidate routes — never a silent fallback that could serve another persona's data.

Use case: Prevents cross-persona / cross-user data leakage through the result cache when identities change on a shared device, connection, or pooled session — a hard multi-tenant and RLS isolation guarantee, consistent with the two-layer governance model (REQ-038) and Stage-2 RLS/masking enforcement (REQ-263).

Code: provisa/cache/key.py, provisa/api/data/endpoint.py

Tests: tests/unit/test_cache_key.py

2. Authentication & Identity¶

REQ-867 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: structural

Table-scoped mutation registration with per-mutation grants across all remote schema adapters (GraphQL remote, OpenAPI, gRPC, Hasura). Mutations are registered as first-class sub-resources, each with its own writable_by list. Empty writable_by = default-deny. The authorization model is protocol-agnostic; shared/universal layer includes table-scoped mutation sub-resources, per-mutation writable_by, global WRITE capability, execute-time enforcement, read-statement taint, ranked-candidate suggestion interface, admin-confirm, and default-deny. What is per-adapter is only the signals used for classification and association.

Use case: Enables fine-grained authorization of table mutations independent of the table's read visibility across all query sources (GraphQL, OpenAPI, gRPC, Hasura), supporting mutation execution as UDFs in the semantic layer.

Code: provisa/security/mutation_authz.py, provisa/security/rights.py, provisa/core/models.py

Tests: tests/unit/test_mutation_authz.py

REQ-868 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: structural

Global WRITE (alias EXECUTE_MUTATION) capability added to the Capability enum. Acts as a coarse master switch; role must hold WRITE AND appear in the specific mutation's writable_by to invoke a mutation.

Use case: Two-gate authorization model: coarse capability check (does role have WRITE at all?) plus fine-grained mutation-level grant check (is role in this mutation's writable_by?). Analogous to Postgres role-level rights + per-table GRANT.

Code: provisa/security/rights.py

Tests: tests/unit/test_mutation_authz.py

REQ-869 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: constraint

Mutations are classified as WRITES by contract (not caller declaration) via protocol-specific classifiers. Per-adapter read/write classifiers: GraphQL uses operation type (type Mutation), OpenAPI uses HTTP method (POST/PUT/PATCH/DELETE = write; GET = read) plus x-provisa-kind override, gRPC uses MethodOptions.idempotency_level (NO_SIDE_EFFECTS = read; IDEMPOTENCY_UNKNOWN and IDEMPOTENT = write; AIP naming and server-streaming are non-normative heuristics only), Hasura uses exposed_as/action_type. Universal default: unknown/unclassifiable → treat as write (default-deny). Any read statement (SELECT) referencing a kind=mutation UDF is promoted to write and subject to both WRITE capability + writable_by check. Execute-time enforcement in _execute_action_field validates role_id ∈ mutation.writable_by.

Use case: Prevents callers from invoking mutations by disguising them as SELECT statements. Protocol-agnostic classification with per-adapter signals ensures mutations cannot bypass write-level authorization regardless of source.

Code: provisa/security/mutation_authz.py, provisa/api/data/endpoint.py

Tests: tests/unit/test_mutation_authz.py

REQ-870 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Admin-only reclassification: legitimate reclassification of a mutation to read-safe is gated by ACCESS_CONFIG capability and recorded as a governance decision. No per-operation read_only opt-out for callers; only admin acts can demote a mutation. Re-introspection of discovered mutations registers with empty writable_by; existing grants preserved by name. DONE (2026-07): two enforceable behaviors. (1) reclassify_kind (provisa/security/ mutation_authz.py) is the single reclassification gate — only a role holding ACCESS_CONFIG (ADMIN bypasses, per has_capability) may demote a mutation to read-safe; the reverse (promoting a read to a write) is rejected for everyone, and there is no caller-supplied read_only flag anywhere in the request path, so classification stays under governance control. A no-op transition is idempotent. (2) upsert_function (provisa/core/repositories/ function.py) now preserves an existing writable_by on re-introspection: a discovered mutation upserts with an empty writable_by that keeps the current admin grant (grants preserved by name), while an explicit non-empty grant still applies. Proven with unit tests for the gate and a live-Postgres test for grant preservation across re-introspection.

Use case: Ensures authorization policy remains under control of governance (admin), not individual callers. Prevents callers from overriding mutation classification, supports safe discovery/re-registration of mutations without accidentally granting them.

Code: provisa/security/mutation_authz.py, provisa/core/repositories/function.py

Tests: tests/unit/test_mutation_authz.py, tests/integration/test_mutation_writable_by_preservation.py

REQ-871 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Mutation↔table association suggestions at registration time via protocol-specific association-suggesters. Universal rule: adapter emits ranked (mutation → table) candidates; suggestions are hints only; admin-confirmed; confirmed associations register with empty writable_by (default-deny); any mutation may bind to any table (manual override always); unmatched mutations remain registerable. GraphQL: walk mutation return type's fields, unwrap NON_NULL/LIST, match leaf types against type_to_table; LIST-valued field = changed-records collection; prioritize single object > list-of-table-type; ignore scalar/stats fields. OpenAPI: align via URL path template, operationId stem, and tags (e.g., POST /users → users table); response schema is tiebreaker only, not primary signal. gRPC: response-message repeated field type + method-name entity stem (weakest signal). Fallbacks (all adapters): operation name affixes (create/update/delete/upsert) and input-type stem for operations whose response carries no queryable type. False negatives expected; no suggestion auto-binds. DONE (2026-07): the suggester engine (provisa/security/association_suggester.py) implements all three protocols and the shared fallbacks exactly as specified, returning a ranked list of TableCandidate(table, score, reason) deduplicated per table (strongest signal wins, corroborating signals add a small tiebreak). Signal strengths encode the spec order: GraphQL single-object return (1.0) > list-of-table-type (0.8); OpenAPI path resource (0.9) > operationId stem (0.7) > tag (0.6) with response schema as a 0.05 tiebreak; gRPC repeated response-field type (0.6) > method-name stem (0.4); name-affix / input-type stem fallback (0.3). Nothing auto-binds — the functions are pure and return hints; an empty list is the honest false-negative case. The GraphQL remote mapper (provisa/graphql_remote/mapper.py) invokes it during registration so every mapped mutation entry carries suggested_associations = [{table, score, reason}] computed from its unwrapped return-type leaves + input-type stem, with writable_by left empty (default-deny). Consuming these hints into an admin confirmation surface is REQ-870; projecting/binding across surfaces is REQ-872.

Use case: Registration of mutations across heterogeneous sources requires protocol-aware association signals. Per-adapter suggesters reduce admin friction by ranking likely alignment; unmatched mutations remain registerable as global functions or require manual binding.

Code: provisa/security/association_suggester.py, provisa/graphql_remote/mapper.py

Tests: tests/unit/test_association_suggester.py, tests/unit/test_graphql_remote_mapper.py

REQ-872 · Authorization¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Registered remote-schema mutations/functions (tracked_functions, tracked_webhooks) MUST be projected into each query surface's native function catalog and be invocable from each surface, routing through the shared executor and enforcing per-mutation writable_by authorization. Single source of truth: tracked_functions + tracked_webhooks registry. Per-surface fidelity: pgwire populates _pg_proc and supports SELECT fn(...), SQL/Explore projects information_schema.routines and supports table-valued-function invocation, Cypher/Bolt binds CALL (args) YIELD ... with optional discovery. DONE (2026-07) — the SQL-surface DISCOVERY projection: the pgwire catalog builds pg_proc and information_schema.routines + information_schema.parameters from the tracked_functions registry (provisa/pgwire/catalog.py::_populate_functions, called from _build_catalog_db), so psql \df, DBeaver and Explore now see registered functions. The registry is the single source of truth; projection is role-scoped by visible_to (unrestricted or the role is granted); table-valued functions (a return_schema) are set-returning (proretset/record); arguments become information_schema.parameters rows in ordinal order with SQL data types; the same function registered under multiple keys (bare + domain-prefixed alias) projects once. DONE (2026-07) — the SHARED EXECUTOR + Cypher INVOCATION: the tracked-function execution core is extracted from the GraphQL action path into invoke_tracked_function(name, args, state, role_id) (provisa/api/data/endpoint.py) — surface-agnostic, enforcing per-mutation writable_by by contract (REQ-869) then running SELECT * FROM "schema"."fn"(args) through the function's source pool. The GraphQL path now routes through it, and the Cypher surface binds CALL (args) YIELD col [AS alias] to it (provisa/api/rest/cypher_router.py::_detect_registered_call / _handle_registered_call): the call is intercepted before parse, args (literals / $params) are coerced positionally, writable_by is enforced in the executor (403 on deny), and YIELD projects the returned columns. DONE (2026-07) — the pgwire / SQL INVOCATION: execute_pgwire_sql intercepts a bare SELECT-of-a-registered-function before governance/routing and runs it through the same executor (provisa/pgwire/function_call.py). Both the table-valued form SELECT * FROM fn(args) and the scalar form SELECT fn(args) are recognized via sqlglot (an Anonymous func whose name is a registered tracked function), literal args are coerced positionally, and the executor's row dicts are adapted back to a pgwire QueryResult — so psql / DBeaver / Explore invoke registered functions exactly like native ones, with writable_by enforced. All surfaces (GraphQL, Cypher CALL, pgwire/ SQL SELECT) now discover AND invoke registered functions through the one shared executor.

Use case: UDF registry currently exposed only through GraphQL; pgwire, SQL/Explore, and Cypher surfaces cannot discover or invoke registered functions. Cross-surface projection and execution binding unifies function access across all query surfaces and enforces consistent authorization.

Code: provisa/pgwire/catalog.py, provisa/pgwire/function_catalog.py, provisa/pgwire/function_call.py, provisa/pgwire/_pipeline.py, provisa/api/data/action_exec.py, provisa/api/rest/registered_call.py, provisa/api/rest/cypher_router.py

Tests: tests/unit/pgwire/test_function_projection.py, tests/unit/pgwire/test_function_call.py, tests/unit/test_tracked_function_invocation.py

9. Live Data & Events¶

REQ-873 · Subscriptions¶

Status: ✗ rejected · Priority: MUST · Type: behavioral

SUPERSEDED BY REQ-874. This requirement duplicated existing watermark/probe change-signal work already captured in REQ-855 (probe transport, opaque token comparison), REQ-260 (RDB watermark_column polling), and REQ-282 (Live Query Engine polling). Its only net-new capability (incremental delta fetch/append to materialized replicas instead of full re-pull) is now carried by REQ-874, which generalizes delta-fetch to any federated dataset and separates WATERMARK (existing change-signal) from DELTA (the new incremental append). This entry is retired; use REQ-874 and REQ-855 instead.

Code: —

Tests: —

4. Source Connectors¶

REQ-874 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Delta fetch for materialization_store REPLICA incremental refresh: an INCREMENTAL RELOAD strategy for datasets whose federation strategy is MATERIALIZED (REQ-826) — data is landed/cached, so full re-pull is the cost delta avoids. Remote schemas (graphql_remote/ openapi/grpc) are the PRIMARY DRIVER but not the sole case; also RDB cached-for-latency, NoSQL, live APIs landed via MATERIALIZED. VIRTUAL and SCAN strategies EXCLUDED (always-fresh / read-in-place; nothing to delta-refresh per REQ-826). KEY SIMPLIFICATION: PROBE == DELTA for monotonic entries. For a monotonic-cursor entry the DELTA QUERY IS THE FRESHNESS EVALUATION. Run the delta query: non-empty result ⇒ changed ⇒ apply the rows; empty ⇒ fresh ⇒ no-op. No separate watermark/probe query. No scalar_path. DEFINITION: ONE authored query (delta_query) — authored, source-native, fully-formed filter with two PLACEHOLDERS that Provisa substitutes: (a) watermark placeholder ($wm bind variable) bound to the stored CURSOR VALUE, (b) fields placeholder ({{fields}}) injected from the table's registered selection set. Provisa substitutes only; never parses the filter. CURSOR FIELD IMPLICIT: the cursor field is the field $wm filters on (single query, single field); after applying delta rows, the stored cursor advances to max(cursor-field) over returned rows (delta_query ordered by cursor field). Same-field alignment therefore trivially guaranteed; no separate field declaration required. Cursor and monotonicity are the registrant's responsibility; Provisa does no dedup or boundary-inclusivity logic. SOURCE-TYPE-SPECIFIC AUTHORING AND EXECUTION: delta_query is AUTHORED IN THE SOURCE'S NATIVE QUERY LANGUAGE, syntax and filter/cursor binding PER-SOURCE-TYPE: GraphQL where: {field: {_gt: $wm}} + {{fields}}; OpenAPI query-param ?updated_since=$wm + selection headers; gRPC request-message filter + $wm + selection; SQL WHERE field > $wm + projection. Only $wm/{{fields}} placeholder substitution and cursor-advance logic are uniform across types. DELTA EXECUTION ROUTES THROUGH THE ADAPTER'S NATIVE CALLER, NOT THE FEDERATION ENGINE: GraphQL delta POSTs the native query to the source (predicate pushed down AT source) and lands rows into materialization_store; does NOT execute as Trino WHERE field > x scan over the federated relation. Native-pushdown path is the efficiency win and reason delta exists rather than relying on Trino watermark_column poll (which re-scans with no pushdown). Per source type: (1) delta template = native query syntax + filter/cursor binding (author-supplied, source-specific); (2) delta execution = routes through source type's existing native caller (GraphQL/HTTP/gRPC/SQL), bypassing federation engine. Pattern mirrors per-adapter mutation classification/suggestion (REQ-869/871): uniform contract with per-source-type native implementation. APPLY: upsert returned rows on the replica's ALREADY-REGISTERED PRIMARY KEY (entity identity from table registration) to replace prior row state; degrades to plain insert for append-only/immutable sources. REPLICA-ONLY: VIEW/mv role is delta-ineligible—REQ-844 Iceberg overwrite-snapshot semantics (one clean snapshot per refresh) for time-travel/provenance/ bitemporal (REQ-372/862) would be violated by upsert. Mutable relational substrate only. RELATIONSHIP TO FRESHNESS: Freshness (REQ-855/856) is pure stale/fresh DECISION with no refresh semantics (REQ-856). Delta is REFRESH ACTION under REQ-826's MATERIALIZED reload — incremental sibling of full re-pull. But for monotonic entries delta query serves as freshness evaluation, so REQ-855's opaque probe is NOT also run: PROBE (opaque token) and DELTA (monotonic cursor) are MUTUALLY EXCLUSIVE PER ENTRY, selected by token type. Opaque-token entries ⇒ REQ-855 probe + full re-pull, no delta. Monotonic entries ⇒ delta-as-probe, no separate probe. Validation gates: (a) delta_query contains both $wm and {{fields}} placeholders; (b) parses; (c) $wm bind-var type is orderable/monotonic-compatible; (d) injected selection valid against row type. DONE (2026-07): the UNIFORM part is implemented — provisa/federation/delta.py: delta_applies gates delta to MATERIALIZED (VIRTUAL/SCAN excluded); render_delta_fields substitutes {{fields}} textually (Provisa never parses the source-native filter) leaving $wm for native binding; has_wm_placeholder is the $wm validation gate; delta_is_fresh is PROBE==DELTA (empty result ⇒ fresh/no-op, non-empty ⇒ changed/apply); advance_cursor advances the stored cursor to max(cursor-field) over the returned rows (empty keeps it). REMAINING (kept in-progress): the per-source-type delta authoring + native execution (GraphQL/HTTP/gRPC/SQL callers, predicate pushed down at source), keyed-upsert apply on the replica, and the delta/opaque-probe mutual-exclusivity wiring with REQ-855.

Use case: For REPLICAS (any MATERIALIZED-strategy dataset whose federation strategy lands data), full re-pull on every refresh is expensive and unnecessary. Delta fetch via monotonic watermark keeps replicas fresh with zero lag (REQ-855 probe) without repeated full upstream fetches. Primary use: remote-schema sources (graphql_remote/openapi/gRPC) avoid full-relation re-fetch and re-materialize. Also benefits RDB sources presently using watermark_column only for subscriptions (REQ-260) — they gain incremental replica refresh: keyed upsert instead of full re-pull, no full-CTAS + mutation-triggered staleness cycle. Strategy-agnostic scope avoids VIRTUAL (live) and SCAN (in-place read) per REQ-826. Replicas in mutable stores absorb deltas; views cannot.

Code: provisa/federation/delta.py, provisa/subscriptions/, provisa/materialization/, provisa/materialization_store/

Tests: tests/unit/test_delta.py, tests/integration/test_materialization_store_lifecycle_e2e.py

6. Execution, Routing, Caching & Performance¶

REQ-875 · Query Routing¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cheap-count route: a count()-shaped query over an UNMATERIALIZED source that exposes an EXACT native count (REQ-673 cardinality capability) is routed to the native count call instead of materializing the full dataset to count it. Parallels Route.CACHE (REQ-865) — a query shape that resolves to a first-class route bypassing the expensive path. Today SELECT count() over a cold API source pulls the entire dataset (endpoint.py _materialize_api_to_trino_cache) to count it; with an exact native count it becomes a single call. THREE GUARDS, all required. (1) SHAPE: a bare count() with no other projection and no WHERE the native call cannot honor; a filtered count the source cannot filter falls back to materialize. (2) EXACTNESS: fires only when the cardinality Estimate is exact; approximate statistics stay in the planner/human-hint lane and are never served as the answer to count(). (3) GOVERNANCE: the route MUST NOT bypass Stage-2 RLS/masking (REQ-263) — when an RLS predicate applies to the table for the requesting persona a native total count would over-count the visible subset, so the route is disabled and falls back to materialize+count. Fail-closed, the same discipline as cache isolation (REQ-866). DONE (2026-07): the three-guard route decision is implemented — provisa/federation/cardinality.py:: can_route_cheap_count(is_bare_count_star, estimate, rls_applies) fires only when the shape is a bare count(), the REQ-673 estimate is EXACT with a value, and NO RLS predicate applies to the persona; any guard failing returns False (fail-closed → materialize+count). REMAINING (kept in-progress): wiring it into the endpoint count() path so an exact-native-count source answers with one call instead of _materialize_api_to_trino_cache.

Use case: Turns SELECT count(*) over a cold remote/API table from a full-dataset HTTP pull into a single cheap call, while preserving exact-answer semantics and governance, so the volume question is answerable interactively without warming the cache.

Code: provisa/federation/cardinality.py

Tests: tests/unit/test_cardinality.py

11. Platform, Infrastructure & Delivery¶

REQ-876 · Deployment & Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Compose host-port parameterization for stack coexistence. Every published host port in docker-compose.core.yml (and the dev/observability overlays) MUST be expressed as ${VAR:-default} so the default preserves current behavior while a second stack can bind a different host port. This lets the dev (provisa), e2e (provisa-e2e), and integration (provisa-test) compose projects run simultaneously instead of colliding on 5432/8080/ 9000/6379/8480 — the failure mode observed when a second project's minio could not bind 9000, started detached from its network, and its minio-init could not resolve minio, hanging the stack at N-1/N. THREE parts: (1) parameterize the published ports in core.yml/dev.yml/observability.yml (left side only; container ports fixed). (2) each test harness project (tests/conftest.py integration, tests/e2e/conftest.py) sets its own offset port set (e.g. PG_PORT/TRINO_PORT/REDIS_PORT/MINIO_PORT/ZAYCHIK_PORT) before compose up. (3) the in-process app under test connects to the matching ports — TRINO_PORT/ZAYCHIK_PORT are already env-driven (app.py:1119/1160); the PROVISA_CONFIG fixture's Postgres host:port must become env-interpolated (or the harness must template it) so the app reaches the offset postgres. Isolation of the compose PROJECT (‑p) already exists; this closes the host-port dimension so lifecycle-independent stacks are also bind-independent.

Use case: A developer or CI can run the dev stack, the e2e stack, and an integration test run at the same time without tearing one down — each provisions its own services on its own host ports, so no run blocks another and no half-attached container corrupts a shared network.

Code: docker-compose.dev.yml, docker-compose.observability.yml, docker-compose.demo.yml, docker-compose.dev-install.yml, docker-compose.mvmvp.yml

Tests: tests/unit/test_infra_requirements.py

5. Data Lineage & Provenance¶

REQ-877 · Row-Level Delta Capture¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Opt-in, per-table row-level delta capture for materialized views — the row-level companion to the column-level lineage trace (REQ-862). Each MV refresh can record which rows changed (insert / update / delete) since the prior refresh, so downstream consumers get change-data-capture on a derived dataset and audit gets row-granular provenance. NATIVE-FIRST — hashing is the fallback, never the default: (1) an Iceberg-target MV derives deltas from a snapshot-to-snapshot changelog scan (no hashing, no retained state); (2) a watermarked source (REQ-260 watermark_column) derives the delta from the watermark-bounded slice — this IS the output-side mirror of REQ-874 delta-fetch; (3) ONLY an RDB target with no watermark falls back to full-MV row hashing. QUERY-GENERATED — MAKE THE DB THE COMPUTE: the row hash and the diff are pushed down as SQL (hash via the engine's md5/sha over the canonical row projection; the delta via a set operation — EXCEPT / anti-join — between the current and the previous hash relation, the previous snapshot persisted as a relation). Provisa issues the query and lands the result (INSERT…SELECT / CTAS in the engine); it MUST NOT SELECT rows out and hash/diff them in Python. The delta is (key, change_type, old_hash, new_hash). COLUMN EXCLUSION: the opt-in specifies which columns are EXCLUDED from the row hash (e.g. volatile audit/updated_at columns), so a change to an ignored column is not reported as a row change — the hash is computed over the included projection only. The hash path is the expensive case and is strictly OPT-IN per table (default off). A SYSTEM-WIDE config sets the row-count ceiling for the hash path — max_row_hashing (global default in provisa.yaml, per-table override) — above which hash-based delta capture is SKIPPED and logged (mirrors the existing MV materialization max_rows guard); never sampled (sampling would make an audit delta lossy and untrustworthy). Deltas land in a SEPARATE append-only ledger keyed by (mv_id, definition_version, refresh_trace_id) — the same store-independent principle as mv_refresh_log (REQ-862), never stamped on the target table. NON-BLOCKING: delta capture — especially the full-MV hash path — MUST run off the refresh critical path in a background thread/task; the refresh completes and marks fresh without waiting on delta computation, and a slow or failed delta capture never delays or fails the refresh (best-effort provenance, logged on failure).

Use case: Gives audit and incremental-consumption workflows row-level "what changed between refresh N and N+1" without a bespoke CDC service, while bounding cost: capable targets/sources (Iceberg, watermarked) get deltas cheaply from the engine, and the expensive full-MV hash path is opt-in and ceiling-guarded so very large MVs never silently pay for it.

Code: provisa/mv/delta.py, provisa/mv/refresh.py, provisa/mv/models.py, provisa/core/schema_org.py

Tests: tests/unit/test_mv_delta_capture.py

REQ-878 · Point-In-Time Reconstruction¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Point-in-time MV reconstruction from the delta ledger (REQ-877). An append-only delta ledger is a temporal substrate: folding change events up to a refresh version N (or reversing them from the live table back to N) reconstructs the view as of N — time-travel for targets that lack it natively. TWO OPT-IN TIERS by fidelity, matching what the ledger stores: (1) HASH-DELTA (REQ-877 default) reconstructs MEMBERSHIP and change history only — which row keys existed as of N, and that/when a row changed — because a hash proves change but does not invert to old values; (2) VALUE-DELTA stores the changed row VALUES, enabling FULL CONTENT reconstruction as of N (true time-travel for an RDB target). Value-delta is a second, heavier opt-in — it reimplements what Iceberg gives natively (checkpoint snapshots + a changelog), so Iceberg targets get content time-travel for free and value-delta buys it for RDB targets at a storage cost. CHECKPOINT-BOUNDED REPLAY: do not fold all history — persist a periodic full snapshot plus deltas between; reconstruct = nearest checkpoint <= N, then apply deltas up to N (Iceberg's model). RECONSTRUCTION IS A QUERY — MAKE THE DB THE COMPUTE (same discipline as REQ-877): as-of-N is a windowed greatest-per-key query over the ledger (last event with version <= N per key, excluding deletes), executed in the engine, never a Python replay. Non-blocking and system-config bounded like REQ-877.

Use case: Lets a user query "the view as of refresh N / time T" for MVs whose target is not Iceberg, without a bespoke temporal store — membership-and-audit reconstruction cheaply from hashes, or full content reconstruction when the value-delta tier is opted in, with replay cost bounded by checkpoints and the whole reconstruction expressed as a single ledger query.

Code: provisa/mv/delta.py, provisa/mv/refresh.py, provisa/core/schema_org.py

Tests: tests/unit/test_mv_delta_capture.py

4. Source Connectors¶

REQ-879 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: structural

Cross-instance MV refresh coordination for a load-balanced fleet sharing one materialization store. Today the live refresh state is per-instance in-memory (MVRegistry._mvs; mark_refreshing flips an in-process status), so instance A's REFRESHING is invisible to B and both refresh the same MV — redundant compute plus concurrent writers racing on the same relation. FIX: promote the materialized_views catalog into the AUTHORITATIVE SHARED refresh-state and drive the refresh loop off it (not the per-instance registry). Add columns: writer, lease_until, materialized_definition_version, materialized_input_version, snapshot_id. PROTOCOL: (1) ATOMIC CLAIM — a single conditional UPDATE that BOTH dedups by version (skip when materialized_input_version == target, the REQ-862 stamp is the already-written key) AND excludes concurrent writers (skip when status=refreshing AND lease_until>now). 0 rows = skip; 1 row = this instance owns the refresh. This is the per-MV election — DECENTRALIZED, no global leader; a heartbeat renews lease_until during a long refresh, and lease expiry reclaims after a crash. (2) FENCED COMMIT — finalize (set materialized_version/status/ snapshot_id) only WHERE writer=me AND lease_until>=now; 0 rows means the lease was lost (slow/crashed-then-revived) so the result is DISCARDED — prevents a stale writer clobbering a newer refresh (fencing token). (3) SOURCE OF TRUTH — the store's committed snapshot (atomic commit, e.g. Iceberg) is authoritative for what is materialized; the catalog is coordination metadata reconciled to the store on lease expiry, so a crash mid-write leaves no partial data. NOT held pessimistic locks (SELECT FOR UPDATE over a long refresh holds a transaction open and a crash leaves a stuck lock) — leases + fencing. The catalog is the same structure that holds freshness (REQ-855) and the version/lineage stamps (REQ-862). CONSISTENCY KNOB: a per-MV consistency setting selects the tier — shared (default) engages this coordination catalog (one snapshot-consistent copy); distributed per-instance-materializes and does NOT engage it. The distributed tier is only eventually consistent under TWO preconditions: (a) a DETERMINISTIC view_sql — enforced at registration (reject now()/random()/uuid/unordered-LIMIT; provisa/mv/determinism.py); and (b) source QUIESCENCE — the source must stop changing long enough within a refresh cycle for all instances to materialize the same source state. A never-settling high-churn source never converges (each instance perpetually reflects a different snapshot), so the distributed tier is inappropriate for it — determinism is necessary but not sufficient. Quiescence is a workload property, not statically checkable, so it is an operational constraint on choosing distributed, whereas determinism is validated.

Use case: Lets an arbitrary number of control-plane/engine instances run behind a load balancer against one shared materialization store without redundant refreshes, write races, or a single-leader bottleneck — each instance races to claim due MVs, exactly one wins per MV, and crashes are handled by lease expiry and fenced commits.

Code: provisa/mv/registry.py, provisa/mv/refresh.py, provisa/mv/coordination.py, provisa/core/schema_org.py

Tests: tests/unit/test_mv_fleet_coordination.py

REQ-880 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: constraint

The materialized-view store (MV cache target for materialized relationships and views) must be configurable as a SQLAlchemy URI instead of pinned to a specific database type. Today it is hardcoded to the Trino catalog literal "postgresql", preventing the materialization backend from using any SQLAlchemy-supported database.

Use case: Supports flexible deployment scenarios where the materialization backend may be a dedicated data warehouse, distributed datastore, or cloud analytics system independent of the source connectors.

Code: provisa/api/app.py, provisa/mv/refresh.py, provisa/core/schema_org.py

Tests: tests/unit/test_federation_materialization.py

REQ-881 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Probe-based freshness gate on the MV REFRESH path, opt-in per MV — the refresh-time application of REQ-855's centralized freshness gate to the materialized-view role. MOTIVATION: relationship-based (join-pattern) MVs are expensive to rebuild — a full Trino CTAS / DELETE+INSERT of the join every cycle. Today the background refresh loop gates purely on TTL (MVRegistry.get_due_for_refresh compares now - last_refresh_at >= refresh_interval), so a due MV pays the full rebuild every interval EVEN WHEN NO SOURCE CHANGED. FIX: add a per-MV freshness_mode (ttl [default, current behavior] | probe | ttl_probe). For probe / ttl_probe, before the expensive materialization the refresh computes an opaque per-MV change token from the REQ-862 per-source input signals (Iceberg snapshot id / RDB watermark MAX, already gathered by provisa/mv/input_signals.py::gather_input_signals) and compares it by equality to the token stored at last refresh: unchanged → skip the rebuild, reset the TTL, keep the materialized rows and FRESH status (zero lag at probe cost); changed → rebuild and store the new token. ttl_probe probes only after the TTL floor elapses (caps probe frequency); probe re-checks every loop ignoring TTL. SAFETY: a token is usable only when EVERY source produces one (len(signals) == len(source_tables)); if any source is token-less the entry degrades to TTL and rebuilds normally — never skip on partial signal (REQ-855 None-degrades-to-TTL, REQ-847 capability-signal-not-silent-fallback). Opt-in is per MV so cheap MVs keep plain TTL and only expensive relationship MVs pay the probe. Mutual exclusivity with delta (REQ-874) and the shared coordination catalog (REQ-879) follow REQ-855: the same materialized_input_version stamp is the probe token and the dedup key.

Use case: Lets a steward gate an expensive relationship-based MV so its costly join is rebuilt only when a source actually changed, trading a cheap per-source probe (snapshot id / watermark) for skipping a full re-materialization every interval — zero-lag freshness where rebuild cost is high, plain TTL where it is not.

Code: provisa/mv/models.py, provisa/mv/registry.py, provisa/mv/refresh.py, provisa/mv/input_signals.py

Tests: tests/unit/test_mv_probe_freshness.py

REQ-882 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Wire the aggregate MV rewrite path into the live query execution path. Populate AggregateMVCatalog from MVRegistry on startup, invoke aggregate rewrite in the endpoint query path, and enforce filter subset-safety before rewriting. DONE (2026-07): (1) POPULATE — MVRegistry.register/unregister now sync the process aggregate-MV catalog (provisa/mv/aggregate_catalog.get_aggregate_catalog()), so every MV registered at startup is visible to rewriting; register() is a no-op for non-aggregate MVs. (2) INVOKE — the endpoint query path calls rewrite_if_aggregate_match after the join-MV rewriter (provisa/mv/rewriter.py); only when the join rewrite did not already fire. It extracts a single-table aggregate query via sqlglot (base table, columns inside aggregate functions, top-level AND filter fragments; queries with a JOIN, multiple tables, or no aggregate are left alone) and, on a covering MV, rewrites FROM to the MV target and sets sources to the MV catalog. (3) SUBSET-SAFETY — find_aggregate_mv now enforces it (the prior code skipped the check its docstring promised): a new MVDefinition.filters records the predicates the aggregate MV was pre-computed with, and an MV is usable only when set(mv.filters) is a SUBSET of the query's filters (an unfiltered MV is always safe); the query's remaining filters (query − MV) are re-applied on the MV at rewrite so no rows the query wants are silently dropped.

Use case: Enables query acceleration via pre-computed aggregates stored in materialized views, allowing queries requesting SUM/AVG/etc over MV-backed tables to read pre-aggregated results instead of materializing raw rows and computing aggregates at query time.

Code: provisa/mv/aggregate_catalog.py, provisa/mv/registry.py, provisa/mv/rewriter.py, provisa/mv/models.py, provisa/api/data/endpoint.py

Tests: tests/unit/test_aggregate_mv_catalog.py

REQ-883 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa's pgwire server must satisfy the capability set that DuckDB's postgres extension exercises when attaching Provisa via ATTACH ... AS x (TYPE postgres), enabling DuckDB federation engines and external DuckDB clients to mount a governed Provisa instance with RLS/masking intact. Concrete capabilities enumerated: (1) libpq connection + auth handshake; (2) catalog introspection queries for schemas/tables/ columns/types/PKs over pg_namespace, pg_class, pg_attribute, pg_type, pg_index, pg_constraint with correct atttypmod and pg_type OIDs; (3) bulk read via COPY (SELECT ...) TO STDOUT (FORMAT binary) with real PostgreSQL binary COPY wire encoding per column type; (4) transaction-control statements (BEGIN/COMMIT/ROLLBACK, incl. SET TRANSACTION READ ONLY / REPEATABLE READ) accepted as no-ops — NO real snapshot isolation is provided or needed; (5) single-stream scan ONLY. Snapshot export, server-side cursors, and ctid-range parallelism are explicit NON-GOALS, validated empirically against DuckDB in the sibling pgwire-calcite project (PGW-004 / PGW-023): DuckDB's ctid parallelism splits a table by physical page range, but Provisa exposes no physical row address, so honoring a ctid range would force a ROW_NUMBER() OVER (ORDER BY ...) partition that scans+sorts the whole table per stream — strictly worse than one stream. The server reports relpages/reltuples ~ 0 in pg_class so DuckDB never elects parallelism and reads a single correct binary-COPY stream. Server version >= 14 (DuckDB's gate) is already met (server_version '14.0.provisa'). DONE (2026-07) capability (3) — BINARY COPY output: COPY ... TO STDOUT (FORMAT binary) now emits a real PostgreSQL binary COPY stream (provisa/pgwire/copy_handler.py): the 11-byte PGCOPY signature + flags + header-extension, per-row int16 field count with length-prefixed per-field bodies (int32 -1 for NULL), and the int16 -1 trailer. Per-column encoding is selected from the column's advertised type (Arrow schema on the Trino Flight bulk path; DuckDB type strings via QueryResult.column_types on the direct path) — bool/int2/int4/int8/float4/float8/bytea encode to their PG binary representations, everything else to UTF-8 text matching the text OID — and the CopyOutResponse overall-format flag is set to 1 (binary). Capabilities (1) libpq handshake and (2) catalog introspection are served by the existing pgwire server + catalog; (4) transaction-control statements are accepted as no-ops by the txn-tag path. DONE (2026-07) the COPY parser now accepts DuckDB's actual (FORMAT "binary") syntax — optional WITH and a quoted format word — which the older regex rejected (all three COPY regexes in provisa/pgwire/copy_handler.py); and a live end-to-end DuckDB 1.5.3 ATTACH ... TYPE postgres test drives all five capabilities against a running server (tests/integration/test_duckdb_attach_pgwire.py): ATTACH + catalog discovery, binary-COPY row/type/NULL round-trip, transaction no-ops via postgres_execute, and relpages=0 single-stream. Capability (5) is confirmed a non-goal, not remaining work. COMPLETE.

Use case: Lets DuckDB (as the REQ-825 federation engine or any external DuckDB/BI client) mount a governed Provisa instance via ATTACH ... TYPE postgres and query it with RLS/masking intact, completing the develop-on-DuckDB story where DuckDB reaches Provisa itself the same way it reaches any Postgres.

Code: provisa/pgwire/server.py, provisa/pgwire/copy_handler.py, provisa/pgwire/catalog.py, provisa/pgwire/_pipeline.py

Tests: tests/unit/pgwire/test_copy_binary.py, tests/integration/test_duckdb_attach_pgwire.py

3. Governance & Observability¶

REQ-884 · Observability¶

Status: ✅ complete · Priority: MUST · Type: behavioral

All Provisa internal logs—starting with the query audit log (REQ-074) and extending to any other operational/observability logs—must be materialized and exposed as first-class registered tables within a dedicated "ops" domain in Provisa's federated catalog.

Use case: Makes operational telemetry queryable through the governed pipeline (pgwire/SQL/GraphQL/Cypher) subject to role + domain access control, instead of only via the Python export path (export_audit_log) or raw control-plane JDBC access which bypass governance. Enables self-observability/dogfooding where an MCP/AI planner can read observability signals through the same governed surface as business data. Closes the gap that query_audit_log is not currently exposed in the federated pgwire catalog.

Code: provisa/api/_meta_views.py, provisa/api/startup_seed.py, provisa/api/app_loaders.py, provisa/audit/query_log.py

Tests: tests/unit/test_ops_domain.py

5. Query Languages, Compilation & Operations¶

REQ-885 · Extensible Functions¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Generalize tracked functions (REQ-205–208: source-resident stored procedures exposed as GraphQL mutations) to support multiple implementation kinds: source procedure (existing), script (local subprocess), http, grpc, and optionally python (in-process). All transports now functional: gRPC uses proto-less generic JSON unary bridge (_grpc_call in function_dispatch.py) via grpc.aio insecure/secure channels, with method path normalization and tls/timeout_s binding keys honored. Implementation kind becomes a selectable dimension at registration; addressing (catalog name) is decoupled from binding (transport+location, swappable). Provisa-hosted functions take relation arguments (kinds: table_ref=lazy, result_set=eager materialized Arrow, column_value=scalar row-wise) and return Arrow buffers materialized as relations, governed at the read-time boundary. Identity model (DEFINER/admin vs INVOKER/user) selected by materialize flag: true ⇒ admin/output-governed; false ⇒ user/input-governed. Composition via view_sql + materialize; no new MV subsystem—reuse MV refresh (REQ-879), CTAS sink, governance, lineage primitives.

Use case: Enables diverse computation platforms (hosted scripts, external services, Python libraries) to act as governed data transformations, returning relations (not just scalar functions). Maintains config-as-truthful-model property with URI addressing and spec-driven discovery (OpenAPI/proto/ script-manifest). Preserves existing source-procedure tracked-function contract while extending to Provisa-hosted and external implementations.

Code: provisa/executor/function_dispatch.py, provisa/api/data/action_exec.py, provisa/api/app.py, provisa/core/models.py, provisa/core/repositories/function.py, provisa/core/schema_org.py, provisa/api/admin/actions_router.py, provisa-ui/src/pages/commands/CommandFormFields.tsx, provisa-ui/src/pages/commands/CommandsPage.tsx, provisa-ui/src/pages/commands/types.ts, provisa-ui/src/pages/commands/api/actions.ts

Tests: tests/unit/test_extensible_functions.py

3. Governance & Observability¶

REQ-886 · Observability¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Engine-emitted I/O tracing for UDF/transformer invocations must be mandatory and non-bypassable, recording UDF name, transport type (script/http/grpc/sql/python), identity context (admin/definer vs user/invoker), declared input refs, output cardinality/size, duration_ms, and status. A trace/correlation ID must be stamped into the UDF's minted pgwire session so audit rows join to the invocation trace for precise lineage reconstruction. UDF-emitted tracing (via W3C traceparent context propagation) is encouraged but optional, allowing UDF interiors and egress calls to nest under the engine-side invocation span.

Use case: Establishes a non-bypassable observability floor (relied on for governance and audit) while encouraging voluntary UDF-emitted interior/egress tracing as the ceiling for debugging and performance analysis. Enables precise lineage reconstruction by correlating UDF invocation traces with pgwire data-access audit rows.

Code: provisa/otel_compat.py, provisa/executor/function_dispatch.py

Tests: tests/unit/test_udf_tracing.py

4. Source Connectors¶

REQ-887 · RDBMS Stored Procedure Auto-Discovery¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Extend database source introspection to auto-discover stored procedures and routines (via information_schema.routines, pg_proc, or vendor equivalents) and auto-register them, rather than requiring hand-registration. Classify discovered procedures as read-returning (registered as parameterized relations; proc arguments become query arguments) or side-effecting (registered as mutations/tracked functions). Discovered proc results flow through Stage-2 governance identically to tables.

Use case: Meet legacy enterprise estates (Oracle/SQL Server shops with large proc inventories) at their current state, ensuring nothing in the estate is invisible to the catalog. Mirrors the OpenAPI discovery pattern (REQ-316/317): introspection + auto-registration avoiding hand-registration. Value scales with proc inventory; low build cost because it reuses existing introspection and OpenAPI-style registration.

Code: provisa/api/admin/introspect.py, provisa/discovery/catalog_cache.py

Tests: tests/unit/test_stored_proc_discovery.py

6. Execution, Routing, Caching & Performance¶

REQ-888 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: constraint

Federation engine contract is wired into the live query path via EngineRuntime, bound mandatorily at AppState construction as state.federation_engine. Engine-specific consumer transports are capability-gated via EngineCapability enum (ROWS/ARROW/ARROW_STREAM); missing capabilities fail closed via UnsupportedCapabilityError with no fallback.

Use case: Mandatory EngineRuntime binding and capability-gating defines the defensible abstraction boundary for platform-specific escape hatches. Named, advertised capabilities prevent arbitrary injected connections and enforce consistent routing (DIRECT → native driver, ENGINE → federated execution) across all consumer paths (pgwire, GraphQL, Arrow Flight, COPY handler, NL executor).

Code: provisa/federation/runtime.py, provisa/api/graphql/, provisa/bolt/, provisa/arrow_flight/, provisa/query_helpers.py

Tests: tests/integration/test_engine_runtime_binding.py, tests/unit/test_engine_capabilities.py

8. Deployment & Infrastructure¶

REQ-889 · Deployment Tier Architecture¶

Status: ✅ complete · Priority: MUST · Type: constraint

The metadata/config/roles store MUST be the embedded single source of truth in every deployment tier. The embedded tier (SQLite or DuckDB) is the metadata home; container tier adds Trino + observability strictly as COMPUTE, never as metadata home. Tier changes MUST be additive and reversible: Trino can be toggled on/off without migrating the catalog.

Use case: Tier changes must be reversible without catalog migrations. If Postgres becomes the metadata home in a container tier, it reintroduces a container dependency and creates a migration problem on every tier toggle, breaking the design invariant that compute is pluggable while metadata stays embedded.

Code: provisa/core/db.py, provisa/core/database.py, provisa/core/desktop_profile.py

Tests: tests/unit/test_arch_invariants.py

4. Source Connectors¶

REQ-890 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Pgwire auth must be a pluggable provider interface selected at launch (trust | local | oidc), superseding today's fixed MD5/cleartext wiring while keeping trust/cleartext as the baseline. (a) A local-accounts provider stores SCRAM-SHA-256 verifiers at rest (RFC 5802 — random salt, iterations, StoredKey/ServerKey; no cleartext stored or required on the wire, secure without TLS) with a CLI to add/remove/list users, and offers SCRAM-SHA-256 as a wire auth mechanism. (b) An external-token provider accepts an OIDC ID token (JWT) presented as the password and verifies it against the issuer's JWKS (signature, exp, aud, iss); issuer-generic (Firebase/Auth0/Google/Entra) via configured issuer URL + audience.

Use case: Pgwire connections to Provisa today use MD5 or cleartext auth, neither of which meet enterprise security standards. SCRAM-SHA-256 (salted, iterated, server-side verification) secures local accounts without requiring TLS. OIDC integration allows enterprises to reuse their identity provider, eliminating separate credential management for database access.

Code: provisa/pgwire/server.py

Tests: tests/unit/test_pgwire_requirements.py, tests/unit/test_auth_providers.py

REQ-891 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Pgwire per-role authorization maps an authenticated account to Calcite/source schema+table visibility, enforced in BOTH discovery and execution by construction: the catalog intercept (provisa/pgwire/catalog.py) filters pg_catalog/information_schema discovery to the role's allowed objects, and query execution rejects access to objects outside the role's grant. Complements existing RLS/masking (which govern rows/columns) with object-level visibility over the wire.

Use case: Pgwire clients (DBeaver, DataGrip, DuckDB) discover database schemas via pg_catalog and information_schema queries. Without object-level authorization filtering, a Pgwire client will display all accessible Calcite schemas/tables to the user even if the user's role should not see them. Enforcing role visibility in both discovery and execution prevents information leakage and aligns object access with existing RLS/masking governance.

Code: provisa/pgwire/catalog.py, provisa/pgwire/_pipeline.py

Tests: tests/unit/pgwire/test_catalog.py

REQ-892 · Connector Abstraction¶

Status: ✅ complete · Priority: MAY · Type: structural

Pgwire may present popular PostgreSQL extension surfaces backed by the federation engine, as a pluggable opt-in-per-deployment mechanism: each surface (a) advertises itself in pg_extension so clients see it "installed", (b) declares its types/OIDs in the single normalization module, (c) maps its operators/functions to an engine function or rejects them explicitly (convert-or-reject-loudly — a surface must NOT claim a capability, e.g. index acceleration, it does not implement). Concrete surfaces: pgvector-compatible (vector type + distance operators <-> L2, <=> cosine, <#> inner product, including ORDER BY emb <-> $1 similarity search, NOT advertising ivfflat/hnsw index acceleration); PostGIS-subset (geometry/geography + ST_Distance/ST_Contains/ST_DWithin/ST_Intersects/&&); JSON operators (-> ->> #> #>> mapped to JSON_VALUE/JSON_QUERY); and compatibility functions (pg_trgm similarity()/%, gen_random_uuid(), pgcrypto digest()).

Use case: Pgwire clients assume PostgreSQL extensions are available (pgvector for ML/embedding workloads, PostGIS for geospatial, JSON operators for document processing). Making these surfaces available over the federation engine eliminates the need to add specialized backends or custom UDFs; a single Provisa pgwire endpoint can serve diverse analytics workloads (ML, geospatial, document) via compatibility operators mapped to federation engine functions.

Code: provisa/pgwire/ext_surfaces.py, provisa/pgwire/catalog_data.py, provisa/pgwire/catalog.py, provisa/pgwire/catalog_populate.py, provisa/pgwire/_pipeline.py

Tests: tests/unit/test_pgwire_ext_surfaces.py

REQ-893 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: MAY · Type: structural

PostgreSQL can act as a pluggable FederationEngine (DriverClass.PARTIAL) via postgres_fdw, enabling Postgres servers to attach foreign tables from other Postgres servers (ATTACH mechanism — no data movement, references in place). FDW types (postgres_fdw, mysql_fdw, file_fdw, mongo_fdw, tds_fdw, etc.) are provisioned at install time via CREATE EXTENSION; per-source connections are created at runtime via CREATE SERVER + CREATE USER MAPPING + IMPORT FOREIGN SCHEMA / CREATE FOREIGN TABLE. This mirrors Trino's architecture: connector types bundled at provision, catalogs dynamically created at runtime (REQ-251).

Use case: Enables Tier 1 deployment: a single PostgreSQL instance serves as both the metadata home and the compute engine for PG-to-PG and cross-FDW federation, eliminating the need for Trino until heterogeneous/broad federation is required. Maintains containerless deployment spirit and completes the multi-tier deployment story. Queries pass through the semantic→IR layer (RLS/masking applied before transpile), ensuring Postgres engine passes governance-parity conformance suite (REQ-827) by construction.

Code: provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/unit/test_postgres_federation_engine.py, tests/integration/test_postgres_federation_engine_e2e.py, tests/integration/test_duckdb_federation_engine_e2e.py

8. Deployment & Infrastructure¶

REQ-894 · Deployment Tier Architecture¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

DuckDB federation engine is a DESKTOP/EDGE/DEV tier embedded federator, not a production multi-user backend. DuckDB is single-process with no server-side connection pooling and no HA; one Provisa process owns one DuckDB file and serves as the multi-user multiplexer (concurrent readers cheap via MVCC snapshots; writers/materialization serialize under Provisa coordination). The "multiple Provisas behind a load balancer with DuckDB" pattern works only for fully independent/partitioned workloads because DuckDB provides no shared state and is single-writer.

Use case: DuckDB is recommended for zero-install single-node desktop/edge/dev deployments with one Provisa instance per DuckDB file. Production multi-user federation at scale requires either Trino (MPP, distributes across a worker cluster) or PostgreSQL with postgres_fdw (single-node shared backend, operational hardening). The tier boundary is SCALE (single-node vs MPP), not just federation breadth.

Code: provisa/federation/engine.py

Tests: tests/unit/test_arch_invariants.py

4. Source Connectors¶

REQ-895 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: MAY · Type: structural

RDBMS-as-single-node-federation-engine pattern generalizes across DB vendors via native foreign-data mechanisms (SQL/MED or equivalent). Faithful SQL/MED: PostgreSQL (most complete, postgres_fdw and ecosystem FDWs) and IBM Db2 (federation via CREATE NICKNAME). Equivalent proprietary mechanisms: Oracle (database links + Heterogeneous Gateways), SQL Server (linked servers + PolyBase external tables), MySQL/MariaDB (FEDERATED / MariaDB CONNECT engine). All are single-node ATTACH-mechanism connectors and belong to the same federation-engine tier as PG/DuckDB; Trino remains the only MPP/broad federator.

Use case: Pluggable "warehouse/RDBMS-native" engine family: a customer's existing warehouse (Db2/Oracle/ SQL Server/PG) could itself be the federation engine via its native foreign-data mechanism, requiring no new infrastructure. These engines are OPTIONAL/MAY and operations-driven, not scale-driven. Unifies the single-node federator family under the SQL/MED abstraction and enables ecosystem partnerships (e.g., Db2, Oracle, SQL Server connectors as plugins).

Code: —

Tests: —

0. Architecture & Design Principles¶

REQ-896 · Federation Architecture¶

Status: ✅ complete · Priority: MUST · Type: structural

Provisa's defensible value is the GOVERNED-FEDERATION MODEL and consistent governance philosophy, not any compute engine or infrastructure. Federation engines (Trino, DuckDB, Postgres, Snowflake, warehouse-native) are interchangeable commodity COMPUTE substrates; governance (RLS, column masking, relationship-join enforcement) is applied in the semantic→IR layer BEFORE any engine sees the query, ensuring governance-parity across all engines regardless of architecture.

Use case: Engine independence allows Provisa to swap compute substrates (Trino ↔ DuckDB ↔ Postgres) with the entire governed federation model unchanged. The connector, CatalogEntry, materialization-store abstractions and single-node→MPP tier graduation remain engine-agnostic; governance invariants hold across deployment tiers and engine boundaries.

Code: provisa/compiler/stage2.py, provisa/pgwire/_pipeline.py

Tests: tests/unit/test_arch_invariants.py

REQ-897 · Federation Architecture¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Federation engines are declared-capability objects; capabilities are planner inputs. Each FederationEngine carries a set of DECLARED capability traits used to make routing/planning/tier decisions. Established traits: driver_class (reach: BROAD/PARTIAL/SELF_ONLY) and mpp (scale-out; orthogonal to reach). Candidate additional traits: file_native, pooled, transactional, streaming (Arrow transport modes), and connector-level pushdown (predicate/join/aggregate).

Use case: Consolidating engine capabilities into one coherent descriptor enables planner decisions such as: push a large federated join to an MPP engine; land API/remote sources into a pooled/transactional store; prefer a file_native engine when sources are files; gate single-node→MPP tier graduation. Capabilities become planner inputs rather than being inferred from incidental plumbing (e.g., connector count or driver class alone).

Code: provisa/federation/engine.py, provisa/federation/connector_base.py, provisa/federation/plan.py, provisa/transpiler/router.py

Tests: tests/unit/test_engine_capability_traits.py

3. Federation Engines & Data Sources¶

REQ-898 · PostgreSQL Deployment¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Curated, prebuilt catalog of PostgreSQL extensions (sqlite_fdw, parquet_fdw/parquet_s3_fdw, pg_lake, pg_duckdb, pg_analytics) indexed by pinned PostgreSQL major version and target OS/platform (manylinux, macOS, Windows), layered onto the pip-installed pgserver runtime to avoid runtime build dependencies.

Use case: Prebuilt extension artifacts eliminate runtime compilation and external build toolchains, preserving pgserver's "no root, pip install" deployment model. Enables sqlite_fdw and Iceberg-oriented FDWs (pg_lake, pg_duckdb, pg_analytics) to work out-of-box with Provisa's Postgres federation engine without requiring libsqlite3, libarrow/libparquet, or DuckDB compilation by the end-user.

Code: provisa/pg_extensions/catalog.py, config/pg_extension_catalog.yaml

Tests: tests/unit/test_pg_extension_catalog.py

4. Source Connectors¶

REQ-899 · Federation Engine Abstraction¶

Status: ✅ complete · Priority: MAY · Type: structural

Catalog of DuckDB extensions eligible to back a federation-source connector, each providing either a relation-producing table function usable as CREATE VIEW ... SELECT * FROM fn(...), or an ATTACH catalog of tables. Only these two mechanisms qualify; scalar/index-only extensions are excluded.

Use case: Enables connector authors to identify which DuckDB extensions can materially contribute federated sources via scanner table-function views or ATTACH catalogs, excluding pure utility/scalar/index extensions that cannot produce relations.

Code: provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/unit/test_duckdb_community_connectors.py

3. Federation Engines & Data Sources¶

REQ-900 · PostgreSQL Deployment¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Postgres federation engine's curated v1 connector set: prebuilt FDW/extension bindings for csv, sqlite, sqlserver, oracle, duckdb, and parquet/iceberg/delta_lake (via pg_duckdb), indexed by (pg_major, platform, libc) triple and layered onto the embedded pgserver runtime. Catalog resolution fails closed: no artifact for a triple means the source is unreachable on that deployment (no silent fallback).

Use case: A deployed Postgres federation engine connects to CSV (file_fdw), SQLite (sqlite_fdw), SQL Server (tds_fdw), Oracle (oracle_fdw), and lakehouse sources (parquet/iceberg/delta via pg_duckdb) without runtime build or external toolchain dependencies. The fixed v1 set avoids sprawl; future connectors (mysql_fdw, mongo_fdw) or write-heavy operations route via separate engines or deferred tiers.

Code: provisa/federation/connector.py, provisa/federation/engine.py, docs/arch/fdw_catalog.md

Tests: tests/integration/test_postgres_federation_engine_e2e.py

REQ-901 · PostgreSQL Deployment¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

pg_duckdb v1.0.0 (bundles DuckDB v1.3.2) is the compatible build for embedded PostgreSQL 16.2; later pg_duckdb versions require backend symbols absent from PG16.2. Extension built from pg source against matching major version and layered into pgserver install via build script.

Use case: Pinned PostgreSQL 16.2 in pgserver 0.1.4 runtime requires pg_duckdb v1.0.0 to avoid symbol resolution failures at extension load. Build process (scripts/build_pg_duckdb.sh) compiles against PG16 source, and the resulting .dylib/.so is dropped alongside other core FDWs (file_fdw, postgres_fdw).

Code: scripts/build_pg_duckdb.sh, provisa/federation/engine.py

Tests: tests/integration/test_embedded_pg_duckdb_engine_e2e.py

REQ-902 · Query Compilation¶

Status: ✅ complete · Priority: MUST · Type: constraint

pg_duckdb's transparent execution path cannot emit PostgreSQL 16 nested-JSON syntax (JSON_OBJECT with colon separators); nested-relationship GraphQL federation through pg_duckdb requires a pg_duckdb-specific JSON emission in the transpiler to collapse JSON_OBJECT into flat json_build_object.

Use case: Governed federation of nested GraphQL pipelines (with JOIN-enforced relationships) through pg_duckdb. Today pg_duckdb engine handles flat federated joins and governance filtering; nested-relationship queries route to other engines until JSON output is adapted.

Code: provisa/transpiler/transpile.py, provisa/federation/pg_backend.py

Tests: tests/unit/test_transpile.py

REQ-903 · PostgreSQL Deployment¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Postgres federation engines validate connector availability and fail explicitly when sources require unavailable connectors. Sources whose connector is unavailable resolve to UnreachableSource (explicit error, never silent fallback). Superseded by REQ-904 (probe-based discovery with runtime functional validation).

Use case: Operators need clear diagnostics when sources are unreachable due to missing connectors, rather than silent downgrade to alternative engines.

Code: provisa/federation/engine.py

Tests: tests/unit/test_federation_engine.py

REQ-904 · PostgreSQL Deployment¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Federation connectors are self-describing prebuilt definitions; each has an async probe() reporting FUNCTIONAL truth (FDW/extension actually loads/works), not mere presence. build_pg_engine() registers all prebuilt Postgres connectors as candidates in precedence order (postgres_fdw; pg_duckdb_csv preferred over file_fdw; pg_duckdb_parquet; pg_duckdb_json). At startup, FederationEngine.discover(fetch, disabled=frozenset(keys)) probes each candidate and sets the active set: a connector struck by the override (key in disabled) is not probed; every other candidate is probed and only kept if available; per source_type the first available candidate wins (richer preferred, stock fallback). A source with no available connector is UnreachableSource (explicit, no silent fallback). The override config is now only a strike-list, not a presence declaration. Runtime probe correctly reports pg_duckdb unavailable until preloaded (shared_preload_libraries check), where a static flag would get it wrong.

Use case: Runtime discovery ensures only functionally available connectors are registered, avoiding silent failures from misconfigured deployments. The strike-list override provides fallback control without requiring static presence declarations.

Code: provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/unit/test_federation_engine.py, tests/integration/test_embedded_pg_duckdb_engine_e2e.py

REQ-905 · Query Execution¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cost-based promotion of VIRTUAL/SCAN sources to MATERIALIZED when the source connector cannot push down row-reducing operators (filter/aggregate) the query requires, and the estimated scan size exceeds the threshold (default 1,000,000 rows). Promoted sources delegate reduction to the engine store, amortized across the freshness TTL. UNKNOWN cardinality never promotes (fail-open).

Use case: When source connectors lack required row-reducing pushdown capability but scans are large, promoting to MATERIALIZED allows the engine store (with full pushdown) to handle reduction efficiency, amortizing the cost across the source's freshness window. Join-only gaps do not promote (non row-reducing).

Code: provisa/federation/promote.py, provisa/federation/strategy.py, provisa/federation/plan.py

Tests: tests/unit/test_promote.py

REQ-906 · Source Connectors¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Seven DuckDB community-extension federation connectors implemented: MSSQL, MongoDB, Snowflake, BigQuery, Firebird, Google Sheets (gsheets), and Airport. Each connector references an external source via ATTACH mechanism and includes a load-only startup probe that verifies the extension loads and registers its scanner/attach symbol without opening a live connection to a source.

Use case: Extends federation to support additional cloud and enterprise source types (MSSQL, MongoDB, Snowflake, BigQuery) plus specialty connectors (Firebird, Google Sheets, Arrow Flight via Airport), each verified at startup via extension probe to ensure engine capability is installed and reachable offline.

Code: provisa/federation/connector.py, provisa/federation/engine.py, provisa/federation/duckdb_extensions.py, provisa/core/models.py, scripts/stage_duckdb_extensions.py

Tests: tests/unit/test_duckdb_community_connectors.py

REQ-907 · Source Connectors¶

Status: ✅ complete · Priority: MUST · Type: behavioral

SqliteFdwConnector and MysqlFdwConnector enable the embedded/BYO Postgres federation engine to attach SQLite files and remote MySQL servers via sqlite_fdw and mysql_fdw, respectively. Each connector declares runtime_deps (tuple of shared libs tagged system/bundled/operator) documenting the packaging surface; probes verify functional availability at startup.

Use case: Federation of SQLite file-based data stores (in-place via sqlite_fdw; libsqlite3 system-provided) and remote MySQL databases (via mysql_fdw; libmysqlclient/mariadb-connector-c bundled) extends the Postgres v1 connector set beyond CSV/Parquet/Iceberg, enabling embedded deployments to query across multiple relational sources without data movement.

Code: provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/integration/test_embedded_pg_sqlite_fdw_e2e.py, tests/unit/test_fdw_connectors.py

REQ-908 · Source Connectors¶

Status: ✅ complete · Priority: MUST · Type: behavioral

PgDuckdbIcebergConnector reads Apache Iceberg tables in-place via pg_duckdb's iceberg_scan (DuckDB iceberg extension) inside the stock embedded Postgres. Extension is STATIC-linked into libduckdb (aws-sdk-cpp, avro-c, roaring) with no extra runtime dylib; build requires USE_MERGED_VCPKG_MANIFEST=1 and iceberg pinned to branch v1.3-ossivalis.

Use case: Federation of Apache Iceberg lakehouse tables stored on S3/GCS/Azure without data movement. Iceberg's time-travel and ACID guarantees are preserved through the query; governance predicates filter rows at the scan level before federation.

Code: provisa/federation/connector.py, provisa/federation/engine.py, scripts/build_pg_duckdb.sh

Tests: tests/integration/test_embedded_pg_duckdb_iceberg_e2e.py, tests/unit/test_pg_duckdb_iceberg_connector.py

4. Source Connectors¶

REQ-909 · Direct-Route Dialect Expansion¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Generic SQLAlchemy fallback driver provides direct-route write capability for any source type with a SQLAlchemy dialect, without requiring a bespoke async driver. Writable sources are the intersection of {has_driver} AND {sqlglot_write_dialect}, where has_driver now includes the SQLAlchemy fallback when no native driver exists.

Use case: Broadens the writable RDBMS set on the direct route without per-source driver implementations. Any SQLAlchemy dialect (whose DBAPI is installed) + sqlglot write dialect support enables writes to that source type without a native async driver.

Code: provisa/executor/writable.py, provisa/executor/drivers/sqlalchemy_driver.py, provisa/executor/drivers/registry.py, provisa/federation/connector.py, provisa/core/models.py, provisa/api/app.py

Tests: tests/unit/test_writable.py, tests/unit/test_drivers.py

REQ-910 · Multi-Route Write Dispatch¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Preference-ordered write-path resolution routes mutations across three tiers: NATIVE (bespoke async driver, direct), SQLALCHEMY (generic fallback driver, direct), and ENGINE (federation engine writes upstream via a write-capable connector). NATIVE and SQLALCHEMY require both a direct driver and SQLGlot write dialect; ENGINE requires only the connector's Capability.write. ENGINE is the only route that reaches sources with no direct driver and no SQLGlot dialect (e.g., NoSQL sources exposed as writable relations by their connector).

Use case: Enables mutation routing to any writable source regardless of per-source driver availability or SQLGlot dialect support. Maximizes write coverage by cascading through direct routes (native first, then SQLAlchemy) before falling back to engine-routed mutations through connectors.

Code: provisa/executor/writable.py, provisa/executor/drivers/registry.py

Tests: tests/unit/test_writable.py

REQ-911 · ClickHouse Federation Engine¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

ClickHouse is a selectable federation engine (PROVISA_ENGINE=clickhouse) alongside trino/pg/duckdb/sqlalchemy. It is a single-node-reach PARTIAL federator, MPP, native_store="clickhouse" that ATTACHes external sources via ClickHouse's native integration engines (Mechanism.ATTACH) rather than an FDW.

Use case: Federate PostgreSQL, MySQL, CSV, Parquet, and MongoDB sources as ClickHouse tables without data movement. PostgreSQL/MySQL sources mount as DATABASE engines exposing all remote tables; CSV/Parquet mount as TABLE engines (S3, URL, or File based on path scheme) with columns inferred by ClickHouse; MongoDB mounts as MongoDB engine with columns supplied from registry metadata.

Code: provisa/federation/engine.py, provisa/federation/connector.py, provisa/federation/clickhouse_backend.py, provisa/federation/clickhouse_runtime.py, provisa/transpiler/transpile.py

Tests: tests/integration/test_clickhouse_runtime_e2e.py, tests/unit/test_clickhouse_connectors.py

REQ-912 · ClickHouse Federation Engine¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The ClickHouse federation runtime supports three interchangeable execution backends selected by URL scheme at initialization: clickhouse:// (HTTP via clickhouse-connect, port 8123), clickhouse+native:// (native TCP via clickhouse-driver, port 9000), or chdb:// (embedded in-process engine, optional filesystem persistence). All backends expose identical SQL and integration engine surfaces.

Use case: Operators can choose between a remote ClickHouse server (HTTP or native protocol), in-process embedded ClickHouse, or persistent embedded store depending on deployment topology and latency requirements.

Code: provisa/federation/runtime.py

Tests: tests/unit/test_clickhouse_connectors.py, tests/integration/test_clickhouse_runtime_e2e.py

0. Architecture & Design Principles¶

REQ-913 · AST-Based Transforms¶

Status: ✅ complete · Priority: MUST · Type: structural

Every query transform — governance (RLS, masking, visibility, row-cap), correlated- subquery rewriting, and dialect lowering — MUST operate on a parsed SQL/relational AST, never on SQL text via regex or string substitution. Each surface language (GraphQL, Cypher, SQL) parses to an AST that is carried through governance and planning and serialized to a dialect string only at final emission; no stage may re-derive query structure (SELECT scope, WHERE injection point, alias binding, LIMIT) from text.

Use case: String/regex transforms silently mis-apply governance: a scalar subquery in the projection moves the masking boundary (PII leak), a non-t0 alias convention skips RLS qualification (scope-ambiguous predicate), and a parameterized LIMIT bypasses the row-cap ceiling. AST transforms make "governance applied" a structural property of the tree rather than a textual coincidence each frontend must keep satisfying.

Code: provisa/compiler/stage2.py, provisa/compiler/rls.py, provisa/compiler/mask_inject.py, provisa/transpiler/transpile.py, provisa/cypher/translator.py

Tests: tests/unit/test_ast_transform_invariants.py, tests/unit/test_ast_transforms.py, tests/integration/test_ast_transform_e2e.py

3. Governance & Observability¶

REQ-914 · Pipeline Stage Tracing¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Each query-pipeline boundary (govern.in, govern.rls, govern.mask, govern.out, and transpile.) emits an OpenTelemetry span event carrying the canonical SQL, so the transform sequence is inspectable per request and the pre/post-governance pair serves as a compliance receipt. SQL capture is OFF by default; PROVISA_TRACE_SQL=redacted blanks every literal to a placeholder before attaching, and =full attaches raw SQL for dev only. Telemetry never fails the request: a redaction parse error is surfaced as a span attribute. PROVISA_TRACE_AST=1 attaches the SQLGlot AST repr, redacted the same way (literals blanked), so it is safe to enable independently of the SQL mode.

Use case: Debugging special-case query transforms previously meant reading regex and guessing where text was mangled; per-stage canonical SQL turns that into a diff. The redaction default prevents literal values (RLS predicate values, PII) from leaking into traces in a governance product.

Code: provisa/observability/stage_trace.py, provisa/compiler/stage2.py, provisa/compiler/rls.py, provisa/compiler/mask_inject.py, provisa/transpiler/transpile.py

Tests: tests/unit/test_stage_trace.py

1. Configuration & Deployment¶

REQ-920 · Config Generation¶

Status: ✅ complete · Priority: MUST · Type: structural

The DDN-to-Provisa connector schema converter (ddn/mapper.py, _map_connectors) emits editable placeholder connection strings when an HML subgraph connector lacks a source-override. Placeholder format: localhost/5432/default/postgres/${env:DB_PASSWORD}. Intentional fallback: DDN is a config-generation tool whose output is reviewed before deployment, not a runtime system.

Use case: Configuration generation tools output templates for human review and edit; the placeholder guides the operator to fill in the correct connection URI with environment variable substitution.

Code: provisa/ddn/mapper.py

Tests: tests/unit/test_ddn.py

6. Materialized Views & Caching¶

REQ-921 · Promotion Coercion¶

Status: ✅ complete · Priority: MUST · Type: behavioral

An unmapped promotion target_type in API JSONB promotions (provisa/api_source/promotions.py, generate_promotion_ddl) yields an empty cast (no explicit JSONB→target coercion). Intentional: only types requiring explicit coercion are in _PG_CAST_MAP; unmapped types are already representable in JSONB without casting.

Use case: Promotion coercion maps only the types that need explicit PostgreSQL casts; types not requiring coercion do not need a cast and the absence signals correct representation.

Code: provisa/api_source/promotions.py

Tests: tests/unit/test_api_promotions.py

7. Connectors & Source Adapters¶

REQ-922 · Change Data Capture¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Debezium CDC provider (provisa/subscriptions/debezium_provider.py) uses a stable sentinel (datetime.min) for missing or unparseable ts_ms, mirroring REQ-343 (RSS provider sentinel). Intentional fallback: ts_ms is optional in Debezium envelopes (snapshot/tombstone records may omit it); using now() would advance the watermark and drop later real events.

Use case: Change-data-capture watermarks must monotonically advance only with real event timestamps, not clock time; sentinel values preserve event ordering when optional fields are absent.

Code: provisa/subscriptions/debezium_provider.py

Tests: tests/unit/test_debezium_provider.py, tests/integration/test_debezium_cdc.py

2. Query Execution & Planning¶

REQ-923 · Trino Introspection¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Trino column introspection (provisa/compiler/introspect.py, introspect_column_types / _fetch_with_startup_retry) retries with backoff when the coordinator reports SERVER_STARTING_UP, up to the ready timeout. Every other Trino error propagates. Intentional resilience: transient coordinator boot can last minutes; retry captures this operational reality without masking genuine errors.

Use case: Coordinator restarts are transient operational events; introspection must tolerate boot time without becoming a silent mask for genuine connection or permission errors.

Code: provisa/compiler/introspect.py

Tests: tests/unit/test_compiler_introspect.py, tests/integration/test_introspect.py

3. Source Registration & Data Modeling¶

REQ-924 · Source Watermarking¶

Status: ✅ complete · Priority: MUST · Type: structural

A source table has at most one watermark column, selected from existing columns only. Sources have no derived/synthetic columns; watermark value is a column name or none.

Use case: The watermark column enables incremental refresh and subscriptions; limiting to one existing column simplifies source configuration and avoids synthetic stamping.

Code: provisa/core/models.py, provisa/federation/residency.py

Tests: tests/unit/test_residency.py, tests/unit/test_watermark_validation.py

REQ-925 · Source Watermarking¶

Status: ✅ complete · Priority: MUST · Type: constraint

If a watermark column is set, it must be monotonic non-decreasing (timestamp or incrementing integer). The column type is validated at selection time; non-monotonic columns are rejected.

Use case: Incremental refresh and subscription reads rely on WHERE wm > cursor predicates; non-monotonic values would miss or duplicate rows, violating correctness.

Code: provisa/core/config_loader.py, provisa/core/ir_types.py

Tests: tests/unit/test_watermark_validation.py

9. Live Data & Events¶

REQ-926 · Change Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The watermark column gates refresh mode and subscribability: watermark set → APPEND refresh (incremental, WHERE wm > cursor) and subscriptions allowed; watermark unset → REPLACE refresh (full DELETE+INSERT) and subscriptions forbidden.

Use case: Subscribability and incremental refresh are coupled; only watermarked sources can reliably serve incremental reads and change notifications.

Code: provisa/core/change_signal.py, provisa/federation/residency.py, provisa/events/injector.py

Tests: tests/unit/test_change_signal.py, tests/unit/test_residency.py, tests/unit/test_injector.py

REQ-927 · Change Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: constraint

Subscriptions require a watermark column. No watermark = no subscriptions and no incremental refresh.

Use case: Subscriptions deliver rows WHERE wm > cursor; without a watermark, there is no monotonic reference point for incremental delivery.

Code: provisa/core/change_signal.py, provisa/core/config_loader.py, provisa/live/reconcile.py

Tests: tests/unit/test_change_signal.py, tests/unit/test_injector.py

REQ-928 · Change Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: constraint

Delivered subscription grain is INSERTS and UPDATES only; HARD DELETES are not supported. An append + (wm > cursor) model cannot observe removed rows; a watermarked landed source is a watermark-ordered append log, not deduplicated current-state.

Use case: Subscribers receive a changelog stream ordered by watermark; deletes leave no advancing watermark and cannot be signaled within the append-log model.

Code: provisa/core/change_signal.py, provisa/events/injector.py

Tests: tests/unit/test_injector.py, tests/unit/test_change_signal.py

REQ-929 · Change Signals¶

Status: ✅ complete · Priority: MUST · Type: structural

The notification/inbound change signal is independent from the watermark column. Allowed values: ttl | probe | probe+ttl | native | debezium | kafka. When a watermark exists, probe defaults to MAX(watermark); otherwise probe is a source-native query or TTL is used. Push variants (native/debezium/kafka) rely on the operator to maintain feed health.

Use case: Change signals are orthogonal to watermarking; different sources may use different notification mechanisms (polling vs. push, automatic vs. operator-managed).

Code: provisa/core/change_signal.py

Tests: tests/unit/test_change_signal.py

6. Materialized Views & Caching¶

REQ-930 · Materialization¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Materialization is derived from federation-engine reachability, not a user knob. A source the engine cannot reach live is materialized/landed (mandatory). A reachable source is queried live, with optional perf-caching (TTL-based). First land of an unreachable source is unconditional; the change signal governs subsequent refresh.

Use case: Materialization ensures correctness when live access is unavailable; it is driven by infrastructure capability, not user preference, eliminating misconfiguration.

Code: provisa/federation/strategy.py, provisa/federation/plan.py

Tests: tests/unit/test_federation_strategy.py

9. Live Data & Events¶

REQ-931 · Change Subscriptions¶

Status: ✅ complete · Priority: MUST · Type: structural

Derived/synthetic watermarks are manufactured in the SQL/view layer, never at the source. A view can project a computed watermark column (e.g. GREATEST(created_at, updated_at), MAX of inputs' watermarks, ROW_NUMBER) and downstream subscribes to that derived column. Source-level synthetic-watermark stamping is intentionally unnecessary.

Use case: Views provide full SQL tooling for composing derived watermarks; sourcing synthetic columns at ingest would duplicate this capability and complicate source registration.

Code: provisa/federation/residency.py

Tests: tests/unit/test_residency.py

REQ-932 · Change Detection Strategy¶

Status: ✅ complete · Priority: MUST · Type: structural

The change_signal→landing-shape mapping (push→CDC upsert/tombstone by PK, poll+watermark→APPEND, else REPLACE) is implemented as store_writer.select_landing_shape + the DB-agnostic Core ops in materialize_exec (land_replace/land_append/apply_cdc via the dialect-agnostic Connection.upsert), NOT engine CTAS in mv/refresh. The named integration tests (test_landing_replace/append_delete/publish_downstream) do not exist; coverage lives in unit tests (test_change_signal, test_materialize_landing, test_subscribe_publish). DONE (2026-07): provisa/federation/store_writer.py, provisa/federation/materialize_exec.py, provisa/core/change_signal.py.

Use case: Eliminates redundant encoding of change-detection intent across three independent fields (change_signal, freshness_mode, live.strategy). Centralizes the authoritative source and clarifies orthogonal axes (detection method, append/replace mode, materialization).

Code: provisa/core/change_signal.py, api/app.py, api/admin/schema.py, api/data/subscribe.py, live/reconcile.py, core/config_loader.py, core/models.py, mv/refresh.py

Tests: tests/unit/test_change_signal.py, tests/unit/test_materialize_landing.py, tests/unit/test_subscribe_publish.py

REQ-933 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: structural

SQLAlchemy generic types are the ONE canonical engine-independent IR type registry (provisa/core/ir_types.py). Sources map native→IR (to_ir), fed engines map IR→physical dialect, the write face maps IR→SQLAlchemy — so the fed engine is never the type authority. to_ir normalizes native/dialect spellings to one canonical IR name (varchar→text, int4→integer, jsonb→text), strips length/precision, and RAISES on an unknown type (no silent varchar default). data_type is an IR name: authoritative for a LANDED table (drives store DDL), informational for a LIVE source (engine's live resolution wins). Types resolved at registration, never lazily backfilled. DONE (2026-07): provisa/core/ir_types.py, provisa/federation/materialize_exec.py; tests: tests/unit/test_ir_types.py.

Code: provisa/core/ir_types.py, provisa/federation/materialize_exec.py

Tests: tests/unit/test_ir_types.py

4. Source Connectors¶

REQ-934 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The materialization store is RECONCILED, not blindly created: an existing landing table whose columns match config is KEPT (survives restart, data intact); a config/table/engine drift that changes the column set RECREATES it (drop+create, re-landed on next refresh); absent → create. Convergent + idempotent. Two triggers, one primitive: boot (after config load) and runtime UI (re)registration. Eager-creates landing tables (DDL) + attaches the engine's read view WITHOUT landing data — DDL/DML split so the catalog is complete at startup; landed sources only, live sources attach live. DONE (2026-07): provisa/federation/store_writer.py (reconcile_table), provisa/federation/native_backend.py (reconcile_landed_tables), provisa/federation/duckdb_runtime.py (attach_landed_source), provisa/federation/runtime.py, provisa/api/app.py, provisa/api/admin/schema.py; tests: tests/unit/test_reconcile_landed.py, tests/unit/test_store_writer.py.

Code: provisa/federation/store_writer.py, provisa/federation/native_backend.py, provisa/federation/duckdb_runtime.py, provisa/federation/runtime.py, provisa/api/app.py, provisa/api/admin/schema.py

Tests: tests/unit/test_reconcile_landed.py, tests/unit/test_store_writer.py

REQ-935 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

At land time, incoming rows are mapped to the target columns by NAME (matched land, unmatched target NULL, extra keys dropped — best-effort); the land is REFUSED when the fraction of target columns present is <= a match floor (default 0.0 fails only at 0% overlap), turned by the caller into an error event — never silently land a mangled table. Distinct from config/design drift (which recreates, authoritative). DONE (2026-07): provisa/federation/store_writer.py (check_source_drift); tests: tests/unit/test_store_writer.py.

Code: provisa/federation/store_writer.py

Tests: tests/unit/test_store_writer.py

9. Live Data & Events¶

REQ-936 · Live Data & Events¶

Status: ✅ complete · Priority: MUST · Type: structural

One event loop over one queue; base sources and MVs are homogeneous nodes. FIVE event kinds in three categories: APPLY {delta, append, replace} → landing shapes; ADVISE {warn} → alert/at-risk/DAG-tint, no enforcement, data still flows; HALT {error} → freeze (landed) / detach (live). Delta is UPSERT by PK (insert/update + tombstone delete); append is INSERT (rows + cursor WHERE watermark > last). Append carries the cursor/rows; a delta payload above the jsonb size ceiling degrades to replace. The emitted kind is set by the node's incremental-maintainability (may differ from consumed; non-maintainable → replace).

Use case: Unified event model provides a single, homogeneous abstraction for all data-movement primitives (sources, MVs, subscriptions) with clear semantics for change operations (delta, append, replace), advisory warnings, and error handling.

Code: —

Tests: tests/unit/test_event_queue.py, tests/unit/test_injector.py

6. Execution, Routing, Caching & Performance¶

REQ-937 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: structural

Every node = generate (fetch for a source | SQL for an MV — the only difference) → post-process (enrich and/or validate/gate) → emit one of the 5 events. Post-processing (enrich and/or validate) applies to sources AND MVs alike. Landing is OPTIONAL: a change event can trigger validation/reaction on LIVE data without materializing.

Use case: Single-threaded node model (generate→post-process→emit) provides a clear, composable architecture for incremental data flows while keeping all processing layers visible and auditable.

Code: —

Tests: tests/unit/test_processor.py, tests/unit/test_handlers.py

REQ-938 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: structural

Every MV is SQL, authored to REFERENCE its inputs (SQLGlot extracts the dependency edges — reuse the exp.Table extraction already in provisa/api/admin/_table_ops.py) and DECLARE its full output schema; columns the SQL doesn't derive are constant placeholders completed by post-processing. The input reference (count/key/max(watermark)) doubles as the node's change/freshness token. Guarantees lineage is always derivable — no node ever has opaque inputs. DAG is acyclic (cycle detection at registration); re-parsed on registration. One event fans out to many dependents → the exclusive unit of work is (event × dependent-table).

Use case: SQL lineage extraction from MV definitions ensures the DAG is always derivable and transparent; combining with change tokens enables correct cascading materialization and versioning.

Code: —

Tests: tests/unit/test_processor.py

REQ-939 · Execution & Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The openapi adapter fetch is wired into SourceRowLoader via make_openapi_loader, which accepts injectable per-type adapter_loaders. make_openapi_loader resolves a table's registered ApiEndpoint and its ApiSource (base_url + auth) from live app state (state.api_endpoints / state.api_sources), calls the operation with its default_params via api_source.caller.call_api, and flattens the response pages into row dicts via api_source.flattener.flatten_response — the same call_api → flatten chain the API-cache path uses. app_wiring builds it from state and registers it under "openapi" source type, so openapi source nodes now land real data. A table with no registered endpoint raises UnsupportedSourceFetch (never a silent empty snapshot). Remaining API/push types (ingest, websocket, rss, grpc_remote, prometheus, google_sheets) still raise UnsupportedSourceFetch until each is wired.

Use case: Openapi sources need a real-time fetch path that reuses the same authentication and response flattening logic as the API-cache write path, without requiring an engine-scannable table.

Code: provisa/events/source_loader.py, provisa/events/app_wiring.py

Tests: tests/unit/test_source_loader.py

4. Source Connectors¶

REQ-946 · Connector Abstraction¶

Status: ✅ complete · Priority: MUST · Type: structural

Trino Connector class is the single source of truth for a source type's Trino catalog and its reach (realizes REQ-842). Each catalogable source type has a Trino Connector (provisa/federation/connector.py) declaring trino_connector (the Trino connector.name for CREATE CATALOG ... USING) and details() (the catalog .properties). build_trino_engine registers the full set via build_trino_connectors(); provisa/core/catalog.py create_catalog/_build_catalog_properties derive name+props from the TRINO_CONNECTORS registry — a source type with no Trino connector gets NO catalog. sqlite/openapi are Mechanism.LAND (data landed into local PG cache, then Trino reads it); all other catalogable types are ATTACH. Consequence: federate() classifies Trino's connector types as VIRTUAL (federated live) rather than MATERIALIZED — engine.connectors is the engine's true reach.

Use case: Centralizing connector registration through a single registry eliminates parallel type→name maps and hardcoded property branches, ensuring consistent catalog generation and reach classification across all source types.

Code: provisa/federation/connector.py, provisa/core/catalog.py

Tests: tests/unit/test_federation_strategy.py, tests/unit/test_federation_engine.py, tests/unit/test_federation_materialization.py, tests/unit/test_sharepoint_connector.py, tests/unit/test_splunk_connector.py, tests/unit/test_trino_catalog_files.py

REQ-947 · Connector Abstraction¶

Status: ✅ complete · Priority: MUST · Type: structural

engine.connectors (the unified connector registry per federation engine) must include a connector for EVERY source type the engine can offer: (a) federated via engine's own connectors, (b) direct-route via Provisa's native async drivers (executor/drivers/registry.py), and (c) adapter-materializable (source_adapters/_ADAPTER_MAP). Only then is the source-creation dropdown a pure projection of engine.connectors; today direct-only sources (e.g., duckdb when used alone) and adapter-only sources (govdata) are absent from engine.connectors and would be dropped from that projection. Introduce a third Mechanism (FETCH) for (c): Provisa's adapter fetches rows; write face lands them; engine reads the replica. This retires the parallel SOURCE_TO_CONNECTOR map and adapter_loaders special-casing.

Use case: engine.connectors must be complete — federatable ∪ Provisa-direct ∪ adapter-materializable — so that the source-creation dropdown dynamically reflects exactly what each federation engine can reach, whether through its native connectors, direct drivers, or adapters. Eliminates hardcoded source-type lists and parallel registry maps.

Code: provisa/federation/connector.py, provisa/federation/engine.py, provisa/events/source_loader.py, provisa/executor/drivers/registry.py, provisa/source_adapters/_ADAPTER_MAP

Tests: tests/unit/test_source_registry_contract.py, tests/unit/test_engine_reach_faces.py

REQ-948 · Driver Management¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Driver availability is per-connector data, not new machinery: Connector.runtime_deps entries are tagged by who provides the driver — "system" (OS-provided), "bundled" (Provisa ships + relocates it), "operator" (BYO, must install) — and Connector.probe() (REQ-904) reports functional availability + operator remediation. A source is shown in the dropdown but rendered disabled with its remediation when its driver is operator-provided and not installed. Bundled subset Provisa distributes: postgres (libpq), duckdb (libduckdb), sqlite (system libsqlite3), csv, parquet, iceberg, sqlserver, mysql, plus any source reachable via SQLAlchemy fallback driver. Niche drivers (e.g. oracle, some SaaS) stay operator/BYO.

Use case: Tagging driver availability per-connector allows stewards to see which sources are available, which require operator installation, and what remediation is needed — without silent failures or hardcoded availability checks.

Code: provisa/federation/connector.py

Tests: tests/unit/test_connector_reach_modes.py

REQ-949 · Connector Abstraction¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Structural debt: provisa/federation/connector.py and provisa/federation/engine.py are monoliths that do not scale as more federation engines are added. Split them into per-engine modules (e.g. provisa/federation/connectors/.py and per-engine engine builders) before/as new engines land.

Use case: As Provisa adds support for additional federation engines (DuckDB, ClickHouse, etc.), centralizing all engine-specific code in monolithic modules becomes unmaintainable. Per-engine module organization enables independent engine contributions without merge conflicts or coupling.

Code: provisa/federation/trino_connectors.py, provisa/federation/clickhouse_connectors.py, provisa/federation/connector.py, provisa/federation/engine.py

Tests: tests/unit/test_federation_engine.py, tests/unit/test_clickhouse_connectors.py, tests/unit/test_connector_reach_modes.py

REQ-950 · Wire-compatible RDB Families¶

Status: ✅ complete · Priority: MUST · Type: structural

Wire-compatible RDB reuses its base wire's native async driver, SQLGlot dialect, and Trino connector via registry data only. Implemented for CockroachDB/YugabyteDB/Greenplum (Postgres-wire) and TiDB (MySQL-wire) via SourceType + SOURCE_TO_CONNECTOR + SOURCE_TO_DIALECT + _PG_WIRE_TYPES/_MYSQL_WIRE_TYPES + jdbc_url + _TRINO_JDBC_TYPES + _DRIVER_FACTORIES. Each federates VIRTUAL on Trino and is direct-route capable.

Code: provisa/core/models.py, provisa/federation/connector.py, provisa/executor/drivers/registry.py

Tests: tests/unit/test_wire_compatible_rdbs.py

REQ-951 · Connector Abstraction¶

Status: ✅ complete · Priority: MUST · Type: structural

Connector declares a CAPABILITY SET of reach modes (mechanisms: frozenset[Mechanism]) instead of a single fixed mechanism. Extend Mechanism to include SCAN and a Provisa-direct materialize member (FETCH). Examples: Trino postgresql = {ATTACH, LAND}; API source (openapi/graphql_remote/grpc_remote) = {FETCH} (Provisa fetches + write-face lands; engine reads replica); lake file on DuckDB = {SCAN}; warehouse-native = {LAND into self}. federate(), strategy.py, CatalogEntry.mechanism, materialization.py, and promote.py read the declared capability set and pick a reach mode per policy/cost instead of a single mechanism plus scattered heuristics (prefer_materialized, cost-based promotion). Materializer's flag (materialize_store) and planner logic are simplified.

Use case: Declaring a full set of reach modes per connector enables federate() and the planner to intelligently select reach modes per policy and cost, eliminating ad-hoc heuristics and making reach flexibility explicit and testable.

Code: provisa/federation/connector.py, provisa/federation/strategy.py, provisa/federation/materialization.py, provisa/federation/promote.py, provisa/core/catalog.py

Tests: tests/unit/test_connector_reach_modes.py

REQ-952 · Source Type Availability¶

Status: ✅ complete · Priority: MUST · Type: ui

The source-creation type dropdown displays the UNION of all possible source types, gated and annotated against the currently-configured federation engine to educate users on the impact of engine choice. Backend engine_registry() exposes per-engine reach faces: live_source_types (types read LIVE via attach connector — ATTACH_RW/ATTACH_R) and reachable_source_types (every configurable type = live-attach ∪ _MATERIALIZE_ONLY ∪ Provisa-direct drivers/adapters that land a refreshed replica). UI tags each option: LIVE (attach, in place, fresh) or REPLICA (Provisa lands a refreshed copy). An unreachable type is disabled and annotated "LIVE w/ {engine labels}" naming the engines that reach it live.

Use case: Users can see at source-creation time which types are available live on the current engine and which would require materialization, enabling informed connectivity choices without trial-and-error or missing sources in the dropdown.

Code: provisa/federation/engine.py, provisa-ui/src/pages/SourcesPage.tsx

Tests: tests/unit/test_engine_reach_faces.py

9. Event Processing¶

REQ-953 · Replica Lifecycle¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Source replicas are built at boot by posting replace events for each source, and refreshed by events based on their cache_ttl cadence or push listeners.

Use case: Ensures all source data is available immediately at startup and stays current during runtime through event-driven updates.

Code: provisa/events/boot.py

Tests: tests/unit/test_boot.py, tests/unit/test_app_wiring.py

4. Source Connectors¶

REQ-954 · Replica Strategy¶

Status: ✅ complete · Priority: MUST · Type: behavioral

To materialize files/sharepoint/splunk sources, Provisa starts the connector's bundled Calcite pgwire server (pgwire-file, pgwire-sharepoint, pgwire-splunk from github.com/kenstott/calcite release engine-v0.28.0), connects to it as a generic PostgreSQL endpoint, and SELECTs from the connector schema to land rows into the materialize store.

Use case: Enables replica materialization for connector-backed sources independent of the federation engine's own connectors, making them reachable via pgwire bridge.

Code: provisa/federation/strategy.py, provisa/federation/pgwire_replica.py, provisa/events/source_loader.py, provisa/events/app_wiring.py

Tests: tests/unit/test_replica_strategy.py

REQ-955 · Replica Strategy¶

Status: ✅ complete · Priority: MUST · Type: structural

Each pgwire server is configured via model/model.json in the bundle with source-specific credentials and storage paths: files → directory (or S3 storageType + storageConfig + AWS env vars); sharepoint → siteUrl, tenantId, clientId, clientSecret (or cert/device-code); splunk → url (or host/port/protocol), token or username/password, app. Provisa manages server lifecycle (start/health/stop) per source. Each server is allocated a unique --port (default 5433) and --calcite-child host:port (default 127.0.0.1:5533) to enable concurrent execution without collision; port allocation is handled transparently by the pgwire- CLI wrapper (./bin/pgwire- --port copies args into python -m pgwire_calcite.launcher). No PGWIRE_PORT env var exists; only --port CLI flag and PGWIRE_PYTHON/PGWIRE_HOME/PGWIRE_CALCITE_CLASSPATH* env vars.

Use case: Centralizes per-source configuration and lifecycle management so each connector's pgwire server is properly authenticated, monitored, and port-isolated for concurrent operation across multiple federated sources.

Code: provisa/federation/strategy.py, provisa/federation/pgwire_replica.py

Tests: tests/unit/test_replica_strategy.py

REQ-956 · Replica Strategy¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Pgwire bundles (pgwire-file, pgwire-sharepoint, pgwire-splunk) are fetched on demand from github.com/kenstott/calcite release engine-v0.28.0 at runtime and cached locally; they are not bundled with Provisa. Version pinning and artifact download are managed via runtime_deps system.

Use case: Decouples Provisa release cycles from pgwire connector releases, and keeps the Provisa distribution lean by fetching connector bundles on-demand.

Code: provisa/runtime_deps, provisa/federation/pgwire_replica.py

Tests: tests/unit/test_replica_strategy.py

9. Live Data & Events¶

REQ-957 · Table Processor (Shared)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A node's processing envelope collapses to ONE optional user hook: preprocess(rows, ctx) -> rows, run after produce (source fetch or MV SQL) and before land. Absent = identity. Returning rows proceeds to land; returning [] is a row-level no-op (nothing lands, no re-post); ctx.warn(reasons) emits an advisory warn event and keeps going (non-fatal); raising emits an error event and short-circuits the land (fatal), fanning the error forward for poison propagation. Reject is not a new outcome — it is an error (REQ-941), just emitted before landing. warn/error severity replaces any separate accept/reject channel. preprocess MUST be deterministic (its output feeds the content hash that gates the re-post; non-determinism ripples forever). Built-ins NOT in the hook: pre-produce poison short-circuit (a claimed upstream error skips produce) and Defer (a lease action, not an emitted event). ctx is read-only: node name/kind, claimed-event summary, columns/schema, per-input frontier, forced flag, last-landed hash/epoch, and window bounds.

Use case: Authors express custom data prep, enrichment, validation, quarantine, and rejection through a single deterministic transform that folds into the existing event-type vocabulary (delta/append/replace/warn/error), with no bespoke gate machinery.

Code: provisa/events/processor.py, provisa/events/handlers.py, provisa/events/boot.py, provisa/mv/preprocess.py, provisa/mv/models.py, provisa/core/models.py

Tests: tests/unit/test_live_core_loop.py, tests/unit/test_mv_preprocess.py

REQ-958 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Temporal MVs are expressed as completeness-gated processing windows, not a separate snapshot pipeline. A node's required-input set is its declared expected-events list (REQ-961), defaulting to all SQL-lineage inputs (extract_inputs, REQ-939), verified as a freshness contract at fire time (no NO_CHANGE events). A window opens on the first fanned-in event, gathers and coalesces subsequent events (fan-in collapse), and fires at the claim's deadline (window.end + allowed_lateness). produce reads as-of ctx.window.end (never now(), to stay deterministic and replayable); the result lands as an append keyed on the window boundary (accumulating lineage), so a point-in-time snapshot (e.g. daily sales) is a read-time filter over the accumulating MV. Matching "received for this window" REQUIRES a window/epoch tag stamped on each event at emit time. Late data policy is per-node: drop, or emit a correction partition (never rewrite a sealed one). A window pegs by time and aligns time-keyed inputs; joins across inputs with no common time key still need frontier machinery. The accumulating-append + read-time-filter form serves PIT as a VIEW over retained RT history and therefore stays INSIDE the NRT ideal (REQ-966) — it is reconstructible, not a departure; the departure line is reconstructible-vs-frozen, and a PIT dataset that CANNOT be rebuilt from current state plus retained event history is a preserved, sealed snapshot (REQ-983), not this.

Use case: Snapshot/temporal MVs (daily sales, period reporting) and torn-read-free joins are served by one primitive — wait for the lineage-required inputs, then compute as-of the window boundary — collapsing the trigger, the as-of peg, the finality proxy, and partial-recompute cost into a single windowing option.

Code: provisa/events/lineage.py, provisa/events/processor.py, provisa/events/deadlines.py, provisa/federation/store_writer.py

Tests: tests/unit/test_live_temporal.py

REQ-959 · Table Processor (Shared)¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Claim scheduling and failover rest on ONE correctness anchor — a compare-and-set on the claim's processor_name at commit — with every other timer demoted to a heuristic that is safe to be wrong. A claim declares a per-claim deadline (fire-by); there is no global lease. Three independent recovery paths, all resolved by the ownership CAS: (1) reassert-on-restart — a returning owner CASes its remembered claims; matched = resume (rehydrate deadline/window from the row), zero rows = a peer took it, drop; (2) optional heartbeat — a lapsed lease lets a peer take over a CRASHED owner before the deadline (failover-latency knob only; false positives cost a wasted recompute, never a double effect); (3) deadline-miss reclaim — deadline+grace passed and not completed lets a peer take over a STUCK-BUT-ALIVE owner (the failure a heartbeat structurally cannot catch). reclaimable = (heartbeat lapsed) OR (deadline+grace passed AND not completed). Deadline is availability-mode (fire partial at deadline) or consistency-mode (fire only on completeness; deadline is an SLA alert, not a trigger), per node. deadline may be self-declared at claim or derived as min(fixed, time-to-deadline).

Use case: Long-held batch/window claims survive worker crash, restart, and stuck-but-alive workers with no coordination service and no false-positive double-processing, because the ownership CAS at commit — not any lease's accuracy — decides who commits.

Code: provisa/events/queue.py, provisa/events/supervisor.py, provisa/events/processor.py

Tests: tests/unit/test_live_core_loop.py, tests/unit/test_processor.py

REQ-960 · Table Processor (Shared)¶

Status: ✅ complete · Priority: MUST · Type: constraint

The processing loop's commit ordering and landing MUST be crash-safe under at-least-once resume. post + fan_out + complete run AFTER land and in one ownership-CAS-guarded control-plane transaction (post-before-complete): a crash between land and commit re-claims and re-runs, never losing the downstream ripple. Because land hits the store DB — a different database from the control-plane queue, so no shared transaction — land MUST be idempotent on the window key (replace overwrites; append upserts by window key, never blind-appends), so a re-run or a rare double-land from a reclaim race is harmless. Combined with recompute-to-current/window-scoped generate (idempotent) and the ownership CAS at commit, this yields exactly-one committed effect from any number of resumed attempts.

Use case: Guarantees no lost fan-out and no duplicate partitions across worker crash, restart, and failover, without distributed transactions — the two code gaps (complete-before-post ordering; land not keyed for idempotency) are closed as an invariant.

Code: provisa/events/processor.py, provisa/federation/store_writer.py, provisa/federation/materialize_exec.py

Tests: tests/unit/test_live_core_loop.py, tests/unit/test_processor.py

REQ-961 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A periodic MV's processing trigger is the CALENDAR boundary (REQ-962), bounded by the claim deadline (window.end + allowed_lateness) — NOT a completeness barrier and NOT a stream of NO_CHANGE events. There is no NO_CHANGE event type. The DECLARED expected-events list is the report's FRESHNESS CONTRACT: the inputs that must be fresh-through-window.end for the output to be trusted, verified by a PULL against per-input freshness state at fire time — not by receiving events. List length is the trust/latency dial: a fact-table subset = periodic fact-gated (dimensions read as-of window.end, never listed); all lineage inputs = strong; empty = calendar-only (compute the closed period, verify nothing). Undeclared default = all lineage inputs. At fire (the deadline; earlier only if a node opts into completeness firing): produce as-of window.end from landed data, then verify each listed input is fresh-through-window.end (its source successfully refreshed to cover the window, with zero or more rows). All fresh → trusted; zero rows on a fresh input → a trustworthy zero. A listed input NOT fresh-through-window.end at the deadline → outage → warn/hold (expected-but-absent) — the accountability the existence gate provides without a human — UNLESS the calendar says the window should not exist (holiday), which removes the expectation and the alarm. Early completeness firing is an optional optimization with little value for periodic reports; the push-completeness signal it needs is only justified for a continuous always-current MV requiring torn-read-free early firing, out of scope here.

Use case: A single declarative freshness contract replaces gating/reference classification and NO_CHANGE punctuation — the author lists which inputs must be current for the report to be trusted, matching a human's expectation of whether an output is due, so an absent output is unambiguously an outage (required input stale) or a legitimate no-op (holiday / fresh zero), decided by input freshness rather than by an event.

Code: provisa/events/processor.py, provisa/events/freshness_contract.py, provisa/events/deadlines.py, provisa/events/lineage.py

Tests: tests/unit/test_live_temporal.py

REQ-962 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Temporal window boundaries come from named, shared, versioned CALENDARS — not fixed intervals. A calendar is a registered definition: base system (Gregorian / fiscal / retail 4-4-5), timezone (DST-aware — days are 23/25h, never a fixed offset), fiscal anchor, and a versioned business-day/holiday set. An MV declares (calendar, grain ∈ daily/weekly/monthly/quarterly/annual/…); the calendar deterministically yields [start,end), a calendar-addressable window id (e.g. 2026-Q1), and the next boundary — driving the as-of peg (window.end), the claim deadline (boundary + per-grain lateness), the scheduler wake, and forced-regen addressing. Grains NEST (day ⊂ week ⊂ month ⊂ quarter ⊂ year) so a coarse roll-up MV's expected set is its constituent sealed sub- windows, and a shared calendar guarantees the nesting is consistent across the DAG. The calendar also GATES WINDOW EXISTENCE: a business-day grain has no window on a non- business-day (holiday) → the MV deterministically does not generate and raises no deadline alarm; a calendar-day grain still opens a window (zero/empty row) — the grain picks which, because "no sales row" and "zero-stock row" are both valid. Holiday data is versioned/immutable-per-window for replay fidelity — never silently drifted. Per-grain: allowed_lateness and the availability-vs-consistency deadline policy (REQ-959).

Use case: Quarterly/annual/daily/fiscal/retail MVs get correct, nesting, DST- and holiday-aware boundaries from one shared calendar registry, and holidays become a deterministic "no window" rather than a silent gap or a spurious deadline alarm.

Code: provisa/events/calendars.py, provisa/events/deadlines.py, provisa/events/processor.py, provisa/events/boot.py

Tests: tests/unit/test_live_temporal.py

REQ-963 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Live (always-current) MVs debounce input churn using the SAME claim+deadline primitive as periodic MVs (REQ-959) — only the deadline SOURCE differs. Periodic: deadline = calendar boundary + lateness (REQ-962). Live: deadline = min(last_change + quiet, first_change + max_delay), a trailing-edge debounce with a hard cap. The max_delay cap is MANDATORY, not optional: pure quiet-reset starves under continuous churn (e.g. a report over 10 frequently-changing tables never goes quiet), so the cap guarantees a refresh and IS the staleness SLA — "wait for a quiet lull, but never be more than max_delay stale"; quiet=0 disables debounce (real-time). The first triggering change opens a claim; it absorbs and coalesces subsequent changes (fan-in collapse — a cross-table burst of N events becomes ONE recompute); it fires once at the deadline. Safe ONLY because recompute- to-current is idempotent — debounce deliberately drops intermediate states and recomputes the final state once. Live MV specifics vs periodic: computes as-of now (no window.end peg) and REPLACE-lands (forgetful current view, not append/accumulate); the expected- events list is the TRIGGER SET (which inputs' changes debounce-fire the recompute), not a freshness contract — unlisted inputs are read at recompute time but do not trigger. quiet and max_delay are per-MV. Debounce controls recompute FREQUENCY, not cost — per-fire cost is the orthogonal incremental-maintenance axis. All REQ-959 failover (reassert, deadline- takeover, ownership CAS) applies unchanged since the debounce deadline is the claim deadline.

Use case: A live report over many frequently-changing tables refreshes without churn — a burst of input changes collapses into a single recompute after a quiet lull, bounded by a max_delay staleness SLA — reusing the periodic MV's claim+deadline machinery with a debounce-derived deadline instead of a calendar-derived one.

Code: provisa/events/processor.py, provisa/events/deadlines.py, provisa/events/queue.py

Tests: tests/unit/test_live_temporal.py, tests/unit/test_processor.py

0. Architecture & Design Principles¶

REQ-964 · Data-Model-Driven Estate¶

Status: ✅ complete · Priority: MUST · Type: structural

The materialization estate is DATA-MODEL-DRIVEN, not process-driven — the key differentiator. You declare the authoritative, governed data you want represented (each MV as SQL) and the RULES for processing/reprocessing (expected-events freshness contract REQ-961, calendar/grain REQ-962, deadline/debounce REQ-959/REQ-963, preprocess REQ-957); the system DERIVES the process — lineage from SQL (REQ-939), fan-out, scheduling, and failover — with nothing hand-orchestrated. This inverts the imperative pipeline model (dbt/Airflow author the process; the data model is emergent) and yields a single, almost- PROVABLE data estate: a deterministic function of (governed sources, SQL definitions, rules). "Almost" is bounded by five explicit PROOF OBLIGATIONS, each pushable toward mechanical enforcement: (1) transform determinism — MV SQL and preprocess must be pure (reject now()/random at registration; sandbox from wall-clock); (2) source freshness honesty — the one irreducible trust boundary, "trusted" is relative to what a source reports; (3) land idempotency on the window key — exactly-one-effect under the store/queue two-database split rests on this + ownership CAS (REQ-960), not a distributed transaction; (4) temporal-join correctness — the barrier guarantees input availability, not point-in- time-correct (SCD-2) joins, which are the SQL author's obligation (lintable); (5) calendar immutability — replay fidelity requires holiday/fiscal data immutable-per-window (REQ-962). Close 1/3/5 mechanically, lint 4, and the estate is provable except where a source lies — the strongest guarantee a federated estate can make.

Use case: Governance and reproducibility are structural properties of the estate, not process hygiene: because lineage is derived (cannot drift), outputs are addressable and deterministically replayable, and trust is explicit via the freshness contract, the whole data estate is auditable as a function of its declared inputs and rules.

Code: provisa/events/lineage.py, provisa/events/boot.py, provisa/events/handlers.py

Tests: tests/unit/test_live_selfdescribing.py

9. Live Data & Events¶

REQ-965 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

An MV has TWO INDEPENDENT operator-declared outcomes, not one: a PERSISTENCE outcome (how the compute result is applied to the MV's own store table — replace / append / upsert, via land_replace / land_append / apply_cdc already in materialize_exec) and a SET of EVENT outcomes emitted downstream: per fire an MV may emit ANY, ALL, or NONE of {replace, append, delta} (alongside warn / error), each shape routed to the dependents that consume it — persistence is a single choice, the event side is a set, not a scalar. Emission is DEMAND- DRIVEN and pay-per-consumer: a shape is produced only if a dependent (or an explicit sink) subscribes to it, so delta's diff cost is paid ONLY when delta has a subscriber. This is what makes operational datasets emergent — a History dataset is the subscriber of the append stream, a DQ dataset the subscriber of the warn/error stream, incremental dependents the subscribers of the delta stream, all from the same MV fire; the estate becomes self- describing (quality, history, freshness, lineage, audit are just more MVs over the streams the machinery already emits). The event outcomes are symmetric with the input side (select_landing_shape / change_signal, REQ-932), so a dependent MV cannot distinguish a base source from an upstream MV — the uniform DAG completed on the output side. The two axes are decoupled along BOTH cost and semantics: (cost) persist=replace + emit=delta recomputes the whole table but diffs new-vs-old ONCE at emit to hand every dependent a precise delta instead of a recompute; persist=upsert + emit=replace maintains precisely but tells downstream only "recompute"; (semantics) persist=replace + emit=append lets a forgetful current-state MV feed a downstream that KEEPS HISTORY — it overwrites itself yet emits its output as an append batch the downstream accumulates. The event outcome is a downstream-facing contract chosen for downstream's needs, not derived from persistence. Emit shapes form a cost/semantics ladder: replace = cheapest, a recompute signal → downstream recomputes; append = emit produced rows as a batch, no diff → downstream accumulates; delta = needs row identity (PK) and possibly a diff at emit if persistence did not already yield the changed rows → downstream applies precisely. Governing rule: event granularity <= (what persistence yields) + an optional emit-diff, whose cost — like the persistence cost — is a DESIGN CHOICE only the operator can make (dataset scale, incrementalizability, prior-state storage, recompute-vs-diff SLA); for very large datasets a diff/delta may be infeasible and the operator declares coarser shapes. The system NEVER infers a shape nor silently downgrades: a declared-but-infeasible outcome on either axis (emit=delta or persist=upsert without a PK, a diff exceeding a declared bound) is an EXPLICIT ERROR surfaced to the operator, never a silent fallback — that would hide churn/cost and break provability (REQ-964). A maintenance mode MAY supply defaults (periodic → append, live → replace) but the operator's declarations are authoritative. Concretely, persist=replace (a limited, performance-sized hot/current table) + emit=append (a downstream history accumulator) keeps PIT DERIVABLE from RT and therefore INSIDE the NRT ideal (REQ-966) — no departure; the frozen, non-reconstructible snapshot is the separate departure (REQ-983).

Use case: An MV is a source to its dependents, and how it stores itself is separable from what it tells them: the operator declares a persistence outcome for its own storage economics and an event outcome for the downstream contract — enabling a forgetful view to feed a historical accumulator, or a full recompute to propagate as a precise delta — so incremental (or historical) change flows through the DAG exactly as the operator chose and can afford, or the estate errors explicitly rather than degrading silently.

Code: provisa/events/outcomes.py, provisa/events/handlers.py, provisa/events/processor.py, provisa/events/boot.py, provisa/events/app_wiring.py, provisa/federation/materialize_exec.py, provisa/federation/store_writer.py, provisa/core/change_signal.py, provisa/mv/models.py

Tests: tests/unit/test_live_mv_outcomes.py

0. Architecture & Design Principles¶

REQ-966 · Data-Model-Driven Estate¶

Status: ✅ complete · Priority: MUST · Type: structural

The zero-config default is the HAPPY PATH of an always-on, real-time enterprise, and it is the model's canonical IDEAL: an MV needs only its SQL — lineage is auto-derived (REQ-939), fan-out auto-wired, failover (ownership CAS + reassert, REQ-959/REQ-960) always-on and config-free — and everything self-organizes into the DAG and reprocesses in near-real-time (event-driven, recompute-to-current, replace-persist, demand-emit). EVERY other primitive is PROGRESSIVE OPT-IN CONFIG that expresses a justified DEPARTURE from that ideal. Every departure traces to exactly TWO ROOT CAUSES, and config SHOULD be tagged with which: (1) RESOURCE constraint — "can't afford NRT" — cost/scale/latency limits behind debounce (REQ-963), deadlines/batch (REQ-959), fact-gating (REQ-961), delta/incremental (REQ-965), and (at the storage level) even pre-cut rollups vs filtering the live stream (REQ-958/ REQ-962); (2) BUSINESS MANDATE — "the biz wants EOD reports and that's the way it is" — period boundaries and history the organization requires (REQ-958/REQ-962/REQ-965). The two have OPPOSITE TRAJECTORIES: resource concessions are DEBT that retires as infrastructure improves (cheaper compute retires debounce, reliable sources retire deadlines, incremental engines make NRT-at-scale free). Business mandates stratify: ENDOGENOUS mandates — internal habits (EOD/monthly cadences) formed because real-time was infeasible — dissolve as NRT normalizes, because capability RESHAPES demand (free live output makes people stop asking for EOD); EXOGENOUS mandates — market close, filing deadlines, counterparty cutoffs — are imposed from outside and no compute or culture-shift retires them (the genuine floor). Defaults are designed, documented baselines, NOT silent fallbacks — the operator always knows the NRT default and departs explicitly. The why-tag makes the config surface a SELF-DESCRIBING LEDGER (itself queryable as a dataset): resource-tagged = retirement backlog / infra-ROI map ("what real-time maturity costs to buy"); endogenous-mandate = adapts as culture catches up; exogenous-mandate = the permanent business-requirement registry ("what you can never buy out of, and who owns it") — and it surfaces FOSSILIZED constraints (EOD because an overnight mainframe batch once required it) as retireable debt masquerading as mandate. But the knobs are PERMANENT for a reason bigger than any floor: business culture and technical capability are two independently-moving frontiers that NEVER FULLY SYNC — tech outruns adoption, biz outruns capability, and the gap is never zero, never static, and differs per dataset. So the config is not merely retireable compromise but the permanent ALIGNMENT LAYER coupling two co-evolving systems — which is WHY it must be declarative (biz expresses intent), granular / per-node (the gap differs per dataset), continuously retunable (the gap moves), and non- destructive (retune re-aligns without re-authoring process). Config measures distance from the ideal (sharpening REQ-964); the knobs keep intent and capability coupled while both move.

Use case: Onboarding is SQL-only and the platform degrades from the NRT ideal one justified, why-tagged config at a time — so the sum of an org's config is a queryable map of its distance from always-on real-time, split into retireable-with-investment vs permanent-business-truth, and reducing that distance is a matter of retiring resource-driven config as infrastructure allows (never re-authoring process) down to the irreducible mandate floor.

Code: provisa/events/boot.py, provisa/events/lineage.py, provisa/events/processor.py

Tests: tests/unit/test_live_selfdescribing.py

REQ-967 · Data-Model-Driven Estate¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The estate is SELF-DESCRIBING: because every change is an event and every derived dataset is an MV, the system's OWN operational metadata is emergent — each operational concern is just an MV over streams the machinery already emits, with no bespoke subsystem. Catalog: DQ / quality = an MV over the warn/error emissions (REQ-957, REQ-941), append-keyed → a scorecard and quality history; HISTORY / SCD-2 = an MV over an append emission from a replace-persist upstream (REQ-965) → a time series, which also RETIRES proof-obligation #4 (REQ-964) — the SCD-2 dataset IS the point-in-time-correct dimension, so temporal joins become a declared dataset instead of an unenforced author burden (this reconstructible SCD-2 view is PIT inside the ideal; a frozen, non-reconstructible snapshot is the REQ-983 departure); FRESHNESS / VINTAGE = an MV over the freshness-contract state (REQ-961) → an SLA / as-of dashboard with queryable outage detection; LINEAGE / IMPACT = an MV over the SQL-derived DAG edges (REQ-939) → impact analysis; AUDIT / PROVENANCE = an MV over the durable event log (REQ-960 queue) → a replayable operational trail; CHURN / COST = an MV over fire and debounce-collapse events (REQ-963) → a tuning dashboard; CONFIG MATURITY = an MV over why-tagged config (REQ-966) → the distance-from-NRT map. The meta-level (how the estate operates) folds into the object- level (governed MVs) — the estate observes itself with its own machinery, so quality, history, freshness, lineage, audit, and cost are first-class governed data, governed and lineage-tracked like any other dataset. This reflexivity is the COMPLETENESS TEST for the primitive set (you cannot reach outside it to describe it) and the strongest form of the REQ-964 thesis: a new operational capability is a DECLARATION — an MV over an existing stream — never a new code path or a separate DQ / history / audit subsystem.

Use case: Observability, quality, history, audit, lineage, and cost are not separate subsystems bolted on — they are declared MVs over the streams the platform already emits, so they inherit the same governance, lineage, and reprocessing guarantees as the business data, and new operational views are added by declaration rather than engineering.

Code: provisa/events/handlers.py, provisa/events/processor.py

Tests: tests/unit/test_live_selfdescribing.py

9. Live Data & Events¶

REQ-968 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Forced regen / replay recomputes a derived node — or an addressable calendar window (e.g. 2026-Q1, REQ-962) — ON DEMAND, bypassing the token/no-op gate (REQ-958). Scope selection: by SOURCE (regen the roots and let lineage fan-out cascade forward — the natural, always-correct form), by MV NODE (regen a specific derived node without re-landing its sources — for a changed SQL def), or by WINDOW-ID (regen one sealed period). It posts synthetic events marked forced ({forced:true, reason, scope}); a forced event MUST bypass the no-op gate (it recomputes regardless of change) yet still runs preprocess (REQ-957) and honors the output outcomes (REQ-965). The forward cascade is the normal fan-out — dependents recompute in topological order. Idempotent land on the window key (REQ-960) makes replay exactly-once; determinism (REQ-964 obligation 1) + calendar-addressable windows (REQ-962) make a window regen EXACT — it recomputes precisely that period from its as-of inputs. Forced events are marked for audit (REQ-967) so a replay is distinguishable from an organic change.

Use case: Operators can restate a period, rebuild after a definition change, or recover from a bad load by declaring a scope to regenerate — sources, a node, or a window-id — and the estate replays deterministically and exactly-once through the existing DAG, with no bespoke backfill job.

Code: provisa/events/processor.py, provisa/events/injector.py, provisa/events/queue.py, provisa/events/deadlines.py

Tests: tests/unit/test_live_regen_snapshots.py

REQ-969 · Derived / MV Processor¶

Status: ✅ complete · Priority: MAY · Type: behavioral

Incremental maintenance computes a derived node's output from an upstream DELTA instead of a full recompute — the cost axis that makes emit=delta (REQ-965) worth declaring. When an input arrives as delta/append (REQ-932/REQ-965), an incrementally-maintained MV applies only the changed rows to its prior landed state (persist=upsert/append) and emits the resulting delta, rather than re-running the full SELECT. Feasibility is per-MV and OPERATOR-DECLARED: it is expressible only when the SQL admits an incremental form (many aggregations/joins do via delta rules; some do not) and a PK exists for row identity — a declared-but-infeasible incremental MV is an EXPLICIT ERROR, never a silent fall-back to full recompute (REQ-964; a silent fall-back hides cost). Realized either as author-provided incremental SQL or a system delta-rule derivation where the SQL shape permits. Recompute-to-current stays the DEFAULT (REQ-966 NRT baseline); incremental is the resource-concession config that lowers per-fire cost for large or hot derived nodes. Conflict to declare around: an incrementally-maintained MV cannot freely drop intermediate deltas the way recompute-to-current can, so it partially conflicts with debounce (REQ-963) — the operator chooses per node.

Use case: Large or high-churn derived nodes refresh by applying upstream deltas instead of full recomputation, making delta emission affordable and incremental propagation real — declared per MV where feasible, erroring explicitly where not, never silently degrading.

Code: provisa/events/handlers.py, provisa/events/lineage.py, provisa/events/boot.py, provisa/federation/materialize_exec.py, provisa/mv/models.py

Tests: tests/unit/test_live_regen_snapshots.py

REQ-970 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A derived node's store-table schema is DERIVED from its SQL SELECT (output column names + types), not taken from a source — the structural contrast with a replica, whose schema comes from its source (REQ-846). At registration the SELECT is analyzed (SQLGlot + type inference, or a describe against the federation engine) to produce the output columns; the MV store table is created/reconciled to that schema via the existing reconcile_table machinery (REQ-846). A definition change that drifts the output schema → RECREATE (drop + reland on next fire), symmetric to source-drift reconcile. A PK for the derived table — required for persist=upsert / emit=delta / incremental (REQ-965/REQ-969) — is operator-declared or inferred from an unambiguous GROUP BY / distinct key; required-but-absent is an explicit error, never a silent replace. The derived schema flows into lineage (REQ-939) so downstream MVs type-check their inputs against it.

Use case: An MV needs only its SQL: its output schema, store table, and PK are derived and reconciled automatically and published into lineage so the DAG is type-checked end to end — no hand- authored DDL, symmetric to but distinct from a replica's source-derived schema.

Code: provisa/federation/store_writer.py, provisa/events/lineage.py, provisa/events/app_wiring.py, provisa/compiler

Tests: tests/unit/test_live_mv_outcomes.py

6. Execution, Routing, Caching & Performance¶

REQ-971 · Data Masking & Expression Pushdown¶

Status: ✅ complete · Priority: MUST · Type: constraint

Column-level data masking MUST be expressed as semantic SQL projection expressions within the governed IR so that masking participates in expression/projection pushdown exactly as RLS predicates participate in predicate pushdown. Where a connector can evaluate the mask expression at the source, the mask MUST push down so the raw, unmasked column value never crosses the wire into the middle tier. Any mask that does NOT push down to the source MUST be evaluated in a bounded-memory streaming stage — either engine-side (the fed engine computes the mask expression and streams results) or per-event/per-batch in the middle tier (as the subscription path already does via _mask_row over an async event stream). It is FORBIDDEN to materialize the full unmasked relation in the middle tier to mask it (no fetchall()-style buffer-then-mask). This streaming constraint ensures masking correctness never depends on fed-engine pushdown capability — a capability gap must never become a data leak — and bounds middle-tier memory to O(batch), not O(relation), minimizing the amount of cleartext resident in the tier at any instant.

Use case: Masking today injects scalar SQL expressions (REGEXP_REPLACE, DATE_TRUNC, constant) into the governed SELECT projection after RLS injection. This works correctly but lacks first-class pushdown capability: the planner cannot reason about whether a mask pushed to the source, and no Capability flag tracks whether the source can evaluate the mask's functions. Without projection pushdown, the raw cleartext value must transit to the engine even if the source could mask it — a confidentiality gap analogous to forcing RLS evaluation server-side when the source supports it. The design must model masking as a pushdown-capable expression so confidentiality-aware masking (including a governance guard that acknowledges non-pushable masks) becomes explicit in the planner. For masks that cannot push down, the fallback must evaluate them in a streaming stage to prevent the middle tier from buffering the full unmasked relation into memory — rationale: masking correctness must never depend on pushdown capability (a gap is never a data leak), and the middle tier must bound memory to O(batch), not O(relation), with minimal cleartext residence time.

Code: provisa/compiler/mask_inject.py, provisa/compiler/stage2.py, provisa/federation/connector.py, provisa/federation/promote.py, provisa/federation/cardinality.py, provisa/api/data/subscribe.py

Tests: tests/unit/test_mask_pushdown.py

11. Platform, Infrastructure & Delivery¶

REQ-972 · Desktop Installation¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Desktop installers default to a self-contained native install: DuckDB federation engine + sqlite control plane + in-memory (fakeredis) cache, with no Docker, VM, Trino, Redis, or MinIO. A native preset in config/capabilities.yaml declares this base tier.

Use case: Enables zero-dependency installation for lightweight desktop and demo scenarios without external infrastructure.

Code: config/capabilities.yaml, provisa/core/desktop_profile.py

Tests: tests/unit/test_desktop_profile.py

REQ-973 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Federation engine is a wizard choice, not a fixed tier: DuckDB (native, default) | Trino-on-Docker | External engine (URL or host+port). Trino-on-Docker and external engines are opt-in overrides layered on the native base.

Use case: Allows users to upgrade from native DuckDB to Trino or an external federation engine without reinstalling the base system.

Code: provisa/core/desktop_profile.py, packaging/macos/first-launch.sh, packaging/windows/first-launch-native.ps1

Tests: tests/unit/test_desktop_profile.py

REQ-974 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

desktop_profile.load_profile gains engine/engine_url/materialize_url/trino_endpoint/otlp_endpoint overrides mapping to PROVISA_ENGINE / PROVISA_ENGINE_URL / PROVISA_MATERIALIZE_URL / TRINO_HOST+PORT / OTEL_EXPORTER_OTLP_ENDPOINT environment variables.

Use case: Enables container orchestrators and deployment tools to override engine and observability endpoints at runtime without modifying the installer-created configuration.

Code: provisa/core/desktop_profile.py

Tests: tests/unit/test_desktop_profile.py

REQ-975 · Desktop Installation¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Observability is always-on built-in (self-telemetry in Admin). The optional Docker observability stack (otel-collector + prometheus + grafana) is an external demonstration, not an on/off switch; selecting it only redirects OTLP export to external infrastructure.

Use case: Ensures telemetry is always collected and visible in Admin UI; external obs-integration is optional for enterprise monitoring without adding complexity to the base install.

Code: provisa/core/desktop_profile.py

Tests: tests/unit/test_desktop_profile.py

REQ-976 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Docker and VMs are provisioned only when needed: engine == Trino-on-Docker, OR the obs-integration demo runs on Docker, OR the demo runs on Docker. Otherwise the install is fully native.

Use case: Minimizes resource footprint for users who do not require Trino or external observability, reducing memory and disk overhead on desktop systems.

Code: packaging/macos/first-launch.sh

Tests: tests/unit/test_desktop_install.py

REQ-977 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

All heavy add-ons (Trino image set, observability stack, demo) are downloadable and resolved local-first: installer-adjacent directory, then mounted volumes, then ~/Downloads, then GitHub release. Enterprise can pre-stage tarballs for airgapped installation.

Use case: Enables fully airgapped deployment by allowing enterprises to pre-stage container images and demo datasets alongside the installer.

Code: packaging/macos/first-launch.sh

Tests: tests/unit/test_desktop_install.py

REQ-978 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Installing the demo is optional and OFF by default. When installed, the launcher opens the UI at ?tour=1 to auto-start the guided tour.

Use case: Separates the base product from demo data; guided tour auto-starts for new users who opt into the demo without manual navigation.

Code: packaging/macos/ProvisaLauncher/Sources/ProvisaLauncher/Views/Setup/SetupWizardView.swift, packaging/windows/provisa-native.ps1, packaging/macos/first-launch.sh

Tests: tests/unit/test_desktop_install.py

REQ-979 · Desktop Installation¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

The native tier builds a Python VENV at first launch from PyPI (pinned to release). The DMG/EXE bundles a bare python-build-standalone interpreter + a wheelhouse. First-launch creates ~/.provisa/venv and installs provisa[embedded] online from PyPI, or offline via pip --no-index --find-links from the bundled wheelhouse. The native tier runs with no Docker or container dependencies.

Use case: Ensures the entire native-tier stack is self-contained and runs on any OS (macOS, Windows, Linux) without requiring external Python environments or container runtimes.

Code: packaging/macos/build-dmg.sh, packaging/windows/build-sfx.ps1, packaging/windows/first-launch-native.ps1, packaging/windows/provisa-native.ps1

Tests: tests/unit/test_desktop_install.py, tests/unit/test_infra_requirements.py

3. Federation Engines & Data Sources¶

REQ-980 · Query Compilation¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Add json as a first-class IR type so native JSON in source columns preserves through to materialized stores as native JSON where supported, instead of collapsing to text. The write face maps IR json to SQLAlchemy's generic JSON (native on Postgres/MySQL/DuckDB, TEXT-affinity on SQLite via the dialect compiler); the land path parses a JSON column's serialized-text value into a Python object before insert so the SQLAlchemy JSON column does not double-encode; and the read path converts an already-parsed dict/list consistently with the string path. Composite/array types (row/array/list/struct/map) remain collapsed to text.

Use case: Federation queries over JSON columns maintain type information and enable native JSON operations in target stores, rather than forcing serialization fallbacks.

Code: provisa/core/ir_types.py, provisa/federation/materialize_exec.py, provisa/executor/serialize.py

Tests: tests/unit/test_ir_types.py, tests/unit/test_materialize_landing.py

9. Live Data & Events¶

REQ-981 · Live Data & Events¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The event loop's content-hash output gate — an always-on no-op ripple suppressor for replace-shaped landings. After a source-land or MV-generate produces its result, the node hashes the landed content (canonical, order-stable over the primary key) and compares it to the hash stored from the prior land. Unchanged → the node does NOT re-post its change event, so no downstream MV recompute cascades; the redundant store write is also skipped. Changed → land, persist the new hash, re-post. This makes a change_signal=ttl / probe_type=none poll (which always re-fetches) cheap on the DOWNSTREAM side: the full fetch still happens, but an unchanged result never ripples the DAG. Applies to replace shapes only (append/CDC deltas are new rows by definition and pass through). The hash is the correctness backstop that probe_type (REQ-981) optimizes the fetch in front of.

Use case: A TTL-polled source or MV that re-queries on cadence but whose content rarely changes no longer forces its entire dependent MV subtree to recompute on every tick — the DAG only ripples when landed content actually differs, eliminating wasted downstream work.

Code: provisa/events/processor.py, provisa/events/handlers.py, provisa/events/content_hash.py, provisa/events/queue.py, provisa/core/schema_org.py

Tests: tests/unit/test_content_hash.py, tests/unit/test_live_change_gates.py, tests/unit/test_handlers.py

REQ-982 · Live Data & Events¶

Status: ✅ complete · Priority: MUST · Type: behavioral

probe_type is the event loop's input-side change-detection axis, orthogonal to the change_signal cadence (ttl | probe | ttl_probe). Values: watermark | hash | count | none. The type both selects the detection transport AND implies the landing shape: watermark -> append (fetch WHERE wm > cursor, needs watermark_column), hash/count/none -> replace. This REPLACES the prior "watermark_column presence implies append" heuristic in select_landing_shape with an explicit probe_type -> shape map. Availability is gated by a per-source capability matrix probe_capabilities(source_type): SQL/engine-scannable sources support {watermark, hash, count, none} (hash = full-scan checksum, expensive, degenerates to the REQ-980 output hash); HTTP/API sources support {hash via ETag/Last-Modified, count via total-header, watermark if a sortable cursor, none}; file/object sources support {hash via mtime+size, none}; push/streaming sources are not on the probe axis. Config validation rejects a probe_type outside the source's capability set; when change_signal in {probe, ttl_probe} and probe_type is unset the default resolves per class (SQL -> watermark if watermark_column else count; API/file -> hash). Each transport implements freshness_token(source, table) -> str | None; a None token degrades the node to TTL cadence (REQ-847 capability signal, never a silent stale fallback). The token baseline is persisted per node and compared through evaluate_freshness (REQ-855). The real probe factory replaces the always-changed _poll_probe_factory in the event-loop boot wiring.

Use case: An operator declares how each source detects change appropriate to its type — a watermark column for an append-only table, an ETag for an HTTP API, mtime for a file — so the loop skips the fetch entirely when the source is unchanged, rather than re-querying every cadence and relying only on the downstream hash gate.

Code: provisa/core/change_signal.py, provisa/core/models.py, provisa/events/injector.py, provisa/events/boot.py, provisa/events/app_wiring.py, provisa/events/probes.py, provisa/events/handlers.py, provisa/federation/residency.py, provisa/federation/store_writer.py, provisa/federation/freshness_gate.py

Tests: tests/unit/test_probes.py, tests/unit/test_injector.py, tests/unit/test_probe_type_config_validation.py, tests/unit/test_live_change_gates.py, tests/mvmvp/test_mv_triggers_e2e.py

REQ-983 · Derived / MV Processor¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A preserved snapshot is a point-in-time dataset that is MATERIALIZED-AND-SEALED because it is NOT reconstructible from current state plus retained event history — the one PIT form that is a genuine DEPARTURE from the NRT ideal (REQ-966), as opposed to the reconstructible PIT of REQ-958/REQ-965/REQ-967 (accumulating append + read-time filter) which stays inside the ideal. It is DECLARED, never inferred, and MUST be why-tagged with its REQ-966 root cause: RESOURCE — the deriving history was never retained or the source cannot be replayed (debt that retires as retention/replay capability improves); or BUSINESS MANDATE — the output must be frozen exactly as computed at a moment (a definition or upstream changed and the prior result must be preserved as-is; an external filing/attestation fixes the number). The snapshot is landed as an IMMUTABLE, addressable partition (a REQ-962 window-id or an explicit snapshot-id), sealed like a calendar window: never rewritten, corrected only by a new partition, replay-exact via determinism (REQ-964) where the inputs still exist, and otherwise preserved verbatim because re-derivation is impossible. Distinct from REQ-958's accumulating-append history (a reconstructible view over RT) and from a live replace-persist hot table (REQ-965): a preserved snapshot is chosen precisely when derive-from-RT is NOT available. Never a silent fallback — the operator declares it with its root cause, or the estate serves PIT as a view.

Use case: Real-world estates cannot always retain full event history or replay every source, and some outputs must be frozen as-computed for audit/attestation — so the platform offers PIT datasets built and preserved at a moment as an explicit, why-tagged departure, keeping derivable PIT the default and making every frozen snapshot a visible entry in the distance-from-NRT ledger (REQ-966).

Code: provisa/events/snapshots.py, provisa/core/schema_org.py, provisa/federation/materialize_exec.py

Tests: tests/unit/test_live_regen_snapshots.py

6. Materialized Views & Caching¶

REQ-984 · Materialized View Lifecycle¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The legacy periodic CTAS materialized-view refresh loop is retired in favor of the event loop as the sole MV compute path. The refresh_loop function is replaced by reclamation_loop, which handles only storage reclamation (drop removed MVs + reap orphaned tables per REQ-234); periodic refresh cadence is now driven by each MV's event-loop poll job, reframing periodic refresh as the REQ-963 debounce deadline source.

Use case: Consolidating MV compute into a single event-driven path eliminates double-computation and simplifies the refresh lifecycle while preserving operator-triggered refresh mutations.

Code: provisa/mv/refresh.py, provisa/api/app_startup.py, provisa/events/boot.py

Tests: tests/unit/test_mv_reclamation_loop.py

6. Execution, Routing, Caching & Performance¶

REQ-985 · Cache¶

Status: ✅ complete · Priority: SHOULD · Type: structural

When cache.enabled is omitted from config, caching defaults to ENABLED. A CacheStore always exists: RedisCacheStore(None) falls back to embedded fakeredis when no Redis URL is configured. Operators must explicitly set cache.enabled: false to disable caching via NoopCacheStore.

Use case: Operators have safe defaults where caching is active unless explicitly disabled, eliminating silent no-cache states and ensuring consistent cache behavior across hot tables and response caches.

Code: provisa/api/app.py, provisa/cache/hot_tables.py

Tests: tests/unit/test_cache_default_enabled.py

3. Federation Engines & Data Sources¶

REQ-986 · Arrow Flight Transport¶

Status: ✅ complete · Priority: MUST · Type: behavioral

ClickHouse federation engine must expose Arrow-native transport through the Provisa Arrow Flight server to honor its declared ARROW and ARROW_STREAM capabilities, enabling columnar data streaming without row materialization.

Use case: ClickHouse can deliver Arrow data natively over HTTP and native TCP protocols, but currently Provisa collapses these to row tuples. Exposing this capability via Arrow Flight matches Trino's architecture and enables efficient columnar streaming for high-throughput federated queries.

Code: provisa/federation/engine.py, provisa/federation/backend.py, provisa/federation/clickhouse_runtime.py, provisa/api/flight/server.py

Tests: tests/unit/test_engine_capabilities.py, tests/unit/test_native_arrow_transport.py, tests/integration/test_clickhouse_runtime_e2e.py, tests/integration/test_arrow_flight_integration.py

REQ-987 · Arrow Flight Transport¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Databricks SQL warehouse must be supported as a first-class Provisa federation engine with Arrow-native transport capabilities for both READ and WRITE paths. Read-side: Arrow columnar data streaming through the Provisa Arrow Flight server via execute_arrow/execute_stream. Write-side: materialization-store landing, residency reconciliation, and CDC snapshot seed must use Arrow-native columnar bulk writes (via ADBC adbc_ingest() or Parquet -> cloud object stage -> COPY INTO / Delta register), never row-oriented INSERT. End-to-end columnar path avoids row materialization overhead in Python and enables Databricks' distributed ingest parallelization. Row INSERT acceptable only for tiny writes where per-file staging overhead dominates.

Use case: Organizations using Databricks SQL warehouses require direct federation capability to Provisa without intermediary routing through Trino. Arrow-native transport for both read and write paths avoids row materialization overhead and matches Trino's architecture for efficient federated query execution over Databricks. Materialized views and landed federation outputs must remain columnar end-to-end, enabling downstream queries to execute at warehouse speed.

Code: provisa/federation/engine.py, provisa/federation/backend.py, provisa/federation/store_writer.py, provisa/api/flight/server.py, provisa/transpiler/transpile.py

Tests: tests/unit/test_databricks_store.py, tests/unit/test_engine_capabilities.py, tests/integration/test_databricks_federation_engine_e2e.py, tests/integration/test_databricks_external_link_e2e.py

REQ-988 · Arrow Flight Transport¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Snowflake federation engine must be promoted from a source-only connector (federated through Trino) to a first-class Provisa federation engine peer to Trino/DuckDB/ClickHouse/Postgres, with Arrow-native transport wired through the Provisa Arrow Flight server. Engine declares ROWS, ARROW, and ARROW_STREAM capabilities; backend implements execute_arrow() and execute_stream() to deliver Arrow columnar data and lazy record-batch streaming without row materialization. Physical SQL transpilation must target Snowflake dialect. Governance (RLS, masking, rate-limiting) applies unchanged above the backend.

Use case: Snowflake is an MPP warehouse with native Arrow support; exposing its columnar capabilities via Provisa's Arrow Flight server enables direct, efficient federated queries without row materialization and eliminates the current Trino-as-middleman routing. Matches architectural parity with Trino (REQ-986 ClickHouse parallel) and serves enterprises whose primary warehouse is Snowflake.

Code: provisa/federation/engine.py, provisa/federation/backend.py, provisa/api/flight/server.py, provisa/executor/trino_flight.py, provisa/transpiler/transpile.py, provisa/source_registry.py, provisa/source_connectors/snowflake_connector.py

Tests: tests/unit/test_engine_capabilities.py, tests/unit/test_transpiler.py, tests/integration/test_snowflake_federation_engine_e2e.py

REQ-989 · Platform Defaults¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Zero-config default stack must be fully embedded and in-process with no external dependencies: default federation engine is duckdb (not trino), default materialize store is duckdb's native embedded store (not platform tenant DB via Postgres), and default control-plane store is sqlite (not embedded_pg). External engines and stores (Trino, Postgres, ClickHouse, Snowflake, Databricks) remain selectable via PROVISA_ENGINE / PROVISA_ENGINE_URL / PROVISA_MATERIALIZE_URL / control_plane_store overrides.

Use case: Organizations and developers need instant local startup with zero Docker, no Trino cluster, no Postgres dependency — everything embedded for true out-of-the-box instant start. The zero-config path must serve as an approachable entry point while preserving full enterprise engine/store pluggability via configuration.

Code: provisa/federation/engine.py, provisa/core/desktop_profile.py, provisa/core/capabilities.yaml

Tests: tests/unit/test_duckdb_store_native.py, tests/unit/test_federation_engine.py

REQ-990 · Materialization Writes¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Materialization-store writes MUST use the target store's columnar or bulk-COPY ingest path wherever available, instead of row-by-row INSERT. Row INSERT is permitted only for targets without bulk support or tiny writes where staging overhead dominates, and this choice must be explicit/capability-gated, never a silent fallback.

Use case: Row-by-row INSERT serializes writes, defeats vectorization, and forces Python row tuples through the network. Bulk columnar paths (Arrow, COPY, adbc_ingest) maintain end-to-end columnar efficiency, reduce network overhead, and leverage target-store parallelization. This is especially critical for high-volume materialization, residency reconciliation, and CDC snapshot seeding.

Code: provisa/federation/store_writer.py, provisa/federation/store_connection.py, provisa/federation/backend.py

Tests: tests/unit/test_store_writer.py, tests/unit/test_databricks_store.py

9. Live Data & Events¶

REQ-991 · Live Delivery Strategy¶

Status: ✅ complete · Priority: MUST · Type: constraint

Live Delivery mechanism (push repeat, append poll, or full-replace poll) is DERIVED SOLELY from the effective change_signal plus watermark column presence. The Live Delivery UI exposes no independent Strategy selector; the delivery mechanism mirrors change_signal.select_landing_shape: push signal → repeat the change stream; poll signal WITH watermark → append; poll signal WITHOUT watermark → full-replace.

Use case: Eliminates redundant strategy selection across detection method and delivery mode. Centralizes authoritative control in Change Signal, reducing cognitive load and preventing inconsistent strategy/detection pairs. The watermark presence is the determining factor for append vs. replace, not a separate configuration.

Code: provisa/live/engine.py, provisa/live/reconcile.py, provisa-ui/src/pages/tables/LiveDeliveryFieldset.tsx

Tests: tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py, tests/unit/test_live_delivery_config.py

REQ-992 · Live Delivery Strategy¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Poll signals without a watermark column execute a full-replace poll mode: LiveEngine._poll_replace re-scans the full result set at each interval and emits a replace-mode LiveSpec only when an order-independent content hash (stored per output in live_query_state) changes. Unchanged scan results do not emit a snapshot, suppressing ripples in small or non-monotonic tables.

Use case: Non-monotonic or small tables scanned via poll (no watermark) need refresh semantics without the churn of repeated identical snapshots. Content-hash-based delivery suppresses ripples while ensuring consistency after real data changes.

Code: provisa/live/engine.py, provisa/live/reconcile.py

Tests: tests/unit/test_content_hash.py, tests/unit/test_live_engine.py, tests/unit/test_live_engine_full.py

REQ-993 · Live Delivery Strategy¶

Status: ✅ complete · Priority: MUST · Type: constraint

The content hash used for full-replace ripple suppression is order-independent (e.g., XOR or sorted multiset hash), stored per-output in live_query_state, and is recomputed on each full-table re-scan. A change in hash always triggers a replace snapshot delivery; an unchanged hash always suppresses delivery.

Use case: Order-independent hashing ensures that logical content changes (row addition/deletion) trigger snapshots even if rows appear in different order, while avoiding spurious ripples when only row order differs. Persistent storage in live_query_state allows next interval's scan to detect real changes.

Code: provisa/live/engine.py

Tests: tests/unit/test_content_hash.py

3. Federation Engines & Data Sources¶

REQ-994 · Data Sources¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Trino/Presto clusters are read as first-class remote SOURCES via SQLAlchemy's trino dialect and landed as REPLICA tables, reaching replication parity with GraphQL, REST, and gRPC sources. Source classification is by read mechanism, not data-at-rest residency; Trino qualifies as both a source and a federation engine, mirroring the dual roles of Snowflake, Databricks, and BigQuery.

Use case: Users need to query live Trino coordinators without deploying a separate federation engine. Trino-as-source unifies the source registration surface: a remote Trino cluster lands via the standard replication path, identical to web-service sources, enabling analytics workloads to federate without ETL.

Code: provisa/executor/drivers/registry.py, provisa-ui/src/pages/sources/constants.ts

Tests: tests/unit/test_trino_source_driver.py

REQ-995 · Federation Engine Capabilities¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Microsoft Fabric federation engine attaches Apache Iceberg data LIVE via OneLake Delta metadata virtualization (OPENROWSET FORMAT='DELTA' over virtualized/shortcut paths), extending live-attach reach from {parquet, csv, delta_lake} to include iceberg. Azure Synapse serverless lacks OneLake virtualization, so Iceberg tables are NOT attachable there and must land as REPLICA materialization.

Use case: Fabric's OneLake Delta virtualization layer enables direct live queries over Iceberg tables without requiring separate Iceberg compute. Synapse's absence of this virtualization necessitates materializing Iceberg data as REPLICA, preserving correctness while leveraging different engine strengths: Fabric for live lakehouse queries, Synapse for post-materialization analytics.

Code: provisa/federation/mssql_warehouse_connectors.py

Tests: tests/unit/test_mssql_warehouse_connectors.py

6. Cross-Source Data Movement¶

REQ-996 · Physical Data Transfer¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Support one-time physical data move expressed as standard SQL CREATE TABLE {schema}.{table} AS SELECT ... FROM {schema2}.{table2}. This physically creates and populates a table in a writable source. It does NOT create or register a Provisa model entity (no Domain, no Table model, no creation_request) — the new table is invisible to the federated model until separately introspected/registered.

Use case: High-speed cross-data-source bulk copy and migration: users express a one-time physical move as a single SQL statement over the federated namespace without external ETL orchestrator, per-source connector code, or intermediate file staging. The CTAS operates at two speed tiers: same-engine zero-copy native CTAS pushdown, and cross-engine bulk-load via the landing face bounded by target bulk-insert driver performance. No semantic model registration required. Primary derived pattern: snapshot/archive via scheduled CTAS with date templating (e.g., CREATE TABLE archive.orders_{{yyyymmdd}} AS SELECT * FROM live.orders on cron). Each run produces distinct table name, so global schema.table uniqueness (REQ-998) holds automatically with no collision/overwrite; snapshots remain unregistered, ideal for time-windowed archives.

Code: provisa/executor/ctas.py, provisa/executor/writable.py, provisa/federation/store_writer.py, provisa/pgwire/server.py

Tests: tests/unit/test_ctas.py

REQ-997 · Execution Routing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Execution routing for CTAS: resolve the write path via existing resolve_write_path(target_source_type, engine). If source and target are on the same engine, emit engine-native CTAS pushed down (zero-copy). If cross-engine, the engine runs the SELECT and rows are bulk-loaded into the target source via the landing write face (provisa/federation/store_writer.py) — the engine is never on the write path. Writes get no retry budget (provisa/executor/direct.py) to avoid duplicate side effects.

Use case: Routing strategy for high-speed bulk migration: same-engine CTAS achieves zero-copy pushdown; cross-engine routes through landing write face to isolate engine from write side effects. No retry budget prevents accidental duplicate table creation. This two-tier routing model is the performance guarantor for federation-wide bulk copy without intermediate staging.

Code: provisa/executor/ctas.py, provisa/executor/writable.py, provisa/federation/store_writer.py, provisa/executor/direct.py

Tests: tests/unit/test_ctas.py

REQ-998 · Name Resolution¶

Status: ✅ complete · Priority: MUST · Type: constraint

The load-bearing invariant: schema.table must be globally unique across the entire model. This is the language contract. A new CTAS target whose schema.table collides with an existing asset is rejected (name taken model-wide).

Use case: Provisa's catalog-free addressing relies on schema.table as the sole global identifier. Allowing collisions would create ambiguity in later reads: schema.table could refer to multiple physical tables across different sources, breaking the semantic model.

Code: provisa/executor/ctas.py

Tests: tests/unit/test_ctas.py

REQ-999 · Name Resolution¶

Status: ✅ complete · Priority: MUST · Type: constraint

Catalog is optional in the grammar and, when present, always names the source. It is never used for resolution when absent (schema.table uniqueness resolves it). When present, catalog MUST equal the source that schema.table resolves to — matches proceed (self-documenting), contradiction is an error, never an override. Grammar stays standard three-part catalog.schema.table so SQLGlot parses it natively (read placement off exp.Table.catalog); the model itself stays catalog-free.

Use case: Catalog in the SQL statement provides optional documentation/validation of placement intent without overriding the model's catalog-free resolution. Explicit catalog allows users to self-document which source they are writing to; mismatch is an error, preventing accidental writes to wrong sources.

Code: provisa/executor/ctas.py

Tests: tests/unit/test_ctas.py

REQ-1000 · Name Resolution¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Placement disambiguation at CREATE only: a new unique schema.table normally lands in the single writable source its schema maps to. In the rare case a schema maps to more than one writable source, catalog becomes required to disambiguate placement; absent an unambiguous target the CREATE is rejected (unique-match-or-error). Never silently pick a target.

Use case: Schemas can map to multiple writable sources in a federation (e.g., a schema with replicas on two different engines). Explicit catalog requirement prevents silent mis-targeting while allowing valid operations when placement is unambiguous.

Code: provisa/executor/ctas.py

Tests: tests/unit/test_ctas.py

REQ-1001 · Transaction Safety¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Transactional safety for the move: create-temp → bulk-load → atomic swap so a mid-stream failure leaves no partial target table.

Use case: Long-running or unreliable bulk-load operations may fail partway through. Atomic swap ensures target table is either completely populated or not created at all, preventing partial or corrupted data.

Code: provisa/federation/store_writer.py, provisa/executor/ctas.py

Tests: tests/unit/test_ctas.py

REQ-1002 · Type Coercion¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cross-engine result-schema → target DDL type coercion policy: the SELECT result schema must map to valid target-dialect DDL (handling engine types like ROW/array/DECIMAL scale). Reuse store_writer ensure_table where possible.

Use case: Source engines emit result schemas with types that may not be natively supported by target engines (e.g., Snowflake ROW types, complex arrays, variable-precision DECIMAL). Type coercion ensures that the target table DDL is valid and preserves logical semantics.

Code: provisa/federation/store_writer.py, provisa/executor/ctas.py, provisa/core/ir_types.py

Tests: tests/unit/test_ctas.py

REQ-1003 · Scheduled Execution¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Extend the APScheduler-backed ScheduledTrigger (provisa/core/models.py:640) to support execution of a user-supplied SQL statement on its cron schedule, not only webhook POST. When a trigger fires, the SQL statement executes against the federated engine. This enables scheduled CTAS snapshot/archive patterns (REQ-996).

Use case: Scheduled CTAS (CREATE TABLE ... AS SELECT) is the primary derived pattern from one-time CTAS (REQ-996), enabling snapshot and archive tables that update on a regular schedule without requiring external ETL orchestrators or manual user intervention. Users express scheduled snapshots as declarative triggers whose SQL statements run on cron.

Code: provisa/scheduler/jobs.py, provisa/scheduler/templating.py, provisa/core/models.py

Tests: tests/unit/test_scheduled_sql.py

REQ-1004 · Template Substitution¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Scheduled SQL statements support run-time substitution of date/timestamp tokens (e.g., {{yyyymmdd}}, {{YYYY-MM-DD}}, {{iso8601}}, {{timestamp}}) into the statement text before execution. Each run substitutes tokens with that run's execution date/time, enabling scheduled CTAS to produce distinct date-varied table names (e.g., archive.orders_{{yyyymmdd}}) without collision or overwrite, preserving the global schema.table uniqueness invariant (REQ-998) automatically. Depends on REQ-1003.

Use case: The snapshot/archive pattern requires each run to produce a uniquely-named table. Date tokens allow users to express target table names declaratively (e.g., archive.orders_{{yyyymmdd}}) so scheduled CTAS automatically creates archive.orders_20260712, archive.orders_20260713, etc. without manual collision detection or user-supplied name-generation logic. This is essential for autonomous time-windowed archives that preserve the model's global schema.table uniqueness (REQ-998) while remaining unregistered semantic entities.

Code: provisa/scheduler/templating.py

Tests: tests/unit/test_scheduled_sql.py

11. Platform, Infrastructure & Delivery¶

REQ-1005 · Desktop Installation¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Windows native tier first-launch (after setup completes and before/after opening the UI) presents next-steps guidance telling the user that the federation engine (Trino), observability stack, and demo data pack are NOT part of the native tier and are added via separate layered installers: Container installer (Provisa-Container-*.exe → WSL2 + containerd + Trino), then Obs installer (requires container tier), then Demo installer (requires Core + Obs). Guidance includes how to initialize the federation engine by running the Container installer.

Use case: The native tier installer provides a standalone local query engine (core + duckdb) but ships no federation engine (Trino), observability stack, or demo data. Users who install native tier receive no indication of how to access these optional features and the app opens directly to the UI without guidance, creating confusion about what Provisa can do. This requirement ensures users understand the tier model and how to extend their installation.

Code: packaging/windows/first-launch-native.ps1

Tests: tests/unit/test_desktop_install.py

REQ-1006 · Infrastructure¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The SPA static server routing is deterministic and driven by the browser Sec-Fetch-Dest request header (never by a maintained path allowlist). Requests with Sec-Fetch-Dest: document or GET whose Accept contains text/html are top-level SPA navigation and receive index.html; all other requests (fetch/XHR/EventSource, iframe subresource, non-GET) are proxied to the API and their real HTTP status — including 404 — is surfaced.

Use case: The previous path-allowlist heuristic silently served index.html (HTTP 200 HTML) for any unlisted API path, causing JSON parse errors and UI hangs on "Loading...". Deterministic header-driven routing eliminates both failure classes and prevents silent fallbacks. It also enables SPA deep-link refreshes on client routes without collision with API prefix handlers like /admin.

Code: provisa/ui_server.py

Tests: provisa-ui/e2e/infrastructure.spec.ts, tests/e2e/test_health_checks.py, tests/unit/test_ui_server_routing.py

REQ-1007 · Multi-Tier Deployment¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Deployment choices — federation engine (DuckDB/Trino/external-SQLAlchemy), observability mode (none/Docker-incluster/collector), and demo install — are exposed uniformly across all deployment surfaces (macOS SwiftUI, Linux AppImage, install.sh, Windows native, Helm, Terraform) with environment-variable overrides and interactive prompts where applicable.

Use case: Deployment parity ensures users can select the same configuration across desktop, server, container, and cloud deployments without relearning the choice model for each platform.

Code: packaging/linux/first-launch.sh, packaging/macos/build-dmg.sh, packaging/windows/first-launch-native.ps1, packaging/windows/install-container.ps1, install.sh, helm/provisa/values.yaml, helm/provisa/templates/demo.yaml, terraform/aws/main.tf, terraform/azure/main.tf, terraform/gcp/main.tf, terraform/deploy.sh

Tests: tests/unit/test_desktop_install.py

8. Client Access & Protocols¶

REQ-1008 · MCP Server¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa exposes a Model Context Protocol (MCP) server so external AI agents (Claude Desktop, claude.ai connectors, Claude Code) can discover catalog metadata and execute SQL through the same governed pipeline as every other client. The server is a protocol adapter with zero new governance logic — every SQL path routes through the existing choke point _govern_and_route(sql, role_id) in provisa/pgwire/_pipeline.py, and catalog reads reuse build_catalog_tables(state). It runs in-process in the FastAPI app (shared state) via the official mcp Python SDK (FastMCP). Two transports remote Streamable HTTP with OAuth (token→provisa role via existing pgwire OIDC) and local stdio with a pinned role for development. A role is REQUIRED on every call; the server never defaults to admin. Tools list_schemas, list_tables(schema), describe_table(schema,table), run_sql(sql) (row-limited/paged), explain_sql(sql) (plan-only), and search_catalog(nl_text). Structured drill-down (list_schemas → list_tables → describe_table) is deterministic over the meta views (registered_tables_meta, table_columns_meta) and needs no vector store. search_catalog is semantic bottom-up detail search using small-to-big retrieval embed narrow column-leaf chunks, match on the detail, then resolve up through deterministic drill-down to return the parent table branch (all columns + FKs) plus schema breadcrumb. Three chunk tiers schema (schema+desc+table names), table (table+desc+column names), column (column+desc+table+schema); all columns embedded. A single formatter get_chunk(schema, table?, column?) is the sole chunk-text contract address depth selects the tier (schema | schema+table | schema+table+column), column requires table, and it returns embed-text as labeled plain prose (NOT markdown — markdown chrome is token noise to the embedder; the pretty structured view is describe_table's job for a different consumer). A separate iter_entities() enumerator walks the meta views yielding addresses that drive get_chunk, keeping the formatter pure (address in, text out). The address IS the provenance the args (schema, table, column) equal the {level, schema_id, table_id, column_id} stored on the vector row, so a hit resolves straight back through get_chunk/describe_table with no separate chunk-id→entity mapping. Schema questions are never answered from stale chunk text. Because one formatter serves both initial index and incremental reindex, indexed and re-indexed text cannot drift. Vector store is DuckDB VSS, full stop — no second backend. The index runs in-process with the always-running Provisa server, alongside the meta views and pg_duckdb it derives from, so there is no external vector store to provision. DuckDB's HNSW index is memory-resident (disk persistence is still experimental), so it is treated as a server-lifetime artifact rebuilt/loaded at startup by the same walk-the-meta-views code path as incremental reindex (a cold build is just the full case). Sizing the index scales with schema cardinality, not row volume, and the catalog is curated, so it stays single-node across the range (fed ~10K, enterprise ~140K, largest curated+transaction ~1.4M column chunks) — well inside DuckDB VSS's range, which is why no pgvector/multi-million fallback is needed. Phase 2 (search_catalog semantic explore) is implemented with full rebuild-on-reload; incremental re-embed (keyed off MV/catalog refresh) remains a future optimization.

Use case: Makes Provisa pluggable into Claude and other MCP-speaking agents for governed metadata discovery and SQL execution, without exposing a second ungoverned surface. Extends the governed-surface-for-AI-planners motivation of REQ-884 from "queryable telemetry" to a first-class agent-facing protocol. Bottom-up search answers detail-topic questions ("which dataset holds the SOFR reset rate") that structured top-down navigation cannot at enterprise catalog scale where the flat table list exceeds an agent's context window.

Code: provisa/pgwire/_pipeline.py, provisa/api/flight/catalog.py, provisa/api/_meta_views.py, provisa/mv/refresh.py, provisa/api/mcp/search.py, provisa/api/mcp/tools.py, provisa/api/mcp/server.py

Tests: tests/unit/test_mcp_server.py, tests/unit/test_mcp_search.py, tests/unit/test_mcp_status.py

11. Frontend UI & UX¶

REQ-1009 · Component Library Standardization¶

Status: ✅ complete · Priority: MUST · Type: structural

Standardize provisa-ui on the Mantine component library (MIT/OSS) as the single modern UI component system. Remove unused @mui/material and Emotion dependencies (dead deps, imported in 0 source files).

Use case: A single, well-maintained component library provides consistent UX, reduces bundle size, and simplifies maintenance. Mantine's hooks-first architecture aligns with React best practices and TypeScript integration.

Code: provisa-ui/src/

Tests: tests/unit/test_ui_modernization_requirements.py

REQ-1010 · Design System¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Establish a consistent visual design theme driven by CSS-variable design tokens, mapped into Mantine's theme. Primary color is indigo (#6366f1); existing App.css token set is the source of truth.

Use case: CSS variables enable dynamic theming (light/dark), reduce redundancy, and provide a single point of truth for colors, spacing, and typography across the application.

Code: provisa-ui/src/App.css

Tests: tests/unit/test_ui_modernization_requirements.py

REQ-1011 · Theming¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Support both Dark and Light color schemes via Mantine's color-scheme system, with a user-facing theme toggle. Current state is dark-only.

Use case: Light mode improves accessibility for users with certain vision conditions; user choice increases accessibility and user satisfaction.

Code: provisa-ui/src/main.tsx, provisa-ui/src/theme/ColorSchemeToggle.tsx

Tests: provisa-ui/e2e/, tests/unit/test_ui_modernization_requirements.py

REQ-1012 · Internationalization¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Internationalize the entire frontend: no hardcoded user-facing strings (~536 exist today). Use react-i18next with an 'en' base catalog; include locale-aware date and number formatting. Pipeline must support adding additional locales.

Use case: i18n enables Provisa to serve global enterprises in their native languages, reducing adoption friction and supporting compliance requirements in non-English jurisdictions.

Code: provisa-ui/src/main.tsx, provisa-ui/src/i18n/

Tests: provisa-ui/e2e/, tests/unit/test_ui_modernization_requirements.py

REQ-1013 · Accessibility¶

Status: ✅ complete · Priority: MUST · Type: constraint

Meet enterprise accessibility requirements: WCAG 2.1 Level AA across the app. Includes focus management and focus-trap in dialogs, full keyboard operability, correct ARIA roles/attributes (e.g. role="dialog"/aria-modal on the 65 modal usages that currently lack them), 4.5:1 text contrast in both color schemes, visible focus indicators, reduced-motion support, form label/error association, and live regions for async notifications.

Use case: WCAG 2.1 AA is required for enterprise and government adoption; accessibility is a legal and ethical requirement, not optional.

Code: —

Tests: tests/unit/test_ui_modernization_requirements.py

REQ-1014 · Accessibility¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Accessibility is a tested, enforced requirement: add automated axe-core checks (@axe-core/playwright) into the Playwright e2e coverage fixture so every e2e run asserts zero accessibility violations.

Use case: Automated accessibility testing catches WCAG violations early, prevents regressions, and enforces compliance as a gate condition for test passage.

Code: provisa-ui/e2e/coverage.ts

Tests: provisa-ui/e2e/coverage.ts, tests/unit/test_ui_modernization_requirements.py

REQ-1016 · Testing¶

Status: ✅ complete · Priority: MUST · Type: constraint

Migrate the provisa-ui test suite (27 Playwright e2e specs + 22 unit/component tests) in lockstep with component modernization (REQ-1009/REQ-1012/REQ-1014). Stop relying on hand-rolled CSS-class selectors (59 locator('.…') uses) and hardcoded literal UI text (58 text-based assertions). Adopt role-based selectors (getByRole) plus stable data-testid attributes on migrated components; text assertions must reference i18n catalog keys (t('key')) rather than literal strings. Migration done per-component in the same pass as that component's Mantine/i18n/a11y conversion, keeping the suite green incrementally.

Use case: Mantine migration removes original classNames and i18n moves strings to catalogs, breaking CSS-class and hardcoded-text selectors. Role-based and data-testid strategies are robust to component library changes; i18n-key assertions reflect the requirements contract. Incremental per-component migration prevents big-bang test breakage and ensures continuous test coverage.

Code: provisa-ui/e2e/, provisa-ui/src/

Tests: provisa-ui/e2e/, tests/unit/test_ui_modernization_requirements.py

11. Platform, Infrastructure & Delivery¶

2. Authentication & Identity¶

REQ-1035 · Demo Tiers & Onboarding¶

Status: 💡 proposed · Priority: MUST · Type: behavioral

REQ-1058 · Public Data Publishing¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

Public-publish anonymous access: unauthenticated requests carry no tenant_id claim; the org is resolved from Host subdomain only, and the request is bound to a built-in read-only 'public' role scoped to that org. TenantMiddleware gains a Host-based resolution path for the no-claim case instead of returning 401.

Use case: Enables orgs to publish datasets anonymously without requiring end-user authentication. Public role allows intentional, opt-in data sharing without building separate anonymous infrastructure. Distinct from the Firebase-anonymous sandbox uid (REQ-1034).

Code: —

Tests: —

REQ-1059 · Public Data Publishing¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

Publishing is a governance grant, not new machinery: each org has a built-in 'public' role starting with ZERO visibility. Publishing an asset means granting that role visibility to a specific table/view; access then flows through existing visibility/RLS/masking (REQ-039/040) uniformly across GraphQL/SQL/pgwire/Bolt. Nothing is public by default; opt-in and explicit.

Use case: Reuses existing access-control infrastructure (visibility, RLS, masking) uniformly across all surfaces. Eliminates special-case code for public access. Clear opt-in model prevents accidental data leaks.

Code: —

Tests: —

REQ-1060 · Public Data Publishing¶

Status: 💡 proposed · Priority: MUST · Type: constraint

Tests: tests/unit/test_encryption_providers.py

REQ-1087 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

AWS KMS encryption provider (envelope wrapping via KMS Encrypt/Decrypt) with support for custom endpoint_url for KMS-compatible private endpoints. Available whenever boto3 is present.

Use case: Enterprises can use AWS-managed KMS keys without re-implementing envelope encryption logic. Private endpoint support enables airgapped deployments.

Code: provisa/encryption/providers/aws_kms.py

Tests: tests/unit/test_encryption_providers.py

REQ-1088 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

HashiCorp Vault (Transit engine) encryption provider with auto-availability when hvac package is installed. Vault address is configurable as a custom endpoint.

Use case: Enterprises using Vault can leverage existing HSM or encryption backends without provisioning cloud KMS. Custom Vault endpoint supports private/airgapped deployments.

Code: provisa/encryption/providers/vault.py

Tests: tests/unit/test_encryption_providers.py

REQ-1089 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Azure Key Vault encryption provider (RSA-OAEP-256 key wrap) with auto-availability when azure-keyvault-keys package is installed.

Use case: Azure-native deployments can leverage customer-managed keys in Azure Key Vault without running separate KMS infrastructure.

Code: provisa/encryption/providers/azure_keyvault.py

Tests: tests/unit/test_encryption_providers.py

REQ-1090 · Encryption¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Enterprises can register custom encryption providers (e.g., in-house KMS/HSM endpoints) without core changes via register_encryption_provider() API, PROVISA_ENCRYPTION_PROVIDER_MODULES environment variable, or provisa.encryption_providers entry-point group.

Use case: Organizations with proprietary KMS or HSM deployments can integrate them without maintaining a Provisa fork.

Code: provisa/encryption/registry.py

Tests: tests/unit/test_encryption_providers.py

10. Admin Console & Governance¶

REQ-1091 · Admin Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Admin encryption tab derives its provider list live from the encryption registry. Built-in and custom providers appear automatically with their declared config fields. Unavailable providers are shown but not selectable.

Use case: Operators can configure encryption without code changes; custom providers are discoverable in the UI without redeploy.

Code: —

Tests: tests/unit/test_admin_config_tabs.py, provisa-ui/src/__tests__/EncryptionTab.test.tsx

11. Frontend UI & UX¶

REQ-1092 · Catalog Explore¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Provisa provides an LLM chatbot at /explore (under the Explore nav group, branded as "Assistant") where users converse in natural language to explore and query the federated catalog. The interface is built with @chatscope/chat-ui-kit-react. A Claude model (default claude-opus-4-8, configurable via ai_models.mcp_chat) drives a manual tool-use loop with governed MCP tools—list_schemas, list_tables, describe_table, search_catalog, run_sql, explain_sql—via provisa/api/mcp/chat.py. Every tool executes under the caller's fixed role (from x-provisa-role), so the assistant respects domain access + governance boundaries; the model cannot choose or escalate the role. The backend streams assistant text and tool activity to the browser as Server-Sent Events via POST /admin/mcp/chat.

Use case: Users can explore and query the federated catalog through conversational AI, discovering tables/columns and running SQL through natural language without schema knowledge, constrained by their role's visibility. The assistant is bound by the same governed pipeline as any client.

Code: provisa/api/mcp/chat.py, provisa/api/mcp/status.py, provisa-ui/src/pages/McpExplorePage.tsx

Tests: tests/unit/test_mcp_chat.py

3. Source Registration & Data Modeling¶

REQ-1093 · Unique Constraints¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A registered Table carries a table-level list of UNIQUE constraints (unique_constraints: [{name, columns[]}]), each an ordered column set that may be single-column or composite. On registration Provisa introspects declared UNIQUE constraints from RDB sources (PostgreSQL/MySQL information_schema.table_constraints + key_column_usage grouped by constraint_name and ordered by ordinal_position; SQLite PRAGMA index_list/index_info with origin='u') and seeds the list. Uniqueness is only recorded from a source-declared constraint, never inferred from sampled data. The admin UI exposes an expandable "Uniques" panel in the register/edit table forms: a "+ Constraint" control adds a row with a Name text field and a multi-select checkbox of the table's columns. pgwire emits each unique constraint as a pg_constraint row with contype 'u' (conkey = ordered attnums, no confrelid) and projects it into information_schema.table_constraints (constraint_type 'UNIQUE') and key_column_usage, so standard clients (DBeaver, psql \d) render the keys. The MCP describe_table tool includes the table's unique_constraints alongside foreign_keys, role-gated.

Use case: Foreign keys can legitimately target any unique key, not just the primary key; surfacing declared unique constraints improves relationship inference, lets the query planner reason about row identity, gives standard SQL clients and the MCP agent the natural keys users actually join and filter on, and makes the ERD/catalog show natural keys instead of only surrogate PKs.

Code: provisa/core/models.py, provisa/discovery/fk_introspect.py, provisa/pgwire/catalog_constraints.py, provisa/api/mcp/tools.py, provisa/api/admin/discovery_schema.py, provisa/compiler/context.py, provisa-ui/src/components/admin/UniquesPanel.tsx, provisa-ui/src/pages/tables/RegisterTableForm.tsx, provisa-ui/src/pages/tables/TableEditForm.tsx

Tests: tests/unit/test_unique_introspect.py, tests/unit/pgwire/test_unique_constraints.py, tests/unit/test_unique_constraints_wiring.py, provisa-ui/src/components/admin/__tests__/UniquesPanel.test.tsx

8. Client Access & Protocols¶

REQ-1094 · PostgreSQL Catalog Compatibility¶

Status: ✅ complete · Priority: MUST · Type: structural

Composite foreign keys are reconstructed at pgwire catalog-build time from the FK→PK invariant. Provisa's relationship model stores each FK column as a separate join with no shared constraint ID; single-column many-to-one joins between the same source/target table pair whose target columns together form the target table's composite primary key are collapsed into one pg_constraint FK row (contype 'f') with multi-element conkey/confkey arrays, ordered by target PK column order. Joins not covering a composite PK (single-column FKs, partial coverage, independent multi-FKs to the same table) remain one single-column row each.

Use case: PostgreSQL clients and tools (e.g., DataGrip, psql) expect composite FKs as single pg_constraint rows with parallel conkey/confkey arrays; reconstructing them ensures correct relationship visualization and validation logic without requiring a V1-prohibited migration.

Code: provisa/pgwire/catalog_constraints.py

Tests: tests/unit/pgwire/test_composite_fk_and_index.py

REQ-1095 · PostgreSQL Catalog Compatibility¶

Status: ✅ complete · Priority: MUST · Type: structural

pg_catalog.pg_index is populated with one row per primary-key and UNIQUE constraint (indrelid = table oid, indkey = ordered key attnums, indisprimary/indisunique flags set). This allows PostgreSQL clients (e.g., DataGrip) that resolve key columns via pg_index.indkey rather than pg_constraint to discover the correct columns via the standard pg_index → pg_attribute join.

Use case: Standard SQL clients resolve primary key and unique constraint columns by joining pg_index to pg_attribute on indkey; populating pg_index ensures clients that use this path (rather than pg_constraint) get correct key column visibility and metadata.

Code: provisa/pgwire/catalog_constraints.py

Tests: tests/unit/pgwire/test_composite_fk_and_index.py

10. UI & Admin Surfaces¶

REQ-1096 · Admin & Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Live config export/diff/patch — an OPT-IN admin capability, distinct from the plain download/upload of REQ-164, that lets an admin see and export exactly what changed since startup. The on-disk config is only the boot seed; admin mutations write to the control-plane DB, so the file goes stale (an MV or relationship created in the UI, or a changed primary key, never appears in it). When enabled, Provisa rebuilds the DB-backed config sections (tables incl. views/MVs, relationships, roles, rls_rules, domains) from live state and overlays them on the file base to produce the CURRENT config. The baseline for diff/patch is a BOOT SNAPSHOT captured ONCE at the end of boot — after all runtime auto-derivation (FK tracking, graphql-remote registration) — so the diff shows only changes made SINCE startup, not derived entities that were never in the file. This is the "lock the starting config after automated changes" behavior: the automated/derived setup is frozen as the baseline, and every subsequent admin edit shows as a clean diff against it. Both sides are normalized identically (deep key-sort + stable entity-sort) so section/key/entity ORDER never differs and the diff surfaces only real changes. The admin UI (Configuration File section) exposes: View Diff (a two-pane diff of baseline vs current, editable), Download Patch (a git-apply/patch-compatible unified diff from baseline to the curated current, for CI/CD reproduction), and Apply Revised (upload the curated config and reload). When the flag is on, uploaded config is normalized on consume and persisted in normalized form so the on-disk file stays byte-faithful to the patch baseline; when off, the file is written verbatim to preserve a hand-authored file's comments/ordering. The feature is coherent only when the generated/normalized config is canonical (config built from installer/demo choices), NOT a hand-authored file whose comments/ordering a normalized patch could not stay faithful to — so it is GATED behind the live_config_export flag (config key live_config_export: true OR env PROVISA_LIVE_CONFIG_EXPORT in {1,true,yes}), exposed to the UI as features.live_config_export, which conditionally renders the View Diff / patch controls. In DEMO MODE the flag MUST be ON: the demo is the canonical-generated-config scenario, so demo builds force live_config_export regardless of an explicit config key.

Use case: Admins running the canonical/demo config need to see exactly what they changed in the UI since startup and export it as a patch for version control or CI/CD, instead of hand-diffing a stale on-disk file that never reflected DB-backed edits. Locking the boot snapshot after automated derivation keeps the diff free of noise. Demo users get the diff/patch surface out of the box because the demo config is generated and therefore always safe to normalize.

Code: provisa/api/app.py, provisa/api/app_startup.py, provisa/api/admin/config_export.py, provisa/api/admin/settings_router.py, provisa-ui/src/pages/AdminPage.tsx, provisa-ui/src/components/admin/ConfigDiffView.tsx

Tests: tests/unit/test_config_export.py, tests/integration/test_settings_router_api.py

7. Connectors & Source Adapters¶

REQ-1097 · Test Coverage¶

Status: ✅ complete · Priority: MUST · Type: constraint

Every datasource type in SourceType enum (49 types: postgresql, mysql, mariadb, sqlserver, oracle, cockroachdb, yugabytedb, tidb, greenplum, singlestore, snowflake, bigquery, databricks, fabric, synapse, redshift, clickhouse, elasticsearch, pinot, druid, exasol, delta_lake, iceberg, hive, hive_s3, mongodb, cassandra, redis, firebird, duckdb, sqlite, openapi, graphql_remote, grpc_remote, govdata, kafka, neo4j, sparql, prometheus, rss, websocket, sharepoint, splunk, google_sheets, csv, parquet, files, airport, ingest) must have an integration or e2e test against a real (Docker or cloud-provisioned) instance. Unit tests that merely reference the type name do not satisfy this.

Use case: Ensures all datasource connectors work correctly in practice against real instances, not just as concept or identity-check in code. Covered with self-provisioning tests: direct-driver — mariadb, tidb, cockroachdb, yugabytedb, sqlserver, oracle, singlestore (license-gated skip); connector (federation engine → Trino catalog) — cassandra (reference pattern), exasol (arch-gated: amd64-only image), redshift + sharepoint (creds/instance-gated, no self-provision); DuckDB community extension — firebird, airport (live vs zaychik), google_sheets (opt-in gated: DuckDB gsheets extension SA-auth, PROVISA_GSHEETS_LIVE=1); Calcite/Trino — splunk (live, self-provisioned). Fixed a product bug found while covering exasol: Source.jdbc_url() omitted exasol/redshift, silently no-op'ing catalog creation. Filed issue #74 (zaychik Arrow-Flight auth/protocol gap, found via airport; the Provisa Flight→Trino transport itself is verified working — tests/integration/test_trino_flight_engine_e2e.py). The heavy multi-container analytics/lake ecosystems are now covered by self-provisioning e2e: pinot and druid (Trino catalog), hive and hive_s3 (Trino hive connector + HMS, S3 on MinIO). kudu and accumulo were removed from SourceType — neither is a SQL-variant with a downloadable driver testable in this harness (Trino 481 ships no kudu/accumulo connector).

Code: provisa/core/models.py, provisa/federation/trino_connectors.py

Tests: tests/integration/test_mariadb_source_e2e.py, tests/integration/test_tidb_source_e2e.py, tests/integration/test_cockroachdb_source_e2e.py, tests/integration/test_yugabytedb_source_e2e.py, tests/integration/test_greenplum_source_e2e.py, tests/integration/test_sqlserver_source_e2e.py, tests/integration/test_oracle_source_e2e.py, tests/integration/test_singlestore_source_e2e.py, tests/integration/test_cassandra_source_e2e.py, tests/integration/test_firebird_source_e2e.py, tests/integration/test_exasol_source_e2e.py, tests/integration/test_redshift_source_e2e.py, tests/integration/test_sharepoint_source_e2e.py, tests/integration/test_google_sheets_source_e2e.py, tests/integration/test_airport_source_e2e.py, tests/integration/test_splunk_source_e2e.py, tests/integration/test_pinot_source_e2e.py, tests/integration/test_druid_source_e2e.py, tests/integration/test_hive_source_e2e.py, tests/integration/test_hive_s3_source_e2e.py, tests/integration/test_trino_flight_engine_e2e.py

11. Platform, Infrastructure & Delivery¶

REQ-1098 · Native Tier Database¶

Status: ✅ complete · Priority: MUST · Type: structural

The native (DuckDB) tier exposes the tenant control-plane as the provisa_admin catalog, enabling meta/ops entities (registered_tables, etc.) to be queryable with parity to the Trino tier. The catalog attaches the tenant SQLite DB READ_ONLY and exposes every table under the compiler-emitted schema org_<id> without a hardcoded table list.

Use case: Provides full parity with Trino tier where provisa_admin is a real attached catalog. Allows native users to query system metadata and operational state via standard SQL, matching Trino/Presto multi-catalog semantics.

Code: provisa/federation/duckdb_runtime.py, provisa/federation/native_backend.py

Tests: tests/unit/test_duckdb_control_plane_attach.py

REQ-1099 · Native Tier Database¶

Status: ✅ complete · Priority: MUST · Type: constraint

The control-plane SQLite (platform + tenant) MUST run in WAL mode (journal_mode=WAL) with busy_timeout enabled. This is the concurrency guarantee that permits the native DuckDB engine to read the tenant DB READ_ONLY while aiosqlite writes config changes without transient "database is locked" errors.

Use case: Enables concurrent read-write access to the control-plane SQLite without lock contention or stalling queries. Required for safe multi-client scenarios where the UI (aiosqlite) updates config while the native engine queries system state.

Code: provisa/core/database.py

Tests: tests/unit/test_admin_database.py

REQ-1100 · Desktop Installation¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

The native demo launcher MUST wait for the demo mock servers (petstore :18080, graphql :4000) to answer before starting the API. The launcher MUST gate the browser-open on the API's /health endpoint (served only after lifespan/config-load completes), not merely on the UI proxy port.

Use case: Ensures the demo environment is fully initialized before opening the browser to the user. Prevents premature browser navigation to a not-yet-ready environment and reduces failed first-user experience.

Code: packaging/windows/provisa-native.ps1

Tests: tests/unit/test_desktop_install.py, packaging/windows/tests/provisa-native.Tests.ps1

REQ-1101 · Desktop MCP Configuration¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Native/desktop tier enables remote MCP server by default, bound to loopback (PROVISA_MCP_HOST=127.0.0.1), with the governed role pinned explicitly (native auth:none = one local admin). All settings overridable via environment variables.

Use case: Provides secure local MCP connectivity for Claude Desktop on native installations without additional configuration, enabling tool use against local Provisa queries.

Code: provisa/api/mcp/server.py, packaging/windows/provisa-native.ps1, scripts/provisa

Tests: tests/unit/test_mcp_status.py

REQ-1102 · MCP Status Endpoint¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The MCP status endpoint returns a resolved, proxy-aware, editable connect URL derived from X-Forwarded-Host/Host headers plus PROVISA_MCP_PORT and /mcp path; PROVISA_MCP_EXTERNAL_URL overrides. Returns null when MCP is disabled.

Use case: Enables UI to display and allow editing of the MCP connection URL, accounting for reverse proxies and external domains that the server cannot reliably infer.

Code: provisa/api/mcp/status.py

Tests: tests/unit/test_mcp_status.py

REQ-1103 · Explore MCP UI Panel¶

Status: ✅ complete · Priority: MUST · Type: ui

The Explore/MCP surface provides a "Connect Claude Desktop" panel that emits a copy-ready claude_desktop_config.json entry with URL prefilled and editable.

Use case: Simplifies Claude Desktop setup by providing a complete, copy-paste configuration entry that users can add to their Claude Desktop config file.

Code: provisa-ui/src/pages/McpExplorePage.tsx

Tests: tests/unit/test_mcp_status.py

REQ-1104 · Claude Desktop Bridge¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

The Claude Desktop connector is Node-free: mcp-proxy (Python stdio-to-Streamable-HTTP bridge) is bundled into the native standalone runtime. The emitted config's command references the bundled interpreter (PROVISA_MCP_BRIDGE_COMMAND). Feature is gated to native tier only; off-tier the panel degrades to "install mcp-proxy yourself". No npx, no Node, no user pip, fully airgapped.

Use case: Eliminates Node.js and npm as dependencies for Claude Desktop MCP connectivity in native installations, reducing distribution complexity and supporting offline/airgapped environments.

Code: packaging/windows/build-sfx.ps1, packaging/macos/build-dmg.sh, pyproject.toml, provisa/api/mcp/status.py, packaging/windows/provisa-native.ps1, scripts/provisa

Tests: tests/unit/test_mcp_status.py

REQ-1105 · MCP Authentication¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Wire resolve_token_role into the FastMCP Streamable HTTP transport to map remote bearer tokens to Provisa roles, enabling secure per-caller MCP authentication before exposing MCP off the loopback.

Use case: Prerequisite for safely exposing MCP to remote callers with proper role-based access control and governance.

Code: provisa/api/mcp/server.py

Tests: tests/unit/test_mcp_bearer_role.py

REQ-1106 · MCP Transport¶

Status: ✅ complete · Priority: MUST · Type: constraint

MCP local-connector transport supports plain HTTP by default with optional TLS (PROVISA_MCP_TLS=1 for public deployment). Per-machine self-signed certificates are generated on first start with OS user-store trust and http fallback if cert creation fails. When MCP binds non-loopback (0.0.0.0), DNS-rebinding Host-header protection is disabled (stdio bridges are not subject to browser-origin attacks); loopback binds retain strict protection.

Use case: Enables local Provisa MCP servers to be reached by Claude Desktop via mcp-proxy stdio bridge without requiring bundled TLS. Supports optional TLS for publicly-exposed deployments while fixing 421 Misdirected Request errors when clients connect via the machine's real IP (e.g. WSL VM IP).

Code: provisa/api/mcp/server.py, provisa/api/mcp/tls.py, provisa/api/mcp/status.py

Tests: tests/unit/test_mcp_status.py

8. Client Access & Protocols¶

REQ-1120 · Served Protocol Surface¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Provisa serves the DuckDB Airport Arrow-Flight protocol as an outbound data-as-a-service surface, allowing external DuckDB clients with the airport community extension to ATTACH Provisa (TYPE AIRPORT) and query its governed federated catalog with server-side governance (RLS/masking) applied.

Use case: DuckDB users can integrate Provisa's federated catalog directly into DuckDB workflows without separate ETL, leveraging DuckDB's in-process analytics while Provisa enforces row-level and column-level access control server-side. This peer protocol surface complements existing bolt (Neo4j), pgwire (Postgres), and Arrow Flight SQL surfaces.

Code: provisa/api/airport/server.py, provisa/api/airport/wire.py, provisa/api/airport/query.py, provisa/api/airport/service.py, provisa/api/app_startup.py

Tests: tests/integration/test_airport_service_e2e.py

REQ-1121 · Query Pushdown & Optimization¶

Status: ✅ complete · Priority: MUST · Type: constraint

The DuckDB airport extension delivers filter and projection pushdown via the endpoints DoAction parameters map (json_filters JSON expression tree + column_ids indices). The server translates these to semantic WHERE + projection, folds them into the DoGet ticket, and injects them through _govern_and_route so sources apply filters/projection with RLS/masking layered on.

Use case: Pushdown minimizes network transfer and leverages source query optimization while maintaining governance invariants. Conformance rule: DoGet must return the FULL advertised schema; projected-out columns are null-filled client-side and DuckDB projects accordingly.

Code: provisa/api/airport/pushdown.py, provisa/api/airport/server.py

Tests: tests/integration/test_airport_service_e2e.py

REQ-1122 · Access Control & Projection¶

Status: ✅ complete · Priority: MUST · Type: constraint

Airport catalog scan projects only registered semantic columns per role, never SELECT *. Unregistered physical columns (e.g. audit created_at) are neither leaked on read nor sent as NULL on INSERT.

Use case: Ensures column-level access control is enforced at the protocol boundary and physical schema details remain hidden from federated consumers, preventing accidental data leakage or schema coupling.

Code: provisa/api/airport/service.py, provisa/api/airport/query.py

Tests: tests/integration/test_airport_service_e2e.py

REQ-1123 · Transaction Management¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Airport transactions use a real in-memory coordinator (provisa/api/airport/transactions.py). create_transaction mints a UUID identifier; get_transaction_status reports active/committed/aborted. Best-effort read-committed isolation: each governed mutation auto-commits.

Use case: Provides transaction identity and status tracking for DuckDB clients performing multi-statement workflows against a governed federated catalog without requiring persistent transaction state across connections.

Code: provisa/api/airport/transactions.py, provisa/api/airport/service.py

Tests: tests/integration/test_airport_service_e2e.py

REQ-1124 · Mutation Enforcement¶

Status: ✅ complete · Priority: MUST · Type: constraint

Governed DML (INSERT / UPDATE / DELETE) via do_exchange decodes the Arrow stream and submits semantic SQL through _compile_govern_execute (the single governed write pipeline), gated on engine writability (is_writable_on / writable.py). Never parallel writers. UPDATE/DELETE key rows by the table's PRIMARY KEY, advertised as the airport row identity (is_rowid pseudo-column filled with the JSON-encoded PK tuple), so the WHERE is formed only from PKs RLS already let the role read and is RLS-filtered again through the pipeline — a role can never mutate a row it cannot see (the attempt is a governed no-op).

Use case: Ensures all mutations flow through the unified governance pipeline with consistent RLS/masking and audit enforcement, preventing bypass via direct insertion or uncontrolled parallel writes.

Code: provisa/api/airport/service.py, provisa/api/airport/query.py, provisa/api/airport/server.py

Tests: tests/integration/test_airport_service_e2e.py

REQ-1125 · Protocol Compliance¶

Status: ✅ complete · Priority: MUST · Type: constraint

Operations refused with a correct Flight protocol error (never a silent no-op). UPDATE/DELETE ARE supported via PRIMARY-KEY row identity: the table's PK is advertised as the airport row identity (is_rowid pseudo-column filled with the JSON-encoded PK tuple) so DuckDB echoes it back on update/delete; the server forms a governed WHERE PK IN (...) submitted through the one write pipeline (RLS/masking/writable-ACL apply). They are refused ONLY per-table, when a table has no primary key (no safe row identity). create_table IS supported: it creates the physical table in the target domain's single writable source (schema-mutation pipeline) and registers it, so the new table joins the catalog and accepts a governed INSERT; refused only when the domain has no writable source bound (or maps to more than one — ambiguous). column_statistics + table_function_flight_info are refused (governed federation has no precomputed stats and advertises no table functions). Physical column/struct ALTERs (add_column/rename_table/add_field/ …) are refused (they need the admin schema-mutation API's source/governance context absent from the airport DDL payload). DDL create_schema/drop_schema map to Provisa domains via the control-plane domain repository.

Use case: Maintains protocol integrity and a consistent governance model: safe, governed operations (PK UPDATE/DELETE, create_table into a writable source) are served through the one pipeline, while operations that cannot be safely federated are rejected — preventing data inconsistency, uncontrolled mutations, or governance bypass.

Code: provisa/api/airport/server.py, provisa/api/airport/query.py

Tests: tests/integration/test_airport_service_e2e.py

9. Deployment & Distribution¶

REQ-1126 · PyPI Distribution¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Provisa ships as a pip-installable wheel via pip install provisa[embedded]. The wheel contains a fully functional embedded Provisa system (SQLite control plane + embedded DuckDB engine) ready to run immediately: provisa run. Supply-chain trust follows existing Artifactory-as-PyPI models already approved and CVE-scanned by regulated enterprises.

Use case: Airgapped/regulated enterprises already trust Artifactory-as-PyPI supply chains (whitelisted, CVE-scanned, hash-pinned via uv.lock) but cannot approve new registries (Docker Hub, notarization services, custom mirrors). Wheel distribution avoids new approval gates.

Code: pyproject.toml, provisa/cli.py, scripts/build-wheel.sh

Tests: tests/e2e/test_pypi_packaging.py

REQ-1127 · UI Packaging¶

Status: ✅ complete · Priority: MUST · Type: structural

React UI is precompiled into the wheel. CI runs vite build (from provisa-ui/package.json), embeds the output into the package tree (e.g. provisa/_ui/), and declares it in [tool.setuptools.package-data] so python -m build includes it. ui_server.py serves from the packaged directory path rather than file.parent.parent/static, eliminating the need for npm/Node at runtime or post-install. The runtime config/ files are staged into provisa/_config/ the same way so the wheel is fully self-contained.

Use case: Airgapped environments have no npm/Node installed. Packaging the precompiled UI into the wheel ensures a self-contained delivery that can be deployed to systems with only Python >=3.12.

Code: provisa/ui_server.py, provisa/core/desktop_profile.py, provisa/pg_extensions/catalog.py, scripts/build-wheel.sh, pyproject.toml

Tests: tests/e2e/test_pypi_packaging.py

REQ-1128 · Console Entry Point¶

Status: ✅ complete · Priority: SHOULD · Type: structural

A provisa console entry point (provisa = provisa.cli:main) is added to [project.scripts] in pyproject.toml, allowing users to launch the system via the provisa run command at the shell without explicitly calling a module or uvicorn.

Use case: Improves user experience and follows Python packaging conventions; users expect a single command (provisa run) rather than python -m invocations.

Code: provisa/cli.py, pyproject.toml

Tests: tests/e2e/test_pypi_packaging.py

REQ-1129 · Embedded Profile Dependencies¶

Status: ✅ complete · Priority: MUST · Type: constraint

The [embedded] optional dependency extra pins SQLite control plane + embedded DuckDB engine (REQ-979: native/desktop tier). The bundled Trino JVM is omitted (PyPI cannot carry a JVM server). Full multi-engine federation is preserved by pointing to a customer-provided ("bring your own") external engine via TRINO_HOST/TRINO_PORT env or federation_engine_host/port config (already supported: provisa/federation/engine.py:1201, provisa/core/desktop_profile.py:153). Embedded DuckDB is the default when no external engine is configured.

Use case: Defines a complete, runnable tier for airgapped enterprises while preserving federation capability via bring-your-own external engines. Omitting the JVM avoids GPL licensing complications and PyPI size/upload limits.

Code: provisa/federation/engine.py, provisa/core/desktop_profile.py, pyproject.toml

Tests: tests/e2e/test_pypi_packaging.py

REQ-1130 · Mirror Preparation¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Python language pin: >=3.12,<3.13. Airgap Artifactory mirrors must pre-seed cp312 platform wheels for all native-wheel dependencies: duckdb, chdb, pyodbc, pyarrow, grpcio, cryptography, bcrypt, psycopg2-binary, snowflake-connector-python, etc. A requirements.lock and "mirror these wheels" checklist is delivered so the customer's Artifactory admin can prepare in one pass.

Use case: Regulated enterprises run PyPI mirrors (Artifactory, Nexus) with whitelisted wheels. Pre-seeding the checklist avoids deployment surprises and allows the admin to verify all transitive deps are approved before go-live.

Code: pyproject.toml, packaging/pypi/requirements-embedded.lock, packaging/pypi/MIRROR-CHECKLIST.md

Tests: tests/e2e/test_pypi_packaging.py

1. Access Governance & Security¶

REQ-1131 · Column Visibility Control¶

Status: ✅ complete · Priority: MUST · Type: constraint

A role with ADMIN capability bypasses per-column visible_to restrictions and always sees every column. The V003 column-visibility check in stage2.py implements this admin override.

Use case: Ensures admins can fully inspect and manage the system without being restricted by user-facing visibility policies that apply to ordinary roles.

Code: provisa/compiler/stage2.py

Tests: tests/unit/test_stage2.py, tests/unit/test_governance.py

REQ-1132 · Semantic-Layer Meta Visibility¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Metadata visibility is tiered by role. The meta domain exposes two column classes: CORE (table_name, column_name, data_type, description, alias, schema_name, domain_id, keys, relationship endpoints) and GOVERNANCE (visible_to, unmasked_to, writable_by, masking rules, view_sql, native_filter_type, scope). DEFAULT tier (every role): sees CORE meta for directly-accessible tables plus 1-hop relationship neighbors (user-defined/semantic relationships only, excluding synthetic catalog edges), bidirectional. Meta DOMAIN GRANT: sees CORE meta for all tables and relationships. Per-relationship opt-out suppresses target metadata from DEFAULT discovery.

Use case: Enables governed discovery over the reachable neighborhood without exposing data. Separates catalog structure knowledge (CORE) from governance configuration (GOVERNANCE), allowing each tier explicit control.

Code: provisa/security/rights.py, provisa/compiler/stage2.py, provisa/core/models.py, provisa/core/schema_org.py, provisa/core/repositories/relationship.py, provisa/api/app.py

Tests: tests/unit/test_meta_governance_visibility.py

REQ-1133 · Domain Access Control¶

Status: ✅ complete · Priority: MUST · Type: constraint

The ops domain behaves like any ordinary data domain: roles must be explicitly granted rights to access it. There is no implicit access to ops tables; admins bypass this via the admin capability override.

Use case: Prevents unintended exposure of operational/internal tables. Each role must be explicitly granted access according to their responsibility.

Code: provisa/api/startup_seed.py

Tests: tests/unit/test_ops_domain.py

REQ-1134 · Governance Column Visibility¶

Status: ✅ complete · Priority: MUST · Type: constraint

The view_governance capability is required to view GOVERNANCE columns in the meta domain (visible_to, unmasked_to, writable_by, mask_type, mask_pattern, mask_replace, mask_value, mask_precision, view_sql, native_filter_type, scope). A role with the meta domain grant does NOT automatically see GOVERNANCE columns; view_governance must be granted independently.

Use case: Separates discovery (CORE meta) from administration/audit (GOVERNANCE meta). Allows data stewards to see governance configuration without granting broad meta-domain access, and vice versa.

Code: provisa/security/rights.py, provisa/compiler/schema_gen.py, provisa/compiler/stage2.py

Tests: tests/unit/test_meta_governance_visibility.py

11. Platform, Infrastructure & Delivery¶

REQ-1135 · Licensing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The installation records a tamper-evident first-use timestamp that survives reinstall. On first startup with no existing anchor, the current date is recorded as first_seen. The anchor payload {first_seen, machine_id} is Ed25519-signed and written to multiple locations outside the application install directory (e.g. ~/.provisa, an OS-native store, and a secondary marker). On every startup the system reads all anchors, takes the earliest (min) first_seen, and rewrites any missing anchor with that value. No network access is required — the mechanism is fully airgapped and time-based.

Use case: Enables a 30-day trial clock that cannot be reset by uninstalling and reinstalling the application, without any license server or phone-home.

Code: provisa/licensing/anchors.py, provisa/licensing/keys.py, provisa/licensing/machine_id.py

Tests: tests/unit/test_licensing.py

REQ-1136 · Licensing¶

Status: ✅ complete · Priority: MUST · Type: constraint

The trial-expiry evaluation is resistant to wall-clock rollback. The system persists a monotonic wall-clock high-water mark and evaluates elapsed time as max(now, high_water) - first_seen. The trial is considered expired when elapsed >= 30 days and no valid license is present. Setting the system clock backward does not extend the trial.

Use case: Prevents a user from evading the nag by moving the machine clock backward.

Code: provisa/licensing/monotonic.py

Tests: tests/unit/test_licensing.py

REQ-1137 · Licensing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

After the trial expires and no valid license is present, a self-contained nag message is surfaced on every access surface. It is emitted in the UI as a persistent shell banner, and through each served protocol's native non-fatal notice channel only (pgwire NoticeResponse, Bolt SUCCESS-metadata notifications, Arrow Flight app_metadata, gRPC trailing metadata, REST X-Provisa-License-Notice header plus warnings[] envelope, MCP notifications/message). The nag never modifies the result body, row stream, or any schema-typed field; if a protocol has no safe out-of-band channel the nag is skipped for that protocol. Emission is rate-limited to once per session/connection. The message states that the license is FREE and is used only to gather usage information because Provisa collects no telemetry, that the trial period has elapsed, lists the fields to send (company, position/title, role, first name, last name, email, phone optional), provides both the provisa.dev registration link and the license request email, includes the machine ID, and states how to apply the received license file. The nag never gates or degrades functionality.

Use case: Nags the user into registering so end-user contact details are captured — the only means of gathering usage since Provisa ships with no telemetry. The license is free; nothing is charged or blocked.

Code: provisa/licensing/nag.py, provisa/licensing/emit.py, provisa/licensing/state.py, provisa/api/app_startup.py, provisa/api/app.py, provisa/pgwire/server.py, provisa/bolt/session.py, provisa/grpc/server.py, provisa/api/flight/server.py, provisa/api/mcp/server.py

Tests: tests/unit/test_licensing.py

REQ-1138 · Licensing¶

Status: ✅ complete · Priority: MUST · Type: structural

A license file is an Ed25519-signed token issued by provisa.dev (which holds the private key). Its payload contains company, position/title, role, first name, last name, email, optional phone, machine_id, and issued_at. The application embeds only the corresponding public key and verifies the license entirely offline. The license is a one-time registration capture and is the alternative to telemetry — applying it does NOT enable, start, or schedule any telemetry, usage reporting, or ongoing data collection. Provisa never phones home before, during, or after licensing.

Use case: Allows offline verification of an authentic, installation-bound license without any runtime call to an issuing server. Registration at install time replaces telemetry entirely; there is no continuous collection.

Code: provisa/licensing/license.py, provisa/licensing/keys.py

Tests: tests/unit/test_licensing.py

REQ-1139 · Licensing¶

Status: ✅ complete · Priority: MUST · Type: behavioral

A license file is applied via provisa license apply <file>, a UI upload in Settings → License, or by placing it at ~/.provisa/license.json. On apply the system verifies the Ed25519 signature against the embedded public key and checks that the license machine_id matches the local installation's stable machine_id (derived from hardware/OS identifiers, not the app directory, so it survives reinstall). A valid, matching license permanently silences the nag on all surfaces. A signature failure or machine_id mismatch is rejected and the nag continues.

Use case: Lets a user remove the nag by applying the installation-bound license they received.

Code: provisa/licensing/license.py, provisa/licensing/machine_id.py, provisa/cli.py

Tests: tests/unit/test_licensing.py

6. Materialized Views & Caching¶

REQ-1140 · Materialized View Governance¶

Status: ✅ complete · Priority: MUST · Type: constraint

A materialized view may only be published if all relationships it depends on are approved. If the requesting user holds the necessary rights to create those relationships, the relationships are automatically created and approved; MV publication proceeds. If the user lacks the required rights, both the relationship creation and MV creation requests are submitted for approval, and MV publication is blocked until all relationships are approved.

Use case: Ensures MVs cannot be published over unapproved relationships, preventing uncontrolled data lineage and access patterns. Distinguishes between users who can auto-provision (trusted data engineers) and those who must request approval (least-privilege roles).

Code: provisa/api/admin/mv_relationship_gate.py, provisa/api/admin/schema_mutation_ops.py

Tests: tests/unit/test_mv_relationship_gate.py

6. Execution, Routing, Caching & Performance¶

REQ-1141 · Caching¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A steward can mark an operational database source as load-sensitive and configure it for scheduled-refresh-only materialization, with a required constraint: at least one of three independent optional gates MUST be configured. The gates are off-peak window (a time range), refresh cadence (refresh_interval), and freshness probe (FreshnessMode.PROBE, checking for source change). The refresh fires on the AND of ONLY the configured gates — an unconfigured gate is vacuously true (ignored). Trigger clocks vary: window-only pulls once per window-open edge; cadence-only pulls every interval regardless of time; probe-only pulls on each evaluation if changed. Probe always gates further: if configured and returns unchanged, the scheduler resets its clock, zero data pull. The query path never touches the source: it serves the last snapshot via FreshnessMode.SCHEDULED (always fresh=true), decoupling query-path load from source access. REQ-1141's contract is scheduler-driven refresh: without a configured gate, there is no scheduler discipline and the source is not load-protected. Boundary principle: MATERIALIZATION REQUIRES A REFRESH POLICY (a REQ-1141 gate OR a cache_ttl). Prefer_materialized + cache_ttl (no gates) is REQ-826 lazy read-through (query-triggered on staleness). Prefer_materialized + no gates + no cache_ttl splits on reachability: if the engine can ATTACH/VIRTUAL the source live, prefer_materialized is inert and the source degrades to live serving; if unreachable (_MATERIALIZE_ONLY type — APIs/NoSQL/streams with no live option), it is a load-once-then-frozen snapshot, only meaningful for static reference data or REQ-847 mutation-driven invalidation.

Use case: Lets a steward protect an operational database by serving it entirely from scheduled snapshots, with fine-grained control over refresh timing (maintenance windows, periodic intervals, or change-driven). The probe signal avoids wasted re-pulls when the source is unchanged. Query traffic never touches the source — only predictable background refresh traffic, bounded by configured gates and staleness by scheduler policy, not query load.

Code: provisa/core/models.py, provisa/federation/scheduled_refresh.py, provisa/federation/freshness_gate.py, provisa/federation/scheduler.py, provisa/federation/read_through.py

Tests: tests/unit/test_scheduled_refresh.py, tests/unit/test_scheduler_driver.py

11. Platform, Infrastructure & Delivery¶

REQ-1142 · Embedded Tier Deployment¶

Status: ✅ complete · Priority: MUST · Type: behavioral

DuckDB extensions (sqlite_scanner, postgres_scanner, iceberg, delta, community) required by the embedded engine are shipped offline via a separate PyPI package provisa-duckdb-ext (py3-none-any universal wheel) that embeds pinned, per-platform .duckdb_extension blobs. On embedded launch, the extensions are staged into a writable local directory without any call to extensions.duckdb.org. Missing extensions fail loud; silent network fallback is disabled.

Use case: Behind enterprise firewalls where only PyPI/Maven/npm/NuGet are proxied (e.g. Artifactory), extensions.duckdb.org is unreachable, breaking the embedded tier and provisa run --demo. Offline delivery via PyPI—already reachable—ensures embedded deployments work in air-gapped environments without network access to DuckDB's extension registry.

Code: provisa/federation/duckdb_extensions.py, provisa/federation/duckdb_runtime.py, packaging/duckdb-ext/

Tests: tests/unit/test_duckdb_ext_staging.py

6. Execution, Routing, Caching & Performance¶

REQ-1143 · Caching¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The table detail view shows a derived, human-readable refresh_policy_summary that expresses the effective refresh/serving behavior per (source, engine) pair, computed server-side from the same resolution logic (federate + freshness_gate + REQ-1141 gates) that the planner uses. The summary states the effective outcome (Live, Scheduled snapshot, or Lazy cache) and warns when prefer_materialized is inert or the table is accidentally frozen.

Use case: A steward cannot decode six interacting config fields (prefer_materialized, gates, cache_ttl, cadence, freshness probe, off-peak window) to understand whether a table is served live, from a scheduled snapshot, or from a lazy cache. The server-derived summary, shown in one sentence, eliminates manual decoding and surface misconfiguration warnings at author time.

Code: provisa/federation/policy_summary.py, provisa/api/admin/_refresh_summary.py, provisa-ui/src/pages/tables/TableEditForm.tsx

Tests: tests/unit/test_policy_summary.py, tests/unit/test_refresh_summary_view.py

11. Platform, Infrastructure & Delivery¶

REQ-1144 · Windows Native Launcher¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The Windows native launcher displays a visible WinForms splash screen (startup monitor) during first launch instead of running fully hidden. The monitor launches first-launch-native.ps1 hidden, tails a progress breadcrumb file (%USERPROFILE%.provisa.startup-status), polls the /health endpoint for readiness, warms first-paint UI endpoints, then opens the browser and closes.

Use case: The prior -WindowStyle Hidden chain showed nothing for tens of seconds during API startup and swallowed fatal staging errors to a hidden console, making failures appear as "nothing happened." A visible monitor surfaces startup progress and fatal errors immediately.

Code: packaging/windows/startup-monitor.ps1

Tests: —

REQ-1145 · Windows Native Launcher¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The launcher surfaces startup progress via breadcrumbs. first-launch-native.ps1 and provisa-native.ps1 write STATE|message lines (STAGING, CONFIG, DEMO, START, WAIT, ERROR) to %USERPROFILE%.provisa.startup-status for the monitor to display. Failed breadcrumb writes must never abort startup (best-effort).

Use case: Breadcrumbs let the monitor and user track each stage of startup (staging runtime, configuring services, starting demo data, waiting for API, etc.), giving visibility into slow stages and enabling the monitor to display progress in real-time.

Code: packaging/windows/first-launch-native.ps1, packaging/windows/provisa-native.ps1

Tests: —

REQ-1146 · Windows Native Launcher¶

Status: ✅ complete · Priority: MUST · Type: constraint

Runtime upgrade must kill ALL processes holding the runtime before replacing it, including the demo mock servers tracked in .demo.pid, not just .native.pid. A leftover demo python.exe locked the runtime DLLs and made re-staging silently abort, pinning the install to its old runtime.

Use case: Without killing all demo processes, a stale Python process holding the runtime directory prevents the upgrade from replacing DLLs, causing the installer to silently fail and continue using the old runtime. This constraint ensures clean, uninterrupted runtime upgrades.

Code: packaging/windows/provisa-native.ps1

Tests: —

REQ-1147 · Windows Native Launcher¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

When the startup monitor drives the launch (env PROVISA_STARTUP_UI=1), provisa-native.ps1 must NOT open the browser itself — the monitor owns the readiness wait and browser open. This avoids a double-open race where both the monitor and the native script attempt to open the same URL simultaneously.

Use case: A double-open race (monitor + native script both calling Start-Process browser) causes unpredictable behavior (duplicate tabs, missed readiness wait, flipped window focus). Centralizing browser open to the monitor ensures a single, coordinated open after confirmed readiness and UI warmup.

Code: packaging/windows/provisa-native.ps1, packaging/windows/startup-monitor.ps1

Tests: —

6. Execution, Routing, Caching & Performance¶

REQ-1148 · Freshness / Change Detection¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A new freshness PROBE transport whose opaque token comes from a zero-byte SENTINEL marker rather than from scanning the data. Reuses the REQ-855 gate unchanged: freshness_token() -> str | None (provisa/events/probes.py:126) compared by equality; a new sentinel drop changes the token -> changed -> re-pull; unchanged -> keep. The sentinel lives at a configured path reachable over a file mount, HTTP, FTP, or SFTP; the token is exists+mtime+size for file/FTP/SFTP, or ETag/Last-Modified for HTTP -- the same HASH token shape files already use (provisa/events/probes.py:46-49). Adds a sentinel probe transport + a sentinel_path (with transport/credentials) config field; capability classes and the gate machinery are untouched. Selected via change_signal in {probe, ttl_probe} with probe_type=hash pointed at the sentinel.

Use case: Lets a steward detect upstream readiness from a published sentinel marker (a near-free poll) instead of scanning or querying the source -- and, combined with REQ-1141 load protection, the cheap sentinel poll replaces hammering the operational source while the heavy re-pull still defers to the off-peak window.

Code: provisa/events/probes.py, provisa/events/sentinel_probe.py, provisa/events/boot.py, provisa/core/models.py

Tests: tests/unit/test_sentinel_probe.py

REQ-1149 · Freshness / Change Detection¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A new change_signal PUSH variant, signal, for a DATA-LESS external trigger (a Kafka control message on a topic, or an HTTP webhook) that says "the source changed / refresh now" WITHOUT carrying the rows. Distinct from the existing CDC push signals (kafka/debezium/native, provisa/core/change_signal.py:29) where the message CARRIES per-row events and lands as upserts (select_landing_shape -> CDC). A signal message instead INVALIDATES the materialized snapshot (marks it stale); the rows are then re-pulled from the SOURCE OF TRUTH on the next refresh -- it does not land the message. It sits between the axes: push-triggered timing, poll-style full/delta re-pull. Adds signal to the push-signal vocabulary + a receiver (kafka control topic / webhook endpoint) mapping a trigger to a staleness mark. Under REQ-1141 the signal is the change-detector and the load-protected scheduler still owns WHEN the heavy pull runs (deferred to the off-peak window) -- decoupling the cheap control signal from the data pull.

Use case: Lets an upstream system announce a change over Kafka/webhook without Provisa polling or the message carrying data; Provisa invalidates and re-pulls from the source on its own schedule -- ideal for load-protected operational sources (REQ-1141) where a control-plane signal must not translate into immediate source load.

Code: provisa/core/change_signal.py, provisa/federation/change_trigger.py, provisa/federation/scheduled_refresh.py, provisa/federation/scheduler.py

Tests: tests/unit/test_change_trigger_signal.py

8. Client Access & Protocols¶

REQ-1156 · Protocol Support¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Registered commands (tracked functions and webhooks -- REQ-205..208, REQ-242, REQ-885) MUST be invocable and discoverable across ALL client-facing protocol surfaces, not only the three that expose them today (GraphQL, pgwire SQL, REST). A command carries two orthogonal dimensions that every surface must honor: kind (query vs mutation) and set-vs-scalar return shape (return_schema / returns = "schema.table" -> set-returning / table-valued). Current state (verified): GraphQL exposes commands as query/mutation fields (provisa/compiler/actions_schema.py:183-185, wired at provisa/compiler/schema_gen.py:846-848); pgwire SQL invokes SELECT fn(...) / SELECT * FROM fn(...) and projects them into the pg_proc/routines catalog (provisa/pgwire/function_call.py:83-87, provisa/pgwire/function_catalog.py:67-90), but SQL invocation is gated by a SINGLE hook maybe_invoke_registered_function that lives ONLY in execute_pgwire_sql (provisa/pgwire/_pipeline.py:528); REST invokes via provisa/api/rest/registered_call.py:162-164. The following surfaces are DARK and MUST be lit: (1) MCP -- run_sql and any tool that runs SQL through _govern_and_route/_execute_plan (provisa/api/mcp/tools.py:270-272) bypasses the function hook, so even SELECT fn(...) does not invoke a command; expose commands as MCP tools and/or route MCP SQL through the shared function hook. (2) Arrow Flight -- ListFlights/catalog lists tables only (provisa/api/flight/catalog.py:87) and do_get runs SQL via _govern_and_route (provisa/api/flight/server.py:536-543), bypassing the hook; commands must be listed as Flights and invocable. (3) gRPC (Provisa's OWN server) -- proto_gen emits only per-table Query{T}/Insert{T} RPCs (provisa/grpc/proto_gen.py:231-239) and never iterates si.functions; generate an RPC per command (query-kind -> unary/server-streaming for set returns, mutation-kind -> unary MutationResponse). (4) Cypher/Bolt -- no tracked-function path exists; APOC-style CALL is rejected (provisa/cypher/parser.py:864) and Bolt RUN routes only to _execute_cypher (provisa/bolt/session.py:284); expose commands via a CALL procedure surface (or equivalent) that reaches invoke_tracked_function. The single shared executor invoke_tracked_function (provisa/api/data/action_exec.py) MUST remain the one invocation path so per-mutation writable_by / governance (REQ-869) is enforced uniformly regardless of surface; the fix is to route each protocol's function-call path through that executor rather than reimplementing dispatch.

Use case: A steward registers a command once and every client -- a GraphQL app, a psql / DBeaver session, a REST caller, an MCP-connected assistant, an Arrow Flight consumer, a gRPC service, and a Neo4j/Bolt/Cypher client -- can invoke and discover it with identical governance, instead of the command silently working on three surfaces and being invisible or a no-op on the rest.

Code: provisa/api/data/action_exec.py, provisa/api/mcp/tools.py, provisa/api/mcp/server.py, provisa/api/flight/server.py, provisa/api/flight/catalog.py, provisa/grpc/proto_gen.py, provisa/grpc/server.py, provisa/cypher/parser.py, provisa/bolt/session.py, provisa/api/rest/registered_call.py

Tests: tests/unit/test_command_surface_discovery.py, tests/unit/test_grpc_command_endpoint.py, tests/unit/test_bolt_command_call.py, tests/unit/test_proto_gen.py, tests/unit/test_tracked_function_invocation.py

5. Query Languages, Compilation & Operations¶

REQ-1157 · Materialized Views & Caching¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A registered relation backed by view_sql (a Provisa-managed view or, with materialize=true, a materialized view -- REQ-885/REQ-543, provisa/core/models.py:496-498) is DERIVED, not a base table, and MUST be exposed query-only: the schema generator MUST NOT emit insert/upsert/update/delete GraphQL mutations for it, and raw-SQL surfaces (pgwire / Flight SQL / Bolt) MUST reject writes to it. Rationale: a write to a view either fails at the source (non-updatable view) or lands in the mv_cache snapshot that the next REQ-879 refresh silently overwrites -- data loss with no error, violating the project's no-silent-failure rule. Implementation: fetch_tables carries view_sql (provisa/api/admin/db_queries.py:114), _build_visible_tables sets _TableInfo.read_only = bool(view_sql) (provisa/compiler/schema_types.py, provisa/compiler/schema_gen.py:_build_visible_tables), and the mutation-field loop skips read_only relations (provisa/compiler/schema_gen.py). The GraphQL read surface (query fields, aggregates, relationships) is UNCHANGED -- only the write surface is withdrawn. A base table is never affected (view_sql is null). Raw-SQL write-rejection: because every raw-SQL surface funnels through the one governed pipeline (REQ-1176), the guard lives in ONE place -- _reject_view_writes (provisa/pgwire/_pipeline.py), called from both _govern_and_route (pgwire / REST /data/sql / Flight SQL / MCP) and _govern_and_route_compiled (Bolt/Cypher / gRPC). It rejects any INSERT/UPSERT/UPDATE/DELETE/MERGE whose TARGET is a view_sql_map relation; a view read in the FROM/USING of a write, and all base-table writes, are unaffected.

Use case: A steward composes an MV over a Provisa-hosted function (REQ-885) and it appears in GraphQL for querying, but insert/update/delete mutations are no longer generated for it -- so a client cannot issue a write that would be lost on the next refresh. The relation reads like any table; it just cannot be written.

Code: provisa/api/admin/db_queries.py, provisa/compiler/schema_types.py, provisa/compiler/schema_gen.py, provisa/pgwire/_pipeline.py

Tests: tests/unit/test_view_readonly_writes.py, tests/unit/test_visibility.py

9. Installation & Deployment¶

REQ-1158 · Package Distribution¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

A universal py3-none-any wheel package provisa-pg-ext carrying the pinned per-platform PostgreSQL extension and FDW binaries, published to PyPI so the embedded tier acquires them via PyPI/Artifactory instead of github.com/kenstott/provisa/releases (unreachable behind PyPI/Maven/npm-only enterprise proxies). Payload (scripts/ci/build_pg_extensions.sh): core contrib file_fdw + postgres_fdw, external sqlite_fdw + mysql_fdw, and pg_duckdb — where pg_duckdb (csv/parquet/json + httpfs + iceberg via DuckDB) SUBSUMES the REQ-898 parquet_fdw / parquet_s3_fdw / pg_lake / pg_analytics functionality in one maintained extension rather than four heterogeneous build systems. Built by the wheel job in .github/workflows/build-pg-extensions.yml, which collects the per-platform build tarballs (packaging/pg-ext/collect.py merges them under provisa_pg_ext/_ext), builds ONE universal wheel (python -m build), and publishes to PyPI. Runtime discovery: packaging/pg-ext/provisa_pg_ext.ext_root() locates the bundle; provisa/pg_extensions/staging.py stage_bundled_pg_extensions() copies the running platform's FDW modules into a pgserver pginstall and provisa/federation/fdw_artifact_catalog.py registers what landed (fails loud if the wheel lacks this platform's bundle — no github.com fallback), keyed by the REQ-898 config/pg_extension_catalog.yaml inventory (provisa/pg_extensions/catalog.py). A pgserver configured with FDW sources thus loads the bundled modules via CREATE EXTENSION straight from the PyPI wheel — the offline analog of the DuckDB extension staging (REQ-1142). The FDW set IS the pgserver federation engine's (PgFederationRuntime, provisa/federation/pg_runtime.py) REACHABILITY matrix: each FDW defines a reachable source type (postgres_fdw→postgres, sqlite_fdw→sqlite, file_fdw→csv, mysql_fdw→mysql, pg_duckdb→parquet/csv/json/httpfs/iceberg). Runtime staging is WIRED: provisa/core/control_plane_pg.py start() calls _stage_bundled_extensions() at embedded-pgserver boot, staging the wheel's modules into the shared pginstall so the pg fed engine can CREATE EXTENSION them offline (best-effort: a BYO-Postgres tier without the wheel is a clean no-op — its FDWs come from the system PG, discovered via the STANDARD pg_available_extensions catalog, connector_duckdb._probe_pg_extension). Reachability is proven end-to-end per FDW through PgFederationRuntime.attach_source: file_fdw + postgres_fdw (test_embedded_pg_fdw_engine_e2e), sqlite_fdw (test_embedded_pg_sqlite_fdw_e2e), pg_duckdb csv/parquet (test_embedded_pg_duckdb_engine_e2e) and iceberg (test_embedded_pg_duckdb_iceberg_e2e), and the runtime directly (test_pg_runtime_e2e); mysql_fdw is best-effort (client lib), matching the CI smoke gate (scripts/ci/smoke_pg_extensions.py, wired in build-pg-extensions.yml, which CREATE EXTENSIONs each on a fresh pgserver). Declarative BYO custom-connector description (adding a source_type→attach mapping for a custom FDW / DuckDB ATTACH extension purely in config) is a SEPARATE requirement, not this packaging one.

Use case: Embedded tier (and enterprise deployments behind restricted proxies) can install PostgreSQL extensions offline via standard PyPI/Artifactory channels instead of fetching from unreachable GitHub releases.

Code: packaging/pg-ext/, packaging/pg-ext/collect.py, scripts/ci/build_pg_extensions.sh, scripts/ci/smoke_pg_extensions.py, .github/workflows/build-pg-extensions.yml, provisa/pg_extensions/staging.py, provisa/pg_extensions/catalog.py, provisa/federation/fdw_artifact_catalog.py, provisa/core/control_plane_pg.py, provisa/federation/pg_runtime.py

Tests: tests/unit/test_pg_ext_staging.py, tests/unit/test_pg_extension_catalog.py, tests/unit/test_control_plane_pg_staging.py, tests/integration/test_embedded_pg_fdw_engine_e2e.py, tests/integration/test_embedded_pg_sqlite_fdw_e2e.py, tests/integration/test_embedded_pg_duckdb_engine_e2e.py, tests/integration/test_pg_runtime_e2e.py

11. Platform, Infrastructure & Delivery¶

REQ-1150 · Embedded CLI Startup¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

The provisa run console command signals startup completion by polling the API /ready endpoint (the warm gate, not /health), prints a "✓ Provisa is ready — " completion line, and opens the browser automatically with ?tour=1 for a --demo run (so the guided tour auto-starts). A new --no-browser flag opts out of the open while still printing the URL. Failures and 300s timeouts are best-effort and non-fatal: the servers keep running, and the URL is printed manually as a fallback.

Use case: Previously provisa run printed URLs then served forever with no completion signal and no browser open, so --demo appeared stuck even though the app was up. Readiness polling, completion line, and browser auto-open provide immediate visual feedback that the system is ready and reachable.

Code: provisa/cli.py

Tests: —

5. Query Languages, Compilation & Operations¶

REQ-1159 · Extensible Functions¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Registered commands (tracked functions) MUST be composable INLINE within a larger SQL statement — e.g. SELECT o.id, e.embedding FROM orders o JOIN enrich_cmd('main.public.orders') e ON o.id = e.id — not only as a standalone top-level SELECT * FROM fn(args) / SELECT fn(args), and this MUST work across ALL client-facing surfaces (GraphQL, pgwire SQL, REST, MCP run_sql, Arrow Flight, Provisa gRPC, Bolt/Cypher), routed through the one shared invoke_tracked_function executor and the shared _govern_and_route / _govern_and_route_compiled pipeline. Implemented as a command-localization rewrite pass (_localize_inline_commands in provisa/pgwire/_pipeline.py → provisa/executor/command_localize.py localize_commands / find_command_calls) inside the shared _govern_and_route pipeline. The standalone hook (provisa/pgwire/function_call.py maybe_invoke_registered_function) now returns None for any composed statement (joined/sub-queried), so it never mis-runs one command as the whole result; the localizer owns composition. Every raw-SQL surface gets it for free by routing through the one pipeline: pgwire, MCP run_sql, Arrow Flight SQL, and REST /data/sql. The drifted second pipeline (provisa/api/data/endpoint_dev.py::_compile_govern_execute) has been DELETED; REST /data/sql, airport writes, and table-profile now route through _govern_and_route/_execute_plan, with as_of (REQ-1163), discovery_mode, provisa view expansion, and govdata folded into the one pipeline. The single chokepoint is enforced: _govern_and_route mints an unforgeable provenance stamp on every plan and _execute_plan refuses any plan lacking one (tests/unit/test_governed_chokepoint.py). gRPC (typed proto → _govern_and_route_compiled, no raw-SQL entry), Bolt/Cypher (command invoked standalone via CALL, REQ-1156 — no inline-SQL syntax) and GraphQL (commands are generated fields, not raw SQL) have no raw-SQL inline-composition surface, so REQ-1159 is N/A there. The pass (a) detects every command call in the parsed tree, (b) executes each to a local relation typed by the command's declared output dataset contract, substituting the call site via a size-adaptive mechanism (VALUES CTE below a row threshold, registered Arrow table above), (c) forces local federation execution (Route.ENGINE — a command cannot be pushed to a remote source), (d) applies governance at the correct DEFINER/INVOKER identity for the command's input plus the caller's governance for the outer SQL, and (e) validates rows in/out against the command's declared input/output dataset schema (fail-loud). Prerequisite: a canonical per-dataset contract — one or more scalar (column_value) args plus one or more TYPED input datasets (result_set/table_ref args, each carrying an ordered relational field-list) and exactly one TYPED output dataset; canonical field-list is the source of truth (single GraphQL-scalar vocabulary), with JSON Schema and GraphQL SDL as edge projections (seed-in / render-out), not stored truth.

Use case: A steward composes a governed external transform directly in ordinary SQL joined against federated tables, on any surface, and — because the command declares and validates its input/output dataset columns — column-level lineage can close across the opaque RPC boundary via taint-closure splicing at the function node.

Code: provisa/pgwire/_pipeline.py, provisa/pgwire/function_call.py, provisa/executor/command_localize.py, provisa/executor/function_dispatch.py, provisa/api/data/action_exec.py, provisa/api/data/endpoint_dev.py, provisa/api/mcp/tools.py, provisa/api/flight/server.py, provisa/processors/contract.py, provisa/core/models.py, provisa-ui/src/pages/commands/CommandFormFields.tsx, provisa-ui/src/api/actions.ts, provisa/api/admin/table_profile_router.py, provisa/api/airport/query.py

Tests: tests/integration/test_command_compose_surfaces.py, tests/integration/test_command_compose_e2e.py, tests/unit/test_command_localize.py, tests/unit/test_mcp_server.py, tests/unit/test_governed_chokepoint.py, tests/e2e/test_transport_contract.py

6. Data Lineage & Provenance¶

REQ-1160 · Column-Level Lineage Graph¶

Status: ✅ complete · Priority: MUST · Type: behavioral

From a SINGLE SQL statement — including one or more inline-composed commands (REQ-1159) — the platform MUST be able to produce a complete column-level lineage DAG for the entire view, from source inputs to final output columns, with command nodes as first-class, contract-validated (non-opaque) nodes in the graph. Computable STATICALLY from the view definition plus each command's declared input/output dataset contract (no execution required); optionally reconciled with runtime refresh-version records (REQ-862 spans / input versions) to stamp which version flowed. Implemented as a full node+edge graph builder (provisa/lineage/graph.py build_column_graph / LineageGraph) that preserves intermediate relations (CTEs, subqueries, joins), alongside the pre-existing leaf-collapsed resolver (provisa/lineage/columns.py resolve_column_lineage, retained for the flat output->source view). Each inline command, once localized into a typed CTE/Arrow relation, contributes real edges: taint-closure by default (each declared output column derives from ALL declared input columns) or precise per-column edges when declared; command nodes are visually distinct opaque-transform boundaries tagged with the command name and its contract. Two grains from one graph: relation-level zoom-out and column-level drill-in. Surfaces (shipped): an API that takes a view/statement and returns graph JSON (nodes + edges) — POST /admin/lineage/graph (provisa/api/admin/lineage_router.py) — and a UI DAG visualization (provisa-ui/src/components/lineage/LineageDag.tsx, LineagePage.tsx); previously column lineage was only OTel span attributes emitted at MV refresh. Each transform (edge) MUST be NAMED and/or DEFINED, not left as an opaque SQL-expression string: the resolver decomposes each projection expression and matches every operation against the union vocabulary of SQL operators/standard functions (SQLGlot expression nodes), engine built-in scalar functions, and registered commands (projected into pg_proc/routines, each carrying its declared contract) — so e.g. lat,lng <- city, state is labeled as the chain "concat/+ (SQL op) -> geocode (command, contract as its def)". This is a sub-DAG inside the edge: the column-level graph states the derivation, and zooming into an edge reveals the named operation chain (expression-level command nodes reuse the same contract / taint-closure machinery). A transform matching no known op is itself a signal worth flagging.

Use case: A steward inspects the end-to-end derivation of any view's output columns — across SQL and opaque command boundaries alike — as a rendered DAG; the visualization doubles as a discipline mirror, since an over-declared "frankenstein" command renders as a wide fan-in node, making sloppy contracts visible.

Code: provisa/lineage/columns.py, provisa/lineage/graph.py, provisa/lineage/versions.py, provisa/api/admin/lineage_router.py, provisa/pgwire/_pipeline.py, provisa-ui/src/components/lineage/LineageDag.tsx, provisa-ui/src/pages/LineagePage.tsx

Tests: tests/unit/test_lineage_graph.py, provisa-ui/src/__tests__/LineagePage.test.tsx

REQ-1161 · Federation-Wide Provenance Graph¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The platform MUST be able to merge every per-view / per-statement column-level lineage DAG (REQ-1160) into ONE federation-wide provenance graph — spanning all registered views, MVs, and inline-composed commands across all sources — by unioning nodes on canonical resolved dataset identity + column name. Base source columns -> every derived column everywhere, with command boundaries transparent via their declared contracts. Node key = resolved dataset identity (physical/semantic, via the centralized dataset identity service, never ad-hoc string comparison — docs/dataset-handling-standards.md "Dataset Identity"): if two references to the same dataset do not resolve to one node, the merged graph fragments silently, which MUST NOT happen. Computed whole but RENDERED progressively: relation-level is the default view, column-level on drill-in, with focus/scoping tools required (ego-graph around a node, upstream-only / downstream-only slice, domain filter) — a literal full-graph render is unusable at federation scale. Built INCREMENTALLY: recompute a view's sub-DAG on its definition change and union incrementally; never rebuild the entire graph. CYCLE handling is detect + CHARACTERIZE + surface, NOT auto-reject: a cycle that crosses a materialization boundary (an MV reading a prior snapshot of a downstream MV) is a LEGAL time-lagged feedback loop — the snapshot is the version boundary that makes it well-defined; a cycle with NO materialization boundary on it is the likely design error (no evaluation order, no stable value). The platform describes the distinction and surfaces it (in the graph and as a diagnostic) for operator judgment; it does not unilaterally reject. IMPLEMENTATION (provisa/lineage/merge.py, provisa/lineage/graph.py, provisa/api/admin/lineage_router.py): merge/union on canonical dataset identity (real-relation node keys, quoting normalized), base→derived stitching with transparent command-contract boundaries, cycle detect+characterize (feedback vs error), progressive slicing (ego-graph / upstream / downstream / depth / domain filter), and the GET /admin/lineage/federation API. Built INCREMENTALLY (build_federation_graph_incremental): each view's parsed sub-DAG is memoized by its SQL, so a request recomputes ONLY the views whose definition changed and unions the rest from cache; an unchanged input returns the prior merged graph without re-unioning — the entire graph is never rebuilt from scratch.

Use case: An operator runs impact analysis (change a source column, see everything affected across every domain), traces any derived column back to its sources across the whole federation, and inspects circular definitions with enough context to judge whether a cycle is an intended time-lagged feedback loop or a design error.

Code: provisa/lineage/merge.py, provisa/lineage/graph.py, provisa/api/admin/lineage_router.py, provisa-ui/src/

Tests: tests/unit/test_lineage_merge.py, tests/unit/test_lineage_incremental.py, tests/unit/test_lineage_graph.py

4. Source Connectors¶

REQ-1162 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A materialized view's time-travel capability is a property of its DEFINITION, guaranteed on any materializing substrate (Iceberg, PostgreSQL, DuckDB, ...). Maintenance is APPEND-ONLY — a version, once written, is never updated or deleted (no valid_to write-back), so it works on engines that cannot cheaply UPDATE a federated store. Each refresh appends one of two ways: snapshot appends the whole fresh dataset stamped with system time; delta appends only the rows the ENGINE computes as changed (anti-joins in INSERT ... SELECT) plus tombstones for removed keys — never folded row-by-row in Provisa. "As of" history is DERIVED at read time from the immutable append log (latest batch for snapshot; a ROW_NUMBER window per entity key for delta), so consumer-facing as_of semantics are identical across every substrate. On Iceberg each append is exactly one snapshot (REQ-844). Declared end to end via mv_bitemporal_mode / mv_bitemporal_key (TableInput -> core Table -> MVDefinition.bitemporal -> refresh_mv) with a UI control in TableEditForm. NOTE: exposing as_of as a query-language argument that routes bitemporal-MV reads through reconstruction is tracked as REQ-1163 (the consumer query surface, analogous to REQ-372 for sources).

Use case: Time-travel previously existed only on Iceberg; when Iceberg was unreachable the substrate degraded to replace/append/upsert, losing history. Append-only bitemporal materialization makes time-travel available on any substrate — the same definition retargets across engines without remodeling.

Code: provisa/mv/bitemporal.py, provisa/mv/refresh.py, provisa/mv/models.py, provisa/api/admin/schema_common.py, provisa/core/schema_org.py, provisa/core/schema.sql, provisa/core/repositories/table.py, provisa/api/admin/schema_helpers.py, provisa/api/app_rebuild.py

Tests: tests/unit/test_bitemporal.py, tests/unit/test_bitemporal_exec.py, tests/unit/test_bitemporal_pg.py, tests/unit/test_bitemporal_iceberg.py, tests/unit/test_bitemporal_refresh_e2e.py, tests/integration/test_bitemporal_http_e2e.py

REQ-1163 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A bitemporal materialized view (REQ-1162) is READABLE through the query languages: consumers get current state by default and can travel in time. (1) Current-by-default — a bitemporal materialized view's inline-expansion entry (state.view_sql_map, app_rebuild.py) is pointed at a reconstruction over the physical append log (bitemporal.py: view_read_sql), so a plain read returns the CURRENT reconstructed state (latest batch for snapshot; latest-version-per-business-key window dropping tombstones for delta), never the raw log. (2) Time travel — a request-level X-Provisa-As-Of header (validated to a safe SQL timestamp literal by parse_as_of; malformed → HTTP 400) applies to the whole query: _prepare_compiled overlays each bitemporal view's expansion entry with an as-of reconstruction over its append log (as_of_view_map + view_read_sql(ts)), giving identical consumer-facing as-of semantics across every substrate. A whole-query as-of context (like a bitemporal system time) rather than a per-field arg keeps it out of the GraphQL schema. Analogous to REQ-372 (as_of for Iceberg/Delta SOURCES).

Use case: Without routing reads through reconstruction a bitemporal MV read would surface all historical versions instead of current state. Current-by-default makes bitemporal MVs usable; the X-Provisa-As-Of header adds opt-in, whole-query time travel.

Code: provisa/api/app_rebuild.py, provisa/api/app.py, provisa/api/data/endpoint.py, provisa/api/data/endpoint_dev.py, provisa/mv/bitemporal.py

Tests: tests/unit/test_bitemporal_exec.py, tests/unit/test_bitemporal_read_integration.py, tests/integration/test_bitemporal_http_e2e.py

REQ-1164 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Declarative entity and fact shortcuts that LOWER to the existing MV / bitemporal / relationship primitives — the two constructs both a star schema and a Data Vault are built from, kept methodology-neutral. (1) entity — a keyed, deduplicated, optionally-historized projection of a source; lowers to a materialized view (view_sql projecting the key + attributes) with materialize=true, and when history is requested a bitemporal MV (REQ-1162): history=scd2 → mode=delta, history=snapshot → mode=snapshot, keyed on the entity key. Serves a star DIMENSION (SCD1/SCD2) and a Data Vault HUB+SATELLITE. (2) fact — join to entity keys, reduce to a declared grain, aggregate measures; lowers to a materialized view (SELECT grain + dimension FK columns + aggregated measures, GROUP BY grain+FKs) plus registered relationships to the entities. Serves a star FACT and a DV LINK (a degenerate fact of keys). The lowering is pure (a spec → the same registration a user would hand-write), so the whole warehouse retargets across engines as IR — the model GENERATES the star/vault, it does not decide the methodology (grain, conformance, SCD choice stay the modeler's).

Use case: Star schemas and Data Vaults are compositions of a tiny primitive set (dimension/fact, hub/ satellite/link). First-class entity/fact sugar lets a modeler declare them once and have Provisa generate the underlying portable MV definitions, instead of hand-writing each view_sql + flags.

Code: provisa/mv/modeling.py

Tests: tests/unit/test_modeling.py

9. Live Data & Events¶

REQ-1165 · Table Processor (Shared)¶

Status: ✅ complete · Priority: MUST · Type: constraint

The inline MV preprocess hook (REQ-957) is RESCOPED from a row transform (rows → rows) to a PREFLIGHT CHECK that returns a verdict (continue / abort / quarantine), not a mutated dataset. Row transforms are explicitly out of scope — they belong in SQL (engine pushdown) or external processors (REQ-940). Rationale: a gate does not mutate the landed set, so it does not feed the content hash (REQ-964), eliminating the justification for full in-memory materialization and the max_rows cap (SKIPPED_SIZE) that prevents scaling to large sources. Accepted preflight transports are SQL pushdown (for SQL-expressible checks, evaluated engine-side as probe queries) and Python+Arrow (for non-SQL checks, evaluated in-process with columnar batches). gRPC is out of scope — it only serves out-of-process/remote validators, which introduces unnecessary overhead (separate process, wire framing) disproportionate to a preflight gate's simplicity and scale requirements. The Python preflight hook input contract is dict[str, pyarrow.RecordBatchReader], keyed by INPUT NODE NAME, providing lazy Arrow streams via FederationEngine.execute_engine_stream. This avoids full materialization of large inputs; the hook consumes batches lazily and short-circuits early. If an engine lacks EngineCapability.ARROW_STREAM and a preflight is declared, the system fails loud at wiring/boot time (no silent skip, no materialize fallback). The land/generate path remains unchanged, using independent streamed reads.

Use case: Rescoping from stateful row transform to stateless verdict gate simplifies the contract, improves memory efficiency, and allows evaluation to scale via engine pushdown (SQL assertions) or streaming short-circuit (non-SQL predicates) rather than buffering the entire dataset in memory before applying the gate. Providing lazy Arrow streams as input ensures the hook can evaluate large datasets without materializing them into Python dicts.

Code: provisa/mv/preprocess.py, provisa/mv/preflight.py, provisa/mv/preflight_sql.py, provisa/mv/preflight_eval.py, provisa/mv/refresh.py, provisa/processors/arrow.py, provisa/events/handlers.py, provisa/events/boot.py, provisa/events/processor.py

Tests: tests/unit/test_preflight_sql.py, tests/unit/test_preflight_verdict.py, tests/unit/test_preflight_refresh.py, tests/unit/test_processor_arrow.py, tests/unit/test_live_core_loop.py, tests/integration/test_preflight_streaming.py, tests/e2e/test_preflight_streaming_e2e.py, tests/features/REQ-1165-preflight-streaming.feature, tests/steps/steps_preflight_streaming.py

REQ-1166 · Materialization Store¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

A materialized view can have a REPEATING CALENDAR TRIGGER attached that cuts a calendar-addressable version at each boundary (end-of-day, end-of-week, end-of-month, end-of-quarter, end-of-year, or custom nth-weekday recurrences). Each boundary is named with a stable window_id (e.g., 2026-Q1, 2026-W03). The calendar is a TRIGGER/ADDRESSING axis (when a version is cut + its stable identifier), orthogonal to the storage strategy used to preserve history. Full-snapshot-per-boundary is one option (expensive); delta append is preferred for most cases. Composes the calendar boundary trigger (REQ-961/962) and freshness-gated firing (per-input freshness contract).

Use case: Time-series analytics and auditing require data captured at calendar milestones (e.g., end-of-month snapshots of customer state or materialized reports keyed by fiscal quarter). Calendar-addressed versions enable users to query "what was the state as-of the end of Q1 2026?" independent of the underlying storage mechanism.

Code: provisa/events/handlers.py, provisa/events/boot.py, provisa/mv/refresh.py

Tests: tests/unit/test_periodic_snapshot_generate.py, tests/integration/test_periodic_snapshot_pipeline.py, tests/steps/steps_periodic_snapshot.py

4. Source Connectors¶

REQ-1167 · Materialization Store¶

Status: ✅ complete · Priority: MUST · Type: constraint

A calendar-addressed TIME-TRAVEL materialized view MUST use one of the append-only or reconstructible storage strategies (bitemporal snapshot mode, bitemporal delta mode, row-level delta ledger with reconstruct_forward/reverse, or reconstructible PIT with accumulating append). A replace/overwrite MV cannot answer calendar-addressed as-of reads: each refresh overwrites the prior version, so only the current state exists. Snapshot (full dataset per boundary) vs delta (changes only) vs ledger vs reconstructible-PIT is a COST choice, not a capability choice: all four strategies enable time-travel reads, but differ in storage footprint and reconstruction latency. Note: snapshot ⟹ time-travel, but time-travel ⇏ snapshot. Preserved snapshots (REQ-983) are a distinct single-instant frozen form and are NOT a time-travel series.

Use case: Append-only and reconstructible-storage designs enable calendar-addressed versions to be queried across time. Replace MVs lose all previous versions on refresh, making time-travel impossible. The codebase supports four such strategies; the cost/latency tradeoff is a deployment choice.

Code: provisa/mv/bitemporal.py, provisa/events/handlers.py

Tests: tests/unit/test_bitemporal_calendar.py, tests/integration/test_periodic_snapshot_pipeline.py

9. Live Data & Events¶

REQ-1168 · Calendar Boundary¶

Status: ✅ complete · Priority: SHOULD · Type: structural

The Grain enum in provisa/events/calendars.py (daily/weekly/monthly/quarterly/annual) cannot express nth-weekday-of-period recurrences (e.g. "3rd Wednesday of each month"). An NthWeekday recurrence rule (weekday + ordinal 1..5 or -1/last) with occurrence math and a tiling window extends the grain vocabulary; parse_grain_spec resolves a nesting grain OR a recurrence spec ("3WE"/"LFR"), and window_for/next_boundary/PeriodicCalendar accept both.

Use case: Business snapshots often occur on variable dates within a period (e.g., last business day of month, 3rd Friday for option expiry). The current fixed-grain enum is insufficient.

Code: provisa/events/calendars.py

Tests: tests/unit/test_live_temporal.py, tests/unit/test_periodic_snapshot_generate.py, tests/unit/test_bitemporal_calendar.py

REQ-1169 · Calendar Management¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Calendars are declarable through the admin surface and the calendar→MV binding is plumbed end to end. A calendar repository + createCalendar mutation + calendars query define named versioned calendars; the MV snapshot-schedule fields (mv_calendar/mv_grain/mv_allowed_lateness/ mv_expected_events/mv_business_day_grain) flow model→registry→event loop; and a collapsible Snapshot Schedule UI panel binds a calendar + grain (nesting or nth-weekday) to a materialized view, making the periodic-snapshot feature reachable without code changes.

Use case: Without config-level calendar definition and UI binding, repeating snapshots cannot be declared by users without code changes. A user-facing config surface is essential for adoption.

Code: provisa/core/repositories/calendar.py, provisa/api/admin/schema_query.py, provisa/api/admin/schema_mutation.py, provisa-ui/src/pages/tables/TableEditForm.tsx

Tests: tests/unit/test_snapshot_schedule_wiring.py, tests/integration/test_periodic_snapshot_graphql_e2e.py

REQ-1170 · Table Load Protection¶

Status: ✅ complete · Priority: SHOULD · Type: constraint

Load protection (REQ-1141 off-peak window) and snapshotting on the same table can conflict on timing: a snapshot boundary may fall outside the off-peak refresh window. The system emits a WARNING when both load protection and periodic snapshot(s) are configured on one table, alerting the user to potential misalignment between the protection window and snapshot schedule.

Use case: If a snapshot boundary coincides with a peak-hours-only window, the snapshot cannot fire. This warning surfaces the timing conflict so the user can adjust the protection window or snapshot schedule.

Code: provisa/api/admin/schema_mutation.py

Tests: tests/unit/test_load_protection_snapshot_conflict.py

REQ-1171 · Calendar Boundary¶

Status: ✅ complete · Priority: SHOULD · Type: structural

Snapshot-schedule grains extend to support the full RFC 5545 RRULE recurrence model (Outlook/iCalendar). RRuleRecurrence (a frozen dataclass with normalized RRULE string) is validated via python-dateutil. parse_grain_spec recognizes "RRULE:FREQ=..." or bare "FREQ=..." specs; window_for/next_boundary dispatch to dateutil.rrule-backed occurrence tiling yielding half-open [start,end) windows addressed by opening occurrence (e.g., "2026-02-18-M3WE"). INTERVAL phase anchors to civil epoch 2000-01-01 for query-independent tiling. Bounded rules (COUNT/UNTIL) are rejected; nesting grains and 4-4-5 fiscal calendars unchanged.

Use case: Beyond nesting grains (daily..annual) and nth-weekday shorthand, users need full recurrence expressivity for complex business schedules (e.g., "every 3 weeks on Monday, until year-end"). RRULE standardizes this across Outlook, Google Calendar, and iCalendar ecosystems.

Code: provisa/events/calendars.py, provisa-ui/src/pages/tables/RecurrenceBuilder.tsx

Tests: tests/unit/test_calendar_rrule.py

5. Query Languages, Compilation & Operations¶

REQ-1172 · Compiler & Schema¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Registered callable field names (tracked functions and webhooks) in GraphQL schemas now apply the active naming convention (apollo_graphql camelCase, hasura_graphql snake_case) at the emit site, matching table field name behavior. Previously callable fields retained their raw registered names regardless of the configured convention.

Use case: Consistent naming across table fields and callable fields improves developer experience and ensures the GraphQL surface respects the configured naming convention uniformly.

Code: provisa/compiler/actions_schema.py, provisa/api/app_loaders.py

Tests: tests/unit/test_callable_field_naming.py

8. Client Access & Protocols¶

REQ-1173 · Protocol Support¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Webhooks (tracked_webhooks from the remote schema registry) are now governed, discoverable commands routable across all protocol surfaces (pgwire, Bolt, Flight, REST, gRPC, MCP), extending REQ-872 from tracked_functions alone. A webhook: enforces writable_by authorization identically to tracked functions; appears in list_visible_commands discovery; is routed through the shared invoke_tracked_function executor (with a webhook-specific branch in invoke_tracked_webhook); and is invocable via SELECT webhook(...) on pgwire, CALL webhook(...) YIELD on Bolt, POST /{domain}/commands/{name} on REST, and as an MCP tool.

Use case: Webhooks (scalar-argument HTTP POST mutations) were previously GraphQL-only and lacked governance. Making webhooks governed, discoverable, and multi-surface-routable unifies command access patterns across all query surfaces and enforces consistent authorization.

Code: provisa/api/data/action_exec.py, provisa/pgwire/function_call.py, provisa/bolt/session.py, provisa/api/rest/generator.py, provisa/api/rest/openapi_spec.py, provisa/grpc/proto_gen.py, provisa/grpc/server.py

Tests: tests/unit/test_webhook_all_surfaces.py

1. Access Governance & Security¶

REQ-1174 · Query Complexity Limits¶

Status: ✅ complete · Priority: MUST · Type: constraint

Per-role query complexity limits (max_query_depth, max_query_nodes, max_query_time_ms) complement per-role request-rate limits (REQ-369). Enforced at the GraphQL→IR compile boundary; over-limit requests return HTTP 413. Limits persist on the roles table and are configurable via admin API.

Use case: Query complexity limits prevent resource exhaustion from complex or malicious queries, complementing request-rate limiting. Provides Hasura api_limits parity.

Code: provisa/compiler/limits.py, provisa/api/models/role.py, provisa/admin/admin_api.py

Tests: tests/unit/test_query_limits.py, tests/unit/test_hasura_api_limits.py, provisa-ui/e2e/security-query-limits.spec.ts

9. Deployment & Release¶

REQ-1175 · Release Integrity¶

Status: ✅ complete · Priority: MUST · Type: constraint

A GitHub Release must never be published in a partial state (missing any installer/download-link asset). All four tag-triggered workflows (build-dmg, release-exports, build-pg-extensions, build-duckdb-extensions) create the release as a DRAFT via ensure-release; sibling workflows attach assets only with gh release upload --clobber (never flipping draft/prerelease). Only build-dmg's publish-release job undrafts the release after verifying the complete installer manifest (macOS core/runtime/obs/demo DMGs, Linux AppImage, Windows native + container installers, JDBC jar) is present via fail_on_unmatched_files; missing assets cause immediate failure leaving the release draft.

Use case: Prevents /dl redirector from serving incomplete builds. In alpha.257/258, a partial release was published when build-dmg failed while sibling workflows succeeded, causing end users to download missing installers.

Code: .github/workflows/build-dmg.yml, .github/actions/ensure-release, site/functions/dl/[platform].js

Tests: tests/unit/test_release_integrity.py

1. Access Governance & Security¶

REQ-1176 · Execution Integrity¶

Status: ✅ complete · Priority: MUST · Type: constraint

All governed SQL execution flows through exactly one pipeline (_govern_and_route/_govern_and_route_compiled → _execute_plan) that mints unforgeable per-plan provenance stamps (256-bit process-private capability tokens). No surface may assemble its own govern+route+execute sequence. Plans lacking valid stamps are rejected before row access, making ungoverned data egress (RLS/masking/column-visibility/write-ACL bypass) mechanically impossible.

Use case: Prevents side-door or hand-crafted execution plans from bypassing governance. A static AST ratchet test enforces that no NEW module co-locates decide_route + apply_governance, and runtime stamp validation rejects any plan without valid provenance.

Code: provisa/pgwire/_pipeline.py

Tests: tests/unit/test_governed_chokepoint.py, tests/e2e/test_transport_contract.py

4. Source Connectors¶

REQ-1177 · Connector Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Operators may declare custom source connectors (proprietary FDWs or ATTACH extensions) in config without code changes. A declarative descriptor per source_type specifies (a) required extension(s) to load, (b) the attach/import template, and (c) source_type mapping. Availability is verified at attach time against each engine's standard extension catalog (pg_available_extensions for Postgres, duckdb_extensions() for DuckDB), failing loud (never silent) when the declared extension is not installable. The two native engines scope this differently because their extension APIs differ. POSTGRES is a GENERIC descriptor: PG FDW is SQL/MED (an ISO standard), so every FDW uses the same DDL shape — CREATE SERVER … FOREIGN DATA WRAPPER OPTIONS(…) + CREATE USER MAPPING + IMPORT FOREIGN SCHEMA (or an explicit CREATE FOREIGN TABLE when the FDW lacks IMPORT, e.g. file_fdw). The descriptor supplies only the per-FDW variance: extension name, SERVER option keys (host vs servername vs dbserver), the USER MAPPING key (user vs username), and a supports_import flag. So an arbitrary standard-conforming FDW is drivable from config. DUCKDB is a CURATED UNION, not arbitrary: DuckDB has no single standard, so we support only extensions that fit one of the TWO mechanisms our engine already drives — (1) ATTACH-TYPE for database extensions (ATTACH '' AS x (TYPE ), generalized today by _DuckDBExtensionConnector / REQ-899: INSTALL [FROM community] + LOAD + ATTACH), and (2) SCAN table-function for file/lake extensions (CREATE VIEW … AS SELECT * FROM read_parquet(…) / read_csv_auto(…) / iceberg_scan(…) / delta_scan(…)). The popular methods (postgres/sqlite/mssql/mongo/mysql; csv/parquet/iceberg/delta) are covered OOTB; the config admits a new DuckDB source_type only when it declares one of these two mechanisms (mechanism + TYPE-name or scan-function + connection/scan template). An extension exposing neither is not drivable and is unsupported by design — no bespoke per-extension code path. CONFORMANCE TARGETS (each is a real extension we do NOT ship an OOTB connector for, chosen to exercise a DIFFERENT descriptor branch than any hardcoded connector — passing them proves the descriptor is general, not a re-expression of what we already hardcode): (1) PG SQL/MED → mongo_fdw (EnterpriseDB/mongo_fdw). Reaches the mongo:7 service ALREADY in docker-compose.test.yml (seeded by db/mongo-init.js, requires_mongodb marker) — no new infra; only the FDW binary (libmongoc) is built. Descriptor: {extension: mongo_fdw, server_options:{address, port}, user_mapping:{username, password}, table_options:{database, collection}, supports_import: false}. Exercises the credentialed CREATE USER MAPPING path, a distinct SERVER key (address), NoSQL collection→foreign-table OPTIONS, and the explicit CREATE FOREIGN TABLE branch — none of which our OOTB PG connectors combine. Also proves cross-engine parity (mongo is reached OOTB by the DuckDB engine via DuckDBMongoConnector; here PG reaches it purely from config). (2) DuckDB ATTACH → ducklake (duckdb/ducklake). Self-contained: ATTACH 'ducklake:cat.ducklake' AS lake (DATA_PATH 'data/') over a local DuckDB/SQLite catalog + local parquet, no server. Exercises the URI-scheme-prefix DSN + extra ATTACH option (DATA_PATH) form, distinct from our stock ATTACH '' AS x (TYPE ). (3) DuckDB SCAN → excel / read_xlsx (duckdb/duckdb-excel). Self-contained: CREATE VIEW v AS SELECT * FROM read_xlsx('f.xlsx', sheet='Q1', header=true) over a local .xlsx, no server. Exercises a scan table-function with NAMED ARGS (sheet/range/header), distinct from our stock read_csv_auto/read_parquet.

Use case: BYO PG-federated or DuckDB-federated deployments with custom/proprietary extensions can extend reachability to new source types purely via config, without code changes or rebuilds. Extends REQ-898 config/pg_extension_catalog.yaml from a packaged-inventory model to an operator-authored connector descriptor.

Code: provisa/federation/custom_connectors.py, provisa/federation/engine.py, provisa/federation/pg_runtime.py, provisa/federation/duckdb_runtime.py, config/custom_connectors.yaml

Tests: tests/unit/test_custom_connectors.py, tests/integration/test_custom_connectors_e2e.py, tests/integration/test_custom_connectors_pg_e2e.py, tests/features/REQ-1177.feature

REQ-1178 · Connector Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Expand ClickHouse federation engine reach beyond the 7 OOTB source types by (1) adding OOTB native engines — SQLite (DATABASE engine, file, no server) and Hudi (lakehouse, zero-copy) — and (2) enabling operators to define config-driven ClickHouse connectors without code, in three kinds mirroring the engine's mechanisms: clickhouse_database (CREATE DATABASE ENGINE=… auto-expose, e.g. Redis), clickhouse_table (per-table CREATE TABLE ENGINE=…, columns from the registry, e.g. the JDBC/ODBC bridge), clickhouse_scan (CREATE TABLE ENGINE=… with ClickHouse inferring the schema, e.g. HDFS/URL). Availability is probed against ClickHouse's STANDARD discovery catalog (system.table_engines), failing loud when a declared engine is absent from the build.

Use case: ClickHouse's 50+ integration engines outreach today's OOTB types. Config-driven connector extensibility (parallel to REQ-1177) gives operators live federation to arbitrary databases without engineering. Streaming ingestion engines (Kafka/RabbitMQ/NATS) are out of scope — they follow the materialized-view ingestion model, not live pull reach.

Code: provisa/federation/clickhouse_connectors.py, provisa/federation/clickhouse_runtime.py, provisa/federation/custom_connectors.py, provisa/federation/engine.py, config/custom_connectors.yaml

Tests: tests/unit/test_clickhouse_connectors.py, tests/unit/test_custom_connectors.py, tests/integration/test_clickhouse_runtime_e2e.py, tests/features/REQ-1178.feature

REQ-1179 · Connector Configuration¶

Status: ✅ complete · Priority: MUST · Type: behavioral

mongo_fdw (EnterpriseDB PG SQL/MED connector) reaches MongoDB live end-to-end as a conformance target for REQ-1177. The extension binary is built from scripts/build_mongo_fdw.sh (mongo_fdw + bundled libmongoc 1.30.2 and json-c linked against embedded PG 16.2). The E2E test installs the extension dylibs into pgserver's embedded PG with @loader_path rpath so that mongo_fdw and libmongoc co-locate and resolve correctly. The descriptor (extension: mongo_fdw, server_options:{address, port}, user_mapping:{username, password}, table_options:{database, collection}, supports_import: false) drives config-only pg_fdw to read docker-seeded MongoDB.product_reviews without import.

Use case: Proves that REQ-1177 config-driven descriptors generalize beyond hardcoded connectors. mongo_fdw exercises a distinct descriptor branch (credentialed, explicit CREATE FOREIGN TABLE, separate SERVER key, NoSQL collection options) not combined by any OOTB PG connector, validating descriptor composability and cross-engine parity (MongoDB is already reachable OOTB via DuckDB).

Code: scripts/build_mongo_fdw.sh, provisa/federation/custom_connectors.py, provisa/federation/pg_runtime.py

Tests: tests/integration/test_custom_connectors_mongo_e2e.py

REQ-1180 · Connector Configuration¶

Status: ✅ complete · Priority: MUST · Type: constraint

GenericPgFdwConnector (provisa/federation/custom_connectors.py) emits a bare CREATE USER MAPPING ... SERVER s with NO OPTIONS clause when user_mapping is present but empty (e.g., user_mapping: {}). If user_mapping is absent, no mapping is created. If keys are present, OPTIONS(...) is emitted. This constraint enables no-auth FDW scenarios (e.g., mongo_fdw against unauthenticated MongoDB), where an empty username would force the driver to attempt a failing auth handshake.

Use case: Unauthenticated FDW targets (no credentials needed) require the bare SQL/MED form to avoid triggering unnecessary auth. This constraint prevents driver-level failures when credentials are not applicable.

Code: provisa/federation/custom_connectors.py

Tests: tests/unit/test_custom_connectors.py

9. Deployment & Release¶

REQ-1181 · Terraform Cloud Deployment¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Terraform GCP provisioning automatically stages release artifacts (AppImage and provisa-core-images-amd64-.zip) from GitHub releases into the target GCS bucket before VM boot via a configurable null_resource.stage_artifacts, gated by a stage_from_github bool (default true). Requires authenticated gh and gsutil on the operator machine.

Use case: Cloud deployments need release artifacts available in GCS before the VM starts. Automating the download and upload reduces manual operator steps and ensures artifact freshness for multi-region or repeated deployments.

Code: terraform/gcp/main.tf, terraform/gcp/variables.tf

Tests: —

REQ-1182 · Identity Provider Configuration¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Cloud deployments support automated identity-provider selection via an auth_provider Terraform variable (none|firebase|basic|keycloak|oauth|oidc). The variable is exported to the Compute Engine node as PROVISA_IDP, driving the server's env-based _auto_configure_idp logic. Firebase is a first-class option via firebase_project_id and firebase_service_account_key variables.

Use case: Cloud deployments require identity providers to be configured at boot time. Environment-based auto-configuration enables one-shot, reproducible deployments without post-launch manual setup or Terraform output re-entry.

Code: terraform/gcp/variables.tf, terraform/gcp/main.tf, provisa/api/setup_router.py

Tests: —

REQ-1183 · Linux Installation & Systemd Integration¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

The Linux first-launch installer persists all IdP-related environment variables (PROVISA_IDP, FIREBASE_PROJECT_ID, FIREBASE_SERVICE_ACCOUNT_KEY, KEYCLOAK_, OAUTH_) into a systemd EnvironmentFile at ~/.provisa/provisa.env, allowing the server running under systemd to resolve these variables for IdP auto-configuration without re-entry from the first-launch process.

Use case: Cloud VMs running under systemd (not a user login shell) require environment variables to be persisted to a file readable by the systemd unit. This decouples the first-launch process from the long-running server.

Code: provisa/cli/linux_installer.py, provisa/api/setup_router.py

Tests: —

REQ-1184 · Linux Installation & Systemd Integration¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

Status: ✅ complete · Priority: MUST · Type: constraint

Tests: —

REQ-1204 · Large Result Redirect & CTAS¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

REQ-1214 · Multi-Protocol Query Routing¶

Status: ✅ complete · Priority: MUST · Type: constraint

Every transport (gRPC, Arrow Flight, airport, GraphQL, JSON:API, Bolt, pgwire) routes through the single governed pipeline (_govern_and_route / _govern_and_route_compiled / _execute_plan) regardless of protocol. For ENGINE-route plans, raw-passthrough transports drain the engine's streaming terminal directly without materializing via _execute_plan; only genuinely-bounded routes (DIRECT native driver, metadata/admin, registered-function output) and format-buffered transports materialize through _execute_plan.

Use case: Centralizing routing through one pipeline ensures uniform governance (RLS, masking, audit), prevents per-transport SQL branches, and allows streaming transports to drain engine terminals directly while keeping buffered transports format-aware.

Code: provisa/executor/_pipeline.py

Tests: tests/unit/test_governed_chokepoint.py, tests/integration/test_grpc_execution.py

8. Deployment & Infrastructure¶

REQ-1215 · Multi-Protocol Exposure (gRPC)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

gRPC server-streaming query RPC (_handle_query in provisa/grpc/server.py) streams ENGINE-route query results lazily. Each row batch is pulled from the sync ResultStream (execute_engine_sync) through loop.run_in_executor, keeping the blocking cursor out of the event loop; peak memory is bounded to one batch. Governance stamp is verified via require_governed_plan before engine execution.

Use case: gRPC aio runs as an async generator over blocking engine cursors; wrapping ResultStream pulls in executor ensures the event loop never blocks and memory never exceeds one batch, enabling large result sets over gRPC.

Code: provisa/grpc/server.py

Tests: tests/unit/test_grpc_server.py, tests/integration/test_grpc_execution.py

REQ-1216 · Multi-Protocol Exposure (Arrow Flight)¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Arrow Flight SQL do_get ENGINE route streams via execute_engine_stream. All Flight-serving engines must declare EngineCapability.ARROW_STREAM only if run_arrow_stream genuinely streams via server-side chunk APIs (fetch_arrow_batches, to_arrow_iterable, fetchmany→Arrow). CONFORMANCE GAP: Databricks, BigQuery, MSSQL declare ARROW_STREAM but call run_arrow then .to_batches(), materializing the whole table. See docs/arch/streaming-uniformity-gap.md Defect 2.

Use case: Streaming via execute_engine_stream replaces full materialization with lazy per-batch pulls, enabling large result sets over Flight. Engine capability declarations must be honest: only declare ARROW_STREAM when the terminal is genuinely lazy via the driver's server-side chunk API.

Code: provisa/api/flight/server.py, provisa/executor/_engine_executor.py

Tests: tests/unit/test_flight_modes.py, tests/integration/test_arrow_flight_integration.py

7. Result Delivery¶

REQ-1217 · Engine Streaming Terminals¶

Status: ✅ complete · Priority: MUST · Type: constraint

Per-engine Arrow streaming terminals (execute_engine_stream) and run_sync must be genuinely lazy. DuckDB uses fetch_record_batch (_ARROW_STREAM_BATCH_ROWS = 65536). Snowflake uses fetch_arrow_batches. CONFORMANCE GAP: Databricks, BigQuery, MSSQL fake laziness by calling run_arrow/.to_batches(), materializing before batching. These must use server-side chunk APIs (BigQuery to_arrow_iterable, Databricks Cloud Fetch iteration, MSSQL cursor fetchmany→Arrow). See docs/arch/streaming-uniformity-gap.md Defects 2-3.

Use case: Genuinely lazy per-batch pulls prevent materializing full result sets in memory and allow the consumer to stop early without forcing the engine to compute remaining batches. Only server-side chunk APIs honor this for warehouse engines; materializing then rebatching violates the design.

Code: provisa/executor/_engine_executor.py

Tests: tests/unit/test_duckdb_stream_terminal.py, tests/integration/test_preflight_streaming.py

REQ-1218 · Protocol-Specific Result Handling¶

Status: ✅ complete · Priority: MUST · Type: constraint

Airport Flight transport drains the single streaming terminal identically to Flight SQL (REQ-1216). Byte-stable schema advertisement comes from the plan's typed output columns (known pre-execution), not by scanning rows. The is_rowid pseudo-column derives from source key metadata, a streamed rowid column, or an off-heap CTAS side-table — never an in-Provisa full-table cache. Airport is a streaming transport, not a materializing catalog-scan transport. See docs/arch/streaming-uniformity-gap.md Defect 5.

Use case: Draining the streaming terminal prevents OOM on large table scans in airport. Schema and rowid metadata are catalog-identity concerns separable from the result-streaming path. Off-heap CTAS satisfies unbounded rowid mappings without materializing in Provisa RAM.

Code: provisa/api/airport/query.py, provisa/api/airport/server.py

Tests: tests/integration/test_airport_source_e2e.py, tests/integration/test_airport_service_e2e.py

1. Cluster Configuration & Startup¶

REQ-1219 · First-Run Setup Wizard¶

Status: ✅ complete · Priority: MUST · Type: constraint

The first-run setup wizard layers an auth section onto a base config. When no config file exists, read_config_for_setup() in provisa/api/admin/_config_io.py falls back to the shipped minimal skeleton provisa-install-base.yaml (system sources/domains + built-in admin role, auth: none). The skeleton guarantees ProvisaConfig can be parsed after the wizard writes its auth section, ensuring _load_and_build always has a valid configuration to parse.

Use case: First-run installs with no prior config file must produce a valid configuration that _load_and_build can parse. The skeleton provides sources/domains/tables/roles (no defaults possible) and allows the wizard to focus on auth layering, guaranteeing config validity on startup.

Code: provisa/api/admin/_config_io.py, provisa/cli.py, scripts/build-wheel.sh

Tests: tests/unit/test_setup_base_config.py

8. Deployment & Infrastructure¶

REQ-1220 · Demo Deployment Parity¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

On demo deployment, the app container (uvicorn main:app) loads a complete config rather than dropping into the first-run wizard. The Dockerfile bakes config/ and demo SQLite sample data into /app/config (demo data under /app/config/demo/files to avoid the ./demo compose bind-mount shadowing). packaging/linux/first-launch.sh::write_demo_overlay() writes ~/.provisa/extensions/demo/docker-compose.demo.yml exporting PROVISA_CONFIG=/app/config/provisa-install.yaml, PROVISA_DEMO=1, PROVISA_DEMO_DIR=/app/config/demo/files when INSTALL_DEMO=y and ROLE=primary (secondaries pull shared config from the primary DB). scripts/provisa auto-includes the overlay.

Use case: Demo deployments must start with a populated config to avoid the first-run wizard and provide users with immediate sample data to explore. Docker/Linux demo parity with native launch (commit f7289d27) requires baking config and demo data into the image and wiring environment variables through the overlay mechanism.

Code: Dockerfile, packaging/linux/first-launch.sh, scripts/provisa

Tests: tests/unit/test_infra_requirements.py

7. Result Delivery¶

REQ-1221 · Arrow Streaming Adapter¶

Status: ✅ complete · Priority: MUST · Type: behavioral

The generic row→Arrow-batch adapter (arrow_batches_from_rows in provisa/federation/runtime_support.py) enables ARROW and ARROW_STREAM transports on ROWS-only federation engines (Postgres via psycopg2, SQLAlchemy). It packs Python rows into memory-bounded Arrow RecordBatches (65536 rows/batch default), locks the Arrow schema from the first batch and casts subsequent batches to it, converts Decimal→float, and maps empty results to a null-typed schema.

Use case: Postgres and SQLAlchemy engines have no native Arrow reader (only row-based results), but clients expect Arrow streaming via Flight-SQL and airport transports. The adapter bridges this gap by materializing batches incrementally instead of buffering the entire result set, enabling lazy streaming over large result sets without requiring zero-copy native Arrow support.

Code: provisa/federation/runtime_support.py, provisa/federation/native_backend.py, provisa/federation/engine.py

Tests: tests/unit/test_runtime_support.py, tests/unit/test_native_arrow_transport.py, tests/unit/test_engine_capability_traits.py

REQ-1222 · Engine Streaming Terminals¶

Status: ✅ complete · Priority: MUST · Type: behavioral

PgFederationRuntime gains run_arrow(sql, params) and run_arrow_stream(sql, params) methods for zero-copy Arrow transport via adbc_driver_postgresql. run_arrow returns a materialized pyarrow.Table; run_arrow_stream returns (schema, batch_generator) with record batches fetched on-demand from a dedicated short-lived ADBC connection. Postgres rows decode directly into Arrow RecordBatches without Python row materialization; streaming variant is memory-bounded by one batch size. Dependency on adbc-driver-postgresql added to pyproject.toml.

Use case: PostgreSQL federation engine can expose Arrow streaming to Flight SQL and airport transports, enabling large result sets without buffering rows in Python memory or materializing the full result set client-side.

Code: provisa/federation/pg_backend.py

Tests: tests/integration/test_streaming_memory_bounded_e2e.py

REQ-1223 · Engine Streaming Terminals¶

Status: ✅ complete · Priority: MUST · Type: behavioral

SqlAlchemyFederationRuntime.run_sync classifies SQL statements by leading keyword (after stripping leading comments) to identify ROW-RETURNING queries (SELECT, WITH, VALUES, TABLE, SHOW, EXPLAIN). Row-returning statements stream via server-side named cursor using SQLAlchemy's stream_results execution option over psycopg2, replacing the prior unnamed cursor that buffered the entire result on execute(). Non-row-returning statements (DDL, DML) execute buffered in a committing transaction because psycopg2 syntax-errors on DECLARE CURSOR FOR DDL/DML. The streaming read runs on a dedicated connection closed when the stream drains.

Use case: Server-side named cursors decouple result fetching from statement execution, allowing large result sets to remain on the server and be pulled incrementally without buffering client-side, enabling memory-bounded streaming on federation engines (Trino, Snowflake, Databricks, etc.) that use SQLAlchemy.

Code: provisa/federation/sqlalchemy_backend.py

Tests: tests/integration/test_streaming_memory_bounded_e2e.py

REQ-1224 · Large Result Redirect & CTAS¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Automatic stream↔CTAS threshold at the single terminal (_execute_plan). Buffered transports (JSON:API, GraphQL, Bolt) land an engine-native CTAS via run_materialize above a configurable row-count/byte threshold and inline the body below it — automatically, without requiring a caller to pass deliver= via side-channel. Threshold is a system configuration (e.g., 10K rows, 100MB). See docs/arch/streaming-uniformity-gap.md Defect 4.

Use case: Buffered transports cannot stream incrementally to the wire and require materialization. An automatic threshold at the terminal (not per-caller) prevents memory OOM on unexpectedly large results by using engine-native CTAS for large sets while keeping small results inline for latency.

Code: provisa/executor/redirect.py, provisa/pgwire/_pipeline.py, provisa/api/jsonapi/generator.py, provisa/api/rest/generator.py, provisa/bolt/session.py

Tests: tests/unit/test_buffered_auto_threshold.py

REQ-1225 · Engine Streaming Terminals¶

Status: ⚙ in-progress · Priority: MUST · Type: behavioral

Warehouse federation runtimes (Snowflake, Databricks, BigQuery, MSSQL, ClickHouse) must implement genuinely streaming run_sync terminals using their respective server-side chunk APIs. Each runtime builds run_sync on its single lazy Arrow primitive via stream_rows_from_arrow(schema, batches) in provisa/federation/runtime_support.py, returning a StreamingQueryResult (single-consume, one batch pulled at a time) rather than a fully materialized QueryResult. The async run terminal materializes via run_async_materialized, which drains the streaming run_sync on an executor thread, mirroring the PostgreSQL runtime split. stream_rows_from_arrow is the inverse primitive of arrow_batches_from_rows.

Use case: Warehouse run_sync currently materializes the full result set, breaking memory boundedness on the pgwire ENGINE route despite REQ-1186. A genuinely streaming run_sync keeps the pgwire route OOM-safe for large results. See docs/arch/streaming-uniformity-gap.md Defect 3.

Code: provisa/federation/snowflake_runtime.py, provisa/federation/databricks_runtime.py, provisa/federation/bigquery_runtime.py, provisa/federation/mssql_warehouse_runtime.py, provisa/federation/clickhouse_runtime.py, provisa/federation/runtime_support.py

Tests: tests/integration/test_streaming_memory_bounded_e2e.py

8. Deployment & Infrastructure¶

Tests: tests/integration/test_airport_service_e2e.py, tests/integration/test_streaming_memory_bounded_e2e.py

3. Multi-tenancy & Organization¶

REQ-1232 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: constraint

Every credential (API key / token / session) is scoped to exactly one organization. Org identity is always determined by the credential's active_org_id in the authentication middleware, never by TLS SNI, HTTP header, or first-membership fallback.

Use case: Ensures that a credential cannot be used to access data from a different organization. Users who hold memberships in multiple organizations must obtain a distinct org-scoped credential per organization.

Code: provisa/auth/middleware.py

Tests: tests/integration/test_auth_org_scoping.py, tests/unit/test_auth_middleware.py

9. Multi-Org SaaS Routing¶

REQ-1233 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: infrastructure

The subdomain {org}.provisa.dev is the org identity. cloud.provisa.dev is the org-agnostic entry point (login + org picker). Every data/control endpoint is addressed by the org subdomain: bolt://{org}.provisa.dev, postgresql://…@{org}.provisa.dev, the Arrow Flight Location, grpc+tls://{org}.provisa.dev, MCP, and https://{org}.provisa.dev/data.

Use case: Subdomain-based org identity is ergonomic for multi-org SaaS and self-integrating with DNS/TLS (every subdomain is a distinct DNS name and TLS certificate request). It requires no protocol-specific multiplexing headers.

Code: provisa/auth/middleware.py, provisa/api/app.py

Tests: tests/integration/test_subdomain_routing.py

REQ-1234 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: behavioral

The subdomain selector is delivered in two transports: (1) HTTP Host header for web/REST/GraphQL, (2) TLS SNI on wire protocols (pgwire, bolt, Arrow Flight, gRPC, MCP). One string, one job. Wire protocol servers currently do wrap_socket(server_side=True) with NO sni_callback; an sni_callback must stash the indicated host per-connection so downstream auth/routing can read it.

Use case: SNI extraction on wire protocols allows the same org-routing logic to apply to pgwire/bolt/Arrow Flight connections without protocol-specific header mechanisms. All clients resolve the org name via DNS; TLS SNI carries it transparently.

Code: provisa/pgwire/server.py:442, provisa/bolt/server.py:213, provisa/auth/middleware.py

Tests: tests/integration/test_wire_sni_extraction.py

REQ-1235 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: constraint

The subdomain never authorizes — it only names the org. The authenticated credential must authorize that org: an org-scoped credential's active_org_id must EQUAL the subdomain org (reject on mismatch); a person-scoped (OIDC) identity must be a MEMBER of the subdomain org (check user_org_memberships) else reject. No default org, no fallback — an unnamed or unauthorized org is a rejection, never a guess.

Use case: Subdomain naming + credential authorization prevents a credential intended for org-acme from accessing org-beta data. A person can hold memberships in multiple orgs but must use org-matched credentials or re-authenticate for each org.

Code: provisa/auth/middleware.py:240-280

Tests: tests/integration/test_auth_org_scoping.py, tests/unit/test_auth_middleware.py

REQ-1236 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: constraint

For wire protocols, the credential is org-scoped (per REQ-1232), so SNI is a cross-check: parse org from SNI, reject the connection if it ≠ the credential's active_org_id. Catches the error case of right-credential/wrong-subdomain.

Use case: Wire SNI cross-check prevents typos or DNS hijack risks where a client connects to the wrong subdomain with a valid but mismatched credential. Catches human error early in TLS handshake.

Code: provisa/pgwire/server.py, provisa/bolt/server.py, provisa/auth/middleware.py

Tests: tests/integration/test_wire_sni_cross_check.py

REQ-1237 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: behavioral

OIDC/social identity is person-scoped; Provisa mints the org-scoped session. Google/OIDC providers (e.g., provisa/auth/providers/oauth.py:74-76, keycloak.py:68-70) return AuthIdentity(user_id=sub, email=…) with NO active_org_id — the IdP proves the person, never the org. After IdP auth completes, Provisa mints its OWN org-scoped session (sets active_org_id) gated by a membership check. The subdomain selects the org pre-auth, so no org-picker screen is needed: land on acme.provisa.dev → OIDC login → membership check on acme → acme-scoped session returned.

Use case: Person-scoped IdP identity + Provisa-minted org-scoped session separates authentication (IdP: who am I?) from authorization (Provisa: which orgs can I access?). Enables subdomain-driven org selection without extra UI screens.

Code: provisa/auth/providers/oauth.py:74-80, provisa/auth/providers/keycloak.py:68-70, provisa/auth/middleware.py:240-280

Tests: tests/integration/test_oidc_to_org_session.py

REQ-1238 · Org Identity & Subdomain Addressing¶

Status: ✓ accepted · Priority: MUST · Type: constraint

The x-org-id header path in provisa/auth/middleware.py:252-253 currently passes the header without checking user_org_memberships — a user could pass any org_id and gain cross-org access. This is the interactive org-switcher lane for a person-scoped session with no subdomain (e.g., POST to /api/switch-org with x-org-id). The passed org MUST be validated against membership; reject if not a member. This corrects a defect that was previously masked by single-org deployments.

Use case: Header-path membership validation closes a privilege-escalation hole where a person-scoped session could be tricked into accessing an org they are not a member of via header manipulation.

Code: provisa/auth/middleware.py:251-253

Tests: tests/integration/test_x_org_id_validation.py, tests/unit/test_auth_middleware.py

REQ-1239 · Deployment & Infrastructure¶

Status: ✓ accepted · Priority: MUST · Type: infrastructure

Wildcard TLS for *.provisa.dev via ACME DNS-01. Obtain a true wildcard *.provisa.dev cert from Let's Encrypt using ACME DNS-01 with a scoped Cloudflare API token (Zone.DNS:Edit on provisa.dev only). One wildcard certificate covers cloud.provisa.dev and every {org}.provisa.dev across all protocols (SNI matches regardless of port). Cert is written to PROVISA_TLS_CERT and PROVISA_TLS_KEY paths that packaging/linux/first-launch.sh:443-451 already adopts. Node-side renewal timer (90-day). Replaces the self-signed dev stand-in for public SaaS deploys.

Use case: .dev is HSTS-preloaded, so a real trusted cert is mandatory for any public deploy. A single wildcard cert covers unlimited subdomains without per-subdomain provisioning or Cloudflare TLS termination (which would break wire protocols). Automated ACME renewal ensures uptime across certificate lifecycle.

Code: packaging/linux/first-launch.sh:443-451, provisa/api/app.py, provisa/pgwire/server.py, provisa/bolt/server.py

Tests: tests/integration/test_wildcard_cert_provisioning.py, tests/e2e/test_acme_dns01_renewal.py

REQ-1240 · Deployment & Infrastructure¶

Status: ✓ accepted · Priority: MUST · Type: infrastructure

Cloudflare DNS split: Zone provisa.dev stays on Cloudflare. provisa.dev + www.provisa.dev remain Cloudflare-proxied (orange) for the marketing site. cloud.provisa.dev and *.provisa.dev are DNS-only (grey-cloud) records pointing at the GCP L4 LB IPs, so raw TCP reaches the LB and in-container TLS terminates. Cloudflare proxy carries only HTTP(S) and would terminate TLS itself, breaking wire protocols. No GCP Cloud DNS zone needed; DNS-01 ACME uses the Cloudflare token.

Use case: DNS-only records for app subdomains allow wire protocols (pgwire, bolt, Arrow Flight, gRPC, MCP) to reach the load balancer with raw TCP, where in-container TLS terminates per-port. Cloudflare proxy on marketing site keeps SEO and CDN benefits without interfering with protocol servers.

Code: packaging/linux/first-launch.sh, infrastructure/cloudflare-setup

Tests: tests/integration/test_dns_split_routing.py

REQ-1241 · Deployment & Infrastructure¶

Status: ✓ accepted · Priority: MUST · Type: constraint

A session is always scoped to exactly one org and therefore one federation engine. state.federation_engine is a per-process singleton built once at startup (provisa/api/app.py:270), read with no org parameter at provisa/api/_query_helpers.py:126, so one process serves one engine. Subdomain-pinned single-org sessions (per REQ-1235) enforce this invariant: each org gets its own Provisa instance (or process) with its own federation engine binding. Until per-org engine routing is built (REQ-1244), sessions cannot switch engines without re-launching the process.

Use case: Single-engine-per-session is a hard invariant that simplifies caching, planning, and freshness logic. Processes are stateless and can be scaled horizontally per-org or per-engine-instance. Cross-engine session switching requires building an org→engine registry and replacing the singleton (future work, REQ-1244).

Code: provisa/api/app.py:270, provisa/api/_query_helpers.py:126

Tests: tests/unit/test_session_org_invariant.py

REQ-1242 · Deployment & Infrastructure¶

Status: ✓ accepted · Priority: SHOULD · Type: behavioral

In-session org switch (e.g., click "switch to acme" in UI) tears down the org-scoped session and mints a new one for the target org (membership-checked), REUSING the already-proven IdP identity (no password/IdP round-trip required), then redirects to {org}.provisa.dev. It is a scoped re-login, NOT a live cross-engine swap — it preserves the single-engine-per-session invariant (REQ-1241). Future work (REQ-1244) will enable live multi-engine switching without re-auth.

Use case: Org-switch without IdP round-trip is ergonomic for multi-org users. Redirecting to the target subdomain re-establishes DNS and TLS SNI for clean cross-org addressing. Re-minting the session (not retaining it) respects the single-engine invariant until multi-engine routing is built.

Code: provisa/api/auth_router.py, provisa/auth/middleware.py

Tests: tests/integration/test_org_switch_ergonomic.py, provisa-ui/e2e/org-switcher.spec.ts

REQ-1243 · Deployment & Infrastructure¶

Status: ✓ accepted · Priority: MUST · Type: structural

Two isolation lanes coexist: (a) Pooled/shared Trino cluster — the free/small tier, "trino-level isolation," all orgs start here. (b) BYO (bring-your-own) cluster — a customer points Provisa at their OWN federation engine instance (e.g., Databricks, Snowflake); self-service, the customer may change their own fed-engine instance, billed separately. Both lanes route through the SAME _govern_and_route/_execute_plan pipeline; routing is org-specific at deployment time (which instance serves which org), not in-session switching (REQ-1244 defers per-org dynamic engine selection).

Use case: Dual-lane model serves free/small orgs economically on shared infrastructure while enabling enterprise customers to run on their own data warehouses. Unified pipeline means no code branching and shared planner/cache/freshness logic.

Code: provisa/api/app.py:270, provisa/compiler/_govern_and_route, provisa/federation/runtime.py

Tests: tests/integration/test_pooled_lane.py, tests/integration/test_byo_lane.py

REQ-1244 · Deployment & Infrastructure¶

Status: 💡 proposed · Priority: SHOULD · Type: constraint

Per-org engine routing is deferred. In-session multi-engine switching requires replacing the per-process singleton (app.py:270 / _query_helpers.py:126) with an org→engine registry that feeds the SAME _govern_and_route/_execute_plan pipeline. Until built, heterogeneous engines are served via subdomain-pinned single-org/single-engine sessions (REQ-1241/1242). Future work: build org-scoped engine registry, keyed by org_id, read at query time to select the federation engine per-session org.

Use case: Dynamic per-org engine routing (without re-auth) enables Provisa to serve one org with Trino, another with Snowflake, and a third with Databricks from the SAME deployment and process. Preserves the single-engine-per-session invariant while allowing org-to-engine mapping to change at deployment time.

Code: provisa/api/app.py:270, provisa/api/_query_helpers.py:126, provisa/federation/runtime.py

Tests: tests/integration/test_per_org_engine_routing.py

REQ-1245 · Deployment & Infrastructure¶

Status: 💡 proposed · Priority: MUST · Type: infrastructure

Control-plane persistence is required for multi-instance fleet deployments. The control_plane store is currently in-memory (provisa/control_plane/store.py:16, "V1: no DB persistence") — a blocker for running multiple Provisa instances behind a load balancer. Multiple instances cannot share state (sources, roles, registered tables, relationships, etc.) and cannot coordinate cache invalidation or governance updates. Persistence must use the same org-scoped control-plane backend (PostgreSQL/SQLite/MySQL) that holds platform state (orgs, users, memberships), ensuring each org's control plane is isolated and durable.

Use case: Persisted control plane enables horizontal scale-out of Provisa instances, load-balanced by subdomain org, each instance serving its own org's control-plane state from the shared backend. Required for production SaaS deployments with high availability.

Code: provisa/control_plane/store.py:16, provisa/core/database.py, provisa/core/schema_admin.py

Tests: tests/integration/test_control_plane_persistence.py, tests/integration/test_multi_instance_coordination.py

2. Authentication & Identity¶

REQ-1246 · Authentication¶

Status: 💡 proposed · Priority: MUST · Type: behavioral

First verified Firebase login provisions a user_profile record, decoupled from the local_users/bcrypt-password path. No password is created for Firebase identities. Firebase ID token serves as the credential. Provisioning occurs in the initial POST /auth/register on a Firebase identity (REQ-121 validates Firebase auth; REQ-124 is hard-gated to provider=="basic" and cannot onboard Firebase identities).

Use case: Enables self-service signup on cloud.provisa.dev with Firebase as the primary identity provider, separating password-managed local users from federated identities.

Code: provisa/api/auth_router.py, provisa/auth/, provisa/core/schema_admin.py

Tests: tests/integration/test_firebase_signup.py, tests/unit/test_firebase_user_provisioning.py

11. Frontend UI & UX¶

REQ-1247 · Authentication¶

Status: 💡 proposed · Priority: MUST · Type: ui

Self-service signup landing on cloud.provisa.dev presents two options to a not-logged-in user: LOG IN or CREATE ACCOUNT. The choice routes to distinct flows: LOG IN → Firebase login (existing user or new self-registration via Firebase), CREATE ACCOUNT → signup flow including org membership selection (REQ-1248 / REQ-1249 / REQ-1250).

Use case: Org-agnostic entry point ergonomically guides new users into account creation while retaining fast login for existing users.

Code: provisa-ui/pages/auth/landing.tsx, provisa-ui/components/auth/signup-choice.tsx

Tests: provisa-ui/e2e/auth-landing.spec.ts

3. Multi-tenancy & Organization¶

REQ-1248 · Org Membership¶

Status: 💡 proposed · Priority: MUST · Type: behavioral

During account creation, a Firebase-authenticated user auto-joins an org if they hold a valid org_invites token (via invite URL or code). Provisioning reuses existing invite validation/consumption logic, upserting user_org_memberships at the inherited role from the invite record. If no invite is held, they proceed to REQ-1249 or REQ-1250 (create new org or request admission to existing org).

Use case: Decouples Firebase signup from org membership: invited users skip org selection, while non-invited users choose between creating a new org or requesting membership in an existing one.

Code: provisa/api/auth_router.py, provisa/api/admin/invites_router.py

Tests: tests/integration/test_signup_with_invite.py

REQ-1249 · Org Membership¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

During account creation, a Firebase user without an org invite may CREATE A NEW ORG (self-service): inserts an orgs row with the user as owner/admin, inserts user_org_memberships at owner/admin role, provisions the per-org schema org_{id}, and registers a control-plane tenant initialized to the shared/pooled Trino cluster (REQ-1243/1244). The createOrg endpoint (currently admin-only /admin/orgs) becomes available as part of the signup flow.

Use case: Enables self-service multi-org SaaS: new users can immediately spin up a workspace (org) without waiting for admin provisioning.

Code: provisa/api/auth_router.py, provisa/api/admin/orgs_router.py, provisa/core/schema_admin.py

Tests: tests/integration/test_signup_create_org.py

REQ-1250 · Org Membership¶

Status: 💡 proposed · Priority: SHOULD · Type: structural

Org membership is pull-based via org_join_requests, the inverse of org_invites. New registry table org_join_requests(user_id, org_id, status, requested_at, decided_by, decided_at) is added to REGISTRY_TABLES in provisa/core/schema_admin.py. Per-org visibility flag orgs.open_to_requests: if False (invite_only, the default), admission requests are silently rejected (no signal that the request was received, to avoid org-name enumeration). If True (open_to_requests), requests are logged pending admin approval. Requesters must be Firebase-authenticated. Anti-abuse: at most one open request per (user, org); rate-limited per requester; stale requests auto-expire (same TTL as org_invites).

Use case: Pull-based membership model enables users to request admission to public orgs without requiring org admins to discover and invite them. Silently rejecting invite_only requests prevents org enumeration attacks.

Code: provisa/core/schema_admin.py

Tests: tests/integration/test_org_join_request_table.py, tests/unit/test_org_join_request_visibility.py

REQ-1251 · Org Membership¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

Admin approval/denial of org_join_requests. When an admin approves a request, the approval path reuses the membership-insert logic (provisa/api/auth_router.py and related) minus the token check, granting the lowest-privilege member role (never admin). Deny path closes the request (status='denied'). Both actions record decided_by (admin user_id) and decided_at.

Use case: Enables org admins to gate membership while supporting the pull-based admission model.

Code: provisa/api/admin/orgs_router.py, provisa/api/auth_router.py

Tests: tests/integration/test_org_join_request_approval.py

10. Admin Console & Governance¶

REQ-1252 · Org Administration¶

Status: 💡 proposed · Priority: SHOULD · Type: ui

Admin UI surface (extend OrgsTab.tsx / add admin router endpoint) to list pending org_join_requests for the org, approve requests, and deny requests. Display requester email, request date, and admin action buttons.

Use case: Provides org admins with a centralized interface to manage pull-based membership requests.

Code: provisa-ui/admin/orgs/OrgsTab.tsx, provisa/api/admin/orgs_router.py

Tests: provisa-ui/e2e/admin-join-requests.spec.ts

8. Deployment & Infrastructure¶

REQ-1253 · Cloud Load Balancing¶

Status: 💡 proposed · Priority: MUST · Type: infrastructure

All protocol surfaces (API port 8000, UI port 3000, Arrow Flight 8815, pgwire 5439, Bolt 7687, MCP 8009, gRPC 50051) must be fronted by a single shared external passthrough load-balancer endpoint per cloud provider, not one LB per protocol. This is required by the subdomain-as-org model (REQ-1233): {org}.provisa.dev must resolve to one A record and reach every protocol by preserving the destination port to the backend node.

Use case: Enables the subdomain-as-org identity model where a single DNS name and IP address serve all protocols. Cloud-specific realizations: GCP uses a backend-service passthrough NLB forwarding rule with all_ports=true on one static IP; AWS uses a single Network Load Balancer with one listener/target-group per protocol port sharing the NLB's endpoint; Azure uses a single Standard Load Balancer with one LB rule per protocol port on a single frontend public IP. Backend liveness is gated by the API HTTPS /health probe. Refs: REQ-1233, REQ-1239, REQ-1240, REQ-1227.

Code: —

Tests: —

REQ-1254 · Cloud Load Balancing¶

Status: ✅ complete · Priority: MUST · Type: infrastructure

Web UI must be published on host port 443 across all cloud Terraform deployments (GCP, AWS, Azure). The UI container continues listening on port 3000 with TLS; docker-compose publishes ${UI_PORT}:3000 with UI_PORT set to 443 in cloud environments. first-launch.sh persists UI_PORT into the systemd EnvironmentFile so provisa start picks it up.

Use case: The .dev gTLD is HSTS-preloaded in browsers, forcing http to https:443. Publishing the UI on port 443 allows cloud.provisa.dev and {org}.provisa.dev URLs to resolve without a port suffix, composing with the single shared passthrough LB (REQ-1253) which forwards all ports to the backend node preserving destination port.

Code: first-launch.sh, terraform/gcp/main.tf, terraform/aws/main.tf, terraform/azure/main.tf, docker-compose.yml

Tests: —

3. Multi-tenancy & Organization¶

REQ-1255 · Org Membership¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

REQ-1260 · Test Infrastructure¶

Status: ✅ complete · Priority: SHOULD · Type: infrastructure

REQ-1268 · Per-Request Multi-Org Data Plane¶

Status: ✅ complete · Priority: MUST · Type: structural

OrgRuntime dataclass encapsulates a single org's slice of AppState; OrgRegistry holds dict[org_id, OrgRuntime] + per-org asyncio.Lock for safe double-checked builds with no TTL. current_org ContextVar selects the active org; AppState property shims resolve the ContextVar-selected runtime or default to the default-org runtime when unset. require_current_org() raises when unset (no silent fallback).

Use case: Enables request-scoped multi-org isolation: each request runs against its designated org's isolated state, catalogs, and connection pools without global state mutation or cross-org leakage.

Code: provisa/app_state.py, provisa/multitenancy/org_runtime.py

Tests: tests/unit/test_org_runtime.py

REQ-1269 · Org-Prefixed Catalog Naming¶

Status: ✅ complete · Priority: MUST · Type: constraint

org_prefixed_catalog(org_id, base, *, default_org) returns bare base name for default org, else org_{id}__{base}. System/fixed-warehouse catalogs are never prefixed. Prevents cross-org catalog name collisions when multiple orgs share the same demo source IDs.

Use case: Avoids catalog name collisions in shared orchestrator environments (e.g., pooled Trino) when multiple orgs instantiate identical demo sources.

Code: provisa/multitenancy/catalog_naming.py

Tests: tests/unit/test_org_isolation.py, tests/integration/test_org_auto_join.py

REQ-1270 · HTTP Org-Routing Middleware¶

Status: ✅ complete · Priority: MUST · Type: behavioral

OrgRoutingMiddleware routes each HTTP request to the active org's runtime, built on-demand from persisted seeded_demo config, gated on multitenancy feature flag. Resolves current_org ContextVar before handlers execute.

Use case: Ensures every request executes within its designated org's isolated AppState, enforcing tenant isolation at the HTTP boundary.

Code: provisa/middleware/org_routing.py

Tests: tests/unit/test_org_isolation.py

REQ-1271 · Self-Service Org Creation¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Any authenticated (non-anonymous) user may POST /admin/orgs and becomes org_admin; anonymous requests rejected 401 under auth. Membership is granted synchronously in the admin plane; org_admin role assignment in the tenant plane granted asynchronously after schema provisioning.

Use case: Enables users to self-provision new orgs without platform admin intervention, supporting multi-org SaaS workflows.

Code: provisa/api/admin/orgs_router.py

Tests: —

REQ-1272 · Async Org Provisioning¶

Status: ✅ complete · Priority: MUST · Type: behavioral

POST /admin/orgs returns immediately with provisioning_state="provisioning"; a background task executes provision_org + build_org_runtime + grant_org_role, then updates provisioning_state to "ready" (or "failed"+provisioning_error). orgs table gains provisioning_state, provisioning_error, seeded_demo columns. Errors are persisted, not swallowed.

Use case: Decouples org creation from slow provisioning operations (schema build, connection pool initialization), enabling responsive UI feedback during multi-org onboarding.

Code: provisa/api/admin/orgs_router.py, provisa/multitenancy/org_provisioning.py

Tests: tests/integration/test_create_org_onboarding.py

REQ-1273 · Org Readiness Notification Seam¶

Status: ✅ complete · Priority: MAY · Type: behavioral

notify_org_ready(org_id, user_id) stub called when async provisioning completes; currently logs only. Defers email implementation; reserved for future notification channels (email, in-app, webhooks).

Use case: Provides a seam to inject notifications (email, dashboard alerts) once an org is ready for use, supporting future multi-channel notification infrastructure.

Code: provisa/multitenancy/org_provisioning.py

Tests: —

REQ-1274 · Org Admin Authorization Fix¶

Status: ✅ complete · Priority: MUST · Type: constraint

_require_org_admin authorization handler allows platform admins any org, confines org_admin to their active_org_id (backed by admin-plane membership row), allows dev/no-auth, rejects all else. Closes authz hole in create_invite/list_invites/revoke_invite endpoints.

Use case: Prevents org admins from inviting users to orgs they do not own, enforcing multi-org isolation boundary at the authorization layer.

Code: provisa/api/admin/auth.py, provisa/api/admin/invites_router.py

Tests: tests/unit/test_tenant_isolation_contract.py

REQ-1275 · UI Onboarding Flow¶

Status: ✅ complete · Priority: SHOULD · Type: ui

Member-less authenticated user (authEnabled && token && orgMemberships.length===0) is gated to OnboardOrgPage. Form accepts org_id, org_name, include_demo checkbox, submits to POST /admin/orgs, polls provisioning status until ready/failed, binds org to localStorage (provisa_org), calls selectOrg, refreshes identity via AuthContext.refresh(), then SPA-navigates to /query. AuthContext now exposes refresh() method.

Use case: Guides newly authenticated users with no org membership through self-service org creation and onboarding, automating post-signup navigation and org binding.

Code: provisa-ui/src/pages/OnboardOrgPage.tsx, provisa-ui/src/context/AuthContext.tsx

Tests: —

REQ-1276 · Subdomain-based Org Resolution¶

Status: ✅ complete · Priority: MUST · Type: behavioral

HTTP request org is authoritative from Host subdomain (leftmost label of acme.provisa.org → org acme), except control-plane host cloud.provisa.* which requires explicit x-org-provisa header. Apex/bare hosts (provisa.org, localhost) resolve to no org (None). Subdomain/header replaces prior active_org token claim and x-org-id header. Membership validation unchanged: org not in user's orgs is rejected 403 (unless user is platform admin).

Use case: Establishes Host subdomain as the authoritative org identity source across HTTP transports, enabling transparent multi-org routing without requiring org claims in tokens or API headers. Exception for control-plane preserves bootstrap and cross-org admin access patterns.

Code: provisa/auth/middleware.py

Tests: tests/unit/test_auth_middleware_multitenancy.py

2. Authentication & Identity¶

REQ-1277 · Member-less User Onboarding¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

REQ-1280 · Commercial Positioning¶

Status: ✓ accepted · Priority: MAY · Type: infrastructure

Provisa SaaS bills on two metered SKUs mapped to the existing gcp-saas topology (REQ-1279): (1) Serving lane — the "active-hour" SKU at $3.25/active-hr, metering warm coordinator uptime (the always-on Trino planner + persistent-protocol listeners: pgwire/bolt/Flight/gRPC/MCP). Any hour the endpoint is active bills a full hour. Margin ~87–97%. (2) Analytical lane — the "worker-hour" SKU at $2.50/worker-hr, metering Spot worker MIG uptime per query, scale-to-zero between queries. Margin ~97% Spot / ~89% on-demand. Egress is a cost-recovery passthrough across both lanes at Hasura parity ($0.13/GB vs $0.12 cost); only result bytes leaving GCP are Provisa's cost (source-cloud egress bills the source owner).

Use case: A customer buys either or both SKUs. Splitting the SKU captures both markets; a single unit misprices one (active-hour framing underprices analytical compute ~8x; worker-hour framing cannot express a warm always-on endpoint).

Code: —

Tests: —

REQ-1281 · SaaS Billing¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

REQ-1284 · Organization Access Control¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Organizations can define an optional email-address rule (regex pattern) for membership. When a user redeems an invite to join that organization, the server validates the authenticated user's email against the rule. If the email does not match, the join is rejected. Organizations with no rule accept any email.

Use case: Organizations need to restrict membership to users with company email addresses (e.g., @acme.com) or other domain-based rules to ensure data governance and compliance within trusted user pools.

Code: provisa/api/auth_router.py, provisa/core/schema_admin.py

Tests: tests/integration/test_redeem_invite.py

REQ-1285 · Organization Access Control¶

Status: 💡 proposed · Priority: SHOULD · Type: behavioral

Organizations may enable an "auto-join" flag with a configured default role (e.g., a low-privilege read-only role like "analyst"). When auto-join is enabled, any newly authenticated user whose email matches the org's email rule (or any user if no email rule exists) is automatically granted membership in that org with the default role — no explicit invite needed. When auto-join is disabled (the default), joining requires an explicit invite. Auto-join depends on the org email-rule feature (REQ-1284) to scope which accounts may self-join.

Use case: Auto-join reduces friction for domain-scoped organizations by eliminating the need for explicit invites when email-based membership rules already verify eligibility. Users matching the email rule gain immediate access with a safe default role, speeding time-to-first-query.

Code: provisa/api/auth_router.py, provisa/core/schema_admin.py

Tests: tests/integration/test_redeem_invite.py

REQ-1286 · Identity Resolution¶

Status: ✅ complete · Priority: MUST · Type: constraint

The default org ID has a single source of truth from the control plane's resolved org_id (the value that names the tenant schema org_). config.default_org_id is None by default and only overrides when explicitly set; if neither resolves, the system raises rather than guessing. Unresolved-identity states (server error, network failure, auth failure) must be distinguished from resolved identities with zero org memberships — the UI must not collapse failed /auth/me into "no account access".

Use case: Three divergent literals ("root" in config, "default" in control plane, "root" hardcoded in registry seed) caused active_org_id to name an org whose org_ schema was never created, leading to /auth/me 500 errors and false "no account access" reports. A single source of truth prevents org ID aliasing. Distinguishing error states from empty memberships ensures the UI routes users forward (to OnboardOrgPage for org creation/join) instead of dead-ending with a generic access-denied message.

Code: provisa/auth/wiring.py, provisa/core/models.py, provisa/core/schema_admin.py, provisa/core/control_plane.py, provisa/api/startup_seed.py, provisa/api/auth_router.py, provisa-ui/src/api/admin.ts, provisa-ui/src/context/AuthContext.tsx, provisa-ui/src/App.tsx, provisa-ui/src/i18n/locales/en/onboardGate.json

Tests: tests/unit/test_default_org_id_resolution.py, provisa-ui/src/__tests__/OnboardGate.test.tsx

3. Multi-tenancy & Organization¶

REQ-1287 · Org Invitations¶

Status: ✅ complete · Priority: SHOULD · Type: behavioral

Onboarding independently answers three questions: whether the user has a Provisa account, whether they have a pending org invitation, and whether they have org membership. Addressed invitations store the invitee's email in the org_invites table, enabling the UI to surface one-click accepts; link invites remain shareable without email.

Use case: Users with pending org invitations can directly accept them on arrival without providing an invite token. The onboarding flow correctly distinguishes between account creation, org invitation acceptance, and org membership scenarios.

Code: provisa/core/schema_admin.py, provisa/api/auth_router.py, provisa-ui/src/pages/OnboardOrgPage.tsx

Tests: tests/integration/test_redeem_invite.py

2. Authentication & Identity¶

REQ-1288 · Platform Admin Bootstrap¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Before any superadmin exists (bootstrap_superadmin enabled and no superadmin_bootstrap row), a public GET /auth/bootstrap-status endpoint returns {"unclaimed": bool}. The login page displays a first-login notice warning that whoever signs in first becomes platform administrator, then shows provider choices. The endpoint bypasses bearer-token validation in _SKIP_PATHS.

Use case: Transparent disclosure of the bootstrap-admin privilege grant prevents surprise superadmin escalation and ensures the first authenticator understands the security implications. The public endpoint enables the UI to probe bootstrap status independently of authentication state.

Code: provisa/api/auth_router.py, provisa/auth/middleware.py, provisa-ui/src/api/admin.ts, provisa-ui/src/pages/LoginPage.tsx, provisa-ui/src/i18n/locales/en/loginPage.json

Tests: tests/integration/test_bootstrap_status.py, tests/unit/test_auth_middleware.py, provisa-ui/src/__tests__/LoginPage.test.tsx

3. Multi-tenancy & Organization¶

REQ-1289 · Onboarding¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Onboarding never dead-ends: a stored credential that the deployment rejects (/auth/me returns 401/403) is discarded by OnboardGate, which returns the user to the sign-in page to authenticate or create an account. Onboarding independently answers three orthogonal questions — does the user have a Provisa account, do they have a pending org invitation, do they have org membership — each with its own forward path. Transient server failures (5xx or unreachable) retain a separate retry screen (REQ-1286) and are not treated as access decisions.

Use case: Users with stale credentials (e.g., post-password-reset, post-org-removal) must be able to re-authenticate rather than encountering a terminal "No account access" message. Platforms with an unclaimed admin slot cannot tell users "ask an administrator for an invitation," so onboarding must independently decompose the account-presence, invitation-presence, and membership-presence decisions and provide forward paths for each.

Code: provisa-ui/src/components/OnboardGate.tsx

Tests: provisa-ui/src/__tests__/OnboardGate.test.tsx

1. Access Governance & Security¶

REQ-1290 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: behavioral

Claiming the sole platform-administrator slot is an explicit act via POST /auth/claim-bootstrap, never a side effect of authentication. AuthMiddleware only reads the superadmin_bootstrap singleton; the endpoint first-writes the id=1 row (first writer wins), returns 404 when bootstrap mode is off and 401 without identity, and reads back the holder on losing claims.

Use case: Prevents stale credentials held by browsers from silently reactivating platform admin on page refresh before the user reads the REQ-1288 first-login disclosure. Explicit claim ensures the user understands the implications and selects a sign-in provider on the login page after reading the disclosure.

Code: provisa/api/auth_router.py, provisa/core/schema_admin.py

Tests: tests/integration/test_redeem_invite.py

REQ-1291 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: behavioral

AuthProvider takes an authVersion prop that is bumped by App whenever a token is stored or dropped. The provider re-runs the identity bootstrap (fetchMe + roles/domains) when authVersion changes, ensuring a just-signed-in user resolves to their real identity instead of staying at userId=null.

Use case: Without authVersion, a just-signed-in user remained at userId=null because the identity fetched at mount was never revisited. OnboardGate then read the fresh token alongside the stale (null) identity, read it as a rejected credential, deleted it, and sent the user back to sign-in — an unbreakable loop.

Code: provisa-ui/src/context/AuthContext.tsx, provisa-ui/src/App.tsx

Tests: provisa-ui/src/__tests__/OnboardGate.test.tsx

REQ-1292 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: behavioral

An unclaimed platform-admin slot outranks org onboarding in the routing gate. A token-holding user is dropped back to sign-in instead of org creation when the admin slot is still unclaimed.

Use case: A valid credential from before the admin-plane was wiped would otherwise skip sign-in entirely (where the first-login disclosure renders) and land on "create an organization" on a deployment with no administrator. This prevents accidental onboarding without understanding the implication of claiming the sole admin slot.

Code: provisa-ui/src/components/OnboardGate.tsx, provisa/auth/middleware.py

Tests: provisa-ui/src/__tests__/OnboardGate.test.tsx

REQ-1293 · Data Isolation¶

Status: ✅ complete · Priority: MUST · Type: constraint

The tenant plane is isolated by schema (org_), not row-level predicates. Admin resolvers must not additionally filter rows by org_id because rows seeded into an org schema carry org_id='root'. A failed tables query must render an error, not an empty grid.

Use case: Schema isolation is the security boundary between tenants. A row-level org_id predicate that failed for seeded rows with org_id='root' caused an org_admin of a non-root org to see vanishing domains, and a failed query rendering as an empty table read as "org has no data" instead of "query failed".

Code: provisa/api/admin/schema_query.py, provisa-ui/src/pages/TablesPage.tsx

Tests: provisa-ui/src/__tests__/TablesPageLoadError.test.tsx

REQ-1294 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: ui

After claiming the platform-admin slot the user is shown a welcome modal explaining they are the platform admin and how to invite others. The flag is read once at mount and cleared when dismissed.

Use case: Claiming the sole admin slot is an irreversible act that happens behind a provider redirect — the user clicks "Sign in with Google" and lands in the app with no statement of what they now are. This modal is that statement, giving the new admin the invite path so they know how to bring others in.

Code: provisa-ui/src/components/PlatformAdminWelcomeModal.tsx, provisa-ui/src/App.tsx

Tests: provisa-ui/src/__tests__/PlatformAdminWelcomeModal.test.tsx

REQ-1295 · Authentication¶

Status: ✅ complete · Priority: MUST · Type: constraint

The UI may only place an ASSIGNED role in the X-Provisa-Role header. When auth is enforced and the roles query fails or yields no assigned role, AuthContext reports the error and leaves the role list empty — it must never fall back to a fabricated full-capability admin role. The DEFAULT_ADMIN_ROLE fallback applies only where auth is not enforced (dev mode).

Use case: The server honors X-Provisa-Role only for roles the user is assigned (provisa/auth/middleware.py), so handing an org_admin a client-side admin role makes every subsequent request 403 "Role 'admin' is not assigned to this user" — the exact failure a self-service org creator hit when a roles query failed and the client fabricated a fallback.

Code: provisa-ui/src/context/AuthContext.tsx, provisa/auth/middleware.py

Tests: provisa-ui/src/context/__tests__/AuthContext.roles.test.tsx, tests/integration/test_org_creator_role_request.py