10 stories tagged pipelines.
Snowflake Postgres data mirroring continuously replicates Postgres tables into Snowflake using Iceberg with 30-second lag. It replaces external CDC connectors with serverless compute and open formats.
Snowflake DCM Projects GA brings declarative infrastructure-as-code to Snowflake with DEFINE statements and plan-then-deploy. Eight capabilities remain in Preview, including masking policies and streams.
Dynamic Iceberg table replication now keeps Snowflake managed Iceberg pipelines in DR plans, but refresh and transfer costs move to the target account.
Snowflake Dynamic Tables now claim up to 2.8x faster refresh on Gen2 warehouses, but the cost win depends on target lag and change volume.
Snowflake Iceberg ADLS support is the GA path for Azure teams to read and write externally managed Iceberg tables without moving storage.
ICEBERG_MERGE_ON_READ_BEHAVIOR is Snowflake's GA switch for Iceberg DML mode. AUTO sends 3 of 4 table cases to merge-on-read.
The Data Productivity Cloud is Matillion's cloud-native platform for building, running, and orchestrating data pipelines with low-code, code, and AI. Here is how the pieces fit and where the cost goes.
Snowflake Iceberg tables store data in open Apache Iceberg format in your own cloud storage, so Snowflake bills zero storage and other engines can read it. The catch: only Snowflake-managed catalog tables get full platform support, external-catalog tables lose clustering, cloning, and replication.
Openflow is Snowflake's managed data integration service, built on Apache NiFi, that moves structured and unstructured data from any source into Snowflake. It runs in two modes: inside Snowflake on container services, or in your own cloud VPC, and each bills and isolates differently.
Dynamic Tables let you declare a SELECT and a target lag and let Snowflake refresh it; Streams and Tasks give you imperative control. Dynamic Tables bottom out at a 60-second minimum lag, so sub-minute freshness still means Snowpipe Streaming or Tasks.