PowderInsights | Snowpipe Streaming vs Openflow vs Kafka Connector: Choosing a Real-Time Ingestion Path

"Real-time into Snowflake" used to mean one thing: the Kafka connector writing files to a stage and Snowpipe loading them a minute or two later. In 2026 there are three credible paths — Snowpipe Streaming, Openflow, and the Kafka connector — and they overlap enough that teams routinely pick the wrong one. This guide lays out the decision, then builds a streaming orders pipeline end to end.

The three paths

Snowpipe Streaming

A row-level ingestion API: your application (or a connector) opens a channel and writes rows directly, with no files and no stage. Snowflake's high-performance architecture for Snowpipe Streaming (new Python/Java/Node SDKs, a REST API, and a PIPE object that defines the target and optional transformation) now sits alongside the classic Java SDK. Latency is seconds; billing is per GB ingested on the new architecture, rather than per-file or per-client-hour.

Best for: application events, IoT, CDC streams you control the producer for, and any case where you want the lowest latency and are willing to run (or use) a client that speaks the API.

Openflow

Snowflake's managed integration service, built on Apache NiFi and GA since June 2025. You assemble flows from connectors — database CDC (SQL Server, Oracle, PostgreSQL, MySQL), Kafka, SaaS sources, object storage, unstructured documents — and Openflow runs them, either in Snowflake's managed runtime (Snowpark Container Services) or in your own VPC. Under the hood it writes to Snowflake with Snowpipe Streaming, so latency is comparable; what you are buying is the connectors and the operations.

Best for: CDC from databases you do not want to instrument yourself, multi-source ingestion where a visual, managed pipeline beats custom code, and teams that want ingestion to be a platform feature rather than an application.

Kafka connector

The Snowflake Connector for Kafka (Kafka Connect sink) has two modes: the original file-and-Snowpipe mode and Snowpipe Streaming mode (snowflake.ingestion.method=SNOWPIPE_STREAMING), which is the one to use now. It consumes topics and writes them to tables, with schema detection and evolution into typed columns.

Best for: you already run Kafka (or Confluent, Redpanda, MSK) and Kafka Connect, and the data is already on topics. Zero application changes.

Decision matrix

	Snowpipe Streaming (direct)	Openflow	Kafka connector (streaming mode)
Latency	Seconds	Seconds to tens of seconds	Seconds to tens of seconds
Source types	Anything your code can read	Databases (CDC), Kafka, SaaS, files, documents	Kafka topics only
You operate	Your producer application	Flows in Snowsight (managed) or a runtime in your VPC	Kafka Connect cluster
Transformation on ingest	Via the `PIPE` definition (new architecture)	Rich, in the flow	Minimal (schema evolution only)
Cost model	Per GB ingested (new architecture)	Openflow runtime credits plus ingestion	Your Connect infra plus ingestion
Schema evolution	Manual or via PIPE	Supported by connectors	Automatic
Fit	Custom apps, IoT, lowest latency	CDC, multi-source, managed	Existing Kafka estates

Rule of thumb: Kafka already? Use the connector. Database CDC, or many sources? Openflow. Your own application producing events? Snowpipe Streaming directly.

High-performance Snowpipe Streaming architecture

The newer architecture separates the definition of ingestion from the client. You create a PIPE that names the target table and an optional COPY-style transformation, then clients open channels against the pipe:

CREATE OR REPLACE TABLE raw.orders_stream (
  order_id      NUMBER,
  customer_id   NUMBER,
  status        STRING,
  amount        NUMBER(12,2),
  event_ts      TIMESTAMP_NTZ,
  _ingested_at  TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

CREATE OR REPLACE PIPE raw.orders_stream_pipe
AS COPY INTO raw.orders_stream (order_id, customer_id, status, amount, event_ts)
   FROM (SELECT $1:order_id, $1:customer_id, $1:status, $1:amount, $1:event_ts
         FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING')));

A producer in Python, using the snowpipe-streaming SDK (pip install snowpipe-streaming, Python 3.9+):

from snowflake.ingest.streaming import StreamingIngestClient

client = StreamingIngestClient(
    client_name="orders-producer",
    db_name="RAW", schema_name="RAW", pipe_name="ORDERS_STREAM_PIPE",
    profile_json="profile.json",   # account url, user, private key (JWT auth)
)
channel, status = client.open_channel("orders-channel-01")
resume_from = status.last_committed_offset_token   # None on first open

for event in order_events(after=resume_from):      # your event source
    channel.append_row(
        {
            "order_id": event.id,
            "customer_id": event.customer_id,
            "status": event.status,
            "amount": str(event.amount),
            "event_ts": event.ts.isoformat(),
        },
        offset_token=str(event.sequence),
    )

channel.wait_for_flush()
channel.close()

Each channel carries an offset token you set on every batch; on restart you read the last committed token back from Snowflake and resume from there, which is how exactly-once works without a dedupe step. Use one channel per producer partition (one per Kafka partition, one per device shard) and keep them long-lived — opening channels is the expensive part. The SDK also offers append_rows for batches (with start/end offset tokens) and wait_for_commit to block until Snowflake has committed a given token; the same operations are available over the REST API for languages without an SDK.

CDC via Openflow

For database sources, skip the producer entirely:

In Snowsight, create an Openflow deployment and runtime (managed, or in your VPC if the database is not reachable from Snowflake).
Add the connector for your source (for example, SQL Server CDC or PostgreSQL logical replication), supply credentials through a Snowflake secret, and choose the tables.
Set the destination database/schema. The connector creates the target tables, writes an initial snapshot, then streams inserts, updates, and deletes continuously, typically into a journal table plus a merged current-state table.

The result lands in RAW the same way the Snowpipe Streaming example does, so downstream design is identical.

Example: streaming orders into a Dynamic Table

Whatever the ingestion path, the consumption layer should be declarative. Given RAW.ORDERS_STREAM receiving events (multiple per order as status changes), a Dynamic Table gives you the current state with a one-minute lag:

CREATE OR REPLACE DYNAMIC TABLE analytics.orders_current
  TARGET_LAG = '1 minute'
  WAREHOUSE = streaming_wh
  REFRESH_MODE = INCREMENTAL
AS
SELECT order_id, customer_id, status, amount, event_ts
FROM raw.orders_stream
QUALIFY ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY event_ts DESC) = 1;

-- and a rolling metric on top, inheriting the lag
CREATE OR REPLACE DYNAMIC TABLE analytics.orders_last_15m
  TARGET_LAG = DOWNSTREAM
  WAREHOUSE = streaming_wh
AS
SELECT DATE_TRUNC('minute', event_ts) AS minute, COUNT(*) AS orders, SUM(amount) AS revenue
FROM raw.orders_stream
WHERE event_ts >= DATEADD(minute, -15, CURRENT_TIMESTAMP())
GROUP BY 1;

Note the second table uses CURRENT_TIMESTAMP(), which is non-deterministic and forces a full refresh; for a small 15-minute window that is fine and cheap, but it is the kind of thing to know rather than discover. The first table is fully incremental.

Operational notes

Latency is end-to-end. Snowpipe Streaming gets rows queryable in seconds, but a Dynamic Table with a one-minute lag and a dashboard that refreshes every five makes the user-visible latency five minutes. Tune the slowest hop.
Monitor ingestion with SNOWPIPE_STREAMING_CLIENT_HISTORY and SNOWPIPE_STREAMING_FILE_MIGRATION_HISTORY (classic architecture), and the PIPE_USAGE_HISTORY / channel status views for the new architecture; Openflow exposes flow metrics in Snowsight.
Schema changes are the operational risk in every path. Land into a stable raw table, or enable schema evolution deliberately and watch for it; do not let an upstream field rename break the Dynamic Table DAG silently.
Cost: streaming ingestion is cheap per GB; the warehouse refreshing Dynamic Tables every minute is where the credits go. Size it X-Small, let it suspend, and measure.

PowderInsights designs and builds real-time ingestion on Snowflake — choosing the path, building the producer or flows, and the Dynamic Table layer on top. Contact us to talk about your use case.