+1 (415) 997-4269

Snowflake Disaster Recovery: Replication Groups, Failover Groups, and a Failover Runbook You Can Actually Test

Most Snowflake programmes get governance, cost, and pipelines right long before anyone asks the awkward question: what happens if our Snowflake region goes away for six hours? Time Travel and Fail-safe do not answer that question — they protect you from bad DML, not from a regional outage. The answer is account replication, and the parts you need are replication groups, failover groups, and client redirect.

This tutorial walks through the whole picture, then gives you a runbook you can rehearse in a maintenance window instead of discovering during an incident.

What Time Travel, Fail-safe and cloning do not do

FeatureProtects againstDoes not protect against
Time TravelBad UPDATE/DELETE/DROP, up to 90 days on Enterprise+Region or account unavailability
Fail-safeSnowflake-side data loss (Snowflake ops only, 7 days)Anything you can self-serve
Zero-copy cloneFast environment copies inside one accountLoss of the account or region
Replication / failover groupsRegional and cloud-provider outages, account lossLogical corruption you replicate onward

That last row matters: replication faithfully copies your mistakes too. DR is a complement to Time Travel, not a substitute.

Replication group vs failover group

Both are account-level objects that replicate a set of objects on a schedule, with point-in-time consistency across everything in the group.

  • A replication group gives you a read-only secondary. Good for reporting offload, data residency copies, or migration dry runs. You cannot promote it.
  • A failover group is a replication group that can be promoted to primary. This is the DR object. Requires Business Critical edition (or higher).

Use groups, not the older per-database ALTER DATABASE ... ENABLE REPLICATION approach. Groups replicate account objects — roles, grants, users, warehouses, resource monitors, network policies, integrations — which is exactly what people forget until failover day, when the data is there but nobody can log in.

What replicates, and what you still have to handle

Replicates inside a failover group (object types you select):

  • DATABASES (tables, views, stages metadata, streams, tasks, pipes, policies, tags)
  • ROLES, USERS, WAREHOUSES, RESOURCE MONITORS, NETWORK POLICIES, ACCOUNT PARAMETERS, INTEGRATIONS

Still needs planning:

  • External volumes and Iceberg tables — the storage lives in a specific cloud region; a cross-region secondary reads across regions unless you replicate the bucket too.
  • External stages and storage integrations — the integration object replicates, but the IAM trust relationship and the bucket may not exist in the DR region.
  • Pipes and Snowpipe Streaming channels — replicated pipes are suspended on the secondary; ingestion must be re-pointed after promotion.
  • Tasks — replicated in a suspended state; you resume them post-promotion (deliberately, so you do not double-run pipelines).
  • Third-party tools — dbt, Fivetran, Airflow, BI tools: each has its own Snowflake connection to redirect.
  • Cortex/AI objects, some connectors, and newer previews — check current support before you assume; treat unsupported objects as rebuild-from-code items.

The honest conclusion: replication covers the platform, your DR plan covers the perimeter. Everything on the second list should exist as code in Git (see our CI/CD tutorial) so the DR account can be reconciled with a pipeline run.

Setting it up

1. Enable replication for both accounts

Run as ORGADMIN in the organisation:

USE ROLE ORGADMIN;
SHOW ORGANIZATION ACCOUNTS;

SELECT SYSTEM$GLOBAL_ACCOUNT_SET_PARAMETER('ACME.PROD_EAST',
  'ENABLE_ACCOUNT_DATABASE_REPLICATION', 'true');
SELECT SYSTEM$GLOBAL_ACCOUNT_SET_PARAMETER('ACME.PROD_WEST',
  'ENABLE_ACCOUNT_DATABASE_REPLICATION', 'true');

2. Create the failover group on the primary

USE ROLE ACCOUNTADMIN;

CREATE FAILOVER GROUP fg_prod
  OBJECT_TYPES = (DATABASES, ROLES, USERS, WAREHOUSES, RESOURCE MONITORS,
                  NETWORK POLICIES, ACCOUNT PARAMETERS, INTEGRATIONS)
  ALLOWED_DATABASES = (RAW, CORE, MARTS, GOVERNANCE)
  ALLOWED_ACCOUNTS  = (ACME.PROD_WEST)
  REPLICATION_SCHEDULE = '10 MINUTE';

Notes that save time later:

  • ALLOWED_DATABASES is explicit on purpose. Leaving out sandbox and scratch databases is the cheapest DR optimisation available.
  • REPLICATION_SCHEDULE accepts '<n> MINUTE' or a cron expression. This is your RPO dial.
  • A database can belong to only one failover group. Plan groups around consistency boundaries: everything in a group is replicated to the same point in time, so keep the pipelines that must agree with each other together.

3. Create the secondary on the DR account

-- In ACME.PROD_WEST
USE ROLE ACCOUNTADMIN;

CREATE FAILOVER GROUP fg_prod
  AS REPLICA OF ACME.PROD_EAST.fg_prod;

ALTER FAILOVER GROUP fg_prod REFRESH;   -- first (large) refresh

The first refresh copies everything; later refreshes are incremental at the micro-partition level.

4. Client redirect so applications do not hardcode a region

-- On the primary
CREATE CONNECTION prod_conn;

-- On the secondary
CREATE CONNECTION prod_conn AS REPLICA OF ACME.prod_conn;

Applications then connect to the organisation-level URL, for example acme-prod_conn.snowflakecomputing.com, and a single ALTER CONNECTION ... PRIMARY moves every client. If your tools point at abc12345.us-east-1.snowflakecomputing.com, your RTO is however long it takes to edit dozens of config files under pressure. Fix the URLs before you need them.

Monitoring lag: are you actually meeting your RPO?

SELECT replication_group_name,
       phase_name,
       start_time,
       end_time,
       DATEDIFF('second', start_time, end_time) AS seconds,
       bytes_transferred / POWER(1024,3)        AS gb,
       credits_used
FROM SNOWFLAKE.ACCOUNT_USAGE.REPLICATION_GROUP_REFRESH_HISTORY
WHERE start_time > DATEADD('day', -7, CURRENT_TIMESTAMP())
ORDER BY start_time DESC;

-- Current state, including last refresh completion
SELECT SYSTEM$SHOW_REPLICATION_GROUP_REFRESH_PROGRESS('fg_prod');

Wire an alert on it (Snowflake alerts and event tables, covered in our Snowflake Trail tutorial):

CREATE OR REPLACE ALERT alert_replication_lag
  WAREHOUSE = ops_wh
  SCHEDULE  = '30 MINUTE'
  IF (EXISTS (
        SELECT 1
        FROM SNOWFLAKE.ACCOUNT_USAGE.REPLICATION_GROUP_REFRESH_HISTORY
        HAVING MAX(end_time) < DATEADD('hour', -1, CURRENT_TIMESTAMP())
      ))
  THEN CALL SYSTEM$SEND_EMAIL('ops_email_int', 'oncall@acme.com',
        'Snowflake replication lag > 1h', 'Check fg_prod refresh history.');

RPO, RTO and cost — pick two, then price them

  • RPO is bounded by your refresh interval plus refresh duration. A 10-minute schedule on a busy warehouse-scale dataset does not give a 10-minute RPO if each refresh takes 25 minutes. Measure, do not assume.
  • RTO is dominated by human steps: deciding to fail over, promoting, resuming tasks, re-pointing ingestion, validating. Promotion itself is seconds.
  • Cost has three parts: compute credits for the refresh operation, cross-region/cross-cloud egress charged by the cloud provider, and duplicate storage in the DR region. Frequent refreshes of high-churn tables are the usual bill surprise.

A pragmatic pattern for cost control: two groups. fg_core (customer-facing marts, 10-minute schedule) and fg_secondary (history, sandboxes, archive, 24-hour schedule). Tiering DR by business criticality routinely cuts replication spend by more than half.

The failover runbook

Rehearse this quarterly. An untested DR plan is a slide, not a capability.

Pre-flight (before the drill)

  1. Confirm the last refresh completed and note the lag.
  2. Snapshot expected row counts / checksums for a handful of critical tables.
  3. Confirm the DR warehouses, roles and network policies are present on the secondary.

Failover

-- On the DR account (ACME.PROD_WEST)
USE ROLE ACCOUNTADMIN;

ALTER FAILOVER GROUP fg_prod PRIMARY;      -- promote
ALTER CONNECTION prod_conn PRIMARY;        -- redirect clients

Post-promotion checklist

  1. Validate row counts/checksums against the pre-flight snapshot.
  2. Resume tasks deliberately: SELECT SYSTEM$TASK_DEPENDENTS_ENABLE('core.pipelines.root_task');
  3. Re-point ingestion — Snowpipe/Snowpipe Streaming, Kafka connectors, Openflow — and confirm the DR-region storage integrations and buckets work.
  4. Re-point orchestration and BI: dbt targets, Airflow connections, semantic layers, dashboards.
  5. Verify a real end-to-end user journey, not just SELECT 1.

Failback

  1. The old primary becomes a secondary automatically once promoted elsewhere; refresh it until lag is small.
  2. During a quiet window, promote it back and redirect the connection.
  3. Write down actual elapsed time per step. That number — not the marketing RTO — is what you tell your auditors and your board.

Common mistakes we see on client accounts

  1. Replicating databases but not account objects. The data arrives; the roles and grants do not. Nobody can query anything.
  2. Hardcoded account URLs. Client redirect exists precisely so failover is one statement.
  3. Never testing. Replication that has never been promoted is an assumption.
  4. Replicating everything. Sandboxes and 10-year raw history at a 10-minute cadence, then surprise at the egress line item.
  5. Treating DR as a substitute for backups. Logical corruption replicates. Keep Time Travel windows and, for critical marts, periodic clones or exports.
  6. Forgetting the pipeline tail. Ingestion and orchestration live outside Snowflake and need their own DR story.

Where to start if you have nothing today

Even on Enterprise edition (no failover groups) you can create a replication group to a second account and get a read-only copy plus a rehearsed refresh process — a big improvement over nothing, and the same object model to build on if you later move to Business Critical. Then: define RPO/RTO per data domain with the business, tier your groups accordingly, put the perimeter (integrations, stages, tasks, tooling) in Git, and book the first drill.

If you would like an experienced Snowflake architect to review your replication topology, price the RPO/RTO trade-offs, or run the first failover drill with your team, get in touch — our consultants do this work on production accounts every week.