Snowflake Trail in Practice: Event Tables, Structured Logging, Traces, and Alerts

Every Snowflake account eventually reaches the point where "the pipeline failed overnight" turns into an hour of detective work: which task, which stored procedure, which retry, and what were the parameters? Snowflake Trail is the observability stack that answers those questions inside the platform — event tables for logs, traces and metrics; alerts and notification integrations for getting told; and the ACCOUNT_USAGE and INFORMATION_SCHEMA views that let you build a pipeline health dashboard without shipping anything to a third-party tool.

This tutorial builds a working setup end to end: an event table, instrumented Python and SQL code, trace spans, a serverless alert that emails on failure, and the queries we use on client engagements to find the slow, expensive, and flaky parts of a warehouse.

What Snowflake Trail actually is

Trail is an umbrella name for capabilities that already exist as concrete objects:

Signal	Where it lands	How you produce it
Logs	Event table (RECORD_TYPE = 'LOG')	logging in Python/Java/Scala UDFs and procedures, SYSTEM$LOG() in SQL
Trace spans	Event table (RECORD_TYPE = 'SPAN', 'SPAN_EVENT')	Auto-instrumented procedures/UDFs plus SYSTEM$ADD_EVENT() and the telemetry API
Metrics	Event table (RECORD_TYPE = 'METRIC')	Automatic for Snowpark Container Services and some workloads
Query telemetry	ACCOUNT_USAGE.QUERY_HISTORY, QUERY_ATTRIBUTION_HISTORY	Automatic
Task/pipeline telemetry	TASK_HISTORY, DYNAMIC_TABLE_REFRESH_HISTORY, COPY_HISTORY	Automatic
Alerts	Serverless alerts + notification integrations	You define them

The piece most accounts are missing is the event table. Without it, logs written by your procedures go nowhere.

Step 1: create the event table

Event tables are a special table type with a fixed schema. Create one, then attach it at account level (you can also attach a different event table per database).

USE ROLE ACCOUNTADMIN;

CREATE DATABASE IF NOT EXISTS observability;
CREATE SCHEMA IF NOT EXISTS observability.telemetry;

CREATE EVENT TABLE IF NOT EXISTS observability.telemetry.events;

ALTER ACCOUNT SET EVENT_TABLE = observability.telemetry.events;

Then decide how chatty you want to be. Levels are set at account, database, schema, or object scope, and the most specific one wins:

-- Quiet by default, verbose for the pipeline schema we are debugging
ALTER ACCOUNT SET LOG_LEVEL = 'WARN';
ALTER ACCOUNT SET TRACE_LEVEL = 'OFF';

ALTER SCHEMA analytics.load SET LOG_LEVEL = 'INFO';
ALTER SCHEMA analytics.load SET TRACE_LEVEL = 'ON_EVENT';

TRACE_LEVEL = 'ALWAYS' captures spans for every call and is the setting that most often surprises people on the bill: event table storage is ordinary table storage, and a busy account at ALWAYS can write tens of gigabytes a month. Start at ON_EVENT, which records spans only when you explicitly add events.

Set retention so the table does not grow forever:

ALTER TABLE observability.telemetry.events SET DATA_RETENTION_TIME_IN_DAYS = 7;

Most teams keep raw events for a week or two and roll up a slim daily summary table for longer history.

Step 2: emit logs from a stored procedure

Here is a load procedure instrumented the way we ship them. Note the logger name — use a stable, hierarchical name so you can filter later.

CREATE OR REPLACE PROCEDURE analytics.load.load_orders(run_date DATE)
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.11'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'main'
AS
$$
import logging
from snowflake import telemetry

logger = logging.getLogger("acme.pipeline.load_orders")

def main(session, run_date):
    logger.info(f"starting load for {run_date}")
    telemetry.set_span_attribute("run_date", str(run_date))

    try:
        rows = session.sql(f"""
            MERGE INTO analytics.orders t
            USING staging.orders_raw s ON t.order_id = s.order_id
            WHEN MATCHED THEN UPDATE SET t.status = s.status, t.amount = s.amount
            WHEN NOT MATCHED THEN INSERT (order_id, customer_id, order_date, amount, status)
              VALUES (s.order_id, s.customer_id, s.order_date, s.amount, s.status)
        """).collect()

        inserted = rows[0]["number of rows inserted"]
        updated  = rows[0]["number of rows updated"]

        telemetry.add_event("load_orders.complete",
                            {"inserted": inserted, "updated": updated})
        logger.info(f"load complete inserted={inserted} updated={updated}")
        return f"OK inserted={inserted} updated={updated}"

    except Exception as e:
        logger.error(f"load failed: {e}", exc_info=True)
        telemetry.add_event("load_orders.error", {"message": str(e)})
        raise
$$;

Three habits worth copying:

1. Log an event, not a sentence. telemetry.add_event with structured attributes is queryable; a free-text string is not.
2. Always raise after logging. Swallowing the exception makes the task succeed and the data wrong — the worst failure mode there is.
3. Set span attributes for the natural partition key (run date, tenant, source system) so you can group traces by it.

In pure SQL you have the same primitives:

CALL SYSTEM$LOG('info', 'starting nightly rebuild');
CALL SYSTEM$ADD_EVENT('rebuild.start', {'layer': 'marts'});

Step 3: query the event table

The event table's RECORD, RECORD_ATTRIBUTES, RESOURCE_ATTRIBUTES, and VALUE columns are VARIANT, so everything is a path expression. Logs first:

SELECT
  TIMESTAMP,
  RESOURCE_ATTRIBUTES:"snow.executable.name"::STRING  AS proc_name,
  RECORD:severity_text::STRING                        AS level,
  VALUE::STRING                                       AS message,
  RESOURCE_ATTRIBUTES:"snow.query.id"::STRING         AS query_id
FROM observability.telemetry.events
WHERE RECORD_TYPE = 'LOG'
  AND SCOPE:name::STRING LIKE 'acme.pipeline.%'
  AND TIMESTAMP > DATEADD('hour', -24, CURRENT_TIMESTAMP())
  AND RECORD:severity_text::STRING IN ('ERROR', 'WARN')
ORDER BY TIMESTAMP DESC;

Then the structured events, which is where the real value lives:

SELECT
  TIMESTAMP,
  RECORD:name::STRING                        AS event_name,
  RECORD_ATTRIBUTES:inserted::NUMBER         AS rows_inserted,
  RECORD_ATTRIBUTES:updated::NUMBER          AS rows_updated
FROM observability.telemetry.events
WHERE RECORD_TYPE = 'SPAN_EVENT'
  AND RECORD:name::STRING = 'load_orders.complete'
  AND TIMESTAMP > DATEADD('day', -14, CURRENT_TIMESTAMP())
ORDER BY TIMESTAMP DESC;

Now you can trend row counts per run and spot the night the source system silently sent 4% of its usual volume — the failure that no exception ever catches.

Span durations, for finding the slow step inside a long procedure:

SELECT
  RESOURCE_ATTRIBUTES:"snow.executable.name"::STRING AS proc_name,
  RECORD:name::STRING                                AS span_name,
  COUNT(*)                                           AS runs,
  ROUND(AVG(DATEDIFF('millisecond', START_TIMESTAMP, TIMESTAMP)))  AS avg_ms,
  MAX(DATEDIFF('millisecond', START_TIMESTAMP, TIMESTAMP))         AS max_ms
FROM observability.telemetry.events
WHERE RECORD_TYPE = 'SPAN'
  AND TIMESTAMP > DATEADD('day', -7, CURRENT_TIMESTAMP())
GROUP BY 1, 2
ORDER BY avg_ms DESC;

Step 4: alert on failures instead of discovering them

An event table you never look at is not observability. Wire a notification integration and a serverless alert.

CREATE OR REPLACE NOTIFICATION INTEGRATION data_team_email
  TYPE = EMAIL
  ENABLED = TRUE
  ALLOWED_RECIPIENTS = ('data-oncall@acme.com');

CREATE OR REPLACE ALERT observability.telemetry.pipeline_errors
  SCHEDULE = '10 MINUTE'
  IF (EXISTS (
    SELECT 1
    FROM observability.telemetry.events
    WHERE RECORD_TYPE = 'LOG'
      AND RECORD:severity_text::STRING = 'ERROR'
      AND SCOPE:name::STRING LIKE 'acme.pipeline.%'
      AND TIMESTAMP BETWEEN SNOWFLAKE.ALERT.LAST_SUCCESSFUL_SCHEDULED_TIME()
                        AND SNOWFLAKE.ALERT.SCHEDULED_TIME()
  ))
  THEN CALL SYSTEM$SEND_EMAIL(
    'data_team_email',
    'data-oncall@acme.com',
    'Snowflake pipeline errors detected',
    'One or more pipeline procedures logged ERROR in the last 10 minutes. Check observability.telemetry.events.'
  );

ALTER ALERT observability.telemetry.pipeline_errors RESUME;

LAST_SUCCESSFUL_SCHEDULED_TIME() is what stops the alert re-sending the same failure every ten minutes. Serverless alerts bill per execution, so a 10-minute cadence on a cheap EXISTS check is fine; a 1-minute cadence that scans a large event table is not.

For task-level failures you often want the alert on TASK_HISTORY instead, because a task that never started produces no logs at all:

SELECT NAME, STATE, ERROR_MESSAGE, SCHEDULED_TIME
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
  SCHEDULED_TIME_RANGE_START => DATEADD('hour', -24, CURRENT_TIMESTAMP())))
WHERE STATE IN ('FAILED', 'CANCELLED')
ORDER BY SCHEDULED_TIME DESC;

Missing-run detection — "the task should have fired by 03:00 and there is no row" — is the check that catches suspended tasks, and it is the one most teams never build.

Step 5: the account-level health queries

Beyond your own instrumentation, four ACCOUNT_USAGE queries cover most of what a reviewer asks for.

Queries that spilled to remote storage (the classic sign of an undersized warehouse):

SELECT QUERY_ID, WAREHOUSE_NAME, USER_NAME,
       BYTES_SPILLED_TO_REMOTE_STORAGE / POWER(1024,3) AS gb_spilled,
       TOTAL_ELAPSED_TIME / 1000 AS seconds
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE START_TIME > DATEADD('day', -7, CURRENT_TIMESTAMP())
  AND BYTES_SPILLED_TO_REMOTE_STORAGE > 0
ORDER BY gb_spilled DESC
LIMIT 50;

Credit attribution by query hash, to find the repeated query that quietly costs the most:

SELECT QUERY_PARAMETERIZED_HASH,
       ANY_VALUE(QUERY_TEXT)          AS sample_sql,
       COUNT(*)                       AS executions,
       SUM(CREDITS_ATTRIBUTED_COMPUTE) AS credits
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY
WHERE START_TIME > DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY 1
ORDER BY credits DESC
LIMIT 25;

Dynamic table refresh failures and lag:

SELECT NAME, STATE, STATE_MESSAGE, REFRESH_START_TIME, REFRESH_END_TIME
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY())
WHERE STATE != 'SUCCEEDED'
  AND REFRESH_START_TIME > DATEADD('day', -3, CURRENT_TIMESTAMP())
ORDER BY REFRESH_START_TIME DESC;

Ingestion errors from COPY_HISTORY, where FIRST_ERROR_MESSAGE usually names the offending column outright.

Remember the latency difference: INFORMATION_SCHEMA table functions are near real-time but limited in history; ACCOUNT_USAGE views go back a year but lag by roughly 45 minutes to three hours depending on the view. Alerts should read INFORMATION_SCHEMA or your event table; dashboards should read ACCOUNT_USAGE.

Step 6: keep the cost of observability honest

Observability that costs more than the pipeline is a failed project. Four controls:

• Trace level. ON_EVENT in production, ALWAYS only while debugging a specific schema, and set it back.
• Retention. Seven to fourteen days of raw events, plus a nightly rollup into a narrow summary table.
• Aggregate, don't scan. Build a daily pipeline_run_summary table (run, rows, duration, status) with a task, and point dashboards at that rather than at the raw event table.
• Measure it. Event table storage shows up in TABLE_STORAGE_METRICS; serverless alert cost shows up in SERVERLESS_ALERT_HISTORY. Check both a month after rollout.

A sensible rollout order

1. Create the event table and attach it at account level. Leave levels conservative.
2. Instrument the three pipelines that page someone most often, using structured events with row counts.
3. Add the error alert plus a missing-run alert for the critical task tree.
4. Build the daily rollup table and one Snowsight dashboard: runs, rows, duration trend, failures.
5. Only then widen instrumentation to the rest of the account.

Teams that do step 4 before step 2 end up with a dashboard nobody trusts. Structured events first, presentation second.

Where this fits

Native Snowflake observability will not replace a full APM tool if your pipelines span half a dozen systems — and Snowflake supports OpenTelemetry export to Datadog, Grafana, and similar if you want signals in one place. But for the work that happens inside Snowflake, Trail removes the excuse for a pipeline whose only monitoring is a user complaining.

If you would like help instrumenting an existing Snowflake estate — event tables, alerting, a run-history model, and the cost guardrails to keep it cheap — the PowderInsights team does exactly this kind of work. Get in touch with your current pipeline stack and where you are losing time, and we will suggest a starting point.