PowderInsights | Unstructured Documents in Snowflake: PARSE_DOCUMENT, Document AI, and Cortex Search

Most enterprises sit on a pile of data that never reaches the warehouse: scanned invoices, supplier contracts, insurance claims, inspection reports, PDFs from a vendor portal. Historically that meant a separate OCR vendor, a Python service somewhere, and a brittle hand-off back into Snowflake.

Snowflake now does the whole pipeline in-platform: stage the files, parse or extract with Cortex, land structured rows, then index the text for retrieval. This tutorial builds that pipeline end to end — stage, PARSE_DOCUMENT, a Document AI extraction model, incremental processing with a stream and task, and a Cortex Search service on top so people (and agents) can ask questions of the documents.

The three extraction paths

Choose before you build; they solve different problems.

Path	Use it for	Output
`AI_PARSE_DOCUMENT` / `PARSE_DOCUMENT`	Turning any PDF, DOCX, PPTX or image into text or layout-preserving markdown	One text/markdown blob per document
Document AI (`<model>!PREDICT`)	Pulling specific named fields out of a recurring document type — invoice number, total, effective date	Typed JSON with per-field confidence scores
Cortex AISQL (`AI_COMPLETE`, `AI_EXTRACT`, `AI_CLASSIFY`)	Ad-hoc extraction, classification or summarisation over text you already have	Whatever you prompt for

A realistic production pipeline uses all three: parse to text, extract known fields with a trained Document AI model, and use AISQL for the long tail.

1. Stage the documents

Documents must be in an internal stage or an external stage with directory tables and, for Document AI, server-side encryption.

CREATE OR REPLACE STAGE docs_stage
  DIRECTORY = (ENABLE = TRUE)
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

-- or an external stage over your own bucket
CREATE OR REPLACE STAGE docs_ext
  URL = 's3://acme-documents/invoices/'
  STORAGE_INTEGRATION = s3_int
  DIRECTORY = (ENABLE = TRUE, AUTO_REFRESH = TRUE)
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

ALTER STAGE docs_stage REFRESH;
SELECT relative_path, size, last_modified FROM DIRECTORY(@docs_stage) ORDER BY last_modified DESC LIMIT 10;

The directory table is what makes incremental processing possible later — it is a real table you can put a stream on.

2. Parse to text or markdown

SELECT
  relative_path,
  AI_PARSE_DOCUMENT(
    TO_FILE('@docs_stage', relative_path),
    {'mode': 'LAYOUT'}
  ) AS parsed
FROM DIRECTORY(@docs_stage)
LIMIT 5;

mode is the only knob that matters most of the time:

OCR — fastest and cheapest, returns plain text. Good for search indexing and summarisation.
LAYOUT — returns markdown that preserves headings, reading order, and tables. Use it whenever the document has tabular content or when the text is going to be chunked for RAG; heading structure gives you natural chunk boundaries.

The function returns an OBJECT; the text lives under content, with page metadata alongside.

CREATE OR REPLACE TABLE doc_text AS
SELECT
  relative_path,
  last_modified,
  AI_PARSE_DOCUMENT(TO_FILE('@docs_stage', relative_path), {'mode': 'LAYOUT'}):content::STRING AS body_md
FROM DIRECTORY(@docs_stage);

Watch the limits: there are per-file page and size caps, and parsing is a serverless (AI-credit) cost per page. Parse once, store the result, never re-parse in a view.

3. Document AI for named fields

Parsing gives you text. When you need the invoice total as a NUMBER, train a Document AI model build in Snowsight:

Create a Document AI model build and upload 20+ representative documents.
Define the fields as natural-language questions — "What is the invoice number?", "What is the total amount due?", "What is the payment due date?".
Review the model's answers on each sample and correct them; that review is the training data.
Train, then publish. Each publish creates a new version.

The published build is a schema-level object with a !PREDICT method:

SELECT
  relative_path,
  ACME_DB.DOCAI.INVOICE_MODEL!PREDICT(
    GET_PRESIGNED_URL(@docs_stage, relative_path), 1     -- 1 = model version
  ) AS prediction
FROM DIRECTORY(@docs_stage)
WHERE relative_path ILIKE '%.pdf';

The prediction is JSON: each field is an array of {value, score} objects. Flatten it and — this is the part teams skip — route on confidence:

CREATE OR REPLACE TABLE invoice_extract AS
WITH raw AS (
  SELECT relative_path,
         ACME_DB.DOCAI.INVOICE_MODEL!PREDICT(GET_PRESIGNED_URL(@docs_stage, relative_path), 1) AS p
  FROM DIRECTORY(@docs_stage)
)
SELECT
  relative_path,
  p:invoice_number[0]:value::STRING              AS invoice_number,
  p:invoice_number[0]:score::FLOAT               AS invoice_number_score,
  TRY_TO_DECIMAL(p:total_due[0]:value::STRING, 12, 2) AS total_due,
  p:total_due[0]:score::FLOAT                    AS total_due_score,
  TRY_TO_DATE(p:due_date[0]:value::STRING)       AS due_date,
  LEAST(p:invoice_number[0]:score::FLOAT,
        p:total_due[0]:score::FLOAT)             AS min_score,
  p                                              AS raw_prediction
FROM raw;

-- Anything below the threshold goes to a human queue, not to finance
CREATE OR REPLACE VIEW invoice_review_queue AS
SELECT * FROM invoice_extract WHERE min_score < 0.80 OR total_due IS NULL;

Pick the threshold from your own labelled sample, not from a blog post. Measure precision at 0.7, 0.8 and 0.9 on a hold-out set of 100 documents and choose the point where the review queue is small enough for the team that owns it. Keep raw_prediction — when someone disputes a number six months later, you need the original JSON and the model version.

4. Process incrementally with a stream and task

Re-running extraction over the whole stage every night is the most expensive mistake in this pipeline. Put a stream on the directory table:

CREATE OR REPLACE STREAM docs_stream ON STAGE docs_stage;

CREATE OR REPLACE TASK process_new_docs
  WAREHOUSE = etl_wh
  SCHEDULE = '10 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('docs_stream')
AS
INSERT INTO invoice_extract (relative_path, invoice_number, total_due, due_date, raw_prediction, loaded_at)
SELECT
  s.relative_path,
  p:invoice_number[0]:value::STRING,
  TRY_TO_DECIMAL(p:total_due[0]:value::STRING, 12, 2),
  TRY_TO_DATE(p:due_date[0]:value::STRING),
  p,
  CURRENT_TIMESTAMP()
FROM (
  SELECT relative_path,
         ACME_DB.DOCAI.INVOICE_MODEL!PREDICT(GET_PRESIGNED_URL(@docs_stage, relative_path), 1) AS p
  FROM docs_stream
  WHERE METADATA$ACTION = 'INSERT'
) s;

ALTER TASK process_new_docs RESUME;

Remember to ALTER STAGE docs_stage REFRESH (or rely on AUTO_REFRESH on an external stage) so the directory table — and therefore the stream — sees new files.

5. Make the text searchable with Cortex Search

Structured fields answer "what is the total on invoice 4471". They do not answer "which contracts have a 90-day termination clause". For that, chunk the parsed markdown and index it.

CREATE OR REPLACE TABLE doc_chunks AS
SELECT
  d.relative_path,
  c.index                                   AS chunk_id,
  c.value::STRING                           AS chunk_text
FROM doc_text d,
LATERAL FLATTEN(
  SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(d.body_md, 'markdown', 1800, 300)
) c;

CREATE OR REPLACE CORTEX SEARCH SERVICE contract_search
  ON chunk_text
  ATTRIBUTES relative_path
  WAREHOUSE = search_wh
  TARGET_LAG = '1 hour'
AS
  SELECT chunk_text, relative_path, chunk_id FROM doc_chunks;

Chunk on markdown structure with an overlap of roughly 15% — 1800/300 is a reasonable default for contracts and reports. Query it from SQL, or wire the service into a Cortex Agent so an assistant can cite the source document:

SELECT PARSE_JSON(
  SNOWFLAKE.CORTEX.SEARCH_PREVIEW(
    'ACME_DB.PUBLIC.CONTRACT_SEARCH',
    '{"query": "termination for convenience notice period", "columns": ["chunk_text","relative_path"], "limit": 5}'
  )
):results AS hits;

Because the search service is built on a normal Snowflake table, row access policies and masking on the underlying data still apply — governance does not fork just because the content is unstructured.

Governance and cost notes

PII in documents is still PII. Classify the extracted columns and apply masking policies to them exactly as you would for structured intake. Consider an AI_CLASSIFY step that tags documents containing personal data before anything is indexed.
Cortex functions bill serverless AI credits per page or per token. Put a SNOWFLAKE.CORE.BUDGET on the schema and monitor CORTEX_FUNCTIONS_USAGE_HISTORY and DOCUMENT_AI_USAGE_HISTORY in ACCOUNT_USAGE from day one.
Version everything. Store the Document AI model version and the parse mode next to every extracted row so results are reproducible and re-processing is targeted.
Region and model availability vary. Check current documentation for which Cortex functions and cross-region inference settings apply to your account before designing around a specific model.

A sensible rollout

Start with one document type and one downstream consumer. Parse to text, ship search first (it is cheap and immediately useful), then add a trained Document AI model for the two or three fields that actually feed a business process, with a confidence threshold and a review queue from day one. Expand to the next document type only after the review queue has been boring for a month.

PowderInsights builds document-processing and RAG pipelines natively in Snowflake — Document AI model design, chunking and Cortex Search architecture, cost controls, and the governance around them. Get in touch with your document type and volumes and we will sketch the pipeline.