Most data teams have already made two decisions, even if they haven't written them down yet. The first is that Apache Iceberg will be the table format their analytical data lives in. The second is that AI agents will be querying that data, not just dashboards and analysts. The Apache Iceberg lakehouse and agentic analytics aren't separate initiatives. They're two halves of the same architecture, and the teams that treat them that way will get to trusted AI years ahead of the teams that don't.
Here's the problem. The path between "we run a warehouse and some databases" and "agents answer business questions against governed Iceberg tables" is full of blockers. Migration risk. Table maintenance. Semantic context for AI. Mountains of unstructured documents. Most vendors solve one of these and leave you to stitch together the rest from three or four other products.
Dremio is built to take you through all four. Its federated query engine lets you start before you migrate anything. Its autonomous management runs the Iceberg lakehouse for you. Its AI Semantic Layer, built-in AI Agent, MCP server, and CLI give agents governed access with real business meaning. And its AI Functions turn PDFs sitting in object storage into Iceberg tables with a single SQL statement.
This post walks through why the Iceberg lakehouse and agentic analytics matter, what blocks teams from getting there, and how Dremio removes each blocker in order.
Problem 2: Managing the Lakehouse So It Doesn't Manage You
Migration gets you to Iceberg. Staying fast on Iceberg is a different job, and historically it's been a thankless one. Tables fragment into small files as writes accumulate. Partition layouts drift away from query patterns. Snapshots and orphan files inflate storage. And acceleration turns into a part-time career: deciding which materialized views to build, scheduling their refreshes, rewriting queries to hit them, and tearing them down when workloads shift.
Dremio's answer is to make the lakehouse autonomous. The platform watches activity through its Active Metadata system, which continuously analyzes query patterns, data relationships, and usage trends, and then it acts on what it learns without waiting for a human.
On the storage side, Dremio runs Automated Table Optimization for Iceberg tables in its Open Catalog: compaction to merge small files into well-sized ones, clustering to physically reorganize data layouts around real access patterns, and vacuum to expire old snapshots and remove orphan files. These run as background maintenance jobs. You don't size them, and you don't get paged when a streaming table quietly accumulates 40,000 tiny files, because Dremio already merged them.
On the acceleration side, the Reflections you used during migration get a serious upgrade once your data is in Iceberg:
Autonomous Reflections remove the design work entirely. Dremio analyzes your query workload over a rolling seven-day window, figures out which materializations would help, then creates, refreshes, and drops Reflections on its own. It targets queries that take at least a second and skips ones already served by cache, so it spends compute exactly where users feel pain. No one on your team decides what to materialize anymore. The platform does, and it revises that decision as workloads change perfect for a world where agent patterns are changing faster than manual acceleration can keep up with.
Live Reflections kill the staleness problem. Because Iceberg exposes table changes through snapshots, Dremio detects when an anchor table changes (polling as often as every 10 seconds) and triggers a refresh immediately. Scheduled refreshes against unchanged data get recognized as redundant and skipped, so you stop burning compute to rebuild things that didn't change.
Incremental Refresh makes those updates cheap. Dremio reads Iceberg's snapshot metadata to identify exactly which records were added, modified, or deleted since the last refresh, and processes only that delta instead of rebuilding the whole materialization. On a 10-billion-row table where last night's load touched 0.2% of rows, that's the difference between minutes and hours of compute.
Then there's the caching stack underneath. The query plan cache stores the physical plan of executed queries, so repeated queries (the lifeblood of BI dashboards) skip compilation and go straight to execution. The results cache goes further: deterministic queries on unchanged Iceberg data return prior results instantly, spooled as Arrow files to distributed storage and shared across coordinators and clients, whether the query arrives over the console, JDBC, ODBC, REST, or Arrow Flight. And the Columnar Cloud Cache (C3) keeps frequently accessed columnar data on local NVMe at the executor nodes, cutting up to 90% of object storage I/O costs and turning cloud-storage latency into local-disk speed.
Stack it up and the operational picture changes shape. Compaction, clustering, vacuum, materialization design, refresh scheduling, and cache management all move from your team's backlog to the platform's job description. Your engineers stop juggling materialized views and start shipping data products. Dremio's claim of 10x data engineering productivity is aggressive, but the mechanism behind it is concrete: the platform absorbed an entire category of recurring work.
Problem 4: Unstructured Data Without a Separate OCR Pipeline
Somewhere in your object storage right now there's a folder of PDFs that matters more than half your tables. Invoices. Contracts. Inspection reports. Resumes. Support transcripts. Industry estimates put 80 to 90% of enterprise data in unstructured form, and almost none of it participates in analytics, because getting it into rows traditionally requires a separate extraction stack: OCR services, document parsers, orchestration, error handling, and a pipeline team to keep it all running.
Dremio's answer is to make documents queryable with SQL. The platform embeds LLM calls directly into the engine as AI Functions: AI_GENERATE, AI_CLASSIFY, AI_COMPLETE, and the table function LIST_FILES. No Python service, no external orchestration, no data leaving your governed environment.
LIST_FILES is the bridge. Point it at a directory in connected storage (S3, ADLS, GCS) and it returns the files as rows, each with metadata plus a file struct you can hand to the other functions. It handles PDFs, images, Word documents, text files, and scanned documents through multimodal vision models. AI_GENERATE then extracts whatever you ask for, and its WITH SCHEMA clause forces the LLM to return typed, named fields rather than a blob of prose.
Put them together and an extraction pipeline collapses into one statement:
CREATE TABLE gold.invoices AS
SELECT
file['path'] AS source_file,
invoice_data.vendor_name,
invoice_data.invoice_number,
invoice_data.total_amount
FROM (
SELECT
file,
AI_GENERATE(
ROW('Extract vendor name, invoice number, and total amount from this invoice.', file)
WITH SCHEMA ROW(
vendor_name VARCHAR,
invoice_number VARCHAR,
total_amount DECIMAL(12,2)
)
) AS invoice_data
FROM TABLE(LIST_FILES('@company_s3/invoices/2025'))
WHERE file['path'] LIKE '%.pdf'
);
Read what that statement actually does. It scans a folder of invoice PDFs in S3, extracts three typed fields from each document, and materializes the results as a governed Apache Iceberg table. The documents become rows. The rows become part of the semantic layer. The semantic layer feeds your agents and dashboards. A workload that used to mean standing up a document-processing service now ships in a SQL Runner tab before lunch.
The other functions round out the toolkit. AI_CLASSIFY constrains the model to one value from a list you supply, which makes it reliable for sentiment labeling, document triage, and routing. AI_COMPLETE handles free-form generation like summaries and descriptions. Model providers are pluggable (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, or Dremio's hosted model), and neither Dremio nor the providers train on your data.
A few production habits make this scale well. Materialize extraction results with CTAS so you pay for each LLM call once instead of on every dashboard refresh. Layer Reflections on the output tables so downstream queries run at interactive speed with zero additional LLM cost. And use workload management rules to route AI-function queries to a dedicated engine so a big extraction job never slows your BI traffic. All three are configuration, not architecture.
.
If the test holds up, the rollout sequence writes itself. Curate wikis and labels on your ten most-asked-about datasets first, because curating semantic context is the most valuable hour an agent program can spend. Hand the MCP connection to one team that already lives in Claude or ChatGPT and let their usage teach you what context is missing. Pick the slowest, most expensive workload in your warehouse as the first view-swap migration candidate, since that's where Iceberg plus Reflections pays back fastest. Then let Autonomous Reflections and Automated Table Optimization run for two weeks and compare your engineering backlog before and after. Each step is reversible, each one delivers value on its own, and none of them requires the others to finish first.
SOCIAL SHARE CARD GENERATOR