All Data and AI Weekly #215: 10 November 2025

All Data and AI Weekly

( AI, Data, NiFi, Iceberg, Polaris, Streamlit, Flink, Kafka, Python, Java, SQL, MCP, LLM, RAG, Cortex AI, AISQL, Search, Unstructured Data )

#215: 10 November 2025

https://www.reddit.com/r/DataEngineeringForAI/hot/

https://lu.ma/PINSAI

This week was dominated by a wave of major announcements from Snowflake BUILD 2025. The central theme is clear: the rapid integration of agentic AI, advanced developer tools, and more open data pipelines directly into the enterprise data stack.

In This Issue:

• 
  The Big Story: Snowflake BUILD 2025 & The Rise of Enterprise AI
  


• 
  AI Deep Dive: Cortex Agents, AI-SQL, and Audio/Video
  


• 
  Data Engineering: Openflow, Postgres Integration, and New Pipelines
  


• 
  Developer Focus: Well-Architected Framework & New Dev Tools
  


• 
  Security & Evaluation: Practical AI Agent Security & Observability
  


• 
  Events & Learning: Upcoming Webinars & Courses
  


• 
  Around the Web: Other Noteworthy Tech

The Big Story: Snowflake BUILD 2025

The main headlines from Snowflake BUILD 2025, focusing on a major push to supercharge enterprise-grade AI development and make data more accessible.

• 
  [Snowflake Press Release]: Snowflake Unveils New Developer Tools to Supercharge Enterprise-Grade Agentic AI Development
  


• 
  [Snowflake Press Release]: Snowflake Delivers the Enterprise Lakehouse with Enhanced Open Data Access
  


• 
  [Official Blog]: Snowflake Intelligence: The Engine for Enterprise AI
  


• 
  [Official Blog]: Fuel Your AI Strategy with Connected, Accessible Data
  


• 
  [Official Blog]: Delivering Intelligent, Governed AI at Scale
  


• 
  [What's New (GitHub)]: See What's New in Snowsight from BUILD 2025
  


• 
  [Video Overview]: Snowflake Intelligence (Video)
  

AI Deep Dive: Cortex AI & Agents

The most significant updates are centered on Snowflake Cortex AI, with new tools for building, managing, and observing AI agents.

• 
  [Getting Started]: Get Started with Snowflake Intelligence
  


• 
  [Cortex Agents]: Getting Started with Cortex Agents
  
  
  
  
  • 
    [Example YAML]: Sample sales_metrics_model.yaml for a Cortex Agent
    
  
  
  • 
    [Awesome List]: Awesome Custom Cortex Agents (REST API & React App)
    
  
  
  
  
  
  



• 
  
  [Cortex AI-SQL]:
  
  
  
  
  • 
    [GA Announcement]: Cortex AI-SQL Operators Now GA
    
  
  
  • 
    [Engineering Blog]: Real-time Text-to-SQL with Snowflake Intelligence
    
  
  
  • 
    [Guide]: Getting Started with Cortex AI-SQL
    
  
  
  • 
    [Examples]: More AI-SQL Examples
    
  
  
  
  
  
  



• 
  
  [Cortex Audio/Video]:
  
  
  
  
  • 
    [GA Announcement]: Cortex AI Transcribe Now GA
    
  
  
  • 
    [User Guide]: Cortex AI Audio Functions Guide
    
  
  
  • 
    [Guide]: Extracting Insights from Multimodal Customer Data
    
  
  
  
  
  
  



• 
  
  [Cortex Search]: Cortex Search Overview
  



• 
  
  [MS Teams Integration]:
  
  
  
  
  • 
    [Blog]: Cortex Agents for Microsoft Marketplace
    
  
  
  • 
    [Guide]: Chat with Snowflake Data from Microsoft Teams
    
  
  
  • 
    [Docs]: Cortex Agents Teams Integration Docs
    
  
  
  
  
  
  

Data Engineering: Openflow, Postgres Integration & New Pipelines

Significant updates for data integration, including the new "pg_lake" for Postgres, Openflow for CDC, and Dynamic Iceberg Tables.

• 
  [Postgres Lakehouse (pg_lake)]:
  
  
  
  
  • 
    [Engineering Blog]: pg_lake: A New Era for Postgres Lakehouse Integration
    
  
  
  • 
    [GitHub Repo]: pg_lake GitHub Repository
    
  
  
  
  
  
  



• 
  
  [Openflow (CDC)]:
  
  
  
  
  • 
    [Guide]: Getting Started with Openflow for CDC on SQL Server
    
  
  
  • 
    [Guide]: Getting Started with Openflow (SPCS)
    
  
  
  • 
    [Guide]: Setup Openflow (BYOC - Bring Your Own Compute)
    
  
  
  • 
    [Demo]: Openflow Structured Data Pipeline Demo
    
  
  
  
  
  
  



• 
  
  [Interactive & Dynamic Tables]:
  
  
  
  
  • 
    [Guide]: Getting Started with Interactive Tables
    
  
  
  • 
    [Docs]: Interactive Tables Documentation
    
  
  
  • 
    [Guide]: Declarative Pipelines with Dynamic Tables
    
  
  
  • 
    [Example]: Dynamic Iceberg Tables (Python Example)
    
  
  
  
  
  
  



• 
  
  [Other Integrations]:
  
  
  
  
  • 
    [GitHub]: GitHub Integration with Snowflake Openflow
    
  
  
  • 
    [DeltaStream]: DeltaStream Enables Real-Time Streaming Natively in Snowflake
    
  
  
  
  
  
  

Developer Focus: Well-Architected Framework & New Dev Tools

Snowflake is doubling down on developer experience with a new Well-Architected Framework, an integrated developer environment, and new ML workflow guides.

• 
  [Well-Architected Framework]:
  
  
  
  
  • 
    [Product Page]: Snowflake Well-Architected Framework
    
  
  
  • 
    [Engineering Blog]: Announcing the Snowflake Well-Architected Framework
    
  
  
  • 
    [Quickstart]: Well-Architected Framework: Security & Governance Guide
    
  
  
  
  
  
  



• 
  
  [Developer Environment]:
  
  
  
  
  • 
    [Blog]: An Integrated Developer Environment for AI
    
  
  
  • 
    [Docs]: Notebooks in Workspaces (Limited Access)
    
  
  
  • 
    [Demo]: Notebooks in Workspaces Demo Repo
    
  
  
  
  
  
  



• 
  
  [MLOps & Workflows]:
  
  
  
  
  • 
    [Guide]: From Dev to Production: Why ML Teams are Migrating to Snowflake
    
  
  
  • 
    [Guide]: Build an End-to-End ML Workflow in Snowflake
    
  
  
  • 
    [Toolkit]: Snowflake AI Toolkit (GitHub)
    
  
  
  
  
  
  

Security & Evaluation: Practical AI Agent Security & Observability

As AI agents become more powerful, security and evaluation are critical. These resources provide frameworks for securing and evaluating your new AI tools.

• 
  [Security]:
  
  
  
  
  • 
    [Meta Blog]: A Guide to Practical AI Agent Security
    
  
  
  • 
    [Snowflake Blog]: Security and Governance Practices for Snowflake Intelligence
    
  
  
  • 
    [Anthropic Blog]: Code Execution with Model-written Code Patches (MCP)
    
  
  
  • 
    [GitHub]: Anthropic's Model-written Code Patches (MCP) Repo
    
  
  
  
  
  
  



• 
  
  [Evaluation & Observability]:
  
  
  
  
  • 
    [Snowflake Blog]: An AI Agent Evaluation Framework: The GPA Framework
    
  
  
  • 
    [Docs]: Cortex AI Observability
    
  
  
  • 
    [TruLens Guide]: Web Search Agent Evaluation with TruLens
    
  
  
  • 
    [Video]: Evaluating AI Agents (Snowflake)
    
  
  
  
  
  
  

Events & Learning

Want to learn more? Here are some upcoming webinars and hands-on labs to get you started.

• 
  [Webinar]: Unlock the Power of Your Data with Snowflake Intelligence (Nov 13, 2025)
  


• 
  [Hands-on Lab]: Build Powerful AI Agents and Apps Using OpenAI in Snowflake Cortex AI (Nov 18, 2025)
  


• 
  [Course]: DeepLearning.AI & Snowflake: Building and Evaluating Data Agents
  


• 
  [Hands-on Lab]: MCP (Model-written Code Patches) HOL from BUILD 2025
  


• https://www.snowflake.com/en/webinars/demo/build-semanticaware-ai-with-snowflake-and-dbt-labs-2025-11-25/

Around the Web: Other Noteworthy Tech

A few other interesting projects and articles that caught our eye this week.

• 
  [Neo4j]: NODES 2025 Talk: From Air Quality to Aircraft, Data Is Everywhere (Slides)
  


• 
  [Data Pipelines]: dlt (Data Load Tool) GitHub Repo
  


• 
  [Supabase]: Supabase ETL - A Postgres-based ETL
  


• 
  [Postgres]: pg_incremental - Incremental View Maintenance for Postgres
  


• 
  [Geospatial]: GeoParquet.org - Open Standard for Geospatial Vector Data
  


• 
  [Developer Tools]: Cursor: Semantic Search in Your Codebase
  


• 
  [Developer Tools]: Task - A Task Runner / Build Tool
  


• 
  [And For Fun...]: How to Calibrate an RTL-SDR (Software Defined Radio)
  

Thanks

https://sessionize.com/tspann

https://github.com/timothyspann

© 2020-2025 Tim Spann https://www.youtube.com/@FLaNK-Stack