[Report] The Latest Form of AI-Ready Data Infrastructure - Achieving Multi-Engine Interoperability with Iceberg #SWTTokyo26

This is Kawabata.

"SNOWFLAKE WORLD TOUR 2026 - TOKYO" was held on September 10–11, 2026.

This article is a report blog for the session
【The Latest Form of AI-Ready Data Infrastructure - Multi-Engine Interoperability Unlocked by Iceberg】.

※Some features are in preview or relate to future roadmap items. Please check the latest official documentation for availability.
※Statements by speakers are summarized based on the author's notes.

Speaker

• Kazutaka Kubo
  • Snowflake Solutions Engineering Division, Solutions Engineer

Why Do Data Copies Multiply?

• The session opened with the question: before you know it, haven't copies of the same data been piling up — for analytics, for machine learning, for sharing with customers, and so on?
  • The session introduced survey results showing that more than half of companies create numerous copies even just in test and development environments
• There are three main causes for copies being created
  • Because engines cannot read data directly, copies are made to suit tools such as BI or Python
  • Access controls are in place, but the only way to share data with others is to hand over a pre-masked copy
  • Permissions can be granted, but since applying them to production takes a month, the person in charge copies and hands over the data
• The problem is not the behavior of the individuals involved. The problem is the "lack of interoperability" that prevents direct access to data where it resides

• Three problems arise when copies multiply
  • Cost: In addition to storage fees, compute costs for ETL pipelines are incurred
  • Governance fragmentation: It becomes impossible to ensure that all copies are under control, increasing the risk of data leakage
  • Loss of business context: Definitions of terms like "revenue" diverge across teams, and even asking AI yields inconsistent answers

• Three things are needed to solve this, and Snowflake can now satisfy all of them
  • Data sovereignty: Decide for yourself where data is stored and in what format
  • Centralized metadata and governance: Controls applied in one place are enforced everywhere
  • Connect to governed data with any engine of your choice

Where to Store Data and in What Format

Iceberg Reduces Dependency on Where Data Is Stored

• The speaker emphasized at the outset that "this is not a talk asking you to put your data in Snowflake"
• Iceberg appears in both Snowflake-managed tables and externally managed tables
  • By holding data in Iceberg format and using compatible catalogs and engines, data can be read and written regardless of where it is stored

Operational Gaps Resolved by Iceberg v3

• Iceberg has existed for several years, but a situation where everything is held in Iceberg has not materialized
  • The speaker's assessment was that this is because there were significant operational gaps in v1/v2
• In v3, features were added that address those gaps on a one-to-one basis

• Additionally, geometry and geography types, nanosecond-precision timestamps, and column default values have been added
• Snowflake already supports v3. It became generally available (GA) on May 7, 2026
  • It can be used simply by specifying ICEBERG_VERSION = 3 in the table definition
• External engines that support the Iceberg REST Catalog (IRC) API can also read from and write to Snowflake-managed v2/v3 tables
  • Writing from external engines became GA on May 26, 2026

The "Iceberg" of Self-Managed Bucket Operations and Snowflake Managed Storage

• Customers who have considered Iceberg often say they want to manage it just like conventional Snowflake tables
  • This is because they realize that managing their own buckets involves operational overhead beyond just storage
• The visible costs are storage fees and API request fees. In reality, the hidden costs are larger
  • Failover, encryption, and the cost of designing and operating those systems

• In response, Snowflake began offering Snowflake-managed storage for Iceberg tables
  • The official name is Snowflake storage for Apache Iceberg tables. GA in June 2026
  • Supports AWS and Azure, and is available in the Tokyo region as well
  • Fail-safe, high availability, encryption, and storage maintenance can all be delegated to Snowflake
• Usage is as simple as specifying EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED' when creating a table
  • No bucket design or IAM configuration is needed, allowing you to use Iceberg with the same experience as conventional tables

※According to the official documentation, encryption for Snowflake managed storage is server-side encryption (SSE) only and does not support customer-managed keys (CMK).

• It is also advantageous in terms of pricing (based on publicly listed prices for AWS US East, comparing monthly costs per TB)
  • Maintenance such as compaction, snapshot expiration processing, and manifest cleanup can be offloaded
  • When accessed from Snowflake's engine, storage request fees are not charged
• The claim is that the difference lies not in "whether Iceberg is supported," but in the aspects that include operations and billing
  • It does not matter where the data is stored. This is positioned as one additional option among many

Snowflake Horizon Catalog

• Horizon Catalog is Snowflake's catalog, and the interoperability layer is built on Apache Polaris
  • Both Snowflake-managed tables and tables in external catalogs can be read and written through this layer
• The three pillars are governance and security, context, and interoperability

Applying Policies to External Engines

• Snowflake's governance is powerful, operating at the row and column level. However, when using data from external engines, it tended to be passed as a copy
• With Horizon Catalog, Snowflake's policies can also be applied to access via the IRC API
• There are two paths
  • Snowflake Connector for Spark: GA. Can be used with almost no changes to existing Spark code
  • Iceberg Scan Plan API: A mechanism for applying policies to engines other than Spark. Entered Public Preview on September 10, 2026
• Using the supported paths, masking policies and row access policies defined once in Snowflake can also be applied to external engines

Semantic Views

• When departments differ, the same question returns different answers. The cause is often a divergence in definitions of terms like "revenue"
• Semantic Views define table joins and metrics in YAML, creating a single source of truth
  • Cortex Analyst, external BI tools such as Power BI, and in-house applications all reference the same definition
• There is no need to write from scratch. Semantic View Autopilot can auto-generate from existing Power BI files and similar sources
• This is not about fine-tuning AI models. It is about defining meaning in data so that AI can interpret it correctly

Reading and Writing via Standard APIs

• Horizon Catalog has standard APIs for Apache Polaris and Iceberg REST Catalog
  • Engines that support the standard API, such as Spark-based services and Trino, can connect to it
  • Not only reading but also writing is supported
• Storage can be either Snowflake storage or cloud storage — either is fine
• Catalogs that were previously set up per engine can be consolidated into one, simplifying the architecture

Consolidating External Catalogs with catalog-linked database

• catalog-linked database (CLD) is a mechanism for linking external catalogs to Snowflake
  • Catalogs such as Apache Polaris, Unity Catalog, and AWS Glue can be linked
• Once linked, external Iceberg tables are automatically discovered, and additions and changes on the other side are automatically tracked
• Policies can also be set on linked tables from the Snowflake side, with the explanation that this reduces the need for dual management between Horizon and the other side
  • However, according to the official documentation, policies cannot be evaluated on paths where external engines read table files directly
  • To read policy-protected tables from external engines, a path such as Snowflake Connector for Spark is required

Centralizing into ACCESS_HISTORY

• As connectivity from any engine becomes possible, auditing becomes more difficult
• Operations from external engines are also recorded in ACCESS_HISTORY
  • Filtering by the "Horizon IRC" condition allows you to review operations via external engines
• Both policies and auditing are consolidated in one place, enabling centralized governance

• MCP is listed among the API suite of Horizon Catalog. The design assumes that AI agents access data through this layer

Demo: Reading and Writing from AWS Glue with Snowflake Governance Applied

• AWS Glue — a third-party managed service outside of Snowflake
• On the Snowflake side, two Iceberg tables were created by specifying EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
  • A table without policies and a table with policies
• Two policies were applied to the table with policies
  • A masking policy on the customer name and a row access policy allowing visibility only into the responsible region
• Behavior was verified on Snowflake
  • With ACCOUNTADMIN, all rows are visible without masking
  • With the role for Glue, only rows for the Tokyo region are visible and customer names are masked
• Executing a single INSERT row from a Glue job (Spark) completed in about one minute. The row could be confirmed on the Snowflake side
  • Without using a Snowflake warehouse, writing to a table on Snowflake managed storage via Horizon Catalog from Glue was successful
• For reading from a Glue notebook, authentication was done using the Glue IAM role without secrets
  • SELECT results showed only rows for the Tokyo region, with customer names masked
• It was demonstrated that Snowflake's policies are applied even to access from an external engine (Glue's Spark)

Answer to Lock-in Concerns

Commitment to Open Standards

• A response to the concern: "It looks great, but won't we be locked in to Snowflake?"
• For working with the same data across engines, specifications need to be common. Standards are built together with the community
  • Apache Iceberg: 5 PMC members from Snowflake. Reportedly reviewing 80% of pull requests
  • Apache Polaris: Donated to the Apache Software Foundation in August 2024. Graduated to a top-level project in February 2026
  • Apache Ossie (Incubating, formerly Open Semantic Interchange): A standard for the semantic layer
• Apache Ossie was introduced as an initiative launched jointly with more than 55 vendors
• Competitors are also participating, which was cited as evidence that this is not about lock-in

Sharing Data with Non-Snowflake Parties

• Means are available to share data even when the recipient is not using Snowflake
  • Open table format sharing, CLD sharing, cross-engine access via IRC, open sharing

• Open Data Sharing (Public Preview) allows data to be delivered to more than 40 IRC-compatible clients via listings
  • An external user called an "external consumer" is created for external connections and a token is issued
  • Only a URL and token need to be provided to the recipient. No Snowflake account is required
  • Connection is possible from IRC-compatible engines as well as from Spark and JDBC

• Not only data but also applications and AI assets can be shared
  • Data products can be distributed with business logic attached
• The answer to lock-in comes down to three points
  • Data remains in an open format
  • The catalog has been relinquished from sole company control
  • Snowflake is not required of sharing recipients

Session Summary

• There is one purpose behind all the updates introduced: "governance and discovery of all data." This has been a consistent effort over two years

• The question to take away is: "When another engine touches your table, what happens?"
  • Are rows filtered?
  • Are columns masked?
  • Is the operation recorded in the audit log?
• If all answers are "yes," the environment is AI-ready. If even one answer is "no," that is a good place to start a PoC

Related Articles

The Day 1 KEYNOTE report is here.

https://dev.classmethod.jp/articles/snowflake-world-tour-2026-tokyo-day1-keynote-swttokyo26/

The official documentation for the features introduced in the session is here.

https://docs.snowflake.com/en/user-guide/tables-iceberg-v3-specification-support

https://docs.snowflake.com/en/user-guide/tables-iceberg-internal-storage

https://docs.snowflake.com/en/user-guide/tables-iceberg-access-using-external-query-engine-snowflake-horizon

https://docs.snowflake.com/en/user-guide/tables-iceberg-query-using-external-query-engine-snowflake-horizon-enforce-access-policies

https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database

https://docs.snowflake.com/en/user-guide/open-data-sharing

Personal Impressions

I attended the session "The Latest Form of AI-Ready Data Infrastructure - Multi-Engine Interoperability Unlocked by Iceberg" at Snowflake World Tour Tokyo 2026 Day 2. It was a session that allowed me to understand Snowflake's open strategy centered on Iceberg in a cohesive way, from three perspectives: storage, governance, and sharing.

I myself had not had a sufficient grasp of the overall picture around Iceberg, so I was glad to get it organized through this session.
I will continue to explore use cases where Iceberg can be leveraged, through hands-on verification.

I hope this article is helpful to someone!