Databricks

On this page:

This document describes the steps required to configure a Databricks compute cluster to connect to a Tabular warehouse.

Requirements

  1. Databricks workspace (you must not have Databricks Unity catalog enabled as this prevents configuring Iceberg catalogs).
  2. A Tabular member credential or Service account credential for authenticating the Databricks cluster with your Tabular warehouse.

Note We recommend using Databricks version 14.x with Spark 3.5 or greater. Spark 3.5 includes native support for record-level query plans such as deletes and merges using Iceberg tables.

Setting up a cluster

Configuring a cluster

  1. On the cluster Configuration page, disable the Use Photon Acceleration option as Photon does not support Iceberg tables.
  2. Provide the following options in the Advanced Configuration -> Spark -> Spark config section. When you do this, be sure to replace <your-warehouse> with your Tabular warehouse name and <tabular-credential> with your Tabular credential:
spark.sql.catalog.<your-warehouse>               org.apache.iceberg.spark.SparkCatalog

spark.sql.catalog.<your-warehouse>.catalog-impl  org.apache.iceberg.rest.RESTCatalog

spark.sql.catalog.<your-warehouse>.uri           https://api.tabular.io/ws

spark.sql.catalog.<your-warehouse>.credential    <tabular-credential>

spark.sql.catalog.<your-warehouse>.warehouse     <your-warehouse>

Installing the Iceberg libraries

  1. On the cluster Libraries page, install the following two libraries from Maven:
    • org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0
    • org.apache.iceberg:iceberg-aws-bundle:1.5.0

Note    Ensure that the version for the iceberg-spark-runtime version matches the version of Spark for your cluster (Spark 3.5 in this example). Also ensure the iceberg-xxx-bundle matches the cloud storage provider for your Tabular warehouse (AWS in this example).

That’s it! When your cluster is up and running you can interact with your Tabular warehouse as you would from any Spark system. Tabular provides access controls to the Iceberg tables and underlying storage system based on the credential used when configuring the cluster. See the Apache Iceberg documentation for more information.

Limitations

  • Iceberg catalogs, including Tabular, are not supported from workspaces that have Unity catalog enabled.
  • Iceberg tables are not supported when using SQL warehouse compute clusters.