Connecting Databricks to ADLS Gen2: A Step-by-Step Guide

1. Introduction

Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft’s enterprise-scale data lake built on top of Azure Blob Storage. It combines the hierarchical namespace and POSIX-compatible ACLs of a true data lake with the massive scalability, low cost, and tiered storage of Azure Blob. It is the go-to landing zone for raw, curated, and enriched datasets in modern lakehouse architectures.

Databricks, running on Apache Spark under the hood, is the most popular compute engine for processing data stored in ADLS Gen2. Together they form the foundation of the Azure Lakehouse pattern — raw data lands in ADLS Gen2, PySpark on Databricks transforms and enriches it, and the results are served back to consumers via Delta Lake tables, dashboards, or downstream APIs.

This blog walks you through every method of connecting Databricks to ADLS Gen2, reading data in popular formats (CSV, JSON, Parquet, Delta), applying transformations, and optimizing for production workloads.

2. Prerequisites

Before writing a single line of PySpark, make sure the following are in place:

An Azure subscription with an ADLS Gen2 storage account (hierarchical namespace enabled).
A Databricks workspace deployed in Azure (any tier — Community Edition works for learning).
A Databricks cluster running Databricks Runtime 11.0 or later (includes Spark 3.3+).
One of the following authentication mechanisms set up (covered in Section 4):
Azure Active Directory (AAD) Service Principal
Managed Identity assigned to the Databricks cluster
Storage Account Access Key (quick dev/test only)
Basic familiarity with PySpark DataFrames and Python.

3. Architecture Overview

Understanding how Databricks talks to ADLS Gen2 before diving into code saves a lot of debugging time. There are two connection paths:

3.1 Direct Access (abfss://)

The recommended modern approach. PySpark reads and writes directly to ADLS Gen2 using the Azure Blob File System Secure (ABFSS) driver. No intermediate mount point is created, and credentials are configured once at the Spark session level or via Databricks Secrets.

			
abfss://<container>@<storage-account>.dfs.core.windows.net/<path>   is the ABFSS URI format.

3.2 Mount Points (dbfs:/mnt/)

The legacy approach. A mount point maps an ADLS Gen2 path to a Databricks File System (DBFS) virtual path. Once mounted, notebooks refer to data as dbfs:/mnt/mydata/file.csv without knowing the underlying storage URI. Mounts are cluster-scoped and persist across restarts.

Mount points are convenient but are being deprecated in Unity Catalog-enabled workspaces. Prefer direct abfss:// access for new projects.

4. Authentication Methods

Method 1 — Service Principal (Recommended for Production)

A Service Principal (SP) is an AAD application identity. You grant the SP the Storage Blob Data Reader (or Contributor) role on the ADLS Gen2 account, then configure Databricks to authenticate as that SP.

Step 1: Register an AAD App and note the credentials

			
# Values obtained from Azure Portal > App Registrations
CLIENT_ID     = "<your-app-client-id>"
CLIENT_SECRET = "<your-app-client-secret>"   # store in Databricks Secrets!
TENANT_ID     = "<your-aad-tenant-id>"
STORAGE_ACCT  = "<your-storage-account-name>"

		

Step 2: Configure Spark session with SP credentials

			
spark.conf.set(
    f"fs.azure.account.auth.type.{STORAGE_ACCT}.dfs.core.windows.net",
    "OAuth"
)
spark.conf.set(
    f"fs.azure.account.oauth.provider.type.{STORAGE_ACCT}.dfs.core.windows.net",
    "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.id.{STORAGE_ACCT}.dfs.core.windows.net",
    CLIENT_ID
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.secret.{STORAGE_ACCT}.dfs.core.windows.net",
    dbutils.secrets.get(scope="kv-scope", key="sp-secret")  # Never hardcode!
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.endpoint.{STORAGE_ACCT}.dfs.core.windows.net",
    f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/token"

		

Method 2 — Access Key (Dev/Test Only)

The simplest but least secure option. The storage account access key grants full access to ALL containers. Never use this in production.

			
spark.conf.set(
    f"fs.azure.account.key.{STORAGE_ACCT}.dfs.core.windows.net",
    dbutils.secrets.get(scope="kv-scope", key="storage-key")
)

Method 3 — Managed Identity

When your Databricks cluster is assigned a User-Assigned Managed Identity (or uses the workspace System-Assigned MI), no credentials need to be managed at all. Azure handles token refresh automatically.

			
# No explicit credential config needed — just ensure the Managed Identity
# has the 'Storage Blob Data Reader' role on the ADLS Gen2 account.
# Then read directly:
df = spark.read.parquet(
    f"abfss://raw@{STORAGE_ACCT}.dfs.core.windows.net/sales/2024/"
)

		

Authentication Methods at a Glance

Method	Security	Scope	Best For
Service Principal + Secret	High	Per storage account	Production pipelines
Managed Identity	Highest	Automatic via Azure	AKS / Databricks on Azure
Access Key	Low	Full account access	Quick dev/test only
SAS Token	Medium	Per container/blob	Temporary external access

5. Mounting ADLS Gen2 to DBFS (Legacy)

If you are on a workspace without Unity Catalog, or need backwards compatibility with legacy notebooks, mount points are still widely used. The mount is created once per workspace; all subsequent clusters see it automatically.

			
configs = {
  "fs.azure.account.auth.type":
      "OAuth",
  "fs.azure.account.oauth.provider.type":
      "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  "fs.azure.account.oauth2.client.id":
      dbutils.secrets.get(scope="kv-scope", key="client-id"),
  "fs.azure.account.oauth2.client.secret":
      dbutils.secrets.get(scope="kv-scope", key="sp-secret"),
  "fs.azure.account.oauth2.client.endpoint":
      f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/token",
}
dbutils.fs.mount(
    source      = f"abfss://raw@{STORAGE_ACCT}.dfs.core.windows.net/",
    mount_point = "/mnt/adls-raw",
    extra_configs = configs
)
# Verify
dbutils.fs.ls("/mnt/adls-raw")

		

After mounting, all notebooks can reference files as /mnt/adls-raw/sales/2024/data.parquet without knowing the underlying ADLS Gen2 URI.

6. Reading Data with PySpark

Once authentication is configured, reading data is straightforward. PySpark’s DataFrameReader supports CSV, JSON, Parquet, Avro, ORC, and Delta Lake out of the box on Databricks.

6.1 Reading CSV

			
CONTAINER    = "raw"
STORAGE_ACCT = "mydatalake"
BASE_PATH    = f"abfss://{CONTAINER}@{STORAGE_ACCT}.dfs.core.windows.net"
df_csv = (
    spark.read
         .format("csv")
         .option("header",          "true")
         .option("inferSchema",     "true")
         .option("sep",             ",")
         .option("nullValue",       "NULL")
         .option("dateFormat",      "yyyy-MM-dd")
         .option("multiLine",       "false")
         .load(f"{BASE_PATH}/sales/2024/*.csv")
)
df_csv.printSchema()
df_csv.show(5, truncate=False)

		

Tip: Always define an explicit schema instead of inferSchema=true for large production datasets. Schema inference forces a full first-pass scan of the data.

6.2 Reading JSON

			
df_json = (
    spark.read
         .format("json")
         .option("multiLine", "true")   # for pretty-printed JSON files
         .option("mode",      "PERMISSIVE")  # log corrupt records
         .load(f"{BASE_PATH}/events/2024/")
)
# Inspect nested schema
df_json.printSchema()

		

6.3 Reading Parquet

Parquet is the preferred format for analytical workloads — columnar storage, built-in compression, and schema embedded in the file footer make it far faster than CSV or JSON for large tables.

			
df_parquet = (
    spark.read
         .format("parquet")
         .load(f"{BASE_PATH}/customers/")
)
# Partition pruning — only reads year=2024/month=03
df_filtered = df_parquet.filter(
    (df_parquet.year  == 2024) &
    (df_parquet.month == 3)
)
df_filtered.show()

		

6.4 Reading Delta Lake Tables

Delta Lake is the default table format in Databricks. It adds ACID transactions, time travel, schema enforcement, and DML support on top of Parquet files in ADLS Gen2.

			
# Read current snapshot
df_delta = (
    spark.read
         .format("delta")
         .load(f"{BASE_PATH}/curated/sales_delta/")
)
# Time travel — read data as of a specific version
df_v5 = (
    spark.read
         .format("delta")
         .option("versionAsOf", 5)
         .load(f"{BASE_PATH}/curated/sales_delta/")
)
# Time travel — read data as of a timestamp
df_ts = (
    spark.read
         .format("delta")
         .option("timestampAsOf", "2024-01-15 00:00:00")
         .load(f"{BASE_PATH}/curated/sales_delta/")
)

		

6.5 Defining an Explicit Schema

For CSV and JSON, always define the schema explicitly in production. This avoids full-file scans during schema inference and ensures type safety.

			
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, DateType, IntegerType
sales_schema = StructType([
    StructField("order_id",    StringType(),  nullable=False),
    StructField("customer_id", StringType(),  nullable=False),
    StructField("product_id",  StringType(),  nullable=True),
    StructField("quantity",    IntegerType(), nullable=True),
    StructField("unit_price",  DoubleType(),  nullable=True),
    StructField("order_date",  DateType(),    nullable=True),
    StructField("region",      StringType(),  nullable=True),
])
df = (
    spark.read
         .schema(sales_schema)
         .option("header", "true")
         .csv(f"{BASE_PATH}/sales/")
)
df.printSchema()

		

7. Working with the Data

Once the DataFrame is loaded, you have the full power of PySpark at your disposal — filtering, aggregating, joining, and applying UDFs.

7.1 Filtering and Projecting

			
from pyspark.sql.functions import col, year, month
df_q1 = (
    df
    .filter(
        (year(col("order_date")) == 2024) &
        (month(col("order_date")).isin(1, 2, 3))
    )
    .select("order_id", "customer_id", "unit_price", "quantity", "region")
    .withColumn("revenue", col("unit_price") * col("quantity"))
)
df_q1.show(10)

		

7.2 Aggregations

			
from pyspark.sql.functions import sum, avg, count, round
revenue_by_region = (
    df_q1
    .groupBy("region")
    .agg(
        sum("revenue").alias("total_revenue"),
        avg("unit_price").alias("avg_price"),
        count("order_id").alias("order_count")
    )
    .withColumn("total_revenue", round(col("total_revenue"), 2))
    .orderBy(col("total_revenue").desc())
)
revenue_by_region.show()

		

7.3 Joining DataFrames

			
df_customers = spark.read.format("delta").load(f"{BASE_PATH}/curated/customers/")
df_enriched = (
    df_q1
    .join(df_customers, on="customer_id", how="left")
    .select(
        "order_id", "customer_id",
        "first_name", "last_name",
        "region", "revenue"
    )
)
df_enriched.show(5)

		

7.4 Registering as a Temp View for SQL

Databricks notebooks support seamless mixing of PySpark and SQL. Register a DataFrame as a temporary view, then query it with %sql magic or spark.sql().

			
df_enriched.createOrReplaceTempView("sales_enriched")
# Option A: spark.sql()
top_customers = spark.sql("""
    SELECT   customer_id, first_name, last_name,
             SUM(revenue) AS total_spent
    FROM     sales_enriched
    GROUP BY customer_id, first_name, last_name
    ORDER BY total_spent DESC
    LIMIT    20
""")
top_customers.show()
# Option B: Use %sql magic in the next notebook cell
# %sql SELECT * FROM sales_enriched WHERE region = 'EMEA'

		

8. Writing Results Back to ADLS Gen2

After transforming data, write it back to the curated or enriched zone in ADLS Gen2. Delta Lake is recommended for all curated outputs.

			
# Write as Delta (recommended)
(
    df_enriched
    .write
    .format("delta")
    .mode("overwrite")            # or 'append'
    .partitionBy("region")        # partition for query pruning
    .option("overwriteSchema", "true")
    .save(f"{BASE_PATH}/curated/sales_enriched/")
)
# Write as Parquet
(
    df_enriched
    .write
    .format("parquet")
    .mode("overwrite")
    .partitionBy("region")
    .save(f"{BASE_PATH}/curated/sales_parquet/")
)

		

9. Performance Optimization Tips

9.1 Partition Pruning

Store data partitioned by date or region and always filter on partition columns. Spark will skip entire directories that don’t match, dramatically reducing I/O.

			
# Writes partitioned by year and month
df.write.partitionBy("year", "month").format("delta").save(path)
# Read with partition filter — only scans year=2024/month=03
df_march = spark.read.format("delta").load(path).filter(
    (col("year") == 2024) & (col("month") == 3)

		

9.2 Delta Lake OPTIMIZE and Z-ORDER

			
from delta.tables import DeltaTable
dt = DeltaTable.forPath(spark, f"{BASE_PATH}/curated/sales_enriched/")
# Compact small files
dt.optimize().executeCompaction()
# Z-ORDER co-locates data by query columns — ideal for high-cardinality filters
dt.optimize().executeZOrderBy("customer_id", "order_date")

		

9.3 Caching Frequently Accessed DataFrames

			
df_customers = spark.read.format("delta").load(f"{BASE_PATH}/curated/customers/")
df_customers.cache()          # persists to Spark memory across actions
df_customers.count()          # trigger caching
# ... run multiple joins against df_customers ...
df_customers.unpersist()      # release memory when done

		

9.4 Avoiding Small File Problems

Many small files in ADLS Gen2 cause high metadata overhead and slow reads. Use the following strategies:

Set spark.sql.shuffle.partitions to match cluster size (default 200 is often too high).
Use coalesce() before writing to reduce output file count.
Run OPTIMIZE periodically on Delta tables to compact small files automatically.
Enable Auto Optimize on Delta tables for streaming or frequent batch writes.

			
# Coalesce before writing to reduce file count
df_enriched.coalesce(8).write.format("delta").mode("append").save(path)
# Auto Optimize properties on a Delta table
spark.sql("""
    ALTER TABLE delta.`abfss://curated@mydatalake.dfs.core.windows.net/sales/`
    SET TBLPROPERTIES (
        delta.autoOptimize.optimizeWrite = true,
        delta.autoOptimize.autoCompact   = true
    )
""")

		

10. Security & Best Practices

Practice	Why It Matters	How to Implement
Never hardcode credentials	Secrets in notebooks leak via version control	Use Databricks Secrets backed by Azure Key Vault
Use Service Principal per pipeline	Least-privilege access per workload	Assign only Storage Blob Data Reader unless writes needed
Prefer abfss:// over mount points	Mount points deprecated in Unity Catalog	Configure Spark session config or cluster init scripts
Define explicit schemas	inferSchema scans all data on read	Use StructType for CSV/JSON in production
Partition data strategically	Skipped partitions = faster queries	Partition by date/region, filter on partition columns
Compact Delta tables regularly	Small files degrade read performance	Schedule OPTIMIZE + VACUUM jobs weekly
Enable VACUUM for Delta	Removes old data files to save storage cost	Set delta.deletedFileRetentionDuration = interval 7 days

11. Common Errors & Fixes

Error	Root Cause	Fix
AuthorizationPermissionMismatch	SP or MI lacks role on storage	Assign Storage Blob Data Reader/Contributor in IAM
Container not found	Wrong container name or typo in abfss:// URI	Verify container name in Azure Portal
AnalysisException: Path does not exist	Wrong file path or no files match glob	Run dbutils.fs.ls() to verify path
Schema mismatch on Delta write	New data schema differs from table schema	Set .option(‘mergeSchema’,’true’) or update schema
Job slow — many small files	Lots of small Parquet/Delta files	Run OPTIMIZE; use coalesce() before writes
Mount already exists	dbutils.fs.mount() called twice	Check dbutils.fs.mounts() and unmount first

12. Conclusion

ADLS Gen2 and Databricks are purpose-built for each other within the Azure ecosystem. Together they cover the full data lifecycle — landing raw files in ADLS Gen2, processing them with PySpark on Databricks, and serving results as Delta Lake tables that support BI tools, ML models, and downstream APIs.

The key takeaways from this guide:

Use Service Principal + Databricks Secrets or Managed Identity for authentication — never hardcode keys.
Prefer direct abfss:// access over DBFS mount points, especially in Unity Catalog workspaces.
Define explicit schemas for CSV and JSON sources; let Parquet and Delta carry their own schemas.
Partition data by date or region and always filter on partition columns to enable pruning.
Keep Delta tables healthy with periodic OPTIMIZE, VACUUM, and Z-ORDER operations.