1. Introduction
Out-of-memory errors, excessive disk spills, slow jobs, and garbage-collection pauses — these are the most common performance killers in PySpark applications, and they all trace back to one root cause: misconfigured or misunderstood memory management.
Apache Spark uses a sophisticated, unified memory model that divides JVM heap space into purpose-built regions for execution, storage, and user data. PySpark adds another layer of complexity by maintaining a separate Python worker process that communicates with the JVM through serialisation. Getting the best out of this system requires understanding both tiers.
This blog provides a comprehensive guide to PySpark memory management — from the foundational architecture through configuration parameters, caching strategies, spill management, garbage collection tuning, and production monitoring — with practical code examples throughout.
2. Spark Unified Memory Architecture
Since Spark 1.6, the default memory manager is the Unified Memory Manager. It divides each executor’s JVM heap into three logical regions and allows the boundary between execution and storage memory to shift dynamically based on demand.
2.1 The Three Memory Regions
| Region | Fraction of Heap | Purpose |
| Reserved Memory | ~300 MB fixed | Internal Spark system usage; not configurable |
| User Memory | (1 – spark.memory.fraction) of (heap – reserved) | User data structures, UDFs, metadata — fully managed by application code |
| Spark Memory Pool | spark.memory.fraction (default 0.6) of (heap – reserved) | Shared pool split between Execution and Storage memory |
2.2 Execution vs Storage Memory
Within the Spark Memory Pool, execution and storage memory compete dynamically. Neither region has a fixed allocation — each can borrow from the other up to configured limits.
| Memory Type | Used For | Eviction Policy |
| Execution Memory | Shuffle buffers, sort operations, join hash tables, aggregation | Cannot be evicted — spills to disk instead |
| Storage Memory | Cached RDDs, DataFrames, broadcast variables | Can be evicted (LRU) to free space for execution |
2.3 Memory Layout Formula
# Total executor memory breakdown (approximate)heap_size = spark.executor.memory # e.g. 8greserved = 300 MB # fixed system reserveusable_heap = heap_size - reservedspark_memory_pool = usable_heap * spark.memory.fraction # default 0.6user_memory = usable_heap * (1 - spark.memory.fraction) # default 0.4storage_memory = spark_memory_pool * spark.memory.storageFraction # default 0.5execution_memory = spark_memory_pool - storage_memory# Example: 8GB executor# usable = 8192 - 300 = 7892 MB# spark pool = 7892 * 0.6 = 4735 MB# user mem = 7892 * 0.4 = 3157 MB# storage = 4735 * 0.5 = 2367 MB (initial — can grow)# execution = 4735 * 0.5 = 2367 MB (initial — can grow)
2.4 Off-Heap Memory
Spark also supports off-heap memory — native memory managed outside the JVM garbage collector. It is used for Project Tungsten’s unsafe operations and can dramatically reduce GC pressure on large heaps.
# Enable off-heap memoryspark.conf.set('spark.memory.offHeap.enabled', 'true')spark.conf.set('spark.memory.offHeap.size', '4g')# Off-heap is primarily used by:# - Tungsten's UnsafeRow binary format# - Sorted external shuffle# - Some Dataset/DataFrame operations
3. PySpark-Specific Memory — The Python Worker
PySpark introduces an additional memory consumer that pure Scala/Java Spark jobs do not have: the Python worker process. Every executor spawns a Python subprocess to run user-defined Python code (UDFs, RDD maps, etc.). This process has its own memory footprint entirely separate from the JVM heap.
| Component | Location | What It Holds |
| JVM Executor | JVM Heap | Spark internals, shuffle, storage, execution |
| Python Worker | Native OS Memory | Python UDF execution, pandas operations, Arrow batches |
| Off-Heap (Tungsten) | Native OS Memory | Unsafe binary row format, sorted shuffle |
| Overhead Memory | Native OS Memory | JVM internals, NIO buffers, Python worker processes |
3.1 spark.executor.memoryOverhead
memoryOverhead is non-heap native memory allocated per executor for OS, JVM internals, and Python worker processes. It defaults to max(executor_memory * 0.1, 384 MB). For PySpark workloads with heavy Python usage, this often needs to be increased significantly.
from pyspark.sql import SparkSessionspark = SparkSession.builder \ .appName('MemoryTuning') \ .config('spark.executor.memory', '8g') \ .config('spark.executor.memoryOverhead', '2g') \ .config('spark.driver.memory', '4g') \ .config('spark.driver.memoryOverhead', '1g') \ .getOrCreate()
3.2 pyspark.worker.memory — Arrow-Enabled Pandas UDFs
When using Pandas UDFs (vectorised UDFs) with Apache Arrow, each Python worker processes Arrow batches. The batch size is controlled by spark.sql.execution.arrow.maxRecordsPerBatch and directly affects how much memory each Python worker consumes.
import pandas as pdfrom pyspark.sql.functions import pandas_udffrom pyspark.sql.types import DoubleType# Control Arrow batch size to limit per-worker memoryspark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'true')spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch', '50000')pandas_udf(DoubleType())def apply_discount(prices: pd.Series) -> pd.Series: return prices * 0.90df = df.withColumn('discounted_price', apply_discount(df['price']))
4. Key Memory Configuration Parameters
| Parameter | Default | Description |
| spark.executor.memory | 1g | JVM heap per executor |
| spark.executor.memoryOverhead | 10% or 384MB | Native overhead per executor (OS, Python workers) |
| spark.driver.memory | 1g | JVM heap for the driver process |
| spark.driver.memoryOverhead | 10% or 384MB | Native overhead for the driver |
| spark.memory.fraction | 0.6 | Fraction of heap for Spark memory pool (execution + storage) |
| spark.memory.storageFraction | 0.5 | Initial fraction of Spark pool for storage (dynamic) |
| spark.memory.offHeap.enabled | false | Enable off-heap Tungsten memory |
| spark.memory.offHeap.size | 0 | Off-heap allocation size |
| spark.sql.shuffle.partitions | 200 | Partitions for shuffle operations (affects per-task memory) |
| spark.default.parallelism | cores * 2 | Default parallelism for RDD operations |
| spark.executor.cores | 1 | Cores per executor (affects task concurrency and memory sharing) |
| spark.sql.execution.arrow.maxRecordsPerBatch | 10000 | Arrow batch size for pandas UDFs |
| spark.rdd.compress | false | Compress serialised RDD partitions |
| spark.serializer | Java | Use KryoSerializer for better performance |
Read this also –
5. Executor Memory Sizing — Fat vs Thin
One of the most impactful decisions in Spark tuning is choosing between fewer large executors (fat) or many small executors (thin). Each approach has distinct memory implications.
| Aspect | Fat Executors (few, large) | Thin Executors (many, small) |
| Memory per executor | Large — more in-memory data per node | Small — spills more likely |
| GC pressure | High — large heaps have long GC pauses | Lower — smaller heaps GC faster |
| Broadcast efficiency | Better — one large broadcast per executor | Worse — more copies across cluster |
| Task parallelism | High — many cores share data in RAM | Limited by small heap per task |
| HDFS throughput | Better — fewer executors, larger HDFS buffers | Moderate |
| Fault tolerance | Lower — losing one executor hurts more | Higher — losing small executor is cheap |
| Recommended for | Shuffle-heavy, broadcast-heavy jobs | Many independent small tasks |
Recommended Executor Sizing Formula
# Rule of thumb for YARN / Kubernetes clusters:# Leave 1 core per node for OS/YARN daemons# Leave 1 GB per node for OS# Max 5 cores per executor (HDFS throughput sweet spot)# Example: Node with 16 cores, 64 GB RAMusable_cores = 16 - 1 = 15executors_per_node = 15 // 5 = 3usable_memory = 64 - 1 = 63 GBmemory_per_executor = 63 // 3 = 21 GBexecutor_memory = 21 * 0.9 = 18 GB # leave 10% as overheadmemory_overhead = 21 - 18 = 3 GB# Spark config:# spark.executor.cores = 5# spark.executor.memory = 18g# spark.executor.memoryOverhead = 3g
6. Caching & Persistence — Using Storage Memory Wisely
Caching stores intermediate DataFrame or RDD results in the Storage Memory region so they can be reused without recomputation. Choosing the right storage level is essential to avoid wasting memory or causing unnecessary disk I/O.
6.1 Storage Levels Explained
| Storage Level | Memory | Disk | Serialised | Replicated | Best For |
| MEMORY_ONLY | Yes | No | No | No | Small datasets; fastest access |
| MEMORY_ONLY_SER | Yes | No | Yes | No | Saving memory at cost of CPU |
| MEMORY_AND_DISK | Yes | Yes | No | No | Medium datasets; spill allowed |
| MEMORY_AND_DISK_SER | Yes | Yes | Yes | No | Large datasets with limited RAM |
| DISK_ONLY | No | Yes | Yes | No | Very large datasets |
| OFF_HEAP | Off-heap | No | Yes | No | Reduce GC; needs offHeap enabled |
| MEMORY_ONLY_2 | Yes | No | No | Yes | Critical data needing fault tolerance |
6.2 Cache vs Persist
from pyspark import StorageLevel# cache() — shortcut for MEMORY_AND_DISK (DataFrame) or MEMORY_ONLY (RDD)df_cached = df.cache()# persist() — choose storage level explicitlydf.persist(StorageLevel.MEMORY_AND_DISK_SER)df.persist(StorageLevel.MEMORY_ONLY)df.persist(StorageLevel.DISK_ONLY)df.persist(StorageLevel.OFF_HEAP)# IMPORTANT: cache/persist is lazy — data is stored only on first actiondf_cached.count() # triggers caching# Always unpersist when done to release storage memorydf_cached.unpersist()df.unpersist(blocking=True) # wait for eviction to complete
6.3 When to Cache
- Cache when the same DataFrame is used in two or more downstream actions.
- Cache when recomputation is expensive (many transformations, complex joins, aggregations).
- Do NOT cache if the DataFrame is used only once — it wastes storage memory.
- Do NOT cache very large DataFrames that barely fit in memory — they cause eviction of other cached data.
- Prefer MEMORY_AND_DISK over MEMORY_ONLY for production jobs where OOM is a risk.
6.4 Checking Cache Status
# View cached DataFrames in Spark UI or programmaticallyspark.catalog.isCached('my_table') # True / False# List all cached tablesfor table in spark.catalog.listTables(): if table.isTemporary: print(f'{table.name}: cached={spark.catalog.isCached(table.name)}')# Check storage memory usageprint(spark.sparkContext.statusTracker().getExecutorInfos())
7. Shuffle Memory Management
Shuffle is the most memory-intensive operation in Spark. It occurs during operations like groupBy, join, distinct, repartition, and orderBy. During a shuffle, data is written to disk (shuffle write) and then read back (shuffle read) across the network. Tuning shuffle memory is critical for avoiding spills and OOM errors.
7.1 How Shuffle Uses Memory
Each task in the map stage uses execution memory to build an in-memory sort buffer (ExternalSorter). When this buffer fills up, data spills to disk. Frequent spills dramatically slow down jobs and can cause disk OOM on small nodes.
7.2 Reducing Shuffle Partitions
# Default: 200 shuffle partitions — often too many for small/medium data# or too few for very large data# For small to medium workloadsspark.conf.set('spark.sql.shuffle.partitions', '50')# For large workloads (10s of GB of shuffle data)spark.conf.set('spark.sql.shuffle.partitions', '2000')# Adaptive Query Execution (AQE) — automatically coalesces shuffle partitionsspark.conf.set('spark.sql.adaptive.enabled', 'true')spark.conf.set('spark.sql.adaptive.coalescePartitions.enabled', 'true')spark.conf.set('spark.sql.adaptive.coalescePartitions.minPartitionNum', '1')spark.conf.set('spark.sql.adaptive.advisoryPartitionSizeInBytes', '128MB')
7.3 Sort-Based vs Hash-Based Shuffle
# SortShuffleManager (default) — uses ExternalSorter with spill support# Better for large data; handles spill gracefully# For small data with few reducers, Spark may use BypassMergeSort# which writes directly to per-reducer files (no sorting overhead)# Tune spill thresholdspark.conf.set('spark.shuffle.spill.compress', 'true')spark.conf.set('spark.shuffle.compress', 'true')spark.conf.set('spark.io.compression.codec', 'lz4') # faster than snappy for CPU# External shuffle service (recommended on YARN)spark.conf.set('spark.shuffle.service.enabled', 'true')
7.4 Broadcast Join to Eliminate Shuffle
The most effective way to reduce shuffle memory pressure is to eliminate the shuffle entirely using a broadcast join. When one side of a join is small enough to fit in memory, broadcast it to all executors so each executor can perform a local hash join.
from pyspark.sql.functions import broadcast# Auto-broadcast threshold (default 10 MB)spark.conf.set('spark.sql.autoBroadcastJoinThreshold', '50MB')# Large tableorders = spark.read.parquet('s3://bucket/orders/')# Small lookup table — broadcast it explicitlyproducts = spark.read.parquet('s3://bucket/products/')# Broadcast join — no shuffle, no sort, O(1) memory on map sideresult = orders.join(broadcast(products), on='product_id', how='left')# Disable auto-broadcast for very large small tablesspark.conf.set('spark.sql.autoBroadcastJoinThreshold', '-1') # force sort-merge join
8. Disk Spill Management
When execution memory is exhausted, Spark spills data to disk — writing intermediate results to a temporary directory and reading them back when needed. Spill is a safety valve that prevents OOM errors but comes at a significant performance cost: disk I/O, serialisation overhead, and increased task duration.
8.1 Detecting Spill
Spill is visible in the Spark UI under the Stages tab. Look for the Spill (Memory) and Spill (Disk) columns. Consistent spill in the same stage indicates the task is under-resourced.
# Monitor spill programmatically via accumulators (Spark 3.x)# In Spark UI: Stages -> look for 'Spill (Memory)' and 'Spill (Disk)' columns# Common causes of spill:# 1. Too many tasks per executor (too many concurrent tasks fighting for execution memory)# 2. Large shuffle operations with insufficient shuffle partitions# 3. GroupBy/join on high-cardinality columns producing large per-task data# 4. Window functions over large partitions
8.2 Strategies to Reduce Spill
- Increase executor memory or reduce executor cores to give each task more memory.
- Increase spark.sql.shuffle.partitions to make each partition (and thus each task) smaller.
- Enable Adaptive Query Execution (AQE) to dynamically right-size shuffle partitions.
- Use broadcast joins to eliminate shuffle for small-large table joins.
- Repartition data before groupBy to distribute load more evenly.
- Pre-filter data aggressively before heavy operations to reduce the data volume each task sees.
# Strategy: repartition before heavy aggregation to reduce per-task datadf_repartitioned = df.repartition(500, 'customer_id')result = df_repartitioned.groupBy('customer_id').agg({'amount': 'sum'})# Strategy: filter early, aggregate lateresult = ( df .filter(df.year == 2024) # reduce data volume first .filter(df.status == 'completed') .groupBy('region') .agg({'revenue': 'sum', 'order_id': 'count'}))# Strategy: use AQE to auto-optimise shufflespark.conf.set('spark.sql.adaptive.enabled', 'true')spark.conf.set('spark.sql.adaptive.skewJoin.enabled', 'true')
9. Garbage Collection Tuning
Spark runs on the JVM and is subject to Java garbage collection pauses. For long-running jobs with large heaps, GC pauses can add minutes to job runtimes. Understanding and tuning the JVM GC is a key part of advanced Spark optimisation.
9.1 GC Types and When to Use Each
| GC Algorithm | Best For | Spark Use Case |
| G1GC (default Java 9+) | Large heaps (8GB+); balanced latency/throughput | Most Spark workloads — recommended default |
| Parallel GC | Maximum throughput; accepts longer pauses | Batch ETL jobs where latency is not critical |
| ZGC (Java 15+) | Ultra-low latency; very large heaps (100GB+) | Near-real-time streaming, interactive queries |
| Shenandoah (OpenJDK) | Low pause times; moderate throughput | Streaming jobs with strict SLA |
9.2 Configuring G1GC for Spark
# G1GC configuration for Spark executorsspark.conf.set('spark.executor.extraJavaOptions', '-XX:+UseG1GC ' '-XX:G1HeapRegionSize=16M ' '-XX:InitiatingHeapOccupancyPercent=35 ' '-XX:+G1PrintRegionLivenessInfo ' '-verbose:gc ' '-XX:+PrintGCDetails ' '-XX:+PrintGCDateStamps ' '-XX:OnOutOfMemoryError=kill -9 %p')# Driver GC optionsspark.conf.set('spark.driver.extraJavaOptions', '-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35')
9.3 Reducing GC Pressure with Kryo Serialisation
Java’s default serialiser creates many short-lived objects, generating heavy GC pressure. Kryo serialisation is 10x faster and creates fewer objects, significantly reducing GC overhead.
from pyspark.sql import SparkSessionspark = SparkSession.builder \ .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \ .config('spark.kryo.registrationRequired', 'false') \ .config('spark.kryo.unsafe', 'true') \ .getOrCreate()# Register frequently used classes for maximum Kryo benefit# (in Scala/Java — not directly from PySpark, but affects JVM-side data)# spark.conf.set('spark.kryo.classesToRegister', 'MyClass1,MyClass2')
10. Monitoring Memory Usage
You cannot tune what you cannot measure. Spark provides several tools for real-time and post-mortem memory monitoring.
10.1 Spark UI — Storage and Executor Tabs
- Storage Tab: shows all cached RDDs/DataFrames, their storage level, memory used, and disk used.
- Executors Tab: shows memory used vs available per executor, GC time, and disk spill per executor.
- Stages Tab: Spill (Memory) and Spill (Disk) columns show spill per stage/task.
- Environment Tab: shows all active Spark configuration values.
10.2 Programmatic Memory Inspection
# Check executor memory statussc = spark.sparkContext# Get executor info (Spark 3.x)exec_info = sc.statusTracker().getExecutorInfos()for info in exec_info: print(f'Executor {info.executorId()}: ' f'maxMemory={info.maxMemory() // (1024**2)} MB')# Get RDD storage infofor rdd_info in sc.statusTracker().getRDDInfos(): print(f'RDD {rdd_info.id()}: ' f'memSize={rdd_info.memSize() // (1024**2)} MB, ' f'diskSize={rdd_info.diskSize() // (1024**2)} MB')
10.3 Metrics via Spark Listener
# Access task-level metrics including spill# Run in a loop after job completion to check metricsdef check_stage_metrics(spark, job_id=None): """Print spill stats from the Spark status tracker.""" tracker = spark.sparkContext.statusTracker() active_jobs = tracker.getActiveJobIds() for job in active_jobs: stage_ids = tracker.getJobInfo(job).stageIds() for sid in stage_ids: info = tracker.getStageInfo(sid) if info: print(f'Stage {sid}: tasks={info.numCompletedTasks()}')# Better: use Spark History Server or integrate with Prometheus/Grafanaspark.conf.set('spark.metrics.conf.executor.source.jvm.class', 'org.apache.spark.metrics.source.JvmSource')
11. Common OOM Scenarios & Fixes
| OOM Scenario | Root Cause | Fix |
| Driver OOM | collect(), toPandas(), or show() on very large DataFrame pulls all data to driver | Avoid collect() on large data; use write() instead; increase spark.driver.memory |
| Executor OOM during shuffle | Per-task shuffle data exceeds execution memory | Increase shuffle partitions; enable AQE; increase executor memory |
| Executor OOM during broadcast | Broadcast variable too large for executor heap | Lower autoBroadcastJoinThreshold or disable; use sort-merge join instead |
| Python worker OOM | Pandas UDF processes too much data per batch | Reduce maxRecordsPerBatch; increase memoryOverhead; use chunked processing |
| OOM on window function | Entire partition loaded into execution memory | Repartition before windowing; add partition column to PARTITION BY |
| GC overhead limit exceeded | JVM spending >98% of time in GC — heap fragmented | Switch to G1GC; reduce object creation; use Kryo; increase heap |
| Cached data evicted unexpectedly | Execution memory pressure evicts storage memory | Reduce data cached; use DISK_ONLY level; increase memory.fraction |
12. Production Memory Tuning Checklist
Cluster Configuration
- Set executor.memory to 70-80% of node memory per executor (leave room for OS and overhead).
- Set executor.memoryOverhead to at least 15-20% of executor.memory for PySpark jobs.
- Use 4-5 cores per executor for optimal HDFS throughput and memory sharing.
- Set driver.memory appropriately — increase if collect() or large broadcasts are used.
Memory Fraction Tuning
- If jobs spill frequently: increase spark.memory.fraction (e.g. to 0.7 or 0.75) to give more to execution.
- If caching is critical: increase spark.memory.storageFraction (e.g. to 0.6).
- Enable off-heap for jobs with very large heaps (>16GB per executor) to reduce GC pause.
Shuffle & Partitioning
- Enable AQE: spark.sql.adaptive.enabled = true.
- Set shuffle.partitions to 2-3x the number of executor cores for typical workloads.
- Use broadcast joins for tables smaller than 100-200 MB.
- Repartition skewed data before heavy aggregations.
Caching
- Only cache DataFrames used 2+ times in the same job.
- Use MEMORY_AND_DISK_SER in production to prevent OOM from unexpected caching pressure.
- Always call unpersist() after the cached DataFrame is no longer needed.
- Monitor cache hit ratio in the Spark UI Storage tab.
GC & Serialisation
- Switch to KryoSerializer for all jobs using RDDs or custom serialised types.
- Configure G1GC with -XX:InitiatingHeapOccupancyPercent=35 for proactive collection.
- Add -XX:OnOutOfMemoryError=’kill -9 %p’ to auto-restart failed executors on OOM.
13. Complete Production SparkSession Configuration
from pyspark.sql import SparkSessionspark = ( SparkSession.builder .appName('ProductionETL') # ── Executor sizing ────────────────────────────────── .config('spark.executor.instances', '20') .config('spark.executor.cores', '5') .config('spark.executor.memory', '18g') .config('spark.executor.memoryOverhead', '3g') # ── Driver sizing ────────────────────────────────── .config('spark.driver.memory', '8g') .config('spark.driver.memoryOverhead', '2g') # ── Memory fractions ────────────────────────────── .config('spark.memory.fraction', '0.7') .config('spark.memory.storageFraction', '0.4') # ── Off-heap ────────────────────────────────────── .config('spark.memory.offHeap.enabled', 'true') .config('spark.memory.offHeap.size', '4g') # ── Shuffle & AQE ───────────────────────────────── .config('spark.sql.shuffle.partitions', '800') .config('spark.sql.adaptive.enabled', 'true') .config('spark.sql.adaptive.coalescePartitions.enabled','true') .config('spark.sql.adaptive.skewJoin.enabled', 'true') .config('spark.sql.autoBroadcastJoinThreshold', '100MB') # ── Serialisation ───────────────────────────────── .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') .config('spark.kryo.unsafe', 'true') # ── Arrow / PySpark ─────────────────────────────── .config('spark.sql.execution.arrow.pyspark.enabled', 'true') .config('spark.sql.execution.arrow.maxRecordsPerBatch', '50000') # ── GC tuning ───────────────────────────────────── .config('spark.executor.extraJavaOptions', '-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 ' '-XX:OnOutOfMemoryError=kill -9 %p') # ── Shuffle compression ─────────────────────────── .config('spark.shuffle.compress', 'true') .config('spark.shuffle.spill.compress', 'true') .config('spark.io.compression.codec', 'lz4') .getOrCreate())# Print effective config for verificationprint('Executor Memory:', spark.conf.get('spark.executor.memory'))print('Memory Fraction:', spark.conf.get('spark.memory.fraction'))print('Shuffle Partitions:',spark.conf.get('spark.sql.shuffle.partitions'))
14. Quick Reference — Memory Tuning Cheat Sheet
| Problem | Symptom | Solution |
| Executor OOM | java.lang.OutOfMemoryError on executor | Increase executor.memory or reduce executor.cores |
| Driver OOM | java.lang.OutOfMemoryError on driver | Increase driver.memory; avoid large collect() |
| Python worker OOM | Python process killed; task fails | Increase memoryOverhead; reduce Arrow batch size |
| Excessive spill | High disk I/O; slow stages | More shuffle partitions; enable AQE; add memory |
| High GC time | >20% GC time in executor tab | Use G1GC; enable Kryo; reduce object creation |
| Broadcast OOM | Broadcast variable fails | Reduce autoBroadcastJoinThreshold or disable it |
| Skewed join | One task takes 10x longer | Enable AQE skew join; salt skewed keys |
| Cache eviction loop | Cache repeatedly evicted and recomputed | Increase storage fraction or use DISK_ONLY level |
| Slow serialisation | High task serialisation time | Switch to KryoSerializer |
| Small file shuffle | Thousands of tiny output files | Enable AQE coalesce; increase advisory partition size |
15. Conclusion
PySpark memory management is a multi-layered discipline that spans JVM heap configuration, Python worker overhead, execution and storage memory balancing, shuffle tuning, spill avoidance, garbage collection optimisation, and real-time monitoring. No single parameter fix solves all problems — effective tuning requires understanding how the pieces interact.
The mental model to keep in mind: every byte of memory is shared between multiple competing consumers — execution tasks, cached data, shuffle buffers, Python workers, and the JVM itself. Your goal is to ensure each consumer gets enough memory at the right time, while minimising GC overhead and disk spills.
Start with proper executor sizing, enable Adaptive Query Execution, switch to Kryo serialisation, and monitor the Spark UI closely. Iterate based on what you observe. With each tuning cycle, your jobs will run faster, more reliably, and at lower infrastructure cost.
Discover more from DataSangyan
Subscribe to get the latest posts sent to your email.