Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Exploring the Big Data ecosystem and Spark’s role in contemporary data platforms
• Comprehending Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate approach for specific scenarios
• Creating and configuring SparkSession to establish a solid foundation for application settings

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data manipulation
• Integrating data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods relative to Spark driver and executor processes — a conceptual view for cluster users rather than administrators
• Tracing the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings to these resources
• Differentiating between controls managed via Spark session parameters versus those handled by the platform team during cluster provisioning — and understanding why this boundary is critical for tuning
• Hands-on exercise: Examine a live Spark-on-Kubernetes job within a sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing executor count versus executor size: evaluating memory-to-core trade-offs and reasoning through optimal configurations
• Managing driver and executor memory allocation, memory overhead, and how these factors translate into pod resource requirements
• Exploring dynamic allocation: understanding its behavior on Kubernetes, identifying when it conserves resources versus when it may not
• Hands-on exercise: Execute identical jobs under varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Assessing how adding or removing nodes impacts job behavior, completion times, and resource usage
• Comparing the performance and cost implications of using many small executors versus fewer large ones
• Refining shuffle behavior and partition tuning, including the estimation of cost impacts from configuration choices
• Hands-on exercise: Expand a sandbox cluster from five to ten nodes to observe the resulting effects on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Addressing the small-files problem: understanding why sources fragmented into many 1–5 MB Parquet files hinder performance and distort partitioning
• Implementing repartitioning and coalescing strategies for optimization
• Regulating partition sizes during both read and write operations
• Writing Parquet efficiently to prevent the recurrence of the small-files issue in downstream processes
• Hands-on exercise: Ingest a dataset composed of numerous small Parquet partitions, apply varying repartitioning strategies, and benchmark performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognizing early warning signs
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Utilizing chunked processing and dtype optimization for resource-constrained environments
• Hands-on exercise: Simulate a common Pandas out-of-memory scenario and resolve it through chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: analyzing performance characteristics, lazy evaluation, and memory behavior
• Understanding how Polars integrates with PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Reimplement a Pandas-heavy transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying principles of configuration, partitioning, and memory management across a realistic ETL pipeline
• Considering optimization factors specific to machine learning workloads running on the same cluster
• Executing a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and a basic model-training step, with participants independently tuning the Spark configuration