[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-ec2478ed-c56d-452a-ab64-8fbaa5058d7e":3,"$fhJN0x-b3CWBeTA0b7sy56fJGeUeJYvsX9dp6-eOMHC4":43},{"id":4,"title":5,"description":6,"categoryId":7,"moduleId":8,"tags":9,"prompt":10,"icon":11,"source":12,"sourceUrl":13,"authorId":14,"authorName":15,"isPublic":16,"stars":17,"runs":18,"createdAt":19,"updatedAt":19,"module":20,"category":27,"packages":34},"ec2478ed-c56d-452a-ab64-8fbaa5058d7e","spark-optimization","优化Apache Spark作业的分区、缓存、shuffle优化和内存调整。用于提高Spark性能、调试慢速作业或扩展数据处理管道。","cat_life_career","mod_other","sickn33,other","---\nname: spark-optimization\ndescription: \"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\"\nrisk: unknown\nsource: community\ndate_added: \"2026-02-27\"\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## Do not use this skill when\n\n- The task is unrelated to apache spark optimization\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Use this skill when\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n\n## Limitations\n- Use this skill only when the task clearly matches the scope described above.\n- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.\n- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.\n","","imported","https:\u002F\u002Fgithub.com\u002Fsickn33\u002Fantigravity-awesome-skills","user_system_seed","SkillOPIC",true,87,954,"2026-05-16 13:41:31",{"id":8,"name":21,"slug":22,"icon":23,"description":24,"sort":25,"createdAt":26},"其他","other","mdi-page-next-outline","其他类型Skill",5,"2026-05-16 12:53:40",{"id":7,"name":28,"slug":29,"icon":30,"description":31,"moduleId":8,"sort":32,"skillCount":33,"createdAt":26},"职场发展","career","mdi-briefcase-outline","面试准备、简历优化、职业规划",4,575,[35],{"id":36,"skillId":4,"version":37,"fileName":38,"fileSize":39,"filePath":40,"fileHash":41,"manifest":42,"createdAt":19},"b668040d-36dd-4819-9e85-3bdcc6c430ea","1.0.0","spark-optimization.zip",5079,"uploads\u002Fskills\u002Fec2478ed-c56d-452a-ab64-8fbaa5058d7e\u002Fspark-optimization.zip","80977c0f3ee0dc507c063ca0d4b81e5193fbd8dbc5baafc66856e3a675afeb76","[{\"path\":\"SKILL.md\",\"isDirectory\":false,\"size\":13657}]",{"code":44,"message":45,"data":46},200,"success",{"items":47,"stats":48,"page":51},[],{"averageRating":49,"totalRatings":49,"ratingCounts":50},0,[49,49,49,49,49],{"limit":52,"offset":49,"hasMore":53,"nextOffset":52,"ratedOnly":16},15,false]