spark-best-practices

From README

Spark Best Practices Use this skill for general Apache Spark guidance when optimizing performance, reliability, and maintainability. 1) Keep execution distributed Avoid collect(), toPandas(), and large take() in core data paths. Materialize to driver memory only for very small control outputs (metrics, IDs, summaries). Keep heavy transformation and write paths in Spark DataFrame execution. 2) Prefer DataFrame APIs to Python loops Use Spark SQL/DataFrame functions so Catalyst can optimize execution plans. Avoid row-by-row Python logic when equivalent DataFrame expressions exist. Keep transformations declarative and composable. 3) Reduce shuffle cost Project and filter early to reduce data volume before joins/aggregations. Repartition intentionally before heavy joins/writes. Use coalesce when reducing output partitions. Watch for skewed keys and apply skew mitigation. 4) Use efficient joins Broadcast small dimension tables when appropriate.