# spark-best-practices

> Claude/Codex skill by [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars · Updated 2026-08-10

General Apache Spark best practices for scalable, maintainable DataFrame jobs: avoid driver materialization, reduce shuffle, join efficiently, and cache carefully. Use when optimizing Spark performance, reviewing PySpark jobs, or writing new DataFrame pipelines.

## Install

```sh
git clone https://github.com/OleanderHQ/claude-plugin /tmp/claude-plugin && ln -s /tmp/claude-plugin/skills/spark-best-practices ~/.claude/skills/spark-best-practices
```

## From README

Spark Best Practices Use this skill for general Apache Spark guidance when optimizing performance, reliability, and maintainability. 1) Keep execution distributed Avoid collect(), toPandas(), and large take() in core data paths. Materialize to driver memory only for very small control outputs (metrics, IDs, summaries). Keep heavy transformation and write paths in Spark DataFrame execution. 2) Prefer DataFrame APIs to Python loops Use Spark SQL/DataFrame functions so Catalyst can optimize execution plans. Avoid row-by-row Python logic when equivalent DataFrame expressions exist. Keep transformations declarative and composable. 3) Reduce shuffle cost Project and filter early to reduce data volume before joins/aggregations. Repartition intentionally before heavy joins/writes. Use coalesce when reducing output partitions. Watch for skewed keys and apply skew mitigation. 4) Use efficient joins Broadcast small dimension tables when appropriate.

## Links

- HTML page: https://omgskills.com/library/oleanderhq/claude-plugin/skills/spark-best-practices/
- GitHub: https://github.com/OleanderHQ/claude-plugin
- Author: [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md)

## More from this repo

- [lake-catalog](https://omgskills.com/library/oleanderhq/claude-plugin/skills/lake-catalog/index.md) — Engine-agnostic oleander lake catalog conventions: catalog.namespace.table naming, hierarchy, and catalog-qualified...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars
- [lake-query](https://omgskills.com/library/oleanderhq/claude-plugin/skills/lake-query/index.md) — Runs lake SQL through oleander's query router: query\_run for reads, query\_submit for writes, spark\_sql\_submit for named...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars
- [polars-submit](https://omgskills.com/library/oleanderhq/claude-plugin/skills/polars-submit/index.md) — Runs Polars queries or scripts on oleander via the CLI, in local or distributed mode, and saves results to the lake...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars