# spark-lineage

> Claude/Codex skill by [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars · Updated 2026-08-10

Preserves connected OpenLineage for oleander Spark jobs by avoiding collect()/toPandas() between read and write, and using env vars for runtime config. Use when lineage looks disconnected, jobs split after collect(), or rewriting Spark pipelines for continuous lineage.

## Install

```sh
git clone https://github.com/OleanderHQ/claude-plugin /tmp/claude-plugin && ln -s /tmp/claude-plugin/skills/spark-lineage ~/.claude/skills/spark-lineage
```

## From README

oleander Spark Lineage Use this skill when Spark lineage in oleander looks disconnected or when rewriting Spark jobs to preserve connected lineage. Lineage context Key behavior from oleander Spark/OpenLineage integrations: collect() is a Spark action that materializes data into driver memory. After collect(), execution is regular Python in-memory logic, not distributed Spark DataFrame execution. Spark can treat "read + collect" and "write from memory" as separate jobs. The OpenLineage Spark integration may not connect those phases as one continuous lineage path. Recommended lineage-safe pattern Prefer this shape for connected lineage: df = spark.table(...) chain DataFrame transforms (select, withColumn, join, aggregate) finalize with df.write(...) If collect() is required, keep it for small side-effects or reporting, not as the core bridge between read and write. Rewrite checklist When fixing lineage gaps: Find points where data leaves Spark (collect, toPandas, driver loops).

## Links

- HTML page: https://omgskills.com/library/oleanderhq/claude-plugin/skills/spark-lineage/
- GitHub: https://github.com/OleanderHQ/claude-plugin
- Author: [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md)

## More from this repo

- [lake-catalog](https://omgskills.com/library/oleanderhq/claude-plugin/skills/lake-catalog/index.md) — Engine-agnostic oleander lake catalog conventions: catalog.namespace.table naming, hierarchy, and catalog-qualified...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars
- [lake-query](https://omgskills.com/library/oleanderhq/claude-plugin/skills/lake-query/index.md) — Runs lake SQL through oleander's query router: query\_run for reads, query\_submit for writes, spark\_sql\_submit for named...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars
- [polars-submit](https://omgskills.com/library/oleanderhq/claude-plugin/skills/polars-submit/index.md) — Runs Polars queries or scripts on oleander via the CLI, in local or distributed mode, and saves results to the lake...
   - [@oleanderhq](https://omgskills.com/library/oleanderhq/index.md) · 4 stars