spark-lineage

From README

oleander Spark Lineage Use this skill when Spark lineage in oleander looks disconnected or when rewriting Spark jobs to preserve connected lineage. Lineage context Key behavior from oleander Spark/OpenLineage integrations: collect() is a Spark action that materializes data into driver memory. After collect(), execution is regular Python in-memory logic, not distributed Spark DataFrame execution. Spark can treat "read + collect" and "write from memory" as separate jobs. The OpenLineage Spark integration may not connect those phases as one continuous lineage path. Recommended lineage-safe pattern Prefer this shape for connected lineage: df = spark.table(...) chain DataFrame transforms (select, withColumn, join, aggregate) finalize with df.write(...) If collect() is required, keep it for small side-effects or reporting, not as the core bridge between read and write. Rewrite checklist When fixing lineage gaps: Find points where data leaves Spark (collect, toPandas, driver loops).