# gcp-spark

> Claude/Codex skill by [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars · Updated 2026-10-23

Develops, optimizes and runs PySpark/Spark code on Managed Spark (Dataproc clusters and Serverless) on Google Cloud. Use when: - Authoring or running Spark/PySpark notebooks (any kernel), incl. data analysis, reports and visualizations. - Writing Spark ETL pipelines or preparing workloads for production. - Training or running inference with ML models on Spark. - Optimizing PySpark or Spark SQL for performance, memory, or OOM risks. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python that doesn't use Spark. - Performing simple SQL queries that can be done directly in BigQuery. - Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting).

## Install

```sh
git clone https://github.com/GoogleCloudPlatform/data-agent-kit-plugin /tmp/data-agent-kit-plugin && ln -s /tmp/data-agent-kit-plugin/skills/gcp-spark ~/.claude/skills/gcp-spark
```

## From README

Managed Spark on Google Cloud \[!IMPORTANT\] You MUST follow the Task Execution Workflow when writing spark code. Notebook Generation Environment Rules \[!CAUTION\] When the task is to generate a Spark notebook (as opposed to executing one), you MUST NOT modify your own local execution environment (i.e. do not run these commands with your shell or other tools). Specifically: NEVER create a virtual environment (python -m venv, virtualenv, conda create, uv venv, poetry init/ , etc.). NEVER packages (pip , %pip , uv pip , conda , poetry add). NEVER run environment discovery loops such as activating a venv, pip list, or probing for ipykernel / google-cloud-spark-connect as a precondition for writing the notebook. Assume the user will select the appropriate kernel and dependencies after the notebook is generated.

## Links

- HTML page: https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/gcp-spark/
- GitHub: https://github.com/GoogleCloudPlatform/data-agent-kit-plugin
- Author: [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md)

## More from this repo

- [accidental-data-loss-prevention](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/accidental-data-loss-prevention/index.md) — \*\*STOP AND VERIFY\*\*: Before running any command or tool that results in irreversible data loss, you MUST obtain...
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars
- [bigquery-ai-ml](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/bigquery-ai-ml/index.md) — Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics.
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars
- [bigquery-bigframes](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/bigquery-bigframes/index.md) — Generates Python code using BigQuery DataFrames (BigFrames).
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars