gcp-spark
Develops, optimizes and runs PySpark/Spark code on Managed Spark (Dataproc clusters and Serverless) on Google Cloud. Use when: - Authoring or running Spark/PySpark notebooks (any kernel), incl. data analysis, reports and visualizations. - Writing Spark ETL pipelines or preparing workloads for production. - Training or running inference with ML models on Spark. - Optimizing PySpark or Spark SQL for performance, memory, or OOM risks. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python that doesn't use Spark. - Performing simple SQL queries that can be done directly in BigQuery. - Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting).
Install
With your agent
Install OMGSkills skill "googlecloudplatform/data-agent-kit-plugin:skills/gcp-spark" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/GoogleCloudPlatform/data-agent-kit-plugin /tmp/data-agent-kit-plugin && ln -s /tmp/data-agent-kit-plugin/skills/gcp-spark ~/.claude/skills/gcp-sparkFrom README
Managed Spark on Google Cloud [!IMPORTANT] You MUST follow the Task Execution Workflow when writing spark code. Notebook Generation Environment Rules [!CAUTION] When the task is to generate a Spark notebook (as opposed to executing one), you MUST NOT modify your own local execution environment (i.e. do not run these commands with your shell or other tools). Specifically: NEVER create a virtual environment (python -m venv, virtualenv, conda create, uv venv, poetry init/ , etc.). NEVER packages (pip , %pip , uv pip , conda , poetry add). NEVER run environment discovery loops such as activating a venv, pip list, or probing for ipykernel / google-cloud-spark-connect as a precondition for writing the notebook. Assume the user will select the appropriate kernel and dependencies after the notebook is generated.
More from this repo
accidental-data-loss-prevention
**STOP AND VERIFY**: Before running any command or tool that results in irreversible data loss, you MUST obtain...
bigquery-ai-ml
Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics.
bigquery-bigframes
Generates Python code using BigQuery DataFrames (BigFrames).