gcp-dataflow
Guides writing, packaging, executing, and troubleshooting Apache Beam pipelines on Dataflow. Use when creating new pipelines, configuring Flex Templates, or analyzing performance of Dataflow jobs. Capabilities include Java/Python/Go setup, Cloud Build integration, and deep diagnostic analysis of job health and autoscaling. Use when: - Creating an Apache Beam Dataflow pipeline. - Creating a Google Dataflow Flex Template. - Using an existing Google Dataflow Template. - Debugging Dataflow pipeline - Troubleshooting Dataflow pipeline - Analyzing Performance of Dataflow pipeline. Key capabilities: Java/Python/Go project setup, Flex Templates (with Cloud Build), and diagnostics for streaming job health, bottlenecks, and autoscaling. Do NOT use for: - General GCP resource management unrelated to Dataflow. - Issues with other GCP services (e.g., GCE, GCS, BigQuery) unless directly impacting Dataflow pipeline execution. - Pipeline technologies other than Apache Beam on Dataflow.
Install
With your agent
Install OMGSkills skill "googlecloudplatform/data-agent-kit-plugin:skills/gcp-dataflow" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/GoogleCloudPlatform/data-agent-kit-plugin /tmp/data-agent-kit-plugin && ln -s /tmp/data-agent-kit-plugin/skills/gcp-dataflow ~/.claude/skills/gcp-dataflowFrom README
Apache Beam Pipelines on Cloud Dataflow Pipeline authoring Use this section when implementing Dataflow pipeline logic using Apache Beam. Check if existing Google Dataflow Template exists Google provides a variety of pre-built, open source Dataflow templates that can be used for common scenarios. Before implementing a pipeline from scratch, you MUST follow the steps below to check whether a Dataflow template for the pipeline logic you need to implement already exists. Step 1: Check for a matching Google Dataflow Template Identify the source and sink (e.g., GCS to BigQuery) from the user's request. Note: You MUST NOT proceed until the source and sink are clearly identified. Action: List templates in the public dataflow-templates bucket: For Classic templates, check gs://dataflow-templates/latest. For Flex templates, check gs://dataflow-templates/latest/flex. Use gcloud storage ls to list the contents. Match templates by name or description to the source and sink.
More from this repo
accidental-data-loss-prevention
**STOP AND VERIFY**: Before running any command or tool that results in irreversible data loss, you MUST obtain...
bigquery-ai-ml
Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics.
bigquery-bigframes
Generates Python code using BigQuery DataFrames (BigFrames).