# gcp-dataflow

> Claude/Codex skill by [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars · Updated 2026-10-23

Guides writing, packaging, executing, and troubleshooting Apache Beam pipelines on Dataflow. Use when creating new pipelines, configuring Flex Templates, or analyzing performance of Dataflow jobs. Capabilities include Java/Python/Go setup, Cloud Build integration, and deep diagnostic analysis of job health and autoscaling. Use when: - Creating an Apache Beam Dataflow pipeline. - Creating a Google Dataflow Flex Template. - Using an existing Google Dataflow Template. - Debugging Dataflow pipeline - Troubleshooting Dataflow pipeline - Analyzing Performance of Dataflow pipeline. Key capabilities: Java/Python/Go project setup, Flex Templates (with Cloud Build), and diagnostics for streaming job health, bottlenecks, and autoscaling. Do NOT use for: - General GCP resource management unrelated to Dataflow. - Issues with other GCP services (e.g., GCE, GCS, BigQuery) unless directly impacting Dataflow pipeline execution. - Pipeline technologies other than Apache Beam on Dataflow.

## Install

```sh
git clone https://github.com/GoogleCloudPlatform/data-agent-kit-plugin /tmp/data-agent-kit-plugin && ln -s /tmp/data-agent-kit-plugin/skills/gcp-dataflow ~/.claude/skills/gcp-dataflow
```

## From README

Apache Beam Pipelines on Cloud Dataflow Pipeline authoring Use this section when implementing Dataflow pipeline logic using Apache Beam. Check if existing Google Dataflow Template exists Google provides a variety of pre-built, open source Dataflow templates that can be used for common scenarios. Before implementing a pipeline from scratch, you MUST follow the steps below to check whether a Dataflow template for the pipeline logic you need to implement already exists. Step 1: Check for a matching Google Dataflow Template Identify the source and sink (e.g., GCS to BigQuery) from the user's request. Note: You MUST NOT proceed until the source and sink are clearly identified. Action: List templates in the public dataflow-templates bucket: For Classic templates, check gs://dataflow-templates/latest. For Flex templates, check gs://dataflow-templates/latest/flex. Use gcloud storage ls to list the contents. Match templates by name or description to the source and sink.

## Links

- HTML page: https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/gcp-dataflow/
- GitHub: https://github.com/GoogleCloudPlatform/data-agent-kit-plugin
- Author: [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md)

## More from this repo

- [accidental-data-loss-prevention](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/accidental-data-loss-prevention/index.md) — \*\*STOP AND VERIFY\*\*: Before running any command or tool that results in irreversible data loss, you MUST obtain...
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars
- [bigquery-ai-ml](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/bigquery-ai-ml/index.md) — Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics.
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars
- [bigquery-bigframes](https://omgskills.com/library/googlecloudplatform/data-agent-kit-plugin/skills/bigquery-bigframes/index.md) — Generates Python code using BigQuery DataFrames (BigFrames).
   - [@googlecloudplatform](https://omgskills.com/library/googlecloudplatform/index.md) · 226 stars