Use the Data Engineering Agent to build and modify data pipelines | BigQuery | Google Cloud Documentation
Skip to main content Console English Deutsch Español Español – América Latina Français Indonesia Italiano Português Português – Brasil עברית 中文 – 简体 中文 – 繁體 日本語 한국어 Sign in BigQuery Start free Home Documentation Data analytics BigQuery Guides Stay organized with collections Save and categorize content based on your preferences. Use the Data Engineering Agent to build and modify data pipelines The Data Engineering Agent in BigQuery and Dataform lets you build, modify, and manage data pipelines to load and process data in BigQuery.
You can use natural language prompts to generate data pipelines from various data sources or adapt existing data pipelines to suit your data engineering needs. The agent generates and organizes data pipeline code directly within Dataform repositories and operates in the Dataform workspace, so Dataform pipelines are automatically available to the agent. You can also interact with the agent using the Data Engineering Agent API , which uses the A2A protocol.
For examples of prompts that you can use, see Sample prompts . The Data Engineering Agent is powered by Gemini for Google Cloud and supports BigQuery AI and ML functions for building batch AI enrichment, semantic search, and machine learning pipelines in Dataform. For more information, see BigQuery AI and ML support . Limitations The Data Engineering Agent has the following limitations: The Data Engineering Agent doesn't support natural-language commands for the following file types: Notebooks Data preparation The Data Engineering Agent cannot execute pipelines.
You must review and run or schedule pipelines. The Data Engineering Agent cannot search any web links or URLs provided through instructions or direct prompts. When importing files in an agent instruction file , the @ import syntax supports only paths that begin with ./ , / , or a letter. The data preview feature is supported only for tables, declarations, or queries with the hasOutput flag set to true . The Data Engineering Agent is subject to the general limitations of AI technology .
When creating pipelines over Apache Iceberg external tables managed by the Lakehouse runtime catalog (formerly BigLake metastore), all Lakehouse runtime catalog limitations apply. Most notably, the agent cannot generate write mutations (such as INSERT , UPDATE , DELETE , or MERGE ) or DDL statements (such as CREATE TABLE or DROP TABLE ) on Iceberg tables. For more information, see Apache Iceberg REST catalog endpoint concepts .
Before you begin Before you use the Data Engineering Agent, perform the steps in this section. Enable Gemini in BigQuery Make sure that Gemini in BigQuery is enabled for your Google Cloud project. For more information, see Set up Gemini in BigQuery . Enable the required APIs Console Enable the following APIs in the Google Cloud console for the Google Cloud project that you use with the Conversational Analytics API, if they aren't already enabled.
Enable the Gemini Data Analytics API Enable the Gemini for Google Cloud API Enable the BigQuery API To enable agent observability to monitor token usage and model latency , enable the following APIs: Enable the Cloud Monitoring API Enable the Cloud Trace API gcloud To enable the Gemini Data Analytics API, the Gemini for Google Cloud API, and the BigQuery API, if they aren't already enabled, use the Google Cloud CLI and run the following gcloud services enable commands: gcloud services enable geminidataanalytics.googleapis.com --project= PROJECT_ID gcloud services enable cloudaicompanion.googleapis.com --project= PROJECT_ID gcloud services enable bigquery.googleapis.com --project= PROJECT_ID To enable agent observability to monitor token usage and model latency , enable the following APIs: gcloud services enable monitoring.googleapis.com --project= PROJECT_ID gcloud services enable cloudtrace.googleapis.com --project= PROJECT_ID Replace PROJECT_ID with your Google Cloud project ID.
Required roles To get the permission that you need to use the Data Engineering Agent, ask your administrator to grant you the following IAM roles on the project: Dataform Code Editor ( roles/dataform.codeEditor ) BigQuery Job User ( roles/bigquery.jobUser ) Gemini Data Analytics Stateless Chat User ( roles/geminidataanalytics.dataAgentStatelessUser ) For more information about granting roles, see Manage access to projects, folders, and organizations .
This predefined role contains the geminidataanalytics.locations.useDataEngineeringAgent permission, which is required to use the Data Engineering Agent. You might also be able to get this permission with custom roles or other predefined roles . For details on the required roles to query Apache Iceberg tables, see Required roles for Lakehouse Apache Iceberg support . Knowledge Catalog integration prerequisites To get the permission that you need to integrate the Data Engineering Agent with Knowledge Catalog, ask your administrator to grant you the Dataplex Catalog Editor ( roles/dataplex.catalogEditor ) IAM role on the project.
For more information about granting roles, see Manage access to projects, folders, and organizations . This predefined role contains the geminidataanalytics.locations.useDataEngineeringAgent permission, which is required to integrate the Data Engineering Agent with Knowledge Catalog. You might also be able to get this permission with custom roles or other predefined roles . If you use a custom service account to run pipelines, you must grant the Dataplex Catalog Editor ( roles/dataplex.catalogEditor ) role on the project or @bigquery entry group.
Additionally, you must enable the Dataplex API . Agent monitoring prerequisites To enable agent monitoring, verify that you have the required permissions to configure observability settings, manage Monitoring dashboards, and view Trace data. For more information, see Monitor agents and conversations . Encrypt data with Cloud Key Management Service keys You can encrypt data at the dataset or project level with the default customer-managed Cloud Key Management Service keys in BigQuery.
For more information, see Set a dataset default key and Set a project default key . You can encrypt pipeline code at the project level by setting a default Dataform Cloud Key Management Service key . Configure VPC Service Controls perimeters If you use VPC Service Controls, you must configure the perimeter to protect Dataform, BigQuery, and the Conversational Analytics API. For more information, see Dataform , BigQuery , and Conversational Analytics API .
Generate a data pipeline with the Data Engineering Agent To use the Data Engineering Agent in BigQuery, select one of the following options: BigQuery pipelines You can use the Data Engineering Agent in the BigQuery pipelines interface by doing the following: Go to the BigQuery page. Go to BigQuery In the query editor, click arrow_drop_down Create new > Pipeline . In the pipeline interface, click Ask agent . In the Ask agent field, enter a natural language prompt to generate a data pipeline—for example: Create dimension tables for a taxi trips star schema from new_york_taxi_trips.tlc_green_trips_2022.
Generate surrogate keys and all the descriptive attributes. After you enter a prompt, click Send . The Data Engineering Agent generates a data pipeline based on your prompt. The Data Engineering Agent generates a proposed draft of a data pipeline. You can click a pipeline node to review the generated SQLX query. To apply the agent-suggested data pipeline, click Apply . Dataform You can use the Data Engineering Agent in Dataform by doing the following: Go to Dataform .
Go to Dataform Select a repository. Select or create a development workspace. In the workspace, click Ask Agent . In the Ask agent prompt that appears, enter a natural language prompt to generate a data pipeline—for example: Create dimension tables for a taxi trips star schema from new_york_taxi_trips.tlc_green_trips_2022. Generate surrog
The rest of this page is on the live site.