Google BigQuery ExecModule

Overview

GoogleBigQueryModule connects ValkyrAI workflows to the Google BigQuery REST API through the native map I/O ExecModule ABI. It covers dataset and table discovery, GoogleSQL queries, paged query results, streaming inserts, and guarded table deletion without exposing OAuth credentials to workflow data.

The connector constructs only the fixed bigquery.googleapis.com API host. It retries bounded GET reads, but query submissions, streaming inserts, and deletes receive one attempt because a lost response can make their outcome ambiguous.

Usage

1. Enable the BigQuery API in the target Google Cloud project.
2. Create a dedicated service identity or OAuth principal with only the required dataset and job permissions.
3. Store the Google Cloud project ID in IntegrationAccount.accountId and a short-lived OAuth access token in the encrypted apiKey SecureField.
4. Set the account to READY and bind it through ExecModuleConfig.authConfig.integrationAccount.
5. Use discovery operations before targeting a dataset or table, and use query parameters for values.
6. Set confirmWrite: true for non-read SQL or streaming inserts. Table deletion separately requires confirmDelete: true.

Workflow input cannot supply a credential, authorization header, arbitrary API host, legacy SQL mode, or multiple statements.

Inputs

Name	Type	Requirement	Default	Description and constraints
operation	string	Required	None	One of the seven documented operations.
projectId	string	Optional	IntegrationAccount.accountId	Google Cloud project ID. A configured override is validated.
datasetId	string	Table operations	None	BigQuery dataset identifier.
tableId	string	Single-table operations	None	BigQuery table identifier.
query	string	run_query	None	One GoogleSQL statement, at most 100,000 characters. Multiple statements are rejected.
queryParameters	array	Optional query binding	None	Up to 1,000 BigQuery {name?,parameterType,parameterValue} objects, capped at 256 KiB.
parameterMode	string	With parameters	NAMED	NAMED or POSITIONAL.
rows	array	insert_rows	None	1–500 non-empty JSON objects, with a total request cap of 1 MiB.
jobId	string	get_query_results	None	BigQuery query job ID.
location	string	Regional query jobs	None	Valid BigQuery location, such as US, EU, or us-west2.
pageToken	string	Optional continuation	None	Opaque provider token, maximum 4,096 characters.
maxResults	integer	Optional	1000	Maximum normalized resources or rows, 1–10,000.
timeoutMs	integer	Optional	30000	Provider/query and HTTP timeout, 100–300,000 ms.
requestId	UUID string	Streaming inserts	None	Caller-owned UUID used to derive stable per-row insertId values.
dryRun	boolean	run_query	false	Validates GoogleSQL and estimates bytes without running it.
skipInvalidRows	boolean	Streaming insert	false	Allows valid rows to commit when another row fails.
ignoreUnknownValues	boolean	Streaming insert	false	Ignores input fields missing from the table schema.
confirmWrite	boolean	Mutation guard	false	Required for non-read SQL and streaming inserts.
confirmDelete	boolean	Destructive guard	false	Required for delete_table.

Outputs

Name	Type	When present	Description
status	string	Always	success or error.
operation	string	Always	Normalized operation.
httpStatus / attempts	integer	Provider request	HTTP status and attempts consumed.
items / count	array / integer	Lists and query rows	Bounded resources or rows mapped by returned schema field name.
data	object	Resource or receipt	Provider table resource, insert receipt, query dry-run receipt, or other non-row response.
jobId / jobComplete	string / boolean	Query response	Query continuation identity and completion state.
nextPageToken	string	More results	Opaque provider continuation token.
totalRows / cacheHit	string / boolean	Query response	Provider row count and cache status.
error	object	Failure	Safe {code,message,httpStatus?,retryable} details.

OAuth tokens and authorization values are redacted from outputs, errors, logs, and workflow events.

IntegrationAccount Requirements

Setting	Requirement
Provider	Google Cloud principal with BigQuery API access
accountName	Human-readable automation identity
accountId	Google Cloud project ID, for example valkyrlabs-analytics
apiKey	Encrypted SecureField containing a short-lived OAuth access token
status	Exactly READY

For discovery, grant only metadata/list permissions. Query execution typically needs bigquery.jobs.create plus read access to the selected datasets. Streaming inserts need table update-data permission. Table deletion needs table delete permission and should use a separate, narrowly scoped identity when practical. Token issuance and rotation remain IntegrationAccount lifecycle responsibilities.

Configuration

{
  "version": "1.0.0",
  "authConfig": {
    "authStrategy": 1,
    "integrationAccount": "integration-account:bigquery-analytics"
  },
  "retryPolicy": {
    "maxAttempts": 3,
    "backoffStrategy": "EXPONENTIAL",
    "initialDelayMs": 1000,
    "maxDelayMs": 60000
  },
  "payloadConfig": {
    "parameters": "{\"operation\":\"run_query\",\"location\":\"US\",\"maxResults\":100}"
  }
}

The integration-account reference is symbolic. Persisted workflows use the generated relationship and never plaintext access tokens.

Operations

Operation	BigQuery behavior	Side effect
list_datasets	Lists all visible datasets in the selected project.	Read-only; bounded retry.
list_tables	Lists tables in one dataset.	Read-only; bounded retry.
get_table	Reads one table resource and schema.	Read-only; bounded retry.
run_query	Submits one Standard SQL query with optional typed parameters.	Read query or confirmed SQL mutation; single attempt.
get_query_results	Reads one page for a query job, optionally in a regional location.	Read-only; bounded retry.
insert_rows	Sends up to 500 JSON rows through tabledata.insertAll with deterministic insert IDs.	Write; confirmed, single attempt.
delete_table	Deletes one dataset table.	Destructive; confirmed, single attempt.

Errors and Failure Modes

Code	Typical cause	Retryable	Resolution
VALIDATION_ERROR	Missing resource, invalid project/identifier/token/parameter, multiple SQL statements, absent confirmation, or oversized payload.	No	Correct the named input; no request was sent.
UNSUPPORTED_OPERATION	Unknown operation.	No	Select a documented operation.
INTEGRATION_ACCOUNT_REQUIRED	No bound account.	No	Bind a Google BigQuery IntegrationAccount.
INTEGRATION_ACCOUNT_NOT_READY	Account is not READY.	No	Repair or reconnect the account.
BIGQUERY_AUTHERROR / BIGQUERY_HTTP_401	Expired or invalid OAuth access token.	No	Refresh the IntegrationAccount token.
BIGQUERY_ACCESSDENIED / BIGQUERY_HTTP_403	The identity lacks project, dataset, job, table, or row permission.	No	Grant only the missing minimum permission.
BIGQUERY_NOTFOUND / BIGQUERY_HTTP_404	Project, dataset, table, job, or location is wrong.	No	Verify the exact resource and job location.
BIGQUERY_RATELIMITEXCEEDED / BIGQUERY_HTTP_429	Provider quota or concurrency limit.	GET reads only	Honor backoff; reconcile write/query state before manual retry.
BIGQUERY_HTTP_5xx	Transient provider failure.	GET reads only	Retry reads; do not blindly repeat an ambiguous POST or DELETE.
NETWORK_ERROR	Timeout, DNS, TLS, or connectivity failure.	GET reads only	Verify connectivity and reconcile job, insert, or table state.
RESPONSE_TOO_LARGE	Response exceeded 10 MiB.	No	Reduce maxResults, narrow the query, or request another page.

BigQuery may return successful HTTP status with row-level insertErrors; these remain in data and must be treated as partial failure when skipInvalidRows is enabled.

Example

Run a parameterized lead query:

{
  "operation": "run_query",
  "query": "SELECT id, name FROM `analytics.leads` WHERE score >= @score ORDER BY score DESC LIMIT 100",
  "queryParameters": [
    {
      "name": "score",
      "parameterType": {"type": "INT64"},
      "parameterValue": {"value": "80"}
    }
  ],
  "parameterMode": "NAMED",
  "location": "US",
  "maxResults": 100
}

Expected result:

{
  "status": "success",
  "operation": "run_query",
  "httpStatus": 200,
  "jobId": "job_analytics_20260810",
  "jobComplete": true,
  "count": 2,
  "items": [
    {"id": "42", "name": "Northstar"},
    {"id": "77", "name": "Valkyr"}
  ],
  "attempts": 1
}

Notes

• Pagination: dataset/table lists and query results return nextPageToken. Pass it back unchanged with the same project, resource, job, location, and page size. The connector retrieves one bounded page per invocation.
• Rate limits: only GET reads retry HTTP 408, 429, 500, 502, 503, and 504 with bounded exponential backoff and Retry-After support. POST query/insert and DELETE table calls always receive one attempt.
• API limits: requests are capped at 1 MiB, responses at 10 MiB, inserts at 500 rows, query parameters at 1,000 entries/256 KiB, and normalized output at 10,000 items.
• Idempotency: streaming inserts require a caller UUID and derive insertId as <requestId>:<row-index>. BigQuery offers best-effort insert deduplication, not a transaction receipt; reconcile table data before repeating an ambiguous call.
• Destructive behavior: delete_table is irreversible through this module and requires confirmDelete: true. Dataset deletion, model/routine deletion, partition expiration changes, IAM changes, load/copy/extract jobs, and job cancellation are intentionally deferred.
• SQL safety: SELECT, WITH, and EXPLAIN run without a mutation confirmation; every other prefix requires confirmWrite. This is a guard, not a SQL parser or substitute for provider IAM, authorized views, row-level security, policy tags, or reservations.
• Authentication: the module accepts OAuth bearer tokens only. Service-account private keys, refresh tokens, and arbitrary API bases are intentionally unsupported in workflow payloads.
• API behavior: nested and repeated query cells remain provider-shaped JSON inside the normalized field value. Consumers needing typed nested expansion should apply a downstream mapper.
• External verification: local tests verify validation, fixed-host routing, OAuth headers, query parameters, pagination, row normalization, retries, mutation guards, insert IDs, redaction, and metadata discovery. Live Google Cloud IAM, billing, quotas, locations, reservations, table schemas, streaming consistency, and SQL execution require separately authorized provider credentials and are not exercised in repository tests.
• Functional references: n8n Google BigQuery integration, BigQuery REST v2, parameterized queries, and streaming inserts.