Elasticsearch ExecModule

Overview

ElasticSearchModule connects a ValkyrAI workflow to one Elasticsearch-compatible HTTPS origin. Version 2.0 replaces the legacy in-memory simulation with real bounded REST requests, complete Workflow Studio metadata, IntegrationAccount-only credentials, normalized outputs, and credential-safe failures.

The module is registered as:

com.valkyrlabs.workflow.modules.integration.ElasticSearchModule

It supports search, get, index, update, delete, and structured bulk operations. It does not expose arbitrary paths or HTTP methods, provider URLs, raw credentials, scripted updates, delete-by-query, index administration, scroll contexts, redirects, mock results, or automatic write retries.

Usage

1. Create an Elasticsearch IntegrationAccount whose accountId is the cluster's HTTPS origin.
2. Store either an Elasticsearch API key or a username/password pair in IntegrationAccount SecureFields and mark the account READY.
3. Bind the account through elasticsearchAccount.
4. Select an operation, index, and its operation-specific inputs.
5. Set requireConfirmation: true for every write.
6. Execute the workflow and inspect status, attempts, the normalized result, and error.

The connector disables redirects and transport-level automatic retries. ValkyrAI itself may retry only search and get after a transient transport failure or HTTP 429, 502, 503, or 504, up to maxReadRetries. Writes send at most one provider request.

Inputs

Input	Required	Description
operation	Yes	search, get, index, update, delete, or bulk.
index	Yes	Lowercase index or data-stream name matching ^[a-z0-9][a-z0-9._-]{0,254}$.
documentId	By operation	Caller-owned ID required for get, index, update, and delete; at most 512 characters.
document	For index and update	JSON object. update treats it as the partial document under Elasticsearch doc. Maximum serialized size: 1 MiB.
query	No	Elasticsearch Query DSL object. search defaults to match_all.
aggregations	No	Elasticsearch aggregations object included under aggs.
sort	No	Array of 1–10 Elasticsearch sort entries. Use a stable tie-breaker such as _id.
searchAfter	No	Opaque values from a prior nextSearchAfter; requires sort.
pageSize	No	Search page size from 1 through 100. Default: 25.
bulkActions	For bulk	Array of 1–500 structured actions. Each requires action (index, update, or delete) and documentId; index/update also require document. An action may override the top-level index.
requireConfirmation	For writes	Must be exactly true for index, update, delete, and bulk.

Raw username, password, apiKey, api_key, token, secret, auth, host, hosts, url, endpoint, elasticsearchUrl, mock, and test fields are rejected in workflow input and module configuration.

Outputs

Output	Description
status	success or error.
operation	Normalized operation.
index	Validated top-level index.
httpStatus	Elasticsearch HTTP status when a response was received.
attempts	Provider attempts. Validation failures report 0; writes report at most 1.
hits	At most pageSize search hits.
total	Elasticsearch total-hit value.
aggregations	Aggregation response when present.
nextSearchAfter	Opaque sort values from the final hit.
hasMore	true only when a full page and a pagination token were returned.
found	Whether get found the document. HTTP 404 for a valid get is normalized as found: false.
document	Retrieved _source for get.
documentId	Provider document ID for document operations.
result	Provider write result or the bulk summary completed / partial_failure.
version	Provider document version when present.
bulkItems	Per-action name, document ID, HTTP status, result, or safe error type. Provider error reasons are excluded.
succeeded / failed	Bulk action counts.
error	Safe object with code, message, and retryable.

Credentials, Authorization headers, endpoint origins, raw provider response bodies, provider error reasons, and request payloads are never returned.

IntegrationAccount Requirements

The bound IntegrationAccount must:

• have status exactly READY;
• put only an HTTPS origin such as https://search.example.com:9243 in accountId;
• put an Elasticsearch API key in the apiKey SecureField, or a Basic-auth username and password in their SecureFields;
• contain no user info, path, query, fragment, or credential inside accountId;
• use a least-privilege role scoped to the documented indexes and operations;
• permit the ValkyrAI runtime's network origin and TLS trust chain.

API key authentication takes precedence when both forms are present. Plaintext HTTP, embedded credentials, and anonymous fallback are rejected.

Configuration

Configuration	Default	Constraint
elasticsearchAccount	None	Required READY IntegrationAccount.
operation	None	Required allowlisted operation.
index	None	Required lowercase bounded name.
documentId	None	Required for document-scoped operations.
query	{"match_all":{}}	JSON object, included only for search.
pageSize	25	1–100.
requireConfirmation	false	Must be true for writes.
timeoutMs	10000	500–30,000 milliseconds, covering connect, write, read, and total call time.
maxReadRetries	1	0–2 additional attempts for search and get only.

Responses are capped at 5 MiB. A single document or search body is capped at 1 MiB. Bulk input is capped at 500 actions and 2 MiB of newline-delimited JSON.

Operations

search

Sends POST /{index}/_search with a bounded Query DSL body, optional aggregations, optional sort, optional search_after, track_total_hits: true, and the bounded page size. The output preserves hit objects because _source, highlights, fields, and sort values are provider-defined.

Use nextSearchAfter unchanged on the next execution. A stable sort with a unique tie-breaker is required to avoid duplicates or omissions while the index changes.

get

Sends GET /{index}/_doc/{documentId}. A provider 404 is a successful absence with found: false, not a fabricated document or workflow failure.

index

Sends PUT /{index}/_doc/{documentId}. The caller must provide the ID and confirmation. Reusing the same ID makes the write reconcilable, but the module still reports non-idempotent behavior because repeated execution can replace a newer document and increment provider versions.

update

Sends one partial-document update to POST /{index}/_update/{documentId}. Scripts and upserts are intentionally unavailable. A transport timeout after submission is ambiguous; inspect the current document before retrying.

delete

Sends one DELETE /{index}/_doc/{documentId}. Deletion is destructive and is never retried automatically.

bulk

Builds provider NDJSON from structured actions. Every action has a caller-owned ID; index and update actions have bounded document objects. The module sends one request to POST /_bulk, then returns safe per-item status and counts. Elasticsearch may commit only part of the request, so partial_failure requires item-by-item reconciliation.

Errors and Failure Modes

Code	Meaning	Recovery
VALIDATION_ERROR	Missing, malformed, unsafe, oversized, or raw credential/provider input.	Correct the named input. No provider request was sent.
UNSUPPORTED_OPERATION	Operation is outside the six-operation allowlist.	Select a documented operation.
CONFIRMATION_REQUIRED	A write omitted requireConfirmation: true.	Review the side effect, then confirm explicitly.
INTEGRATION_ACCOUNT_ERROR	Account is absent, not READY, or lacks supported SecureFields.	Repair and rebind the IntegrationAccount.
ELASTICSEARCH_AUTH_ERROR	HTTP 401 or 403.	Verify credential validity and least-privilege index permissions.
ELASTICSEARCH_NOT_FOUND	A non-get resource returned HTTP 404.	Verify the index and document ID.
ELASTICSEARCH_RATE_LIMITED	HTTP 429 remained after bounded read retries, or occurred on a write.	Respect cluster capacity. Reconcile writes before retrying.
ELASTICSEARCH_TRANSIENT_ERROR	HTTP 502, 503, or 504 remained after bounded read retries, or occurred on a write.	Verify cluster health. Reconcile writes before retrying.
ELASTICSEARCH_HTTP_ERROR	Another non-2xx status.	Inspect provider logs with authorized tooling; the workflow output omits provider bodies.
ELASTICSEARCH_TRANSPORT_ERROR	TLS, DNS, connection, timeout, request, or bounded-read failure.	Verify connectivity. Writes may be ambiguous and must be reconciled.
ELASTICSEARCH_RESPONSE_ERROR	A successful provider response was invalid JSON or violated output bounds.	Inspect provider/proxy behavior without exposing the body to the workflow.

Example

Search active summaries with stable search_after pagination:

{
  "operation": "search",
  "index": "customer-summaries",
  "query": {
    "term": {
      "status": "active"
    }
  },
  "sort": [
    { "updatedAt": "asc" },
    { "_id": "asc" }
  ],
  "pageSize": 25
}

Expected normalized result shape:

{
  "status": "success",
  "operation": "search",
  "index": "customer-summaries",
  "httpStatus": 200,
  "attempts": 1,
  "total": 42,
  "hits": [
    {
      "_id": "customer-42",
      "_source": { "status": "active" },
      "sort": [1786262400000, "customer-42"]
    }
  ],
  "nextSearchAfter": [1786262400000, "customer-42"],
  "hasMore": false
}

Notes

• Pagination is bounded and stateless. The module does not create Elasticsearch scroll or point-in-time resources.
• hasMore is a conservative convenience signal: it is true only for a full page with returned sort values. A full page does not prove that another hit exists.
• Search and get retries are limited to transient transport failures and HTTP 429, 502, 503, or 504. They reuse the exact request and never follow redirects.
• Index, update, delete, and bulk are never retried automatically, even for rate limits or transient failures. Reconcile provider state first.
• Bulk HTTP 200 does not mean every action succeeded. Always inspect failed, result, and bulkItems.
• Delete is destructive. Index can replace existing state. Update can overwrite fields. The module has no dry-run mode.
• The repository suite does not use a live Elasticsearch credential. It deterministically verifies request construction, authentication selection, validation, bounded read retries, pagination mapping, bulk NDJSON, partial failures, write single-attempt behavior, secret redaction, annotation scanning, and catalog serialization. Live TLS, provider version compatibility, and role permissions remain deployment-time boundaries.