Skip to main content

Active Context — ValkyrAI Workflow Engine Enhancement

Current Mission: TAKE VALKYRAI WORKFLOW ENGINE TO THE NEXT LEVEL!​

Master crispy wants the ValkyrAI Workflow Engine elevated to absolute next level - production-quality, world-class, simple, elegant, and powerful.

Primary Objectives​

🎯 Core Production Requirements​

  1. Smooth UX with Complete Error Handling

    • Perfect CRUD operations: add, save, update, delete workflows
    • Seamless load/edit cycle in workflow editor
    • Bulletproof error handling and user feedback

  2. Complete Workflow Operations UX

    • Load, save, update, delete, launch, pause, stop workflows
    • Change permissions and RBAC integration
    • Status monitoring and real-time feedback

  3. ExecModule Parameter Mapping

    • UX for editing ExecModule parameters
    • Data flow mapping between modules
    • Input Maps and WorkflowState object handling
    • Pass data from one task to the next seamlessly

  4. Built-in Data Selection/Gather ExecModule

    • QBE (Query By Example) support for API calls
    • Input variables from forms/environment
    • Data selection and filtering capabilities

  5. Proven Testable Default Workflow

    • Select data from API using QBE + input variables
    • Iterate over selected data
    • Send emails to list of selected Principals
    • Full end-to-end working example

  6. Async Branching Support

    • Branching that resolves "eventually"
    • Complex workflow orchestration
    • Conditional logic and parallel execution

  7. World-Class UX Completeness

    • Fully usable, production-quality interface
    • Simple, elegant, powerful workflow designer
    • Comprehensive dashboard for monitoring

Technical Foundation​

Backend (✅ Solid)​

  • Quartz-based scheduling system
  • 20+ ExecModules (Email, REST, Files, Social, Payments)
  • Spring Security RBAC/ACL integration
  • WebSocket real-time monitoring
  • Database persistence with graph storage

Frontend (🚀 Enhancement Target)​

  • React Flow visual designer
  • WorkflowStudio component with drag-drop
  • Real-time execution highlighting
  • Multiple UI components for workflow management

Implementation Strategy​

  1. Memory Bank Updates - Document current architecture
  2. Enhanced UX Components - Build production-level interfaces
  3. Data Flow Management - ExecModule parameter mapping
  4. QBE ExecModule - Built-in data selection capabilities
  5. Default Workflow - Proven email blast example
  6. Async Branching - Advanced workflow orchestration
  7. Integration Testing - Comprehensive validation

Success Criteria​

  • Workflow engine that rivals world-class automation platforms
  • Intuitive UX that non-technical users can operate
  • Robust error handling and recovery
  • Seamless data flow between workflow steps
  • Production-ready monitoring and management
  • Comprehensive testing coverage

Next Actions​

  1. Update memory bank with current architecture analysis
  2. Enhance WorkflowStudio with production-level UX
  3. Build ExecModule parameter mapping system
  4. Create QBE data selection ExecModule
  5. Implement proven email workflow example
  6. Add async branching capabilities
  7. Comprehensive testing and validation

Workflow Engine Mission Plan

Stabilize Core Experience: Harden the create → save → reload loop. Introduce optimistic UI state with transactional rollback, explicit validation/error surfaces, and auto-draft snapshots so editors never lose work.

CRUD + Lifecycle Dashboard: Elevate the workflow list into a command center with inline actions for launch/pause/stop, RBAC-aware sharing controls, and audit trails. Add dedicated detail pages for run history and configuration diffs.

Parameter Mapping UX: Build a two-panel mapper—left shows upstream outputs/workflow context, right shows ExecModule inputs. Support drag-to-map, type-aware validation, inline transformers, and snippets for WorkflowState paths.

ExecModule Authoring System: Provide richer forms per module via schema-driven metadata. Add module previews, validation badges, and test-run buttons to execute a module in isolation with stubbed inputs.

Data Gather (QBE) Module: Ship a first-class Query Builder ExecModule with REST connector presets, parameterized filters, preview table, paging controls, and input-variable binding. Persist query templates for reuse.

Reference Workflow Template: Offer a seeded “Principal Outreach” workflow demonstrating QBE fetch, iteration, and templated email send. Include unit/integration tests plus a guided wizard to clone/adapt it.

Async Branching Engine: Extend the runtime with fan-out/fan-in primitives, join nodes, timeout policies, and compensating actions. Surface branch progress in the UX with live status pills and retry controls.

Observability & Ops: Integrate Micrometer/Prometheus, structured logging, and distributed tracing IDs across nodes. In the UI, add real-time run monitors, performance analytics, and failure drill-down with rerun options.

Resilience & Testing: Expand automated coverage (backend integration + frontend RTL/Cypress). Add chaos scenarios for module failure, rate limits, and long-running branches to ensure graceful degradation.

Polish & Accessibility: Enforce consistent theming, responsive layouts, keyboard shortcuts, and screen-reader support. Layer in tooltips, inline docs, and onboarding tours to keep the designer approachable.

Implementation Blueprint — Requirements 1 & 2​

Data + Service Strategy​

  • Prefer generated ThorAPI RTK Query clients: @thorapi/redux/services/WorkflowService for CRUD and @thorapi/model/Workflow / WorkflowState payloads.
  • Layer lifecycle commands via existing redux/services/WorkflowCommandService.tsx; extend with helper thunks in redux/features/workflows/workflowLifecycle.ts (new) for composed launch/pause/stop flows.
  • Introduce selector helpers in redux/features/workflows/workflowSelectors.ts (new) to derive optimistic UI state, loading/error flags, and permission gates without touching generated files.
  • Use redux/services/VAIWorkflowEventService.tsx for WebSocket-backed status pushes; add an adapter in redux/features/workflows/workflowEvents.ts to normalize incoming payloads.

UX Hardening Tasks​

  1. Workflow Console (CRUD + Lifecycle Hub)

    • Build components/WorkflowConsole/WorkflowConsole.tsx (new) consuming selectors + command hooks.
    • Provide data grid with inline create/edit/delete, optimistic row updates, and surfaced error/resetError actions.
    • Add lifecycle action drawer (launch/pause/stop/reload) tied to new thunks; reflect real-time status via event service.
    • Wire RBAC-aware permission editor modal leveraging existing ACL ThorAPI services (no generated edits).

  2. Editor Stability Enhancements

    • In components/WorkflowStudio/ add useWorkflowDraft hook (new) for auto-save snapshots (draft persisted in Redux slice + localStorage fallback).
    • Validate before save using Thor model schemas (type guards) and show structured errors via toast + inline message well.
    • Implement redux/features/workflows/workflowEditorSlice.ts (new) to track dirty state, validation results, and last persisted revision id for diffing.

  3. Error & Feedback System

    • Centralize request feedback with components/Common/ErrorBoundaryWorkflow.tsx (new) wrapping console + editor.
    • Use generated error shapes; map HTTP issues to user-friendly copy with retry hooks.
    • Log significant failures through existing telemetry pipeline (utils/telemetryLogger.ts).

  4. Status Monitoring UI

    • Extend components/WorkflowMetricsDashboard/WorkflowMetricsDashboard.tsx to subscribe to new event adapter selectors for live run metrics.
    • Add timeline view component to show run history (pulling from ThorAPI WorkflowStateService).

Testing & Validation​

  • Add RTK Query integration tests in web/typescript/valkyr_labs_com/src/test/workflows/ stubbing Thor endpoints.
  • Create Cypress journey: CRUD workflow → launch → observe status update → pause/stop → permission edit.
  • Unit-test slices/thunks for optimistic rollback (Jest).

Dependencies / Coordination​

  • No schema changes expected; reuse existing fields (status, role, workflowState, ACL tables).
  • Coordinate with backend if new lifecycle command endpoints are required; otherwise rely on vaiworkflow/* routes already exposed.

Parameter Mapping & Module Authoring Progress​

  • Added Redux slice workflowMappings to persist module input bindings without touching ThorAPI code.
  • New ParameterMapperPanel renders workflow state + upstream outputs with drag-to-map, transformer hooks, and constant/environment sources.
  • ModuleDesigner auto-generates parameter forms from schema metadata, synchronizes mappings into moduleData, and exposes a test-run button hitting /v1/vaiworkflow/module/test.
  • QBE module schema (QbeDataGatherModule) ships with query builder UI (filters, paging, connector selection) and maps outputs for downstream tasks.

Roadmap — Remaining Pillars​

  1. Reference Workflow Template

    • Seed a Principal Outreach workflow (ThorAPI id: workflow_principal_outreach).
    • Includes tasks: QBE Fetch Principals (QbeDataGatherModule), Prepare Email Batch (Branching module), Send Outreach Email (EmailModule).
    • Provide guided wizard in UI (WorkflowConsole extension) to clone template with custom filter + email copy.
    • Backed by integration test hitting mocked ThorAPI endpoints (src/test/java/.../PrincipalOutreachWorkflowIT).

  2. Async Branching Engine

    • Implement runtime fan-out nodes (ParallelGatewayTask) with join semantics stored in WorkflowState.joinTokens.
    • Add scheduler support for eventual consistency (persist branch ledger table workflow_branch_ledger).
    • UI: new node type with live status pills using event stream (workflowEvents).

  3. Observability & Ops

    • Backend: add Micrometer registry, Prometheus scraping endpoint /actuator/prometheus.
    • Correlate runs with trace ids (use Sleuth/Micrometer Tracing).
    • UI dashboard widgets: throughput heatmap, failure drilldown, rerun controls (hooks in WorkflowConsole).

  4. Resilience & Testing

    • Expand integration suite with docker-compose DB + mocked services.
    • Frontend: add RTL tests for ParameterMapper and ModuleDesigner; Cypress flow covering template -> launch -> monitor.
    • Chaos tasks to simulate module failure + rate limiting (feature flag workflow.chaos.enabled).

  5. Polish & Accessibility

    • Global UX audit: consistent theming, keyboard shortcuts for canvas operations, screen reader labels (ARIA) across WorkflowStudio.
    • Add inline tooltips (Tooltip usage) plus onboarding walkthrough via react-joyride.
    • Responsive tweaks for tablet/mobile layout.

Wizard Simplification Plan (Phase 1)​

  • Replace six-step tab UX with three-stage wizard: 1. Define Basics, 2. Data & Workflows, 3. Review & Generate.
  • Use carousel/stepper (single active panel) driven by central state; hide tab strip.
  • Auto-populate generator defaults by application archetype; reduce type list to curated presets (Full Stack, Frontend, API).
  • Move bundle selection into optional Advanced Settings panel with smart defaults (baseline RBAC + recommended extras).
  • Keep schema designer (step 2) but preload starter schema when archetype chosen.
  • Third step summarises selections, shows bundle preview, enables generation triggers.

Implementation Notes​

  • Introduce useApplicationWizard hook to manage step index, form state, progress.
  • Replace Tab.Container structure in Application/index.tsx with a WizardCarousel component.
  • Create AdvancedOptionsDrawer for bundles/flags, hidden by default.
  • Update validation schema to align with simplified flow (type selection optional, defaults applied).
  • Maintain compatibility with existing ThorAPI service calls; submit via updated handler.