N8N Killer Workflow Designer Initiative

🎯 Mission

Transform ValkyrAI's WorkflowStudio into the most intuitive, powerful workflow designer that makes N8N look primitive. Build production-grade, enterprise-ready automation with rock-solid UX.

Deadline: Production-ready with all features working flawlessly.

---

📋 Architecture Overview

Current State

• Frontend: React + ReactFlow canvas, Redux Toolkit Query
• Backend: Spring Boot 3.4.6, ValkyrWorkflowService
• Data Model: Workflow → Task → ExecModule (chain execution)
• ExecModules: 20+ connectors for Email, REST, Stripe, AWS, Social, etc.

The Challenge

1. ❌ ExecModules within a Task are not visually chainable ("snap together like LEGO")
2. ❌ No unified configuration editor for each module's specific requirements
3. ❌ No data lookups for REST API auto-complete
4. ❌ No workflow trigger endpoint (POST object → execute workflow with data)
5. ❌ Server-side module execution logic needs tightening
6. ❌ Create/edit workflows have sync issues between UI and backend
7. ❌ JPA ID generation issues when creating new objects

---

🔧 Phase 1: UX/Usability Improvements (Days 1-2)

1.1 Unified ExecModule Configuration Editor

Goal: Every module type (Email, REST, Stripe, etc.) gets a rock-solid form builder.

Tasks:

• Create ExecModuleConfigBuilder component

  • Introspect ExecModule schema via ThorAPI
  • Generate form fields for each module type
  • Include field validation, descriptions, examples

• Build module-specific form schemas (for each ExecModule type)

  • EmailModule: recipient, subject, body, attachments
  • RestModule: URL, method, headers, body template
  • StripeModule: API key, charge amount, customer ID
  • AwsModule: region, service, operation params
  • SocialModule: platform, auth, message content
  • etc. (all 20+ modules)

• Implement rich input field types:

  • Text, textarea, number, dropdown, multi-select
  • Date/time picker
  • JSON editor (for complex configs)
  • QBE lookups for data from REST APIs
  • API browsing to select endpoints dynamically

1.2 ExecModule Chaining Visualization

Goal: Show how modules feed into each other like LEGO blocks.

Tasks:

• Update WorkflowCanvas to display modules within Tasks

  • Add nested module nodes inside task nodes
  • Show data flow between modules (arrows, connections)
  • Support drag-to-reorder modules within task
  • Snap modules together visually

• Create ModuleChainViewer component

  • Shows input/output mapping
  • Visual feedback for data transformation
  • Hover to see data payload

• Implement drag-and-drop for chaining

  • Connect output of module A to input of module B
  • Auto-suggest compatible data fields

1.3 Task UX Improvements

Goal: Better visibility and control over task execution context.

Tasks:

• Create task-level configuration panel

  • Task name, description
  • Execution role (SYSTEM, ASSISTANT, USER)
  • Retry policy (max retries, backoff strategy)
  • Timeout settings
  • Error handling (fail-fast vs continue)

• Show module list attached to task

  • Visual list with module icons and names
  • Drag to reorder execution
  • Quick edit / quick delete buttons
  • Connection status (success/error from last run)

1.4 Workflow Trigger Endpoint & Object-to-Workflow Mapping

Goal: Allow external systems to trigger workflows by POSTing data objects.

Tasks:

• Create POST /v1/vaiworkflow/{workflowId}/trigger endpoint

  • Accepts JSON body with workflow input data
  • Validates input against workflow schema
  • Returns execution ID (for polling/monitoring)

• Build API browsing UI component

  • Browse available REST APIs from environment
  • Select endpoint → fetch schema
  • Map request/response to workflow variables
  • Test trigger with sample data

• Implement data mapping UI

  • Map external object fields → workflow input variables
  • Show data transformation preview
  • Validate mappings before save

---

🛠️ Phase 2: Server-Side Tightening (Days 2-3)

2.1 Workflow Execution Service Refactor

File: ValkyrWorkflowService.java

Goals:

• Tighten task/module execution logic
• Fix lazy-loading issues (no more LazyInitializationException)
• Ensure proper state persistence
• Implement correct data flow between modules

Tasks:

• Audit executeTask() method

  • Pre-load all nested collections (tasks, modules) in transaction
  • Use Hibernate.initialize() before async boundaries
  • Test with deep module chains

• Implement module input/output mapping

  • Parse task.getInputMapping() correctly
  • Thread workflow state through module chain
  • Capture module output, merge into globalState

• Implement branching logic correctly

  • BranchingModule evaluates condition on input state
  • Routes to appropriate next task
  • Joins converging branches

• Add comprehensive logging & metrics

  • Log at each module execution
  • Track execution time per module
  • Emit structured events for monitoring

2.2 ExecModule Service Tightening

File: ValkyrExecModuleService.java

Goals:

• Proper CRUD with ThorAPI service layer
• Validate module configuration
• Ensure no JPA ID issues on creation

Tasks:

• Audit createExecModule() method

  • NEVER set ID on new objects (JPA handles it)
  • Use generated ThorAPI service methods
  • Set default values for required fields
  • Return fresh object from DB (verify persistence)

• Implement validateModuleConfig() method

  • Check required fields per module type
  • Validate field values (URLs, enums, etc.)
  • Return validation errors

• Implement getModuleConfigSchema() method

  • Introspect each ExecModule type
  • Return JSON schema for frontend form builder
  • Include examples, descriptions

2.3 Task Service Improvements

File: ValkyrTaskService.java

Goals:

• Proper task creation/update lifecycle
• Correct module chaining handling

Tasks:

• Audit createTask() method

  • Initialize empty module list
  • Don't set workflow.id manually (relationship handled by repository)
  • Return fresh task from DB

• Implement addModuleToTask() helper

  • Set correct moduleOrder
  • Update task.modules list
  • Persist task (not just module)

• Implement updateModuleOrder() helper

  • Re-number moduleOrder fields
  • Persist changes

---

📊 Phase 3: Data Lookup & API Integration (Days 3-4)

3.1 REST API Auto-Complete Service

Goal: Auto-suggest values for module inputs based on REST API responses.

Tasks:

• Create RestApiLookupService (Java)

  • Maintain cache of known APIs (OasOpenAPISpec)
  • For a given module, detect APIs it can call
  • Pre-fetch sample responses

• Create RestApiLookupComponent (React)

  • Dropdown showing available APIs
  • Select API → fetch available fields
  • Auto-suggest field values from API response
  • Test query against API

• Implement QBE (Query By Example) for module inputs

  • Example: EmailModule needs recipient → query User API with QBE
  • Show results in dropdown
  • Allow user to select value

3.2 API Schema Browser

Goal: Browse available REST APIs and their schemas.

Tasks:

• Create ApiBrowserComponent

  • List all available OasOpenAPISpec objects
  • Search by name/tag
  • Show endpoints, parameters, response schema
  • Test endpoint with sample data

• Integrate with module configuration

  • When user selects "REST" module type
  • Show API browser to select endpoint
  • Auto-populate module config from schema

---

🚀 Phase 4: Workflow Lifecycle & Sync (Days 4-5)

4.1 Create Workflow Flow

Goal: Rock-solid workflow creation with instant server sync.

Frontend (CreateWorkflowFlow)

1. User clicks "New Workflow"
2. Show modal with workflow name, description
3. User clicks "Create"
4. POST /Workflow with name (DO NOT SET ID)
5. Server returns fresh workflow object WITH id set
6. UI navigates to editor with workflowId from server
7. All subsequent operations use that ID

Backend (WorkflowController.postWorkflow)

1. Receive Workflow object (ID should be null/empty)
2. Validate (name required, etc.)
3. Save via WorkflowService.saveOrUpdate(workflow)
4. JPA auto-generates UUID
5. Return saved object (with ID) to client
6. Client uses that ID for all future ops

4.2 Edit Workflow Flow

Goal: Dirty-state tracking, auto-save, conflict resolution.

Tasks:

• Implement dirty-state tracking in Redux

  • Track which fields changed
  • Show "unsaved changes" indicator
  • Prevent navigation without save prompt

• Implement auto-save (debounced)

  • 2 second debounce on field changes
  • PATCH endpoint for partial updates
  • Show save status (saving, saved, error)

• Implement update conflict detection

  • On save, check workflow.updatedDate on server
  • If stale, show conflict dialog
  • Options: keep local, accept server, manual merge

4.3 Workflow Validation

Goal: Prevent invalid workflows from being saved.

Tasks:

• Implement validateWorkflow() on frontend

  • At least 1 task in workflow
  • Each task has at least 1 module
  • All required fields filled
  • No cycles in task graph

• Implement validateWorkflow() on backend

  • Same validation server-side
  • Return 400 if invalid
  • Include validation errors in response

---

🧪 Phase 5: Testing & Quality (Days 5-6)

5.1 Unit Tests

Frontend:

• ExecModuleConfigBuilder.test.tsx
• ModuleChainViewer.test.tsx
• WorkflowCanvas.integration.test.tsx

Backend:

• ValkyrWorkflowServiceTest.java
• ValkyrTaskServiceTest.java
• ValkyrExecModuleServiceTest.java

5.2 Integration Tests

E2E Workflow Creation:

• Create workflow via API
• Add task to workflow
• Add 3 ExecModules to task with chaining
• Save workflow
• Retrieve workflow, verify structure
• Execute workflow, verify state transitions

5.3 Manual Testing

• Create workflow in UI
• Configure each module type with validation
• Save, refresh, verify persistence
• Edit existing workflow
• Test dirty-state tracking
• Test auto-save
• Test workflow execution
• Test error handling

---

📝 Implementation Checklist

Essential Rules

• ✅ NO ID SETTING ON NEW OBJECTS: Let JPA handle it
• ✅ USE ThorAPI GENERATED SERVICES: Never write custom queries
• ✅ EAGER LOAD BEFORE ASYNC: Prevent LazyInitializationException
• ✅ VALIDATE SERVER-SIDE: Never trust client input
• ✅ TRANSACTION BOUNDARIES: Use @Transactional(REQUIRES_NEW) for async state persistence
• ✅ SECURITY CONTEXT: Propagate auth to async threads explicitly
• ✅ TESTING: Every feature needs tests

Deliverables

• ExecModule unified config editor (TypeScript + form schemas)
• Module chaining visualization (ReactFlow nested nodes)
• REST API lookup & auto-complete service
• Workflow trigger endpoint (POST /v1/vaiworkflow/{id}/trigger)
• Tightened workflow execution service (ValkyrWorkflowService)
• Tightened ExecModule service (ValkyrExecModuleService)
• Create/Edit workflow flows with sync
• Comprehensive unit & integration tests
• Updated documentation (ADR, README, API docs)

---

🔍 Key Files to Modify/Create

Frontend (React/TypeScript)

web/typescript/valkyr_labs_com/src/components/WorkflowStudio/
├── ExecModuleConfigBuilder.tsx          (NEW - unified config editor)
├── ModuleChainViewer.tsx                (NEW - visualize module chain)
├── ApiBrowserComponent.tsx              (NEW - browse & select APIs)
├── RestApiLookupComponent.tsx           (NEW - auto-complete)
├── WorkflowCanvas.tsx                   (MODIFY - show nested modules)
├── ExecModuleEditModal.tsx              (MODIFY - use new builder)
├── InspectorPanel.tsx                   (MODIFY - module list with drag-reorder)
├── execModuleCatalog.ts                 (MODIFY - add form schemas for each module type)
└── utils/execModule.ts                  (MODIFY - add validation helpers)

redux/
├── features/workflows/workflowSlice.ts  (MODIFY - dirty-state tracking)
└── services/WorkflowService.ts          (MODIFY - auto-save, conflict detection)

Backend (Java)

valkyrai/src/main/java/com/valkyrlabs/workflow/
├── service/ValkyrWorkflowService.java           (MODIFY - tighten execution)
├── service/ValkyrExecModuleService.java         (MODIFY - tighten CRUD)
├── service/ValkyrTaskService.java               (MODIFY - tighten CRUD)
├── controller/WorkflowController.java           (MODIFY - add trigger endpoint)
├── validation/WorkflowValidator.java            (NEW - validation logic)
└── config/ExecModuleSchemaProvider.java         (NEW - module config schemas)

---

🎨 Design System

Color Palette (by Module Type)

• Flow Control: Blue (#38bdf8) - Start, End, Branch
• Communication: Purple (#a78bfa) - Email, SMS, Slack
• Data: Green (#4ade80) - REST, Database, Spreadsheet
• External: Orange (#f97316) - Stripe, AWS, Shopify
• Transform: Yellow (#facc15) - Map, Filter, Aggregate
• Error: Red (#f87171) - Error handlers

Component Hierarchy

WorkflowStudio (root)
├── WorkflowCanvas (ReactFlow)
│   ├── TaskNode (renders Task with nested ModuleNodes)
│   │   ├── ModuleNode (renders ExecModule)
│   │   │   ├── ModuleIcon (colored icon)
│   │   │   ├── ModuleName
│   │   │   └── ModuleHandle (connection points)
│   │   └── ModuleChainViewer (shows data flow)
│   └── TaskEdges
├── FloatingExecModulesToolbar
│   ├── SearchBar
│   ├── CategoryTabs
│   └── ModuleList
├── InspectorPanel
│   ├── SelectedNodeDetails
│   ├── ModulesList (for selected task)
│   ├── ExecModuleEditModal
│   └── ModuleChainViewer (detail view)
└── BottomConsole
    ├── ValidationErrors
    ├── ExecutionLog
    └── SaveStatus

---

🚀 Success Criteria

1. ✅ Users can create workflows in < 5 minutes
2. ✅ All 20+ module types have validation & config forms
3. ✅ Module chaining works visually (snap together like LEGO)
4. ✅ API lookups auto-complete user inputs
5. ✅ Workflows save/load reliably
6. ✅ Server-side execution is bulletproof (no LazyInit, proper state)
7. ✅ Tests cover 80%+ of code
8. ✅ Performance: rendering 100+ node workflows < 1s
9. ✅ Zero JPA ID generation errors
10. ✅ Production-ready, enterprise-grade UX

---

📅 Timeline

Phase	Duration	Deliverable
1	2 days	UX improvements, module config editor
2	1 day	Server-side tightening
3	1 day	API lookup & auto-complete
4	1 day	Workflow lifecycle & sync
5	1 day	Testing & QA
Total	6 days	Production-ready workflow designer

---

🎯 Next Steps

1. Start with Phase 1: Build ExecModule config builder
2. Get validation schemas for all 20+ modules
3. Implement module chaining visualization
4. Then move to server-side improvements (Phase 2)
5. Add API browsing & lookups (Phase 3)
6. Finalize create/edit flows (Phase 4)
7. Comprehensive testing (Phase 5)

Let's build the N8N killer! 🚀