Advanced CMS-Workflow Integration & Module Configuration System

🎯 Overview

This comprehensive upgrade transforms ValkyrAI's CMS (ContentData) and Workflow engine integration from raw JSON configuration editing to a production-grade system with:

1. Structured Module Configuration - No more JSON editing; full UI controls with dropdowns, lookups, and validation
2. Advanced Template Merging - Handlebars/Mustache with filters, conditionals, and loops
3. CMS-First Workflow Design - Load templates from ContentData with automatic metadata merging
4. Type-Safe Configuration - Backend schema definitions drive frontend UI generation

---

📋 Architecture

Backend Components

1. ModuleConfigSchema (com.valkyrlabs.workflow.config.ModuleConfigSchema)

Defines the structure for module configurations with full UI metadata:

ModuleConfigSchema schema = new ModuleConfigSchema("EMAIL", "Email Module", "Send emails with CMS templates");

schema.addFields(
  new ConfigField("integrationAccountId", "Email Integration", FieldType.LOOKUP)
    .required(true)
    .section("Account")
    .apiLookup(new ApiLookup("/api/v1/integrationaccounts", "id", "accountName")
      .filterParam("type", "EMAIL")
      .searchable(true)),

  new ConfigField("templateId", "CMS Template", FieldType.LOOKUP)
    .section("Template")
    .apiLookup(new ApiLookup("/api/v1/contentdata", "id", "title")
      .filterParam("contentType", "template")
      .searchable(true))
);

Key Features:

• FieldType Enum: TEXT, EMAIL, NUMBER, TEXTAREA, CHECKBOX, SELECT, MULTISELECT, LOOKUP, JSON, CODE, TEMPLATE_EDITOR, RECIPIENT_LIST, etc.
• ApiLookup: Define how to populate dropdown options from API endpoints with filtering, search, and pagination
• Conditional Fields: Use dependsOn() to show/hide fields based on other field values
• Validation Rules: Built-in validation (url, email, number, etc.)
• Nested Sections: Group related fields with section() for better UX

2. ModuleConfigSchemaRegistry (com.valkyrlabs.workflow.config.ModuleConfigSchemaRegistry)

Centralized registry of all module configuration schemas. Pre-registers schemas for:

• EMAIL (with ContentData template lookup)
• REST (with method selector, auth accounts, header/query builders)
• CONTENT_TEMPLATE (advanced merging modes)
• BRANCHING (condition editor)
• LLM_ROUTING (model selection)
• SLACK, TEAMS (with channel selectors)

@Component
public class ModuleConfigSchemaRegistry {
  // Get schema for specific module type
  public Optional<ModuleConfigSchema> getSchema(String moduleType);

  // Register custom module schemas
  public void registerSchema(String moduleType, ModuleConfigSchema schema);

  // Get all schemas
  public Map<String, ModuleConfigSchema> getAllSchemas();
}

3. ExecModuleConfigService (com.valkyrlabs.workflow.service.ExecModuleConfigService)

Type-safe configuration parsing and validation:

// Parse moduleData JSON safely
Map<String, Object> config = configService.parseModuleData(module);

// Get typed values with defaults
String templateId = configService.getStringConfig(module, "templateId", "");
boolean escapeHtml = configService.getBooleanConfig(module, "escapeHtml", true);

// Validate against schema
ValidationResult result = configService.validateConfig(module);
if (!result.isValid()) {
  Map<String, String> errors = result.getErrors();
}

// Merge with schema defaults
Map<String, Object> configWithDefaults = configService.mergeWithDefaults(module);

4. ModuleConfigSchemaController (com.valkyrlabs.workflow.controller.ModuleConfigSchemaController)

REST endpoints for frontend:

• GET /api/v1/module-schemas - Get all schemas
• GET /api/v1/module-schemas/{moduleType} - Get specific schema

---

Enhanced ContentTemplateAdvancedModule

Location: com.valkyrlabs.workflow.modules.ContentTemplateAdvancedModule

Significantly improved template rendering:

Supported Template Engines

Mustache (simple variable substitution):

Hello {{firstName}} {{lastName}}!
Your email: {{email | lowercase}}

Handlebars (advanced control flow):

{{#if isPremium}}
  Welcome VIP member {{name}}!
{{/if}}

{{#unless isSubscribed}}
  <a href="/subscribe">Subscribe now</a>
{{/unless}}

{{#each items}}
  <li>Item {{@index}}: {{this.name}} - ${{this.price | currency}}</li>
{{/each}}

Built-in Filters

All filters can be piped: {{value | uppercase | trim}}

Filter	Input	Output
uppercase / upper	"hello"	"HELLO"
lowercase / lower	"HELLO"	"hello"
capitalize / cap	"hello world"	"Hello world"
trim	" hello "	"hello"
truncate	"very long text..."	"very long text..." (50 chars)
reverse	"hello"	"olleh"
length / size	"hello"	"5"
html_encode	""<div>""	"<div>"
url_encode	"hello world"	"hello%20world"
currency / money	"123.456"	"$123.46"
percent	"0.75"	"75%"
ceil / ceiling	"3.2"	"4"
floor	"3.8"	"3"
round	"3.7"	"4"

Handlebars Loop Variables

Inside {{#each}} blocks:

• {{this}} - Current item
• {{@index}} - Zero-based index
• {{@key}} - Key for object iteration
• {{@first}} - True if first item
• {{@last}} - True if last item
• {{@odd}} - True if odd index
• {{@even}} - True if even index

Configuration Example

{
  "mode": "single",
  "contentId": "uuid-of-template",
  "mergeEngine": "handlebars",
  "escapeHtml": true,
  "contextVars": {
    "companyName": "Acme Corp"
  }
}

Search Mode (QBE)

{
  "mode": "search",
  "search": {
    "type": "template",
    "category": "email",
    "limit": 5
  }
}

---

Frontend Components

1. AdvancedModuleDesigner.tsx

Beautiful, responsive module configuration UI that replaces raw JSON editing:

import AdvancedModuleDesigner from "@/components/WorkflowStudio/AdvancedModuleDesigner";

<AdvancedModuleDesigner
  schema={moduleSchema}
  currentConfig={execModule.moduleData ? JSON.parse(execModule.moduleData) : {}}
  onConfigChange={(newConfig) => {
    // Save to backend
    updateExecModule({
      ...execModule,
      moduleData: JSON.stringify(newConfig),
    });
  }}
  loading={loading}
  error={error}
/>;

Features:

• ✅ Tabbed interface (Configuration, JSON Preview, Preview)
• ✅ Grouped field sections with collapsible headers
• ✅ Real-time validation with error messages
• ✅ Conditional field visibility based on dependencies
• ✅ Monaco Editor for JSON and Handlebars templates
• ✅ API-powered lookup fields with autocomplete and search
• ✅ Recipient list builder for email workflows
• ✅ Drag-and-drop for complex field configurations

Field Type Renderers:

Type	Control	Example
TEXT, EMAIL	TextField	"user@example.com"
NUMBER	TextField[type=number]	42
TEXTAREA	TextField[multiline]	Multi-line text
CHECKBOX	Checkbox	Toggle switch
SELECT	dropdown	"mustache", "handlebars"
MULTISELECT	Multi-select	Multiple selections
LOOKUP	Autocomplete	IntegrationAccount selector
JSON	Monaco Editor	Custom JSON configs
CODE	Monaco Editor (code)	Lambda expressions
TEMPLATE_EDITOR	Monaco Editor (handlebars)	Email templates
RECIPIENT_LIST	List builder	Email recipients

2. useModuleConfigSchema.ts Hook

Fetch and cache module schemas from backend:

import {
  useModuleConfigSchema,
  useAllModuleConfigSchemas,
} from "@/hooks/useModuleConfigSchema";

// Get schema for specific module type
const { schema, loading, error } = useModuleConfigSchema(execModule.moduleType);

// Get all available schemas
const { allSchemas, loading } = useAllModuleConfigSchemas();

// Prefetch schema for better UX
prefetchModuleSchema("EMAIL");

// Clear cache after schema updates
clearSchemaCache();

---

🚀 Usage Examples

Email Module Configuration

Before (Raw JSON):

{
  "integrationAccountId": "...",
  "templateMode": "cms",
  "templateId": "...",
  "mergeEngine": "handlebars",
  "subjectTemplate": "Welcome {{firstName}}"
}

After (Beautiful UI):

<AdvancedModuleDesigner
  schema={emailSchema}
  currentConfig={config}
  onConfigChange={handleChange}
/>

Renders with:

• Account dropdown (filtered to EMAIL type)
• Template mode selector (CMS/Inline/URL)
• CMS template lookup (searchable, filtered to "email" category)
• Merge engine selector
• Subject line template editor
• Recipient mode selector
• Context variables JSON editor

Content Template Module

Load multiple templates from CMS and compose them:

{
  "mode": "multiple",
  "templateIds": ["header-id", "body-id", "footer-id"],
  "mergeEngine": "handlebars",
  "customFilters": {}
}

Or search for templates by type:

{
  "mode": "search",
  "search": {
    "type": "template",
    "category": "email",
    "limit": 10
  }
}

REST Module Configuration

{
  "url": "https://api.example.com/users",
  "method": "POST",
  "integrationAccountId": "auth-account-id",
  "headers": {
    "Content-Type": "application/json"
  },
  "queryParams": {},
  "body": "{{requestBody | json}}",
  "responseMapping": {
    "userId": "result.id",
    "userName": "result.name"
  }
}

---

📊 Email Template Example

CMS ContentData (stored template):

<!DOCTYPE html>
<html>
<head>
  <title>{{subject}}</title>
</head>
<body>
  <h1>Hello {{firstName | capitalize}}!</h1>

  {{#if isPremium}}
    <p class="vip-badge">VIP Member</p>
  {{/if}}

  {{#unless isUnsubscribed}}
    <section>
      <h2>Your Recent Orders</h2>
      {{#each orders}}
        <div class="order">
          <p>Order #{{@index}} - {{this.name}}</p>
          <p>Total: {{this.amount | currency}}</p>
          {{#if this.completed}}
            <span class="badge">Completed</span>
          {{/if}}
        </div>
      {{/each}}
    </section>
  {{/unless}}

  <footer>
    <p>&copy; {{companyName | html_encode}}</p>
  </footer>
</body>
</html>

Workflow Config:

{
  "mode": "single",
  "contentId": "template-uuid",
  "templateMode": "cms",
  "mergeEngine": "handlebars",
  "integrationAccountId": "sendgrid-account-id",
  "subjectTemplate": "Welcome {{firstName}}!",
  "recipientMode": "dynamic",
  "recipientDataPath": "recipients",
  "escapeHtml": true
}

Workflow Input Data:

{
  "recipients": [
    {
      "email": "john@example.com",
      "firstName": "john",
      "isPremium": true,
      "orders": [
        { "name": "Item A", "amount": 29.99, "completed": true },
        { "name": "Item B", "amount": 49.99, "completed": false }
      ]
    }
  ],
  "globalContext": {
    "companyName": "Acme Corp"
  }
}

Rendered Email:

<!DOCTYPE html>
<html>
  <head>
    <title>Welcome john!</title>
  </head>
  <body>
    <h1>Hello John!</h1>

    <p class="vip-badge">VIP Member</p>

    <section>
      <h2>Your Recent Orders</h2>
      <div class="order">
        <p>Order #0 - Item A</p>
        <p>Total: $29.99</p>
        <span class="badge">Completed</span>
      </div>
      <div class="order">
        <p>Order #1 - Item B</p>
        <p>Total: $49.99</p>
      </div>
    </section>

    <footer>
      <p>&copy; Acme Corp</p>
    </footer>
  </body>
</html>

---

🔌 Integration Points

Adding a New Module Type

1. Create Module Configuration Schema:

@Component
public class MyCustomModuleSchemaProvider {
  @Autowired
  private ModuleConfigSchemaRegistry registry;

  @PostConstruct
  public void registerSchema() {
    ModuleConfigSchema schema = new ModuleConfigSchema(
      "MY_MODULE",
      "My Custom Module",
      "Does something awesome"
    );

    schema.addFields(
      new ConfigField("apiKey", "API Key", FieldType.TEXT)
        .required(true)
        .section("Configuration")
    );

    registry.registerSchema("MY_MODULE", schema);
  }
}

2. Implement Module Execution:

@Component("myCustomModule")
@VModuleAware
public class MyCustomModule extends BaseMapIOModule {
  @Autowired
  private ExecModuleConfigService configService;

  @Override
  public void execute() {
    // Use config service for type-safe access
    String apiKey = configService.getStringConfig(execModule, "apiKey", "");

    // Do work...
    outputMap.put("result", "success");
  }
}

3. Frontend automatically gets UI from schema definition!

---

🧪 Testing

Backend Tests

@SpringBootTest
class ExecModuleConfigServiceTests {
  @Autowired
  private ExecModuleConfigService configService;

  @Test
  void testConfigParsing() {
    ExecModule module = new ExecModule();
    module.setModuleData("{\"key\": \"value\"}");

    Map<String, Object> config = configService.parseModuleData(module);
    assertThat(config).containsEntry("key", "value");
  }

  @Test
  void testValidation() {
    ExecModule module = new ExecModule();
    module.setModuleType(ModuleTypeEnum.EMAIL);
    module.setModuleData("{}");

    ValidationResult result = configService.validateConfig(module);
    assertThat(result.isValid()).isFalse();
    assertThat(result.getErrors()).containsKey("integrationAccountId");
  }
}

Frontend Tests

import { render, screen, waitFor } from "@testing-library/react";
import AdvancedModuleDesigner from "@/components/WorkflowStudio/AdvancedModuleDesigner";

test("renders configuration fields from schema", async () => {
  const schema: ModuleConfigSchema = {
    moduleType: "TEST",
    moduleName: "Test Module",
    description: "Test",
    fields: [
      {
        name: "testField",
        label: "Test Field",
        fieldType: "TEXT",
        required: true,
        order: 1,
        validationRules: [],
        fieldOptions: {},
      },
    ],
    metadata: {},
  };

  render(
    <AdvancedModuleDesigner
      schema={schema}
      currentConfig={{}}
      onConfigChange={jest.fn()}
    />
  );

  await waitFor(() => {
    expect(screen.getByLabelText("Test Field")).toBeInTheDocument();
  });
});

---

📚 Migration Guide

From Raw JSON to Schema-Based Configuration

Old Approach:

• Manually edit moduleData JSON in database
• No validation
• No dropdown options
• Hard to discover features

New Approach:

• Define schema in ModuleConfigSchemaRegistry
• Frontend auto-generates UI
• Built-in validation
• Discoverable field options via API lookups
• Better UX for end-users

Database Migration (Optional)

Existing moduleData JSON continues to work. New fields added to schemas apply defaults automatically via configService.mergeWithDefaults().

---

🎓 Best Practices

1. Always use ConfigField builder for readability:

   new ConfigField("field", "Label", FieldType.LOOKUP)
     .description("Help text")
     .required(true)
     .section("Grouping")
     .order(1)

2. Filter API lookups for better UX:

   .filterParam("type", "EMAIL")
   .filterParam("status", "active")

3. Make fields conditional based on dependencies:

   .dependsOn("templateMode")  // Only show if templateMode is set

4. Use appropriate FieldType for each use case:

   • TEMPLATE_EDITOR for Handlebars/Mustache
   • LOOKUP for API-powered selectors
   • JSON for complex nested configs
   • RECIPIENT_LIST for email recipients

5. Validate in both backend and frontend:

   • Backend: configService.validateConfig()
   • Frontend: AdvancedModuleDesigner shows errors

---

🔄 Version History

• v1.0.0 (2025-10-20)
  • Initial release: ModuleConfigSchema, Registry, Controller
  • ContentTemplateAdvancedModule enhancements
  • AdvancedModuleDesigner React component
  • Email, REST, ContentTemplate, Branching, LLM modules
  • Full Handlebars/Mustache support with filters
  • CMS integration with ContentData lookups

---

📞 Support

For issues or questions:

1. Check existing module schemas in ModuleConfigSchemaRegistry
2. Review example configurations in workflow definitions
3. Test with AdvancedModuleDesigner UI before saving
4. Check validation errors via ExecModuleConfigService

---

This is NOT just an upgrade—it's a complete reinvention of how modules are configured in ValkyrAI. No more JSON editing madness! 🎉