GrayMatter Operational Playbook

This guide is for operators, SREs, and account managers responsible for keeping GrayMatter healthy and cost-efficient.

Credit charges are runtime-configured. Use the main site Pricing page for current credit rates, and use this playbook for operating patterns and volume control.

First Response: Run the Doctor

Before changing credentials, credits, MCP config, or plugin files, run the GrayMatter doctor from the installed GrayMatter repo or plugin directory:

scripts/gm-doctor --quick

From the ValkyrAI repository, use the wrapper:

scripts/graymatter-doctor.sh --quick

Use the full smoke path when you want to prove live write/query behavior and are comfortable spending the small query cost:

scripts/gm-doctor

or, from ValkyrAI:

scripts/graymatter-doctor.sh

The doctor checks:

Layer	What It Proves	Typical Fix
Self-update	Plugin/repo can check the source-of-truth GrayMatter repo	Run `scripts/gm-self-update force` from a clean checkout
Install/auth	`jq`, keychain/env auth, and token permissions are usable	Run `scripts/gm-login`, then `scripts/gm-install-check`
Memory layer	The current account can use durable memory	Check credits, account roles, and `/v1/auth/me`
Graph/strategy/KPI layers	Cached OpenAPI exposes the business graph and operating domains	Run `scripts/gm-openapi-sync` or inspect api-0 availability
MCP contract	Agent hosts can discover the expected memory and graph tools	Repackage/reinstall the GrayMatter plugin
Deferred replay	Locally queued writes can be replayed after recovery	Run `scripts/gm-replay-deferred` after auth/credits are restored
Smoke	Live write/query works end to end	Resolve auth, credits, or API transport errors before agents rely on memory

--quick skips the live smoke query. It still verifies auth, layer status, OpenAPI cache, MCP contract, and replay readiness. If live OpenAPI sync is slow but a cached schema exists, the doctor reports a warning and continues from cache; that is acceptable for short-lived diagnostics, but operators should refresh the cache before declaring a release gate clean.

Daily Monitoring Routine

1. Check Memory Health (5 min)

Every morning, visit /graymatter-memory dashboard and review:

Metric	Target	Red Flag
Hit Rate	>90%	<70% = reindex needed
Context Utilization	60-80%	>85% = compact needed
Waste %	<20%	>30% = prune recommended
Burn Rate	Stable	2x usual = investigate

Action: If any red flag appears, move to appropriate action (below).

2. Review Recommendations

The dashboard shows sorted recommendations by impact:

COMPACT if waste is high → recovers 5-20% space
REINDEX if hit rate dropped → improves quality 5-10%
PRUNE if cold entries accumulate → removes stale data
EXPAND only if sustained demand → rarely needed

Action: Execute top recommendation if credits available.

3. Monitor Credits Balance

Check /credits/{accountId}? and verify:

Available: Should cover 1 week of normal operations
Burn rate: Credits/hour is consistent
Trend: Spike = investigate usage

Action: If <3 days of credits remain, top up via /buy-credits.

Operational Decisions Matrix

Scenario: Hit Rate Drops Below 80%

Symptoms:

Agents report "can't find relevant memories"
/memory/stats shows hitRate < 0.80
Users manually re-searching for entries

Root Causes:

Semantic drift: old embeddings became irrelevant
Incomplete tags: new entries not properly tagged
Memory pollution: too much low-signal data

Action Sequence:

Run REINDEX when the current credit balance supports it
```
POST /v1/memory/reindex?dryRun=false
```
- Duration: 15-30 min
- Expected improvement: +3-8% hit rate
- Credit charge: runtime configured
- Success rate: 85%
If hit rate still low, run PRUNE
```
POST /v1/memory/prune?confidenceThreshold=0.5&dryRun=false
```
- Removes low-confidence entries
- Credit charge: runtime configured
- Expected improvement: +2-5% hit rate
If still low, escalate to team
- May indicate architectural issue
- Conduct memory audit (see below)

Scenario: Context Utilization Exceeds 85%

Symptoms:

Dashboard shows usedPct > 0.85
Alerts say "context pressure high"
New writes may start failing with quota errors

Root Causes:

Too many entries stored
Large text fields consuming space
Aggressive entry creation without pruning

Action Sequence:

Run COMPACT when the current credit balance supports it
```
POST /v1/memory/compact?dryRun=false
```
- Consolidates similar entries
- Duration: 5-15 min
- Expected result: -10-20% utilization
- Credit charge: runtime configured
If still high (>70%), run PRUNE
```
POST /v1/memory/prune?confidenceThreshold=0.6&unusedDaysThreshold=90&dryRun=false
```
- Removes old, unused entries
- Credit charge: runtime configured
- Expected result: -15-30% utilization
If utilization still high, run DRY-RUN EXPAND to forecast
```
POST /v1/memory/expand?capacityIncrementMb=250&dryRun=true
```
- Validates expansion would help
- Credit charge: no charge for dry-run forecast when supported
- If helpful, execute full EXPAND using the current pricing matrix

Scenario: Burn Rate Spikes 2x

Symptoms:

/memory/stats shows burnRate: 100+ credits/hour (vs. usual 30-50)
Credit balance depleting faster than expected
No obvious new workload

Diagnosis Steps:

Check recent runs
```
GET /v1/memory/runs?limit=20
```
- Look for unexpected reindex, compact, or prune operations
- Check who initiated them and when
Analyze memory queries
```
GET /v1/memory/stats
```
- memory.retrievalCost: Should be stable month-to-month
- Spike here = more reads than usual
Review usage logs
```
GET /v1/memory/runs?limit=100
```
- Filter by action: READ and action: WRITE
- Group by principalId to find heavy user

Action Sequence:

If spike is from automated reindex/compact:
- Verify it was scheduled (expected)
- If not scheduled, disable automation temporarily
If spike is from increased read volume:
- Contact top users
- Ask if new integrations or processes started
- Offer optimization consultation
If spike is from increased write volume:
- Review for memory bloat pattern (see Pitfall #1 above)
- May indicate agent writing too aggressively
- Adjust agent write policies or prune low-value entries

Weekly Administration Tasks

Task 1: Inspect Low-Confidence Entries

Entries with confidence < 0.6 accumulate over time:

GET /v1/memory/entries?confidenceThreshold=0.0&confidenceMax=0.6&limit=50

Review: Do these deserve to stay? Examples:

❌ "Maybe use exponential backoff?" (vague, low confidence)
✅ "Use exponential backoff: 2^n * 100ms" (clear, high confidence)

Action: Delete low-value entries or update with more helpful text.

Task 2: Check for Duplicate Entries

Manually search for similar-looking memories:

GET /v1/memory/entries?query=payment%20retry&limit=20

Look for entries with nearly-identical content. COMPACT should remove some, but manual deduplication:

DELETE /v1/memory/entries/{id}  # Delete the older or less clear one

Credit charge: Runtime configured; see Pricing.

Task 3: Verify Encryption and ACLs

Spot-check that sensitive memories are encrypted:

GET /v1/memory/entries/{id}

Verify response includes:

encrypted: true
keyId: <valid key ID>
Appropriate principalId permissions

Task 4: Review Audit Log for Anomalies

Check for unusual patterns:

GET /v1/memory/runs?limit=100

Look for:

Bulk deletions (possible data loss)
Deletions by unexpected principals
Deletions without corresponding create events

Monthly Health Review

1. Capacity Planning

GET /v1/memory/stats

Analyze trends:

Entries growth: +100 entries/month? Sustainable?
Utilization trend: Gradually increasing? Compact more often.
Cost per entry: If rising, memory getting stale → prune more aggressively.

Recommendation: If utilization growth >5% MoM, plan EXPAND.

2. Hit Rate Analysis

GET /v1/memory/stats

Review memory.hitRate over past 30 days:

Stable >90%: Excellent. No action needed.
Declining (80-90%): Schedule monthly REINDEX.
Below 80%: Investigate root cause (see decision matrix above).

3. Cost Optimization

GET /v1/memory/runs?limit=100

Calculate:

Total credits used
Credits per entry (total used / current totalEntries)
ROI of operations: Did COMPACT save enough to justify cost?

Target: Establish a credit-per-active-memory baseline, then reduce it over time.

4. Communicate Health to Stakeholders

Create simple report:

GrayMatter Monthly Report - March 2026

📊 Metrics
- Hit Rate: 94% (↑ 2% from Feb, excellent)
- Context Utilization: 68% (stable, healthy range)
- Total Entries: 1,247 (↑ 3% from Feb)
- Total Credits Used: 450 (down 8% from Feb due to optimizations)

🎯 Actions Taken
- 1× REINDEX (successful, +2% hit rate)
- 2× COMPACT (recovered 180MB space)
- 1× PRUNE (removed 89 stale entries)

⚠️  No red flags. System healthy.

💡 Recommendations
- Monitor utilization; plan EXPAND in Q3 if growth continues
- Schedule monthly REINDEX to maintain hit rate

Incident Response

Incident: Memory Queries Returning Empty Results

Severity: High (agents can't access memories)

Response:

Verify authentication (5 min)

curl -H "Authorization: Bearer $TOKEN" \
  https://api-0.valkyrlabs.com/v1/memory/stats

If 401: Re-authenticate
If 403: Check ACL permissions

Check if entries exist (5 min)
```
GET /v1/memory/entries?limit=100
```
- If total: 0: Data loss; escalate to engineering
- If entries exist: Query issue; go to step 3
Verify query parameters (10 min)
- Try broad query: GET /v1/memory/entries?query=*&limit=10
- Try exact title match: GET /v1/memory/entries?query=exact-title-text
- Check if tags are correct
Run REINDEX if queries work but results poor (30 min)
```
POST /v1/memory/reindex?dryRun=false
```
Escalate if unresolved (>15 min)
- Contact engineering with:
  - Query examples
  - Expected vs. actual results
  - Last successful query time

Incident: Credits Exhausted / 402 Payment Required

Severity: Medium (workflows blocked)

Response:

Immediate (1 min)

POST /v1/credits/{accountId}/payment \
  -d '{"credits": 1000, "paymentMethod": "card"}'

Purchase emergency credits
Workflows resume immediately

Investigation (during purchasing)
```
GET /v1/memory/stats
```
- Check burnRate; is it abnormally high?
- Check memory.totalEntries; are we storing too much?
Prevention (after recovery)
- Set automatic top-up at a threshold based on observed burn rate
- Review burn rate and optimize if necessary
- Adjust pricing model if needed

Incident: Data Loss / Accidental Deletion

Severity: Critical (unrecoverable data)

Response:

STOP all operations immediately (1 sec)
- Halt any ongoing compact/prune/reindex
- Stop new writes to memory
Verify deletion (2 min)
```
GET /v1/memory/entries/{id}
```
- Returns 404? Entry is gone.
- Check audit log for deletion timestamp
Check for backups (5 min)
- ValkyrAI infrastructure maintains hourly backups
- Contact engineering to restore from backup
- Estimate: 30-120 min recovery time (depends on backup age)
Post-incident review
- Why was entry deleted? Human error? Bug?
- Implement safeguards (e.g., require confirmation for bulk delete)

Automation & Scheduled Tasks

Recommended Cron Schedules

# Daily at 2 AM UTC: Check health and auto-compact if needed
0 2 * * * curl -X POST \
  https://api-0.valkyrlabs.com/v1/memory/compact?dryRun=false

# Weekly (Sunday at 4 AM UTC): Full reindex
0 4 * * 0 curl -X POST \
  https://api-0.valkyrlabs.com/v1/memory/reindex?strategy=FULL

# Monthly (1st at 6 AM UTC): Prune low-confidence entries
0 6 1 * * curl -X POST \
  https://api-0.valkyrlabs.com/v1/memory/prune?confidenceThreshold=0.5

# Every 6 hours: Monitor and alert on burn rate
0 */6 * * * curl https://api-0.valkyrlabs.com/v1/memory/stats | \
  jq '.cost.burnRate' | xargs -I {} \
  [ {} -gt 100 ] && send_alert "High burn rate: {}"

Implementation (Python)

import schedule
import requests
from datetime import datetime

def auto_compact():
    resp = requests.post("https://api-0.valkyrlabs.com/v1/memory/compact",
                        headers={"Authorization": f"Bearer {TOKEN}"})
    if resp.status_code == 200:
        print(f"✅ Compact at {datetime.now()}: success")
    else:
        print(f"⚠️  Compact failed: {resp.status_code}")

def auto_reindex():
    resp = requests.post("https://api-0.valkyrlabs.com/v1/memory/reindex",
                        json={"strategy": "FULL"},
                        headers={"Authorization": f"Bearer {TOKEN}"})
    if resp.status_code == 200:
        print(f"✅ Reindex at {datetime.now()}: success")
    else:
        print(f"⚠️  Reindex failed: {resp.status_code}")

# Schedule tasks
schedule.every().day.at("02:00").do(auto_compact)
schedule.every().sunday.at("04:00").do(auto_reindex)

while True:
    schedule.run_pending()
    time.sleep(60)

Cost Control Best Practices

Set a monthly budget: Know your max acceptable spend.
Implement quotas: Limit per-agent memory writes (e.g., 10/day).
Tag ruthlessly: Narrow queries with precise tags.
Prune quarterly: Remove entries >180 days old and unused.
Monitor individual agents: Who's the top memory consumer?
Automate operations: Don't manually run compact/reindex—schedule them.

Scaling Considerations

From 1K to 100K Entries

Challenges:

Queries slow down (linear scan)
Indexing takes longer
Storage balloons

Solutions:

Add tags to everything (enables filtered queries)
Run COMPACT monthly → cuts entries by 20-30%
Implement retention policy (prune >180 days old)
Consider sharding by principalId if per-user memory grows large

From 1 Agent to 100 Agents

Challenges:

Contention on memory endpoints
Shared entries become stale for some agents

Solutions:

Use ACLs to isolate per-agent memories
Tag memories by agent (agent:valoride-1, agent:valoride-2)
Implement memory eviction: agents don't read >1 week old entries
Consider federated memory: each agent gets its own namespace

First Response: Run the Doctor​

Daily Monitoring Routine​

1. Check Memory Health (5 min)​

2. Review Recommendations​

3. Monitor Credits Balance​

Operational Decisions Matrix​

Scenario: Hit Rate Drops Below 80%​

Scenario: Context Utilization Exceeds 85%​

Scenario: Burn Rate Spikes 2x​

Weekly Administration Tasks​

Task 1: Inspect Low-Confidence Entries​

Task 2: Check for Duplicate Entries​

Task 3: Verify Encryption and ACLs​

Task 4: Review Audit Log for Anomalies​

Monthly Health Review​

1. Capacity Planning​

2. Hit Rate Analysis​

3. Cost Optimization​

4. Communicate Health to Stakeholders​

Incident Response​

Incident: Memory Queries Returning Empty Results​

Incident: Credits Exhausted / 402 Payment Required​

Incident: Data Loss / Accidental Deletion​

Automation & Scheduled Tasks​

Recommended Cron Schedules​

Implementation (Python)​

Cost Control Best Practices​

Scaling Considerations​

From 1K to 100K Entries​

From 1 Agent to 100 Agents​

First Response: Run the Doctor

Daily Monitoring Routine

1. Check Memory Health (5 min)

2. Review Recommendations

3. Monitor Credits Balance

Operational Decisions Matrix

Scenario: Hit Rate Drops Below 80%

Scenario: Context Utilization Exceeds 85%

Scenario: Burn Rate Spikes 2x

Weekly Administration Tasks

Task 1: Inspect Low-Confidence Entries

Task 2: Check for Duplicate Entries

Task 3: Verify Encryption and ACLs

Task 4: Review Audit Log for Anomalies

Monthly Health Review

1. Capacity Planning

2. Hit Rate Analysis

3. Cost Optimization

4. Communicate Health to Stakeholders

Incident Response

Incident: Memory Queries Returning Empty Results

Incident: Credits Exhausted / 402 Payment Required

Incident: Data Loss / Accidental Deletion

Automation & Scheduled Tasks

Recommended Cron Schedules

Implementation (Python)

Cost Control Best Practices

Scaling Considerations

From 1K to 100K Entries

From 1 Agent to 100 Agents