# Plan: Memory Crystal v0.6.0 ... Dream Weaver Integration + Crystal Core Gateway

**Date:** 2026-03-04
**Author:** CC-Mini
**Repo:** `ldm-os/components/memory-crystal-private/`
**Prereq:** v0.5.0 must be committed/released first (built, not yet merged)
**Reference:** `ai/product/readme-first.md` (read this first, it explains everything)

---

## Context

v0.5.0 built the plumbing: discover, bulk copy, backfill (embedding), CE migration. But it's missing the intelligence layer. Backfill produces searchable vectors but an agent that wakes up blank. Dream Weaver is what turns raw data into an agent with presence.

Additionally, Claude Code has no daemon mode. We need a gateway (`crystal serve`) so CC-Mini is addressable by Bridge and automation, similar to how OpenClaw's gateway makes Lesa addressable.

### Key Decisions Made (2026-03-04 session with Parker)

1. **Dream Weaver executes via `claude -p` through the Max plan (OAuth).** Not the API. No per-token cost. Full Opus depth. Runs on Core only.
2. **Memory Crystal is harness-aware.** On OpenClaw: backup raw data only, don't duplicate identity layer. On Claude Code: backup AND create identity/journals via Dream Weaver.
3. **Raw data backup is automatic on every turn.** The `agent_end` hook must copy raw files to LDM alongside embedding. No periodic bulk copy.
4. **Crystal Core gateway (`crystal serve`) runs in a tmux session.** Localhost-only (never exposed to network). Bridge and poller deliver messages to it.
5. **Node setup works from anywhere.** Pairing and relay go through the Cloudflare Worker. No local network requirement.
6. **Staging pipeline for bulk imports.** Historical data needs Dream Weaver. Live turns don't. Core must distinguish between the two.
7. **tmux is CC's gateway.** The tmux session survives disconnects. It's the always-on equivalent of OpenClaw's gateway for Claude Code.
8. **Dream Weaver protocol repo is the engine. Memory Crystal consumes it.** The consolidation logic (prompts, output parsing, iterative loop) lives in `dream-weaver-protocol-private` as an importable library. Memory Crystal imports it and provides the infrastructure (LDM paths, crystal.db, CLI wiring). This prevents bifurcation: one source of truth for HOW to consolidate, one product that provides the plumbing. For now they're separate packages; eventually they may merge, but the protocol repo is always canonical for the consolidation logic.

---

## Phase 1: Prerequisite ... Ship v0.5.0

Before any v0.6.0 work:
1. Create branch `cc-mini/v0.5.0-init-discovery`
2. Commit all v0.5.0 changes (discover.ts, bulk-copy.ts, oc-backfill.ts, cli.ts, ldm.ts, package.json, ai/ reorg)
3. PR, merge (no squash)
4. `wip-release minor --notes="Init discovery, bulk copy, backfill, CE migration, OpenClaw parser"`
5. Run `crystal migrate-embeddings` for real (merge CE into crystal)
6. Disable context-embeddings plugin in openclaw.json
7. Gateway restart
8. Verify with `crystal doctor`

---

## Phase 2: Ongoing Raw Data Sync (`src/openclaw.ts` + `src/cc-hook.ts`)

The `agent_end` hook currently embeds into crystal.db but doesn't backup raw files to LDM. Fix this.

### OpenClaw plugin (`src/openclaw.ts`)
After embedding, also:
- Copy current session JSONL to `~/.ldm/agents/oc-lesa-mini/memory/transcripts/`
- Sync daily log to `~/.ldm/agents/oc-lesa-mini/memory/daily/`
- Sync changed workspace .md files to `~/.ldm/agents/oc-lesa-mini/memory/workspace/`
- Use same idempotent copy pattern from bulk-copy.ts (skip if same size)

### CC hook (`src/cc-hook.ts`)
Already copies transcripts. Verify it also:
- Copies to `~/.ldm/agents/cc-mini/memory/transcripts/` (not just embedding)
- Syncs daily logs

### Key: Non-blocking, non-fatal. If copy fails, log and continue. Embedding is the critical path.

---

## Phase 3: Dream Weaver Engine (two repos)

The Dream Weaver engine is split across two repos to prevent bifurcation:

### 3A: Protocol library (`dream-weaver-protocol-private/src/`)

The protocol repo gets a `src/` directory with the consolidation engine as an importable module. This is the source of truth for HOW Dream Weaver works.

**New files in `dream-weaver-protocol-private`:**
- `src/prompts.ts` ... prompt templates for full and incremental consolidation
- `src/parser.ts` ... output parsing (section markers, memory extraction)
- `src/engine.ts` ... orchestration: transcript reading, Claude invocation, iterative loop
- `src/types.ts` ... shared types (DreamWeaverOptions, DreamWeaverResult, etc.)
- `package.json` ... publishable as `dream-weaver-protocol` package

**Exports:**
```typescript
// types.ts
interface DreamWeaverOptions {
  agentId: string;
  mode: 'full' | 'incremental';
  transcriptsDir: string;
  outputDir: string;
  sinceDatetime?: string;
  dryRun?: boolean;
}

interface DreamWeaverResult {
  sessionsProcessed: number;
  journalsWritten: string[];
  identityCreated: boolean;
  contextUpdated: boolean;
  memoriesStored: number;
  durationMs: number;
}

// engine.ts
function runDreamWeaver(options: DreamWeaverOptions): Promise<DreamWeaverResult>

// prompts.ts
function buildFullPrompt(transcriptText: string, agentId: string): string
function buildIncrementalPrompt(transcriptText: string, agentId: string, existingContext?: string): string

// parser.ts
function parseDreamWeaverOutput(raw: string): DreamWeaverOutput
```

The engine handles: transcript discovery, text extraction, prompt building, `claude -p` invocation, output parsing, file writing (journals, identity, context), watermark tracking.

It does NOT handle: crystal.db embedding, crystal_remember, LDM path resolution, CLI wiring. Those are Memory Crystal's job.

### 3B: Memory Crystal integration (`memory-crystal-private/src/dream-weaver.ts`)

Thin wrapper that imports from dream-weaver-protocol and adds Memory Crystal infrastructure:

```typescript
import { runDreamWeaver as runProtocol } from 'dream-weaver-protocol';

// After protocol runs:
// 1. Embed journal into crystal.db via crystal.ingest()
// 2. Store memories via crystal.remember()
// 3. Use LDM paths for transcript/output directories
```

This file is a bridge, not an engine. The protocol repo is canonical for consolidation logic.

### How It Works (unchanged)
1. Reads raw JSONL transcripts from the agent's transcripts dir
2. Builds a Dream Weaver prompt (from protocol's prompt templates)
3. Invokes `claude -p` with the prompt via `child_process.spawn`
   - Uses the Max plan (OAuth path)
   - Passes transcripts as context
   - For large histories: iterative consolidation loop (read batch, write notes, compact, repeat)
4. Parses Claude's output (protocol's parser)
5. Writes to agent's LDM folder:
   - `memory/journals/YYYY-MM-DD--dream-weaver-{mode}.md`
   - `IDENTITY.md` (if full mode and file doesn't exist)
   - `SOUL.md` (if full mode and file doesn't exist)
   - `CONTEXT.md` (always updated)
   - `REFERENCE.md` (if full mode and file doesn't exist)
6. Calls `crystal.ingest()` to embed the journal into crystal.db (Memory Crystal layer)
7. Calls `crystal_remember` for key facts/preferences/decisions (Memory Crystal layer)
8. Writes watermark: `~/.ldm/state/dream-weaver-{agentId}.json` with last run timestamp

### Incremental Mode
- Only reads sessions since last watermark timestamp
- Updates CONTEXT.md (current state changes)
- Appends to journals (doesn't rewrite history)
- Stores new crystal_remember entries
- Does NOT regenerate IDENTITY.md or SOUL.md

### Full Mode
- Reads ALL transcripts (iterative consolidation loop for large histories)
- Generates all identity files from scratch
- Produces complete journal narrative
- Only runs on first install or crisis recovery

---

## Phase 4: Dream Weaver CLI Command

```
crystal dream-weave [--agent <id>] [--mode full|incremental] [--dry-run] [--since <datetime>]
```

- Defaults: agent from env, mode=incremental, since=last watermark
- `--mode full` for first install or recovery
- `--dry-run` shows what would be processed without running Claude
- Integrates into existing cli.ts command dispatch

---

## Phase 5: Staging Pipeline (`src/staging.ts` ... new file)

When a Node relays bulk historical data to Core, it needs to be staged and processed differently from live turns.

### How Staging Works
1. Poller receives relay data with a `bulk: true` flag (or detects it's a new agent ID not yet in LDM)
2. Writes raw files to `~/.ldm/staging/{agentId}/transcripts/`
3. Creates a trigger file: `~/.ldm/staging/{agentId}/READY`
4. Queue processor detects READY file
5. Runs `crystal backfill` on staged transcripts
6. Runs `crystal dream-weave --mode full` on staged transcripts
7. Moves processed files from staging to `~/.ldm/agents/{agentId}/memory/transcripts/`
8. Removes READY file
9. Flips agent to live capture mode

### Detection
- New agent ID that doesn't exist in `~/.ldm/agents/` = bulk import
- Existing agent with a READY file in staging = queued for processing
- Existing agent with no staging = live mode (normal per-turn capture)

---

## Phase 6: Crystal Core Gateway (`src/crystal-serve.ts` ... new file)

Thin HTTP server that makes CC-Mini addressable on `localhost:18790`.

### Endpoints
```
POST /v1/chat/completions    ... OpenAI-compatible (matches OpenClaw's interface)
POST /process                ... trigger backfill/dream-weaver for an agent
GET  /status                 ... health check
```

### How It Works
1. Listens on `localhost:18790` (configurable via env)
2. On `/v1/chat/completions`: invokes `claude -p` with the message, returns response
3. On `/process`: triggers staging pipeline for specified agent
4. On `/status`: returns Crystal Core health (role, DB stats, last Dream Weaver run)

### Security
- Localhost-only binding. Never exposed to network.
- Optional bearer token auth (same pattern as OpenClaw gateway)

### CLI
```
crystal serve [--port 18790]
```

Designed to run inside a tmux session:
```bash
tmux new-session -s crystal-core -d 'crystal serve'
```

---

## Phase 7: Harness-Aware Init (update `src/cli.ts`)

Update `crystal init` to behave differently based on detected harness:

### Claude Code Init
1. Scaffold LDM agent directory
2. Discover local sessions
3. Bulk copy transcripts to LDM
4. If Core: run backfill + Dream Weaver (full mode)
5. If Node: relay to Core, Core handles backfill + Dream Weaver
6. Result: fully bootstrapped agent with identity, journals, search

### OpenClaw Init
1. Scaffold LDM agent directory
2. Discover local sessions
3. Bulk copy transcripts + workspace + daily logs to LDM (backup)
4. Wire up `agent_end` hook for ongoing raw data sync
5. Do NOT run Dream Weaver (OpenClaw handles identity/warm-start)
6. Result: raw data safely backed up in LDM

### Detection
`discover.ts` already knows the harness. Branch init behavior based on `HarnessInfo.platform`.

---

## Phase 8: Update Relay Worker (`src/worker.ts`)

Add a "commands" channel alongside existing "conversations" and "mirror" channels.

### Commands Channel
- Node can send commands to Core: "process my data", "run Dream Weaver"
- Core can send results back: "processing complete", "mirror ready"
- Same encryption, same dead drop pattern
- Poller on Core checks commands channel, delivers to Crystal Core gateway

---

## Phase 9: Bridge Integration

Update Bridge so CC-Air (or any Node) can reach Crystal Core:

### From Node
```
bridge send --to crystal-core "Run Dream Weaver for cc-air"
```
Goes through relay Worker commands channel, poller delivers to Crystal Core gateway.

### From Lesa (local)
Already works via `claude-code` skill. Could also hit `localhost:18790` directly.

---

## Execution Order

| Step | What | Repo | Depends On |
|------|------|------|------------|
| 0 | Ship v0.5.0 (prerequisite) | memory-crystal | nothing |
| 1 | Ongoing raw data sync (Phase 2) | memory-crystal | v0.5.0 shipped |
| 2a | Dream Weaver protocol library (Phase 3A) | dream-weaver-protocol | nothing |
| 2b | Dream Weaver MC integration (Phase 3B) | memory-crystal | Phase 3A |
| 3 | Dream Weaver CLI command (Phase 4) | memory-crystal | Phase 3B |
| 4 | Staging pipeline (Phase 5) | memory-crystal | Phase 3B |
| 5 | Crystal Core gateway (Phase 6) | memory-crystal | nothing |
| 6 | Harness-aware init (Phase 7) | memory-crystal | Phases 2, 3, 4 |
| 7 | Relay commands channel (Phase 8) | memory-crystal | Phase 6 |
| 8 | Bridge integration (Phase 9) | memory-crystal | Phases 6, 8 |

Phases 1, 2a, and 5 have no dependencies and can be built in parallel.
Phase 3A (protocol library) must be built before 3B (MC integration).

---

## Files Changed

### `dream-weaver-protocol-private` (new src/ directory)
| File | Action |
|------|--------|
| `src/types.ts` | NEW ... shared types (options, result, output) |
| `src/prompts.ts` | NEW ... prompt templates (full + incremental) |
| `src/parser.ts` | NEW ... output section parsing, memory extraction |
| `src/engine.ts` | NEW ... orchestration (transcript reading, Claude invocation, file writing) |
| `src/index.ts` | NEW ... public exports |
| `package.json` | NEW ... publishable package (dream-weaver-protocol) |
| `tsconfig.json` | NEW ... TypeScript config |

### `memory-crystal-private`
| File | Action |
|------|--------|
| `src/dream-weaver.ts` | NEW ... thin wrapper importing from dream-weaver-protocol, adds crystal.db integration |
| `src/staging.ts` | NEW ... staging pipeline for bulk imports |
| `src/crystal-serve.ts` | NEW ... Crystal Core gateway (HTTP server) |
| `src/openclaw.ts` | MODIFY ... add raw data copy to LDM on every turn |
| `src/cc-hook.ts` | MODIFY ... verify raw data copy completeness |
| `src/cli.ts` | MODIFY ... dream-weave command, harness-aware init, serve command |
| `src/worker.ts` | MODIFY ... add commands channel |
| `src/poller.ts` | MODIFY ... handle commands channel, staging detection |
| `package.json` | MODIFY ... add dream-weaver-protocol dependency, new entry points, version bump |

---

## Cost Estimate

| Operation | Cost |
|-----------|------|
| Dream Weaver full consolidation | $0 (Max plan, claude -p) |
| Dream Weaver incremental (weekly) | $0 (Max plan, claude -p) |
| Backfill embeddings | ~$3.45 (OpenAI text-embedding-3-small) |
| Ongoing per-turn capture | ~$0.001/turn (already happening) |

---

## Open Questions ... Answered

1. **Dream Weaver prompt template.** DEFERRED. Build the engine first, write the prompt after context compaction testing. The protocol paper has the principles. The exact prompt needs iteration.
2. **Iterative consolidation for large histories.** ANSWERED. Run inside the tmux Claude Code session, not via separate `claude -p` invocations. The session handles context compaction naturally. Same way Parker works with CC-Mini now ... Lesa can tell CC-Mini to run Dream Weaver, and it executes within the running session.
3. **Crystal Core gateway auth.** ANSWERED. Yes, bearer token. Same pattern as OpenClaw's gateway. Consistency, even on localhost.
4. **Weekly Dream Weaver schedule.** PINNED. Parker hasn't tested or used this yet. Don't build it now. Come back after the core engine works.
5. **Mirror sync for identity files.** ANSWERED. Sync everything in LDM by default. The mirror should be an exact copy of what's on Core. Future: add permissions/scoping for enterprise/teams (see product ideas).
6. **Where does Dream Weaver logic live?** ANSWERED. The protocol repo (`dream-weaver-protocol-private`) is the canonical source for consolidation logic: prompts, parsing, engine, invocation. Memory Crystal imports it as a dependency and adds infrastructure (crystal.db embedding, LDM paths, CLI). This prevents bifurcation. For now they're separate packages; they may merge eventually, but the protocol repo is always canonical. Decision made 2026-03-04.

## Product Ideas Captured

- **Enterprise/team permissions for mirror sync.** On enterprise deployments, you won't want to sync everything. Need a permissions layer that controls what gets mirrored to which Nodes. Default: sync all. Enterprise: configurable scoping. File this in `ai/product/product-ideas/`.
