# Dev Update: v0.5.0 ... Init Discovery + Raw Data Safety + Memory Consolidation

**Date:** 2026-03-04 00:30 PST
**Author:** CC-Mini
**Version:** 0.5.0 (built, not yet released)

## What Happened

### 1. Raw Data Bulk Copy (DONE)

All raw session data is now safely copied to LDM:

| Data | Source | LDM | Status |
|------|--------|-----|--------|
| Lesa session JSONLs | 748 (154MB) | 748 | Complete |
| Lesa workspace MDs | 133 | 133 | Complete |
| CC session JSONLs | 444 | 476 | Complete |

LDM grew from 5.8GB to 6.4GB. The raw files are the ground truth. Everything else is an index.

### 2. Three New Source Files

- **`src/discover.ts`** ... detects Claude Code and OpenClaw installations on the current machine, returns session file paths. Per-harness only (installing on Air only finds Air's sessions).
- **`src/bulk-copy.ts`** ... copies raw session files from source to `~/.ldm/agents/{id}/memory/transcripts/`. Idempotent (skips existing same-size files). Also handles workspace .md files.
- **`src/oc-backfill.ts`** ... parses OpenClaw JSONL format (different from Claude Code). Same `ExtractedMessage` output as cc-poller.

### 3. Three New CLI Commands

**`crystal backfill [--agent <id>] [--dry-run] [--limit <n>]`**
- Scans `~/.ldm/agents/{id}/memory/transcripts/*.jsonl`
- Auto-detects format (Claude Code vs OpenClaw)
- Extracts messages, embeds into crystal.db
- Core/Standalone: embeds locally
- Node: placeholder for relay (not yet implemented)
- Dry run tested:
  - cc-mini: 32 files, 16,600 messages, $0.05 est.
  - oc-lesa-mini: 748 files, 24,930 messages, $0.19 est.

**`crystal migrate-embeddings [--dry-run]`**
- Reads context-embeddings.sqlite (read-only, never modified)
- For each chunk: SHA-256 hash, check if crystal has it, insert if unique
- Copies embedding blob directly (same model: text-embedding-3-small, 1536d)
- Zero API calls, zero cost
- Backs up crystal.db before any writes
- Dry run result: 15,618 unique chunks would migrate, 161 duplicates skipped

**Updated `crystal init`**
- After scaffold, runs session discovery
- Shows found files by platform with sizes
- Offers bulk copy to LDM (Y/n prompt, or `--yes` for non-interactive)
- Suggests next step (backfill) based on role
- `--skip-discover` flag to skip

### 4. LDM Updates

- Added `workspace` to `LdmPaths` interface and `scaffoldLdm()`
- Path: `~/.ldm/agents/{id}/memory/workspace/`
- Fixed boolean flag parsing (`--yes`, `--skip-discover`, `--include-secrets`)

### 5. Memory Architecture Investigation

Full investigation written to `ai/plan/mem-consolidation/`:
- `investigation.md` ... all three databases documented, overlap analysis, migration plan
- `raw-data-gaps.md` ... what LDM was missing (~1,292 files, ~813MB)

Key findings:
- Three vector databases: main.sqlite (6.1GB, OpenClaw built-in), context-embeddings.sqlite (124MB, our plugin), crystal.db (1.6GB, our product)
- All use text-embedding-3-small, 1536d, float32. Embeddings directly compatible.
- CE has ~15,618 unique chunks not in crystal
- main.sqlite is OpenClaw core (not ours to consolidate)
- CE is ours to consolidate into crystal

## What's NOT Done Yet

### CE Migration (ready to execute)
- `crystal migrate-embeddings` is built and dry-run tested
- **Has NOT been run for real yet**
- Need to run it, verify, then disable CE plugin in openclaw.json

### CE Plugin Disable (waiting on migration)
- Remove `context-embeddings` from `openclaw.json` plugins list
- Gateway restart to pick up the change
- CE database stays on disk as backup (never deleted)
- Can re-enable by adding it back to openclaw.json

### lesa-bridge Update (separate task)
- `lesa_conversation_search` currently searches CE
- After CE is frozen, either point it at crystal.db or remove it
- `lesa_memory_search` stays regardless

### Node Relay Backfill (future)
- Backfill command has placeholder for Node mode
- When running on Air as Node, should relay to Core instead of local embed
- Not blocking... Air install comes after this is stable on Mini

### wip-release + PR
- v0.5.0 built but not committed/PRed/released
- Need branch, PR, merge, wip-release

## Files Changed

| File | Action |
|------|--------|
| `src/discover.ts` | NEW |
| `src/bulk-copy.ts` | NEW |
| `src/oc-backfill.ts` | NEW |
| `src/ldm.ts` | MODIFIED (workspace path) |
| `src/cli.ts` | MODIFIED (3 commands, init discovery, flag fixes) |
| `package.json` | MODIFIED (v0.5.0, new entry points) |
| `ai/plans/2026-03-04--cc-mini--init-discovery-and-consolidation.md` | NEW |
| `ai/plan/mem-consolidation/investigation.md` | NEW |
| `ai/plan/mem-consolidation/raw-data-gaps.md` | NEW |

## Test Results

```
crystal init --skip-discover         OK (scaffold only)
crystal init --yes                   OK (scaffold + discover + copy)
crystal backfill --dry-run           OK (32 CC files, $0.05)
crystal backfill --agent oc-lesa-mini --dry-run  OK (748 OC files, $0.19)
crystal migrate-embeddings --dry-run OK (15,618 unique, 161 dupes, $0.00)
crystal doctor                       OK (170,568 chunks, 1 failure: no API key in env)
npm run build                        OK (ESM + DTS clean)
```

## Decision Log

- **Init is per-harness.** Installing on CC-Air only finds CC-Air sessions. Not global.
- **Dream Weaver runs on Core only.** Node sends raw files to Core via relay.
- **CE migration copies embeddings directly.** Same model, same dimensions, zero cost.
- **Raw files are ground truth.** Databases are indexes. LDM must have copies of everything.
- **CE stays on disk forever.** Never deleted. Insurance policy.
