# Reusable Agent-Memory-Move Runbook

## Purpose

This runbook documents the repeatable process for moving a person’s AI/archive history into a governed, provenance-backed AI memory system.

It is based on Bert’s first `ai-archive` / Open Brain preparation pass and is intentionally written so we can do it again with more precision.

## Core principle

Do not confuse archive search with durable memory.

```text
Raw archive = evidence and source material
Local index = searchable retrieval layer
Candidate memory = proposed long-term memory
Approved memory = reviewed partnership substrate
Current session = temporary working surface
```

The archive can be large. Memory should be selective.

---

# Phase 0 — Define scope before parsing

## Goal

Prevent the system from becoming a dump of everything.

## Inputs

- User’s explicit must-keep domains
- Any protected/private boundaries
- Desired review rules
- Target memory system: Hermes native memory, Open Brain/OB1, MCP, vector DB, SQLite FTS, etc.

## Output

A scope/governance document.

For Bert’s current pass:

```text
docs/memory-scope-and-review-governance.md
```

## Bert-specific first-install scope

1. Barely, But Here and all article drafts across backups
2. Bert identity / professional positioning / writing voice
3. AI Systems Assessment campaign for urgently needed income

## Governance rule

Personal memory candidates must be reviewed by Bert before they are approved or installed.

---

# Phase 1 — Preserve raw sources unchanged

## Goal

Create a stable source of truth.

## Pattern

```text
raw/
  chatgpt/
  claude/
  notion/
  substack/
  other/
normalized/
index/
scripts/
docs/
review/
approved-memory/
```

## Rules

- Never edit raw exports.
- Ignore OS metadata like `.DS_Store`.
- Record export source and date when known.
- Treat sidecar files as potentially valuable, not noise.

## Bert-specific source caveat

Claude had clean conversation JSON plus sidecars:

```text
raw/claude/conversations.json
raw/claude/memories.json
raw/claude/projects/*.json
raw/claude/users.json
```

ChatGPT/OpenAI expanded export looked more like a Notion-style markdown/doc archive than simple role-based chat transcripts.

---

# Phase 2 — Inspect export shapes before coding assumptions

## Goal

Avoid parser hallucinations and bad schema assumptions.

## Checks

- File counts by extension
- Known raw files and folder names
- JSON root type: array/object/lines
- Message structure and role fields
- Markdown/doc structure
- Attachment/media presence
- Sidecar export files
- Unicode line separator or paragraph separator characters

## Bert-specific discoveries

- Claude main export: clean JSON array of conversations.
- Claude sidecars contained high-value memory/project context.
- ChatGPT side: heterogeneous markdown/docs/assets, not pure conversation transcript.
- Unicode line separators could break naive JSONL reading with `.splitlines()`.

---

# Phase 3 — Normalize to durable intermediate files

## Goal

Create source-neutral records that future tools can trust.

## Outputs

```text
normalized/chatgpt_conversations.jsonl
normalized/claude_conversations.jsonl
normalized/conversations.jsonl
normalized/messages.jsonl
```

## Minimum conversation/document fields

```json
{
  "id": "stable id",
  "source": "claude | chatgpt | notion | substack | ...",
  "title": "...",
  "created_at": "...",
  "updated_at": "...",
  "metadata": {
    "raw_path": "...",
    "source_specific_ids": "..."
  }
}
```

## Minimum message/section fields

```json
{
  "id": "stable id",
  "conversation_id": "...",
  "role": "user | assistant | system | document | memory | project",
  "text": "...",
  "created_at": "...",
  "metadata": {}
}
```

## JSONL safety rules

- Write one JSON object per physical newline.
- Sanitize `\u2028` and `\u2029` before writing JSONL.
- Read JSONL with file iteration, not `.splitlines()`.
- Store metadata as valid JSON, not Python `str(dict)`.

---

# Phase 4 — Chunk for retrieval, not memory

## Goal

Create precise search units that still point back to source evidence.

## Outputs

```text
normalized/chunks.jsonl
index/archive.db
```

## Chunk requirements

Every chunk must include:

```json
{
  "chunk_id": "...",
  "conversation_id": "...",
  "source": "...",
  "title": "...",
  "text": "...",
  "message_start": 1,
  "message_end": 3,
  "metadata": {
    "raw_path": "...",
    "split_reason": "oversized_message | normal",
    "part": 1,
    "parts_total": 4
  }
}
```

## Quality checks

- No missing `raw_path`.
- No giant chunks.
- Search works for punctuation-heavy phrases.
- Known important phrases return expected results.
- SQLite index tables contain valid JSON metadata.

## Bert-specific verified examples

- `Barely, But Here`
- `not-fun places`
- `staccato rhythm`
- `AI Systems Assessment`

---

# Phase 5 — Re-audit parsing before memory promotion

## Goal

Make sure the parser is good enough for long-term memory, not just search.

## Required audit questions

1. Are all source types represented?
2. Are sidecars imported?
3. Are raw paths carried to chunks?
4. Are chunks bounded?
5. Are sensitive/private categories identifiable?
6. Are project domains separable?
7. Can we reconstruct the original source from a candidate?
8. Are search results precise enough to generate candidates?

## Bert-specific issue fixed

Initial chunks did not carry direct raw-path provenance. That was patched before memory candidate generation.

---

# Phase 6 — Generate memory candidates inside agreed scope only

## Goal

Create reviewable proposals, not installed memory.

## Candidate packet location

```text
review/*.md
```

## Bert-specific packets

```text
review/barely-but-here-memory-candidates.md
review/bert-identity-voice-memory-candidates.md
review/ai-systems-assessment-memory-candidates.md
```

## Review decisions

```text
APPROVE
EDIT
REJECT
SOURCE_ONLY
PENDING
```

## Important distinction

`SOURCE_ONLY` means the material is important and must remain retrievable, but should not become durable memory.

This is especially important for drafts.

---

# Phase 7 — Human review gate

## Goal

Protect personal memory from silent installation.

## Process

1. User reviews packet.
2. User marks each candidate.
3. Export script separates approved/source-only/rejected/edit/pending.
4. Assistant inspects approved set before install.
5. Nothing enters long-term memory without explicit approval.

## Bert’s current status at time of this runbook

Bert has reviewed 2 of 3 packets and plans to finish `bert-identity-voice` in the morning.

Do not assume all candidates are approved until the export script confirms it.

---

# Phase 8 — Export reviewed decisions

## Command

```bash
cd /Users/bertmahoney/workspace/ai-archive
python3 scripts/export_review_decisions.py
```

## Outputs

```text
approved-memory/approved-memories.jsonl
approved-memory/source-only-index.jsonl
approved-memory/rejected-memory-log.jsonl
approved-memory/needs-edit-memory-candidates.jsonl
approved-memory/pending-memory-candidates.jsonl
```

## Pre-install checks

- Count approved memories.
- Count pending memories.
- Inspect sensitive/restricted items.
- Confirm provenance fields exist.
- Confirm no raw credentials/secrets are present.
- Confirm `SOURCE_ONLY` items are not being treated as approved memory.

---

# Phase 9 — Install into memory substrate

## Preferred architecture

```text
Raw archive / local FTS
  +
Approved memory / Open Brain
  +
MCP tools
  +
Hermes native compact memory
```

## What goes where

| Layer | Contents |
|---|---|
| Hermes native memory | Small durable facts/preferences that should be in every session |
| Open Brain / Supabase | Approved structured memory with metadata/provenance |
| Local archive index | Large source retrieval, drafts, evidence |
| Current chat | Temporary working context |

## Do not

- bulk import all chunks into Open Brain as memory
- install unreviewed candidates
- use approved memory without source metadata
- treat every draft paragraph as durable memory

---

# Phase 10 — Connect via MCP and verify runtime behavior

## Goal

Make memory available to AI partners through tools rather than prompt stuffing.

## Useful MCP tools

```text
search_approved_memories(query, project?, memory_type?)
get_memory(memory_id)
search_archive(query, source?, project?)
get_source_chunk(chunk_id)
propose_memory_from_archive(chunk_id, rationale)
list_memory_candidates(status)
approve_memory(candidate_id)
reject_memory(candidate_id, reason)
```

## Restart/reload rule

Use a new Hermes session or MCP reload after adding/changing MCP servers or toolsets.

Do not restart because the archive needs to be “loaded.” The archive should remain queryable, not stuffed into context.

---

# Phase 11 — Produce diagrams and narrative documentation

## Goal

Make the system understandable to Bert and reusable as a public/private essay/process artifact.

## Current diagrams

```text
docs/hermes-openbrain-memory-workflow.svg
docs/hermes-openbrain-memory-workflow-by-location.svg
```

## Current narrative docs

```text
docs/memory-review-workflow.md
docs/openbrain-bert-memory-architecture.md
docs/archive-parsing-reaudit-openbrain.md
docs/memory-scope-and-review-governance.md
docs/reusable-agent-memory-move-runbook.md
```

---

# Precision improvements for next run

## Before ingest

- Create a source inventory with export date, export tool, and canonical purpose.
- Create a `sources.yaml` manifest.
- Assign stable source IDs before parsing.
- Define project/domain taxonomy before candidate generation.
- Define sensitivity taxonomy before candidate generation.

## During parsing

- Store parser version in every normalized record.
- Store content hash for every raw file and every chunk.
- Keep an import log with warnings and skipped files.
- Add unit tests for each export shape.
- Add fixture samples from Claude, ChatGPT, Notion-style markdown, and sidecars.

## During candidate generation

- Separate candidate types more carefully:
  - stable fact
  - project rule
  - writing voice rule
  - source excerpt
  - strategy
  - offer/campaign detail
  - preference
  - procedure
- Generate shorter distilled candidate content, plus source excerpt separately.
- Add `why_candidate_matters` and `risk_if_wrong` fields.
- Add `suggested_layer`: Hermes native, Open Brain, source-only, archive-only.

## During review

- Use a structured review UI or CSV/JSON review table, not only Markdown.
- Allow bulk actions for obvious `SOURCE_ONLY` draft excerpts.
- Require explicit approval for sensitive/restricted items.
- Keep an audit log of who approved what and when.

## During install

- Dry-run import first.
- Install to staging memory table first.
- Query-test approved memories before enabling globally.
- Keep rollback/export scripts.

---

# Repeatable command checklist

```bash
cd /Users/bertmahoney/workspace/ai-archive

# 1. Import sources
python3 scripts/import_chatgpt.py
python3 scripts/import_claude.py

# 2. Normalize and chunk
python3 scripts/normalize.py
python3 scripts/chunk.py

# 3. Build index
python3 scripts/build_index.py

# 4. Smoke test search
python3 scripts/search.py 'Barely, But Here' --limit 3
python3 scripts/search.py 'not-fun places' --limit 3
python3 scripts/search.py 'AI Systems Assessment' --limit 3

# 5. Build review packets
python3 scripts/build_memory_review_packets.py --domain all --per-query-limit 8

# 6. After Bert reviews packets
python3 scripts/export_review_decisions.py
```

---

# Final reminder

The winning architecture is not “AI remembers everything.”

The winning architecture is:

```text
AI can search everything,
but only carries forward what Bert approved as durable memory.
```