# Archive Parsing Re-Audit for Open Brain Memory

Date: 2026-05-22

## Reason for re-audit

Bert wants to try an Open Brain / OB1-style shared memory system because long-term AI partnerships require reliable memory. The goal is not merely archive search; the goal is a governed memory layer:

```text
OB1-inspired shared memory layer
+
Bert-specific schema/governance
+
archive provenance
```

## Current parser coverage

### Raw files observed

```text
.csv   138
.jpg    21
.json    4
.md   1311
.pdf     1
.png    31
.txt     4
```

### Normalized records after re-audit fixes

```text
chatgpt_conversations: 1450
claude_conversations:   88
all conversations:     1538
messages:              3336
chunks:                2214
```

### Source breakdown

```text
conversations:
  chatgpt:        1450
  claude:           83
  claude_memory:     1
  claude_project:    4

chunks:
  chatgpt:        1744
  claude:          461
  claude_memory:     2
  claude_project:    7
```

## Important fixes made

### 1. Chunk-level provenance

Before: conversation records had `raw_path`, but chunks did not carry direct raw-path provenance.

After: every chunk metadata object now includes:

- `raw_path`
- `conversation_created_at`
- `conversation_updated_at`
- source conversation ID
- message start/end range

This matters because Open Brain memories need traceable evidence.

### 2. Oversized chunk splitting

Before: at least one message/chunk was extremely large, unsuitable for promotion or embedding.

After: oversized single-message records are split into ~900-word chunks with:

- `split_reason: oversized_message`
- `part`
- `parts_total`

Current chunk size audit:

```text
chunks: 2214
word length min/median/p95/max: 3 / 85 / 901 / 916
missing raw_path: 0
```

### 3. Phrase-safe search

Before: SQLite FTS queries with punctuation, such as `Barely, But Here`, could fail.

After: normal search treats user input as a literal phrase and safely quotes it. Raw FTS syntax is still available with `--raw-fts`.

### 4. JSON metadata in SQLite

Before: metadata was stored as Python string representations.

After: metadata is stored as valid JSON strings in SQLite, making downstream MCP/OB1 integration cleaner.

### 5. Claude sidecar parsing

Before: only `raw/claude/conversations.json` was parsed.

After: parser also imports:

- `raw/claude/memories.json` as `claude_memory`
- `raw/claude/projects/*.json` descriptions/docs as `claude_project`

This was important. Claude memory/project data contains high-value long-term context, including Barely, But Here editorial standards and project docs.

## Notable recovered memory-grade material

Searches now retrieve durable context such as:

- Barely, But Here tagline and title conventions
- “not-fun places” as fixed/non-negotiable language
- editorial voice: staccato rhythm, California-casual register, minimal punctuation for effect, intentional profanity, raw honesty
- no self-help framing, redemption arcs, or inspirational tone
- content warning / disclaimer / support-resource conventions
- Claude project docs such as `Functional Depression: The Exhausting Art of Showing Up.md`

These are strong candidates for curated long-term memory, but they should still pass review before becoming approved Open Brain memories.

## Remaining limitations

### ChatGPT export ambiguity

The local `raw/chatgpt` directory behaves like a Notion-style markdown/document export, not a clean ChatGPT `conversations.json` transcript export. It is ingested as documents/pages, not as role-separated conversation transcripts.

This is acceptable for archive search and source excerpt retrieval, but it means ChatGPT-specific turn-level semantics are limited unless a separate canonical ChatGPT export JSON is supplied later.

### Media files are indexed only as paths

Images/PDF are not OCR’d or visually indexed yet. Current pipeline focuses on `.md`, `.txt`, and `.csv` text content plus Claude JSON.

Future improvement:

- OCR PDF/image assets
- attach extracted text as source documents
- store media provenance

### Memory candidates are not distilled yet

Current `scripts/export_memory_candidates.py` exports source-backed candidates from archive search results. It does not yet use an LLM to distill concise durable memories.

Recommended next step: add an LLM-assisted distillation pass with review states.

## Open Brain readiness verdict

The archive is now ready to support Open Brain implementation as a source-backed memory pipeline.

Recommended posture:

```text
Archive = searchable source of truth
Open Brain = curated, durable, governed memory
MCP = access layer for AI partners
```

Do not bulk-import every chunk into Open Brain. Promote reviewed memory candidates only.

## Files created/updated

- `src/ai_archive/chunking.py`
  - direct chunk provenance
  - oversized message splitting

- `src/ai_archive/indexing.py`
  - JSON metadata storage

- `src/ai_archive/importers.py`
  - Claude memory/project sidecar import

- `scripts/import_claude.py`
  - uses full Claude export importer

- `scripts/search.py`
  - phrase-safe search
  - provenance display

- `scripts/export_memory_candidates.py`
  - creates reviewable memory candidates from archive search

- `docs/openbrain-bert-memory-architecture.md`
  - Bert-specific Open Brain architecture/governance

- `docs/openbrain-bert-supabase-extension.sql`
  - OB1-compatible indexes/views for governed memory

## Recommended next implementation sequence

1. Install OB1 foundation: Supabase + pgvector + remote MCP server.
2. Apply `docs/openbrain-bert-supabase-extension.sql` after the base OB1 table exists.
3. Generate candidate memories from priority topics:
   - Barely, But Here
   - writing voice
   - offer / AI Systems Assessment
   - personal/professional identity
   - AI partnership preferences
4. Review candidates before inserting into Open Brain.
5. Add MCP tools that can search approved memory and retrieve source chunks separately.