codex/codex-rs/memories/README.md

Memories

This directory owns reusable memory crates and the memory pipeline documentation.

Runtime orchestration for Phase 1 and Phase 2 still lives in codex-core under codex-rs/core/src/memories/.

Crates

• codex-rs/memories/read (codex-memories-read) owns the read path: memory developer-instruction injection, memory citation parsing, and read-usage telemetry classification.
• codex-rs/memories/write (codex-memories-write) owns the write path: Phase 1 and Phase 2 prompt rendering, filesystem artifact helpers, workspace diff helpers, and extension resource pruning.

Prompt Templates

Memory prompt templates live with the crate that uses them:

• The undated template files are the canonical latest versions used at runtime:
  • read/templates/memories/read_path.md
  • write/templates/memories/stage_one_system.md
  • write/templates/memories/stage_one_input.md
  • write/templates/memories/consolidation.md
• In codex, edit those undated template files in place.
• The dated snapshot-copy workflow is used in the separate openai/project/agent_memory/write harness repo, not here.

When it runs

The pipeline is triggered when a root session starts, and only if:

• the session is not ephemeral
• the memory feature is enabled
• the session is not a sub-agent session
• the state DB is available

It runs asynchronously in the background and executes two phases in order: Phase 1, then Phase 2.

Phase 1: Rollout Extraction (per-thread)

Phase 1 finds recent eligible rollouts and extracts a structured memory from each one.

Eligible rollouts are selected from the state DB using startup claim rules. In practice this means the pipeline only considers rollouts that are:

• from allowed interactive session sources
• within the configured age window
• idle long enough (to avoid summarizing still-active/fresh rollouts)
• not already owned by another in-flight phase-1 worker
• within startup scan/claim limits (bounded work per startup)

What it does:

• claims a bounded set of rollout jobs from the state DB (startup claim)
• filters rollout content down to memory-relevant response items
• sends each rollout to a model (in parallel, with a concurrency cap)
• expects structured output containing:
  • a detailed raw_memory
  • a compact rollout_summary
  • an optional rollout_slug
• redacts secrets from the generated memory fields
• stores successful outputs back into the state DB as stage-1 outputs

Concurrency / coordination:

• Phase 1 runs multiple extraction jobs in parallel (with a fixed concurrency cap) so startup memory generation can process several rollouts at once.
• Each job is leased/claimed in the state DB before processing, which prevents duplicate work across concurrent workers/startups.
• Failed jobs are marked with retry backoff, so they are retried later instead of hot-looping.

Job outcomes:

• succeeded (memory produced)
• succeeded_no_output (valid run but nothing useful generated)
• failed (with retry backoff/lease handling in DB)

Phase 1 is the stage that turns individual rollouts into DB-backed memory records.

Phase 2: Global Consolidation

Phase 2 consolidates the latest stage-1 outputs into the filesystem memory artifacts and then runs a dedicated consolidation agent.

What it does:

• claims a single global phase-2 lock before touching the memories root (so only one consolidation inspects or mutates the workspace at a time)
• loads a bounded set of stage-1 outputs from the state DB using phase-2 selection rules:
  • ignores memories whose last_usage falls outside the configured max_unused_days window
  • for memories with no last_usage, falls back to generated_at so fresh never-used memories can still be selected
  • ranks eligible memories by usage_count first, then by the most recent last_usage / generated_at
• computes a completion watermark from the claimed watermark + newest input timestamps
• syncs local memory artifacts under the memories root:
  • raw_memories.md (merged raw memories, stable ascending thread-id order)
  • rollout_summaries/ (one summary file per selected rollout)
• keeps the memories root itself as a git-baseline directory, initialized under ~/.codex/memories/.git by codex-git-utils
• prunes stale rollout summaries that are no longer selected
• prunes memory extension resource files older than the extension retention window, so cleanup appears in the workspace diff
• writes phase2_workspace_diff.md in the memories root with the git-style diff from the previous successful Phase 2 baseline to the current worktree
• if the memory workspace has no changes after artifact sync/pruning, marks the job successful and exits

If the memory workspace has changes, it then:

• spawns an internal consolidation sub-agent
• builds the Phase 2 prompt with the path to the generated workspace diff
• points the agent at phase2_workspace_diff.md for the detailed diff context
• runs it with no approvals, no network, and local write access only
• disables collab for that agent (to prevent recursive delegation)
• watches the agent status and heartbeats the global job lease while it runs
• resets the memory git baseline after the agent completes successfully; the generated diff file is removed before this reset so deleted content is not kept in the prompt artifact or unreachable git objects
• marks the phase-2 job success/failure in the state DB when the agent finishes

Selection and workspace-diff behavior:

• successful Phase 2 runs mark the exact stage-1 snapshots they consumed with selected_for_phase2 = 1 and persist the matching selected_for_phase2_source_updated_at
• Phase 1 upserts preserve the previous selected_for_phase2 baseline until the next successful Phase 2 run rewrites it
• Phase 2 loads only the current top-N selected stage-1 inputs, syncs rollout_summaries/ directly to that selection, renders raw_memories.md in stable ascending thread-id order to avoid usage-rank churn, then lets the git-style workspace diff surface additions, modifications, and deletions against the previous successful memory baseline
• when the selected input set is empty, stale rollout_summaries/ files are removed and raw_memories.md is rewritten to the empty-input placeholder; consolidated outputs such as MEMORY.md, memory_summary.md, and skills/ are left for the agent to update

Watermark behavior:

• The global phase-2 lock does not use DB watermarks as a dirty check; git workspace dirtiness decides whether an agent needs to run.
• The global phase-2 job row still tracks an input watermark as bookkeeping for the latest DB input timestamp known when the job was claimed.
• Phase 2 recomputes a new_watermark using the max of:
  • the claimed watermark
  • the newest source_updated_at timestamp in the stage-1 inputs it actually loaded
• On success, Phase 2 stores that completion watermark in the DB.
• This avoids moving the recorded completion watermark backwards, but does not decide whether Phase 2 has work.

In practice, this phase is responsible for refreshing the on-disk memory workspace and producing/updating the higher-level consolidated memory outputs.

Why it is split into two phases

• Phase 1 scales across many rollouts and produces normalized per-rollout memory records.
• Phase 2 serializes global consolidation so the shared memory artifacts are updated safely and consistently.