mirror of
https://github.com/openai/codex.git
synced 2026-05-15 08:42:34 +00:00
01ab25dbb5
## Why This PR make the `morpheus` agent (memory phase 2) use a git diff to start it's consolidation. The workflow is the following: 1. The agent acquire a lock 2. If `.codex/memories` does not exist or is not a git root, initialize everything (and make a first empty commit) 3. Update `raw_memories.md` and `rollout_summaries/` as before. Basically we select max N phase 1 memories based on a given policy 4. We use git (`gix`) to get a diff between the current state of `.codex/memories` and the last commit. 5. Dump the diff in `phase2_workspace_diff.md` 6. Spawn `morpheus` and point it to `phase2_workspace_diff.md` 7. Wait for `morpheus` to be done 8. Re-create a new `.git` and make one single commit on it. We do this because we don't want to preserve history through `.git` and this is cheap anyway 9. We release the lock On top of this, we keep the retry policies etc etc The goals of this new workflow are: * Better support of any memory extensions such as `chronicle` * Allow the user to manually edit memories and this will be considered by the phase 2 agent As a follow-up we will need to add support for user's edition while `morpheus` is running ## What Changed - Added memory workspace helpers that prepare the git baseline, compute the diff, write `phase2_workspace_diff.md`, and reset the baseline after successful consolidation. - Updated Phase 2 to sync current inputs into `raw_memories.md` and `rollout_summaries/`, prune old extension resources, skip clean workspaces, and run the consolidation subagent only when the workspace has changes. - Tightened Phase 2 job ownership around long-running consolidation with heartbeats and an ownership check before resetting the baseline. - Simplified the prompt and state APIs so DB watermarks are bookkeeping, while workspace dirtiness decides whether consolidation work exists. - Updated the memory pipeline README and tests for workspace diffs, extension-resource cleanup, pollution-driven forgetting, selection ranking, and baseline persistence. ## Verification - Added/updated coverage in `core/src/memories/tests.rs`, `core/src/memories/workspace_tests.rs`, `state/src/runtime/memories.rs`, and `core/tests/suite/memories.rs`. --------- Co-authored-by: Codex <noreply@openai.com>
75 lines
2.5 KiB
Rust
75 lines
2.5 KiB
Rust
//! SQLite-backed state for rollout metadata.
|
|
//!
|
|
//! This crate is intentionally small and focused: it extracts rollout metadata
|
|
//! from JSONL rollouts and mirrors it into a local SQLite database. Backfill
|
|
//! orchestration and rollout scanning live in `codex-core`.
|
|
|
|
mod extract;
|
|
pub mod log_db;
|
|
mod migrations;
|
|
mod model;
|
|
mod paths;
|
|
mod runtime;
|
|
|
|
pub use model::LogEntry;
|
|
pub use model::LogQuery;
|
|
pub use model::LogRow;
|
|
pub use model::Phase2JobClaimOutcome;
|
|
/// Preferred entrypoint: owns configuration and metrics.
|
|
pub use runtime::StateRuntime;
|
|
|
|
/// Low-level storage engine: useful for focused tests.
|
|
///
|
|
/// Most consumers should prefer [`StateRuntime`].
|
|
pub use extract::apply_rollout_item;
|
|
pub use extract::rollout_item_affects_thread_metadata;
|
|
pub use model::AgentJob;
|
|
pub use model::AgentJobCreateParams;
|
|
pub use model::AgentJobItem;
|
|
pub use model::AgentJobItemCreateParams;
|
|
pub use model::AgentJobItemStatus;
|
|
pub use model::AgentJobProgress;
|
|
pub use model::AgentJobStatus;
|
|
pub use model::Anchor;
|
|
pub use model::BackfillState;
|
|
pub use model::BackfillStats;
|
|
pub use model::BackfillStatus;
|
|
pub use model::DirectionalThreadSpawnEdgeStatus;
|
|
pub use model::ExtractionOutcome;
|
|
pub use model::SortDirection;
|
|
pub use model::SortKey;
|
|
pub use model::Stage1JobClaim;
|
|
pub use model::Stage1JobClaimOutcome;
|
|
pub use model::Stage1Output;
|
|
pub use model::Stage1StartupClaimParams;
|
|
pub use model::ThreadGoal;
|
|
pub use model::ThreadGoalStatus;
|
|
pub use model::ThreadMetadata;
|
|
pub use model::ThreadMetadataBuilder;
|
|
pub use model::ThreadsPage;
|
|
pub use runtime::DeviceKeyBindingRecord;
|
|
pub use runtime::RemoteControlEnrollmentRecord;
|
|
pub use runtime::ThreadFilterOptions;
|
|
pub use runtime::ThreadGoalAccountingMode;
|
|
pub use runtime::ThreadGoalAccountingOutcome;
|
|
pub use runtime::ThreadGoalUpdate;
|
|
pub use runtime::logs_db_filename;
|
|
pub use runtime::logs_db_path;
|
|
pub use runtime::state_db_filename;
|
|
pub use runtime::state_db_path;
|
|
|
|
/// Environment variable for overriding the SQLite state database home directory.
|
|
pub const SQLITE_HOME_ENV: &str = "CODEX_SQLITE_HOME";
|
|
|
|
pub const LOGS_DB_FILENAME: &str = "logs";
|
|
pub const LOGS_DB_VERSION: u32 = 2;
|
|
pub const STATE_DB_FILENAME: &str = "state";
|
|
pub const STATE_DB_VERSION: u32 = 5;
|
|
|
|
/// Errors encountered during DB operations. Tags: [stage]
|
|
pub const DB_ERROR_METRIC: &str = "codex.db.error";
|
|
/// Metrics on backfill process. Tags: [status]
|
|
pub const DB_METRIC_BACKFILL: &str = "codex.db.backfill";
|
|
/// Metrics on backfill duration. Tags: [status]
|
|
pub const DB_METRIC_BACKFILL_DURATION_MS: &str = "codex.db.backfill.duration_ms";
|