logseq/docs/agent-guide/075-sync-e2ee-password-persistence.md

Sync E2EE Password Persistence Implementation Plan

Goal: Persist and consume sync E2EE passwords only as refresh-token-encrypted data across CLI, desktop app, web app, and db-worker-node.

Architecture: Keep password encryption and verification logic in frontend.worker.sync.crypt as the single source of truth.

Architecture: Use runtime-specific persistence behind worker platform boundaries, with IndexedDB-backed secret storage on browser runtime and <data-dir>/e2ee-password file storage on node runtime.

Architecture: Route CLI sync start --e2ee-password and sync download --e2ee-password, plus desktop UI password input, through the same worker-side validation and persistence flow.

Tech Stack: ClojureScript, Promesa, db-worker platform adapters, babashka.cli command parsing, Electron renderer-to-worker UI request protocol.

Related: Builds on docs/agent-guide/047-logseq-cli-sync-command.md, docs/agent-guide/055-logseq-cli-login-logout.md, and docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md.

Problem statement

Current behavior is inconsistent across runtimes and command paths.

frontend.worker.sync.crypt/<save-e2ee-password encrypts password text by refresh-token, but password sourcing and persistence locations differ by flow.

CLI still accepts and persists plain :e2ee-password in cli.edn, which conflicts with secure persistence requirements.

Browser runtime currently persists non-native password file via OPFS (e2ee-password) instead of IndexedDB.

CLI headless fallback currently includes config password and can return :ui-interaction-required rather than a direct e2ee-password not found outcome.

Desktop interactive flow can decrypt via UI path, but the verify-and-persist behavior is not defined as an explicit shared worker API reused by CLI.

The table below summarizes current state and required change.

Area	Current state	Required state
CLI config	`sync config set	get
Password crypto key	Worker encrypt/decrypt helpers already accept refresh-token, but CLI fallback still depends on config and auth-token candidates.	Use refresh-token as the only password crypto key source for persisted password.
Browser persistence	Non-native flow reads/writes OPFS file e2ee-password.	Store encrypted password in IndexedDB.
Node persistence	Non-native flow reads/writes <data-dir>/e2ee-password.	Keep <data-dir>/e2ee-password as canonical node location.
CLI UX	sync start/status does not provide a dedicated missing-password UX contract.	If password is unavailable, sync start/status should report e2ee-password not found with actionable hint.
CLI input path	No sync start --e2ee-password or sync download --e2ee-password option.	Add --e2ee-password to sync start and sync download, verify it, encrypt it, then persist it.
Desktop UI reuse	UI decrypt path exists, but shared CLI-equivalent verify/persist contract is not explicit.	Desktop missing-password interaction should call the same worker verify-and-persist flow as CLI.

Target architecture

CLI / Desktop UI / Web runtime
        |
        | password input (optional)
        v
frontend.worker.sync.crypt
  - verify password against user encrypted private key
  - encrypt/decrypt persisted password with refresh-token
  - return typed missing-password errors
        |
        +--> Browser runtime: IndexedDB secret slot
        |
        +--> Node runtime: <data-dir>/e2ee-password

Testing Plan

I will add worker unit tests in src/test/frontend/worker/sync/crypt_test.cljs for runtime-specific storage routing, refresh-token-only encryption key usage, and typed missing-password behavior in CLI-headless mode.

I will add CLI command tests in src/test/logseq/cli/command/sync_test.cljs for sync start --e2ee-password, sync download --e2ee-password, deprecation of config key handling, and missing-password reporting in sync start/sync download/sync status paths.

I will add CLI config tests in src/test/logseq/cli/command/sync_test.cljs and src/test/logseq/cli/format_test.cljs to verify :e2ee-password is no longer treated as file config and to verify updated user-facing output.

I will add desktop UI request tests in src/test/frontend/handler/db_based/sync_test.cljs or nearby handler tests to verify missing-password UI roundtrip invokes the shared worker verify-and-persist API instead of bespoke flow-only logic.

I will run targeted tests first, then run bb dev:lint-and-test for final validation.

NOTE: I will write all tests before I add any implementation behavior.

Implementation plan

Phase 1. Define shared worker password vault behavior.

1. Add failing tests in src/test/frontend/worker/sync/crypt_test.cljs that describe a new shared worker API for verify-and-persist password.

2. Add failing tests that enforce refresh-token-only persisted password decrypt path in headless mode and reject config-only fallback.

3. Implement a single worker helper in src/main/frontend/worker/sync/crypt.cljs that verifies a candidate password by decrypting user encrypted private key and then persists encrypted password payload.

4. Expose the helper through a thread API that can be invoked by both CLI and desktop flows.

5. Remove :e2ee-password config candidate usage from crypt.cljs headless candidate logic.

6. Ensure CLI-headless path does not mask missing-password with :ui-interaction-required when no interactive channel exists.

7. Add typed error payload and hint contract for missing persisted password.

8. Run worker crypt tests and confirm failing cases are now passing.

Phase 2. Enforce browser and node persistence locations.

1. Add failing tests for browser runtime persistence using IndexedDB secret storage rather than OPFS file route.

2. Add failing tests for node runtime persistence at <data-dir>/e2ee-password.

3. Update src/main/frontend/worker/sync/crypt.cljs storage branching to select persistence backend by runtime (:browser vs :node) rather than legacy native URL heuristics for this password feature.

4. Keep node runtime read/write via platform/read-text! and platform/write-text! with path e2ee-password.

5. Keep browser runtime read/write via platform/read-secret-text and platform/save-secret-text! so storage lands in IndexedDB-backed kv store.

6. Add migration fallback in browser path to read legacy OPFS e2ee-password once, then re-save to IndexedDB, then remove legacy file.

7. Add clear/delete behavior that correctly removes persisted password from both runtime backends.

8. Run frontend.worker.sync.crypt-test to verify storage routing and migration behavior.

Phase 3. Deprecate cli.edn :e2ee-password.

1. Add failing CLI tests in src/test/logseq/cli/command/sync_test.cljs that reject sync config set|get|unset e2ee-password.

2. Add failing config tests in src/test/logseq/cli/command/sync_test.cljs showing :e2ee-password in old cli.edn is ignored and sanitized out without any deprecation warning.

3. Remove e2ee-password from config-key-map and sync config command examples in src/main/logseq/cli/command/sync.cljs.

4. Remove :e2ee-password from sync-config payload generated by CLI for worker runtime config.

5. Add :e2ee-password into removed-config-keys in src/main/logseq/cli/config.cljs so persisted file config is automatically sanitized.

6. Update src/main/logseq/cli/format.cljs and src/test/logseq/cli/format_test.cljs for changed sync-config output expectations.

7. Run sync command and format tests to verify deprecation behavior.

Phase 4. Add CLI --e2ee-password on sync start and sync download, and enforce missing-password errors.

1. Add failing parse/build-action tests in src/test/logseq/cli/command/sync_test.cljs for --e2ee-password on both sync start and sync download.

2. Add failing execute tests that both commands invoke worker verify-and-persist before start or download work begins.

3. Add CLI auth helper test coverage in src/test/logseq/cli/command/sync_test.cljs for resolving refresh-token from auth.json in runtime command execution path.

4. Extend src/main/logseq/cli/command/sync.cljs command spec to include --e2ee-password for sync start and sync download.

5. Add CLI-side runtime auth context helper usage so both flows have id-token and refresh-token available at execution time.

6. When option is provided, call worker shared verify-and-persist API and fail fast on invalid password.

7. In sync start and sync download execution paths, plus sync status reporting path, map missing-password diagnostics to explicit e2ee-password not found errors with actionable hint text.

8. Ensure sync status returns error status and non-zero exit code when e2ee password is required but missing.

9. Keep machine-readable error code stable for automation.

10. Run sync command tests and relevant integration tests.

Phase 5. Reuse CLI flow from desktop missing-password interaction.

1. Add failing UI request tests for missing persisted password path in desktop interactive runtime.

2. In src/main/frontend/worker/sync/crypt.cljs, route interactive fallback to request password input, then call the same shared verify-and-persist helper used by CLI.

3. Update src/main/frontend/worker/ui_request.cljs and src/main/frontend/handler/worker.cljs action contract so UI request returns password input and worker does verification/decryption centrally.

4. Update desktop password modal flow in src/main/frontend/handler/events/rtc.cljs and src/main/frontend/components/e2ee.cljs only as needed to collect password and forward it to worker.

5. Remove duplicate validation logic from UI-side code paths after worker shared flow is active.

6. Run frontend handler/component tests and existing e2ee-related tests.

Phase 6. Documentation and acceptance.

1. Update docs/cli/logseq-cli.md to remove sync config ... e2ee-password references.

2. Document sync start --e2ee-password and sync download --e2ee-password usage, plus missing-password troubleshooting for sync start/sync download/sync status.

3. Document persistence locations explicitly.

4. Browser runtime stores encrypted password in IndexedDB.

5. Node runtime stores encrypted password at <data-dir>/e2ee-password.

6. Add migration note for users with legacy cli.edn :e2ee-password and legacy browser OPFS password file.

7. Run a final lint and test pass.

8. Confirm no docs still recommend cli.edn :e2ee-password.

Edge cases to cover

Expired id-token with valid refresh-token in CLI should still allow password verification and persistence after auth refresh.

Missing refresh-token in auth.json should produce a login-required error before any password persistence attempt.

Wrong --e2ee-password should fail verification without overwriting previously stored encrypted password.

Node file exists but is corrupted transit payload should return typed missing/invalid-password diagnostics and preserve existing UX hints.

Browser migration should handle missing legacy OPFS file without error noise.

Desktop interactive flow cancellation should not persist empty password and should keep prior stored password untouched.

Multiple sync start invocations should avoid race conditions that overwrite password storage with stale data.

Verification commands

Run targeted worker tests.

bb dev:test -v frontend.worker.sync.crypt-test

Run targeted CLI sync tests.

bb dev:test -v logseq.cli.command.sync-test

Run CLI format tests.

bb dev:test -v logseq.cli.format-test

Run CLI command tests around sync/auth/config behavior.

bb dev:test -v logseq.cli.command.sync-test

Run full lint and test suite before merge.

bb dev:lint-and-test

Testing Details

The key behavior assertions are that password persistence is always encrypted with refresh-token, storage backend is runtime-correct, and user-facing CLI/desktop UX is deterministic when password is missing or invalid.

Tests will validate command outcomes and worker behavior contracts instead of only checking internal data structures.

Implementation Details

• Introduce one shared worker API for verify-and-persist E2EE password and reuse it from CLI and desktop.
• Remove :e2ee-password from CLI file config lifecycle and sync config subcommands.
• Keep refresh-token as the sole crypto key for persisted password encryption and decryption.
• Move browser persistence to IndexedDB secret storage and keep node persistence at <data-dir>/e2ee-password.
• Add legacy migration read path from browser OPFS password file to IndexedDB.
• Add explicit missing-password diagnostics for CLI sync start/sync download/sync status, with sync status returning error when password is required but missing.
• Add sync start --e2ee-password and sync download --e2ee-password options with verification-before-persist semantics.
• Keep :e2ee-password deprecation behavior as silent ignore with no compatibility warning.
• Preserve machine-readable error codes and CLI structured output compatibility.
• Update user docs and troubleshooting guidance to match deprecation and new option flow.

Question

No open questions.

Decisions locked.

Add --e2ee-password to sync start and sync download only.

Ignore cli.edn :e2ee-password silently with no compatibility path.

Return error for missing password in sync status.

---