logseq/docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md

# db-worker-node `invoke-main-thread` Removal and Worker-Local Refactor Plan

Goal: Remove all production `invoke-main-thread` usage from db worker logic and make `db-worker-node` fully self-contained for API execution.

Goal: Refactor each current main-thread API dependency into db-worker-owned implementations that work in both Node and Browser runtimes.

Goal: For UI-dependent interactions, replace direct worker->main-thread invocation with a request/response protocol where worker sends `postMessage`, and main thread actively calls db-worker thread APIs to respond.

Goal: Enforce request isolation with `request-id` across concurrent UI requests.

Architecture: Keep runtime-specific logic behind worker platform adapters (`frontend.worker.platform`, `frontend.worker.platform.node`, `frontend.worker.platform.browser`) and keep shared logic runtime-agnostic.

Related:
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/state.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/ui_request.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/auth.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/assets.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/e2ee.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/browser.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/worker.cljs`

## Problem statement

`db-worker-node` currently sets a main-thread stub that always rejects (`main-thread is not available in db-worker-node`).

Historically, shared worker modules called `worker-state/<invoke-main-thread` for auth refresh, asset upload/download metadata, E2EE key/password operations, and search-idle checks.

That created runtime divergence:
- Browser worker could call main-thread APIs through Comlink.
- Node daemon could not, so behavior depended on fallback code quality or failed outright.

Target state: db worker owns all non-UI logic and uses only explicit message protocol for UI interactions.

## Progress snapshot (current)

### Completed migrations

- Auth refresh dependency on `:thread-api/ensure-id&access-token` was removed from worker sync auth path.
- Asset upload/download/metadata dependencies on main-thread thread APIs were removed and replaced by worker-local implementations through platform adapters.
- Search idle dependency on `:thread-api/input-idle?` was removed from `db_core`; it now uses a **main-thread push + worker TTL cache** model:
  - main thread periodically pushes `:thread-atom/search-input-idle-status` with `{repo {:idle? ... :ts ...}}`
  - worker consumes that state with a TTL window and falls back to non-blocking behavior when state is missing/stale.
- E2EE UI-interaction path in `frontend.worker.sync.crypt` was migrated to a worker-owned request manager:
  - worker now emits `:db-worker/ui-request` with `request-id`
  - main thread responds via `:thread-api/resolve-ui-request` / `:thread-api/reject-ui-request`
  - worker crypt flow no longer directly calls `worker-state/<invoke-main-thread`
  - pending UI requests are cancelled on close/shutdown paths (`close-db`, `db-sync-close-db`, `unsafe-unlink-db`, browser `stop-db-worker!`)

### Remaining direct `invoke-main-thread` production usages

- None in production worker modules.
- `frontend.worker.state/<invoke-main-thread` remains as legacy infrastructure, but has no active production call sites.

## Historical inventory of `invoke-main-thread` dependencies (baseline before migration)

| Call site | Current main-thread API | Category | Runtime risk in `db-worker-node` |
| --- | --- | --- | --- |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/auth.cljs` (`<resolve-ws-token`) | `:thread-api/ensure-id&access-token` | Auth/token refresh | Token refresh path may fail without worker-local refresh |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`upload-remote-asset!`) | `:thread-api/rtc-upload-asset` | Asset I/O + network | Upload path depends on main-thread asset code |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`download-remote-asset!`) | `:thread-api/rtc-download-asset` | Asset I/O + network | Download path depends on main-thread asset code |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`request-asset-download!`) | `:thread-api/get-asset-file-metadata` | Asset metadata | Existence/checksum check depends on main thread |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-save-password-text!`) | `:thread-api/native-save-e2ee-password` | Password persistence | Native keychain path unavailable in node daemon |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-read-password-text`) | `:thread-api/native-get-e2ee-password` | Password persistence | Same |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-delete-password-text!`) | `:thread-api/native-delete-e2ee-password` | Password persistence | Same |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<generate-and-upload-user-rsa-key-pair!`) | `:thread-api/request-e2ee-password` | UI prompt | Requires explicit UI handshake |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<decrypt-private-key`) | `:thread-api/decrypt-user-e2ee-private-key` | UI/session-dependent decrypt | Must be worker-local + UI fallback protocol |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`:thread-api/init-user-rsa-key-pair`) | `:thread-api/request-e2ee-password` | UI prompt | Same as above |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` (`<wait-for-search-index-idle!`) | `:thread-api/input-idle?` | UI state query | Tight loop currently requires main-thread sync point |

## Design constraints and invariants

1. `db-worker-node` must never depend on main-thread APIs.
2. Shared worker code must run in both Node and Browser runtimes.
3. UI interactions must use explicit async protocol with `request-id` correlation.
4. Error shapes must remain stable (or be intentionally versioned) across transport layers.
5. Migration should be incremental with behavior-compatible phases.

## Target architecture

### 1) Remove worker-side direct main-thread invocation from business paths

Keep `frontend.worker.state/<invoke-main-thread` only as legacy infrastructure for browser compatibility during migration, then shrink/remove usage in production worker modules.

Acceptance target after migration:
- No production call sites to `<invoke-main-thread` outside temporary migration guards.
- `db-worker-node` main-thread stub remains reject-only and never hit in normal flows.

### 2) Move main-thread business logic into worker-owned modules

#### Auth refresh (`:thread-api/ensure-id&access-token`)
- Move token refresh implementation into worker sync auth module.
- Reuse current refresh token and auth state from worker state (`:auth/id-token`, `:auth/refresh-token`) that is already synchronized.
- Keep main thread responsible only for UI/session orchestration, not worker internal token lifecycle.

#### Asset operations (`:thread-api/rtc-upload-asset`, `:thread-api/rtc-download-asset`, `:thread-api/get-asset-file-metadata`)
- Introduce worker-side asset I/O + transfer module (shared, runtime-agnostic).
- Move file read/write/checksum/upload/download logic from main-thread handler into worker with platform adapter calls.
- Keep progress events emitted from worker via existing broadcast channel.

#### E2EE password storage (`native-save/get/delete`)
- Replace main-thread keychain dependency with worker platform capabilities.
- Add explicit storage abstraction in platform adapters for encrypted password persistence.
- Keep existing encrypted payload format unchanged where possible.

#### Private key decrypt (`:thread-api/decrypt-user-e2ee-private-key`)
- Make worker-local decrypt path primary.
- Request UI password only when worker-local credentials are unavailable.

### 3) UI request/response protocol (request-id isolated)

For UI-dependent operations, worker should not call main-thread thread APIs directly.

Use this flow:

```text
1) Worker needs UI input
   -> postMessage event to main thread
      {:type :db-worker/ui-request
       :request-id <uuid-v4>
       :action <keyword>
       :payload <map>
       :timeout-ms 60000}

2) Main thread handles UI action (prompt/dialog/user interaction)

3) Main thread actively calls db-worker thread API:
   :thread-api/resolve-ui-request [request-id result]
   or
   :thread-api/reject-ui-request [request-id error]

4) Worker resolves or rejects the pending promise by request-id.
```

Required worker internals:
- `*ui-requests-in-flight` map keyed by `request-id`.
- `request-id` is generated as UUID v4.
- default request timeout is `60000ms` (60s), with optional per-action override.
- timeout handling + cleanup.
- duplicate/late response protection.
- cancellation support for graph switch or worker shutdown.

Required acceptance:
- concurrent requests do not cross-resolve.
- timed-out request cannot be resolved later.
- all terminal states remove map entries.

#### UI interaction contract: headless vs interactive

The implementation uses two explicit modes instead of implicit fallback behavior.

- **Headless mode (default for `db-worker-node`/CLI):**
  - If an operation requires UI input and no interactive channel is available, return a typed error.
  - Error code should be stable and machine-readable: `:ui-interaction-required`.
  - Error payload should include at least `:action` and optional `:hint` (for example, configured password fallback).

- **Interactive mode (browser app):**
  - Worker emits `:db-worker/ui-request` with `request-id`.
  - Main thread performs UI interaction and actively calls worker thread APIs to resolve or reject the request.
  - Worker resumes flow only when matching `request-id` is resolved.

This dual contract keeps headless behavior deterministic while preserving async interactive UX in browser runtime.

### 4) Platform abstraction updates for Node/Browser parity

Extend `frontend.worker.platform` contract with only capabilities required by migrated APIs.

Likely additions:
- secure encrypted secret storage (or equivalent persisted encrypted blob)
- binary asset read/write/stat helpers
- HTTP transfer helpers with progress hooks (if not already available in shared runtime)
- monotonic clock/timer helpers for timeout handling

Node implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`.

Browser implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/browser.cljs`.

## Per-API migration matrix

| Legacy main-thread API | New worker-owned implementation | UI request protocol needed? | Node/Browser notes |
| --- | --- | --- | --- |
| `:thread-api/ensure-id&access-token` | Worker-local refresh function in sync auth module | No | Must work without UI in node daemon |
| `:thread-api/rtc-upload-asset` | Worker asset upload API | No | Use platform file + HTTP adapters |
| `:thread-api/rtc-download-asset` | Worker asset download API | No | Same |
| `:thread-api/get-asset-file-metadata` | Worker metadata/checksum API | No | Same |
| `:thread-api/native-save-e2ee-password` | Worker secure persistence API | No | Use platform adapter implementations |
| `:thread-api/native-get-e2ee-password` | Worker secure read API | No | Same |
| `:thread-api/native-delete-e2ee-password` | Worker secure delete API | No | Same |
| `:thread-api/request-e2ee-password` | Worker emits UI request and waits on `resolve-ui-request` | Yes | Node can return typed `:ui-interaction-required` unless configured fallback exists |
| `:thread-api/decrypt-user-e2ee-private-key` | Worker-local decrypt + optional UI request for password | Yes (fallback) | Keep headless path for node |
| `:thread-api/input-idle?` | Main thread pushes idle-state updates; worker consumes a local TTL cache for indexing decisions | Yes | Push+TTL is the default model to avoid high-frequency request storms |

## Implementation phases

### Phase 0: Safety rails and observability

1. Add explicit metric/log points for any call to `<invoke-main-thread` in worker code.
2. Add request-id propagation fields to db-worker-node invoke/event logs for traceability.
3. Add temporary feature flag to switch between old and new path per migrated domain.

### Phase 1: Non-UI API migration

1. Migrate auth refresh from main thread to worker.
2. Migrate asset metadata/upload/download to worker module + platform adapters.
3. Migrate E2EE encrypted password persistence to worker platform APIs.
4. Keep compatibility shims temporarily, but default to worker-local path.

Exit criteria:
- No non-UI flow depends on main thread.
- CLI and db-worker-node sync paths pass regression tests.

### Phase 2: UI request protocol

1. Introduce generic worker UI request manager (`request-id`, timeout, cleanup).
2. Add worker thread APIs for resolving/rejecting UI requests.
3. Add main-thread handler wiring in `frontend.handler.worker` for `:db-worker/ui-request` events.
4. Route E2EE prompt/decrypt UI-dependent flows through new protocol.

Exit criteria:
- concurrent password requests are isolated by `request-id`.
- graph switch/shutdown does not leak pending requests.

### Phase 3: Search idle and remaining UI-state dependencies

1. Replace `:thread-api/input-idle?` direct invocation with **main-thread push + worker TTL cache** model.
2. Define TTL window and stale-state behavior for indexing loops.
3. Validate search indexing throughput and UI responsiveness.

Exit criteria:
- no direct worker->main-thread API invocation in search flow.
- push+TTL idle-state model is active and covered by tests.
- no regression in search build behavior.

### Phase 4: Remove legacy dependencies

1. Remove remaining production usages of `worker-state/<invoke-main-thread` in worker sync/core modules.
2. Update `non-repo-methods` and API inventory if method ownership changed.
3. Keep browser compatibility wrappers only if still required by bootstrapping internals.

Final exit criteria:
- `invoke-main-thread` usage in production code is zero (or explicitly limited to bootstrap-only internals with no business logic).
- `db-worker-node` full API suite works without main-thread API dependencies.

## Tests and verification plan

### Unit tests

- Worker auth tests: token refresh in worker without main-thread stubs.
- Worker asset tests: metadata/upload/download via worker module and platform mocks.
- Worker crypt tests: password storage/decrypt paths do not call main thread.
- UI request manager tests: request-id isolation, timeout, duplicate response rejection.

### Integration tests

- Browser flow: UI request roundtrip (`postMessage` -> main thread -> `resolve-ui-request`).
- Node flow: same API paths run headless and return typed errors when UI is mandatory.
- Search indexing flow: idle-state protocol behavior under repeated checks.

### Regression checks

- `eca__grep` gate: no production `worker-state/<invoke-main-thread` call sites in migrated modules.
- `db-worker-node` daemon invoke smoke tests for sync + e2ee + asset paths.
- Existing CLI and worker tests remain green.

## Risks and mitigations

1. **E2EE flow regressions**
   - Mitigation: migrate with feature flags and run dedicated crypt regression suite first.

2. **Asset transfer behavior drift**
   - Mitigation: keep payload format/checksum semantics unchanged during extraction.

3. **Request map leaks / stale requests**
   - Mitigation: enforce timeout + shutdown cleanup + finalizer assertions.

4. **Search performance impact**
   - Mitigation: add request throttling and local TTL cache for idle-state checks.

5. **Protocol mismatch between main thread and worker**
   - Mitigation: define strict message schema and validate fields at both ends.

## Acceptance criteria

1. Every current production `invoke-main-thread` dependency listed in this plan has a worker-owned replacement.
2. `db-worker-node` runs without any required main-thread API bridge.
3. UI-dependent flows use `postMessage` request + main-thread active callback to worker thread API.
4. `request-id` enforces isolation for concurrent UI requests.
5. In headless mode, UI-required operations return typed error `:ui-interaction-required` with stable machine-readable payload.
6. Node and Browser runtimes both pass targeted tests for migrated paths.
7. No unresolved pending UI requests remain after graph close or worker shutdown.

## Suggested execution order

1. Auth + non-UI crypt storage.
2. Asset API migration.
3. Generic UI request protocol.
4. E2EE prompt/decrypt migration.
5. Search idle-state migration.
6. Legacy call-site removal and cleanup.

## Confirmed decisions

1. `request-id` format: UUID v4.
2. Default UI request timeout: 60s (`60000ms`), optional per-action override.
3. Search idle state model: main-thread push + worker TTL cache.
4. Headless behavior: UI-required operations return typed error `:ui-interaction-required`.
5. Execution order: follow the phased order defined in this document.

## Notes

This plan intentionally prioritizes behavior parity and migration safety over immediate deep refactor of all related modules.

The protocol and platform abstraction should be stabilized first; broad code movement should follow once test harnesses are in place.