diff --git a/docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md b/docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md new file mode 100644 index 0000000000..039f05b395 --- /dev/null +++ b/docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md @@ -0,0 +1,297 @@ +# db-worker-node `invoke-main-thread` Removal and Worker-Local Refactor Plan + +Goal: Remove all production `invoke-main-thread` usage from db worker logic and make `db-worker-node` fully self-contained for API execution. + +Goal: Refactor each current main-thread API dependency into db-worker-owned implementations that work in both Node and Browser runtimes. + +Goal: For UI-dependent interactions, replace direct worker->main-thread invocation with a request/response protocol where worker sends `postMessage`, and main thread actively calls db-worker thread APIs to respond. + +Goal: Enforce request isolation with `request-id` across concurrent UI requests. + +Architecture: Keep runtime-specific logic behind worker platform adapters (`frontend.worker.platform`, `frontend.worker.platform.node`, `frontend.worker.platform.browser`) and keep shared logic runtime-agnostic. + +Related: +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/state.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/auth.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/assets.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/e2ee.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/browser.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/worker.cljs` + +## Problem statement + +`db-worker-node` currently sets a main-thread stub that always rejects (`main-thread is not available in db-worker-node`). + +However, shared worker modules still call `worker-state/ postMessage event to main thread + {:type :db-worker/ui-request + :request-id + :action + :payload + :timeout-ms 60000} + +2) Main thread handles UI action (prompt/dialog/user interaction) + +3) Main thread actively calls db-worker thread API: + :thread-api/resolve-ui-request [request-id result] + or + :thread-api/reject-ui-request [request-id error] + +4) Worker resolves or rejects the pending promise by request-id. +``` + +Required worker internals: +- `*ui-requests-in-flight` map keyed by `request-id`. +- `request-id` is generated as UUID v4. +- default request timeout is `60000ms` (60s), with optional per-action override. +- timeout handling + cleanup. +- duplicate/late response protection. +- cancellation support for graph switch or worker shutdown. + +Required acceptance: +- concurrent requests do not cross-resolve. +- timed-out request cannot be resolved later. +- all terminal states remove map entries. + +#### UI interaction contract: headless vs interactive + +The implementation uses two explicit modes instead of implicit fallback behavior. + +- **Headless mode (default for `db-worker-node`/CLI):** + - If an operation requires UI input and no interactive channel is available, return a typed error. + - Error code should be stable and machine-readable: `:ui-interaction-required`. + - Error payload should include at least `:action` and optional `:hint` (for example, configured password fallback). + +- **Interactive mode (browser app):** + - Worker emits `:db-worker/ui-request` with `request-id`. + - Main thread performs UI interaction and actively calls worker thread APIs to resolve or reject the request. + - Worker resumes flow only when matching `request-id` is resolved. + +This dual contract keeps headless behavior deterministic while preserving async interactive UX in browser runtime. + +### 4) Platform abstraction updates for Node/Browser parity + +Extend `frontend.worker.platform` contract with only capabilities required by migrated APIs. + +Likely additions: +- secure encrypted secret storage (or equivalent persisted encrypted blob) +- binary asset read/write/stat helpers +- HTTP transfer helpers with progress hooks (if not already available in shared runtime) +- monotonic clock/timer helpers for timeout handling + +Node implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. + +Browser implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/browser.cljs`. + +## Per-API migration matrix + +| Legacy main-thread API | New worker-owned implementation | UI request protocol needed? | Node/Browser notes | +| --- | --- | --- | --- | +| `:thread-api/ensure-id&access-token` | Worker-local refresh function in sync auth module | No | Must work without UI in node daemon | +| `:thread-api/rtc-upload-asset` | Worker asset upload API | No | Use platform file + HTTP adapters | +| `:thread-api/rtc-download-asset` | Worker asset download API | No | Same | +| `:thread-api/get-asset-file-metadata` | Worker metadata/checksum API | No | Same | +| `:thread-api/native-save-e2ee-password` | Worker secure persistence API | No | Use platform adapter implementations | +| `:thread-api/native-get-e2ee-password` | Worker secure read API | No | Same | +| `:thread-api/native-delete-e2ee-password` | Worker secure delete API | No | Same | +| `:thread-api/request-e2ee-password` | Worker emits UI request and waits on `resolve-ui-request` | Yes | Node can return typed `:ui-interaction-required` unless configured fallback exists | +| `:thread-api/decrypt-user-e2ee-private-key` | Worker-local decrypt + optional UI request for password | Yes (fallback) | Keep headless path for node | +| `:thread-api/input-idle?` | Main thread pushes idle-state updates; worker consumes a local TTL cache for indexing decisions | Yes | Push+TTL is the default model to avoid high-frequency request storms | + +## Implementation phases + +### Phase 0: Safety rails and observability + +1. Add explicit metric/log points for any call to `main-thread API invocation in search flow. +- push+TTL idle-state model is active and covered by tests. +- no regression in search build behavior. + +### Phase 4: Remove legacy dependencies + +1. Remove remaining production usages of `worker-state/ main thread -> `resolve-ui-request`). +- Node flow: same API paths run headless and return typed errors when UI is mandatory. +- Search indexing flow: idle-state protocol behavior under repeated checks. + +### Regression checks + +- `eca__grep` gate: no production `worker-state/clj (js/JSON.parse text) :keywordize-keys true))] + (if (.-ok resp) + {:id-token (:id_token data) + :access-token (:access_token data)} + (throw (ex-info "worker auth refresh failed" + {:code :auth-refresh-failed + :status (.-status resp) + :token-url token-url + :body data}))))))) + (defn {:auth/id-token id-token} + (seq access-token) (assoc :auth/access-token access-token))) + id-token) (p/resolved token)))) (defn get-user-uuid diff --git a/src/main/frontend/worker/sync/crypt.cljs b/src/main/frontend/worker/sync/crypt.cljs index 6ffaeccc1c..644c14e79b 100644 --- a/src/main/frontend/worker/sync/crypt.cljs +++ b/src/main/frontend/worker/sync/crypt.cljs @@ -16,6 +16,7 @@ (defonce ^:private *graph->aes-key (atom {})) (defonce ^:private *user-rsa-key-pair-inflight (atom {})) (defonce ^:private e2ee-password-file "e2ee-password") +(defonce ^:private e2ee-password-secret-key "logseq-encrypted-password") (defonce ^:private native-env? (let [href (try (.. js/self -location -href) (catch :default _ nil))] @@ -31,15 +32,15 @@ (defn (p/let [native-text ( (#'db-sync/ (#'db-sync/ (p/with-redefs [sync-crypt/native-worker? (fn [] true) + crypt/ (p/with-redefs [sync-crypt/native-worker? (fn [] true) + platform/current (fn [] platform-map) + platform/read-secret-text (fn [_platform' _key] + (p/rejected (ex-info "secret store unavailable" {}))) + platform/read-text! (fn [platform' path] + (swap! fallback-read-calls inc) + (is (= platform-map platform')) + (is (= "e2ee-password" path)) + (ldb/write-transit-str {:cipher "payload"})) + crypt/