clone/logseq
1
0
Fork 0
mirror of https://github.com/logseq/logseq.git synced 2026-05-25 21:24:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(db-worker,wip): remove <invoke-main-thread (1)

Browse Source
This commit is contained in:
rcmerci
2026-04-03 16:05:55 +08:00
parent 4b139bb6f2
commit 30671bf4e1
12 changed files with 571 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

297
docs/agent-guide/074-db-worker-node-invoke-main-thread-refactor.md Normal file
View File

@@ -0,0 +1,297 @@
# db-worker-node `invoke-main-thread` Removal and Worker-Local Refactor Plan
Goal: Remove all production `invoke-main-thread` usage from db worker logic and make `db-worker-node` fully self-contained for API execution.
Goal: Refactor each current main-thread API dependency into db-worker-owned implementations that work in both Node and Browser runtimes.
Goal: For UI-dependent interactions, replace direct worker->main-thread invocation with a request/response protocol where worker sends `postMessage`, and main thread actively calls db-worker thread APIs to respond.
Goal: Enforce request isolation with `request-id` across concurrent UI requests.
Architecture: Keep runtime-specific logic behind worker platform adapters (`frontend.worker.platform`, `frontend.worker.platform.node`, `frontend.worker.platform.browser`) and keep shared logic runtime-agnostic.
Related:
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/state.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/auth.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/assets.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/e2ee.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/browser.cljs`
- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/worker.cljs`
## Problem statement
`db-worker-node` currently sets a main-thread stub that always rejects (`main-thread is not available in db-worker-node`).
However, shared worker modules still call `worker-state/<invoke-main-thread` for auth refresh, asset upload/download metadata, E2EE key/password operations, and search-idle checks.
This creates runtime divergence:
- Browser worker can call main-thread APIs through Comlink.
- Node daemon cannot, so behavior depends on fallback code quality or fails outright.
Target state: db worker should own all non-UI logic and only use explicit message protocol for UI interactions.
## Full inventory of current `invoke-main-thread` dependencies
| Call site | Current main-thread API | Category | Runtime risk in `db-worker-node` |
| --- | --- | --- | --- |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/auth.cljs` (`<resolve-ws-token`) | `:thread-api/ensure-id&access-token` | Auth/token refresh | Token refresh path may fail without worker-local refresh |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`upload-remote-asset!`) | `:thread-api/rtc-upload-asset` | Asset I/O + network | Upload path depends on main-thread asset code |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`download-remote-asset!`) | `:thread-api/rtc-download-asset` | Asset I/O + network | Download path depends on main-thread asset code |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/assets.cljs` (`request-asset-download!`) | `:thread-api/get-asset-file-metadata` | Asset metadata | Existence/checksum check depends on main thread |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-save-password-text!`) | `:thread-api/native-save-e2ee-password` | Password persistence | Native keychain path unavailable in node daemon |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-read-password-text`) | `:thread-api/native-get-e2ee-password` | Password persistence | Same |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<native-delete-password-text!`) | `:thread-api/native-delete-e2ee-password` | Password persistence | Same |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<generate-and-upload-user-rsa-key-pair!`) | `:thread-api/request-e2ee-password` | UI prompt | Requires explicit UI handshake |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`<decrypt-private-key`) | `:thread-api/decrypt-user-e2ee-private-key` | UI/session-dependent decrypt | Must be worker-local + UI fallback protocol |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` (`:thread-api/init-user-rsa-key-pair`) | `:thread-api/request-e2ee-password` | UI prompt | Same as above |
| `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` (`<wait-for-search-index-idle!`) | `:thread-api/input-idle?` | UI state query | Tight loop currently requires main-thread sync point |
## Design constraints and invariants
1. `db-worker-node` must never depend on main-thread APIs.
2. Shared worker code must run in both Node and Browser runtimes.
3. UI interactions must use explicit async protocol with `request-id` correlation.
4. Error shapes must remain stable (or be intentionally versioned) across transport layers.
5. Migration should be incremental with behavior-compatible phases.
## Target architecture
### 1) Remove worker-side direct main-thread invocation from business paths
Keep `frontend.worker.state/<invoke-main-thread` only as legacy infrastructure for browser compatibility during migration, then shrink/remove usage in production worker modules.
Acceptance target after migration:
- No production call sites to `<invoke-main-thread` outside temporary migration guards.
- `db-worker-node` main-thread stub remains reject-only and never hit in normal flows.
### 2) Move main-thread business logic into worker-owned modules
#### Auth refresh (`:thread-api/ensure-id&access-token`)
- Move token refresh implementation into worker sync auth module.
- Reuse current refresh token and auth state from worker state (`:auth/id-token`, `:auth/refresh-token`) that is already synchronized.
- Keep main thread responsible only for UI/session orchestration, not worker internal token lifecycle.
#### Asset operations (`:thread-api/rtc-upload-asset`, `:thread-api/rtc-download-asset`, `:thread-api/get-asset-file-metadata`)
- Introduce worker-side asset I/O + transfer module (shared, runtime-agnostic).
- Move file read/write/checksum/upload/download logic from main-thread handler into worker with platform adapter calls.
- Keep progress events emitted from worker via existing broadcast channel.
#### E2EE password storage (`native-save/get/delete`)
- Replace main-thread keychain dependency with worker platform capabilities.
- Add explicit storage abstraction in platform adapters for encrypted password persistence.
- Keep existing encrypted payload format unchanged where possible.
#### Private key decrypt (`:thread-api/decrypt-user-e2ee-private-key`)
- Make worker-local decrypt path primary.
- Request UI password only when worker-local credentials are unavailable.
### 3) UI request/response protocol (request-id isolated)
For UI-dependent operations, worker should not call main-thread thread APIs directly.
Use this flow:
```text
1) Worker needs UI input
-> postMessage event to main thread
{:type :db-worker/ui-request
:request-id <uuid-v4>
:action <keyword>
:payload <map>
:timeout-ms 60000}
2) Main thread handles UI action (prompt/dialog/user interaction)
3) Main thread actively calls db-worker thread API:
:thread-api/resolve-ui-request [request-id result]
or
:thread-api/reject-ui-request [request-id error]
4) Worker resolves or rejects the pending promise by request-id.
```
Required worker internals:
- `*ui-requests-in-flight` map keyed by `request-id`.
- `request-id` is generated as UUID v4.
- default request timeout is `60000ms` (60s), with optional per-action override.
- timeout handling + cleanup.
- duplicate/late response protection.
- cancellation support for graph switch or worker shutdown.
Required acceptance:
- concurrent requests do not cross-resolve.
- timed-out request cannot be resolved later.
- all terminal states remove map entries.
#### UI interaction contract: headless vs interactive
The implementation uses two explicit modes instead of implicit fallback behavior.
- **Headless mode (default for `db-worker-node`/CLI):**
- If an operation requires UI input and no interactive channel is available, return a typed error.
- Error code should be stable and machine-readable: `:ui-interaction-required`.
- Error payload should include at least `:action` and optional `:hint` (for example, configured password fallback).
- **Interactive mode (browser app):**
- Worker emits `:db-worker/ui-request` with `request-id`.
- Main thread performs UI interaction and actively calls worker thread APIs to resolve or reject the request.
- Worker resumes flow only when matching `request-id` is resolved.
This dual contract keeps headless behavior deterministic while preserving async interactive UX in browser runtime.
### 4) Platform abstraction updates for Node/Browser parity
Extend `frontend.worker.platform` contract with only capabilities required by migrated APIs.
Likely additions:
- secure encrypted secret storage (or equivalent persisted encrypted blob)
- binary asset read/write/stat helpers
- HTTP transfer helpers with progress hooks (if not already available in shared runtime)
- monotonic clock/timer helpers for timeout handling
Node implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`.
Browser implementation: `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/browser.cljs`.
## Per-API migration matrix
| Legacy main-thread API | New worker-owned implementation | UI request protocol needed? | Node/Browser notes |
| --- | --- | --- | --- |
| `:thread-api/ensure-id&access-token` | Worker-local refresh function in sync auth module | No | Must work without UI in node daemon |
| `:thread-api/rtc-upload-asset` | Worker asset upload API | No | Use platform file + HTTP adapters |
| `:thread-api/rtc-download-asset` | Worker asset download API | No | Same |
| `:thread-api/get-asset-file-metadata` | Worker metadata/checksum API | No | Same |
| `:thread-api/native-save-e2ee-password` | Worker secure persistence API | No | Use platform adapter implementations |
| `:thread-api/native-get-e2ee-password` | Worker secure read API | No | Same |
| `:thread-api/native-delete-e2ee-password` | Worker secure delete API | No | Same |
| `:thread-api/request-e2ee-password` | Worker emits UI request and waits on `resolve-ui-request` | Yes | Node can return typed `:ui-interaction-required` unless configured fallback exists |
| `:thread-api/decrypt-user-e2ee-private-key` | Worker-local decrypt + optional UI request for password | Yes (fallback) | Keep headless path for node |
| `:thread-api/input-idle?` | Main thread pushes idle-state updates; worker consumes a local TTL cache for indexing decisions | Yes | Push+TTL is the default model to avoid high-frequency request storms |
## Implementation phases
### Phase 0: Safety rails and observability
1. Add explicit metric/log points for any call to `<invoke-main-thread` in worker code.
2. Add request-id propagation fields to db-worker-node invoke/event logs for traceability.
3. Add temporary feature flag to switch between old and new path per migrated domain.
### Phase 1: Non-UI API migration
1. Migrate auth refresh from main thread to worker.
2. Migrate asset metadata/upload/download to worker module + platform adapters.
3. Migrate E2EE encrypted password persistence to worker platform APIs.
4. Keep compatibility shims temporarily, but default to worker-local path.
Exit criteria:
- No non-UI flow depends on main thread.
- CLI and db-worker-node sync paths pass regression tests.
### Phase 2: UI request protocol
1. Introduce generic worker UI request manager (`request-id`, timeout, cleanup).
2. Add worker thread APIs for resolving/rejecting UI requests.
3. Add main-thread handler wiring in `frontend.handler.worker` for `:db-worker/ui-request` events.
4. Route E2EE prompt/decrypt UI-dependent flows through new protocol.
Exit criteria:
- concurrent password requests are isolated by `request-id`.
- graph switch/shutdown does not leak pending requests.
### Phase 3: Search idle and remaining UI-state dependencies
1. Replace `:thread-api/input-idle?` direct invocation with **main-thread push + worker TTL cache** model.
2. Define TTL window and stale-state behavior for indexing loops.
3. Validate search indexing throughput and UI responsiveness.
Exit criteria:
- no direct worker->main-thread API invocation in search flow.
- push+TTL idle-state model is active and covered by tests.
- no regression in search build behavior.
### Phase 4: Remove legacy dependencies
1. Remove remaining production usages of `worker-state/<invoke-main-thread` in worker sync/core modules.
2. Update `non-repo-methods` and API inventory if method ownership changed.
3. Keep browser compatibility wrappers only if still required by bootstrapping internals.
Final exit criteria:
- `invoke-main-thread` usage in production code is zero (or explicitly limited to bootstrap-only internals with no business logic).
- `db-worker-node` full API suite works without main-thread API dependencies.
## Tests and verification plan
### Unit tests
- Worker auth tests: token refresh in worker without main-thread stubs.
- Worker asset tests: metadata/upload/download via worker module and platform mocks.
- Worker crypt tests: password storage/decrypt paths do not call main thread.
- UI request manager tests: request-id isolation, timeout, duplicate response rejection.
### Integration tests
- Browser flow: UI request roundtrip (`postMessage` -> main thread -> `resolve-ui-request`).
- Node flow: same API paths run headless and return typed errors when UI is mandatory.
- Search indexing flow: idle-state protocol behavior under repeated checks.
### Regression checks
- `eca__grep` gate: no production `worker-state/<invoke-main-thread` call sites in migrated modules.
- `db-worker-node` daemon invoke smoke tests for sync + e2ee + asset paths.
- Existing CLI and worker tests remain green.
## Risks and mitigations
1. **E2EE flow regressions**
- Mitigation: migrate with feature flags and run dedicated crypt regression suite first.
2. **Asset transfer behavior drift**
- Mitigation: keep payload format/checksum semantics unchanged during extraction.
3. **Request map leaks / stale requests**
- Mitigation: enforce timeout + shutdown cleanup + finalizer assertions.
4. **Search performance impact**
- Mitigation: add request throttling and local TTL cache for idle-state checks.
5. **Protocol mismatch between main thread and worker**
- Mitigation: define strict message schema and validate fields at both ends.
## Acceptance criteria
1. Every current production `invoke-main-thread` dependency listed in this plan has a worker-owned replacement.
2. `db-worker-node` runs without any required main-thread API bridge.
3. UI-dependent flows use `postMessage` request + main-thread active callback to worker thread API.
4. `request-id` enforces isolation for concurrent UI requests.
5. In headless mode, UI-required operations return typed error `:ui-interaction-required` with stable machine-readable payload.
6. Node and Browser runtimes both pass targeted tests for migrated paths.
7. No unresolved pending UI requests remain after graph close or worker shutdown.
## Suggested execution order
1. Auth + non-UI crypt storage.
2. Asset API migration.
3. Generic UI request protocol.
4. E2EE prompt/decrypt migration.
5. Search idle-state migration.
6. Legacy call-site removal and cleanup.
## Confirmed decisions
1. `request-id` format: UUID v4.
2. Default UI request timeout: 60s (`60000ms`), optional per-action override.
3. Search idle state model: main-thread push + worker TTL cache.
4. Headless behavior: UI-required operations return typed error `:ui-interaction-required`.
5. Execution order: follow the phased order defined in this document.
## Notes
This plan intentionally prioritizes behavior parity and migration safety over immediate deep refactor of all related modules.
The protocol and platform abstraction should be stabilized first; broad code movement should follow once test harnesses are in place.

4
src/main/frontend/components/repo.cljs
View File

@@ -452,7 +452,9 @@
(state/<invoke-db-worker :thread-api/set-db-sync-config
{:enabled? true
:ws-url config/db-sync-ws-url
:http-base config/db-sync-http-base})
:http-base config/db-sync-http-base
:oauth-domain config/OAUTH-DOMAIN
:oauth-client-id config/COGNITO-CLIENT-ID})
(p/let [rsa-key-pair (state/<invoke-db-worker :thread-api/db-sync-ensure-user-rsa-keys)]
(set-e2ee-rsa-key-ensured? (some? rsa-key-pair))))
(p/catch (fn [e]

4
src/main/frontend/handler/events/ui.cljs
View File

@@ -332,7 +332,9 @@
(state/<invoke-db-worker :thread-api/set-db-sync-config
{:enabled? true
:ws-url config/db-sync-ws-url
:http-base config/db-sync-http-base})
:http-base config/db-sync-http-base
:oauth-domain config/OAUTH-DOMAIN
:oauth-client-id config/COGNITO-CLIENT-ID})
(state/<invoke-db-worker :thread-api/db-sync-ensure-user-rsa-keys))
(p/catch (fn [error]
(log/error :db-sync/ensure-user-rsa-keys-failed error)

4
src/main/frontend/persist_db/browser.cljs
View File

@@ -143,7 +143,9 @@
_ (state/<invoke-db-worker :thread-api/set-db-sync-config
{:enabled? true
:ws-url config/db-sync-ws-url
:http-base config/db-sync-http-base})
:http-base config/db-sync-http-base
:oauth-domain config/OAUTH-DOMAIN
:oauth-client-id config/COGNITO-CLIENT-ID})
_ (state/pub-event! [:rtc/sync-app-state])
_ (log/info "init worker spent" (str (- (util/time-ms) t1) "ms"))
_ (sync-ui-state!)

18
src/main/frontend/worker/platform.cljs
View File

@@ -75,6 +75,24 @@
(f path text)
(throw (ex-info "platform storage/write-text! missing" {:path path}))))
(defn save-secret-text!
[platform key text]
(if-let [f (get-in platform [:crypto :save-secret-text!])]
(f key text)
(throw (ex-info "platform crypto/save-secret-text! missing" {:key key}))))
(defn read-secret-text
[platform key]
(if-let [f (get-in platform [:crypto :read-secret-text])]
(f key)
(throw (ex-info "platform crypto/read-secret-text missing" {:key key}))))
(defn delete-secret-text!
[platform key]
(if-let [f (get-in platform [:crypto :delete-secret-text!])]
(f key)
(throw (ex-info "platform crypto/delete-secret-text! missing" {:key key}))))
(defn websocket-connect
[platform url]
(if-let [f (get-in platform [:websocket :connect])]

22
src/main/frontend/worker/platform/browser.cljs
View File

@@ -60,6 +60,24 @@
[k value]
(idb-keyval/set k value @kv-store))
(def ^:private secret-prefix "worker-secret###")
(defn- secret-key
[key]
(str secret-prefix key))
(defn- save-secret-text!
[key text]
(kv-set! (secret-key key) text))
(defn- read-secret-text
[key]
(kv-get (secret-key key)))
(defn- delete-secret-text!
[key]
(kv-set! (secret-key key) nil))
(defn- install-opfs-pool
[sqlite pool-name]
(.installOpfsSAHPoolVfs ^js sqlite #js {:name pool-name
@@ -126,5 +144,7 @@
:close-db (fn [db] (.close db))
:exec (fn [db sql-or-opts] (.exec db sql-or-opts))
:transaction (fn [db f] (.transaction db f))}
:crypto {}
:crypto {:save-secret-text! save-secret-text!
:read-secret-text read-secret-text
:delete-secret-text! delete-secret-text!}
:timers {:set-interval! (fn [f ms] (js/setInterval f ms))}})

13
src/main/frontend/worker/platform/node.cljs
View File

@@ -298,6 +298,12 @@
[state]
(transit/write kv-transit-writer state))
(def ^:private secret-prefix "worker-secret###")
(defn- secret-key
[key]
(str secret-prefix key))
(defn- kv-store
[data-dir]
(let [kv-path (node-path/join data-dir "kv-store.json")
@@ -359,5 +365,10 @@
:backup-db (fn [db path]
(let [backup-fn (gobj/get db "backup")]
(backup-fn path)))}
:crypto {}
:crypto {:save-secret-text! (fn [key text]
((:set! kv) (secret-key key) text))
:read-secret-text (fn [key]
((:get kv) (secret-key key)))
:delete-secret-text! (fn [key]
((:set! kv) (secret-key key) nil))}
:timers {:set-interval! (fn [f ms] (js/setInterval f ms))}})))

65
src/main/frontend/worker/sync/auth.cljs
View File

@@ -2,10 +2,10 @@
"Auth and endpoint helpers for db sync."
(:require [clojure.string :as string]
[frontend.worker-common.util :as worker-util]
[logseq.common.util :as common-util]
[promesa.core :as p]
[frontend.worker.state :as worker-state]
[frontend.worker.sync.util :as sync-util]
[frontend.worker.state :as worker-state]))
[logseq.common.util :as common-util]
[promesa.core :as p]))
(defn ws-base-url
[db-sync-config]
@@ -36,16 +36,59 @@
(catch :default _
true))))
(defn oauth-token-url
[db-sync-config]
(or (:oauth-token-url db-sync-config)
(when-let [domain (not-empty (:oauth-domain db-sync-config))]
(str "https://" domain "/oauth2/token"))))
(defn <refresh-id&access-token
[]
(let [refresh-token (:auth/refresh-token @worker-state/*state)
db-sync-config @worker-state/*db-sync-config
token-url (oauth-token-url db-sync-config)
oauth-client-id (:oauth-client-id db-sync-config)]
(when-not (seq refresh-token)
(throw (ex-info "worker auth refresh requires refresh token"
{:code :missing-refresh-token})))
(when-not (seq token-url)
(throw (ex-info "worker auth refresh requires oauth token url"
{:code :missing-oauth-token-url})))
(when-not (seq oauth-client-id)
(throw (ex-info "worker auth refresh requires oauth client id"
{:code :missing-oauth-client-id})))
(let [form-data (js/URLSearchParams.)]
(.set form-data "grant_type" "refresh_token")
(.set form-data "client_id" oauth-client-id)
(.set form-data "refresh_token" refresh-token)
(p/let [resp (js/fetch token-url #js {:method "POST"
:headers #js {"content-type" "application/x-www-form-urlencoded"}
:body (.toString form-data)})
text (.text resp)
data (when (seq text)
(js->clj (js/JSON.parse text) :keywordize-keys true))]
(if (.-ok resp)
{:id-token (:id_token data)
:access-token (:access_token data)}
(throw (ex-info "worker auth refresh failed"
{:code :auth-refresh-failed
:status (.-status resp)
:token-url token-url
:body data})))))))
(defn <resolve-ws-token
[]
(let [token (sync-util/auth-token)]
(if (and (not (sync-util/cli-node-owner?))
(id-token-expired? token))
(p/let [resp (worker-state/<invoke-main-thread :thread-api/ensure-id&access-token)
refreshed-token (:id-token resp)]
(when (string? refreshed-token)
(worker-state/set-new-state! {:auth/id-token refreshed-token})
refreshed-token))
(let [token (sync-util/auth-token)
token-expired? (id-token-expired? token)]
(if (and (not (sync-util/cli-node-owner?)) token-expired?)
(p/let [{:keys [id-token access-token]} (<refresh-id&access-token)]
(when-not (seq id-token)
(throw (ex-info "worker auth refresh returned empty id-token"
{:code :auth-refresh-empty-id-token})))
(worker-state/set-new-state!
(cond-> {:auth/id-token id-token}
(seq access-token) (assoc :auth/access-token access-token)))
id-token)
(p/resolved token))))
(defn get-user-uuid

20
src/main/frontend/worker/sync/crypt.cljs
View File

@@ -16,6 +16,7 @@
(defonce ^:private *graph->aes-key (atom {}))
(defonce ^:private *user-rsa-key-pair-inflight (atom {}))
(defonce ^:private e2ee-password-file "e2ee-password")
(defonce ^:private e2ee-password-secret-key "logseq-encrypted-password")
(defonce ^:private native-env?
(let [href (try (.. js/self -location -href)
(catch :default _ nil))]
@@ -31,15 +32,15 @@
(defn <native-save-password-text!
[encrypted-text]
(worker-state/<invoke-main-thread :thread-api/native-save-e2ee-password encrypted-text))
(platform/save-secret-text! (platform/current) e2ee-password-secret-key encrypted-text))
(defn- <native-read-password-text
[]
(worker-state/<invoke-main-thread :thread-api/native-get-e2ee-password))
(platform/read-secret-text (platform/current) e2ee-password-secret-key))
(defn- <native-delete-password-text!
[]
(worker-state/<invoke-main-thread :thread-api/native-delete-e2ee-password))
(platform/delete-secret-text! (platform/current) e2ee-password-secret-key))
(defn- <save-e2ee-password
[refresh-token password]
@@ -55,9 +56,16 @@
(defn- <read-e2ee-password
[refresh-token]
(p/let [text (if (native-worker?)
(<native-read-password-text)
(platform/read-text! (platform/current) e2ee-password-file))
(p/let [platform' (platform/current)
text (if (native-worker?)
(-> (p/let [native-text (<native-read-password-text)]
(if (seq native-text)
native-text
(platform/read-text! platform' e2ee-password-file)))
(p/catch (fn [e]
(log/error :native-get-e2ee-password {:error e})
(platform/read-text! platform' e2ee-password-file))))
(platform/read-text! platform' e2ee-password-file))
data (ldb/read-transit-str text)
password (crypt/<decrypt-text-by-text-password refresh-token data)]
password))

2
src/test/frontend/handler/db_based/sync_test.cljs
View File

@@ -386,7 +386,7 @@
(p/then (fn [_]
(is (= 1 (count @worker-calls)))
(let [[op graph graph-uuid graph-e2ee?] (first @worker-calls)]
(is (= :thread-api/db-sync-download-graph op))
(is (= :thread-api/db-sync-download-graph-by-id op))
(is (string/ends-with? graph "demo-graph"))
(is (= "graph-1" graph-uuid))
(is (= false graph-e2ee?)))

108
src/test/frontend/worker/db_sync_test.cljs
View File

@@ -14,6 +14,7 @@
[frontend.worker.sync :as db-sync]
[frontend.worker.sync.apply-txs :as sync-apply]
[frontend.worker.sync.assets :as sync-assets]
[frontend.worker.sync.auth :as sync-auth]
[frontend.worker.sync.client-op :as client-op]
[frontend.worker.sync.crypt :as sync-crypt]
[frontend.worker.sync.handle-message :as sync-handle-message]
@@ -219,34 +220,99 @@
:a-child-1 (db-test/find-block-by-content @conn "a child 1")
:b-child-1 (db-test/find-block-by-content @conn "b child 1")}))
(deftest resolve-ws-token-refreshes-when-token-expired-test
(deftest resolve-ws-token-prefers-worker-local-refresh-when-token-expired-test
(async done
(let [refresh-calls (atom 0)
(let [fetch-calls (atom [])
main-thread-calls (atom 0)
main-thread-prev @worker-state/*main-thread
worker-state-prev @worker-state/*state]
(reset! worker-state/*state (assoc worker-state-prev :auth/id-token "expired-token"))
worker-state-prev @worker-state/*state
sync-config-prev @worker-state/*db-sync-config
fetch-prev js/fetch]
(reset! worker-state/*db-sync-config {:feature-flags {:worker-auth-refresh? true}
:oauth-token-url "https://auth.example.com/oauth2/token"
:oauth-client-id "worker-client-id"})
(reset! worker-state/*state (assoc worker-state-prev
:auth/id-token "expired-token"
:auth/refresh-token "refresh-token"))
(reset! worker-state/*main-thread
(fn [qkw _direct-pass? _args-list]
(if (= qkw :thread-api/ensure-id&access-token)
(do
(swap! refresh-calls inc)
(p/resolved {:id-token "fresh-token"}))
(p/resolved nil))))
(with-redefs [db-sync/auth-token (fn [] "expired-token")
db-sync/id-token-expired? (fn [_token] true)]
(when (= qkw :thread-api/ensure-id&access-token)
(swap! main-thread-calls inc))
(p/resolved {:id-token "legacy-token"})))
(set! js/fetch
(fn [url opts]
(swap! fetch-calls conj {:url url :opts opts})
(let [resp (js-obj)]
(aset resp "ok" true)
(aset resp "status" 200)
(aset resp "text"
(fn []
(p/resolved "{\"id_token\":\"fresh-worker-token\",\"access_token\":\"fresh-worker-access-token\"}")))
(p/resolved resp))))
(with-redefs [sync-util/auth-token (fn [] "expired-token")
sync-auth/id-token-expired? (fn [_token] true)]
(-> (#'db-sync/<resolve-ws-token)
(p/then (fn [token]
(is (= 1 @refresh-calls))
(is (= "fresh-token" token))
(is (= "fresh-token" (worker-state/get-id-token)))
(reset! worker-state/*main-thread main-thread-prev)
(reset! worker-state/*state worker-state-prev)
(done)))
(is (= 1 (count @fetch-calls)))
(is (= 0 @main-thread-calls))
(is (= "fresh-worker-token" token))
(is (= "fresh-worker-token" (worker-state/get-id-token)))
(is (= "fresh-worker-access-token"
(:auth/access-token @worker-state/*state)))))
(p/catch (fn [error]
(reset! worker-state/*main-thread main-thread-prev)
(reset! worker-state/*state worker-state-prev)
(is nil (str error))
(done))))))))
(is nil (str error))))
(p/finally (fn []
(set! js/fetch fetch-prev)
(reset! worker-state/*main-thread main-thread-prev)
(reset! worker-state/*state worker-state-prev)
(reset! worker-state/*db-sync-config sync-config-prev)
(done))))))))
(deftest resolve-ws-token-does-not-fallback-to-main-thread-when-feature-flag-disabled-test
(async done
(let [fetch-calls (atom 0)
main-thread-calls (atom 0)
main-thread-prev @worker-state/*main-thread
worker-state-prev @worker-state/*state
sync-config-prev @worker-state/*db-sync-config
fetch-prev js/fetch]
(reset! worker-state/*db-sync-config {:feature-flags {:worker-auth-refresh? false}
:oauth-token-url "https://auth.example.com/oauth2/token"
:oauth-client-id "worker-client-id"})
(reset! worker-state/*state (assoc worker-state-prev
:auth/id-token "expired-token"
:auth/refresh-token "refresh-token"))
(reset! worker-state/*main-thread
(fn [qkw _direct-pass? _args-list]
(when (= qkw :thread-api/ensure-id&access-token)
(swap! main-thread-calls inc))
(p/resolved {:id-token "fresh-legacy-token"})))
(set! js/fetch
(fn [_url _opts]
(swap! fetch-calls inc)
(let [resp (js-obj)]
(aset resp "ok" true)
(aset resp "status" 200)
(aset resp "text"
(fn []
(p/resolved "{\"id_token\":\"fresh-worker-token-2\",\"access_token\":\"fresh-worker-access-token-2\"}")))
(p/resolved resp))))
(with-redefs [sync-util/auth-token (fn [] "expired-token")
sync-auth/id-token-expired? (fn [_token] true)]
(-> (#'db-sync/<resolve-ws-token)
(p/then (fn [token]
(is (= 1 @fetch-calls))
(is (= 0 @main-thread-calls))
(is (= "fresh-worker-token-2" token))
(is (= "fresh-worker-token-2" (worker-state/get-id-token)))))
(p/catch (fn [error]
(is nil (str error))))
(p/finally (fn []
(set! js/fetch fetch-prev)
(reset! worker-state/*main-thread main-thread-prev)
(reset! worker-state/*state worker-state-prev)
(reset! worker-state/*db-sync-config sync-config-prev)
(done))))))))
(deftest update-online-users-dedupes-identical-messages-test
(let [client {:repo test-repo

58
src/test/frontend/worker/sync/crypt_test.cljs
View File

@@ -131,6 +131,64 @@
(is false (str e))))
(p/finally done)))))
(deftest save-e2ee-password-native-uses-platform-secret-storage-test
(async done
(let [platform-map {:runtime :test}
secret-calls (atom [])
invoke-calls (atom 0)]
(-> (p/with-redefs [sync-crypt/native-worker? (fn [] true)
crypt/<encrypt-text-by-text-password (fn [_refresh-token _password]
{:cipher "payload"})
platform/current (fn [] platform-map)
platform/save-secret-text! (fn [platform' key text]
(swap! secret-calls conj {:platform platform'
:key key
:text text})
(p/resolved nil))
platform/write-text! (fn [_platform' _path _text]
(p/rejected (ex-info "should not fallback for successful native save" {})))
worker-state/<invoke-main-thread (fn [& _]
(swap! invoke-calls inc)
(p/rejected (ex-info "should not call invoke-main-thread" {})))]
(#'sync-crypt/<save-e2ee-password "refresh-token" "password"))
(p/then (fn [_]
(is (= 1 (count @secret-calls)))
(is (= platform-map (:platform (first @secret-calls))))
(is (= "logseq-encrypted-password" (:key (first @secret-calls))))
(is (string? (:text (first @secret-calls))))
(is (zero? @invoke-calls))))
(p/catch (fn [e]
(is false (str e))))
(p/finally done)))))
(deftest read-e2ee-password-native-falls-back-to-file-when-secret-read-fails-test
(async done
(let [platform-map {:runtime :test}
fallback-read-calls (atom 0)
invoke-calls (atom 0)]
(-> (p/with-redefs [sync-crypt/native-worker? (fn [] true)
platform/current (fn [] platform-map)
platform/read-secret-text (fn [_platform' _key]
(p/rejected (ex-info "secret store unavailable" {})))
platform/read-text! (fn [platform' path]
(swap! fallback-read-calls inc)
(is (= platform-map platform'))
(is (= "e2ee-password" path))
(ldb/write-transit-str {:cipher "payload"}))
crypt/<decrypt-text-by-text-password (fn [_refresh-token _data]
"decrypted-password")
worker-state/<invoke-main-thread (fn [& _]
(swap! invoke-calls inc)
(p/rejected (ex-info "should not call invoke-main-thread" {})))]
(#'sync-crypt/<read-e2ee-password "refresh-token"))
(p/then (fn [password]
(is (= "decrypted-password" password))
(is (= 1 @fallback-read-calls))
(is (zero? @invoke-calls))))
(p/catch (fn [e]
(is false (str e))))
(p/finally done)))))
(deftest save-e2ee-password-native-fallback-uses-platform-write-text-test
(async done
(let [platform-map {:runtime :test}