diff --git a/docs/agent-guide/047-logseq-cli-sync-command.md b/docs/agent-guide/047-logseq-cli-sync-command.md new file mode 100644 index 0000000000..945d496e5e --- /dev/null +++ b/docs/agent-guide/047-logseq-cli-sync-command.md @@ -0,0 +1,225 @@ +# Logseq CLI Sync Command Implementation Plan + +Goal: Add `logseq sync` subcommands to inspect and operate db-sync through existing db-worker-node APIs. + +Architecture: The CLI parser and executor will gain a dedicated sync command module that maps subcommands to `:thread-api/db-sync-*` calls via `/v1/invoke`. +Architecture: A small worker API addition will expose runtime sync status, and sync config commands will support headless token setup through CLI-managed config values. +Architecture: The design will reuse existing graph lock and repo binding behavior in `logseq.cli.server/ensure-server!` and `frontend.worker.db-worker-node/repo-error`. + +Tech Stack: ClojureScript, babashka.cli, promesa, db-worker-node HTTP API, frontend.worker.sync. + +Related: Builds on `docs/agent-guide/031-logseq-cli-doctor-command.md`, `docs/agent-guide/033-desktop-db-worker-node-backend.md`, and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +The current CLI exposes graph, server, doctor, list, upsert, remove, query, and show commands, but it does not expose db-sync control or observability. + +`frontend.worker.db-core` already exposes operational db-sync thread APIs such as `:thread-api/db-sync-start`, `:thread-api/db-sync-stop`, `:thread-api/db-sync-upload-graph`, and `:thread-api/db-sync-grant-graph-access`. + +`frontend.worker.db-worker-node` already routes these methods over `/v1/invoke` with repo-lock safety checks, so the missing piece is a CLI command surface and one read API for status inspection. + +This plan keeps scope tight by reusing current transport and server lifecycle code, and only adds new worker behavior where inspection data is currently unavailable. + +I will use @planning-documents for naming, @writing-plans for task granularity, @logseq-cli for CLI integration expectations, and @test-driven-development for implementation sequence. + +## Testing Plan + +I will add parser and action unit tests that fail first for new `sync` command help, option validation, and action shaping. + +I will add command execution tests that fail first and verify `logseq.cli.transport/invoke` receives the exact method names and argument shapes for each sync subcommand. + +I will add format tests that fail first and verify human output for `sync status` and action commands, while keeping JSON and EDN behavior unchanged. + +I will add worker API tests that fail first for the new sync inspection API exposed through `/v1/invoke`. + +I will add one CLI integration test that fails first and verifies an end-to-end `sync status` flow on a temp graph and a started db-worker-node process. + +I will run targeted tests after each behavior slice and then run `bb dev:lint-and-test` before final review. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and CLI surface + +| CLI command | Purpose | Worker method | Repo required | +|---|---|---|---| +| `sync status [--graph ]` | View current db-sync runtime state and counters. | `:thread-api/db-sync-status` (new). | Yes. | +| `sync start [--graph ]` | Start db-sync websocket client for the graph. | `:thread-api/db-sync-start`. | Yes. | +| `sync stop [--graph ]` | Stop db-sync client for the running daemon. | `:thread-api/db-sync-stop`. | Yes, to target a graph daemon deterministically. | +| `sync upload [--graph ]` | Upload current graph snapshot and mark graph remote metadata. | `:thread-api/db-sync-upload-graph`. | Yes. | +| `sync download [--graph ]` | Download remote graph data and apply it to local graph storage. | `:thread-api/db-sync-download-graph` (new). | Yes. | +| `sync remote-graphs` | List remote graphs visible to current auth context. | `:thread-api/db-sync-list-remote-graphs` (new). | No. | +| `sync ensure-keys` | Ensure user RSA keys required by e2ee are present. | `:thread-api/db-sync-ensure-user-rsa-keys`. | No. | +| `sync grant-access --graph-id --email [--graph ]` | Grant encrypted graph key access to a target user email. | `:thread-api/db-sync-grant-graph-access`. | Yes. | +| `sync config set ` | Set one config value by key. | `:thread-api/set-db-sync-config`. | No. | +| `sync config get ` | Read one config value by key. | `:thread-api/get-db-sync-config` (new). | No. | +| `sync config unset ` | Remove one config value by key. | `:thread-api/set-db-sync-config`. | No. | + +The first release intentionally excludes asset download and raw kv import commands because they need more user-facing safety rails and payload tooling. + +`sync config set` supports `ws-url`, `http-base`, and `auth-token`, and `config set auth-token ` is the headless authentication entrypoint. + +`sync config get` and `sync config unset` reject unknown config keys. + +`sync status` will return normalized fields even when sync is not configured, so scripts can branch deterministically. + +`sync remote-graphs` and `sync download` require auth-token to be configured in headless mode. + +## Architecture and integration points + +```text +logseq sync + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs (parse/build/execute) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs (new) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs (ensure graph daemon) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs (POST /v1/invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs (repo checks + invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs thread APIs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs and sync/crypt.cljs +``` + +Worker additions will be minimal, with no protocol changes to cloud endpoints. + +CLI additions will follow existing `graph` and `server` command module patterns for spec, `entries`, `build-action`, and `execute-*` helpers. + +## Implementation plan + +### Phase 1. Add failing parser and help tests. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that top-level help includes `sync` and `sync status`. +2. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `logseq sync` shows subgroup help like `server` and `graph`. +3. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `sync config set|get|unset` and `sync grant-access` show in sync group help. +4. Run `bb dev:test -v logseq.cli.commands-test/test-help-output` and confirm failure references missing `sync` command rows. + +### Phase 2. Add failing action and execution tests for sync command module. + +5. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` with failing tests for `build-action` graph requirement on `sync status`. +6. Add a failing test for `build-action` rejection when `sync config set` is missing name or value. +7. Add a failing test for `build-action` rejection when `sync grant-access` misses `--graph-id` or `--email`. +8. Add a failing test for `build-action` graph requirement on `sync download`. +9. Add a failing execution test that stubs `logseq.cli.server/ensure-server!` and `logseq.cli.transport/invoke` and expects `:thread-api/db-sync-start` with `[repo]`. +10. Add a failing execution test that expects `:thread-api/db-sync-stop` with `[]` and still routes through `ensure-server!` using selected repo. +11. Add a failing execution test that expects `:thread-api/db-sync-upload-graph` with `[repo]`. +12. Add a failing execution test that expects `:thread-api/db-sync-download-graph` with `[repo]`. +13. Add a failing execution test that expects `:thread-api/db-sync-list-remote-graphs` for `sync remote-graphs`. +14. Add a failing execution test that expects `:thread-api/db-sync-ensure-user-rsa-keys` without repo. +15. Add a failing execution test that expects `:thread-api/db-sync-grant-graph-access` with `[repo graph-id email]`. +16. Add a failing execution test that expects `:thread-api/get-db-sync-config` for `sync config get `. +17. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config set ` and payload merge behavior. +18. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config unset ` and key removal behavior. +19. Add a failing execution test that verifies `sync config set auth-token ` updates worker-consumable token config for headless mode. +20. Run `bb dev:test -v logseq.cli.command.sync-test` and confirm failures are only from missing sync implementation. + +### Phase 3. Implement CLI sync command wiring. + +21. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` with sync option specs, `entries`, `build-action`, and `execute-*` functions. +22. Register `sync-command/entries` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` command table. +23. Extend `finalize-command` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` with sync-specific required-option checks. +24. Extend single-token group help routing in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to include `sync`. +25. Extend `build-action` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to call `sync-command/build-action`. +26. Extend `execute` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to route sync action types. +27. Add `sync` to top-level command grouping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs`. +28. Run `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` and `bb dev:test -v logseq.cli.command.sync-test` until green. + +### Phase 4. Add read-only worker APIs for sync inspection. + +29. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` accepts `thread-api/get-db-sync-config` without repo. +30. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` for `thread-api/db-sync-status` enforces repo and returns structured status. +31. Add `:thread-api/get-db-sync-config` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning current config map. +32. Add `:thread-api/db-sync-status` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning ws state, graph id, and sync counters for a repo. +33. Add or expose a small helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to compute status without requiring websocket side effects. +34. Add `:thread-api/db-sync-list-remote-graphs` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement cloud graph listing through worker sync HTTP helpers. +35. Add `:thread-api/db-sync-download-graph` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement remote snapshot download plus local import flow. +36. Update worker sync auth token resolution so `sync config set auth-token ` is used in headless mode when state token is missing. +37. Register `:thread-api/get-db-sync-config` and `:thread-api/db-sync-list-remote-graphs` in `non-repo-methods` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`. +38. Run `bb dev:test -v frontend.worker.db-worker-node-test` and fix only sync-related regressions. + +### Phase 5. Add output formatting tests and implementation. + +39. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync status`. +40. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync remote-graphs`. +41. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of sync action commands such as start, upload, and download. +42. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` verifying token redaction for `sync config get auth-token` in human output. +43. Implement sync human formatters in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with stable keys and token redaction. +44. Confirm JSON and EDN output behavior by running `bb dev:test -v logseq.cli.format-test`. + +### Phase 6. Add integration coverage and CLI docs. + +45. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a graph and runs `sync status` with `--output json`. +46. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config set auth-token` then `sync config get auth-token` behavior. +47. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config unset auth-token`. +48. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync remote-graphs --output json`. +49. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync download --graph ` flow with mocked remote snapshot response. +50. Implement any missing glue for integration stability in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs`. +51. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` command docs with sync command examples and error behaviors. +52. Run `bb dev:test -v logseq.cli.integration-test` to verify end-to-end behavior. + +### Phase 7. Final verification and cleanup. + +53. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and confirm exit code `0`. +54. Run manual smoke commands with a temp graph and confirm both `--output human` and `--output json` are stable. +55. Review help text alignment and command ordering to match existing CLI aesthetics. + +## Edge cases and error handling + +`sync status` must return a valid map even when `:ws-url` is missing, with an explicit inactive state rather than throwing. + +`sync start` must keep current behavior where missing `ws-url` or missing graph uuid results in no crash and a deterministic status response. + +`sync grant-access` must surface cloud errors with existing `http-error` path and preserve status code and body context. + +`sync config get auth-token` must redact token values in human output while keeping full value available in JSON and EDN output for scripting. + +`sync config set auth-token ` must write to the config file selected by `--config` (default `~/logseq/cli.edn`) so headless auth survives daemon restarts. + +`sync remote-graphs` must return a deterministic empty list when user has no remote graphs instead of returning nil. + +`sync download` must fail fast when the target local graph is missing required auth or remote graph metadata, and must report a clear sync-specific error code. + +Repo mismatch and lock ownership behavior must remain enforced by db-worker-node and must not be bypassed in CLI command code. + +All new options must keep kebab-case keyword naming and avoid introducing `_` forms. + +## Verification commands + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.command.sync-test` | Sync command unit tests pass with no failures. | +| `bb dev:test -v logseq.cli.commands-test/test-help-output` | Help output includes `sync` group and subcommands. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Worker invoke tests pass including new sync read APIs. | +| `bb dev:test -v logseq.cli.format-test` | Human and structured output tests pass including sync formatters. | +| `bb dev:test -v logseq.cli.integration-test` | CLI integration tests pass for sync status and config flow. | +| `bb dev:lint-and-test` | Full lint and unit suite passes with exit code `0`. | +| `node ./dist/logseq.js sync status --graph demo --output json` | Returns `{"status":"ok","data":...}` with sync status fields. | +| `node ./dist/logseq.js sync remote-graphs --output json` | Returns remote graph list in structured output. | +| `node ./dist/logseq.js sync download --graph demo` | Downloads remote graph snapshot and imports it into local graph data. | +| `node ./dist/logseq.js sync config set auth-token ` | Sets headless auth token for db-sync API calls. | +| `node ./dist/logseq.js sync config get auth-token --output json` | Returns configured token value in structured output. | +| `node ./dist/logseq.js sync config unset auth-token` | Removes configured token and returns success message. | + +## Testing Details + +The new tests verify behavior at parser level, action-building level, transport payload level, worker invoke contract level, output formatting level, and end-to-end CLI invocation level. + +The tests assert external behavior such as command availability, returned status payloads, and worker method invocations, instead of asserting internal helper implementation details. + +## Implementation Details + +- Add new file `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` for sync command ownership. +- Keep `sync` command wiring inside existing dispatch points in `commands.cljs` and do not introduce a second dispatcher. +- Add core worker sync inspection APIs, `get-db-sync-config` and `db-sync-status`, and reuse existing `set-db-sync-config` for config writes. +- Add worker sync APIs for remote graph listing and graph download to support `sync remote-graphs` and `sync download`. +- Reuse `transport/invoke` with existing `direct-pass?` handling and default to transit mode. +- Keep `sync status` output fields stable for scripting, including `repo`, `graph-id`, `ws-state`, and pending counters. +- Keep human output terse and redact auth-token values. +- Update `command.core/top-level-summary` and group-help routing so `sync` behaves like existing command groups. +- Keep all new keyword names kebab-case and avoid shadowed local names such as `bytes`. +- Update `docs/cli/logseq-cli.md` with command list, examples, and expected error hints. +- Run full lint and tests after targeted green passes. + +## Question + +No open question. + +This plan adopts option A and includes `sync config set|get|unset` with `config set auth-token ` as the token setup path. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 725b4b22b1..7f7c908c40 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -90,6 +90,30 @@ Server ownership behavior: - `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. - `server list` human output includes an `OWNER` column, and `server status` / `server list` include owner metadata in structured output (`--output json|edn`). +Sync commands: +- `sync status --graph ` - show db-sync runtime state for a graph daemon +- `sync start --graph ` - start db-sync websocket client for a graph +- `sync stop --graph ` - stop db-sync client on a graph daemon +- `sync upload --graph ` - upload local graph snapshot to remote +- `sync download --graph ` - download remote graph `` into a same-name local graph directory +- `sync remote-graphs [--graph ]` - list remote graphs visible to the current auth context +- `sync ensure-keys [--graph ]` - ensure user RSA keys for sync/e2ee +- `sync grant-access --graph --graph-id --email ` - grant encrypted graph access to a user +- `sync config set [--graph ] ws-url|http-base|auth-token|e2ee-password ` - set db-sync runtime config key +- `sync config get [--graph ] ws-url|http-base|auth-token|e2ee-password` - get db-sync runtime config key +- `sync config unset [--graph ] ws-url|http-base|auth-token|e2ee-password` - remove db-sync runtime config key + +Sync download behavior: +- `sync download` requires `--graph `. +- If a local graph with the same name already exists, the CLI returns `graph-exists`. +- If no remote graph with that name exists, the CLI returns `remote-graph-not-found`. +- For e2ee remote graphs in headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before download. + +Sync config persistence: +- `sync config set/unset` writes to the CLI config file selected by `--config`. +- If `--config` is not provided, the default config path is `~/logseq/cli.edn`. +- `sync config get` reads from that same config source. + Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 3241ac9a06..0f198be4eb 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -394,6 +394,14 @@ (reset! worker-state/*db-sync-config config) nil) +(def-thread-api :thread-api/get-db-sync-config + [] + @worker-state/*db-sync-config) + +(def-thread-api :thread-api/db-sync-status + [repo] + (db-sync/status repo)) + (def-thread-api :thread-api/db-sync-start [repo] (db-sync/start! repo)) @@ -418,10 +426,40 @@ [] (sync-crypt/ensure-user-rsa-keys!)) +(def-thread-api :thread-api/db-sync-list-remote-graphs + [] + (db-sync/list-remote-graphs!)) + (def-thread-api :thread-api/db-sync-upload-graph [repo] (db-sync/upload-graph! repo)) +(def-thread-api :thread-api/db-sync-download-graph + [repo] + (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph! repo) + row-count (count rows) + _ (when (seq rows) + ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) + repo rows true graph-id remote-tx graph-e2ee?))] + {:repo repo + :graph-id graph-id + :remote-tx remote-tx + :graph-e2ee? graph-e2ee? + :row-count row-count})) + +(def-thread-api :thread-api/db-sync-download-graph-by-id + [repo graph-id graph-e2ee?] + (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph-by-id! repo graph-id graph-e2ee?) + row-count (count rows) + _ (when (seq rows) + ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) + repo rows true graph-id remote-tx graph-e2ee?))] + {:repo repo + :graph-id graph-id + :remote-tx remote-tx + :graph-e2ee? graph-e2ee? + :row-count row-count})) + (def-thread-api :thread-api/set-infer-worker-proxy [infer-worker-proxy] (reset! worker-state/*infer-worker infer-worker-proxy) @@ -987,22 +1025,30 @@ (defn- prev-graph close-db!) - (when graph - (if (= graph prev-graph) - service - (do - (log/info :db-worker/init-service {:graph graph - :prev-graph prev-graph - :import-type (:import-type start-opts)}) - (p/let [service (shared-service/js fns) - #(on-become-master graph start-opts) - broadcast-data-types - {:import? (:import-type? start-opts)})] - (assert (p/promise? (get-in service [:status :ready]))) - (reset! *service [graph service]) - service)))))) + (cond + (nil? graph) + (do + (some-> prev-graph close-db!) + nil) + + (and (= graph prev-graph) service) + service + + :else + (do + (when (and prev-graph (not= graph prev-graph)) + (close-db! prev-graph)) + (log/info :db-worker/init-service {:graph graph + :prev-graph prev-graph + :import-type (:import-type start-opts)}) + (p/let [service (shared-service/js fns) + #(on-become-master graph start-opts) + broadcast-data-types + {:import? (:import-type? start-opts)})] + (assert (p/promise? (get-in service [:status :ready]))) + (reset! *service [graph service]) + service))))) (defn- notify-invalid-data [{:keys [tx-meta]} errors] diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 7bbaf63bec..f84b8e2102 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -129,7 +129,9 @@ (def ^:private non-repo-methods #{:thread-api/init :thread-api/set-db-sync-config + :thread-api/get-db-sync-config :thread-api/db-sync-stop + :thread-api/db-sync-list-remote-graphs :thread-api/db-sync-update-presence :thread-api/db-sync-ensure-user-rsa-keys :thread-api/list-db diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index 9cf77a0e88..64235a62da 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -57,6 +57,22 @@ :remote-tx remote-tx :graph-uuid graph-uuid}))) +(defn status + [repo] + (let [client (current-client repo) + counts (or (sync-counts repo) {}) + ws-url (:ws-url @worker-state/*db-sync-config) + ws-state (or (some-> client :ws-state deref) + (if (seq ws-url) :stopped :inactive))] + {:repo repo + :graph-id (or (:graph-id client) (:graph-uuid counts)) + :ws-state ws-state + :pending-local (or (:pending-local counts) 0) + :pending-asset (or (:pending-asset counts) 0) + :pending-server (or (:pending-server counts) 0) + :local-tx (:local-tx counts) + :remote-tx (:remote-tx counts)})) + (defn- normalize-online-users [users] (->> users @@ -133,7 +149,8 @@ (string/replace base #"/sync/%s$" ""))))) (defn- auth-token [] - (worker-state/get-id-token)) + (or (worker-state/get-id-token) + (:auth-token @worker-state/*db-sync-config))) (defn- id-token-expired? [token] @@ -408,6 +425,102 @@ :url url :body body})))))) +(defn- require-auth-token! + [context] + (when-not (seq (auth-token)) + (fail-fast :db-sync/missing-field (assoc context :field :auth-token)))) + +(defn- ->uint8 + [payload] + (cond + (instance? js/Uint8Array payload) payload + (instance? js/ArrayBuffer payload) (js/Uint8Array. payload) + (string? payload) (.encode text-encoder payload) + :else (js/Uint8Array. payload))) + +(defn- decode-snapshot-rows + [payload] + (sqlite-util/read-transit-str (.decode text-decoder (->uint8 payload)))) + +(defn- frame-len + [^js payload offset] + (let [view (js/DataView. (.-buffer payload) offset 4)] + (.getUint32 view 0 false))) + +(defn- concat-payload + [^js a ^js b] + (cond + (nil? a) b + (nil? b) a + :else + (let [combined (js/Uint8Array. (+ (.-byteLength a) (.-byteLength b)))] + (.set combined a 0) + (.set combined b (.-byteLength a)) + combined))) + +(defn- parse-framed-chunk + [buffer chunk] + (let [payload (concat-payload buffer chunk) + total (.-byteLength payload)] + (loop [offset 0 + rows []] + (if (< (- total offset) 4) + {:rows rows + :buffer (when (< offset total) + (.slice payload offset total))} + (let [len (frame-len payload offset) + next-offset (+ offset 4 len)] + (if (<= next-offset total) + (let [frame-payload (.slice payload (+ offset 4) next-offset) + decoded (decode-snapshot-rows frame-payload)] + (recur next-offset (into rows decoded))) + {:rows rows + :buffer (.slice payload offset total)})))))) + +(defn- finalize-framed-buffer + [buffer] + (if (or (nil? buffer) (zero? (.-byteLength buffer))) + [] + (let [{:keys [rows buffer]} (parse-framed-chunk nil buffer)] + (if (and (seq rows) (or (nil? buffer) (zero? (.-byteLength buffer)))) + rows + (fail-fast :db-sync/incomplete-snapshot-frame + {:rows (count rows) + :remaining-buffer-bytes (some-> buffer .-byteLength)}))))) + +(defn- gzip-payload? + [^js payload] + (and (some? payload) + (>= (.-byteLength payload) 2) + (= 31 (aget payload 0)) + (= 139 (aget payload 1)))) + +(defn- payload->stream + [^js payload] + (js/ReadableStream. + #js {:start (fn [controller] + (.enqueue controller payload) + (.close controller))})) + +(defn- stream payload) + decompressed (.pipeThrough stream (js/DecompressionStream. "gzip")) + resp (js/Response. decompressed) + array-buffer (.arrayBuffer resp)] + (->uint8 array-buffer)) + (p/rejected (ex-info "gzip decompression not supported" + {:type :db-sync/decompression-not-supported})))) + +(defn- uint8 array-buffer)] + (if (gzip-payload? payload) + (js (with-auth-headers {:method "GET"}))) + _ (when-not (.-ok resp) + (fail-fast :db-sync/snapshot-download-failed {:repo repo + :graph-id graph-id + :status (.-status resp)})) + payload ( diff --git a/src/main/frontend/worker/sync/crypt.cljs b/src/main/frontend/worker/sync/crypt.cljs index 2b1d65b31b..d4ed5cec12 100644 --- a/src/main/frontend/worker/sync/crypt.cljs +++ b/src/main/frontend/worker/sync/crypt.cljs @@ -1,7 +1,6 @@ (ns frontend.worker.sync.crypt "E2EE helpers for db-sync." - (:require ["/frontend/idbkv" :as idb-keyval] - [clojure.string :as string] + (:require [clojure.string :as string] [frontend.common.crypt :as crypt] [frontend.common.thread-api :refer [def-thread-api]] [frontend.worker-common.util :as worker-util] @@ -15,7 +14,6 @@ (defonce ^:private *graph->aes-key (atom {})) (defonce ^:private *user-rsa-key-pair-inflight (atom {})) -(defonce ^:private e2ee-store (delay (idb-keyval/newStore "localforage" "keyvaluepairs" 2))) (defonce ^:private e2ee-password-file "e2ee-password") (defonce ^:private native-env? (let [href (try (.. js/self -location -href) @@ -61,7 +59,8 @@ password)) (defn- auth-token [] - (worker-state/get-id-token)) + (or (worker-state/get-id-token) + (:auth-token @worker-state/*db-sync-config))) (defn- auth-headers [] (let [token (auth-token)] @@ -160,8 +159,8 @@ (defn- error ex-message) + data (ex-data error)] + (or (= :thread-api/decrypt-user-e2ee-private-key (:method data)) + (and (string? message) + (string/includes? message "main-thread is not available"))))) + + [] + (seq configured-password) (conj {:source :config + :value configured-password}) + (seq refresh-token) (conj {:source :saved-password + :value refresh-token}) + (and (seq auth-token) + (not= auth-token refresh-token)) + (conj {:source :saved-password + :value auth-token}))] + (letfn [( (p/let [password ( (decrypt-on-main-thread encrypted-private-key) + (p/catch (fn [error] + (if (main-thread-unavailable? error) + (graph repo) :import-type import-type :input input - :allow-missing-graph true}})) + :allow-missing-graph true + :require-missing-graph true}})) (defn execute-graph-list [_action config] diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs new file mode 100644 index 0000000000..8d6aa2602a --- /dev/null +++ b/src/main/logseq/cli/command/sync.cljs @@ -0,0 +1,273 @@ +(ns logseq.cli.command.sync + "Sync-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.config :as cli-config] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) + +(def ^:private sync-grant-access-spec + {:graph-id {:desc "Remote graph UUID"} + :email {:desc "Target user email"}}) + +(def entries + [(core/command-entry ["sync" "status"] :sync-status "Show db-sync runtime status" {}) + (core/command-entry ["sync" "start"] :sync-start "Start db-sync client" {}) + (core/command-entry ["sync" "stop"] :sync-stop "Stop db-sync client" {}) + (core/command-entry ["sync" "upload"] :sync-upload "Upload current graph snapshot" {}) + (core/command-entry ["sync" "download"] :sync-download "Download remote graph snapshot" {}) + (core/command-entry ["sync" "remote-graphs"] :sync-remote-graphs "List remote graphs" {}) + (core/command-entry ["sync" "ensure-keys"] :sync-ensure-keys "Ensure user RSA keys for sync/e2ee" {}) + (core/command-entry ["sync" "grant-access"] :sync-grant-access "Grant graph access to an email" sync-grant-access-spec) + (core/command-entry ["sync" "config" "set"] :sync-config-set "Set sync config key" {}) + (core/command-entry ["sync" "config" "get"] :sync-config-get "Get sync config key" {}) + (core/command-entry ["sync" "config" "unset"] :sync-config-unset "Unset sync config key" {})]) + +(def ^:private config-key-map + {"ws-url" :ws-url + "http-base" :http-base + "auth-token" :auth-token + "e2ee-password" :e2ee-password}) + +(defn- missing-repo + [label] + {:ok? false + :error {:code :missing-repo + :message (str "repo is required for " label)}}) + +(defn- invalid-options + [message] + {:ok? false + :error {:code :invalid-options + :message message}}) + +(defn- parse-config-key + [raw-key] + (let [raw-key (some-> raw-key string/trim string/lower-case) + config-key (get config-key-map raw-key)] + (if config-key + {:ok? true + :key config-key} + (invalid-options (str "unknown config key: " raw-key))))) + +(defn build-action + [command options args repo] + (case command + (:sync-status :sync-start :sync-stop :sync-upload) + (if-not (seq repo) + (missing-repo (name command)) + {:ok? true + :action {:type command + :repo repo + :graph (core/repo->graph repo)}}) + + :sync-download + (if-not (seq repo) + (missing-repo (name command)) + {:ok? true + :action {:type :sync-download + :repo repo + :graph (core/repo->graph repo) + :allow-missing-graph true + :require-missing-graph true}}) + + :sync-remote-graphs + {:ok? true + :action {:type :sync-remote-graphs}} + + :sync-ensure-keys + {:ok? true + :action {:type :sync-ensure-keys}} + + :sync-grant-access + (if-not (seq repo) + (missing-repo "sync grant-access") + (let [graph-id (some-> (:graph-id options) string/trim) + email (some-> (:email options) string/trim)] + (cond + (not (seq graph-id)) + (invalid-options "--graph-id is required") + + (not (seq email)) + (invalid-options "--email is required") + + :else + {:ok? true + :action {:type :sync-grant-access + :repo repo + :graph (core/repo->graph repo) + :graph-id graph-id + :email email}}))) + + :sync-config-get + (let [[name] args + key-result (parse-config-key name)] + (if-not (seq (some-> name string/trim)) + (invalid-options "config key is required") + (if-not (:ok? key-result) + key-result + {:ok? true + :action {:type :sync-config-get + :config-key (:key key-result)}}))) + + :sync-config-set + (let [[name value] args + key-result (parse-config-key name)] + (cond + (not (seq (some-> name string/trim))) + (invalid-options "config key is required") + + (not (seq (some-> value str string/trim))) + (invalid-options "config value is required") + + (not (:ok? key-result)) + key-result + + :else + {:ok? true + :action {:type :sync-config-set + :config-key (:key key-result) + :config-value value}})) + + :sync-config-unset + (let [[name] args + key-result (parse-config-key name)] + (cond + (not (seq (some-> name string/trim))) + (invalid-options "config key is required") + + (not (:ok? key-result)) + key-result + + :else + {:ok? true + :action {:type :sync-config-unset + :config-key (:key key-result)}})) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown sync command: " command)}})) + +(defn- invoke-with-repo + [config repo method args] + (let [sync-config {:ws-url (:ws-url config) + :http-base (:http-base config) + :auth-token (:auth-token config) + :e2ee-password (:e2ee-password config)}] + (p/let [cfg (cli-server/ensure-server! config repo) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config]) + result (transport/invoke cfg method false args)] + result))) + +(defn- invoke-global + [config method args] + (let [base-url (:base-url config) + sync-config {:ws-url (:ws-url config) + :http-base (:http-base config) + :auth-token (:auth-token config) + :e2ee-password (:e2ee-password config)}] + (if (seq base-url) + (p/let [_ (transport/invoke config :thread-api/set-db-sync-config false [sync-config])] + (transport/invoke config method false args)) + (p/let [repo (or (core/resolve-repo (:graph config)) + (p/let [graphs (cli-server/list-graphs config)] + (some-> graphs first core/resolve-repo))) + cfg (if (seq repo) + (cli-server/ensure-server! config repo) + (p/rejected (ex-info "graph name is required" + {:code :missing-graph}))) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config])] + (transport/invoke cfg method false args))))) + +(defn execute + [action config] + (case (:type action) + :sync-status + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-status + [(:repo action)])] + {:status :ok + :data result}) + + :sync-start + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-start + [(:repo action)])] + {:status :ok + :data {:result result}}) + + :sync-stop + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-stop + [])] + {:status :ok + :data {:result result}}) + + :sync-upload + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-upload-graph + [(:repo action)])] + {:status :ok + :data (if (map? result) + result + {:result result})}) + + :sync-download + (p/let [remote-graphs (invoke-global config + :thread-api/db-sync-list-remote-graphs + []) + remote-graph (some (fn [graph] + (when (= (:graph action) (:graph-name graph)) + graph)) + remote-graphs)] + (if-not remote-graph + {:status :error + :error {:code :remote-graph-not-found + :message (str "remote graph not found: " (:graph action)) + :graph (:graph action)}} + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-download-graph-by-id + [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] + {:status :ok + :data (if (map? result) + result + {:result result})}))) + + :sync-remote-graphs + (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] + {:status :ok + :data {:graphs (or graphs [])}}) + + :sync-ensure-keys + (p/let [result (invoke-global config :thread-api/db-sync-ensure-user-rsa-keys [])] + {:status :ok + :data {:result result}}) + + :sync-grant-access + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-grant-graph-access + [(:repo action) (:graph-id action) (:email action)])] + {:status :ok + :data {:result result}}) + + :sync-config-get + (p/let [current config] + {:status :ok + :data {:key (:config-key action) + :value (get (or current {}) (:config-key action))}}) + + :sync-config-set + (p/let [_ (cli-config/update-config! config {(:config-key action) (:config-value action)})] + {:status :ok + :data {:key (:config-key action) + :value (:config-value action)}}) + + :sync-config-unset + (p/let [_ (cli-config/update-config! config {(:config-key action) nil})] + {:status :ok + :data {:key (:config-key action)}}) + + (p/resolved {:status :error + :error {:code :unknown-action + :message "unknown sync action"}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index c427971775..70c1b0dd58 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -10,6 +10,7 @@ [logseq.cli.command.remove :as remove-command] [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] + [logseq.cli.command.sync :as sync-command] [logseq.cli.command.upsert :as upsert-command] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -101,7 +102,8 @@ remove-command/entries query-command/entries show-command/entries - doctor-command/entries))) + doctor-command/entries + sync-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -267,6 +269,10 @@ (not (seq (:graph opts)))) (missing-graph-result summary) + (and (= command :sync-download) + (not (seq (:graph opts)))) + (missing-graph-result summary) + :else (command-core/ok-result command opts args summary)))) @@ -284,7 +290,7 @@ (empty? args) (command-core/help-result summary) - (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query"} (first args))) + (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query" "sync"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) :else @@ -328,7 +334,9 @@ (defn- ensure-missing-graph [action config] - (if (and (= :graph-import (:type action)) (:repo action)) + (if (and (:repo action) + (or (:require-missing-graph action) + (= :graph-import (:type action)))) (p/let [graphs (cli-server/list-graphs config) graph (command-core/repo->graph (:repo action))] (if (some #(= graph %) graphs) @@ -406,6 +414,11 @@ :doctor (doctor-command/build-action options) + (:sync-status :sync-start :sync-stop :sync-upload :sync-download + :sync-remote-graphs :sync-ensure-keys :sync-grant-access + :sync-config-set :sync-config-get :sync-config-unset) + (sync-command/build-action command options args repo) + {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) @@ -451,6 +464,10 @@ :server-start (server-command/execute-start action config) :server-stop (server-command/execute-stop action config) :server-restart (server-command/execute-restart action config) + (:sync-status :sync-start :sync-stop :sync-upload :sync-download + :sync-remote-graphs :sync-ensure-keys :sync-grant-access + :sync-config-set :sync-config-get :sync-config-unset) + (sync-command/execute action config) {:status :error :error {:code :unknown-action :message "unknown action"}}))] @@ -460,4 +477,5 @@ :schema :source :target :update-tags :update-properties :remove-tags :remove-properties - :export-type :file :import-type :input]))))) + :export-type :file :import-type :input + :graph-id :email :config-key :config-value]))))) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index f27152b557..e050e113f3 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -45,9 +45,16 @@ [{:keys [config-path]} updates] (let [path (or config-path (default-config-path)) current (or (read-config-file path) {}) - filtered-current (dissoc current :auth-token :retries) - filtered-updates (dissoc (or updates {}) :auth-token :retries) - next (merge filtered-current filtered-updates)] + filtered-current (dissoc current :retries) + filtered-updates (dissoc (or updates {}) :retries) + nil-keys (->> filtered-updates + (keep (fn [[k v]] + (when (nil? v) + k)))) + merged (merge filtered-current filtered-updates) + next (if (seq nil-keys) + (apply dissoc merged nil-keys) + merged)] (ensure-config-dir! path) (.writeFileSync fs path (pr-str next)) next)) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index fb8ffd0da4..14dd44e5e0 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -280,6 +280,60 @@ (cond-> [(str "Server " (name status) ": " repo)] (and host port) (conj (str "Host: " host " Port: " port)))))) +(def ^:private redacted-token "[REDACTED]") + +(defn- format-sync-status + [{:keys [repo graph-id ws-state pending-local pending-asset pending-server local-tx remote-tx]}] + (string/join "\n" + [(str "Sync status") + (str "repo: " (or repo "-")) + (str "graph-id: " (or graph-id "-")) + (str "ws-state: " (or ws-state :unknown)) + (str "pending-local: " (or pending-local 0)) + (str "pending-asset: " (or pending-asset 0)) + (str "pending-server: " (or pending-server 0)) + (str "local-tx: " (or local-tx "-")) + (str "remote-tx: " (or remote-tx "-"))])) + +(defn- format-sync-remote-graphs + [graphs] + (format-counted-table + ["GRAPH-ID" "GRAPH-NAME" "ROLE" "E2EE"] + (mapv (fn [{:keys [graph-id graph-name role graph-e2ee?]}] + [graph-id + graph-name + (or role "-") + (if (nil? graph-e2ee?) + "-" + (if graph-e2ee? "true" "false"))]) + (or graphs [])))) + +(defn- format-sync-action + [command {:keys [repo email]}] + (case command + :sync-start (str "Sync started: " repo) + :sync-stop (str "Sync stopped: " repo) + :sync-upload (str "Sync upload requested: " repo) + :sync-download (str "Sync download requested: " repo) + :sync-ensure-keys "Sync keys ensured" + :sync-grant-access (str "Sync access granted: " email " (repo: " repo ")") + "Sync updated")) + +(defn- format-sync-config-get + [{:keys [key value]}] + (let [display-value (if (contains? #{:auth-token :e2ee-password} key) + redacted-token + (if (some? value) value "-"))] + (str "sync config " (name key) ": " display-value))) + +(defn- format-sync-config-set + [{:keys [key]}] + (str "sync config set: " (name key))) + +(defn- format-sync-config-unset + [{:keys [key]}] + (str "sync config unset: " (name key))) + (defn- format-upsert-block [{:keys [repo source target update-tags update-properties remove-tags remove-properties]} result] (if (vector? result) @@ -375,6 +429,13 @@ :server-status (format-server-status data) (:server-start :server-stop :server-restart) (format-server-action command data) + :sync-status (format-sync-status data) + :sync-remote-graphs (format-sync-remote-graphs (:graphs data)) + (:sync-start :sync-stop :sync-upload :sync-download :sync-ensure-keys :sync-grant-access) + (format-sync-action command context) + :sync-config-get (format-sync-config-get data) + :sync-config-set (format-sync-config-set data) + :sync-config-unset (format-sync-config-unset data) :list-page (format-list-page (:items data) now-ms) :list-tag (format-list-tag (:items data) now-ms) :list-property (format-list-property (:items data) now-ms) diff --git a/src/test/frontend/worker/db_core_test.cljs b/src/test/frontend/worker/db_core_test.cljs index bb1462bb81..e1d616adfd 100644 --- a/src/test/frontend/worker/db_core_test.cljs +++ b/src/test/frontend/worker/db_core_test.cljs @@ -1,7 +1,9 @@ (ns frontend.worker.db-core-test - (:require [cljs.test :refer [deftest is]] + (:require [cljs.test :refer [async deftest is]] [frontend.common.thread-api :as thread-api] - [frontend.worker.db-core])) + [frontend.worker.db-core :as db-core] + [frontend.worker.shared-service :as shared-service] + [promesa.core :as p])) (deftest db-core-registers-db-sync-thread-apis (let [api-map @thread-api/*thread-apis] @@ -14,3 +16,29 @@ (is (contains? api-map :thread-api/db-sync-ensure-user-rsa-keys)) (is (contains? api-map :thread-api/db-sync-upload-graph)) (is (contains? api-map :thread-api/db-sync-import-kvs-rows)))) + +(deftest init-service-does-not-close-db-when-graph-unchanged + (async done + (let [service {:status {:ready (p/resolved true)} + :proxy #js {}} + close-calls (atom []) + create-calls (atom 0) + *service @#'db-core/*service + old-service @*service] + (reset! *service ["graph-a" service]) + (with-redefs [db-core/close-db! (fn [repo] + (swap! close-calls conj repo) + nil) + shared-service/ (#'db-core/ (stop!) (p/finally (fn [] (done)))) (done)))))))) +(deftest db-worker-node-sync-status-requires-repo-and-returns-structured-status + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-sync-status") + repo (str "logseq_db_sync_status_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + {:keys [status body]} (invoke-raw host port "thread-api/db-sync-status" []) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true) + _ (is (= 400 status)) + _ (is (= false (:ok parsed))) + _ (is (= "missing-repo" (get-in parsed [:error :code]))) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + status-result (invoke host port "thread-api/db-sync-status" [repo])] + (is (= repo (:repo status-result))) + (is (contains? status-result :ws-state)) + (is (contains? status-result :pending-local)) + (is (contains? status-result :pending-asset)) + (is (contains? status-result :pending-server)) + (is (contains? status-result :local-tx)) + (is (contains? status-result :remote-tx)) + (is (contains? status-result :graph-id))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) diff --git a/src/test/frontend/worker/sync/crypt_test.cljs b/src/test/frontend/worker/sync/crypt_test.cljs index 611306fb04..f3e3c04820 100644 --- a/src/test/frontend/worker/sync/crypt_test.cljs +++ b/src/test/frontend/worker/sync/crypt_test.cljs @@ -166,6 +166,89 @@ (set! js/fetch fetch-prev) (done))))))) +(deftest fetch-graph-aes-key-for-download-uses-platform-kv-clear-test + (async done + (let [fetch-prev js/fetch + graph-id (str (random-uuid)) + expected-key (str "rtc-encrypted-aes-key###" graph-id) + platform-map {:runtime :test} + kv-set-calls (atom [])] + (set! js/fetch + (fn [url _opts] + (cond + (string/includes? url "/e2ee/user-keys") + (js/Promise.resolve + #js {:ok true + :text (fn [] + (js/Promise.resolve + "{\"public-key\":\"public-key\",\"encrypted-private-key\":\"encrypted-private-key\"}"))}) + + (string/includes? url (str "/e2ee/graphs/" graph-id "/aes-key")) + (js/Promise.resolve + #js {:ok true + :text (fn [] + (js/Promise.resolve + "{\"encrypted-aes-key\":\"remote-encrypted\"}"))}) + + :else + (js/Promise.resolve + #js {:ok false + :status 404 + :text (fn [] (js/Promise.resolve "{\"message\":\"not-found\"}"))})))) + (-> (p/with-redefs [sync-crypt/e2ee-base (fn [] "https://example.com") + worker-state/get-id-token (fn [] "token") + worker-state/ (p/with-redefs [ldb/read-transit-str (fn [_] :encrypted-private-key) + worker-state/ (p/let [_ (sync-command/execute {:type :sync-start + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-start false ["logseq_db_demo"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-stop + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-stop + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-stop false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-upload + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-upload-graph false ["logseq_db_demo"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []] + [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download-uses-graph-config-when-base-url-missing + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? false}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:graph "demo" + :data-dir "/tmp"})] + (is (= [[{:graph "demo" + :data-dir "/tmp"} + "logseq_db_demo"] + [{:graph "demo" + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []] + [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download-remote-graph-not-found + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "other-id" + :graph-name "other-graph"}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :remote-graph-not-found (get-in result [:error :code]))) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-remote-graphs + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved []))) + (-> (p/let [_ (sync-command/execute {:type :sync-remote-graphs} + {:base-url "http://example" + :http-base "https://sync.example.com" + :ws-url "wss://sync.example.com/sync/%s" + :auth-token "test-token" + :e2ee-password "pw" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url "wss://sync.example.com/sync/%s" + :http-base "https://sync.example.com" + :auth-token "test-token" + :e2ee-password "pw"}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-ensure-keys + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-ensure-keys} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-ensure-user-rsa-keys false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-grant-access + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-grant-access + :repo "logseq_db_demo" + :graph-id "graph-uuid" + :email "user@example.com"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-config-get + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-get + :config-key :auth-token} + {:base-url "http://example" + :auth-token "abc" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-config-set + (async done + (let [orig-invoke transport/invoke + orig-update-config! cli-config/update-config! + invoke-calls (atom []) + update-calls (atom [])] + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil)) + ) + (set! cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (merge {:ws-url "wss://old.example/sync/%s"} updates))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-set + :config-key :auth-token + :config-value "token-value"} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token "token-value"}]] + @update-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (set! cli-config/update-config! orig-update-config!) + (done))))))) + +(deftest test-execute-sync-config-unset + (async done + (let [orig-invoke transport/invoke + orig-update-config! cli-config/update-config! + invoke-calls (atom []) + update-calls (atom [])] + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil))) + (set! cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (dissoc {:ws-url "wss://old.example/sync/%s" + :auth-token "token-value"} + :auth-token))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-unset + :config-key :auth-token} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token nil}]] + @update-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (set! cli-config/update-config! orig-update-config!) + (done))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index aee1a906c0..44eacce467 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -66,6 +66,7 @@ (is (string/includes? plain-summary "doctor")) (is (string/includes? plain-summary "graph")) (is (string/includes? plain-summary "server")) + (is (string/includes? plain-summary "sync")) (is (string/includes? plain-summary "Path to db-worker data dir (default ~/logseq/graphs)")) (is (contains-bold? summary "list page")) (is (contains-bold? summary "list tag")) @@ -86,6 +87,8 @@ (is (contains-bold? summary "graph create")) (is (contains-bold? summary "server list")) (is (contains-bold? summary "server start")) + (is (contains-bold? summary "sync status")) + (is (contains-bold? summary "sync start")) (is (contains-bold? summary "--help")) (is (contains-bold? summary "--graph")) (is (re-find #"\u001b\[[0-9;]*mCommands\u001b\[[0-9;]*m:" summary)) @@ -205,6 +208,28 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-help-sync-group + (testing "sync group shows subcommands" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["sync"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "sync status")) + (is (string/includes? plain-summary "sync start")) + (is (string/includes? plain-summary "sync stop")) + (is (string/includes? plain-summary "sync upload")) + (is (string/includes? plain-summary "sync download")) + (is (string/includes? plain-summary "sync remote-graphs")) + (is (string/includes? plain-summary "sync ensure-keys")) + (is (string/includes? plain-summary "sync grant-access")) + (is (string/includes? plain-summary "sync config set")) + (is (string/includes? plain-summary "sync config get")) + (is (string/includes? plain-summary "sync config unset")) + (is (contains-bold? summary "sync status")) + (is (contains-bold? summary "sync config set")) + (is (contains-bold? summary "sync grant-access"))))) + (deftest test-parse-args-help-upsert-group (testing "add group is removed" (let [result (commands/parse-args ["add"])] @@ -1217,6 +1242,11 @@ (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) + (testing "sync download requires graph" + (let [result (commands/parse-args ["sync" "download"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + (testing "graph import rejects unknown type" (let [result (commands/parse-args ["graph" "import" "--type" "zip" @@ -2460,6 +2490,28 @@ (set! cli-server/ensure-server! orig-ensure-server!) (done))))))) +(deftest test-execute-sync-download-rejects-existing-graph + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server!] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))) + (-> (p/let [result (commands/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :allow-missing-graph true + :require-missing-graph true} + {})] + (is (= :error (:status result))) + (is (= :graph-exists (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (done))))))) + (deftest test-execute-graph-export (async done (let [invoke-calls (atom []) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 6c0e91623b..1c461500bf 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -104,5 +104,16 @@ contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] (is (= "new" (:graph parsed))) - (is (not (contains? parsed :auth-token))) + (is (= "secret" (:auth-token parsed))) (is (not (contains? parsed :retries))))) + +(deftest test-update-config-removes-nil-values + (let [dir (node-helper/create-tmp-dir "cli") + cfg-path (node-path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:graph \"old\" :auth-token \"secret\"}") + _ (config/update-config! {:config-path cfg-path} + {:auth-token nil}) + contents (.toString (fs/readFileSync cfg-path) "utf8") + parsed (reader/read-string contents)] + (is (= "old" (:graph parsed))) + (is (not (contains? parsed :auth-token))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 6189b2d7ef..38b3fb6d71 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -235,6 +235,96 @@ {:output-format nil})] (is (= "Imported sqlite from /tmp/import.sqlite" result))))) +(deftest test-human-output-sync-status + (testing "sync status renders runtime and queue fields" + (let [result (format/format-result {:status :ok + :command :sync-status + :data {:repo "demo-graph" + :graph-id "graph-uuid" + :ws-state :open + :pending-local 2 + :pending-asset 1 + :pending-server 3 + :local-tx 10 + :remote-tx 13}} + {:output-format nil})] + (is (string/includes? result "Sync status")) + (is (string/includes? result "demo-graph")) + (is (string/includes? result "graph-uuid")) + (is (string/includes? result "pending-local")) + (is (string/includes? result "pending-asset")) + (is (string/includes? result "pending-server"))))) + +(deftest test-human-output-sync-remote-graphs + (testing "sync remote-graphs renders a table" + (let [result (format/format-result {:status :ok + :command :sync-remote-graphs + :data {:graphs [{:graph-id "graph-1" + :graph-name "Alpha" + :role "manager" + :graph-e2ee? true} + {:graph-id "graph-2" + :graph-name "Beta" + :role "member" + :graph-e2ee? false}]}} + {:output-format nil})] + (is (string/includes? result "GRAPH-ID")) + (is (string/includes? result "GRAPH-NAME")) + (is (string/includes? result "ROLE")) + (is (string/includes? result "Alpha")) + (is (string/includes? result "Beta")) + (is (string/includes? result "Count: 2"))))) + +(deftest test-human-output-sync-actions + (testing "sync start renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-start + :context {:repo "demo-graph"}} + {:output-format nil})] + (is (string/includes? result "Sync started")) + (is (string/includes? result "demo-graph")))) + + (testing "sync upload renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-upload + :context {:repo "demo-graph"} + :data {:graph-id "graph-uuid"}} + {:output-format nil})] + (is (string/includes? result "Sync upload")) + (is (string/includes? result "demo-graph")))) + + (testing "sync download renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-download + :context {:repo "demo-graph"}} + {:output-format nil})] + (is (string/includes? result "Sync download")) + (is (string/includes? result "demo-graph"))))) + +(deftest test-human-output-sync-config-get-token-redaction + (testing "sync config get auth-token redacts value in human output" + (let [token "super-secret-token-value" + result (format/format-result {:status :ok + :command :sync-config-get + :data {:key :auth-token + :value token}} + {:output-format nil})] + (is (string/includes? result "auth-token")) + (is (string/includes? result "[REDACTED]")) + (is (not (string/includes? result token)))))) + +(deftest test-human-output-sync-config-get-e2ee-password-redaction + (testing "sync config get e2ee-password redacts value in human output" + (let [password "super-secret-password" + result (format/format-result {:status :ok + :command :sync-config-get + :data {:key :e2ee-password + :value password}} + {:output-format nil})] + (is (string/includes? result "e2ee-password")) + (is (string/includes? result "[REDACTED]")) + (is (not (string/includes? result password)))))) + (deftest test-human-output-graph-info (testing "graph info includes key metadata lines" (let [result (format/format-result {:status :ok